Workstations for Hong Kong’s troubled air traffic system have required more frequent reboots than recommended by its manufacturer, according to an official user guide obtained by FactWire.
FactWire revealed in May that the Civil Aviation Department (CAD) had started performing rotational restarts fortnightly on 47 air traffic workstations and weekly on six other workstations since February, in a bid to prevent computer crashes and sluggish performance of the system.
But the efforts were apparently in vain after it emerged earlier this month that glitches had intensified since April, prompting additional reboots during peak hours. CAD had previously said that regular reboots were part of its “routine maintenance procedure” recommended by the system’s manufacturer Raytheon.
However, a copy of AutoTrac III System Administrator’s Guide written by Raytheon has now come to light, detailing in about 300 pages steps and procedures on how to operate and service the air traffic management system.
According to Section 2.8 Housekeeping, restarts of workstation software and workstation reboot are included in the 11 preventive and general maintenance procedures, along with other measures such as periodic archiving, purging of log files and directed switchover.
But restarts are only recommended to take place every 30 days as part of regular maintenance of the system and should only be performed during non-busy hours to minimise potential side effects.
Recommendations in the administrator’s guide run counter to the current practice adopted by CAD, which had admitted to rotationally rebooting 47 workstations every fortnight and six other workstations every week.
Since late April, CAD has also been restarting some workstations and their software during peak hours from 7am to midnight despite the manufacturer’s guidelines, although the department had said that additional reboots only occurred during period of “relatively low air traffic flow.”
In a statement on November 9, CAD said: “Major overseas ATCCs (air traffic control centres) also in general regularly reboot individual workstations based on their needs, which is part of their routine housekeeping procedures.” The department also told FactWire earlier in June that airports in Singapore, Shanghai and Beijing would restart certain air traffic workstations every fortnight when required.
However, information shows that all three airports CAD raised as examples use air traffic management systems developed by French company Thales, instead of Raytheon, which made Hong Kong’s AT3.
The Civil Aviation Authority of Singapore did not respond to a request for comment, but a source knowledgeable of its operations said although some air traffic workstations at Changi Airport did require regular reboots after commissioning of a new air traffic management system due to performance issues, the practice was temporary and only lasted four to five months.
Restarting some workstations as necessary is also different to CAD’s current practice of regularly rebooting all workstations, he said. According to sources, workstation rebooting was originally adopted by CAD as a temporary measure for use in non-busy hours.
After six incidents of sluggish performance since the system’s full commissioning last November, Raytheon at the time suggested rebooting some workstations as a preventive maintenance measure.
The first reboot took place on 26 November last year to prevent sluggish performance, which was followed by preventive reboots of more than ten other workstations on 2 December after another four cases of sluggishness. The practice continued on 26 December, 24 and 26 January when one, four and seven workstations were restarted respectively.
Reboots then became part of a routine maintenance procedure after CAD drew up a schedule in February for rebooting all workstations rotationally every fortnight – a practice that has continued ever since.
Sources said the performance of the workstations did improve for a while, but glitches like sluggishness, frozen screen and unresponsiveness to commands resurfaced in April, causing additional reboots, with most happening in peak hours (see table below).
In response to FactWire’s revelation on 9 November that glitches had caused about 20 reboots between April and October as well as of the details of 14 incidents, CAD denied any occurrence of “system crash,” but said “there were five confirmed cases of slight sluggishness of the screen display.”
It added that radar screens did occasionally experience sluggishness while mice and keyboards were not responsive to command, but insisted some cases FactWire revealed – without specifying which – did not happen according to the department’s record.
Frontline air traffic controllers said CAD had “conned the public with their wording trick” by narrowly defining “system crash’ as “frozen radar screen or suspension of operation” and disregarding situations such as unresponsive screen or sluggish performance.
A source also confirmed that all 14 incidents in the summary were reported to the department after maintenance staff inspected and rebooted the problematic workstations, but since the system failed to record some of the cases, CAD regarded those cases as “unsubstantiated” and did not refer them to Raytheon for further investigations.
Routine maintenance normally includes measures such as archiving and inspecting of hard disks which would not interfere with operations, according to the source, while it is also rare for other air traffic management systems to adopt monthly rebooting of workstations as a permanent maintenance procedure.
“Bi-weekly restarting of every workstation is just a very desperate way of doing preventive maintenance,” he said. “But if even with such preventive measures, the problem still surfaced, the system’s reliability is questionable.”
