Chance for Valuable Insights Into Safety Issues Lost

Tue, May 13, 2008 — David Evans


Premature burial of an air-safety survey means an opportunity is being lost to better understand safety trends at a time when the Federal Aviation Administration (FAA) is being accused in Congress of being too cozy with the airlines it is required to regulate.

Although not expressed this bluntly, that is the central conclusion that can be drawn from a 31 March 2008 National Air & Space Administration Inspector General (NASA/IG) audit of a research project that interviewed thousands of pilots and produced results suggesting safety problems were worse than indicated by the FAA’s database.

The audit was of the National Aviation Operations Monitoring Service (NAOMS), a NASA project to compile safety-relevant trends through structured interviews with thousands of pilots and, eventually, similar interviews with flight attendants, mechanics and air traffic controllers. Before the effort could be expanded beyond pilots, it was shut down, despite the interest of the Science & Technology Committee in the U.S. House of Representatives, which held a hearing on the abortive NAOMS project 31 October 2007 (see Air Accident Digest, November 2007).

The NASA/IG found:

“The Government may have missed an opportunity to foster a deeper understanding of the aviation safety environment from 2001 through 2004 because its working groups were unable to reach a consensus on the validity or value of the NAOMS data. As a result, NASA was reluctant to publish a report detailing research and conclusions garnered from the collected NAOMS survey data.”

Last fall the Associated Press was denied a Freedom of Information Act (FOIA) request for release of the survey data, creating controversy in the press that led to the Congressional hearing. At the hearing, NASA Administrator Michael Griffin criticized the NAOMS effort as flawed, primarily because its survey methodology had not been peer reviewed. Those experts closely involved in the study claimed, on the contrary, that the NAOMS effort had been peer reviewed every step of the way and that its results were eminently valid.

The FAA was concerned that the survey results could give the impression to the general public that aviation was not as safe as presumed. For instance, NAOMS indicated a higher rate of runway incidents and incursions than recorded by the FAA database or by the Aviation Safety Reporting System (ASRS).

ASRS is a confidential reporting system maintained by NASA on behalf of the FAA. Through ASRS, a pilot can confidentially report a safety violation without fear of retribution.

As the NASA/IG reported:

“Because ASRS is a voluntary reporting system and not suitable for comparative statistical analysis of changes in the safety of the National Aviation System … NAOMS [would] complement and enhance the ASRS information. NAOMS would include survey data from pilots, mechanics, air traffic controllers, and others, which could provide insights into the performance and safety of the National Aviation System. By routinely evaluating the data, decision makers of the aviation community could quantitatively measure safety, assess trends, identify factors driving those trends, and evaluate the effects of new technologies and procedures introduced into the National Aviation System.”

Interviews with approximately 25,000 airline pilots from 2001-2004 demonstrated that the survey was sound and would provide useful input for trend assessment. Because ASRS requires a voluntary report, it is not useful for statistically valid trend analysis, which is why the NAOMS effort was seen as a necessary complement to ASRS. NAOMS would also tend to provide higher rates than reported by ASRS, which depended on the initiative of the individual reporter, whereas NAOMS was based on a statistically valid sampling of all reporters within the system. As indicated in the 2002 presentation slide to NASA’s Aviation Safety & Security Program Office, NAOMS data indicated event rates greater than that of other aviation safety databases:


Not surprisingly, the FAA was concerned with the results of the survey and the data presented, expressing a lack of confidence in the survey methodology due to the event rates it produced.

In other words, any data that showed the system was not as safe as proclaimed was suspect. However, not all NAOMS event rates showed negative trends, as indicated in this 2003 graph:


Despite the fact that the NAOMS project meets the FAA’s test of “data-driven” safety, it was stopped before expansion to include air traffic controllers, who certainly could provide additional insights into runway incursions. And including mechanics in the survey protocol would provide additional revelations concerning aircraft maintenance, the subject of recent airworthiness directive-related groundings and allegations that FAA oversight of airline maintenance was lacking.

Apparently, if a system is not operational to report negative findings, threats to aviation safety do not exist. It is easy to dismiss ASRS reports as anecdotal and not reflective of a system wide problem. Not so with NAOMS.

NASA’s Associate Administrator for Aeronautics Research, Jaiwon Shin, wrote in response to the IG report that the interviews, which were stopped at the end of 2004, are less relevant with the passage of time.

As the IG noted, “The Associate Administrator nonconcurred with our recommendation to publish a detailed report analyzing the NAOMS research …”

The IG apparently is reluctantly willing to let the matter lay there. In fact, the IG arrived at a fairly insipid conclusion, that the survey methodology may be more valuable than “the data itself” for evaluating the current aviation safety environment. This is like saying, “We like the way Einstein went about developing the theory of relativity but we find that we are unable to validate or support his findings.”

Barring a Congressional demand, and funding, to resurrect the NAOMS program, its opportunity to advance the state of knowledge about what works, and what doesn’t, to make aviation safer lies under a tombstone. On that tombstone are inscribed the following words: “Here lies aviation safety, killed by bureaucrats who feared data-driven embarrassment.”

(The full NASA/IG report may be viewed at

Comments are closed.

Nolan Law Group