Analysis of the Organization Designation Authorization in Aircraft Certification: Differences to the Brazilian and European Approaches

ABSTRACT

This paper aims the identification of the Organization Designation Authorization (ODA) regulatory approaches adopted by the United States, using the Boeing 737 MAX 8 certification process as a basis, by revising the ODA’s certification role and what can be enhanced on the ODA program, considering the data from the Federal Aviation Administration (FAA), the U.S. Department of Transportation, AND the Office of Inspector General (OIG), and comparing them to the regulatory basis in Europe and in Brazil. An assessment of undue pressure on ODA Unit members and other aspects, such as conflicting restraints, analysing their consequences and roles for the process’ safety, is presented, as well as a comparison with the independent design assurance system adopted by the other two agencies. Therefore, the discussion around the ODA’s aircraft certification processes highlights the ODA model drawbacks, and what can be learned from other models to improve it, so the industry and the society can fully benefit from this system.

Keywords
Delegation; Organization Designation Authorization; Aircraft certification; Design organization; Level of involvement

INTRODUCTION

After the Boeing 737 MAX 8 fatal accidents in 2018 and 2019, questions were raised about the American Federal Aviation Administration (FAA) certification process, especially the delegation process (USA 2020a).

In 2005, FAA created the Organization Designation Authorization (ODA) program through an amendment to 14 CFR Part 183 (FAA 2021b). This regulation defines the requirements for the organizations that want to be recognised as a designee and specifies the authorized functions once the applicant meet all the criteria and have been approved as an ODA (FAA 2011).

In summary, the ODA allows an organization to perform specified functions on behalf of the FAA on subjects related to engineering, manufacturing, operations, airworthiness, and maintenance, according to the FAA Order 8100.15B (FAA 2018).

While, up to now, it is not clear the role of the ODA in those accidents, it is also possible to notice that the delegation approach is not used by Brazilian (ANAC 2021) and European (EASA 2021) authorities, which implemented a different strategy in the certification process performed through design organizations. Therefore, the present paper will evaluate the FAA-ODA process and its differences to the European Union Aviation Safety Agency (EASA) and Brazilian Agência Nacional de Aviação Civil (ANAC) approaches, in order to understand how those differences could influence the certification process of the 737 MAX 8, particularly the overall function of Boeing ODA during this process, analysing the certification process itself, the delegation system and its structure, and the aircraft’s development and its non-usual features, such as the Manoeuvring Characteristics Augmentation System (MCAS) and the FAA oversight structure.

METHODS

This paper is an exploratory work, in which an analysis of reports, such as the United States Department of Transportation (U.S. DOT), the Office of the Inspector General (OIG) reports about the development of the 737 MAX 8, surveys conducted by The MITRE Corporation within the FAA, the pertinent regulatory standards from the FAA, EASA, ANAC and the International Civil Aviation Organization (ICAO), other papers and books, is performed.

RESULTS

The Certification Process in U.S.

The U.S. certification process is regulated by the regulations on 14 CFR Part 21 (FAA 2021a) and the guidance is provided by several guidance material, remarkably FAA Orders 8110.48A (2017a) and 8110.4C (2017b). Following these guidelines, section 21.21 of 14 CFR Part 21 (2021a) determines that FAA shall issue a document of approval stating that a specific aircraft model is compliant with applicable regulations, and that no feature or characteristic makes it unsafe for the category in which certification is requested.

Per 14 CFR Part 21 (FAA 2021a), the design approval can address a completely new aircraft design, which requires the emission of a type certificate (TC), or a modification of an approved design, which can be an amendment to the TC (ATC), or a supplemental TC (STC). The first one is applicable only when the change to the TC is made by the TC holder, while the STC can be approved either by the TC holder, or by a third party (FAA 2011). Considering that the case that motivated the questions on the ODA process is a modification performed by the TC holder, the focus of this study will be on the ATC process, and any reference to change or modification of the design will refer to an ATC unless otherwise specified.

The main difference between a TC and an ATC, according to OIG (2021), is that under the ATC only areas that present significant changes in design need to be brought up to current airworthiness requirements, following the section 21.101 of the 14 CFR Part 21 (FAA 2021a), the changed product rule. There are, nonetheless, exceptions that can be applied and the applicant can comply with earlier requirements if, according to section 21.101 of the 14 CFR Part 21, an area, system, component, equipment, or appliance are not affected by the change, if compliance with a later amendment does not materially improve safety, or if compliance with the latest amendment is impractical.

Another aspect established by the section 21.101 of the 14 CFR Part 21 is the time span of five years to complete the certification process from the moment the applicant files for the ATC (FAA 2021a). If the design is approved during this time range, the applicant must comply with the airworthiness requirement in effect on the date of the application for the change, and with the environment protection regulations.

The FAA ODA Program

As described before, in 2005, FAA (2021b) created the ODA program through an amendment to 14 CFR Part 183 to standardize and consolidate these delegations under a single initiative for manufacturers of a product or articles produced under a TC.

Through the ODA, FAA may delegate a substantial amount of critical work during the certification process, respecting the assumptions on 49 U.S.C. 44702(d) (USA 2020b), a legislation that specifies the issuance of certificates, and have suffered two amendments after its introduction in 1958, with the latest and most relevant amendment made in 2020, after the Boeing 737 MAX crashes. This last amendment determines that the FAA may not delegate any finding of compliance with applicable airworthiness standards or review of any system safety assessment required for the issuance of a certificate, including a TC, or amended or STC, until the FAA has reviewed and approved all assumptions related to human factors.

According to DOT (USA 2020a) the concept of delegation has been promoted for decades by the U.S. Congress, which encouraged its expansion on the 2012 and 2018 FAA Reauthorization Acts.

Also, as per the FAA Order 8100.15B (2018), ODA is the authorization to perform a variety of previously approved functions on behalf of the FAA, while ODA Holder is the organization that obtains this authorization from the FAA through a Letter of Designation, and finally, an ODA Unit is an identifiable group of two or more individuals within the ODA Holder’s organization that apply and perform the previously authorized functions following the FAA Order 8100.15B.

To be able to get an ODA, the organization must submit an application to the FAA, containing the functions for which the authorization is being requested, the qualifications of the applicant, such as proof of sufficient resources, personnel, facilities, and relevant experience to correctly perform the pledged functions, and also have previous and vast experience with FAA requirements, processes and procedures (FAA 2017c).

According to the 14 CFR Part 183 (FAA 2021b), the applicant must have a description of their organizational structure and a proposed ODA Unit within its existing structure, as well as a proposed procedures manual, containing the functions and limitations authorized by the FAA and the procedures for performing those functions. It must also contain an organizational structure and the responsibilities of the ODA Unit and the ODA Holder, as well as a description of the infrastructure and facilities used by them. The manual must also underline the training requirements for ODA Unit personnel, a process and a procedure of an ODA Unit periodic audit by the ODA Holder in compliance with FAA’s regulatory material, which must be acquired and maintained by the ODA Holder.

Furthermore, FAA (2017c) states that the manual must have a description of the experience required and knowledge required for each position within the ODA Unit, procedures for revising the manual and procedures for performing the activities for product certification and operational approvals. After the process approval by the FAA, according to the 14 CFR Part 183 (2021b), the ODA Holder receives its Letter of Designation, and must comply with the procedures contained in its approved procedures manual in order to maintain the effectiveness of the Letter of Designation, which can be revoked sooner by the FAA if the ODA Holder requests its termination or suspension before the due date, or if the Holder has not performed its duties properly and no longer meets the qualifications required to perform the authorized functions that have been previously authorized by the FAA.

For any approval or certificate for a product, part or appliance the ODA Holder must monitor reported service problems related to certificates or approvals and notify the FAA if a product, part or appliance has a condition that could result to be unsafe, or another one that does not meet the applicable airworthiness requirements, according to the 14 CFR Part 183 (FAA 2021b). The FAA can demand an investigation to the ODA Holder to find any suspected unsafe condition or after the discovery of noncompliance with the airworthiness requirements, producing the information necessary to implement corrective actions to end with the nonconformities, with all data submitted to the FAA at the end of the process.

The ODA goal, according to DOT (USA 2020a, p. 22), is:

The structured, safety-focused delegation system bolsters aviation safety and encourages innovation, efficiency, and industry growth. Delegation processes, including ODA, provide space for innovation and technical expertise while enabling the FAA to maintain its oversight processes and maintain established safety standards. By making use of delegation, the FAA is able to use a risk-based approach to focus its attention on the most critical certification areas.

The EASA DOA

The Design Organisation Approval (DOA) program is established by EASA under Part 21 Subpart J, of the Commission Regulation (EU) No. 748/2012, which standardizes the rules for airworthiness and environmental certification of aircraft and related products, parts and appliances, as well as the certification of design and production organizations.

The program was developed to establish a designee program, especially an organization, just as its counterpart program in the U.S. In terms of application, however, some things differ from the FAA program. At first, the applicant may fill a handbook provided by EASA with all the data necessary, such as the company’s organizational structure, information about the procedures adopted in the product development, the assurance of compliance, establishing a design assurance system for the control and supervision of the design, changes in design, products, parts and appliances.

This handbook must be handed to the agency, as well as any amendment necessary to keep it up to date to any changes in the company. The design organization must also state the management staff qualifications and experience as well as the qualifications of those who are linked to any decision that affects airworthiness and environmental issues in the organization (EASA 2021).

Once every aspect discussed previously have been scrutinized by the authority, an unlimited duration design organization approval is issued stating the specific terms of approval, such as the types of design work, the categories of products, parts and appliances for which the design organization holds a design organization approval. For DOA continued validity the holder must be able to demonstrate compliance with all the applicable requirements under EASA Part 21 Subpart J, the agency must be able to perform any investigation, including investigations of partners and subcontractors, and the holder must allow the agency to make any inspection, review any report and perform any test necessary to check the validity of the compliance statements made by the holder during the certification process. The design assurance system must also maintain a satisfactory control and supervision of the changes and the design of products, for the limitless duration of the approval to continue. The EASA can also revoke any certificate under applicable administrative procedures at any time (EASA 2021).

Any non-compliance with the applicable requirements under EASA Part 21 of a DOA holder found during an investigation are categorized in three distinct levels: A level one finding is any non-compliance which could affect the safety of the aircraft and which could lead to uncontrolled non-compliances with applicable requirements, the holder has no more than 21 working days to demonstrate necessary corrective action to fulfil the agency’s needs. A level two finding is any non-compliance which is not classified as level one, the corrective action period granted by EASA must not exceed three months at first; however, in certain circumstances, this period can be extended by the agency until a satisfactory corrective action plan is presented. A level three finding is anything that has been identified by direct evidence, to contain potential problems that could lead to a non-compliance, and it does not require immediate action by the designee. In case of levels one and two findings, the design organisation approval can be partial or fully suspended or revoked (EASA 2021).

During the certification process the DOA holder is entitled to perform activities within its scope of approval. Following the compliance documents, the approval of flight conditions for a permit to fly, a type-certificate or approval of a major change to a type design, a supplemental type- certificate, as well as a major repair design approval can be accepted by EASA without further verification, but can be thoroughly investigated by the agency if it judges to be necessary. The holder is also entitled to classify any changes to type design and repairs as major or minor, to approve minor changes to type design, minor repairs and the design of major repairs to products or auxiliary power units for which it holds a European Technical Standard Order (ETSO) authorization, a type-certificate or a supplemental type-certificate, to approve the conditions under which a permit to fly, related to the safety of the design, can be issued. The designee can also approve minor revisions to the aircraft flight manual and supplements, and publish instructions or information issuing statements standardized by EASA (2021).

Furthermore, the holder of a design organization approval must maintain its handbook in conformity with the design assurance system and ensure that it is used as a basic working document within the organization. The designee must provide EASA with statements and the necessary documentation to confirm the compliance with the requirements of the design of products, changes or repairs, and provide the required information or instructions related to airworthiness directives. Finally, the organization must determine the conditions under which a permit to fly can be issued, establishing compliance with the agency’s requirements (EASA 2021).

The ANAC COPj

ANAC implemented a Certified Design Organization in the amendment 3 of its Brazilian Civil Aviation Regulations (RBAC) 21 regulation known as COPj (ANAC 2018; 2021). The approach used by Brazilian authority is similar to the European one, however, differently from EASA, the organization certification is not a condition to obtain a TC. Therefore, the TC Holder, in Brazil, could be a certified organization, with a design assurance system, or a person that demonstrates a product complies with applicable regulations (ANAC 2021).

Another crucial point is that, even considering the RBAC 21 adopted a similar requirement to EASA, the European regulation already implemented changes that were not absorbed by Brazilian RBAC yet, such as the privileges of the organization and, therefore, it is possible that differences could exist between a design organization in each country (ANAC 2021). However, for the differences raised above with FAA approach, ANAC strategy can be considered identical to European one, with the certification of the organization instead the delegation of authority.

Differences between ODA and DOA

Both agencies have differences when issuing design approvals, especially with the type of information transferred to the authority by the designee, as well as some specific bureaucracies for the issuing of the certificate of designation and the certification itself.

A difference between the FAA and EASA designee programs is the approval of operational requirements, while the FAA certification process includes only the requirements for the approval of the aircraft design; in other words, the operational requirements can be met after the issuing of a TC, but before the entry into service. The EASA certification process is a little bit different from its counterpart, the compliance with the operational requirements and standards must be entirely fulfilled prior to the issuance of a TC (De Florio 2016).

While there are a lot of similarities between both approaches, there is a core difference: the ODA is a delegated structure inside the design organization, i.e., it is a system that acts on behalf of FAA (Yang and Liu 2011). Therefore, the ODA will perform the same activities of the FAA, which means that they will verify the compliance with the requirement based on a determined level of involvement, which specify what verifications tasks will be performed directly by FAA employee, the ones that will be performed by ODA unit member, and the ones that will not require a verification. This verification, performed under paragraph 21.33(a) of 14 CFR Part 21 (FAA 2021a), is a second layer to identify failures in the show compliance performed by the design organization (GAO 2022).

On the other hand, the independent Design Assurance System (DAS) from EASA DOA is a more embracing system, where the verification is only one part of the certified system, with the DAS also covering the development and show compliance processes. Furthermore, it is important to notice two differences to ODA verification: The DOA verification is performed over all show compliance data and artifacts, instead only a part as in FAA ODA; and the verification part of the DAS is performed using processes established by the DOA, that are adequate for their environment and are responsive to the independent monitoring findings, while ODA must follow the same processes and documents that FAA employees use (GAO 2022).

The aforementioned difference is clearer when we evaluate the self-monitoring of both system: while FAA ODA self-audit is focused on the ODA activities (i.e., only in the verification), the EASA DOA independent monitoring system cover all the certification process (GAO 2022).

Moreover, under EASA DOA, the design organization is responsible for all certification compliance findings, that is certified to perform these tasks, what is important to create the sense of accountability on the design organization. Otherwise, the FAA ODA, as stated above, must perform its activities as delegated by FAA, what could undermine this sense of accountability (GAO 2022).

Therefore, there are some points that need to be improved on the ODA program, and they will be further analysed on the next chapters.

CASE STUDY

Current Boeing ODA Unit

Currently, the Boeing 737 MAX and 777X certification programs use the ODA system, employing FAA guidelines to appoint and evaluate ODA Unit members, and following the FAA-approved Boeing Commercial Airplanes ODA Procedures Manual. In 2020, Boeing had 1399 ODA Unit Members, 1004 assigned to engineering and 395 assigned to manufacturing (USA 2020a).

These Unit members are Boeing employees from areas such as engineering, and production, that act on FAA’s behalf while performing ODA functions during the certification process. This structure is supervised by the FAA’s Boeing Aviation Safety Oversight Office (BASOO).

BASOO is part of AIR’s System Oversight Division (AIR-800) and is based in the Seattle area. It oversees large-scale inspections, routine audits and oversight, as well as participating in flight tests, ground tests, inspections and test witnessing.

737 MAX Development

On January 27, 2012, Boeing officially filed an ATC application for the 737 MAX 8 with FAA. This milestone marks the beginning of the aircraft’s certification process.

The proposed certification basis for the Boeing 737 MAX 8 followed the requirements in the section 21.101 of the 14 CFR Part 21 (FAA 2021a), the changed product rule, using the guidelines within AC 21.101-1B (FAA 2016) and Order 8110.48A (FAA 2017a).

Following the requirements established by this regulation, the certification basis was defined, and Boeing identified 12 changes labelled as significant from the Boeing 737-800, that was used as the baseline model for the Boeing 737 MAX 8. Among these modifications, there was a crucial addition to the aircraft systems, the MCAS, which was needed to compensate for aerodynamic changes from the baseline model, due to its larger, heavier engines and their placements on the wing.

This system would later be a pivot of a crisis within the 737 MAX program and would put in doubt the credibility of the FAA certification process, especially the delegation program (OIG 2021).

ODA level of involvement during the 737 MAX certification process

According to U.S. DOT (USA 2020a), the FAA determined which areas of the Boeing 737 MAX 8 would be involved, via delegation or directly, through a risk-based decision approach, establishing a risk assessment process that was both quantitative and qualitative, by analysing elements such as historical operational data derived from the baseline aircraft, the standard practices regarding maintenance, regulation criticality and standard operational practices intended to maintain a compliant product.

From February 15, 2013 to November 14, 2013, the FAA reviewed and accepted the Master Certification Plan, which described the acceptable means of compliance during the certification process, which items would be delegated to the ODA and which items and areas would remain under the FAA. After the initial approval, the FAA had retained the sections related to the stabilizer, including MCAS, and the flight controls.

The FAA level of involvement during the certification is at the Agency’s discretion at any given moment during the program and based on a risk assessment (USA 2020a). In some cases, the FAA may choose to leverage applicant expertise to aid on critical areas. This, however, is defined after a proper risk assessment of the information provided by the applicant. The U.S. DOT (USA 2020a) also stress the importance of FAA level of oversight, and its confidence in an organization that propitiates a higher quantity of delegations to the manufacturer’s ODA.

Furthermore, OIG (2021, p. 25) states that, during the 737 MAX 8 certification program,

Notably, the number of certification activities that FAA delegated increased significantly throughout the certification process, which, according to FAA managers, is typical as systems mature and the Agency gains confidence in Boeing’s capabilities through its initial involvement.

Initially, only 28 out of 87 (32%) of the detailed certification activities were delegated to the Boeing ODA for approval. This number, however, eventually reached 79 out of 91 (87%) by November 2016, including the system revisions, containing MCAS, and the flight controls modifications, sections of the Master Certification Plan originally retained by FAA.

Moreover, as stated by OIG (2021, p. 26):

FAA can delegate specific deliverables within each certification plan, such as system safety assessments, even if FAA retains the plan itself. These delegations can also change over the course of the project, as was the case for the over 1,700 Boeing 737 MAX deliverables.

However, in 2015, OIG (2015) reported a concern about the FAA’s lack of a risk-based oversight approach to ODA. This was ratified when FAA identified some problems regarding the quality of ODA certification documents that needed to be undertaken, as well as violations of Boeing’s document control system, and insufficient certification documents. By December of that year, Boeing was condemned to pay a civil penalty of US$ 12 million due to these transgressions.

According to OIG (2021), later oversight encountered indications that Boeing did not comply with all affairs raised by the FAA. Another issue identified by Boeing and FAA was reported undue pressure on ODA personnel at multiple Boeing facilities, leading to a formal compliance action against Boeing issued by the FAA in November 2018, these predicaments will be detailed at the following section.

Possible Failures of Boeing ODA System

The section 183.57 of the 14 CFR 183 (FAA 2021b) requires ODA companies to grant ODA Unit members sufficient authority to perform their authorized duties, without any interference that affect the member’s performance to deal with ODA functions and not give any conflicting non-ODA assignments.

A thriving ODA, according to FAA Order 8100.15B (2018), is an ODA which provides the necessary authority and time to each Unit member, without influence or pressure from other branches of the organization. This later description outlines what is known as undue pressure.

Another recurrent issue addressed by FAA Order 8100.15B (2018) is the necessity of no conflicting restraints or responsibilities for the members that conflicts with the ones from the ODA Unit. This, however, is not extensively explored by the regulations, with the definition of conflicting restraints becoming vague, and it does not specifically impede an engineer to both demonstrate and then evaluate compliance on the same design.

According to OIG (2021), during interviews with FAA and Boeing ODA, they confirmed that there were cases during the Boeing 737 MAX 8 certification that the same company engineer worked on a particular design and then approved it as an ODA Unit member, as employees are only considered ODA Unit members when they are performing tasks on FAA’s behalf. OIG (2021, p. 36) states that this situation “may not provide enough independence and could cause a conflict of duties for those Unit members.”

A survey conducted by MITRE (2020a) from November 20 to December 9, 2019 with FAA employees under the FAA Office of Aviation Safety (AVS), including the staff from the FAA Aircraft Certification Service, indicated that more than 40% of the respondents did not feel the FAA appropriately delegated certification activities to organizations and individual designees.

Another issue identified in survey responses was the difficulty of the FAA AVS to deal with influences of industry, lobbyists and other political pressures. There were some concerns raised, such as the organization’s tendency to put profit over safety, external influence and a too close a relationship with industry.

With the FAA proposing two civil penalties against Boeing and two separate letters of investigation, issued in June 2019 and March 2020 against Boeing’s South Carolina facility, detailing work interference and undue pressure of ODA Unit members by Boeing management. Penalties totalled $1.25 million (MITRE 2020b).

FAA and ODA holders are aware of such safety concerns derived from undue pressure on the ODA Unit members. The section 183.53 of the 14 CFR Part 183 (FAA 2021b) requires ODAs to establish and include in their procedures manuals processes for preventing interference, including performing periodic self-audits of every aspect of the ODA Unit, from the staff to processes, policies and compliance with ODA regulations. These self- audits are valuable, and, according to OIG (2021), both Boeing ODA and FAA have identified instances of potential undue pressure on Unit members from at least 2013.

Over the course of 2013 to 2019, the Boeing ODA found concerns regarding undue pressure on eight of eleven self-audits conducted at Boeing facilities in South Carolina and Washington. For OIG (2021), although none of the audits found violations of FAA regulations and Unit members were confident using the undue pressure reporting process, there were reports that members lacked the confidence that this process could reach a satisfactory conclusion or even protect the Unit members, and the perception of inadequate protection from actions by leadership outside of ODA.

Furthermore, in 2016, an undue pressure survey of the Boeing ODA Unit was conducted by the company, as seen in Fig. 1. There were 523 respondents, and while 97% of them understood the process for reporting undue pressure, close to 40% responded that they had encountered situations in which they perceived undue pressure, and almost 25% had experienced undue pressure beyond their direct reporting structure while performing ODA duties.

Figure 1
Results of Boeing’s 2016 Survey on ODA Undue Pressure.

Reports of pressure from high workloads, potential undue pressure due to the dual role of a Unit member, confusion and the desire for shared information about other cases of undue pressure within the company to help other Unit members learn from them were common during the survey. One respondent claimed that while: “upper management will never issue a direct order for [a Unit member] to do the wrong thing, [they] will create situations to indirectly pressure the [Unit member] to do the wrong thing” (OIG 2021, p. 39).

After this 2016 survey, Boeing addressed all formally reported cases of undue pressure without FAA action. However, the Agency observed the necessity of further oversight of the undue pressure systems and processes. In 2018, five engineering Unit members reported instances of interference or conflicting duties with their Unit member roles. This led to a formal compliance action against Boeing by the FAA, arriving in 2020.

The formal responses to FAA’s enforcement actions and results from Boeing’s 2016 internal survey provided a variety of causes for undue pressure, such as schedule pressure, which comes to the surface due to cost concerns, lack of knowledge from the Boeing management about their ODA roles, lack of communication between Unit members and ODA management, the delineation of company and ODA roles. That could lead to a pressure on the Boeing ODA Unit member to approve items or confirm compliance with regulations without the proper time to perform a review, acting against their own judgement and expertise, potentially impacting the aircraft safety (OIG 2021).

Moreover, the FAA AVS employees expressed their perception about external pressure for MITRE (2020b), feeling an intense pressure by industry to meet their production deadlines, as well as the way the industry perceives AVS employees and ODA Unit members to be standing in the way, and escalating it to senior leadership and even the Congress, resulting on the reversal of staff engineering recommendations and the replacement of individuals, for example.

Another source of pressure, according to MITRE (2020b), is the unwritten code to be more liberal-minded, supported by the Congress, FAA senior leadership and the industry, in terms of safety/risk, to find win-win solutions that benefit industry, putting pressure on technical staff that are responsible for identifying safety issues and concerns.

Other common worry identified by MITRE (2020b) was the concern that the FAA has delegated too much authority to industry, under 49 U.S.C. 44702(d), negatively affecting the safety of the National Airspace. According to employees overheard by MITRE (2020b), the current delegation system and ODA Model are causing FAA to move away from its safety mission, as well as creating confusion about FAA’s role as a regulator and as a promoter of safety.

For MITRE (2020b), the ODA System relies on the safety mindset and culture of the designees, and when they are compromised or inadequate, the system becomes compromised and less effective.

Boeing ODA and the MCAS

In 2016, less than a year before the end of certification, a Boeing ODA Unit member advisor expressed, in an internal email, his concerns regarding a lack of clarity by Boeing when notifying FAA about changes made late in the certification process, as well as its lack of confidence that FAA and ODA Unit members clearly understood, after the flight-testing certification campaign started, what they were approving when Boeing made changes during this period (OIG 2020).

In May 2016, Boeing introduced the MCAS Revision D, and while mentioning some aspects about the system during a briefing with FAA flight test personnel, such as the increased maximum range of the horizontal stabilizer movement under MCAS from 0.55° to 2.5° and the changed parameters that permitted MCAS to be activated at much slower airspeeds than before, ranging from 0.2 to 0.84 Mach, whereas previously it could only be activated at speeds above 0.67 Mach, the FAA certification engineers were unaware of the significant changes to this system and did not correctly understand how it worked (OIG 2021).

There were multiple FAA branches, groups, divisions and offices involved in the certification process, specifically employees from the Aircraft Certification Service and the Flight Standards Service with its Aircraft Evaluation Group (AEG).

The AEG was responsible to determine the appropriate type rating and levels of pilot training required for the 737 MAX 8 aircraft, and, according to OIG (2020), the inspectors were also unaware of the MCAS full capabilities. As a result, they were making vital decisions regarding the information provided to pilots without fully understanding the function, like approving the complete remotion of any mention of MCAS from flight crew manuals in 2016. An AEG employee that was responsible for approving the change stated that: “FAA, as a result of the information they had at the time, based the decision on the understanding that MCAS remained as originally designed” (OIG 2021, p. 22).

Furthermore, Boeing’s objective to keep the same type rating as the 737 NG, and to keep costs down by bypassing simulator training for 737 MAX pilots, according to OIG (2021, p. 22), led to: “internal messages discussed how Boeing wanted to present it to FAA as an additional function of the existing speed trim system, as well as its ODA’s concurrence with that approach, while still using the term ‘MCAS’ internally.” By using this approach, the MCAS was not an area of emphasis regarding pilot training, which was more focused at the new flight displays and banking warnings.

While Boeing tested a single, inadvertent activation of MCAS, multiple activations were not tested, assuming that repeated activations of MCAS would not be worse than a single activation. When developing the risk assessment, Boeing engineers and test pilots made an engineering assumption that commercial pilots would react to an unintended MCAS activation as a runaway stabilizer event (OIG 2021).

In addition, even after the 737 MAX 8 certification there were communication gaps between Boeing and FAA. In August 2017, Boeing became aware that not all 737 MAX 8 were equipped with an AOA disagree alert, as seen in Fig. 2, to notify pilots when the two AOA sensors disagree by more than 10° for at least 10 s.

Boeing later included the AOA disagree alert message in updated certification documents, that were approved by a Boeing ODA Unit member in September 2017. According to OIG (2020), the company considered that this modification would not have an operational impact, and did not submit a formal notification to the FAA oversight office. FAA was only officially notified by Boeing about this issue after the Lion Air accident.

Figure 2
AOA Disagree Message and AOA Indicator.

CONCLUSION

The regulations of counterpart programs in Europe and Brazil, which have a more embracing system, could be used as an example to strengthen the regulatory basis of the ODA program. The independence assured by both regulations, as well as the clear separation of roles provided by the organization certification approach are a step forward to the ODA program, and should be considered by FAA to consolidate its regulatory basis, and properly address the conflicting restraints issues.

Therefore, the ODA model adopted for the Boeing ODA during the Boeing 737 MAX 8 certification requires some important enhancements, and while changes are necessary for an overall improvement on the certification process, the separation of roles, the effective determination of the authority’s involvement and the definition of the accountability and liability of the ODA Holder are key factors, combined with improvements on the dialog with FAA, to enhance safety.

Finally, this paper described different approaches to certification, using the 737 MAX 8 certification program and the FAA, EASA and ANAC regulatory basis, analysing the possible failures occurred during this process and what could be optimized and improved on the ODA program, while evaluating the proposed changes to the program.

ACKNOWLEDGEMENTS

Not applicable.

• Peer Review History: Single-Blind Peer Review.

• DATA AVAILABILITY STATEMENT

  All data sets were generated or analysed in the current study.

• FUNDING

  Not applicable.

REFERENCES

• [ANAC] Agência Nacional de Aviação Civil (2018) Resolução nº 495, de 14 de novembro de 2018. Brasília. [accessed Jul 21 2022]. https://www.anac.gov.br/assuntos/legislacao/legislacao-1/resolucoes/2018/resolucao-no-495-14-11-2018
  » https://www.anac.gov.br/assuntos/legislacao/legislacao-1/resolucoes/2018/resolucao-no-495-14-11-2018
• [ANAC] Agência Nacional de Aviação Civil (2021) RBAC 21. Brasília. [accessed Apr 12 2021]. https://www.anac.gov.br/assuntos/legislacao/legislacao-1/rbha-e-rbac/rbac/rbac-21
  » https://www.anac.gov.br/assuntos/legislacao/legislacao-1/rbha-e-rbac/rbac/rbac-21
• De Florio F (2016) Airworthiness: An Introduction to Aircraft Certification and Operations. 3 ed. Cambridge: Elsevier.
• [EASA] European Union Aviation Safety Agency (2021) Part 21. Cologne.
• [FAA] Federal Aviation Administration (2011) Order 8100.8D. Washington D.C. [accessed Nov 19 2021]. https://www.faa.gov/regulations_policies/orders_notices/index.cfm/go/document.information/documentid/1019601
  » https://www.faa.gov/regulations_policies/orders_notices/index.cfm/go/document.information/documentid/1019601
• [FAA] Federal Aviation Administration (2017a) Order 8110.48A - How to Establish the Certification Basis for Changed Aeronautical Products. Washington D.C. [accessed Aug 22 2021]. https://www.faa.gov/regulations_policies/orders_notices/index.cfm/go/document.information/documentID/1031586
  » https://www.faa.gov/regulations_policies/orders_notices/index.cfm/go/document.information/documentID/1031586
• [FAA] Federal Aviation Administration (2017b) Order 8110.4C - Type Certification - With Change 6. Washington D.C. [accessed Jul 14 2021]. https://www.faa.gov/regulations_policies/orders_notices/index.cfm/go/document.information/documentid/15172
  » https://www.faa.gov/regulations_policies/orders_notices/index.cfm/go/document.information/documentid/15172
• [FAA] Federal Aviation Administration (2017c) The FAA and industry guide to product certification. Washington D.C. [accessed Jun 23 2021]. https://www.faa.gov/aircraft/air_cert/design_approvals/media/cpi_guide.pdf
  » https://www.faa.gov/aircraft/air_cert/design_approvals/media/cpi_guide.pdf
• [FAA] Federal Aviation Administration (2018) Order 8100.15B - Organization Designation Authorization Procedures, Including Change 3. Washington D.C. [accessed Apr 27 2021]. https://www.faa.gov/regulations_policies/orders_notices/index.cfm/go/document.information/documentid/1021171
  » https://www.faa.gov/regulations_policies/orders_notices/index.cfm/go/document.information/documentid/1021171
• [FAA] Federal Aviation Administration (2021a) 14 CFR Part 21. Washington D.C. [accessed Apr 04 2022]. https://www.faa.gov/regulations_policies/rulemaking/committees/documents/media/Part21ARC-10052012.pdf
  » https://www.faa.gov/regulations_policies/rulemaking/committees/documents/media/Part21ARC-10052012.pdf
• [FAA] Federal Aviation Administration (2021b) 14 CFR Part 183. Washington D.C. [accessed Feb 11 2022]. https://www.ecfr.gov/current/title-14/chapter-I/subchapter-K/part-183
  » https://www.ecfr.gov/current/title-14/chapter-I/subchapter-K/part-183
• [GAO] United States Government Accountability Office (2022) GAO-22-104480. Washington D.C. [accessed Aug 02 2022]. https://www.gao.gov/assets/gao-22-104480.pdf
  » https://www.gao.gov/assets/gao-22-104480.pdf
• MITRE. The MITRE Corporation (2020a) Federal Aviation Administration (FAA) Aviation Safety Organization (AVS) Safety Culture Survey Findings, February 28, 2020. McLean. [accessed Feb 20 2022]. https://transportation.house.gov/imo/media/doc/FAA%20Records%20Set%20for%20Public%20Release.pdf
  » https://transportation.house.gov/imo/media/doc/FAA%20Records%20Set%20for%20Public%20Release.pdf
• MITRE. The MITRE Corporation (2020b) Federal Aviation Administration (FAA) Aviation Safety Organization (AVS) Safety Culture Assessment Report, February 28, 2020. McLean. [accessed Feb 20 2022]. https://transportation.house.gov/imo/media/doc/FAA%20Records%20Set%20for%20Public%20Release.pdf
  » https://transportation.house.gov/imo/media/doc/FAA%20Records%20Set%20for%20Public%20Release.pdf
• [OIG] Office of Inspector General (2015) FAA lacks an effective staffing model and risk-based oversight process for organization designation authorization. Washington D.C. [accessed Dec 22 2021]. https://www.oig.dot.gov/library-item/32742
  » https://www.oig.dot.gov/library-item/32742
• [OIG] Office of Inspector General (2020) Timeline of activities leading to the certification of the Boeing 737 MAX 8 aircraft and actions taken after the October 2018 Lion Air accident. Washington D.C. [accessed Nov 17 2021]. https://www.oig.dot.gov/library-item/37940
  » https://www.oig.dot.gov/library-item/37940
• [OIG] Office of Inspector General (2021) Weaknesses in FAA’s certification and delegation processes hindered its oversight of the 737 MAX 8. Washington D.C. [accessed Mar 14 2022]. https://www.oig.dot.gov/library-item/38302
  » https://www.oig.dot.gov/library-item/38302
• [USA] United States of America (2020a) U.S. Code. 49 USC 44702: Issuance of certificates. Washington D.C.
• [USA] United States of America (2020b) U. S. Department of Transportation. Official Report of the Special Committee to Review the Federal Aviation Administration’s Aircraft Certification Process. Washington D.C.
• Yang K, Liu C (2011) Comparison on the ways of airworthiness management of civil aircraft design organization. Procedia Eng 17:388-395. [accessed Mar 16 2022] https://doi.org/10.1016/j.proeng.2011.10.045
  » https://doi.org/10.1016/j.proeng.2011.10.045

Edited by

Section editor: Eric Nioya

Publication Dates

• Publication in this collection
  28 Nov 2022
• Date of issue
  2022

History

• Received
  30 May 2022
• Accepted
  08 Sept 2022