Waking From the Nightmare : 787 Dreamliners Now Grounded

Wednesday afternoon saw the US FAA suddenly ground all 787 planes, requiring proof of the safety of the plane’s onboard Lithium ion batteries before it would allow the plane to resume commercial flights.  Other countries with airlines operating 787s quickly came out with matching orders, and all the planes are now withdrawn from commercial service for an unknown/indefinite period.

The FAA order would seem to contradict their extremely carefully worded announcement on Friday last week that the plane was ‘safe but needs review’ – a process which, as envisaged last Friday, would not have involved any restraints on the plane’s continued operations.

Last Friday the FAA held a carefully choreographed press conference, complete with Boeing executives struggling to look positive rather than po-faced, at which it indicated that it would launch a wide-ranging review of the plane’s critical systems and the processes by which they had previously been certified as satisfactory and safe.  However, the FAA was at great pains to indicate that this highly unusual procedure did not indicate any concerns about the airplane’s safety.

The FAA has found itself in the difficult position of first having certified the plane as safe to fly little more than a year ago, in a process which Boeing proudly proclaims, even now, as ‘the most robust and rigorous certification process in the history of the FAA’.  Since the plane’s approval, and in response to concerns that have been regularly voiced, the FAA regularly repeated its belief that the plane was safe to fly.

Then, last Friday it managed to describe black as white when announcing that it was going to hold a ‘comprehensive review’ of the plane’s safety, while simultaneously saying that the plane was safe (and so surely didn’t need any review).

But further serious 787 problems since Friday, leading to a voluntary decision by ANA and JAL to ground their planes, have meant that now on Wednesday this week, the FAA felt compelled to take the extreme step of a total grounding of the plane – not just for a very brief period of time for some specific fix to be carried out, but rather for an open-ended time frame until the plane was proven to be safe again, with no clear indication of what proof would be acceptable or how it would be gained – indeed, there isn’t yet a consensus or clear understanding of what the problems are, let alone what the appropriate solutions may be.

Indeed, since our newsletter’s comments on the 787 last Friday morning, there have been seemingly daily new problems with the 787 – not all critical problems, but adding to the crush of bad news about the plane.

The last time the FAA grounded an airplane type was in 1979 with the DC-10, and the FAA was an unwilling participant in that grounding, goaded into action by a court order filed by an airline passenger association.

These Are Not Teething Problems

We’re going to flip over to ‘the other side’ for a minute.  Yes, we do agree that some of the problems are indeed ‘teething problems’ and don’t necessarily mean the entire plane is flawed.  A cockpit windshield cracking, for example, and fuel tank leaks.  These are possibly like the minor rattles and problems you get with a new car – irritating nuisances but nothing that make you want to return the car and buy a different make/model of car.

We also agree that the problems with the planes’ engines are not so much a Boeing problem as an engine manufacturer problem.

But we disagree that ‘teething problems’ (not exactly a scientific exact term used by the FAA to indicate acceptable failures and problems, but rather a nice safe sounding phrase used by Boeing’s apologists) should be allowed at the end of ‘the most robust and rigorous certification process in the history of the FAA’.  This is especially inappropriate when the outcome of that allegedly marvelous certification process was nearly instant ETOPS authority to fly the plane for five and a half hours over the water away from the nearest airport.  Even benign teething problems may sometimes require the plane to divert and be safely on the ground in less than 330 minutes of further flying.

Even so, the so-called teething problems are perhaps acceptable as long as the safety of the plane is not at risk.  Unfortunate, but acceptable.

What is not acceptable is problems with the plane’s battery systems.

The Nature of the Key 787 Problem

A lot has been written about the plane being made, in significant part, by carbon fiber type materials rather than by ‘old fashioned’ aircraft grades of aluminum and aluminum alloys.  That’s another whole huge area of potential concern, but one which has yet to manifest itself in any ‘teething troubles’.  Indeed, our concern over the carbon fiber construction has always been (and remains) longer term.

But the battery system has not attracted so much attention in the press – at least, not until now.  In order to save weight, Boeing has switched a lot of the systems from being powered by bleed air or hydraulics from the engines, as has been the case in past airplane models, and instead, they are now powered by electrical actuators, indeed, the plane generates an enormous 1.5MW of power (about the same as 2,000 hp) to drive all its systems.

Now that’s no big deal, in and of itself – electrically powered devices are far from untried and untested.  The big issue is what happens if there is a problem with the engines and generators/alternators that cause them to stop generating the electricity needed for all the airplane’s systems – many times more power than has been required by earlier less-electric planes?

Earlier planes have some backup battery power provided by Ni-Cad (Nickel cadmium) type rechargeable batteries.  Ni-Cad batteries are very old technology, and have various limitations associated with them.  From Boeing’s point of view, the biggest limitation is that the Ni-Cad batteries are bulky and heavy.

In most non-airplane applications, Ni-Cad batteries have been replaced by Ni-MH batteries (Nickel metal hydride).  They are a bit more expensive, but they hold more energy per unit of volume and weight.

In cases where either space is critical, or cost not so important, Ni-MH batteries have in turn been replaced by some type of Lithium (Li) based (and still rechargeable of course) battery.  These batteries hold even more charge per unit of weight and volume than Ni-MH batteries.  The most common form of Lithium based battery is referred to as a Lithium Ion (Li-ion) battery, and there are various slightly different types of these.  Li-ion batteries are what power most cell phones and tablets, because space and battery longevity is of paramount concern, and cost slightly less important.

In Boeing’s case, the company decided that the extra cost of Li type batteries was justified by the space and weight saving they offered, and in selecting between the various different types of Li-ion batteries, chose a cobalt type one that is generally considered as the most dangerous – but also the ‘best’ in terms of storing the most charge in the least amount of space and weight.

In choosing to add any type of Li batteries to the 787, Boeing entered a curious zone of irony.  The FAA has placed restrictions on having Lithium type batteries carried in airplane holds and in air freight, and many airlines are keen for even more severe restrictions, due to concerns about what might happen if one of these batteries fails and explodes/bursts into flames in an airplane hold in the middle of a flight.  But here is Boeing, wanting to place substantial quantities of the most dangerous type of these batteries in electronic bays underneath the passenger compartment – in other words, doing the very same thing that the FAA restricts commercial shippers from doing and which the airlines want to see even further restricted.

Accordingly, the FAA focused very closely on the safety of the batteries as part of its certification process.  Or, at least, it says it did, although its evaluation process was contentious at the time and challenged as being inadequate by a whistle-blower back in 2006.  Here are some details of the ambiguity of the FAA’s apparent thorough testing; and of course the two different battery problems in the last week now cast some doubt – if not on the FAA’s testing, definitely on their conclusions.

A special rider was added to their certification approval, which reads

Accordingly, pursuant to the authority delegated to me by the Administrator, the following special conditions are issued as part of  the type certification basis for the Boeing Model 787-8 airplane. In lieu of the requirements of 14 CFR 25.1353(c)(1) through (c)(4),  the following special conditions apply. Lithium ion batteries on the  Boeing Model 787-8 airplane must be designed and installed as follows: (1) Safe cell temperatures and pressures must be maintained during any foreseeable charging or discharging condition and during any  failure of the charging or battery monitoring system not shown to be extremely remote. The lithium ion battery installation must preclude explosion in the event of those failures.

Clearly the two battery problems over the last week both invalidated that requirement.

Which puts the FAA in a very difficult position.  Much of what it will be reviewing – and potentially finding fault with – is its own testing and certification process.  Is the FAA the best organization to audit and review its own processes and procedures?

Although a lot of the focus is on the end result – the batteries spectacularly exploding – it is not currently clear whether this is due to a problem within the batteries themselves (a manufacturing quality control problem), or an environmental problem (perhaps the temperature they are kept at, which can reach elevated levels that, if nothing else, materially shorten the lives of these battery packs, or perhaps also there are changes of pressure in the storage compartment that creates physical stress on the batteries) or something to do with the external charging system which might be overcharging or mismanaging the batteries.

Finding out what the problem is of course a key part of the process than now has to be undertaken.

One more thing – to put all this into context, the new batteries are saving about 500lbs on the plane.  The plane has a maximum takeoff weight of just over 500,000 lbs.  The safety of the plane has apparently been put at risk by a saving of one tenth of one percent of its total weight.

The Danger of Lithium Batteries

It is helpful to understand the nature and magnitude of the problem, which is all to do with what happens to rechargeable Li-ion batteries (not to be confused with single use Lithium metal batteries) when they ‘catastrophically fail’.

You may have heard occasional stories of iPhones or whatever else ‘exploding’ or ‘bursting into flames’.  But take a minute or two to have a look at this educational video that shows the battery inside a laptop failing under laboratory test conditions – and note how each individual cell in the battery (laptops typically have from four to twelve cells) causes the next cell to subsequently fail, and in each case, with more violence that the preceding one.  Furthermore, towards the end, the failure of the battery inside the laptop and the heat and flames generated causes an adjacent multi-cell battery to then fail as well.

So the domino effect can not only cause the cells in one battery to successively fail, but the first battery’s failure can spill over to a second and subsequent batteries that are adjacent to the first as well.

The 787 has multiple batteries, each comprising multiple cells (this is believed to be the underlying cell type).  A single cell failure can risk the entire battery, and one battery failure may then spread to the adjacent batteries.

These spectacular failures are made worse by the fact that the electrolyte (one of the chemicals in the battery) is moderately flammable, and the batteries, when heated, release oxygen.  Something that burns, some oxygen for it to burn, and the heat to get it started – as any fireman will tell you, that’s a very bad combination and not something you want to have happen on a plane, 5 1/2 hours away from the nearest airport.

This is bad enough with one tiny battery weighing a couple of ounces in an iPhone, or a pound or two in a laptop, but when you have 60lbs or more of battery all together, then it can become a huge problem.

How Long Will the Planes Be Grounded?

Fortunately, the grounding of the 50 delivered 787s won’t be very impactful on US travelers.  The only US airline with any 787s is United, with a mere six.  Of the balance, 17 are operated by ANA and seven by JAL, five each by Qatar Airways and Air India, four by Ethiopian Airlines, two by LOT and three by Lan Chile.

How long before the FAA allows them to fly again?  That is very hard to say, particularly due to the broad nature of the testing required to reassure the FAA that the planes are safe.  Furthermore, when the testing actually uncovers the issues causing the problems, how long will it take to resolve those issues and then to retest the fix to ensure it is now safe?

The only other across the board grounding – that of the DC-10 in 1979 – was a 37 day event, but the very different circumstances of that grounding don’t give us any meaningful guidance for how long this grounding may last.

We can’t see how a thorough battery testing protocol could take less than a month.  Indeed, late Thursday afternoon the Japanese manufacturer of the batteries surfaced with their first public statement on the issue, in which they said a full investigation ‘may take months’.

Something (actually, apparently, two different things – one affecting the batteries for the APU and the other affecting the main battery compartment) have caused two different problems over the last week.  Boeing and its suppliers need to identify the cause of each of these problems, then come up with satisfactory fixes for the problems, then get the fix installed on the planes before they can fly again.

Add some bureaucratic time for agreeing on the scope and extent of testing, the fail/pass parameters, and so on, and then, after identifying the root causes, and coming up with solutions, applying those solutions to the grounded planes, and months zip by very quickly indeed.  Indeed, we’ll go as far as to say that if the testing and re-approval process takes less than a month – especially now that the battery manufacturer has said their review and troubleshooting might take multiple months, then one has to be concerned about its valid comprehensiveness and completeness.

Meantime, Boeing keeps churning out new planes from its two 787 production lines (although the dynamics of halting the production lines are such as to make that close to impractical).  Its overall 787 program, which due to all its past problems and delays and cost overruns was thought might take as many as 1,000 airplane sales to finally reach profitability, now looks to be even further away from finally becoming the success that Boeing needs it to be.

The Ultimate Lesson?

Boeing’s proud boasting of the 787 having been subjected to  ‘the most robust and rigorous certification process in the history of the FAA’ is now made to look rather hollow by the post-certification problems.

New technologies pose both known unknowns and unknown unknowns.  Computer modeling can only explore the possible impacts of known variables.  Beyond that, there’s unfortunately no substitute for real world testing, and as we’ve seen, it has taken many thousands of hours of commercial flying spread over some 40 operational planes for these two recent battery problems to appear.

The FAA needs to slow down its 330 minute ETOPS certification procedures, not speed them up.  The FAA needs to conduct the thorough and wide-ranging review of any new plane’s critical systems, such as it promised last Friday to now undertake with the 787, before rather than after it has approved the plane as suitable for unrestricted commercial operations.

If you’d like more information on this situation, we recommend reading back through the 787 items that are sprinkled through Ben Sandiland’s excellent Plane Talking blog, and also reading the various relevant items in the Seattle Times, starting perhaps here.