Well, it’s been a few days and all the headlines still seem to be appended with “…cause not yet pinpointed”. This never stops the media from hastily putting together some scripts with all the right buzz words in them however. I have to admit, I was enjoying the announcer’s commentary on the outage as much as the game and commercials (skechers was my personal fav) themselves. Said one, “As you can see a power surge has hit the stadium and now the lights are slowly getting brighter as they restore power”. Umm…. not really I thought. They restored power 20 min ago, and the HID metal halide lighting is just going through its normal restrike and run-up cycle. Also, a surge, normally defined as a multi-cycle moderate voltage increase, is a fairly rare power anomaly and was almost certainly not the cause. In a way the reporting reminded me of the Fukushima nuclear accident in Japan – listening to the news as a former nuclear engineer, the fabrications of respected news agencies were embarrassing. I could picture some producer behind the scenes telling the reporters, “look, no one understands this stuff anyway, just be creative and fill in the gaps”. It’s not only the reporters troweling out these inaccuracies. Greg Boyce, CEO of Peabody said the outage was a “convincing visual demonstration to counter those who’ve envisioned a world without coal”. Really Greg? Regardless of your views on global warming, I’m pretty sure coal was not a root cause here.
Anyway, it is also unlikely that an overloaded feeder problem existed, as has been reported. In large part, the superdome is either on or off. The lighting, ventilation, the guy cooking hot dogs, none of them use any more power for the superbowl than any other event. There is not much of a per-occupant electrical burden, and the place only has a fixed number of seats anyway. Further, design engineers are notoriously conservative in sizing wires and breakers, because they don’t have to pay for the oversizing and the consequences of undersizing are severe. In general, owners under-appreciate the cost savings of right-sizing, but the engineer will get slapped with back charges or even sued if things are undersized – we recently investigated just such a case and estimated corrective action to be over 175% of initial project cost due to demolition expense.
Ok then, so what was the cause and how might it have been prevented?
My guess, and without having personally investigated that’s all it is, is the incident was related to improper protective action coordination. In my experience, improper trip coordination is the biggest cause of unintended partial facility outages. I’ve never done a coordination study and not found instances of mis-coordination. In simple terms, the purpose of coordinated protective action in electrical distribution systems is to ensure that faults are isolated with minimal impact to the rest of the distribution. With thousands of pieces of equipment running in the superdome, the probability of a fault developing in one of them is fairly large – we should expect it. What is important is that the branch circuit breaker feeding the faulted equipment has a trip curve that is below and to the left on a time-current plot of the feeder breaker(s) upstream. In plain english, this means that when something goes wrong electrically, the branch breaker trips first and isolates the fault without any power interruption to other loads. With metal halide (MH) stadium lighting, it is particularly important to rigorously apply coordination to protect against any voltage dips on the panels feeding the lighting. Why? MH lighting (and other HID lighting) is vulnerable to even brief power interruptions, and once it’s off, as we all saw, it takes a long time to restart. Even a 15% voltage drop for a fraction of a second (and a fault in 1 decent size motor can easily cause this) is enough to knock out MH lighting. These lights work by vaporizing mercury in an arc tube, and the tube temperature & pressure have to be allowed to decrease by natural cooling for 10 or more minutes before “re-striking” the arc. Manufacturers do have “instant restrike” lights available, but they are expensive. Once the arc is re-ignited, it then can take another 15 min or so to reach full intensity as the cooled metal halides are again fully re-vaporized. This causes the “sunrise” effect that we saw in the stadium.
Assuming my mis-coordination theory is correct, the design engineer’s coordination study should have involved plotting the lighting manufacturers time-voltage tolerance curve right on same plot as the adjacent breaker trip curves. This method makes it immediately apparent whether it is possible for any other branch breakers to “let through” a fault that could allow the MH lighting arcs to extinguish. If so, we make adjustments to the trip settings, select different breakers, or even add fusing and plot the curves again until we achieve adequate coordination. Notwithstanding code requirements, the rigor with which this process is applied must increase when the economic impact of the unavailability of the involved loads increases. In other words, if the HID lights in a Walmart parking lot go out for 30 min, it’s not a huge deal, but when 1/3 of the country is watching an event that suddenly goes dark…..
Hopefully the outage didn’t ruin your superbowl experience. If the 49ers had pulled it off, I think the NFL might have someday been using words like “legendary” and “famous” to describe this power outage and its alleged effect on the game. Personally, I didn’t mind it. I just feel bad for the facilities guys down in the pits of the stadium. Think about it – they were probably getting yelled at on their radios before the 7 second network delay allowed them to see it on their little TV in the electrical shack. Then they ran down the hall, and reset the tripped breaker, lets say, within 3 minutes. Then for the next 31 minutes, they were getting blasted with irate inquiries and demands, trying to explain things like “restrike time” to a bunch of execs. But in fact they had already done all they could and it was just a waiting game. Now for weeks they will be interrogated by experts and investigators sniffing around for clues, writing their reports. Poor guys. Anyway, I sent my CV over to Doug Thornton, superdome manager, so maybe with a little luck I’ll get to meet them.