Mis-steps at Indian Point Nuclear Plant

, director, Nuclear Safety Project | March 1, 2016, 6:00 am EST
Bookmark and Share

Disaster by Design/Safety by Intent #21

Disaster by Design

Growing up, I remember seeing a test pattern of the Emergency Broadcast System appear on our black and white television set accompanied by a really annoying high-pitched constant sound, followed by a voice telling us that “If this had been an actual emergency, you would have been instructed where to tune in your area for news and official information.”

Source

(Source: Filiapao 28)

The United States discontinued the Emergency Broadcast System in 1997, but workers at the Indian Point nuclear plant in New York needed a voice a couple of years later telling them “Because this is an actual emergency, follow the instructions to make it stop.” With that voice silenced, many of them headed home and let their “secret” emergency worsen.

The Event

The Indian Point Unit 2 pressurized water reactor was operating at 99% power on Tuesday, August 31, 1999. Instrument and control technicians were performing maintenance on Channel 3 of the reactor protection system when a spurious voltage spike occurred on Channel 4. Signals from both channels triggered the automatic shut down of the reactor at 2:31 pm.

The reactor’s shutdown caused the automatic shut down of the turbine and generator by design. The generator had been producing electricity that passed through the switchyard and out via power transmission lines to the electrical grid. Some of the electricity produced by the generator went to powering the unit’s equipment. The generator’s shut down deprived in-plant equipment of their normal source of power when then unit was operating. The design called for power supplies to automatically shift from the generator to the offsite electrical grid. Electrical relays made these transfers. But four minutes later, the electrical breakers re-opened, stopping the flow of power from the offsite grid to the four key electrical circuits.

Sensors detected low voltage on these four key electrical circuits, triggering the automatically startup of the three emergency diesel generators. The emergency diesel generators started and connected to the four electrical circuits to re-power equipment supplied from these circuits. Fourteen seconds later, the breaker connecting emergency diesel generator 23 to electrical circuit 6A re-opened, causing that one electrical circuit to lose alternating current (ac) power again. The breaker’s re-opening affected the other three electrical circuits in another way—the loss of ac power to electrical circuit 6A prevented the other three electrical circuits from being re-connected to the offsite electrical grid.

Battery 24 was another backup power supply to electrical circuit 6A. Thus, some of the vital equipment on electrical circuit 6A continued to receive direct current (dc) power from the battery. Other vital equipment (including a motor-driven auxiliary feedwater pump, a charging water pump, a component cooling water pump, and an auxiliary service water pump) had no power supplies available.

The Emergency Not Called

The unit’s emergency response procedures dictated that an Unusual Event, the least serious of the NRC’s four emergency classification levels, be declared when offsite power is unavailable to the four key electrical circuits for longer than 15 minutes. Even though that condition existed by 2:50 pm, the operators misinterpreted the procedure and did not declare an emergency at that time.

The Non-Emergency Response

At 4:00 pm, management convened a meeting to discuss tasks that needed to be performed before the reactor could be restarted. Restoring ac power to electrical circuit 6A was deemed most important of the steps to restart, but a lot of other housekeeping chores were added on the To Do list.

At 4:30 pm, the Station Nuclear Safety Committee (SNSC) met to review a procedure for work scheduled to be performed before restart. The NRC’s report on the event stated “that the SNSC meeting, which covered a topic unrelated to the trip and recovery, distracted some plant personnel from efforts to evaluate Bus 6A and recover from the event.”

Some workers simply went home.

I attended the public meeting conducted in the NRC’s Region I offices in King of Prussia, Pennsylvania with representatives from Indian Point Unit 2 regarding this event. An NRC manager told me about the agency’s frustration during the event getting the plant’s management to take it seriously. When I asked what this meant, the manager said that many workers headed for home at the normal end of the work day, despite the unit being unable to get power from the offsite grid and some vital equipment only getting powered from batteries that would soon become depleted. The manager told me that some of the recovery efforts were delayed by the departing staff resources. For example, the Plant Manager and Vice President-Nuclear for Indian Point Unit 2 headed home before 6:00 pm.

The Watch Engineer performed an online risk assessment at 6:40 pm and concluded that the condition was Red due to a Daily Risk Factor of 196. The typical Daily Risk Factor was less than 1.0 (zero being lowest risk) and a Red condition had never been calculated for Unit 2 before. The risk was estimated to be 1.8×10-3, or approximately 2 in a 1000 chance of reactor core damage—or nearly 200 times greater risk than that associated with normal reactor operation. And things went downhill from there.

The Emergency Belatedly Called

At 9:55 pm, the voltage from Battery 24, normally at 118 volts, dropped below 105 volts, causing it to stop powering safety equipment. The operators declared an Unusual Event, the least serious of the NRC’s four emergency classification levels, because the depleted battery disabled about 75% of the alarms in the control room.

The operators notified all state and local agencies by 10:09 pm. The operators notified the NRC about the emergency at 10:39 pm.

The Overdue End of the Emergency

The operators reconnected electrical circuit 5A to the offsite power grid at 2:24 am on September 1, 1999 and shut down emergency diesel generator 21 eighteen minutes later.

The operators reconnected electrical circuits 2A and 3A to the offsite power grid at 2:50 am and shut down emergency diesel generator 22 six minutes later.

At 3:30 am, the operators terminated the Unusual Event.

The Well-Earned Emergency

The spurious voltage spike on reactor protection system Channel 4 had occurred several times prior to this event, most recently on August 26, 1999—just five days earlier. Workers initiated a condition report that day to troubleshoot and repair the problem. But the condition report was closed out on the morning of August 31 without any troubleshooting or repairing.

The recurring but uncorrected problem recurred again, this time with worse consequences.

After the generator tripped, the supply of power to in-plant equipment automatically transferred to the offsite power grid per design. But the electrical breakers for four key electrical circuits re-opened four minutes later. Workers had modified the control logic for these breakers in 1995. The modification specified voltage conditions for closing and opening these breakers. But those voltage values had not been revised in calibration and maintenance procedures. Thus, when workers calibrated the control logic in June 1997, they used the wrong values. During the August 31, 1999, event, these breakers performed as calibrated instead of as designed to perform.

Emergency diesel generator 23 automatically started and connected to electrical circuit 6A per design. But its output breaker re-opened 14 seconds later. In 1997, workers modified the overcurrent protection relays for the emergency diesel generator output breakers, reducing the setpoint for opening the breakers from 7,500 amps to 6,000 amps. Workers tested the relays and verified that they opened the breakers at 6,000 amps. But the test they used was deficient. When workers re-tested the relays after the August 31 event using a proper procedure, the breakers opened instead at 3,000 amps. When emergency diesel generator 23 started and connected to electrical circuit 6A, other logic circuits restarted equipment every few seconds to avoid overloading the emergency diesel generator with too much demand (i.e, electrical current needs) at the same time. The 6,000 amp design limit accommodated all the staggered demands; the 3,000 amp actual limit did not.

The Well-Earned Fine

On February 25, 2000, the NRC imposed an $88,000 fine for several violations of safety regulations. Events were happening at Indian Point faster than the NRC could write tickets—the operators manually tripped the reactor on February 15, 2000, due to a broken tube inside a steam generator. Workers had inspected that tube in 1997 and got indications it was damaged more than allowed by safety regulations, but had mis-diagnosed those indications without fixing the damaged tube.

Safety by Intent

An event happening a year earlier at the Davis-Besse nuclear plant in Ohio demonstrated how things should have gone at Indian Point.

The Event

The Davis-Besse pressurized water reactor was operating at 99% power on Wednesday, June 24, 1998. At 8:44 pm, a tornado touched down onsite. The control room operators manually tried starting both emergency diesel generators upon receiving reports of the tornado. One emergency diesel generator failed to start, so operators went to the diesel generator room and manually started it using the local panel. One minute later, at 8:47 pm, the tornado damaged the switchyard, disconnecting the plant from its offsite power grid and causing the automatic shut down of the reactor, turbine, and generator.

The operators declared an Alert, the second least serious of the NRC’s four emergency classification levels at 9:18 pm. Despite the tornado’s damage disabling two of the three telephone systems at the plant, workers completed all the emergency notifications by 9:36 pm.

The loss of power from the offsite grid, even with both emergency diesel generators running, meant that none of the reactor coolant pumps could be operated to force cooling water through the reactor core. The emergency response procedures directed the operators to cool down the reactor water at a rate of less than 10°F per hour to avoid forming a steam bubble in the upper dome space of the reactor vessel.

During the slow and steady cool-down, the temperature inside one of the rooms containing a running emergency diesel generator warmed to over its 120°F maximum design limit due to a faulty damper in its ventilation system. Workers installed portable cooling fans and monitored the emergency diesel generator’s parameters for indications of diminished performance. They continued running the emergency diesel generator, but declared it to be inoperable because of the high room temperature.

The plant’s technical specifications directed that the reactor water be cooled to less than 280°F within seven hours of one emergency diesel generator being declared inoperable. The lead operator decided that continuing the slow and steady cool-down was safer than accelerating it to drop below 280°F within the specified timeframe, so he invoked a clause in the NRC’s regulations that permitted a requirement to be intentionally violated if plant conditions warranted it.

The power company’s workers repaired damage to the offsite transmission lines and towers, restoring offsite power to the plant. The operators downgraded the emergency classification to an Unusual Event at 2:00 am on June 26 and terminated it later that day.

Probablistic Risk Assessment

These Indian Point and Davis-Besse events illustrate how probabilistic risk analyses are conducted for nuclear power plants in general and in response to specific conditions.

An event tree, in this case (Fig. 2) for a small-break loss of coolant accident (LOCA) looks at the array of systems installed to mitigate it. When that system functions successfully, the event tree moves upward and onward to the next decision point. When that system fails, the event tree moves downward to the next decision point. The probability of success and failure are derived from past experience.

Fig. 2

Fig. 2 (Source: NRC with annotations by UCS)

As the event tree illustrates, a failure and sometimes even multiple failures can be tolerated without leading to meltdown as long as some of the mitigation measures succeed. Called defense-in-depth, safety is best served when there are as many paths to Okay as possible and the odds of wandering down a path to Meltdown are as small as achievable.

Too many of the crossroads faced during the Indian Point moved along the negative path—the connections to offsite power were lost due to a flawed modification, one of the onsite backup power sources was lost due to another flawed modification, the operators failed to recognize and declare an emergency, and so on. While the event did not result in a meltdown, too many of the mitigation measures failed, pushing the reactor closer to meltdown than necessary.

The majority of the crossroads faced during the Davis-Besse event moved along the positive path—the failure to start an emergency diesel generator from the control room was remedied within two minutes, the loss of two telephone systems complicated but did not impair timely emergency notifications, the overheating of an emergency diesel generator was readily detected and appropriate compensatory measures put in place, and so on. The event did not result in a meltdown and the many successful mitigation measures provided a comfortable margin to a meltdown.

Nuclear plants are not meltdown-proof. Dodging a meltdown, as Indian Point did in August 1999, is great. Extracting lessons from near misses and effectively implementing solutions that make nuclear plants more meltdown-resistant is better.

—–

UCS’s Disaster by Design/ Safety by Intent series of blog posts is intended to help readers understand how a seemingly unrelated assortment of minor problems can coalesce to cause disaster and how effective defense-in-depth can lessen both the number of pre-existing problems and the chances they team up.

Posted in: Disaster by Design, Nuclear Power Safety Tags: , , , ,

Support from UCS members make work like this possible. Will you join us? Help UCS advance independent science for a healthy environment and a safer world.

Show Comments


Comment Policy

UCS welcomes comments that foster civil conversation and debate. To help maintain a healthy, respectful discussion, please focus comments on the issues, topics, and facts at hand, and refrain from personal attacks. Posts that are commercial, self-promotional, obscene, rude, or disruptive will be removed.

Please note that comments are open for two weeks following each blog post. UCS respects your privacy and will not display, lend, or sell your email address for any reason.

  • Leonard Suschena

    Well, it’s a good thing ConEd and NYPA got out of the nuclear business and Entergy took over.

  • atomikrabbit

    Maybe someone could explain to me why UCS is so “concerned” about an event at IP seventeen years ago that harmed no one (and even if it had progressed through several more failures would still have harmed no one), but has never once seen fit to analyze or discuss an actual failure at a renewable energy infrastructure site that killed 171,000?

    Should they give up the pretext they are “scientists”?
    Is their agenda a “concern” for human life and wellbeing, or is it something else?

    • JenniWest

      I agree, Atom. It smells like a witch hunt on nuclear to me. As well, adding in the other times we’ve seen the UCS be quite irrational about nuclear power.

  • Leonard Suschena

    “Battery 24 was another backup power supply to electrical circuit 6A. Thus, some of the vital equipment on electrical circuit 6A continued to receive direct current (dc) power from the battery.
    Bus 6A provides 480VAC to various safety related components. The motors are specifically designed to operate using 480VAC ONLY. These components will not run with DC power supplied to them, no way, no how. As a concerned nuclear engineer, I’m concerned that you have sufficient experience in your field of expertise to be commenting on nuclear power, since you don’t even know basic electrical principles.

    • atomikrabbit

      Good catch Leonard.

      As you said, the 125VDC battery banks supply backup DC power when their battery chargers lose AC power. This DC is used for control circuitry for motor operated valves and breakers, and also feeds inverters supplying 120VAC for vital instrument busses.

      The trouble is, guys like Lochbaum know a lot more about nuclear power plant engineering than Joe Public (as is entirely natural, as Joe and Jane have a lot more important things to deal with in their lives), but a lot less than any number of thousands of actual practicing nuclear plant engineers who don’t have the time or interest in blogging or social media.

      So to the public, not having the training, education, or experience to discern the difference, everything he says as plausible, if not authoritative. Meanwhile the press, who generally have no better background in science or engineering than the general public, speed-dial guys like Lochbaum or Lyman to provide “balance” every time there is a nuclear related issue.

      • Leonard Suschena

        Well Atomik, if this is the “director of the UCS Nuclear Safety Project, is one of the nation’s top independent experts on nuclear power.” UCS is in sad shape. Has he ever even driven past a nuclear plant? He knows nothing about power distribution, even less about tech specs, shutdown requirements and cool down rates. Obviously never worked at one.

        • atomikrabbit

          Actually, for an anti, I consider Lochbaum one of the more knowledgeable and less unreasonable. Rod Adams had a long chat with him recently: atomicinsights(dot)com/atomic-show-237-dave-lochbaum-ucs/

          The official bio from his Wiki article says “Prior to joining UCS in October 1996, Mr. Lochbaum served as a Senior Engineer for Enercon Services, Inc., System Engineer for General Technical Services, Reactor Engineer/Shift Technical Advisor for the Tennessee Valley Authority, BWR Instructor for General Electric, and Junior Engineer for Georgia Power.”

          Translation: “After getting a BSNE degree from UT in 1979, Dave got an entry level engineer job at Georgia Power (probably at Plant Hatch), then after post-TMI regs mandated them, quit GP for TVA to became a non-licensed Shift Technical Advisor. Then he quit TVA and spent a decade at several job shops as a contract instructor or engineer, so he can cite ‘working at’ dozens of NPPs. About 1993 he raised an issue his bosses disagreed with, elevated it beyond the NRC to a sympathetic member of congress, got media spotlighted as a whistleblower, and rode that into a position at UCS in 1996.”

          The thing about Lochbaum that frosts me is that he signed up to be an instructor at the NRC Technical Center in Chattanooga, allowed the taxpayers to pay his moving expenses back to his old stomping grounds in Tennessee, then quit after almost exactly one year (the minimum to not have to reimburse them). Now he uses that NRC credential on his inflated resume and his anti followers swoon at his “expertise”.

          In the commercial world, it can take nine months to even qualify as an SRO certified simulator instructor. I wonder how many classes he actually taught?

          • Leonard Suschena

            Good info Rabbit. But I can tell, that aside from what his resume says, he really knows very little plant operations or design.

  • Leonard Suschena

    “The plant’s technical specifications directed that the reactor water be cooled to less than 280°F within seven hours of one emergency diesel generator being declared inoperable.”

    Davis-Besse Technical Specifications: http://pbadupws.nrc.gov/docs/ML0830/ML083010076.pdf

    See page 3.8-1-2, item B does not require shutdown at all, as long as backup power sources are verified operable. If they are, then they can operate for 7 days with one DG inoperable.

    Page 3.8.1-4, F requires them be in Mode 3 in 6 hours be in Mode 5 in 36 hours.
    Mode 3 in RCS temperature less than 350 degrees and Mode 5 is less then 200 degrees.

    I’d don’t have D-B’s specific operating cool down procedures, but I asked the operator at another PWR, who’s cool-down rate of 100 degrees/hour on one unit and 60 degrees/hour on the other. Have no idea where you found 10 degrees/hour or 280 degrees.

    Also, if your going to site Tech Specifications, try Google, site name and Technical Specifications and it will bring up the actual technical specification from the NRC website.
    Might also help if you took some nuclear training so you can get to speed on nuclear because after all, you are the “director of the UCS Nuclear Safety Project, is one of the nation’s top independent experts on nuclear power.”

    • atomikrabbit

      Davis-Besse is a B&W PWR, but I’ve never heard of a Mode 5 change starting at 280 degrees.

      The 10 degree/hour cooldown limitation (as you say, normally 100/hr) may be a procedural requirement in their EOPs for when there is no power to the Control Rod Drive Mechanism fans, to prevent voiding in the upper head.

      David’s audience generally doesn’t know the difference between Tech Specs and applesauce, and just believes him implicitly.