The Story of a Troubled Tank

Tuesday, 31 August 2010 09:52 By Richard A Oppel Jr and Richard A Fineberg, | Report | name.

Review of the Alyeska Pipeline Service Company's closely held internal investigation report on the May 25 oil spill at Trans-Alaska Pipeline System (TAPS) Pump Station #9 (PS9) reveals a disturbing picture of chronic problems on the aging, partially revamped pipeline that carries upwards of 550,000 barrels of oil per day (bpd) from the northern edge of the continent on an 800 mile journey across Alaska to reach tankers headed to the West Coast.

About 5,000 barrels (210,000 gallons) of crude oil overflowed the pressure relief tank (TK-190) at PS9 when the automated TAPS control system released an estimated 33,000 barrels from the pipeline during an emergency shutdown. The spill was set in motion by cascading events that began when the pipeline's critical control systems crashed at PS9 due to the failure of an uninterruptible power supply (UPS) battery system that was supposed to be rock-steady. The spilled oil remained within the lined retaining walls of the station's tank farm compound, but the significance of the event transcends the limited environmental consequences of the spill itself and revolves around this question: Was this spill an anomaly, or was it a harbinger of future mishaps on the line?

TAPS resumed shipping three days after the spill and presently carries an average of 550,000 barrels per day (bpd) with a gross market of more than $40 million for roughly 3% of the nation's daily liquid petroleum consumption. (1) Meanwhile, this writer's review of the background to the May 25 spill reveals new facts and troubling questions:

  • When pipeline was restarted May 28, the pipeline operators did not know what caused the UPS system to fail, leading to the spill and shutdown. As a condition of restart, the federal Pipeline and Hazardous Materials Safety Administration (PHMSA) required Alyeska to send a special oversight group to PS9, a normally automated facility, to watch the pipeline 24 hours a day.
  • Nearly three months later, TAPS is still operating with the cause of failure unresolved (insofar as is publicly known), and with the same special watch team in place. The prolonged, temporary staffing of the normally unmanned pump station is an indication of the unease that hangs over the aging, reconfigured pipeline as information on the spill slowly comes to light.
  • According to Alyeska's closely held internal investigation report, the pipeline company failed to address a string of separate warnings and lessons from prior incidents that should have alerted the pipeline operators to potential risks during the planned maintenance work during the May 25 shutdown that went awry. The investigation report levels frank criticism at Alyeska management for these oversights.
  • In both press releases and a company-wide internal message July 1 summarizing the internal review, Alyeska failed to present clear and complete descriptions of the May 25 events and relevant past incidents. Alyeska's selective release of information has deflected public attention from a series of troubling mishaps, the latest of which appears to have been triggered by the pipeline company's own operating, maintenance and management deficiencies.
  • Two salient questions emerge from this analysis: If nobody had been present May 25 at PS9, a normally automated facility, when and how would this spill have been detected? Does Alyeska's troubled electrification and automation makeover, formally sanctioned in 2004 and now in place at three of the pipeline's four operating pump stations, increase the pipeline's vulnerability to a major spill?

Concerns about the safety of TAPS operations take on added significance in the glare of the fire and explosion that wrecked BP's Deepwater Horizon oil platform five weeks before the PS9 spill. BP, the major owner of TAPS with a 46.93% share, (2) supplied Alyeska with its current President, Kevin Hostler. The Alyeska president, who came to the pipeline company in 2005, announced his planned retirement July 7, while congressional staffs were investigating worker complaints that Alyeska's repressive and cost-cutting environment jeopardizes safe operations. (3)

A July 15 congressional staff memorandum provides additional perspective on TAPS and other pipeline issues, while this writer's May 5, 2010 briefing on TAPS issues, prepared three weeks before the spill, will introduce readers to the range of challenges that Alyeska faces. Additional information to help readers evaluate Alyeska's previously unreleased incident investigation report on the May 25 mishap will follow, presented in four steps: (1) inquiry into initial accounts of the May 25 incident; (2) review of PS9's critical position as the final pumping station in the pipeline's reconfigured, automated operating framework; (3) examination of mishaps at PS9 that preceded the events of May 25, which may be understood as inadequately addressed warning signs; and (4) consideration of Alyeska's work environment, as experienced by employees. Taken together, these steps lay the groundwork for a summary and analysis of Alyeska's previously unreleased incident investigation report and subsequent concluding comments. 

1. "Power Failure:" Explanation or Mask?

In each of four press releases issued between the day of the spill (May 25) and pipeline restart three days later, Alyeska steadfastly asserted that the spill occurred when a power failure caused PS9 pressure relief tank valves to open "as they are designed to do." The press releases and daily fact sheets provided little substantive information on the cause and nature of the power failure and the cascade of events it set in motion, leading to the unrecognized oil discharge into the secondary containment area surrounding the 55,000-barrel tank. This account seeks to fill the gaps.

As reported, the May 25 spill occurred during a planned, 6-hour maintenance shutdown at PS9, after workers shut off the primary power feed from the local utility. The shut-off was the first step in a scheduled test of the station's emergency fire response system; the workers apparently assumed that the station's emergency power system for critical controls would come on immediately, as described in an Alyeska brochure on the pipeline's pump stations.

But that didn't happen.

When the UPS system that was supposed to provide emergency backup power for the station's critical communication and control systems did not deliver, the planned safety test turned into an actual emergency. Personnel on site for the planned maintenance tests were unable to restart the main power system and could not figure out the reason the UPS system did not work.

Meanwhile, the resulting break in electronic communications with the pipeline's remote operations control center (OCC) triggered an automatic diversion of oil from the main pipeline (already idling for the planned maintenance shutdown) to the pressure relief tank. The oil diversion from the main pipeline continued, unrecognized, until the relief tank was observed overflowing, about 40 minutes after the worker-initiated power failure. It took another 1-1/4 hours for workers to stop the overflow diversion by manually closing the relief valves that had opened automatically.

Alyeska officials have been reluctant to divulge information - if they possess it - that might shed critical light on the events of May 25. The pipeline company declined to release its June 22 internal investigation report, TK-190 Overfill Incident Root Cause Analysis Report And Post Accident Review, shielding the document from public review by stamping every page "[c]onfidential and proprietary information protected from public disclosure." The Alaska State Pipeline Coordinator's Office, responding to a public records request under Alaska statutes, released the company investigation report to this writer August 4, withholding ten attachments for security purposes and protection of trade secrets, as well as the names of persons interviewed. The SPCO release came two days after Jason Leopold published information about the report, with direct quotes from its findings, in the second of his Truthout investigative reports on TAPS problems. (4)

A major surprise in Alyeska's internal investigation report was that the investigating team had conducted a Technical Failure Analysis (TFA) devoted to the UPS systems and a protective breaker within the UPS system for critical controls that was discovered - apparently sometime after the spill - to have tripped open, cutting off the theoretically uninterruptible power supply. The investigation report also noted that an external switch, apparently placed on the control system outer panel to show the breaker's position, was shielded by a protective guard that may have prevented the switch from functioning, and that PS9 maintenance records showed outstanding work orders to replace weak UPS system battery cells. (5) During testing, the investigating team found the UPS system was functioning correctly and therefore "was not able to specifically determine the physical failure cause;" in sum, "no specific root cause [could be] identified." (6)

An opaque summary Alyeska's investigation report findings transmitted to all employees by Operations Vice President Mike Joynor in a July 1 e-mail made no mention of: (a) the TFA line of inquiry; (b) that investigation's focus on the UPS emergency system for critical communication and control systems; or (c) the recommendation for additional testing and attention to "[a]ny known maintenance issues." (7) This e-mail acknowledged that a backup power system failure led to the communications beak between PS9 and the pipeline's supervisory control systems -- a step forward over Alyeska's earlier press releases, but still providing no details.

The June 22 investigation report makes clear what Joynor's July 1 summary did not: When the pipeline was restarted three days after the spill (May 28), the exact cause of the failed emergency power supply for the critical control systems at PS9 was unknown. It should be noted that by the time Joynor issued his opaque July 1 summary, speculation about what Alyeska was not telling the world was already a hot topic among the small universe of interested observers, including concerned Alyeska employees.

The mischief created by the absence of clear and well-documented public information about the causes PS9 spill was evident two weeks later at a July 15 congressional hearing on pipeline safety issues in Washington, DC. At that hearing, Alaska Congressman Don Young used a rather bizarre interpretation of the breaker information discussed above as he tried to convince his colleagues that the spill was not a big deal. The congressman offered this explanation of the spill:

"There was a human error factor - uh, there was a breaker that was forgot not [sic] to be checked . . . oil that did spill at a pump station was contained as it was designed, and it worked excellently." (8)

No big deal? To buy into Congressman Young's position, one must overlook at least three important points that follow from the information in Alyeska's closely-held incident investigation report:

  • Since the breaker worked properly when tested under various circumstances, some undiscovered factor must have been creating problems; what made the breaker trip open in the first place?
  • If the purpose of the UPS system is to provide seamless emergency power when needed, why would anyone design a UPS system with a protective breaker that could take it out of service, without a backup system or plan to ensure uninterrupted power?
  • Given the vulnerability of the UPS system - and, in train, the pump station's critical control and communication systems - to this single point of failure: what program deficiencies permitted the pipeline company to allow installation of a protective guard that (a) prevented the external breaker from working properly and (b) made the breaker more difficult to troubleshoot, without ever discovering the problem it had created?

In any event, Congressman Young's statement must have astonished persons who recognize the necessity of carefully checking both equipment and procedures to assure safe operations. The congressman's explanation was tantamount to saying, "So what if the pilot skipped pages on his pre-flight safety check?" The congressman's excursion demonstrates that when it comes to TAPS operations, seemingly simple answers frequently point the way to more important questions with broader implications.

Breaker confusion was just the tip of the iceberg. Before considering the submerged issues that Alyeska's incident investigators confronted and reported, background on TAPS facilities and operations will be useful.

2. PS9 and Strategic Reconfiguration (SR)

During its salad days in the late 1980s, when TAPS carried two million bpd of North Slope crude oil, PS9 was the eighth of ten pipeline pump stations. To relieve pipeline pressure during transient conditions and shutdowns Alyeska made use of the 55,000-barrel pressure relief tank at each of the pump stations along the 800-mile TAPS route between Prudhoe Bay and Valdez, along with a much larger pressure relief facility on the south flank of the Brooks Range at PS5 -- a facility that never pumped oil but has protected the pipeline on its descent from Atigun Pass since 1977.

Keeping pace with the decline in North Slope production, Alyeska has closed six pumping units since 1996, taking the pressure relief tanks at each station out of service. On the morning of May 25, PS9 provided the only relief tank on the pipeline between Valdez and the relief tank at PS5. (9) In the aftermath of the May 25 spill, the tank at PS9 is now out of service. (10)

With fewer pressure relief tanks, safe transport on TAPS is now more dependent than ever on its critical communications and control systems to identify changing hydraulic conditions and respond with proper sequencing and timing of valve opening and closure. (11)

Today, with throughput at less than one-third the historical peak, PS9 is also the pipeline's fourth and final pump unit, pushing North Slope oil across the last 250 miles of TAPS, over the Alaska Range and into the pipeline terminal at Valdez. (12) Under a massive, multi-year project known as Strategic Reconfiguration (SR) that began in 2004, at three of the four current pumping units on TAPS Alyeska has replaced jet engine-powered pumps with new electric pumps (more efficient and capable of handling today's lower throughput) and has installed new control systems that are supposed to be fully automated.

PS9, the SR flagship, was the site of Alyeska's first pump station conversion. (13) When the SR project was formally announced, Alyeska said the project would be completed by the end of 2005 at an estimated cost of $250 million, (14) but things did not go as planned: It was 2007 before Alyeska was able to put its flagship reconfigured and fully automated pump station into operation. The pipeline company now anticipates completion of SR at the fourth and final station - PS1 at Prudhoe Bay - in 2013. In the intervening years, the project cost has more than tripled. (15)

In addition to the fact that the cost and scheduling estimates were so far off the mark, a broad range of implementation problems call the planning for this project into question. (16) Of particular relevance here is the fact that the May 25 spill marked the third time in four years that Alyeska encountered problems associated with a diversion of pipeline flow to the relief tank at PS9 during an unplanned shutdown. The two prior events at the PS9 relief tank were:

  • the brief but intense fire on January 6, 2007, mentioned above, which began when an improperly deployed portable heater ignited gas vented from the relief tank during an unplanned pipeline diversion, that roared over the PS9 tank farm (but quickly subsided); and
  • a breakdown of the newly-installed pipeline control system two and a half months later, on March 22, 2007, during which the TAPS operations control center (OCC) temporarily went blind and lost communications with PS9, resulting in an ungoverned relief tank diversion.

These two events will be discussed in the following section.

3. Near Misses: The Troubled Tank at PS9

The January 2007 fire occurred when workers brought a portable heater into the tank farm and set up a temporary structure in order to work at a temperature of 25 degrees below zero. An electrician was checking valve wiring that had been modified for the new Safety Integrity Pressure Protection System (SIPPS), a component of the automated control system. In violation of basic safety procedures, the portable heater was placed beneath a relief tank vent and the electrician was working without adequate radio contact with the pump station control room, a few hundred yards away. The Joint Pipeline Office (JPO) report on the tank farm fire also noted the absence of an on-site alarm system that would have enabled workers to evacuate the tank farm area whenever a relief event was beginning. These unsafe working practices almost immediately caught up with Alyeska when the new SIPPS system sent an erroneous signal from 150 miles south of the pump station, automatically triggering an unexpected shutdown event. The pipeline controller tried unsuccessfully to warn the electrical worker as the relief tank began to fill, expelling flammable gasses that were quickly ignited by the improperly placed temporary heater. The fire that erupted high over the tank farm went out five minutes later, when the relief valves were closed: a terrifying near miss that might have destroyed the station. (17)

PHMSA, the U.S. Department of Transportation's pipeline safety unit, found Alyeska to be in violation of a slew of safety procedures at the PS9 tank farm that day and hit the company with a proposed penalty of $506,000 -- the largest the agency issued in 2007 against any pipeline operator. (18) Three years later, Alyeska is still contesting this proposed penalty. Its lengthy (308 page) and rather lame litany rejecting PHMSA's proposed penalty assessments, filed in February 2008, Alyeska claims, among other things, its radio communications were adequate and should not be penalized because the legal standards for portable radio communications are not explicit. (19)

The new, fully automated pipeline supervisory control system at PS9 was activated shortly after the fire at the PS9. On March 22, 2007, the new system malfunctioned. Alyeska was still in the process of "running in" new pipeline pumps and control equipment on that day. Although the technical language of that day's incident report is difficult to decipher, the parallels to the event at the same site three years later are unmistakable: On March 22, 2007, the OCC lost communication with PS9, an event that initiated an automatic pipeline shutdown and oil diversion from the mainline into the relief tank, while the OCC -- which was supposed to be in complete control of the automated system -- was temporarily blind and could not tell what was happening at the pump station. (20)

The incident report on the March 22, 2007 shutdown contained at least three references to possible warning and alarm system inadequacies. This problem had surfaced two months earlier in the near-miss fire event and arose again three years later with the May 25 relief tank overflow. (21)

Alyeska's report on the March 22, 2007 incident also discussed the relief tank overfill issue. At that time, the report noted, technicians were on site while the new system was being put into place; they halted the flow to the relief tank before it could overflow. Nevertheless, the 2007 investigation report expressed concerns about future occasions, when no one would be at the fully automated facility. Considering the implications of the fail-safe diversion to the relief tank, the investigators questioned the design strategy but noted that a future situation in which the OCC would not be able to observe and stop the overfill was "highly unlikely." The report, however, made no recommendation on this issue. (22)

In the January-February 2007 issue of Alyeska Monthly, President Hostler, referring to the death of an employee in 2006, acknowledged that, "[b]y our standards, we did not have a good year. . . . We will not tolerate poor safety performance on TAPS." But two months later Hostler proclaimed that "[e]very day our professional workforce continues to deliver high safety and quality standards to maintain integrity in all business practices." (23) In November 2007, the month that PHMSA issued its proposed penalty on the PS9 tank vent fire, in the hard-copy version of the year-end (November-December 2007) Alyeska Monthly newsletter, Hostler stated that the occurrence of the fire was "unacceptable," noting that "[w]e launched the Unified Plan to improve upon [work practices . . . and] identify improvement areas for safety, integrity management and risk management on TAPS." (24)

The Unified Plan, Alyeska management's response to problem indicators that included the two relief tank incidents at PS9 early in 2007, coordinated nearly two dozen Management Action Plans (MAPs) dealing with people, processes and facilties. (25) Some of these initiatives will be discussed later in this analysis.

4. Cost-Cutting and Open Work Environment Issues

Alyeska's investigation report of the May 25, 2010 spill incident provides a measure of the pipeline company''s success (or lack thereof) in addressing TAPS problems. Problems at PS9 -- presented here as probable warning flags unheeded and the results of the pipeline company's long-running struggle to implement the SR program -- intersect another set of TAPS issues: employee concerns and the pipeline owners' predilection for cost-cutting. This report does not assert direct links between specific fiscal and management pressures and the May 25 spill. However, it is clear as a general principal that cost-cutting measures are liable to have corrosive effects on safety -- and on reports, which are not written in a vacuum. Before considering Alyeska's internal investigation report on the May 25 spill, readers may wish to consider the following background information on Alyeska's work environment.

While cost cutting on TAPS has been frequently source of employee concerns, in the last year this pressure has become so intense that Alyeska worker appeals were somehow turned over to the BP ombudsman for investigation. (26)

At the July 15 hearing of the U.S. House Transportation and Infrastructure Committee's Subcommittee on Railroads, Pipelines and Hazardous Materials in Washington, DC (the same session at which Congressman Young sounded off about checking breakers) an Alyeska plan to economize by moving TAPS workers from Fairbanks to Anchorage was a focus of interest. Once again Congressman Young was off base as he tried to convince congressional colleagues that Alyeska is a separate entity that operates independently from BP. Speaking of the pipeline company, the congressman said: "[i]t's a separate entity in title, by itself. It does run itself by itself." (27)

A 2002 letter from the head of the TAPS owners committee to the Alyeska president, turning down project financing requests and asking instead for across-the-board budget reductions, demonstrates that Congressman Young is mistaken on this issue. Then, as now, a senior BP official was speaking to another member of the BP team, on loan to Alyeska from the parent company. This kind of misinformation flourishes in many circles in the 49th state, where, all too often, the industry - dominated by the three companies that control more than 90% of both North Slope production and TAPS - calls the tune.

Documentary indication of BP's cost-cutting proclivities in Alaska would not have surprised State Representative David Guttenberg of Fairbanks. Invited to testify before the U.S. House subcommittee, Rep. Guttenberg outlined his concerns with this issue:

My involvement in this issue began in December 2009 when I received word that Alyeska was planning to transfer a group of employees from Fairbanks to Anchorage. I was told that the engineers, technicians and scientists proposed for transfer are critical to monitoring and maintaining the integrity, public safety and environmental compliance of the Trans-Alaska Pipeline System (TAPS).

The proposed transfer raised alarm bells with me for two reasons: First, those were good jobs moving out of my community. Second, what standard did Alyeska use to determine that moving personnel responsible for the pipeline safety and integrity 350 miles away from the pipeline would be prudent and responsible? My initial thought was that it makes sense for these positions to be located in Fairbanks because it is a transportation-hub centrally located on the pipeline right-of-way. When something goes wrong or needs to be checked out on the pipeline, these employees can get to the problem location quickly. Anchorage is nowhere near the pipeline. In just about every scenario, it is quicker for these employees to reach the pipeline from Fairbanks.

When I began speaking out publicly, several Alyeska employees contacted me and confirmed my concerns. It was explained to me that many in the company shared my sentiment, but attempts to express those concerns were squashed at the highest levels by senior managers who feared retaliation for going against the mandate of Alyeska's president. At that point it became clear to me that Alyeska's "open-working-environment" was not working at all, allowing poor decisions to go unchecked that could have severe consequences for the state of Alaska. (28)

In December 2009, Alyeska President Kevin Hostler made no bones about his cost-cutting mind-set when he came to Fairbanks to explain to employees and the Fairbanks community Alyeska's plans to consolidate offices and transfer workers to Anchorage. According to the Fairbanks Daily News-Miner:

Hostler characterized the changes as part of a broad response to rising costs, including property taxes, and falling oil volume in the pipeline.

With throughput on the line expected to keep falling, the company's marginal costs - the costs of operation per barrel of oil - are expected to keep going up, Hostler said.

"We're trying to be as efficient and effective as we can," Hostler said, calling the restructuring a "wake-up call" to a statewide economy that leans heavily on the oil business. (29)

Underscoring the TAPS owners' obsessive interest in economizing is the fact that the TAPS owners did not authorize funding for the SR project until they were convinced that investment in the project would boost the rate of return on pipeline expenditures.(30) Many observers, including this writer, believe the industry's penchant for cost-cutting creates an unhealthy work environment that exacerbates the difficulties of ensuring safe operations on the aging pipeline.

5. Alyeska's Internal Investigation Report (Summary and Analysis)

The Alyeska internal investigation report offers a fascinating look at pipeline operations, the challenges Alyeska faces and the manner in which the company meets those challenges. The investigating team, working in this pressure cooker, delivered a report that contains facts and critical observations that some corporate managers and other die-hard defenders of Alyeska might not welcome. To appreciate how the internal investigation team has handled its task, in addition to understanding the context of this undertaking one must struggle through technical terminology, arcane methodology and issues and frequently awkward construction. To facilitate understanding of the incident investigation report, the following summary analysis and comments are offered.

The Alyeska internal investigation report employed two distinct approaches: The narrowly focused Technical Failure Analysis (TFA), discussed at the outset of this article, was not able to determine a specific cause of the identified breaker problem. On the other hand, the companion Root Cause Analysis (RCA), whose broader purpose was to identify potential management deficiencies, developed two root causes, three contributing causes and recommendations for addressing each. Here is a brief summary of the RCA results:

Root Cause #1 dealt with technical and design issues. The technical issues were narrow in focus, including electrical system issues such as the unexplained breaker that tripped open and the failure of alarm systems. The design issues were much broader in scope, focusing on subjects such as the overall design basis for SR, the use of relief tanks and the configuration of valves during pipeline shutdowns. (31)

Root Cause #2 asserted that Management Action Plans (MAPs) intended to implement past investigation recommendations have failed to arrest a pattern of significant incidents occurring on the pipeline. This finding also noted organization-wide inadequacies in communicating reports and recommendations and applying lessons learned to major maintenance activities and pipeline shutdowns. (32)

Contributing Cause #1 described less than adequate situational awareness in responding to abnormal situations on the part of both the maintenance team on site at PS9 May 25 and the OCC staff in Anchorage. (33)

The two remaining contributing causes identified weaknesses in shutdown preparation procedures and the standards, policies and administrative controls for the shutdown. (34)

Further discussion of these findings follows.

Root Cause #1 - Design Less Than Adequate (LTA). One of the subjects flagged for further work by Root Cause #1 of this investigation is the TFA's unsuccessful quest to explain what caused the failure of the PS9 emergency power system, discussed above. Whatever shakes out of this investigation, this much is clear: Alyeska has been less than forthcoming publicly about the unexplained and unexpected failure of the pump station's UPS system for critical control and communication systems. According to the Alyeska brochure describing a reconfigured pump station:

If primary power [supplied at PS9 by the Golden Valley electric utility] fails, emergency power will run life safety systems (fire and other critical systems such as critical controls, SCADA, telecommunications, and security systems) until backup power can be brought online. . . . critical systems will have up to 4 hours of emergency power via an uninterruptible power supply (UPS) system, which can be extended by a small 65-kilowatt (kW) diesel generator.

Information from the background on TAPS issues presented in preceding sections bears directly on many elements of Alyeska's multi-faceted investigation report. Consider, for example, the timing of the current investigation report recommendation for a broad review to validate the overall SR design: How did the massive and long-running SR project, now in its seventh year and in effect over much of the line, arrive at this late date without having addressed the design deficiencies uncovered May 25? Were past reviews adequate and kept up to date? These questions warrant further attention, but the immediate task is to summarize and understand what the investigators saw, reported and recommended.

Leaving the festering electrical system problems, the report's recommendation of design review for another SR component - the use and capacity of the breakout (relief) tanks - raises similar concerns. Despite the fact that the current investigation report includes the March 22, 2007 overfill near-loss as one of six "significant incidents . . . [that] demonstrate a trend of operational discipline deficiencies similar to those involved with the TK-190 overfill," facts and concerns about this prior incident critical to the present inquiry were not provided. What the May 25, 2010 incident investigation offered was a summary list of 18 key recommendations, lumped together from the six past incidents dealing with a variety of issues; two of those incidents were from other facilities and two were not concerned with relief tank overflow.

The current investigation report stated that "[t]hese recommendations appear to have been completed within the context of each individual incident in question and were believed to have been effective toward mitigating likelihood and consequences of further incidents."(36) If relevant past recommendations had, in fact, been effective, would the events of May 25 have transpired, and would new reviews be necessary at this time?

On point in this regard are two background items from the investigation report on the March 22, 2007 near-miss that stand out among the relevant information that did not find its way into Alyeska's current investigation report: (1) The conclusion to the report on the incident three years ago questioned the philosophy of the fail-safe system which allowed a possible relief tank overfill as protection against over-pressuring the mainline. As noted earlier, that report did not make a specific recommendation on this issue. (2) In the same discussion, however, the 2007 report estimated that if an overflow situation were recognized by persons present at the automated facility (if anyone happened to be there at the time), or at the remote OCC, they would have 70 minutes to address this highly unlikely situation;(37) incident chronology indicates that when the unlikely relief tank overfill occurred three years later, the tank began to overflow in about 40 minutes. (38)

Regarding the fail-safe mechanism that kicked into action when the UPS system crashed and PS9 lost electronic contact with the OCC, the press releases Alyeska issued the week of the spill steadfastly maintained that the valves that opened automatically to divert oil to the relief tank at PS9 functioned as designed. Some veteran observers question the design philosophy of the Alyeska operating system. "Fail safe [equipment] should not put you in an unsafe condition," says veteran pipeline safety engineer Richard Kuprewicz of Redmond, Washington. "This would suggest that the entire line needs to go through a hazard review."

In addition to studying the difficulties the pipeline operators encountered under SR and the problem of shutting down the pipeline without overflowing the relief tank at PS9, the incident investigation has also called for "review of the philosophy and operating practice regarding the configuration of the mainline valves (RGVs and BLs) during shutdowns." (39) In a rational world, one would think the system hydraulics and contingencies that affect pipeline shutdowns would have been fully considered before Alyeska and the TAPS owners started closing pump stations in 1996 and formally launched the SR automation program in 2004.

The final recommendation under Root Cause #1 is a plan to ensure implementation of upgrades at PS9 to incorporate improvements now in place at PS3 and PS4. This recommendation can best be understood in the context of SR history outlined earlier in this assessment. When the SR program was sanctioned and work began in 2004, Alyeska management anticipated installation would be completed by the end of 2005. In fact, the project proved so much more complicated and time-consuming than anticipated that it was 2007 before Alyeska put its first reconfigured pump station -- PS9 -- into service. In order to take advantage of knowledge that would be gained during the implementation process, management decided to perform the next installations (PS3 and PS4) in sequence. Now that PS3 and PS4 are reconfigured and running with newer electrical and automation equipment, PS9 has to catch up.

Root Cause #2 - Previous Incident MAPs & Lessons Learned LTA. The salient conclusion of this root cause is that "a pattern of significant incidents" continues on TAPS, despite "Management Action Plans (MAPs) intended to implement recommendations identified during investigations" and other "efforts . . . to address previous incidents and learn from work experience." The investigation report describes in less than flattering terms the organizational environment in which this pattern of significant incidents continues. According to the investigation report:

  • "Lessons Learned are routinely conducted throughout the organization for activities that include major maintenance completion, pipeline shutdowns, spill drills and incident response," but "[a]s an organization, we are not optimizing our opportunities to learn."
  • "Personnel are working hard to complete all requirements and remain in compliance, but the completion of actions intended to prevent incidents and the opportunities to learn from work activities have not been effective in influencing the culture or behaviors."
  • Remedial actions tend to be case-specific and lacking in follow-up to assure implementation and company-wide dissemination. "The Operations Incident Review Board has not been meeting as routinely as intended and has not effectively communicated incident learning's [sic] throughout the organization."
  • "There is usually no continuity between the Incident Investigation Team and the MAP Development Team."

To deal with these observations, the investigating team made the following recommendations:

  • Ensure PS9 incident and future incident investigation team representation during MAP development and implementation;
  • Enhance incident investigation and loss prevention manuals "to provide direction and detail on MAP purpose, accountabilities, Investigation Team/MAP continuity, development, communication, tracking, and validation. . . . Also, provide guidance to the Operations Incident Review Board to incorporate knowledge sharing and a learning culture;" and
  • "Improve methods to provide easy and reasonable access to incident investigation reports, Lessons Learned, risk assessments, and hazard analysis . . . . Establish expectations for personnel to utilize the tools to foster a culture of knowledge sharing and learning throughout the organization." (40)

The investigation report asserts subpar performance by portions of the Alyeska management structure involved with incident investigation, giving especially low marks to the MAP processes and making a bid for investigation team representation in MAP development and implementation proceedings in order to improve those undertakings.

Although these recommendations are constructive, past history suggests that they do not, in and of themselves, guarantee success. The Management Action Plans launched in 2007 in response to events that included the January fire and the March communications loss and shutdown at PS9 were initiated with similar hopes. For example, in plan approvals at that time, Alyeska promised to "significantly improve our Incident Investigation Process," committing to complete training on improved investigative techniques applied to serious incidents by March 31, 2008 and assuring "a much higher level of understanding and learning from incidents." (41) Another part of the MAP initiative, approved two months later, noted that "Alyeska has identified the need to improve its Incident Investigation and Root Cause Analysis Processes," with a goal of having a robust process that results in identification of true root cause(s) associated with near-loss and actual loss incidents." (42)

Contributing Cause #1 - Situational Awareness LTA. According to the investigation report, During the May 25 incident, OCC & field personnel "did not react in a manner that supported the safety and integrity of TAPS." Apparently they failed to recognize that when OCC lost electronic communications from PS9, oil would automatically be diverted to the relief tank, setting up conditions for the overflow that occurred. "This lack of action and preparedness prevailed in spite of a communication in 2009 which noted the fire system testing 'will not shut down the station unless the relief system UPS is not up to snuff'." The investigators also noted that "situational awareness was identified in the previous PS09 Piping Overpressure Event report and the fact that it was identified again as an issue during this incident provides direct linkage to Root Cause #2." (43)

It should be noted that in discussing this contributing cause the investigation report has quietly referred to two clear warnings in 2009 about issues that directly contributed to the May 25 incident:

  • The pipeline over-pressure event at PS9 in July; (44) and
  • In October 2009, PS3 workers identified the importance of close coordination with OCC and the risks of the UPS system failure when conducting fire system testing work and sent out an advisory e-mail discussing the potential problem. (45)

The investigating team recommended that Alyeska deal with this contributing cause by instituting a panoply of fixes that includes:

  • enhancing process safety management training;
  • improving situational Awareness training programs;
  • assessing industry best practices to improve management of OCC alarms; and
  • enhancing investigation and lessons learned processes by incorporating a focus on situational awareness deficiencies.

Both the statement of the cause and the proposed actions make sense. The warnings cited above provide additional support for the assertion that the significance of the May 25 incident is not the spill itself, but the cause it gives for concern with Alyeska's performance, illustrated by the multiple institutional failures that were evident at PS9.

Contributing Cause #2 - Safe Operating Committees LTA. The crux of this issue is the absence or inadequate performance of Safe Operating Committee exercises in preparation for the May 25 shutdown and fire safety testing at PS9. The recommendation: establish clear guidelines regarding occasions for and conduct of this standard preparatory procedure.

Contributing Cause #3 - Standards, Policies, and Administrative Controls (Procedures) LTA. The report identified a number of other procedures, standards, policies and administrative controls that require improvement to resolve inconsistencies and improve communications during shutdowns. The recommendation is simple: Determine what's right, and do it.

6. Conclusions

The environmental, social and economic consequences of the Alaska spill are in no way commensurate with the Gulf catastrophe, which killed 11 people and unleashed the ongoing environmental disaster on the Gulf of Mexico. Nevertheless, BP's heavy-handed assertion of an owner's prerogative on TAPS and the pipeline's recurrent problems bear striking similarities to the conditions that led to the Gulf Deepwater disaster. Mounting evidence -- much of it compiled and chronicled recently by investigative reporter Jason Leopold of Truthout -- confirms TAPS employee concerns that a repressive, cost-cutting work environment jeopardizes safe operations.

Meanwhile, the recurrent near-miss mistakes at PS9 summarized in this article call into question Alyeska's ability to manage its operations in a manner that will deliver safe operations. The record suggests that there are huge potential risks of more serious events to come if Alyeska continues to fail to get things right on TAPS.

Some observers may take comfort in the fact that the Alyeska investigation report on the May 25 spill at PS9 candidly acknowledged that, "[d]espite the efforts made to address previous incidents and to learn from previous work activities, there continues to be a pattern of significant incidents occurring." (46) However, as noted in this analysis, the investigation report recommendations to address current problems at PS9 give cause to recall promises made after similar near-miss events at the same troubled station and tank in 2007.

In this regard, it must be noted that the names of the two senior Alyeska signatories to the current investigation report - Senior Vice President of Operations Joynor and Director of Health, Safety and environmental Quality Rod Hanson - appear on other relevant documents. Both are signatories to a 2007 MAP document, quoted above, approving a promise to "significantly improve our Incident Investigation Process." (47) Hanson is also the signer of the second 2007 MAP plan approval referenced above, while Joynor sent the watered-down July 1, 2010 e-mail to all employees less than ten days after signing the current incident investigation report.

It is not clear from the available record whether Alyeska management team deserves more commendation for candor than criticism for failure to deliver. But the presence of the same names at both ends of this tunnel of mishaps calls attention to the small number of persons responsible for the company's large number of challenges. The possibility that a very small cadre of people maintain an effective choke-hold on decision-making at Alyeska may help explain why the pipeline company experiences seemingly perpetual difficulties establishing and maintaining a truly open work environment and a safe pipeline.

In any event, Alyeska's record stands in marked contrast to the company's carefully nurtured public image of a company whose "environmental program is infused in everything we do," a company that "proactively minimizes environmental impacts." (48)

The similarities between the 2007 and 2010 statements urging improvement to incident investigation and follow-up efforts suggest that Alyeska may be reinventing a wheel that, to date, has not rolled smoothly toward an open work environment, where robust discussion would foster safe operations. The pipeline company's reluctance to disclose information indicates flat spots in the wheel. However understandable Alyeska management's desire to avoid embarrassment may be, the company's withholding of its findings is not consistent with the spirit of the investigating team's recommendations which advocate access to incident investigation reports and related materials.

Despite the problems at PS9, during the first two months after the spill TAPS carried an average of about 550,000 barrels per day (bpd) - roughly 100,000 bpd less than the pipeline's pre-spill throughput. (49) With oil trading at an average price of about $75 per barrel during this period, every day the Alaska crude oil flowing through TAPS brings in more than $40 million in gross revenue, including an estimated $11 million in net profits. Most of this money goes to BP and two other companies -ConocoPhillips and ExxonMobil by dint of their overlapping interests in TAPS and North Slope production interests. Together, these three companies take upwards of 90% of the industry's net profits from the North Slope and TAPS. The state of Alaska takes in an even larger share of the gross revenue than the industry -- on the order of $15 million per day.(50)

The role the aging pipeline across Alaska plays in this economic dynamo (frequently considered on this web site in past pipeline tariff analyses), will receive further attention at a later date. For the moment, it will suffice to note that at today's oil prices total TAPS costs account for approximately six percent of the gross revenue generated by North Slope crude oil; this small percentage, by the way, includes a guaranteed profit for TAPS owners on pipeline investment and operating costs as a regulated utility. Under these circumstances, one might think the industry would spare no expense to guarantee safe transport of oil. But there remains a curious contradiction between (1) the essential role TAPS plays in this highly profitable economic enterprise and (2) the apparent risks Alyeska and the pipeline owners take with its cargo - and with Alaska's environment. This dissonance demands further inquiry.


(1) Aug. 19, 2010 ANS production = 619,436 barrels, avg. price (8/1 - 8/20) = $77.41; total domestic daily consumption, July 2010 = 19,063,000 barrels. See: Alaska North Slope production and price: Alaska Dept. of Revenue, "ANS Oil Production," accessed Aug. 20, 2010 at and "Crude Oil Prices, Bloomberg," accessed Aug. 20, 2010 at U.S. daily consumption: U.S. Energy Information Administration, "Petroleum Trade: Overview," Monthly Energy Review, July 2010, p. 41 (Table 3.3a; "Product Supplied").

(2) BP, with a 46.93% share of TAPS, is joined on the pipeline by ConocoPhillips (28.32%), ExxonMobil (20.34%), Koch Industries (3.08%) and Unocal Pipeline Co. (1.36%.(Facts) Together, these three transnational corporations own 95% of the pipeline and control approximately the same percentage of North Slope production. Alyeska Pipeline Service Company, the facts: trans alaska pipeline system, 2007, p. 6. Together, the three transnationals own 95% of the pipeline and control approximately the same percentage of North Slope production (production shares estimated from Alaska Dept. of Natural Resources data).

(3) See: Subcommittee on Railroads, Pipelines, and Hazardous Materials Staff, "Summary of Subject Matter - Hearing on 'The Safety of Hazardous Liquid Pipelines (Part 2): Integrity Management'," July 14, 2010, pp. 5-9. (Memorandum to Members of the Subcommitee on Railroads, Pipelines, and Hazardous Materials, Committee on Transportation and Infrastructure, U.S. House of Representatives.)

(4) Jason Leopold, "Confidential Report Blames BP Executive For Distress at Alyeska Pipeline," Truthout, August 2, 2010 (accessed Aug. 20, 2010 at; and "Dangerous Cost Cuts at Alyeska Pipeline: 'Yet Another Example of How BP Runs Things'," Truthout, July 6, 2010 (accessed Aug. 20, 2010 at

(5) Alyeska Pipeline Service Company, TK-190 Overfill Incident Root Cause Analysis Report And Post Accident Review (TK-190 Overfill Incident Review), June 22, 2010, pp. 8, 11 and 12 and Appendix 8 (Technical Failure Analysis), pp. 1-4.

(6) TK-190 Overfill Incident Review, pp. 4 and 11.

(7) "Technical Failure Analysis," p. 4.

(8) Subcommitee on Railroads, Pipelines, and Hazardous Materials, "'The Safety of Hazardous Liquid Pipelines (Part 2): Integrity Management'" (Hearing), July 15, 2010 (hearing accessed July 15, 2010 at

(9) the facts: trans alaska pipeline system, 2007, p. 34.

(10) Michelle Egan, Communications Director, Alyeska Pipeline Service Company, Aug. 5, 2010 (e-mail).

(11) For a brief description of TAPS operations factors, see Alyeska Pipeline Service Company, Operating the Trans Alaska Pipeline, June 1988, pp. 15-16.

(12) the facts, 2007, passim.; Alyeska Pipeline Service Company, "Pipeline Facts" (on-line), at (accessed Aug. 20, 2010).

(13) Alyeska Pipeline Service Company, "Strategic Reconfiguration" (on-line), at (accessed Aug. 20, 2010).

(14) Alyeska Pipeline Service Company, "Pipeline Reconfiguration Project Overview: Pump Stations and Control Systems Upgrade -- Project Completion by End of 2005," March 2005, pp. 2, 4 (accessed Aug. 20, 2010 at; and "$250 Million TAPS Upgrade Approved: Alyeska starting biggest TAPS project since construction," Alyeska Monthly (on-line newsletter), March 2004; and "Pipeline Facts" (on-line), ibid.

(15) PS1: Alan Bailey, "Trimming back: 60 jobs at Alyeska to go in 2010 as pipeline oil flow continues to decline; pump station 1 electrification to be delayed by one year," Petroleum News, Nov. 15, 2009. Costs Tripled: In 2007 the Alaska Department of Revenue reported that the estimated expenditure for SR was approximately $750 million (Alaska Department of Revenue, Fall 2007 Revenue Sources Book, p. 44).

(16) See: Richard A. Fineberg, "Trans-Alaska Pipeline System Strategic Reconfiguration: A Narrative Case Study," June 4, 2009 (prepared for the Alaska Forum for Environmental Responsibility and the Alaska Wilderness League; submitted as public comment to the Alaska Department of Environmental Conservation re: State of Alaska Oil & Gas Infrastructure Risk Assessment Project; accessed Aug. 20, 2010 at

(17) See: Alyeska Pipeline Service Co., "Pump Station 9 Tank Vent Fire Root Cause Incident Investigation & Executive Summary - Final Report," March 9, 2007;" and Joint Pipeline Office, "Investigation of the January 6, 2007 PS 9 Tank Farm Fire," March 7, 2007 (prepared by Ray Ellevan [Alaska Dept. of Labor Safety Liaison]; Technical Report Number ANC-07-E-001).

(18) U.S. Department of Transportation, Pipeline and Hazardous Materials Safety Administration (PHMSA), Notice of Probable Violation, Proposed Civil Penalty and Proposed Compliance Order ("NOPV;" CPF 5-2007-5041), p. 12; and Notice of Amendment (CPF 5-2007-5042M), Nov. 27, 2007 (letters from Chris Hoidal [Director, Western Region, PHMSA] to Mr. Jim Johnson [Pipeline Vice President, Alyeska]).

(19) Alyeska Pipeline Service Co., "Re: Notice of Probable Violation CPF No. 5-2007-5041" (letter from Joseph P. Robertson, P.E. [JPO/DOT Liaison Director, Alyeska] to Chris Hoidal [Western Region Director, PHMSA], with attachments [308 pages], including Response to Notice of Probable Violation #9); as of Aug. 22, 2010, PHMSA enforcement records indicate this case is still open.

(20) Alyeska Pipeline Service Co., "March 22, 2007 Pump Station 09 Shutdown Incident," submitted to the Joint Pipeline Office April 17, 2007. See also: John Governale (JPO), "Site Visit top PS-9 for Post Startup Oversight of Pipeline Strategic Reconfiguration Project, 5-6 April 2007," April 10, 2007 (TAPS Technical Report; JPO No. ANC-07-E-012).

(21) "March 22, 2007 Pump Station 09 Shutdown Incident," p. 3.{"insufficient warning methods"), p. 9 ("There was no alarm to OCC indicating the faulted condition."); and p. 18 ("Alarm management is currently under review for the SR project. . . . The number of alarms available in the SR system configuration is quite large. There are a large number of automatic diagnostic features . . . that can flood the operator with too much information.").

(22) "March 22, 2007 Pump Station 09 Shutdown Incident," pp. 21-22 (Conclusions). The concerns articulated by the 2007 investigating team on this issue follow:

Worst Case Scenario for Tank Overflow

If there had been no one at PS09 and if OCC had not recognized the abnormal situation through other means, the relief event could conceivably have continued, and after approximately 70 minutes from the start of the relief event, the tank would overflow. The overflow would no longer be measured and accounted for as tank inventory in the Leak Volume Balance detection system, soon afterwards the leak detection system would have alerted the OCC controllers that a loss was occurring on the pipeline. The overflow would be contained within the tank farm dike area. The dike would have provided an additional 6-hour of containment at this relief rate.

This scenario is very uynlikely after implementation of the recommendations from this report.

Design Strategy:

The design strategy for this type of malfunction was to allow the oil to relieve in the tank until the station was recognized through other means by the OCC controller, who would then close the RGVs in the event the relief could not be stopped either by raising the set point or closing the RB block valves. This situation is highly unlikely, but the strategy is to overfill the tank which is contained within a dike area if absolutely necessary instead of risking the integrity of the pipeline by closing the relief valves at the wrong time."

(23) Kevin Hostler, "Committed to Improving Safety on TAPS" (President's Message), Alyeska Monthly, Jan. / Feb. 2007(, and "Open Work Environment on TAPS" (President's Message), Alyeska Monthly, April 2007 (

(24) Kevin Hostler, "Moving Into the New Year" (President's Message), Alyeska Monthly, Nov. / Dec. 2007 (hard copy). The article with this statement was pulled from the internet version of the newsletter and replaced with a repeat copy of the previous month's "President's Message," in which Hostler talked instead about how much he enjoyed the opportunity to visit rural Alaska. See: "Visiting Rural Alaska" (President's Message), Alyeska Monthly, Nov. / Dec.. 2007 (, and

(25) See: Kevin Hostler, "Unified Plan developed to improve safety, integrity management on TAPS" (President's Message), Monthly News, Jan. 2008 (accessed Aug. 20, 2010 at and Alyeska Pipeline Service Company, "Unified Plan Presentation to TAPS Oversight & Regulatory Agencies," Nov. 28, 2007.

(26) See, for example: Jeff Richardson, "Alyeska plans to transfer Fairbanks jobs," Fairbanks Daily News-Miner, Feb. 21, 2010; and Dermot Cole, "Alyeska Move to Anchorage won't pencil out, critic charges," Fairbanks Daily News-Miner, c. Feb. 21, 2010.

(27) "'The Safety of Hazardous Liquid Pipelines (Part 2): Integrity Management'" (Hearing).

(28) Rep. David Guttenberg, "Testimony of Representative David Guttenberg," Subcommitee on Railroads, Pipelines, and Hazardous Materials, "'The Safety of Hazardous Liquid Pipelines (Part 2): Integrity Management'" (Hearing), July 15, 2010.

(29) Christopher Eshleman, "Alyeska to trim union role, Fairbanks offices," Fairbanks Daily News-Miner, Dec. 3, 2009, p. A1.

(30) See: Jerry Allison (Alyeska) and Pat Flood (Conoco-Phillips), "Pipeline Elecrification: Analysis of Risks and Cost Probability Distribution," Dec. 2003. (This Alyeska Pipeline Service Company document found that "[e]conomic risks are substantially greater for inertia than for electrification, with an estimated base case after tax net present value on investment of 27% for electrification of four pump stations.)

(31) TK-190 Overfill Incident Review, p. 12.

(32) TK-190 Overfill Incident Review, pp. 12-13.

(33) TK-190 Overfill Incident Review, pp. 13-15.

(34) TK-190 Overfill Incident Review, pp. 15-17.

(35) Alyeska Pipeline Service Company, Strategic Reconfiguration Power Generation System (accessed July 14, 2010 at

(36) TK-190 Overfill Incident Review, p. 9.

(37) "March 22, 2007 Pump Station 09 Shutdown Incident," pp. 21-22.

(38) TK-190 Overfill Incident Review, pp. 7-8.

(39) TK-190 Overfill Incident Review, p. 12.

(40) TK-190 Overfill Incident Review, pp. 12-13.

(41) Alyeska Pipeline Service Company, "Management Action Plan -- in response to Common Cause Assessment From Serious Incident Reports (Conger & Elsea - June 25, 2007)" October 5, 2007," pp. 4, 6.

(42) Alyeska Pipeline Service Company, "Management Action Plan -- for Incident Investigation and Root Cause Analysis Process Improvements Initiative," Nov. 20, 2007, p. 2.

(43) TK-190 Overfill Incident Review, pp. 13-14.

(44) TK-190 Overfill Incident Review, p. 13.

(45) TK-190 Overfill Incident Review, p. 10.

(46) TK-190 Overfill Incident Review, pp. 12-13.

(47) "Management Action Plan -- in response to Common Cause Assessment From Serious Incident Reports (Conger & Elsea - June 25, 2007)" October 5, 2007," p. 4.

(48) Alyeska Pipeline Service Company, "Alyeska's Environmental Program," (accessed Aug. 18, 2010).

(49) See: Alaska Department of Revenue, "ANS Oil Production," accessed Aug. 20, 2010 at, etc.

(50) ANS daily profit and state "take" estimated from: Alaska Department of Revenue, "Basic Data Used for ANS Oil & Gas Production Taxes," Revenue Sources, Fall 2009, Fig. 4-7, p. 34.

Last modified on Tuesday, 31 August 2010 12:36