Anatomy of a Sewage Leak

Last month's high-volume sewage spill into the Rappahannock River from a waterfront Caroline Street pump station was most likely caused by an improperly reinstalled sewage pump. It was the latest and most severe in a long string of technical issues whose repair set the stage for a rare double failure of two high capacity sewage pumps. The resulting spill caused a massive spike in E. Coli bacteria found in water samples taken downtstream of the leak in the days immediately afterward.

Alarm logs and maintenance records obtained through a Freedom of Information Act request detail an ongoing series of technical issues which culminated in one of the two Caroline Street pumps failing within hours of being reinstalled and brought online following maintenance. Both pumps had been stripped down and inspected within 30 days of the failure.

Emails between city officials in the days after the leak detail how the situation was managed. News of the leak trickled out at first, receiving little media attention until five days afterwards. This, despite an effort by city staff to alert the public of the leak through notices posted to the city website and sent to local media.

A History of Problems

The Caroline Street pump station consists of two redundant wastewater pumps sitting on baseplates inside of a concrete wet well. Each pump is about the size of a large shop vacuum. If one fails, the other can take over. They don't run constantly, only when the water level triggers a sensor, and without the sensor they are blind.

The Caroline Street pumps had been causing headaches down at the city shop for months.

Since late July, the pumps had sent out a plethora of apparently bogus alarms: 151 pump failure alarms, 14 power failure alarms, and 14 high water level alarms. For weeks, no one seemed to be able to figure out why.

In late July and early August, electrical contractors inspected the pump on site for a damaged seal or high temperature conditions but found no problems in the pump. Notes from city alarm logs show that the problem would be "fixed" only to crop up again hours later.

Some amount of false alarms are expected in the normal operation of the sewage system, said Doug Fawcett in an interview. Usually, the problem would go away if, like a buggy computer, the pumps were simply rebooted. This option was becoming less and less effective over the summer and fall.

Between 10:36 p.m., Aug. 27 and 4:38 p.m., Aug. 28, the alarms for the Caroline Street pump station were cycling between "all normal", "pump failure" or "power failure" roughly once every minute.

"Have been receiving alarms from Caroline St. pump station constantly," reads one entry in the alarm log during the early morning hours on August 28.

False Alarms Set Stage For, But Don't Directly Cause, Failure

In early September a purchase order was approved for the Caroline Street pump to be sent out for emergency repairs to Industrial Motor Services in Richmond. The $4,000 bill included a full disassembly, inspection and cleaning of the pump's components.

An electrical contractor from Ashland based Enviromation, Inc. replaced the alarm panel to the tune of $3,750 on top of $604 in labor costs.

Enviromation also performed troubleshooting and repairs on the pump station following Hurricaine Irene for $1,400 in early September.

Still, the false alarms persisted off and on until October 5, when the first of the two pumps was sent to Industrial Motor Services and reinstalled on November 14. Four days later, on November 18, the second pump was taken out and sent off for the same repairs. It returned and was installed on November 22, the day the sewage leak began.

Within a few hours of installation both pumps would be inoperative. The first pump failure alarm was set off at 6:49 p.m. by 7:07 p.m., the high water alarm sounded. This time the alarm rang true, and public works staff were on the scene before the wet well had overflowed.

Down at the Caroline Street pump station, public works staff peered inside a stinking wet well slowly filling with waste.

"Pump 2 failure light flashing," reads log notes left by the night crew which discovered the leak. "Pushed reset button, nothing. Well was a little high, no pump came on."

There, beneath the muck, newly re-installed pump 2 had moved off of its baseplate and severed a wire powering pump 1's water sensor. Without it, the redundant pump couldn't turn on.

By 8 p.m., raw sewage was trickling into the Rappahannock, according to a environmental pollution incident report filed with the DEQ by city staff.

The City's Response

Overnight, the situation was attended to by the public works night crew. But lacking a backup pump, there wasn't much they could do. The response kicked into high gear the next day.

Within City Hall, word of the leak spread fast. Dave King, assistant director of public works, briefed Fawcett on the situation by phone shortly after 7:30 a.m. on November 23. At 9:08 a.m., City Manager Beverly Cameron sent an email to members of the City Council alerting them of the situation.

"Obviously this is a serious matter and a reportable event to DEQ," wrote Cameron. "We will work around-the-clock to fix this problem….sorry for the bad news."

At the pump station, the first order of business was to get a backup pump and generator in place to drain the wet well and haul the broken pumps out for repair, a task which lasted much longer than normal. Two parts needed to run the pump, rented from Ashland based contractor Godwin Pumps, had to be gathered from disparate construction sites in Richmond and suburban Maryland. Once those pieces were in place at around 2:47 p.m., it took another five hours to get the pump working. In all, it wasn't until almost 24 hours after the leak was discovered that the backup was brought online.

"We have always relied on Godwin Pumps in situations such as this, and they have always responded promptly 24/7," wrote Fawcett in an email to  Cameron around 8 p.m. on November 23, shortly after the backup pump was brought online. "This is the first time they've had any significant difficulty getting the pump to operate after it has arrived and been set up."

The leak dramatically raised the levels of harmful E. Coli bacteria in the river downstream from the leak thousands of times over normal background levels. Using a bacterial analysis called the Most Probable Number technique, the first set of samples came back with E. Coli MPN counts averaging roughly 6,500. It would be a week after the leak was first discovered before the E. Coli MPN counts dipped back down to normal MPN counts between 90 and 180.

Trouble Leaking the Leak

It didn't take long for City Officials to make an effort to alert regulators and members of the public of the situation. Signs were ordered to be posted at the City Dock and staff at the non-profit river stewardship Friends of the Rappahannock were told of the leak.

At 9:31 a.m., Kimberly Klock, chief operator of the wastewater treatment plant, notified the Virginia Department of Environmental Quality. Localities are required to report leaks of more than 1,000 gallons of wastewater to VDEQ. 

At 9:47 , Fawcett drafted up a public notice to be posted on the city website and sent to local media. This notice was only sent to the Free Lance-Star, where it was emailed to city hall reporter Robyn Sidersky, editor Richard Hammerstrom and a newsroom email account that goes to every news editor. Despite this, the Free Lance-Star did not report on the matter until five days later when reporter Rusty Dennen broke the story on his blog.

"I can't directly communicate with large numbers of people without the assistance of the media," said Fawcett in an interview. "The local media has a way of reaching people that we don't have."

City officials also did not make use of the Fredericksburg Alert text-messaging system, currently used by a variety of city departments to keep residents updated on park closures, school activities and weather alerts.  Fawcett says that the option to use the system never occurred to Public Works staff.

"We didn't make a conscious decision to not use it in this case," said Fawcett. "If a similar incident happened today we would certainly use the alert system."

Lessons Learned?

The two failed pumps are now back in operation at the Caroline Street Pump Station. City officials have about as clear of a picture of the causes of the leak as they expect to get, says Fawcett.

Fredericksburg officials are now in the midst of a review of the city's response to the spill. The effectiveness of the city's communication of the situation to the general public and the length of time it took to get a backup pump running at the site are two main focuses of review, according to Fawcett.

So, is this another story of aging infrastructure failing the public? Not necessarily.

"If the paper trail said that these pumps were 40 or 50 years old and hadn't been maintained regularly, then maybe," said Fawcett. "I don't think that's really the case here. We have rebuilt both of the pumps within 30 days. I would certainly lean towards saying that this is a different set of circumstances."

Indeed, the full record of the city's troubles at the Caroline Street Pump Station show city staff promptly responding to technical difficulties.

"We obviously take the situation very seriously," said Cameron in an interview. "From my perspective public works responded and did all that they could given the resources that they had."

Correction: The city's email to the Free-Lance Star at newsroom@freelancestar.com only goes to the news editors and not the entire newsroom.