NASHVILLE, Tenn. – The vulnerability of the telecommunications system in Nashville and beyond became clear Christmas Day when AT&T’s central office in downtown became the site of a bombing.
Mayor John Cooper called the blast on Second Avenue an attack on infrastructure. The effects of that attack are sure to ripple through the region for weeks, as the telecom giant scrambles to restore services while maintaining the integrity of an active investigation site teeming with federal agents.
State and local officials and experts say the fact that a multistate region could be brought to its knees by a single bombing is a “wake-up call,” exposing vulnerabilities many didn’t know existed and predicting it would lead to intense conversations about the future.
The bombing and the damage to the AT&T office was a “single-point of failure,” said Douglas Schmidt, the Cornelius Vanderbilt professor of computer science at Vanderbilt University.
“That’s the Achilles’ heel. The weak link,” he said. “When one thing goes wrong and everything comes crashing down.”
Now, the Tennessee Emergency Communications Board has called a special meeting for next week to address the “impact to 911 operations as a result of the bombing in downtown Nashville,” according to a public notice of the meeting set for Monday.
Crews hustle for heat:Vandalism leaves 3,500 customers without heat or hot water amid freezing temperatures in Colorado
While still piecing together a motive, investigators Monday said suspected bomber Anthony Quinn Warner sought “more destruction than death.”
He parked an RV outside the nondescript windowless red-brick building on the historic district, which houses a facility that includes connection points for regional internet and wireless communications.
Flames broke out in the building and 3 feet of water pooled in the basement. Temporary battery power kept services intact in the hours following the explosion, but fire and flooding damaged backup power generators to power those batteries.
The disruption brought communications in the region, from Georgia to Kentucky, to a halt, affecting 911 call centers, hospitals, the Nashville airport, government offices and individual mobile users. Issues with credit card devices hamstrung businesses big and small.
TBI director:Nashville bomber Anthony Q. Warner’s motive appears linked to ‘more destruction than death’
AT&T reported Monday morning that the majority of services in Nashville had been restored through a combination of fixes, including generator repairs and a temporary network set up at Nissan Stadium.
“Having a critical facility in a major metropolitan area next to a street without any other protections than a thick wall is crazy,” Schmidt said.
“The silver lining here is nobody was killed,” he said. “But this is a wake up call that, if people treat it right, will help with future situations and be better prepared.”
‘Our systems are not redundant enough’
When the situation settles down, state Sen. Paul Bailey, R-Sparta, who most recently served as chairman of the Senate commerce committee, hopes the Tennessee legislature can hear from AT&T representatives about what type of plan they’re implementing to prevent this type of outcome if another similar disaster occurs.
“They need to have better redundancies in place,” Bailey said, referring to AT&T’s backup systems to prevent widespread outages. “It’s just very concerning that we have 911 centers go down. Lots of emergency services losing communications. That’s really concerning to me.”
Nashville Metro Council member Freddie O’Connell, who represents the downtown area, said the city must also follow up on how to create more redundancy in critical communication systems in the aftermath of the bombing.
“How does a city as a whole function if we go through something like this again or a natural disaster?” he said. “We learned our systems are not redundant enough when one major provider goes offline.”
Police officers on the scene Friday were issued burner phones, according to Metro police spokesperson Don Aaron. Nashville’s police department uses FirstNet network, a priority network for first responders to use on existing AT&T cell towers for voice and data.
Nashville’s 911 line remained operational but officials were without access to administrative phone lines through Friday evening, according to Stephen Martini, director of the Nashville Department of Emergency Communications.
In the absence of non-emergency phone lines, residents were encouraged to request services through hubNashville online, which officials monitored for a three-day period.
Martini said communications to emergency personnel via radio was never impacted over the weekend.
He declined to share details on how the department remained operational, citing sensitive public safety information, but said a redundancy plan, dubbed the PACE method (Primary, Alternate, Contingent, Emergency), was in place.
Nashville’s director of information and technology services, Keith Durbin, said Verizon phones had to be driven to some staff on Christmas Day.
“This was one of the worst case scenarios that happened,” Durbin said. “… To have (AT&T)completely taken out … was even broader impact than we thought.”
Luckily, he said, none of the city’s “internal network backbone” was affected, with issues primarily coming from smaller Metro facilities. Some were continuing to experience issues Monday, including the Davidson County Clerk’s office.
The city was able to switch from AT&T to a secondary internet carrier Friday. But the city doesn’t have a backup for phone services. It’s something officials have considered in recent years.
Those talks, Durbin said, will be revived after the bombing. He said he’s confident it’ll now get wide support.
Bailey said he heard from 911 center directors in his district reporting outages nearly 100 miles away from Nashville. Residents in the area received reverse emergency calls to inform them not to dial 911, but instead to use another phone number to get in touch with dispatchers.
And then there were the retail stores, pharmacies, businesses and hospitals that were impacted, he said.
He credits AT&T for working quickly to restore service. But he said it’s concerning that one incident could wipe out so much of the region’s communications capability.
“This affected our entire Southeast region,” Bailey said. “There were multiple states that had issues because of this.”
But as for whether the Tennessee General Assembly wields much power to compel action from AT&T, “the short answer is no, we don’t,” Bailey said.
The state’s Public Utility Commission, a five-member board consisting of political appointees, also has limited ability to regulate for-profit communications companies. Much of that would be a federal issue, Bailey noted.
State and hospitals face outages
The city wasn’t alone in experiencing communication outages.
The Tennessee General Assembly, which has offices adjoining the Capitol downtown, also had outages over the weekend. The email system was down for a portion of the day Saturday, and staff were told to work from home Monday after the building experienced phone outages until Sunday evening.
State government office buildings remained closed Monday due to safety hazards that the outages continued to pose, said Lola Potter, spokesperson for the Department of Finance and Administration.
Fire and safety alarm systems in state buildings in Nashville still weren’t fully functioning.
Other state services were impacted over the weekend, though state employees found workarounds and alerted authorities in some situations, such as with reporting child abuse.
“Obviously we were concerned about it, but we took precautions by reaching out to law enforcement,” said Jennifer Donnals, chief of staff for the Department of Children’s Services.
A state web form and app that field child abuse reports remained in service, Donnals said, and staff in different regions also alerted major children’s hospitals around the state to the hotline being down.
The timing of the disruptions occurring on a weekend – and a holiday weekend at that – meant the agency would likely have been receiving fewer complaints than on weekdays anyway.
Meanwhile, hospitals in the region have also had to work around outages since the weekend, mostly stemming from their landline phone systems going down and being unable to receiving incoming calls.
Vanderbilt University Medical Center reported it was back in service by late Monday afternoon, while TriStar Centennial Medical Center was experiencing some intermittent outages.
Both hospitals had to set up new, temporary phone numbers for people to call, according to spokespeople for both hospital systems
Health records and other IT infrastructure needed to care for patients were not affected at Vanderbilt, nor were employees’ ability to make internal calls within the hospital.
There were remaining limitations to flight corridors in and out of the Nashville International Airport on Monday, but it did not have a significant impact on flight departures and arrivals, according to Tom Jurkovich, vice president of communications and public affairs for the airport.
Flights were grounded at Nashville Airport on Christmas afternoon due to telecommunication issues stemming from the explosion. Jurkovich estimated up to 45 of 116 flights scheduled for departure that day were delayed.
Follow reporters Yihyun Jeong and Natalie Allison on Twitter: @yihyun_jeong, @natalie_allison
More:The ways Donald Trump and Republicans have tried to overturn Joe Biden’s 2020 win
More:Red tape, confusion block COVID homeless aid from cash-strapped cities, counties across US
Source Article from https://www.usatoday.com/story/news/nation/2020/12/29/nashville-bombing-area-communications-network-exposed-achilles-heel/4070797001/
Comments