DALLAS - It was the start of a steamy Friday two Augusts ago when Jason Whisler settled in for a working breakfast at the Coffee Ranch restaurant in the Texas Panhandle city of Borger. The most pressing agenda item for city officials that morning: planning for a country music concert and anniversary event.
Then Whisler's phone rang. Borger's computer system had been hacked.
Workers were frozen out of files. Printers spewed out demands for money. Over the next several days, residents couldn't pay water bills, the government couldn't process payroll, police officers couldn't retrieve certain records. Across Texas, similar scenes played out in nearly two dozen communities hit by a cyberattack officials ultimately tied to a Russia-based criminal syndicate.
In 2019, ransomware had yet to emerge as one of the top national security concerns confronting the United States, an issue that would become the focus of a presidential summit between Washington and Moscow this year. But the attacks in Texas were a harbinger of the now-exploding threat and offer a vivid case study in what happens behind the scenes when small-town America comes under attack.
Texas communities struggled for days with disruptions to core government services as workers in small cities and towns endured a cascade of frustrations brought on by the sophisticated cyberattack, according to thousands of pages of documents reviewed by The Associated Press and interviews with people involved in the response. The AP also learned new details about the attack's scope and victims, including an Air Force base where access to a law enforcement database was interrupted, and a city forced to operate its water-supply system manually.
In recent months, a ransomware attack led to gasoline shortages. Another, tied to the same hacking gang that attacked the Texas communities, threatened meat supplies. But the Texas attacks - which, unlike these prominent cases, were resolved without a ransom payment - make clear that ransomware need not hit vital infrastructure or major corporations to interrupt daily life.
"It was just a scary feeling," Whisler, Borger's emergency management coordinator, recounted in an interview.
In the early morning of Aug. 16, as most Texans were still asleep, hackers half a world away were burrowing into networks. They encrypted files and left ransom notes.
That afternoon, with the attack's impact becoming apparent, the city manager of Vernon emailed colleagues about a "ransom type" virus affecting the police department. The city near the Oklahoma state line could get back online by paying the $2.5 million the hackers were demanding, he wrote, but that was "obviously" not the plan.
"Holy moly!!!!!" replied city commissioner Pam Gosline, now the mayor.
The culprits were affiliated with REvil, the Russia-linked syndicate that last spring extorted $11 million from meat-processor JBS and more recently was behind a Fourth of July weekend attack that crippled businesses around the globe. In the Texas case, however, communities were ultimately able to recover most of their data and rebuild their systems without anyone paying ransom.
The hackers gained their foothold through an attack on a Texas firm that provides technology services to local governments, branching through screen-sharing software and remote administration to seize control of the networks of some of the company's clients.
Because the city had paid for offsite remote backup, Borger had the capability to reformat servers, reinstall the operating system and bring data back over. A newly purchased server that had yet to be installed came in handy. The police department, however, retained its data locally and the attack hampered officers' access to previous incident reports.
As they worked to resolve the problem, officials shared draft press releases that offered reassurances that critical emergency operations would continue and that the attacks weren't a reflection of any misstep by the city.
Other communities preemptively took potentially vulnerable systems offline. In the Austin suburb of Leander, the city shut off the program that police used to check license plates for 24 hours as IT staff worked to confirm that it hadn't been exposed.
Emails reveal moments of exasperation as problems persisted.
The impact wasn't limited to local governments. Sheppard Air Force Base confirmed to AP that its access to a statewide law enforcement database used for background checks on visitors was temporarily interrupted, causing delays for issuing passes. Operations were otherwise unaffected.
State officials didn't immediately know which communities had been victimized. They called around asking, "Were you impacted? Were you impacted? Were you impacted?" said Nancy Rainosek, Texas' chief information security officer.
"There was one place that we contacted and they said, 'no, no, we're not hit,'" Rainosek said. Then, days later, "they said, 'yes, we were.'"
State officials spent a full week inside their command post - built to withstand a nuclear blast - and used a map to chart the attack's spread. All told, some 23 government entities were ultimately shaded to indicate they'd been hit.
"It's a bit of a mind struggle because you're trying to stay focused and present on the folks that you know about," said Amanda Crawford, executive director of the Texas Information Resources Department. "But you're continually worrying about, 'Is there something you're missing? Or are there others, that you're going to get another call that somebody else has been hit?'"
By Wednesday evening, records show, most city services in Borger were restored, including utility payments, vital statistics and most employee computers. The situation had stabilized; the city ended up with about 80% of its data back and the concert Whisler was planning happened as scheduled.
Still, in a city with a roughly $31 million budget, Borger had overtime IT expenses to contend with and purchased $44,000 worth of new computers. It's invested in additional cybersecurity protections, including some $30,000 in annual costs for additional remote backup.
Borger officials in the weeks before the hack had discussed upgrading the threat level from cyberattacks. Those considerations are now more than theoretical.