AT&T Restores Service After Massive, Nationwide Outage (cnn.com) 55
An anonymous reader quotes a report from CNN Business: AT&T's network went down for many of its customers across the United States Thursday morning, leaving customers unable to place calls, text or access the internet. By a little after 3 pm ET, roughly 11 hours after reports of the outage first emerged, the company said that it had restored service to all impacted customers. "We have restored wireless service to all our affected customers. We sincerely apologize to them," AT&T said in a statement. The company added that it is "taking steps to ensure our customers do not experience this again in the future."
The Federal Communications Commission confirmed Thursday afternoon that it is investigating the outage. The White House says federal agencies are in touch with AT&T about network outages but that it doesn't have all the answers yet on what exactly led to the interruptions. Although Verizon and T-Mobile customers reported some network outages, too, they appeared far less widespread. T-Mobile and Verizon said their networks were unaffected by AT&T's service outage and customers reporting outages may have been unable to reach customers who use AT&T.
Thursday morning, more than 74,000 AT&T customers reported outages on digital-service tracking site DownDetector, with service disruptions beginning around 4 am ET. That's not a comprehensive number: It tracks only self-reported outages. Reports had been rising steadily throughout the morning but leveled off in the 9 am ET hour. By 12:30 pm ET, the DownDetector data showed some 25,000 AT&T customers still reporting outages. By 2 pm ET, fewer than 5,000 customers were still reporting issues. Earlier Thursday, AT&T acknowledged that it had a widespread outage but did not provide a reason for the system failure. By late morning, AT&T said most of its network was back online, and it confirmed Thursday afternoon that service was fully restored. According to an anonymous industry source, the issue for the outage appears to be related to how cellular services hand off calls from one network to the next, a process known as peering. They said there's no indication that it was the result of a cyberattack or other malicious activity.
The FCC confirmed that it is investigating the incident. "We are aware of the reported wireless outages, and our Public Safety and Homeland Security Bureau is actively investigating," the FCC said in a statement posted on X. "We are in touch with AT&T and public safety authorities, including FirstNet, as well as other providers."
The Federal Communications Commission confirmed Thursday afternoon that it is investigating the outage. The White House says federal agencies are in touch with AT&T about network outages but that it doesn't have all the answers yet on what exactly led to the interruptions. Although Verizon and T-Mobile customers reported some network outages, too, they appeared far less widespread. T-Mobile and Verizon said their networks were unaffected by AT&T's service outage and customers reporting outages may have been unable to reach customers who use AT&T.
Thursday morning, more than 74,000 AT&T customers reported outages on digital-service tracking site DownDetector, with service disruptions beginning around 4 am ET. That's not a comprehensive number: It tracks only self-reported outages. Reports had been rising steadily throughout the morning but leveled off in the 9 am ET hour. By 12:30 pm ET, the DownDetector data showed some 25,000 AT&T customers still reporting outages. By 2 pm ET, fewer than 5,000 customers were still reporting issues. Earlier Thursday, AT&T acknowledged that it had a widespread outage but did not provide a reason for the system failure. By late morning, AT&T said most of its network was back online, and it confirmed Thursday afternoon that service was fully restored. According to an anonymous industry source, the issue for the outage appears to be related to how cellular services hand off calls from one network to the next, a process known as peering. They said there's no indication that it was the result of a cyberattack or other malicious activity.
The FCC confirmed that it is investigating the incident. "We are aware of the reported wireless outages, and our Public Safety and Homeland Security Bureau is actively investigating," the FCC said in a statement posted on X. "We are in touch with AT&T and public safety authorities, including FirstNet, as well as other providers."
Re:enable wi-fi calling (Score:5, Insightful)
It's not clear at this point that the problem was related to, or could have been avoided by, enabling wifi calling. Even with wifi calling enabled, the AT&T infrastructure that routes texts and calls still has to be operational, for your communication to succeed. Wifi calling only changes one hop in the chain: instead of connecting to a nearby tower, it gets to AT&T via the internet. Once there, your call still flows through AT&T's infrastructure like any other call.
Re:enable wi-fi calling (Score:5, Informative)
Re: (Score:2)
I have two AT&T phones. While in my house, I tried them both. One worked, and the other didn't.
Re: (Score:2)
Wifi texting worked for me even when I was getting nothing but SOS on the mobile side.
Re:enable wi-fi calling (Score:5, Funny)
Re: (Score:2)
^^^ Somebody mod the comedian up!
What is the real number? (Score:3, Interesting)
Re: (Score:2)
Only ATT would know and they're probably not exactly keen to publish that
Re: (Score:2)
Oh, they'll absolutely have to report it to the FCC.
These sorts of things make AT&T executives nervous. . . . . . especially with the company being trigger happy for the last several years with layoffs.
Re: (Score:2)
Hanlon crisis?
Re:overworked or undertrained (Score:5, Funny)
We are talking about ATT, I think we have to assume malice.
Re: (Score:2)
AT&T used to stick random junk fees onto our bill, such as "phone insurance", and were slow to remove them when we complained. I suspect Wells Fargo-ism.
Re: (Score:3)
Oh, no, trust me.
AT&T is far more incompetent than it is malicious.
Upper management is far too stupid / short sighted to be malicious unless it's by accident.
Re: (Score:1)
Re: (Score:2)
I completely agree. With proper regulation it is also a crime, because phone networks are critical infrastructure.
Re: overworked or undertrained (Score:2)
There is the troublesome word "other"
Outage Cause? (Score:5, Informative)
Years ago AT&T had a similar widespread outage on their long distance network. It turned out they had dropped a new software load into their switches overnight that brought the whole thing down.
Re: (Score:3)
Re: (Score:3)
Re: (Score:2)
According to some reddit posters, AT&T was having a problem with some new Cisco firmware. Take it with a grain of salt.
Re: (Score:2)
I wonder if they even QA tested these. :(
Re: (Score:3)
QA ?
HAHAHAHAHAHAHA
True story: AT&T is deploying untested Cisco hardware into production networks and is of the opinion that Cisco will fix the bugs as we come across them.
So, seeing random parts of the network catch fire and sink to the bottom of the ocean does not come as a surprise to me at all.
Re: (Score:2)
On a related note... (Score:5, Interesting)
It looks like Electrify America's EV charging stations use the AT&T network for payment and account processing. Many of them were reported in "free to use because the network is down" mode today.
Re: (Score:3, Insightful)
It looks like Electrify America's EV charging stations use the AT&T network for payment and account processing. Many of them were reported in "free to use because the network is down" mode today.
I'm going to assume that taxpayers might want to know more about any citizen-funded subsidies that are in place that cover the costs of that, to include built in profits, overhead salaries, and bonuses.
(Name another business that can afford to hang the "everything is free, 'cause outage" sign on the door..)
Re:On a related note... (Score:5, Insightful)
I'm going to assume that taxpayers might want to know more about any citizen-funded subsidies that are in place that cover the costs of that, to include built in profits, overhead salaries, and bonuses.
(Name another business that can afford to hang the "everything is free, 'cause outage" sign on the door..)
Your outrage is misplaced. (Muh TAX DOLLA$!!!!)
It is not uncommon in service industries to write off as "customer goodwill" charges when there is an inability to accept payments. It goes in the books as a marketing expense.
It is done because the reputational harm to the business from failing to provide service outweighs the cost of providing the service for free.
Re: (Score:1)
I'm going to assume that taxpayers might want to know more about any citizen-funded subsidies that are in place that cover the costs of that, to include built in profits, overhead salaries, and bonuses.
(Name another business that can afford to hang the "everything is free, 'cause outage" sign on the door..)
Your outrage is misplaced. (Muh TAX DOLLA$!!!!)
It is not uncommon in service industries to write off as "customer goodwill" charges when there is an inability to accept payments. It goes in the books as a marketing expense.
It is done because the reputational harm to the business from failing to provide service outweighs the cost of providing the service for free.
So, the IRS is happy to accept a business writeoff under the pretense that an EV charging station handing out free power, would seriously need to "market" that, as if every EV in a 50-mile radius somehow did not know within a blink of social media, because wait, did you say free?
Meanwhile, I heard some15-year old kid down the street is getting audited for collecting one too many lunch money cashapp transactions from a suspect domesticated threat commonly tagged as "Mum", which the FBI suspects is a known a
Re: (Score:2)
Re: (Score:2)
No one in the US uses the term mum, unless you're in Texas around October. But that is something completely different.
True enough. It was probably deemed a "racist" term back when they blackballed Aunt Jemima.
Fewer and fewer kids in the US know any term for father. But that's something completely different.
No, its because we're not British. We say mom, and because a mummy is a mummified corpse and not a food.
Re: (Score:2)
Too bad that's only service industries. It's always fun when all the cash registers at the supermarket go down and the lines are backed up for hours.
Re: (Score:1)
Cell jammer for the free charging win?
Re: On a related note... (Score:2)
You're saying all I need to get free charging is a Faraday cage? :-)
Network vs. Network? (Score:3)
Since "networks" are becoming more and more uniform (that is, "networks" for carrying calls, distributing television broadcasts, or browsing the Web are more like the same thing than they were 30 years ago), does AT&T's network outage mean their home Internet, government services and everything else were down, or just their cellular "network"?
Re:Network vs. Network? (Score:4)
Re: (Score:2)
not saying it's aliens, but (Score:5, Funny)
...both start with an "A", just sayin'
This impacted far more people than they claim.... (Score:5, Informative)
I work for a transportation company where we have several thousand drivers carrying smartphones using AT&T in most cases, running a kiosked app we use for scanning packages as they're picked up or dropped off on routes. Our entire system was pretty much non-functional from late last night until sometime this afternoon. And I'd be surprised if ANY of those drivers were signing in to DownDetector and reporting anything.
Looks like coincidence now, but in my city, we lost Spectrum broadband during pretty much this same time-frame. (The rumors I'm hearing though say a truck hit a phone pole and severed some fiber, and people saw many Spectrum trucks in that area all morning/afternoon working on it.)
I also find it suspect that Verizon claims their network was "unaffected" because I've read a number of individual reports of people with Verizon phones having no service or ability to call out this morning. They weren't talking about being unable to complete calls to AT&T customers. Might well be some AT&T hand-off code they updated/changed and broke the whole thing, including competitor's cell towers that started mishandling their own customer's signal/service as a result of it?
Re: (Score:1)
Re: This impacted far more people than they claim. (Score:2)
Maybe 4G vs 5G?
Non-Stop Layoffs at AT&T possible root cause? (Score:5, Interesting)
From what I have heard it has been non-stop layoffs at AT&T since 2023. Comparing 2018 AT&T to 2024 AT&T it is 40% fewer employees.
https://www.lightreading.com/a... [lightreading.com]
You can only cut, outsource, and AI so much until either bad things happen or people get upset and let bad things happen, either intentionally or unintentionally.
Re: (Score:2)
Indeed. Never try to do things cheaper than possible.
At some point cost-cutting becomes very expensive. Competent manager identify that point and then stay at a safe distance. Incompetent ones are excited to save a few more cents and lose millions and sometimes billions in the process. Like some former engineering company that has forgotten how to install bolts, for example.
That said, this is a regulated industry. Seems to me the regulator was asleep. Hopefully they have woken up now and audit and fine the
Headcount decline not primarily from layoffs (Score:4, Informative)
From what I have heard it has been non-stop layoffs at AT&T since 2023. Comparing 2018 AT&T to 2024 AT&T it is 40% fewer employees.
In that time span, AT&T spun off both Time Warner and DirecTV, and sold Crunchyroll and their adtech business. Those actions account for the bulk of the change in headcount.
Re: (Score:2)
2023 ?
Your dates are a bit off I'm afraid.
AT&T has been actively laying off their workforce for far longer than that. January of 2017 was a bloodbath where any management employee not working near a " Collaboration Zone " ( specific sites designated by the company ) were simply laid off with zero regard to the employee's skill set, seniority, knowledge or how good / bad of an employee they may have been. They simply got axed.
Entire teams vanished practically overnight. Teams that were actively mainta
Apple caused this (Score:2)
Just some real life âoeproduct placementâ for its Emergency SOS feature.
Re: (Score:1)
Re: Apple caused this (Score:2)
Quotation marks cannot be gay. They cannot have sex with other quotation marks. They might be idiotic, stupid, or lame. But not gay.
Love The Conspiracy Theories (Score:2)
TikTok was already full of conspiracy theories about this. They all claimed it affected every cell network, which wasn't true. The reports of outages for Verizon and T-Mobile were really from people trying to call AT&T customers and being unable to do so.
Folks were claiming it was a government test of an EMP. Others believing it's the Russians because Biden allowed them. And still others believing it was a test before an invasion.
Re: Love The Conspiracy Theories (Score:1)
Very likely Trump / MAGA self inflicted injury. "Great economy? We're going to trash it! -MAGA crazies"
another testatment (Score:2)
Another testament to the great technical acumen of today's oligarchic companies.
Solar Flares (Score:2)