Fire At AT&T Facility Causes Outage For Over a Million U-Verse Fiber Customers In Texas (wfaa.com) 54
New submitter JustChapman writes: Local Dallas/Fort Worth WFAA is reporting a major outage of AT&T U-Verse fiber internet, due to a lightening strike at a switching facility in Richardson, TX. Apparently the strike took out primary and secondary power systems, setting fire to the building. One commenter states a representative allegedly said that 1.5 million customers are currently without service.
Yup, it's been a stormy afternoon here... (Score:4, Funny)
Thank the maker I don't have AT&T as my ISP, but it remains to be seen if I have internet at home.
However, given AT&T's past transgressions, Somehow it makes sense that the Maker is mad with ATT and struck them with lighting.
Re: (Score:3)
Re: (Score:2)
Well with directv as long as you have power and clear view of the sky you have TV.
Re: (Score:2)
Well with directv as long as you have power and clear view of the sky you have TV.
Yea, but why? Even with a thousand channels, there is never anything good on any more.
Personally, I just have a DVR full of interesting stuff and a drawer of DVD/Blu-ray disks for when the internet happens to be down. Not to mention the book shelves full of good lit books from our "home schooling the kids" phase that just ended. I literally have a life time or two of entertainment...
Re: Yup, it's been a stormy afternoon here... (Score:3)
Youâ(TM)re not paying for redundancy at the POP level. You said yourself it seemed like a great deal, indicating price was one of your major drivers. 99.99% of end users donâ(TM)t want to pay what redundancy costs for the small amount of outages, especially now that everyone has a cell device which can handle 10s of Mb/sec as a âfall backâ(TM). If you really need wired redundancy, have a cable company put in a line, and manage that with a router with prioritization. You donâ(TM)t wa
Re: (Score:3)
Re: Yup, it's been a stormy afternoon here... (Score:3)
I like on DFW, and I've had inverse for a year, and so far this outage has gone 12 hours, or half a day.
Network reliability is measured in "nines", as in "5 nines", which means the network is available 99.999% of the time.
2 nines is 99%, or about 90 hours of downtime per year.
3 nines is 99.9%, or about 9 hours of down time per year.
4 nines is 99.99%, or about 1 hour of downtime per year.
5 nines, the holy grail of availability is about 5 minutes of downtime per year.
Right now, U-Verse is at about "3 nines",
Re: Yup, it's been a stormy afternoon here... (Score:2)
The problem is a large "tech" company thinks it can shove all of their services for a large population into one building.
This is how telcos achieve 'economy of scale', they have done this for about a century - same for municipal water and electricity
Re: (Score:2)
Youâ(TM)re not paying for redundancy at the POP level.
Right but 1.5 million customers affected seems a bit higher up the network than "POP level"
Re: Yup, it's been a stormy afternoon here... (Score:2)
Nonsense.
This isnâ(TM)t a classical data center like google or amazon, this is for local access - by definition it has to be near itâ(TM)s customers/users. I live in the impacted area, and my internet service was restored by midnight, perfectly acceptable for consumer internet service IMHO.
For those running businesses on a consumer ISP like this, this is why you should invest in business class service.
Re: (Score:3)
Sadly I do have U-verse TV and internet. Seemed like a great deal for 1Gb internet service. Service at the house has been down since 10:30 this morning. They won't even attempt to provide an ETA on repairs or service restoration. What kind of tech company doesn't have a redundancy plan? Ohhhh, AT&T isn't a tech company. My bad.
I just heard a local report on this.. Apparently the building burned pretty badly and the Roof collapsed. Unless they have a totally redundant system in some other location, which I find improbable for a host of reasons, it's going to be a LONG time before this gets fixed. IF customer wiring goes though this building (which I find HIGHLLY likely) I'm going to guess it is going to take a lot of time to rewire everything to some new location.
In short.. I'd be asking AT&T to let you out of any contracts ba
Re: (Score:2)
Re: Yup, it's been a stormy afternoon here... (Score:2)
It was resolved (for me, anyway) by midnight Monday - about 15 hours downtime.
Re: Yup, it's been a stormy afternoon here... (Score:2)
The issue was a lightning strike that took out the redundant power feed for the facility when a fire started in the power room.
Sure, they should have had two power rooms, but at some point there is diminishing return on the redundancy.
Re: (Score:3)
Not to worry, with their extensive redundancy in the critical departments, the bills will go out on time and the legal staff is at full power.
Re: (Score:2)
But why did the lightning miss Comcast?
Re: (Score:2)
With comcast there is a few super heads end that if they go down a BIG part of there service will go out.
Re: Yup, it's been a stormy afternoon here... (Score:2)
U-Verse had redundant power, the problem was the place where those two diverse power sources converged is where the fire was. Absent investing in a fully-redundant facility outside the disaster zone of the first facility this type of problem is unavoidable.
This wasnâ(TM)t a classical âdatacenterâ(TM), where itâ(TM)s actual location is irrelevant, this was a facility where multiple access points converged. This facility had connections to community-level distribution (head) points, which
Re: (Score:2)
Re: (Score:2)
Lightening (Score:3)
Not what you think it is: https://www.dictionary.com/bro... [dictionary.com]
Re: (Score:2)
They were struck by someone painting all the grey walls with white paint. It reflected more light, so the building got too cold. This cause the air handlers to freeze up, at which point the equipment all melted down. It makes perfect sense to me.
lightening vs lightning (Score:2, Interesting)
Lightening is where you lighten something up that's dark such as a photo in digital photo editing software.
Lightning is the high-energy burst that almost rivals a nuclear bomb, causes audio compression (i.e. a thing called "thunder"), and can start building fires even if the building is properly grounded.
Hopefully, this has been enlightening for JustChapman. Get a dictionary!
Re: lightening vs lightning (Score:1)
That's odd... (Score:2)
Lightening strikes usually result in liftoff and sometimes an abrupt crash, equally devastating, and fire is a common feature...
Like Hinsdale all over again (Score:2)
This sounds eerily like the 1988 fire at the switching center in Hinsdale, IL [mit.edu]. Hopefully they didn't ignore alarms as happened then.
Primary and Secondary Power Systems together!!? (Score:2)
You really have to hand it to these people. They put the primary and secondary power systems together in the same room.
This is cluelessness on a grand scale.
Re: Primary and Secondary Power Systems together!! (Score:2)
Primary power = Commercial AC
Secondary = Generators
Battery strings and rectifiers in that mix somewhere. Trivia: Many telco systems are DC. Usually -48v
Lightning strike probably destroyed batteries and / or rectifiers meaning even IF the generator came out unscathed, it would not matter.
If it is a building that housed a pass-through fiber node, it will not take long to reroute it.
If it is a fiber drop / handoff point, may take a bit longer but AT&T will roll their mobile crisis units out ( essentially
Re: Primary and Secondary Power Systems together! (Score:2)
there likely was a failure to to ground something, and the lightning came in on whatever that was. Probably a wireless antenna.
Seriously? Why would there be a wireless antenna there?
Re: (Score:2)
Re: Primary and Secondary Power Systems together!! (Score:2)
Right, they should have had completely redundant wiring for the two power sources, not just two power sources for the facility.
At some point your redundant systems have to converge.
If your company has two ISP connections, diversity of service, they likely have both routers in the same rack in the same closet.
But no, you're right, they need to put two power rooms on alternate sides of the building, with both running to a third room, where there is a cutover in case one fails... oh wait, what if that room cat
Just Dallas/Fort Worth area, not all of Texas (Score:2)
Re: Redundancy dammit. (Score:2)
I'm in DFW, my U-Verse Internet is down, yet my AT&T cellphone works fine - it acts as a backup WiFi via hotspot service.
Impossible. (Score:2)