Forgot your password?
typodupeerror
Wireless Networking Cellphones Communications Networking

A Possible Cause of AT&T's Wireless Clog — Configuration Errors 217

Posted by timothy
from the three-card-monty-design dept.
AT&T customers (iPhone users notably among them) have seen some wireless congestion in recent months; Brough Turner thinks the trouble might be self-inflicted. According to Turner, the poor throughput and connection errors can be chalked up to "configuration errors specifically, congestion collapse induced by misconfigured buffers in their mobile core network." His explanation makes an interesting read.
This discussion has been archived. No new comments can be posted.

A Possible Cause of AT&T's Wireless Clog — Configuration Errors

Comments Filter:
  • Wouldn't be the first time, except maybe for AT&T.
    • Re:First Time (Score:5, Insightful)

      by Fluffeh (1273756) on Sunday October 25, 2009 @09:33PM (#29868643)

      Wouldn't be the first time, except maybe for AT&T.

      I don't think that it's limited to just AT&T - I am in Australia, so have never even had to deal with them, but I am finding that in the vast majority of Australian companies as well, simple back to basics work quality is plummeting. Everything seems to be about making everything as cheap as possible - whether or not it even functions the way it is supposed to. That also goes for the majority of customer service dealings as well.

      It seems that the "Do it once but do it properly" mentality is limited to very few people and businesses. I work as a business analyst and the amount of arguing I have to do with each project to get extra money spent to do things properly (the majority of the time it saves money in the long run anyhow for other projects - I am not even taking into account the maintenance and support savings into that equation) yet I seem to always have to fight the same battles over and over.

      • Re:First Time (Score:5, Interesting)

        by MichaelSmith (789609) on Sunday October 25, 2009 @10:25PM (#29868847) Homepage Journal

        I know somebody who works on network infrastructure for Telstra. I suggested to him that a lot of traffic which currently goes through wireless and wired LANs will soon run through the cellular networks. He was horrified at the idea. Apparently TCP/IP traffic from 3G cells has to go all the way back to the internet backbone, so anything resembling P2P still saturates the links between the base stations and the back end. Thats a minor issue just now but in addition the links to the 3G cells are only just keeping up with demand right now.

        I pointed to the European environment where 3G data is much cheaper and more bandwidth is available. He says that we don't do that kind of investment here. So at the end of the day its a money problem. Lots of profit being taken while they can get away with it.

        • Re:First Time (Score:4, Interesting)

          by bertok (226922) on Sunday October 25, 2009 @11:40PM (#29869091)

          I know somebody who works on network infrastructure for Telstra. I suggested to him that a lot of traffic which currently goes through wireless and wired LANs will soon run through the cellular networks. He was horrified at the idea. Apparently TCP/IP traffic from 3G cells has to go all the way back to the internet backbone, so anything resembling P2P still saturates the links between the base stations and the back end. Thats a minor issue just now but in addition the links to the 3G cells are only just keeping up with demand right now.

          I pointed to the European environment where 3G data is much cheaper and more bandwidth is available. He says that we don't do that kind of investment here. So at the end of the day its a money problem. Lots of profit being taken while they can get away with it.

          Yeah, I love the lack of forward planning by Telcos in Australia.

          Some years ago, there was talk of building some huge fiber-optic ring around the Pacific, connecting a bunch of countries. The only telco in Australia at the time that could afford to buy into the project was Telstra. One of the VPs of Telstra was quoted as saying "we have sufficient bandwidth right now". Think about it: the VP of a telco couldn't quite understand the need to maintain exponential growth in bandwidth right when broadband was taking off. Thanks to morons like that overpaid suit, Australia has been bandwidth-starved for a decade, which is why you don't see that many truly "unlimited" plans or free WiFi access points like in other countries.

      • Re: (Score:3, Insightful)

        by rubi (910818)

        I think that what you are seeing is just the result of how business is conducted these days and how the money is allocated. currently I have the same perception about things where I work and I believe the change came when the company hired several "genius" executives that had degrees in finance, administration and such from reputable universities.

        They came with the current trends in economic analisys "pre-programmed" (to be truthful, that is what is being teched at most universities now) and this type of m

        • Re:First Time (Score:4, Insightful)

          by Fluffeh (1273756) on Sunday October 25, 2009 @10:47PM (#29868909)

          any money spent "doing things properly" is money spent now, not in six months or two years or longer, so the calculate that a project finished early that just needs "tweaking" in a future date is better and cheaper than a project done the way it is supposed to be done (but taking longer)

          Yup, that's exactly what I am talking about, and I find it very frustrating. The time between project end and the final "tweaking" implementation where the project deliverable finally works as it is supposed to is both frustrating for the users, has a high support cost from a systems point and the "tweaks" normally end up adding much more to the cost itself than just doing it properly the first time.

          I am reasonably lucky that these days I am involved in the early stages of some of the projects that I work on, and I start on the offensive for the most part, and ask for detailed analysis from project managers that I work with on the cost of the "cheap" and "proper" solutions over the space of a year or two if the project looks like it is trying to cut too many corners - and take that analysis to the program office - it's coming out of their pockets after all or on occasion directly to the business that is footing the bill for the project. While it works for the majority of the time, it's still amazingly frustrating to have to fight the same damned fight each time so that things are done properly. In my eye's it's up to the project managers to be ensuring that their projects are done properly and not end up as massive drains on support/systems.

          Sadly, few of them see it that way. It's all about being cheap and cost cutting and meeting budget KPI's rather than arguing that the budgets are set too low.

          • Re: (Score:3, Insightful)

            by cgenman (325138)

            I believe we need to change the "They don't make em' like they used to" mentality.

            They cut corners and cheapened out on stuff in the past too. It's just, none of those survived. So sure we can disassemble radio transmitters made in the 40's, see the craftsmanship that went into each one, and sigh that our equipment isn't made nearly as well. But there were a heck of a lot of transmitters and things made in the 40's that simply didn't survive because they were cheap junk.

            It's not helpful to go to the uppe

        • Re: (Score:2, Insightful)

          by Anonymous Coward
          Posting Anonymously because I don't feel like being associated with an unpopular viewpoint:

          Sometimes 'good enough' really is the best option for the business as a whole. Techies and engineers often have a hard time accepting this until they've actually run a department, but it's true.

          If we have $10m today, we might benefit more by doing 10 $1m projects 'pretty well' than by doing 7 $1.4m projects 'perfectly', for a number of reasons.

          It could get something to market faster... it could be that the
          • Re:First Time (Score:5, Insightful)

            by jyx (454866) on Monday October 26, 2009 @01:50AM (#29869649)

            Oh noes, I'm feeding the trolls again.

            But whatever... it doesn't matter. Because at the end of the day, the techie nerds will continue to have no respect for management... and then they'll wonder why they're treated with no respect in return.

            So you think the techies that have taken the time to explain all the reasons *why* something needs to be done are stupid.

            But you can sit back and say 'loose 8% from your budget - go do it'. No reasons, no explanations just a demand. (Brillant!).

            I'm guessing your also the same arsehole that screams at the 'stupid' techies for not being able to restore that sales contract from two months ago that you accidentally deleted - Forgetting about that replacement broken tape drive you refused to pay for last quarter.

            As a manager you have got to be the conduit between the workers and the directors. Here's a tip, how about try talking to your techies. No seriously, talk to them. Show them your budget, show them your overheads. Ask them to provide assistance in setting the priorities instead of telling them to get stuffed.

            You may end up *earning* some respect from the people who are actually keeping your company running and who don't play musical employers when things start getting to hard.

            • Re: (Score:3, Insightful)

              by MrNemesis (587188)

              Best thing our new management has done was to open the budgets. We all got an email telling us the expenditure for the previous year and expected expenditure for this year - the bosses were going to ask the managers to get the techie types, like myself, to go over each item and see if they were neccesary... but before they could organise a meeting they were already inundated with "this contract expired two years ago!" and "we can totally negotiate a better price on this!" messages from the front lines.

              A man

              • by umghhh (965931)
                I may have reservations about the ideas like scrum or generally agile methods but if done well they apparently involve technical people in decision and budget work as a matter of principle which is good as it enriches the perception engineers have thus enabling better understanding of constrains that one has to work with.
          • Re: (Score:3, Informative)

            by umghhh (965931)
            I was on the other side of the divide than you number of times and I must say I can understand the trade-offs well enough for an idiot techie. I have great doubts that this is the same on your time of the divide though.

            I make it simple for you as you are obviously on the other side. There are basically 3 basic ways to save money while doing a project and consequently 4 different decisions you can make:

            • not doing a project at all. In absolute terms it is a bummer - no costs! It has a disadvantage of not hav
      • Re: (Score:3, Insightful)

        by zoloto (586738)
        You're one of the few and I'm glad there are more of us around. It brings a little sense of peace to my world.

        I've worked in the aerospace engineering and IT industries (both non-military/military companies) and it's like pulling teeth from a hippo to make sure some things are done properly. Only _one_ engineer understood this "do it once or don't do it at all" (verbatim, his words) philosophy. it was a quality you could see in his 45 years as a professional.

        "They don't build them like they used to" was an
      • Re:First Time (Score:5, Insightful)

        by bertok (226922) on Sunday October 25, 2009 @11:37PM (#29869081)

        Wouldn't be the first time, except maybe for AT&T.

        I don't think that it's limited to just AT&T - I am in Australia, so have never even had to deal with them, but I am finding that in the vast majority of Australian companies as well, simple back to basics work quality is plummeting. Everything seems to be about making everything as cheap as possible - whether or not it even functions the way it is supposed to. That also goes for the majority of customer service dealings as well.

        It seems that the "Do it once but do it properly" mentality is limited to very few people and businesses. I work as a business analyst and the amount of arguing I have to do with each project to get extra money spent to do things properly (the majority of the time it saves money in the long run anyhow for other projects - I am not even taking into account the maintenance and support savings into that equation) yet I seem to always have to fight the same battles over and over.

        There's a simple reason for that: money is trivial to measure. Quality is much harder to measure. For example, failure rates like MTBFs often don't directly correlate into straight dollars and cents, but a small percentage chance that it might cost a large but unknown amount at some point in the future. This kind of thing confuses people, so they stick to the simple stuff. In an Excel spreadsheet, the solution that costs fewer up-front dollars is just "better" in the world view of most people.

        I've had a conversation recently with the CIO of a major business who didn't quite understand why backups were worthwhile. He said something along the lines of "how does this help the business sell more widgets?".

        I see the same thing, but often much worse, in big government or big bureaucracies. Project management is complex, so to simplify things, they just ignore the rest of the business or potential future requirements like they don't even exist. In the past, I've tried to point out that, say, with an additional 10% spend on one project they could halve the cost of a dozen future projects, but that's basically crazy talk to a project manager that has to minimize the cost of this project, right now. I've given up trying, and I bet a lot of other people have too.

        • Re: (Score:3, Insightful)

          by Angostura (703910)

          I'm not so sure that attempting to measure quality is necessarily the way forward when having the discussion with that CIO. Perhaps an alternative approach is to talk more about the value of the data which is being backed up. It can be quite illustrative to go through the data that is being backed up and work out the cost that went into generating it: "That took those staff 3 person-weeks to generate, their average salary is X" etc. It's fairly easy to get a notional cost of most data. Then you can look at

  • by bertoelcon (1557907) <berto.el.con@noSpAm.gmail.com> on Sunday October 25, 2009 @09:03PM (#29868467)
    This is not really news at all. They spend little to nothing to keep their network up to the devices they have on it. This misconfiguration of buffers (if that is really a cause at all) is probably because they might not hire people with any knowledge of what they are doing to keep costs low.
    • They keep cost and quality low because that is what their customers actually want, or at least, that is what they are willing to pay for.

      Let's face it. The western consumer values one thing above all else; price. The cheaper the better. The public has shown repeatedly that it will value cost above quality. AT&T's customers are still with it after all. Why should AT&T attempt to improve the quality of its network if people are a) willing to pay for what they currently have, and b) won't pay for any attempts AT&T will make to improve quality. In the telecoms business, ordinary people can and will jump on the cheapest package available.

      It's been a race to the bottom in more industries than this one. So we really can't complain when such shocking lapses in quality occur even in the largest companies.

      • by SuperKendall (25149) on Sunday October 25, 2009 @09:24PM (#29868581)

        The public has shown repeatedly that it will value cost above quality

        Right, that is why Apple laptop sales have tanked in the downturn. Oh, wait.

        Why should AT&T attempt to improve the quality of its network if people are a) willing to pay for what they currently have, and b) won't pay for any attempts AT&T will make to improve quality.

        Because those people if they dislike the network enough, will leave eventually. That is the motivation to improve on what they have now, never mind they want to stop the customers bitching who are losing them new customers right now. They have plenty of reasons, they even have plenty of money from the influx of iPhone people. There's more than enough motivation, it's more a question of execution now.

        People will pay for quality. For some the cost is financial. For others, the "cost" is that they will not buy an iPhone while the AT&T network has issues.

        • Re: (Score:3, Informative)

          by hitmark (640295)

          quality, and fashion, one not having to be the other...

          • by SuperKendall (25149) on Sunday October 25, 2009 @09:48PM (#29868695)

            quality, and fashion, one not having to be the other...

            Indeed, and that is why many companies built atop the foundations of showy fashion are gone now. Fashion is transient and fickle. Apple however delivers a quality product that delivers new customers through loyalty and word of mouth. If this were not so Apple would not be a tenth of what it is now.

            It doesn't hurt that it is fashionable, too. But that is not why I and so many other people buy Apple products.

        • by MBCook (132727) <foobarsoft@foobarsoft.com> on Sunday October 25, 2009 @09:29PM (#29868617) Homepage

          Because those people if they dislike the network enough, will leave eventually.

          This is the problem. Thanks to the competitive barriers (such as the inability to move phones between all but two of the top four networks, and none of the top 3) moving can take a long time (2 year contract must expire) before someone can move networks unless they want to pay a large fee.

          And then, you probably lose your phone. So even if you like it, you have to buyer either a different phone from the new provider, or the same one in their version. Both will cost you even more money, unless you're willing to be stuck on another 2 year contract.

          The US system is very well setup, as far as carrier lock in goes.

          It's rather amazing how many people go to AT&T for the iPhone. I think they said about 1/3 of their iPhone customers are coming from other networks. I wonder how many more people would get iPhones if it wasn't for their current contract? That's a big reason for many people I've talked to. The rest who want an iPhone are in the "I'd love it but I'm not touching AT&T again" camp.

          • This is the problem. Thanks to the competitive barriers (such as the inability to move phones between all but two of the top four networks, and none of the top 3) moving can take a long time (2 year contract must expire) before someone can move networks unless they want to pay a large fee.

            You say that like it does not matter because the period of switchover is two years. But it matters a great deal still, because many people will still leave then (or if they are mad enough pay the fee). A company like AT

        • by Darkness404 (1287218) on Sunday October 25, 2009 @09:54PM (#29868719)

          Right, that is why Apple laptop sales have tanked in the downturn. Oh, wait.

          I think that will change once Windows 7 is mainstream. Everyone hated Vista. Now it seems like everyone loves 7 and Snow Leopard only got a "meh" response from reviewers (not because Snow Leopard is bad it just doesn't have anything revolutionary, the fact that 7 runs at a decent speed is considered to be "revolutionary" in the PC world). There are two people who use Macs, people who have grown up using Macs and people who prefer Macs. When faced with Vista, a lot of people started to realize they prefer Macs.

          Because those people if they dislike the network enough, will leave eventually

          "Eventually" isn't very soon when you have a 2 year contract with early termination fees that are through the roof.

          That is the motivation to improve on what they have now, never mind they want to stop the customers bitching who are losing them new customers right now

          All 4 major carriers suck though. Lets see here, AT&T has network issues and isn't cheap, T-Mobile might have great customer service, good phones but it has a pathetic amount of 3G coverage compared to the others. Verizon might have a great network, but it isn't exactly cheap and a lot of their phones (at least used to) suck terribly with many features being stripped out of them. Sprint might be cheap but their coverage isn't great.

          And none of them have a phone with as many apps as the iPhone, yes, Android and WebOS are great, but they still don't have the amount of apps as the iPhone nor as much support from companies such as game developers and the like. And don't get me started on Windows mobile.....

          For others, the "cost" is that they will not buy an iPhone while the AT&T network has issues.

          ...And who is going to look at a few "geek" articles about the iPhone and decide not to get it? Yeah, sure, we all know about how AT&T's network is crap, but people see the iPhone and want that. They only see the network once AT&T has them hooked on a few years agreement.

          • by SuperKendall (25149) on Sunday October 25, 2009 @10:05PM (#29868763)

            I think that will change once Windows 7 is mainstream.

            I don't think people care that much one way or the other. In fact studies have shown previous Windows releases increased Mac sales, and this will too - if you have to refresh a whole system, if you have to learn a new UI - why not a Mac?

            I just can't see how Windows7 will have any impact at all in slowing down the Mac train.

            All 4 major carriers suck though.

            From experience with them all I totally agree, which is why I am not as much bothered by some people with the iPhone being AT&T only.

            Verizon might actually improve if they don't tamper with the Droid much. ...And who is going to look at a few "geek" articles about the iPhone and decide not to get it?

            It's not that at all. It's having a friend who complains about dropped calls all the time, or if he tries to the use the network and it's failing a lot. This kind of damage is all done at the word of mouth level.

        • Re: (Score:3, Insightful)

          by beej (82035)

          The public has shown repeatedly that it will value cost above quality

          Right, that is why Apple laptop sales have tanked in the downturn. Oh, wait.

          Macs have a 10% market share. I'm not sure that really supports the suggestion that people value quality over cost, with 9/10 people voting against "quality".

          Either that, or people don't think Macs are quality.

      • by MojoRilla (591502) on Sunday October 25, 2009 @09:33PM (#29868645)

        The public has shown repeatedly that it will value cost above quality.

        Then why are people flocking to AT&T for the IPhone? It certainly isn't the least expensive smartphone out there. Perhaps it is because it is the best smartphone out there, and people are willing to put up with a crappy provider to get the device. Perhaps quality does sell, at least for devices.

        • Re: (Score:2, Insightful)

          by obarthelemy (160321)

          Until I see reliability data that proves otherwise, I don't think Apple is about quality/reliability. At least, not in my experience.

          Aplle is about design, ease of use, and trendiness. My in-laws iPhone broke after 2 months. My brother's iMac, after 2 years, several thing in a row (CPU fan, HD). And let's not get even close to the Mighty Mouse I got suckered into buying in for a bday.

          • Re: (Score:3, Interesting)

            by RudeIota (1131331)
            To be fair, MojoRilla's argument was it's one of the "best smartphones out there", not the highest quality and certainly not the most reliable.

            The iPhone has managed to put itself in the hands of many people who've never had a very nice phone, so I think the iPhone is far better quality than a large portion of its user base is used to and comparable to other phones in its class.

            For what it is worth, I believe Apple's selling points are in this order: Features, quality, price (the last two are very close,
        • Re: (Score:3, Insightful)

          The public has shown repeatedly that it will value cost above quality.

          Then why are people flocking to AT&T for the IPhone? It certainly isn't the least expensive smartphone out there. Perhaps it is because it is the best smartphone out there, and people are willing to put up with a crappy provider to get the device. Perhaps quality does sell, at least for devices.

          I think a fundamental problem with discussing this here is that there are (at least) two different perspectives at work. There's the Slashdotter who has read "AT&T network problem" time and time again over the last few months and then there's the iPhone customer who, rather than basing his decision solely on these headlines, knows several people with an iPhone who have never complained about the network. Somebody who has made up their mind that they don't want to touch it at all cannot understand the

          • My own personal anecdote: AT&T is HORRIBLE. I switched to AT&T for the iPhone and am getting the hell away from it as soon as my 2 year contract expires. I will switch to (in my opinion) an inferior phone (the Droid/Sholes/whatever they are calling it now) looks promising, to get away from AT&T. I absolutely love my iPhone, but it does me no good when I can't access the internet or make/receive calls when I want to.

            The difference from Verizon is astounding. I can count on two hands the number

            • by dgatwood (11270)

              It depends on where you are. My CDMA phone on Verizon was dropping three and four calls per night. After switching to AT&T, I drop calls only in a few dead spots.

        • Re: (Score:3, Insightful)

          by SheeEttin (899897)

          Then why are people flocking to AT&T for the IPhone?

          Uh... They aren't. They're flocking to the iPhone. AT&T is incidental.
          If it had been Sprint, T-Mobile, or some other provider, sales would be nearly identical. (Actually, given the amount of bitching about AT&T, sales would probably have been slightly higher.)

      • by forand (530402) on Sunday October 25, 2009 @09:43PM (#29868677) Homepage
        Europe has a vastly superior cell network than the US and on average consumers do not pay significantly more than US consumers. Furthermore US consumers are locked into a contract which ensures a steady income for the service providers. The major difference being populations density but this should not be an issue since the cell providers are given a subsidy to build out into rural areas. I think that the real issue is not the Western consumer but the US corporation which extracts every last cent of value from the current consumer to give it to investors and or executives with little to no thought towards how the company will make money in the future. So while the US consumer may strive to keep prices low (as they should in a free market) the US corporations are taking the profit they have and investing nothing for the future.
        • Re: (Score:2, Interesting)

          by dUN82 (1657647)
          Totally agree ! Have being lived in 3 countries for some considerable time and as a mobile phone user in China (9 yrs), UK(5yrs), and the US(3yrs), I have to say: US has the worst cellphones selection, worst cellphone call quality + coverage, highest tariffs, lengthiest contract, and most unfair contract. I have to settle with AT&T when I arrived in the US since I have my own GSM phone which keeps out of Sprint/Verizon, and honestly speaking, I am glad I brought my own phone, because there really isn't
        • Re: (Score:3, Interesting)

          by Alpha830RulZ (939527)

          Europe has many more customers in a much smaller geographic area. I wonder if it isn't a lot cheaper to service them.

        • Re: (Score:3, Interesting)

          by darkpixel2k (623900)

          Furthermore US consumers are locked into a contract which ensures a steady income for the service providers.

          There's a challenge for you. Go and buy an unlocked cell phone. Then go to any of the major carriers and try to signup for service without a contract.

          I tried this a few years ago and not a single carrier would sign me up without a two year contract. (What's the point of buying an unlocked phone if you can't take it from network to network without locking in to a contract. I might as well get the damn subsidized phone.)

        • by beej (82035)

          My German friend gets some-number-of-megabit tethering on his phone for something around $40/month. It's positively criminal.

        • by mveloso (325617) on Monday October 26, 2009 @12:12AM (#29869181)

          1. the US is much bigger than Europe, with multiple overlapping jurisdictions. It's easy to cover any of the European countries, because they're small and there wasn't a technology transition.

          2. there isn't as much rural subsidy for cellphones. Universal service was for landlines, mainly.

          3. the problem is cost vs coverage. You can build out rural areas, but you make less money because there are less people. For urban areas, you start running into interference problems. Plus, you have to constantly build out your infrastructure (see AT&T's infrastructure problems).

          Europe has it easy. It's not evil corporations (which, to be frank, is a retarded and simplistic view of how things work) - it's pure cost/benefit.

          Example:

          France: 211,207 square miles
          Texas: 268,601 square miles
          US: 3,537,441 square miles

          If AT&T only had to operate in Texas, it would be able to do pretty well. AT&T's footprint is national, however. Do you develop Texas completely, or do you cover Michigan and Texas? How about extending to Missouri? etc etc.

          Then that's current coverage; what about LTE? How about maintaining that infrastructure?

          • by bruce_the_loon (856617) on Monday October 26, 2009 @01:01AM (#29869419) Homepage

            Be careful of equating land area coverage with the size of the networks. There's nearly twice as many people in the EU than the US, and even ignoring the possibility that a greater percentage have cell phones, (Finland had 100% penetration) that's almost 100% more customers on the network.

            The European providers have to manage greater customer density than the US ones do. It's harder to do, making sure frequencies overlap properly, reducing interference etc.

          • Re: (Score:3, Insightful)

            by dkf (304284)

            1. the US is much bigger than Europe, with multiple overlapping jurisdictions. It's easy to cover any of the European countries, because they're small and there wasn't a technology transition.

            There was a technology transition, and there most certainly are overlapping jurisdictions. Just because you don't see them from your perspective doesn't mean that they aren't there. What there is though is more of a willingness to do something about it.

            2. there isn't as much rural subsidy for cellphones. Universal service was for landlines, mainly.

            Who cares about the boonies? Why is there such bad service in US urban areas? You'd think that there'd be plenty of people there to pay...

            3. the problem is cost vs coverage. You can build out rural areas, but you make less money because there are less people. For urban areas, you start running into interference problems. Plus, you have to constantly build out your infrastructure (see AT&T's infrastructure problems).

            Yet Europe is more urbanized and yet manages to solve it with more companies in the market. It can't be an insurmountable

      • by Kenja (541830)
        Which totally ignores that elsewhere you get higher quality for lower cost...
      • by sjames (1099)

        Just how expensive is it to trim the buffers? Hiring one routing genus could solve all their problems.

      • by sjames (1099)

        Let's face it. The western consumer values one thing above all else; price. The cheaper the better.

        That's a natural reaction to the nearly complete lack of truth in advertising enforcement and corporate cost (and quality) cutting. When every purchase is a crapshoot for quality no matter how reputable or unknown the brand, consumers will naturally buy the cheapest every time on the principle that if they're going to get screwed, they might as well get screwed for the least amount possible.

        Unfortunately, unless or until strong consumer laws start nailing the liars to the wall, a company wanting to compete

      • by cgenman (325138)

        Let's face it. The western consumer values one thing above all else; price. The cheaper the better.

        DSL was cheaper than cable, but it became marginalized because the network was unreliable and slow.
        Dial-up is now a fifth of the cost of most high-speed internet connections in the US. But it's not even considered an option now for most people.

        I agree with the sentiment, but don't forget that the western consumer is also lazy and impatient. Combined with cheap, these three things form the legs of the cripple

  • Hm (Score:5, Insightful)

    by Anonymous Coward on Sunday October 25, 2009 @09:07PM (#29868489)

    His explanation makes an interesting read.

    I'd like to think that's a given, considering it's a news story. At any rate, from TFA:

    The bottleneck link is the over-the-air link, i.e. the connection from radio access network or UTRAN to the Mobile Statation (MS) in the above diagram, therefore the critical buffers are those at the UTRAN. In practice the UTRAN includes both the basestations (called Node-Bs) and the Radio Network Controllers (RNCs) which coordinate handovers between basestations (among other things). Because of hand-overs, the amount of data buffered at the Node-B is relatively small. It's the buffers at the RNC that must be large enough to deal with the delay variations in the radio network and yet small enough to induce packet loss when the network gets congested.

    I am not a network engineer, but how exactly could 8 second ping time be not noticed by the AT&T engineers who set up, configured, and monitored their OTA link? I would think that we're not talking about some dude's set of bridged dd-wrt linksys routers, but some serious heavy-duty RF equipment. I'm thinking on the order of several zeros...

    • Re: (Score:3, Insightful)

      by zoloto (586738)
      incompetence
    • Re: (Score:3, Informative)

      by Anonymous Coward

      Because they don't monitor it. They don't even send a guy to check on it unless several people on the same cell log a complaint, something which may take several calls to customer service.

    • Re: (Score:2, Interesting)

      by Anonymous Coward

      I think the key word you assumed was being followed is "monitored". They setup the network and walk away and never monitor. I just upgraded my ATT DSL service but tested the speed before I submitted the change request. After the request was fulfilled, the speed remained the same. I called and 20 minutes later they had "corrected" the problem. Set it and forget it is the problem.

    • Re: (Score:2, Funny)

      by mikep554 (787194)

      Maybe some PHB equates packet loss with dropped calls, and told the engineers that packet loss would also equate to job loss. Not the first time a person in authority forces a bad configuration choice based on a complete misconception of a situation.

    • Re:Hm (Score:4, Interesting)

      by phantomfive (622387) on Monday October 26, 2009 @01:14AM (#29869497) Journal
      I don't have an iPhone and don't have experience with this particular problem, but in general there aren't automatic monitoring devices for mobile networks out in the field, so if AT&T wants to know what is happening on the devices, they have to send a team out with tools and monitoring devices to check. If this is a problem that only happens when several iphone users get together in an area at the same time, then the problem may have gone away by the time a team comes out to check (if they come at all).
  • by Anonymous Coward on Sunday October 25, 2009 @09:10PM (#29868503)

    You see, most blokes, you know, will be buffering at ten. You're on ten here, all the way up, all the way up, all the way up, you're on ten on your buffer. Where can you go from there? Where?

    I don't know.

    Nowhere. Exactly. What we do is, if we need that extra push over the cliff, you know what we do?

    Put it up to eleven.

    Eleven. Exactly. One more buffered.

  • Software Robustness (Score:3, Interesting)

    by dziman (415307) on Sunday October 25, 2009 @09:13PM (#29868513)

    I find it just as problematic that applications software on Windows Mobile and other similar mobile OSes do not handle large network delays gracefully.

    There is often very little feedback to the user of the software that actual progress is being made in attempt to communicate over the network. Sure, we can use the fuzzy "bars" indicator on the device to help diagnose what may be the cause of our trouble, but that doesn't indicate actual network conditions due to capacity. We also have animated indicators that web browsers and other applications use, but these still don't indicate any kind of actual success to communicate. In web browsers we get text alluding the DNS lookup, and connection attempt, but when you combine 'Connecting to...' with a simple spinning indicator or progress bar, that often doesn't convey that the message reached any destination or how long until you can expect any response from your local network based on its operating conditions.

    The writers of the software may not fully understand the implications of being on a network with high packet loss or long round trip times. So they timeout or have errors that could be resolved by more delay or retry. In a mobile OS we should probably take this into account at the OS level, and opt out of this behavior only when the programmer or user specifies (if that's exposed).

    • by Techman83 (949264)
      Some advanced info would be handy in apps, but the people that generally benefit from that information can find it out in other ways or already understand the nature of the Internet. Most people would either glaze over or swear at their machine regardless of the information presented.
    • by kaiser423 (828989) on Sunday October 25, 2009 @09:49PM (#29868699)
      Blackberries are awesome about this with the bi-directional communication arrows. When I'm with friends in an area of low reception, they're all walking around randomly trying to call every two yards, and waiting 15 seconds before determining that its not going to work. I walk around until I see an incoming arrow. I freeze and then make a call. Works wonderously.
      • I walk around until I see an incoming arrow. I freeze and then make a call. Works wonderously.

        And what does the outgoing arrow mean? A call is a two way conversation so I don't see what point you are trying to make.

        • by nick0909 (721613) on Sunday October 25, 2009 @11:21PM (#29869027)
          The arrows show data traffic as well as voice traffic. It is very nice to see a whole lot of up, down, or both arrows flashing when an app is sitting "unresponsive." You know data is flying so nothing is wrong, just wait and the app will respond when it has the data it needs. The arrows (at least on my 8330) are large for the faster network, and thin for the slow network so I even know when it will take longer because of poor network coverage. I used a Windows Mobile phone for a week and it drove me mad not knowing what was going on with the network data.
      • by Anonymous Coward on Sunday October 25, 2009 @10:50PM (#29868931)

        I walk around until I see an incoming arrow. I freeze and

        people and cars crash into me

    • by ratboy666 (104074) <fred_weigel@NoSPaM.hotmail.com> on Sunday October 25, 2009 @11:37PM (#29869079) Homepage Journal

      TCP/IP completely shields the application from the underlying transport.

      A call is made to resolve a name (dns), a connection is opened, and... data flows. Or not.

      So, your speculated "connectivity feedback" information has to come from a lower level than the application. It has to be in the stack.

      Some platforms do incorporate this feedback from the stack. It just isn't an application responsibility. Even on platforms with good feedback (Blackberry), the applications are not aware.

      And the application layer programmer should DEFINITELY not be making these decisions. If the application wants something other than TCP, the developer does have the option of using UDP.

  • safari sux (Score:3, Interesting)

    by peterflat (1326469) on Sunday October 25, 2009 @09:17PM (#29868547)
    it doesn't help that the safari client that the iphone uses will double load a page. Even if the user closes safari for a couple minutes, when reopening the browser the current page will reload. lose lose for everyone.
  • by Anonymous Coward
    How can you expect to have zero packet loss on a wireless network? That's just stupid.

    Please fix this ATT!

    (I'm not holding my breath... )
    • by MBCook (132727) <foobarsoft@foobarsoft.com> on Sunday October 25, 2009 @09:34PM (#29868649) Homepage

      Wow. This is kind of amazing.

      Nothing on this page (as I type) talks about zero packet loss, except you. That means you read the article.

      Of course, the article says that AT&T has set their buffers large enough to prevent packet loss due to congestion in transit, not that they expect no radio packet loss. The problem is that TCP/IP needs packet loss to tell it when it's going too fast and AT&T's decision causes this to fail spectacularly at times.

      The trolls read the articles. Weird.

  • by Constantin (765902) on Sunday October 25, 2009 @09:22PM (#29868577)

    Every time I deal with AT&T I am amazed that anything works at all over there. My phone almost always shows five bars at home, yet frequently calls don't cause the phone to ring - they go to voicemail after pretending to ring. The jaded amongst us could suspect a deliberate misconfiguration of phones and signal strength monitoring. Similarly, it would not surprise me if AT&T data networks weren't about as reliable as the signal strength indicator on my phone. The recent alleged blurb from an Apple "genius" in NYC that 1/3 of all iPhone calls get dropped seems to point in that direction.

    That a cell-phone won't work everywhere and perfectly every time is a given. However, wouldn't it be nice if the companies that stood behind these networks would actually be held accountable for some of the advertising statements they make? What it comes down to is that we're dealing with an oligopolistic market, where only a few carriers can achieve the scale and the coverage to satisfy most mobile customers most of the time. On the flipside, that also means that said carriers can be truly dismal when it comes to customer service, back-end efficiency, etc. since consumers don't have many choices. Considering the ongoing consolidation in the industry, the only way out seems to be a trust-busting activity on the part of the DoJ to regulate the industry.

    Not sure that is the better alternative... nor what the best structure for a regulated market would be.

    • by socsoc (1116769)

      Not sure that is the better alternative... nor what the best structure for a regulated market would be.

      There's a map for that.

    • Re: (Score:3, Informative)

      by Not_Wiggins (686627)
      I presume you have an iPhone. A friend of mine that has jailbroken his phone pointed out to me that the bars represent the 3G signal strength, and not necessarily the regular network strength. I was considering AT&T because the T-Mobile strength in my house is terrible. He had 3-4 bars on his IPhone, but when he turned off 3G and went to EDGE only, it averaged 1-2 bars. Point is, I don't think the signal strength always means what we think it means. 8/
      • Re: (Score:3, Informative)

        by socsoc (1116769)
        Cause they aren't necessarily the same tower? Your scenario makes perfect sense. Except that jailbroken has nothing to do with it and if you are getting 3G signal, that is your "regular network strength." It only degrades to EDGE if you can't get a 3G signal (which for my major metropolitan area, is nearly any building you walk into).
    • by Savage650 (654684) on Monday October 26, 2009 @03:02AM (#29869949)

      My phone almost always shows five bars at home, yet frequently calls don't cause the phone to ring - they go to voicemail after pretending to ring. The jaded amongst us could suspect a deliberate misconfiguration of phones and signal strength monitoring.

      Signal strength alone does not guarantee the ability to make/receive calls. Even if your mobile is registered in the network, making and receiving calls depends on the availability of various scarce resources, namely:

      1. a "slot" on the over-the-air network (# of active connections per cell is limited)
      2. a "switching path" inside your operators network
      3. a "switching path" inside one or more transfer networks (owned by someone else)
      4. a "switching path" inside the network the caller/callee is connected to
      5. a "slot" on the over-the-air network on the others side (if the caller/callee is mobile)

      In case of "lots of missed calls" in a particular area (your home) one could assume that

      • your home cell is overcrowded (all slots in use) or
      • there is a bottleneck in the upstream network

      Note: outgoing calls should have the same problems; if they "fail less" it could be because your operator has chosen to reserve a (possibly large?) percentage of the slots/lines for outgoing calls. (Which obviously reduces the chances of incoming calls even more)

    • Re: (Score:3, Funny)

      by Mr_Silver (213637)

      The jaded amongst us could suspect a deliberate misconfiguration of phones and signal strength monitoring.

      There is actually no standard scale for signal bars on a mobile phone. As such, the mobile phone manufacturer implements a scale pretty much however they want. The upshot of this is that when the signal strength is the same, one model might show 4 bars but another only 2.

      Years ago Nokia was notorious for showing a stronger signal than other phones did - meaning that when people sat around and compared t

  • by NynexNinja (379583) on Sunday October 25, 2009 @09:24PM (#29868585)

    I worked for AT&T in several parts of the country on their core networks, and in the early 2000's they had misconfigured all of their Solaris boxes and I worked with the infrastructure group to implement a startup script on Solaris to tune all the ndd settings for performance. The problem with Solaris is that by default all the TCP, UDP, Ethernet, etc settings are set for a Desktop workstation, not a server. Most system admins know to tune these settings, otherwise in a lot of cases a multi-CPU box will perform as slow as a 1 CPU box. Anyway, at specific companies I worked with (AT&T Broadband / Worldnet in St. Charles, MO was one big one), all the servers were configured without the proper settings for a server, so we had all kinds of issues as a result, a big one is that the tcp accept queue is not set high enough and so connections to daemons will drop after a low number of connections, making it appear that the box can't handle the connections...., As a result, they had spent millions on numerous servers (in one situation they had over twenty 12-cpu servers just for smtp...

    These changes seem small, however, changing "ndd" kernel parameters on a Solaris box is not a single task, it is an infrastructure-wide task, and therefore requires the coordination of dozens of different groups, it really took a long long time to get this script implemented. It was called "S99nddfix" and it had all the ndd tunable parameters in it. Later when I worked at a different AT&T group in a different state, I noticed my script had been implemented on all the Solaris servers in the 200+ server environment.

    • by QuoteMstr (55051) <dan.colascione@gmail.com> on Sunday October 25, 2009 @09:40PM (#29868667)

      That's why to the greatest degree possible, libraries, programs, and algorithms should be auto-tuning. You can provide all the knobs you want, but people won't actually touch them. They'll choose which library, application, or operating system they use based on the default settings, so you'd better damn well make sure the default settings are good --- or even better, that you don't need settings at all.

    • by Kumiorava (95318) on Sunday October 25, 2009 @10:12PM (#29868789)

      I haven't worked for AT&T but at one point I tried to see what traceroute from San Diego to Finland would show me because ping was really slow. The traceroute I run jumped from west coast to east coast twice before going over the atlantic. I suspect that routing rules might need some fine tuning as well. It doesn't really matter if you have very fast network if the data keeps on jumping between servers creating extra traffic, I can imagine in my case the packet could have reached the destination with much fewer jumps.

      Of course that visual traceroute I used might not really give accurate locations of the servers.

  • Non-obvious cause (Score:5, Informative)

    by NixieBunny (859050) on Sunday October 25, 2009 @09:58PM (#29868737) Homepage
    If you take the time to RTFA, you will see that the problem with TCP management (as Mr. Turner describes it) is that you have to cause the system to drop packets occasionally when it's near but not quite at saturation, to let the TCP device at the other end know that the network is getting congested. If there are no dropped packets, TCP ups the packet rate until the network becomes clogged.

    So in this case, zero packet loss is a setup for disaster instead of a desirable quality.

    The trouble is that it's not an intuitive solution to a problem, the introduction of occasional packet loss. It's usually something to avoid.

    • Re: (Score:3, Insightful)

      by sjames (1099)

      The big problem is that it's exactly the opposite of the way ATM networks are managed. They can't seem to wrap their heads around the way it's done on an IP network.

    • by j-stroy (640921) on Monday October 26, 2009 @02:02AM (#29869715)
      As I recall, the story went: Mandelbrot was a mathematician at IBM lab. The engineers were attempting high speed data networking, but were encountering data/signal loss due to some noise. So like good engineers, they made things more robust, better isolation, grounds, shielding, etc. but the darn noise was still there.. They could not get rid of it. Determined to find the cause, they went to Mandelbrot with the request to analyze the noise, to determine its cause, in order to eliminate it.

      Mandelbrot examined the data and found that there were periods of clear signal interrupted by noise. He examined the noise and found that within it were periods of clear signal, interrupted by noise and so on. Hmmm... He astutely determined that "shit happens" and what was needed was a redundant protocol, not better shielding. The noise you see, was inherent in a damped and driven system.

      It was from this that he began his explorations of fractals and chaos theory, and we got robust network protocols.
  • Zero Packet Loss (Score:5, Interesting)

    by Anonymous Coward on Sunday October 25, 2009 @10:03PM (#29868761)

    Zero packet loss may sound impressive to a telephone guy, but it causes TCP congestion collapse and thus doesn't work for the mobile Internet!

    I was in the standardisation group that specified the RLC/MAC layer (ETSI SMG2, later called 3GPP TSG GERAN) and our priorities were not the behaviour of TCP. We were designing the radio layer to provide a bearer service for the higher layer protocols, at that time they were X25, IP (UDP and TCP). The "problem" we were trying to solve was the tendancy of the radio layer to fade, have multipath and generally lose packets. The RLC layer was designed to deliver error-free packets, in sequence over the radio layer. Generally that is exactly what it does, and does well. If it didn't then tehre would be no mobile internet.

    What we did find to be a significant performance problem was the asymetric channel. The uplink is usually the root of the TCP performance issues, UDP works much better. When the discrepancy is higher than 10, the downlink is ten times faster than the uplink, then the TCP Acks don't arrive in time and it stalls. Sadly a faster uplink is difficult and expensive to provide.

    • Re: (Score:3, Interesting)

      by NixieBunny (859050)
      It sounds like the solution might be to implement a custom version of TCP that takes the asymmetry of the physical radio channels into account. Since most mobile platforms have a much higher downlink packet count, a group ack method could provide relief to the unreliable uplink channel.

      Disclaimer: I've only designed one wireless packet data link system in my life, and it was symmetrical.

      • Re: (Score:2, Interesting)

        by BodeNGE (1664379)
        OP here. It could possibly be done between the IP stack of the device and the GGSN. The six layer stack diagram in the article does show the IP layer going between the two, but in reality it doesn't. On the mobile side there are usually two IP stacks. The GSM one to the GGSN access point (APN), and the stack presented to the client AP, or the the PC connected to it (calleed Terminal Equipment in GSM speak). If you tracert from the device you will usually see the device IP as a loopback address. The GP
  • by JoshuaZ (1134087) on Sunday October 25, 2009 @10:09PM (#29868779) Homepage
    Is to let AT&T engage in all the packet shaping and other fun stuff they want. Down with net neutrality? Wait? What's that? You think they'd manage to screw up their network even more then and probably fuck over other people in the process? Say it ain't so!
    • by JoshuaZ (1134087)
      And once again I can't spell even with the help of the preview. I'll go back into my little hole again.
    • Traffic shaping does not necessarily effect net neutrality. Also, I'd see the packet shaping techniques of yours, are the outgoing packets rounder?

  • "A Possible Cause of AT&T's Wireless Clog — Configuration Errors"

    Oh wait, this is slashdot, we test on the live system.

  • Router fairness (Score:5, Informative)

    by Animats (122034) on Sunday October 25, 2009 @11:24PM (#29869033) Homepage

    TCP measures round trip time, and doesn't need packet loss to tell it that the round trip time is long. The retransmit interval will go up appropriately. TCP will behave reasonably with a long round trip time. If you're trying to do a bulk transfer, there's nothing wrong with this. The problem comes when short messages and bulk transfers are sharing the same channel. The short messages can spend too much time in the queue.

    The solution is reordering the packets, not dropping them. That's what "fair queuing" is about. It may be worthwhile to implement fairness at the port-pair level, rather than the IP address level, at entry to the air link. Then low-traffic connections will get through faster.

    "Quality of service" can help, but it's not a panacea. The network layer can't tell which of the TCP connections on port 80 is highly interactive and which is a bulk download, other than by traffic volume.

    (I used to do this stuff [faqs.org].)

    • Re: (Score:3, Informative)

      by Trepidity (597)

      I was about to post a comment on this article complaining that "TCP doesn't work unless routers drop packets" is oversimplifying how TCP works at best, partly by citing RFC 896, then I come and see the author of the damn RFC beat me to it. This discussion in the article of buffers so large that they never fill (so can in effect be considered infinite) makes RFC 970 [faqs.org] seem relevant, also.

    • Re: (Score:3, Informative)

      When TCP receives an ACK it can dump it's whole window size of packets onto the wire. A large number of TCP streams sharing a link with large transmit buffers and a small common bottleneck will saturate your transmit buffers. Yes TCP will tend to back off. But when you have heaps of streams all trying to transmit data, they collectively try to ramp up again too quickly.

      I used to share an ADSL link with a neighbour of mine. I would often see the round trip latency over the ADLS link and back go up to 1 to 1

The major difference between bonds and bond traders is that the bonds will eventually mature.

Working...