Wrong Number: Why Phone Companies Overcharge For Data 105
MrSeb writes "A recent study (PDF) conducted by UCLA professor Chunyi Peng shows that carriers generally count data usage correctly, but those customers who commonly use their device in areas with weak signal strength or to stream audio or video are often overcharged. Peng and three other researchers used data gleaned from an app installed on Android smartphones on two different carriers. The issue appears to be in how the system is set up to count data usage. Under the current scenario, data is charged as it is sent from the carrier's network to the end user. What does not exist is a system to confirm whether the packets are received, and thus preventing charges for unreceived data. Peng demonstrated this in two extreme circumstances. In one case, 450 megabytes of data was charged to an account where not a single bit of it had been received. On the flipside, Peng's group was able to construct an app which disguised data transfers as DNS requests, which are not counted by the carriers as data usage. Here they were able to transfer 200 megabytes of data without being charged. Overall, the average overcharge is about 5-7% for most users. While that does not seem like much, with unlimited plans gone and data caps in style that could pose potential problems for some heavy data users. Could you be going over your data allotment based on data you never received? It's quite possible."
DNS not counted? (Score:3)
DNS requests, which are not counted by the carriers as data usage.
I'd love to see a source for this.
Well, a source to anything actually, but it seems to much to ask around here these days...
Re:DNS not counted? (Score:5, Funny)
Re: (Score:2)
Hey man, quiet, please! Like every single airport/etc which tries to charge (exorbitantly) for connectivity passes DNS through, so they can intercept connections and display their paywall page. If more people know about iodine, these holes will soon be plugged. A good part of countries has nazi laws forbidding open hotspots, so you can't get those, too.
Re: (Score:3)
Re: (Score:3)
That's assuming port 53 traffic is free rather than DNS traffic being free - not the same thing.
Re: (Score:2)
Re: (Score:3)
Actually many hotspots *do* require you to use the local DNS server. It's pretty trivial, since that's part of the DHCP config, so it's not like end-users have to do anything to make that work. In those cases your data needs to be close enough to the DNS protocol to get propagated upstream, which is basically what iodine does.
Also note the SSH more or less requires TCP, so even if you allowed port 53 outbound you could limit access to the UDP protocol and effectively block SSH and many other tunneling tools
Re: (Score:2)
If port 53 is unhampered, iodine will just put the traffic there -- but its primary purpose is encapsulating packets in real DNS packets.
Imagine queries like "TXT packet.123.dns-encoded.contents.are.abcdefgh123456.iodine.slashdot.org". The encoding is similar to uuencode, except for a different set of legal characters, and not all characters being legal at the beginning or end of a word.
Countermeasure implimentation in (Score:1)
3...2...1...
Seriously though, if airports, etc., wanted to enforce their paywall, they can just drop all packets that aren't destined for "inside the free walled garden" and be done with it.
DNS or other UDP or TCP access to a machine inside the walled garden e.g. a "pay me" web site? Traffic allowed.
DNS or other port UDP or TCP access to outside the garden? Drop or re-route to a fake server at that address.
As for ISPs and cell-phone-data providers that offer free DNS lookups, expect most of them to start
Re: (Score:3, Insightful)
Re: (Score:2, Interesting)
Re:DNS not counted? (Score:5, Informative)
The operators that my employer delivers charging solutions to rarely use the byte counters from the GGSN because they charge he traffic types differently, e.g. "free facebook" while other traffic is charged based on byte counters from the DPI box. It is true that some of the newer GGSNs/ASN-GWs have adequate DPI capabilities so they can classify the traffic with enough granularity and the byte from them can be used.
There is one slightly fishy thing in the study (yes, I read the fine paper). Their test with logging on, go idle, move to radio-inaccessible room, then have server start steaming UDP to the phone (which will be dropped due to inaccessibility). In my experience the SGSN/GGSN quickly signals that the user has gone offline (PDP session termination), and the stream of UDP from the server is blocked at the DPI or the GGSN. Sounds like the operator that the study used has a major bug in its charging setup where PDP session termination doesn't also stop the IPuser association.
Re: (Score:2)
I'm not sure about sensationalism, but the article certainly has a strong bias.
Re: (Score:2, Insightful)
A strong bias and a poor conclusion. Rather, a poor methodology. Since you also appear to work in the industry - I'm wondering if you notice what I notice? The two methodologies they used for detecting traffic at the mobile... They used an android application that does not require root access (e.g. isn't taking pure IP level statistics) and the Traffic Status API.
Two pretty big problems. 1) They don't measure TCP Retransmissions... This is uhh a big "duhh" - but more disturbing they don't measure IP
Re: (Score:2)
No, I didn't notice that they didn't measure the L3/L4 overhead. I'm not familiar with the two APIs/tools they used. But if that is the case then it invalidates the majority of the study.
Re: (Score:3)
I'd love to see a source for this. Well, a source to anything actually, but it seems to much to ask around here these days...
Seems to me the study is the source. Called a "primary source" where I come from.
Re: (Score:2)
If only it worked both ways (Score:5, Funny)
Well I sent a check for my monthly bill... not my fault you didn't receive it.
Re:If only it worked both ways (Score:5, Funny)
Reminds me of a joke. Telephone lineman is recruited into the army. Gets to the rifle range for practice. Everyone is shooting, targets are being hit by all but the lineman. Sarge comes up and asks why his target got no hits on it. Lineman ties a rag over the end of his rifle and fires off a round, shows the sarge the rag with a hole in it and says, "No problem at this end."
Link to study (Score:5, Informative)
Re:Wow, your mobile networks SUCK. (Score:5, Funny)
Thank you for your suggestion. at&t(R) is committed(tm) to rebuilding the nation's largest 4G network this year with your input. To pay for the buildout* you will notice a 10% 2012 Nation's Largest 4G Network Improvements Fee along with a 200% SMS price increase. Thank you again for choosing at&t(R), with the nation's largest 4G network.
*Buildout subject to cancellation without notice. Just kidding about that last part, we cancelled it while you were reading that sentence. The fee stands to pay for the costs of typing the prior two and commissioning a forthcoming Gartner study to validate their market impact.
they could charge more... (Score:5, Funny)
Step 1 (Score:4, Funny)
1. write an app that disguises streaming porn as a DNS request
2. ???
3. Profit!
Re: (Score:3)
http://code.kryo.se/iodine/ [code.kryo.se]
It turns out you can already disguise all of your traffic as a DNS request.
Re: (Score:2)
Re: (Score:2)
But what if I need to disguise a DNS request (Score:1)
or disguise a disguesed DNS request
or, oh, curses, refoiled again!
Re: (Score:3)
The nature of TCP includes positive acknowldgement and retransmission. If you don't get that acknowledgement, don't charge the user for that packet.
Re: (Score:3)
To get rid of the taste of the food.
Re: (Score:1)
Because they don't have thumbs...
Re: (Score:2)
Probably because of the peanut butter they smeared on there.
The trouble is. . . (Score:5, Insightful)
Resending a packet due to a missed ACK takes up air time, just like it did sending it the first time, and the carriers have no control on where the user will be. If they make their systems robust enough to move their present average packet reception rate from an already-good 93-95% to, say, 99%, this will only enable their users to move down another floor in their sub-basements, or another few city blocks, or another cubicle row deeper into the building, before the average goes back down again -- after all, wireless systems have limited range. The cost of the new infrastructure would be roughly twice that of the previous one ("increasing coverage is increasingly expensive"), and you're going to pay for the cost of the infrastructure either way in your air-time charges.
Look at it this way: Even if the company only charged for packets successfully received, it would just increase their rates by (1/0.95) - 1 = 5.3% to (1/0.93) - 1 = 7.5% to maintain the same cash flow. Plus it would have to start keeping track of the success or failure of each packet transmitted, and put that into its billing scheme. That's a database PITA I don't want, thank you very much.
Re:The trouble is. . . (Score:4, Informative)
Keep track isn't hard. Telco's already have optimisation for tcp re-delivery from the mobile gateway so that the distant sender doesn't have to re-send the missing packet, the telco can do that.
This service improves tcp performance over a mobile network and is important for customer retention.
Maybe not all telco's do it but I doubt it.
http://www.iith.ac.in/~tbr/teaching/docs/transport_protocols.pdf [iith.ac.in]
http://www.cs.sunysb.edu/~jgao/CSE370-spring06/lecture17.pdf [sunysb.edu]
Re:The trouble is. . . (Score:4, Informative)
Indeed, all sorts of TCP splitting techniques exist. However, there is only so much data such a device can temporarily queue to keep retransmission on the terrestrial side. If you run a network with 10 million subscribers, it becomes very interesting.
The mismatch comes from the fact that operators collect CDRs on the terrestrial side of their GGSNs. So even if the mobile subscriber does not need to resend a segment, the terrestrial retransmission is still accounted, as are the duplicate ACKs sent by the Internet host.
You simply can't expect both having the cake and eating it. High latency links come with trade-offs.
I work for a provider with much higher RTTs (~1200ms). The 5% reported by the study is exactly what we are seeing.
Re:The trouble is. . . (Score:5, Interesting)
Also, most people have no idea of the optimization techniques operators use.
Navigate to any content heavy website. If your mobile browser allows you to, try to see the source of the page.
Chances are you will see all whitespace trimmed, all CSS and JS inlined. All pictures are compressed in a lossy
fashion to reduce their size.
There is also HTTP request coalescing. If you request "/", the whole page will be retrieved, then processed as
above and delivered to the mobile browser in a single reply. How many GET requests do you save? A lot.
If there were no such techniques, one's mobile bill would be almost twice as high and the browsing experience
would be 4 times as slow.
Re: (Score:3, Interesting)
HTTP request coalescing is a mixed bag, since the all the stuff you inline will be resent when you request the next page. If you have carefully reduced the size of your html pages and set your CSS and JS to be basically cacheable forever, you may want to add Cache-Control: No-Transform to your headers to request that no such thing is done. The same header should theoretically prevent your images from being re-compressed with an absurdly bad JPEG quality setting.
Re: (Score:3)
Why is it then, that bad wifi on my phone is immensely faster than good Cell (3G)? One or two bars on wifi loads so much faster than full bars on my 3G? Is their network that over saturated that even with a good signal that they just can't transmit? It is even more obvious on mobile optimized sites. Just sayin.
Re: (Score:1)
Even with bad WiFi, your RTT should still be an order of magnitude smaller than on the 3G network.
I suppose it also depends on your 3G network. I am in the UK and my service works well enough that
I don't have to bother finding WiFi spots at all, I just use 3G.
Re: (Score:2)
That's doubtful. There is a feedback loop where transmission gains encourage webisite operators to be profligate with their content, because the hardware side can "take it".
Look at Google's homepage for a dramatic example of this over the last 10 years. If there hadn't been all the efficiency gains at the network level, chances are we'd have a faster we
Re: (Score:1)
Whoops, replied to myself a 2nd time instead.
Also, most people have no idea of the optimization techniques operators use.
Navigate to any content heavy website. If your mobile browser allows you to, try to see the source of the page.
Chances are you will see all whitespace trimmed, all CSS and JS inlined. All pictures are compressed in a lossy
fashion to reduce their size.
There is also HTTP request coalescing. If you request "/", the whole page will be retrieved, then processed as
above and delivered to the mob
Re: (Score:2, Insightful)
and if the phones allowed ad-blocking, your data use will drop significantly.
I'm fine with ad's in a unlimited ecosystem. but in a bandwidth limited, receiver pays, system? Screw all ad's. It's why I jailbroke my iphone and installed an ad blocker, and my android phone also had ad blocking added to it.
Re: (Score:2)
Re: (Score:1)
Mobile operators collect CDRs on the terrestrial side of the GGSN. Even if over the air there are no retransmissions,
there will still be some on the IP network. I work for a provider with higher RTTs (~1200ms) and the extra 5% is pretty
much what we are seeing. We all use the same TCP splitting techniques (Vegas on the slow link and Reno on the
fast link), because the primary purpose is to improve the user experience.
Navigate to any content heavy website. If your mobile browser allows you to, try to see the s
Re: (Score:2)
Keep track isn't hard. Telco's already have optimisation for tcp re-delivery from the mobile gateway so that the distant sender doesn't have to re-send the missing packet, the telco can do that.
Keeping track of data that's delivered may not be hard, but if you accept that the Telcos charge so much money for data because it costs a lot of money to deliver data to your phone, it's that last mile of wireless delivery that's the most expensive. It costs them nearly nothing to get the data from the internet to their cell tower, the expensive part is in maintaining the infrastructure to get the data from the tower to your phone.
As long as they are only counting costs for data when it leaves the cell tow
Re: (Score:1)
Change your database....
Re: (Score:2)
If they get to charge for packets never received, you can never know what your bill will be. If they raise their rates to account for 'breakage' (like a typical store will), then at least you know what the bill is going to be and you have a fair comparison of costs between carriers.
Also, we want to give the telcos an incentive to improve their network, not to just let it degrade and use the packet loss as a stealth price increase.
The water works doesn't charge you for their leaky main, the power company doe
Re: (Score:3)
Data takes up network capacity whether the device receives it or not.
if you're an operator..
step 1. hire some flooders.
step 2. profit.
essentially this is why they should sell by speed, not by usage.
Re: (Score:3)
step 1. hire some flooders.
While I don't expect legit carriers to crap-flood for profit, it seems this presents a perverse economic incentive not to invest in improving the quality of their networks.
Vodafone Netherlands (Score:5, Interesting)
* Android Droidstats usage logger: 369 MB (2012-07-31 22:16h)
* Android "My Data Manager": 337 MB (2012-07-31 22:16h)
* Vodafone online usage monitor: 307 MB (up to 2012-07-30 17:46h)
* Phone bill for July: 343 MB (since a couple of months they actually mention the total; before I needed to use a perl script to parse the PDF invoice and add the data usage of some 200 separate data sessions)
When I asked about the differences a few months ago, the Vodafone customer service told me: "The information on your Vodafone account online is the real usage. Numbers from data usage apps are not reliable." But I highly doubt that I used 36 MB over the last day of the month, so it seems that within Vodafone they have different systems.
My train commute (where I use most of my data) passes through an area with bad coverage, so I would have expected a bigger difference based on the theory that packet loss accounts for most of the difference.
Re: (Score:2)
When I asked about the differences a few months ago, the Vodafone customer service told me: "The information on your Vodafone account online is the real usage. Numbers from data usage apps are not reliable." But I highly doubt that I used 36 MB over the last day of the month, so it seems that within Vodafone they have different systems.
Most likely the numbers in their on-line usage monitor are not truly up-to-date. The Vodafone website here in the Netherlands is not always the best example of engineering.
Re: (Score:3)
With AT&T, I discovered that data usage takes about a day to be entered into the system, so the extra data could have come from the prior billing cycle.
In my case, I had about 800 MB of data left on the last day of my billing cycle so I decided to download a 300 MB file. The data usage didn't show up in AT&T's system until the next day, so despite showing the correct usage date on my bill, the data was applied to next months data cap.
Basically, don't trust usage monitors, apps or even your bill to
Word Play (Score:2, Insightful)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Word Play with UPS (Score:2)
Don't get so hung up on the UPS Service Guarantee [ups.com] (section 47, pdf page 32, paper page 29):
In the event UPS fails to attempt delivery within the time published on the UPS website, or as provided when 1-800-PICK-UPS® is called, UPS, at its option, will either credit or refund the transportation charges for each such package to the payer only, upon request, provided the conditions set forth in the UPS Service Guarantee are met. Transportation charges do not include other fees or charges that may be assessed by UPS including, but not limited to, fuel surcharges. This is the sole remedy available under the UPS Service Guarantee.
UPS shall not be liable for any damages whatsoever for delayed delivery, except as specifically provided for shipments made under the UPS Service Guarantee. Under no circumstances shall UPS be liable for any special, incidental, or consequential damages including, but not limited to, damages arising from delayed delivery or failure to attempt on-schedule delivery.
UPS may cancel or suspend the UPS Service Guarantee for any service(s), and for any period of time, as determined by UPS in its sole discretion, and without prior notice.[Emphasis added.]
What follows (in Section 47.1) is seven bullets of conditions, followed (in Section 47.2) by eleven bullets of exclusions.
I don't have a problem with UPS -- they've always treated me, and my packages, well -- but I'm not under any illusions that I could actually get a court judgement from them based on their terms of service, should they decide not to refund their shipping charges on a lost parcel, and I decided to su
Carriers are businesses (Score:4, Insightful)
Re: (Score:1)
What people forget is that it costs a *lot* of money to run a network, especially if that network is going to be any good. No one is forcing you to use the service - if you don't want to spend the money, then just step away from the mobile phone, and go home and use your ADSL or Cable, or even
Re: (Score:3)
Re: (Score:2)
The problem is that carriers also charge for a voice call what it takes to connect that voice call, and keep the call that the message traveled over running -- even though it's the same infrastructure used for text messages. Nothing like double billing.
Of course not. Employee salaries are an expen
Client side... (Score:2)
The telco can only rely on the data from their own equipment, that is they know the data was transmitted to the user but have no way to reliably tell if it was received.
If they were to ask the user's device, then the device could lie in an attempt to get free use...
Re: (Score:2)
For TCP, they could look at retransmissions. For UDP, it's protocol dependent, of course.
However, the deeper flaw with the article is the assumption that you should only pay for what you receive. In fact, your request of the data is what uses up bandwidth, regardless of whether you receive it or not, so you should actually be charged by what's transmitted, not what's received.
Re: (Score:2)
The telco can only rely on the data from their own equipment, that is they know the data was transmitted to the user but have no way to reliably tell if it was received.
If only TCP had some way to acknowledge that a packet was received by the endpoint so that connection participants knew if a packet was lost....
Re: (Score:2)
Actually, they can. If the client doesn't ACK the data sent, the connection stalls while the same data is sent again. Because of that, a phone that never acked data would have no useful data connection at all.
Re: (Score:2)
For TCP yes, but for UDP based protocols its much harder... So you'd be able to get round the bandwidth caps by using a custom udp based tunneling protocol where it was impossible to tell if a packet had been acknowledged.
Re: (Score:2)
Or they could also just count UDP based on transmission (since TCP is the bulk of the traffic).
Incorrect statement in summary (Score:2)
>"While that does not seem like much, with unlimited plans gone and data caps in style that could pose potential problems for some heavy data users."
I guess the author never heard of Sprint, which has unlimited plans and without data caps or throttling.
Perhaps the statement was out of context, I don't know. But it is misleading.
Re: (Score:2)
> I guess the author never heard of Sprint, which has unlimited plans and without data caps or throttling.
When your average EVDO speeds are 80-100kbps on a GOOD day, there's no need to cap or throttle. Someone did a study a few months ago comparing data speeds in South Florida, and even MetroPCS was faster than Sprint.
Re: (Score:2)
In many areas, Sprint's 3G speeds are quite slow. Unfortunately, they are here too. The main problem is that Sprint is over-subscribed. But not all areas are slow. Plus, the areas that had WiMax had plenty of speed. Unfortunately, I am also in an area that never got WiMax. PLUS the LTE areas are very fast. Unfortunately, we don't have that yet either... but it is due pretty soon (thank God).
I will point out that there was and is no cap or throttling on Sprint's WiMax or LTE...
Re: (Score:2)
Sprint's wimax was a disappointment, but it's what made them tolerable while their EVDO service deteriorated to levels that would have been laughed at by someone paying 250 Rupees/month for unlimited 1xRTT 100kbps service in India. The fatal mistake made by Sprint's management, which I believe is going to cost them *dearly* over the next 12 months in terms of lost customers, was to throw away everything they gained with wimax by not giving us a year or two of phones with dual-mode LTE+wimax 4G radios. Had S
Conflict of Interest (Score:2)
Surely this means that network operators have no incentive to increase their coverage?
IP over DNS using Iodine (Score:2)
http://code.kryo.se/iodine/ [code.kryo.se]
I've played with IP over DNS, and it works surprisingly well. It can break through most firewalls. I think there was something like a 50% performance hit, but considering how convoluted it is, that's pretty good.
It shouldn't be difficult to port it to Android if you install a kernel with tun/tap support.
Re: (Score:2)
Yes, it's much better than IP over SMS. It depends on the DNS server, but you can sometimes get most of the 1500 bytes available, minus the encapsulation overhead. You have to set up your own domain and DNS server, and it's most efficient if you can send DNS directly to your own server, but that's not required.
Iodine doesn't have encryption or compression, so it's recommended to open an ssh connection and tunnel everything through that. Of course, it's pretty obvious what you're doing if anyone looks at
I still have an unlimited data plan (Score:3)
So wait... (Score:3)
All I need to do is run a website or service on UDP/53 and mobile users wont ever get charged? It can't be that easy.
why is this an "overcharge"? (Score:2)
The article makes the assumption that people should only be charged for packets actually received. But the company's wireless infrastructure is busy while it is transmitting packets, whether received or not. Bandwidth could be entirely saturated even if no client ever receives anything.
It seems to me that people should pay the way they are, and if they don't like paying for dropped packets, they should turn off data when in marginal areas. Of course, coverage should also be measured that way: if you are
Why?! (Score:1)
Why Phone Companies Overcharge For Data
Shouldn't that read "How Phone Companies Overcharge For Data??"
(I would think the why would be obvious to anyone who's not a mouth-breather...)
That why i have Sprint. (Score:1)
Re: (Score:2)
Re: (Score:2)
Check s4gru.com for updates!
Cheers
This article is BS - let's clarify (Score:1)
First, carriers "optimize" traffic which actually saves the subscriber money. I won't get into the details - but most carriers around the world deploy optimization techniques that save their subscribers money at the carrier's expense.
Second, subscribers are charged for IP packets, this is true. That includes TCP retransmissions. Subscribers are not charged for air-interface retransmissions. What I mean to say here is that there are multiple layers that involved in ensuring packets make it to user equipm
Another Exploit (Score:2)
Re: (Score:3)
Another thing is, you could attempt to download far more data than your connection is capable of. For example, making a portable device participant in a BitTorrent network, where the client intentionally attempts to download as much data as possible -- at a high loss rate, the data your phone isn't capable of receiving still gets sent by the tower, and therefore still congests the wireless portion of their network, but your device never receives the data, because it's not capable of it.
Packet loss is