Twitter Throttling Hits Third-Party Apps

Barence writes "Twitter's battle to keep the microblogging service from falling over is having a dire affect on third-party Twitter apps. Users of Twitter-related apps such as TweetDeck, Echofon and even Twitter's own mobile software have complained of a lack of updates, after the company imposed strict limits on the number of times third-party apps can access the service. Over the past week, Twitter has reduced the number of API calls from 350 to 175 an hour. At one point last week, that number was temporarily reduced to only 75. A warning on TweetDeck's support page states that users 'should allow TweetDeck to ensure you do not run out of calls, although with such a small API limit, your refresh rates will be very slow.'"

75 updates per hour

[VisiX]

Any information that needs to be distributed more than once per minute probably shouldn't be relying on twitter.

Protocol overhead

[ickleberry]

I wonder if it would have much of an impact if they switched from the verbose JSON/XML over HTTP formats for the API to a binary UDP-based protocol. Twitter seems well suited to such a protocol since it is so simple and the messages ar so short

Is it that they are doing too much processing on the data, wasting too much bandwidth or is their database causing trouble? Since its twitter obviously any bandwidth used is a waste, but you know what I mean

Re:Are They Employing an Event/Listener Paradigm?

[Late Adopter]

I agree, that's the "right" way to tackle subscription mechanisms. But it's not the right way to tackle Twitter, because one of the defining features of Twitter is its ubiquity: i.e. if you have a phone/computer/netbook that's capable of running any sort of app whatsoever, you can run a Twitter app. As it stands now to write a Twitter client, you need to be able to do HTTP GET requests (every modern environment provides for this) and parse XML. That's it. But to do pub/sub, you'd presumably need to be able to listen, which you can't always do, say, on a smartphone or a Firefox extension.

Uhh, what are you talking about?

[Anonymous Coward]

What are you talking about? That pattern works well under very controlled circumstances, like UIs, but falls apart over networks.

What happens when a client is behind a heavily NATed network, or behind a firewall, or forced to use a proxy? Twitter can't contact them directly to push the new data. That's one of the benefits of the web; the client pulls the data, rather that it being pushed to them, which often isn't an option.

What about clients who don't have a constant connection to the Internet, or who have a dynamic IP? Now twitter has to poll them, to see if they exist. You end up with the same situation, except worse.

What happens when devices disappear, but no longer alert twitter to them no longer being a subscriber? If my smartphone gets run over by a truck, it won't have a chance to alert twitter that it's no longer a subscriber, because it'll be totally fucked up. Will twitter keep trying to push updates to it constantly?

This is a good example of why software design patterns are dangerous. People like you don't seem to understand where they can apply, and where they totally fall apart.

Re:It's time to ditch the NoSQL bullshit.

[Amouth]

Lowes hardware does - there is a local server in the store that serves as a caching server only if the main trunk fails.

Re:It's time to ditch the NoSQL bullshit.

[Knux]

Any telecom does way more than that.

I've worked in a big telecom with 40mi+ clients and I've seen an 8 nodes Oracle RAC responsible for the whole pre-paid client database handle far, far more transactions and queries than Twitter says it does.

Each regional server responsible for authorizing the calls has a 2 node Oracle RAC and it too handles far more transactions and queries than Twitter.

So, there you go... The excuse to use NoSQL was that it is quicker in some cases. It's not, time to move back to RDBMS.