Twitter Throttling Hits Third-Party Apps 119
Barence writes "Twitter's battle to keep the microblogging service from falling over is having a dire affect on third-party Twitter apps. Users of Twitter-related apps such as TweetDeck, Echofon and even Twitter's own mobile software have complained of a lack of updates, after the company imposed strict limits on the number of times third-party apps can access the service. Over the past week, Twitter has reduced the number of API calls from 350 to 175 an hour. At one point last week, that number was temporarily reduced to only 75. A warning on TweetDeck's support page states that users 'should allow TweetDeck to ensure you do not run out of calls, although with such a small API limit, your refresh rates will be very slow.'"
Re:Monty Python (Score:5, Insightful)
Company bases a business model on offering their resources for free, only to discover to their chagrin that people will take them up on it. Where oh where have I heard this one before?
175/hr is slow? (Score:5, Insightful)
Re:175/hr is slow? (Score:5, Insightful)
But I am inclined to comment about this bit of "news"... Big. Woop. Twitter's just trying to stay alive. If the service falls over NO UPDATES will happen... at all... Inconvenient, yes, but totally necessary.
Re: (Score:2)
Incoming conspiracy theory in 3, 2, 1...
Re: (Score:1)
Re: (Score:2)
Thanks for that, the article wasn't clear if it was X/hr for an app run by a given user in total, or X/hr for that app for a given person they're following.
It seems inefficient that TweetDeck is sending 10 different requests; can Twitter's API not handle a "tell me if anyone I'm following has updated" request, to allow 10 requests to be rolled in to one? Admittedly, that would put additional burden on the twitter servers to keep track of what "anyone I'm following" means.
Re:175/hr is slow? (Score:4, Informative)
The API rate limit is per hour per user (if authenticated) and per IP if not authenticated. Unfortunately the Twitter API does not allow you to aggregate requests even if their web site does (e.x. status updates for all of the people I'm following and all of the things people I'm following have retweeted). If you go through the API docu, you'll find all sorts of horrid seeming inefficiencies and awkwardness with the API.
For instance when you request a status (or a list of statuses or whatever) you'll get back: the contents of the tweet, the user name, user id, URL for the user avatar, URL for the user's profile page background image, whether that user is following you, their real name, the number of tweets that user has made, and so-on and so forth. A lot of this information could easily be cached by the client, but is instead sent for every tweet you get back.
Re: (Score:2)
If you're only following a single feed. But I have like 10 lists in TweetDeck that all get individually queried, and there are some who have WAY more than that.
So why does it take multiple API queries? Shouldn't it grab everything all at once, perhaps with the option to prioritize the current window worth of results? I can understand requiring a separate call to view profiles, or perform a search, but whether I'm following 1 person or 100 people, I expect it should take just 1 call to receive all the tweets.
Re: (Score:2)
I think he means that he has TweetDeck set up to monitor several lists (a feature in twitter to allow a feed of updates from users not on your main "follow" list) which means that the requests for what's in each list will have to happen independently. Correct me if I'm wrong...
What Twitter really needs to do is require heavy-hitting API-using apps like TweetDeck to maintain their own mirror of tweet activity, a replicated database of some sort that way when users are craving full length updates of ten list
Re: (Score:2)
Yeah their API is very simplistic - if they made some changes this wouldn't be an issue.
As it stands right now, if you wanted to check all your messages, once per minute you would need
1 Request for Direct Messages
1 Request for messages @ you
1 Request for timeline (people you follow)
1 Request for each list's timeline (you can list people you don't follow)
So if the user has 5 lists that's potentially 8 requests every 60 seconds.
The other problem is that you can't get more than 200 messages per request - so if
Re: (Score:2)
What they should probably do is make an all inclusive query that allows you to specify that you want DM,Timeline,LIST1,LIST45 updates, and it would provide you with all of those, with an XML/JSON field that indicated what the message source was. They would have to increase the messages per request limit to something reasonable like 1000-5000, and allow requesting say 10 different sources at once.
I think 200 messages is a reasonable return rate, if only one universal query is required, instead of multiple. That's still an average of one API call per 20 seconds (say, query all every minute, that still allows two other calls per minute for messages out or overflow updates) which should be enough for most reasonable uses.
If you're getting more than 200 messages per minute, there might need to be a reevaluation of your Twitter usage. That's a lot for an individual to read. Anyone who wants to do so
Re: (Score:2)
200 isn't a lot if you're doing filtering on the client to weed out messages you didn't want to see. Not everything a person tweets may be interesting, you might want a subset of their tweets - or maybe all tweets without URLs, etc.
This is especially true related to language filtering, which is currently broken server-side on Twitter (and has been for OVER SIX MONTHS). I guess I shouldn't be surprised, considering that even their website has a minimum of one bug every time I try to use it.
Re: (Score:2)
I (@garydunn808) use TweetCaster on my Android phone with refresh interval at 30 min. Two interuptions per hour is all my right brain can handle.
Re: (Score:1)
This is really how it works? Come on, what decade is this? I've been on the user side and now I'm on the vendor side of packet based application performance products. Think wireshark or the defacto standard certain brand name that jumps into your head. A primary part of the job is showing people how inefficient their database calls are when they either ask for everything every time and don't cache it, or they get tiny bits and pieces a few bytes at a time instead of larger more efficient downloads.
So Tw
Re: (Score:2)
If twitter doesn't want to fall over, it should stabilize itself by scaling up its hardware, not by jabbing its cane into its users' asses.
Because that's a good way to stay upright, and be alone.
Re: (Score:2)
It seems somewhat silly if you need to check feeds separately. Why not say, "Have A,B,C, or D said anything?", and get a batch of replies? After all, in theory Twitter already knows who you are following, so you probably don't even need to ask for most things.
Also, why do we care about per-second updating from Twitter? Perhaps some people do -- ma
Re: (Score:2)
Twitter's just trying to stay alive. If the service falls over NO UPDATES will happen
What's the expression I'm looking for? Oh, yeah: "And nothing of value would be lost."
Re: (Score:1, Insightful)
A RSS reader will consume one update per person you follow on twitter. Following a moderate number of people with a 15 minute refresh will easily break the cap.
Re:175/hr is slow? (Score:5, Funny)
Isn't that an update nearly every 20 seconds? How fast do people need to see that you're currently wiping your butt?
It seems you have forgotten how full of shit the average Twit is.
Re: (Score:2)
Re: (Score:2)
Must ... not ... click ... poop ... link ...
Re: (Score:1)
popular != valuable (Score:2)
Move along to the next topic, please.
Re: (Score:2)
Naw, it's millions of people checking to see if Ashton Kutcher is wiping his butt.
75 updates per hour (Score:4, Interesting)
Re: (Score:1, Insightful)
You need to think in terms of API calls. If it takes an API call to get one of your follower's updates, following 100 people could push your refresh rate over an hour.
Re: (Score:2)
Does it really work this way rather than one call to get all new messages of people you are following?
Are They Employing an Event/Listener Paradigm? (Score:5, Informative)
Over the past week, Twitter has reduced the number of API calls from 350 to 175 an hour.
Okay, if you're making that many calls to Twitter then there might be an inherent flaw with their RESTful interfaces. I think for a long time, the "web" as we know it has suffered from the lack of the Event/Listener paradigm. This is a pretty simple design concept that I'm going to refer to as the Observer [wikipedia.org]. Let's say I want to know what Stephen Hawking is tweeting about and I want to know 24/7. Now if you have to make more than one call, something is wrong. That one call should be a notification to Twitter who I am, where you can contact me and what I want to keep tabs on--be it a keyword or user. So all I should ever have to do is tell Twitter I want to know everything from Stephen Hawking and everything with #stephenhawking or whatever and from that point on, it will try to submit that message to me via any number of technologies. Simple pub/sub [wikipedia.org] message queues could be implemented here to alleviate my need to continually go to Twitter and say: "Has Stephen Hawking said anything new yet? *millisecond pause* Has Stephen Hawking said anything new yet? *millisecond pause* ..." ad infinitum. I'm not claiming Twitter does this but a cursory glance at the API [twitter.com] looks like it's missing this sort of Observer paradigm that allows for the scalability they need.
...
I'm not leveling the finger at Twitter, it's a widespread problem that even I have been a part of. Ruby makes coding RESTful interfaces so easy that it's very very tempting to just throw up a few controllers that are basically CRUD interfaces for databases and to call it a day. I suspect that Twitter is feeling the impending pain of popularity right about now
Re: (Score:1, Funny)
Informed opinions based on logical thought and understanding of programming concepts are not welcome here.
You must be new.
Re: (Score:1, Informative)
Re: (Score:2)
The biggest problem is that when Twitter (or whoever) goes to deliver the update, at the user's home network a router or firewall will block Twitter from connecting. Of course this can be overcome if the client sends a heartbeat packet via UDP at regular intervals to Twitter so that the router thinks you're actively communicating, so when Twitter pushes data back via UDP the router knows who it's for and lets it in.
Of course, UDP isn't exactly a standard web tool. I know ASP.NET supports it through .NET,
Re: (Score:3, Interesting)
Re: (Score:2)
Correctumundo! Tell him what he's won.
Re: (Score:3, Informative)
XMPP would at least take advantage of established listening pathways (GTalk clients on mobile devices, etc).
Uhh, what are you talking about? (Score:1, Interesting)
What are you talking about? That pattern works well under very controlled circumstances, like UIs, but falls apart over networks.
What happens when a client is behind a heavily NATed network, or behind a firewall, or forced to use a proxy? Twitter can't contact them directly to push the new data. That's one of the benefits of the web; the client pulls the data, rather that it being pushed to them, which often isn't an option.
What about clients who don't have a constant connection to the Internet, or who have
Re: (Score:3, Insightful)
What about clients who don't have a constant connection to the Internet, or who have a dynamic IP? Now twitter has to poll them, to see if they exist. You end up with the same situation, except worse.
E-mail seems to be doing just fine, despite these "shortcomings".
Re: (Score:1)
Re: (Score:3)
Yawn, just because the client isn't polling (REST, is just a way of saying polling to make people feel better), doesn't mean this doesn't work on just about every damn device out there. TCP keep-alives are supported by all the major TCP stacks and all the minor ones I've ever used (although not strictly required per RFC 1122). With reasonable configuration parameters for maintaining connections with little data transfer, its possible to keep a port open for basically an indefinite time period. Once the port
Re: (Score:2)
Simple pub/sub [wikipedia.org] message queues [...]
And therein lies the rub. The problem is that pub/sub message queues are neither simple nor scalable as the problem size increases. From the WP article you cited:
As noted above, while pub/sub scales very well with small installations, a major difficulty is that the technology often scales poorly in larger ones.
You pretty much don't get a larger organization than the public internet, assuming that a service becomes popular enough. I've seen these problems myself in one particular large internet company I worked for. Message queue systems are great tools, but there are a ton of practical problems in implementing an event/push based system broadly.
On th
Re:Are They Employing an Event/Listener Paradigm? (Score:5, Informative)
Now if you have to make more than one call, something is wrong. That one call should be a notification to Twitter who I am, where you can contact me and what I want to keep tabs on--be it a keyword or user.
That's not easy to do on a large scale. A persistent connection has to be in place between publisher and subscriber. Twitter would have to have a huge number of low-traffic connections open. (Hopefully only one per subscriber, not one per publisher/subscriber combination.) Then, on the server side, they'd have to have a routing system to track who's following what, invert that information, and blast out a message to all followers whenever there was an update. This is all quite feasible, but it's quite different from the classic HTTP model.
It's been done before, though. Remember Push technology [wikipedia.org]? That's what this is. PointCast sent their final news/stock push message [cnet.com] in February 2000. There's more support for "push" in HTML5, incidentally.
If you really wanted to scale this concept, the thing to do would be to rework a large server TCP implementation so that it used a buffer pool shared between connections, rather than allocating buffers for each open connection. The TCP implementation needs to be optimized for a very large number of mostly-idle connections. Then implement an RSS server with slow polling, so that the client makes an RSS query which either returns new data, waits for new data, or times out in a minute or two and returns a brief "no changes" reply. Clients can then just read the RSS feed, and be informed immediately when something changes. A single server should be able to serve a few million Twitter-type users in this mode.
The client side would encode what it was "following" in the URL parameters. The server side needs a fabric between data sources such that changes propagate from sources to front servers quickly, and then on each front server, all the RSS feeds for all the followers for the changed item get an update push.
There's a transient load problem. If you have 50,000,000 users, each following a few hundred random users, load is relatively uniform and it works fine. If you have 50,000,000 people following World Cup scores, each update will force 50,000,000 transactions, all at once. All the clients get a notification that something has changed. So they immediately make a request for details (the picture of someone scoring, for example). All at the same time. However, if you arrange things so that the request for details hits a server different from the one that's doing the notifications, ordinary load-balancing will work.
Re: (Score:2)
It seems like a standardised pub/sub protocol ought to be cacheable, and everyone has an ISP, and ISPs themselves take feeds from networks - so wouldn't it make sense for every local network to have a proxy-like box which subscribes to feeds requested downstream, and therefore reduce the load on upstream boxes?
An open, fully decentralised infrastructure like that would probably come out looking like Usenet, but secure and for micro-transactions. And that seems like it ought to be a much smarter way of doing
Re: (Score:2)
It seems like a standardised pub/sub protocol ought to be cacheable, and everyone has an ISP, and ISPs themselves take feeds from networks - so wouldn't it make sense for every local network to have a proxy-like box which subscribes to feeds requested downstream, and therefore reduce the load on upstream boxes?
Like NNTP [wikipedia.org].
Re: (Score:2)
Sounds a lot like something that might get solved with wider application of XMPP and PubSub...
Re: (Score:2)
Of course o
Re:Are They Employing an Event/Listener Paradigm? (Score:4, Informative)
For messaging architectures (like, say, the internet), the pattern is usually described as "Publish/Subscribe". All serious messaging protocols support it (XMPP, AMQP, etc.) and some are dedicated to it (PubSubHubbub). The basic problem with using it the whole way to the client is that many clients are run in environments where it is impractical to run a server which makes recieving inbound connections difficult.
There are fairly good solutions to that, mostly involving using a proxy for the client somewhere that can run a server which holds messages, and then having the client call the proxy (rather than the message sources) to get all the pending messages together.
Given what's been published about Twitter in the past (including them at one point building their own message queueing system because none of the existing ones that they tried seemed adequate), I don't think what they've done is as simplistic as that on the back-end, though they be forcing third-party apps through an API which makes it seem like that's what is going on (and produces inefficiencies in the process.)
Re: (Score:1)
This allows you to receive tweets in real-time over a persistent HTTP connection.
It's rather well hidden though, perhaps they don't want people finding out about it for whatever reason (performance?).
Re: (Score:2)
Re: (Score:2)
or........ get polled continuously.......... or........... exit the business
affect/effect? (Score:1)
on no! my news is gone.... (Score:1)
It's time to ditch the NoSQL bullshit. (Score:4, Insightful)
It's high time that the so-called "Web 2.0" companies ditch the NoSQL bullshit they've started to put into place. It's not bringing the scalability benefits they all claimed it would, and it's leading to data with very questionable reliability otherwise (not that their data is particularly valuable in the first place...)
A lot of these scalability problems could be solved by using a proper RDBMS on proper hardware that's designed to handle huge concurrent workloads. This level of traffic isn't new by any means. There are many POS systems around the world, from retail operations to airlines, that deal with a similar level of "traffic".
It doesn't matter if they go with a database and hardware stack from Oracle, or a DB2 and hardware stack from IBM, or even use Sybase's ASE on hardware from HP. They just need to invest in some real hardware and some real database systems that are meant for dealing with absolutely huge loads.
Ditch NoSQL databases. Ditch shitty servers. Start using real software, and start using real hardware. That's what other businesses do when they "grow up". If twitter is a viable business, it's time for them to grow up, too.
Re: (Score:2)
okay just what Point of Sale System handles as many transactions a day as Twitter?
I doubt that even WalMart pushes every POS transaction to it's central database in realtime. Frankly it would be stupid to design the system that way. You could have a ore many stores all go down if their was a cable cut.
Odds are that WalMart has servers in each store that push data to a central server every x amount of time.
Also lets be honest Walmarts transactions are each far more valuable than twitters.
I would think that 1
Re: (Score:1, Insightful)
International fast food chains, national lotteries, telecommunication service providers, and others.
Twitter should really look at how telecommunications billing is done. It's realtime, it's at a much greater volume than twitter handles, and they sure as hell don't bother with NoSQL "technologies".
Re: (Score:2)
Fast food chains use a server in each store and then bundle the data to be transmitted. So not really.
National lotteries? maybe but I know that my state lottery also bundles data because they stop selling tickets about an hour before the drawing.
telecomms maybe but then the data value is much higher than twitters.
Re: (Score:2)
Those examples wouldn't even come close to handling the kind of traffic that Twitter does.
Re: (Score:3, Interesting)
Lowes hardware does - there is a local server in the store that serves as a caching server only if the main trunk fails.
Re: (Score:2)
Do you really think Lowes does that many transactions?
Re: (Score:2)
well how about a better one..
Amazon
Re:It's time to ditch the NoSQL bullshit. (Score:5, Informative)
Debit card processing systems require real-time access to the full network for every single transaction. PIN numbers cannot be cached locally, and must be validated before completing the transaction.
Re: (Score:2)
Debit card processing systems require real-time access to the full network for every single transaction. PIN numbers cannot be cached locally, and must be validated before completing the transaction.
Actually they do not. As evidence for that, I cite the fact that I can purchase train tickets from the conductor while the train is in a tunnel where there is definitely no signal. In the absence of magic which lets radio waves penetrate a hundred metres or more of sandstone, what must be happening is that some sort of validation is occurring locally (though presumably it's formally unsafe) and then later on it is being reconciled with the bank. The train company takes the risk that the transaction will fai
Re: (Score:3, Interesting)
I've worked in a big telecom with 40mi+ clients and I've seen an 8 nodes Oracle RAC responsible for the whole pre-paid client database handle far, far more transactions and queries than Twitter says it does.
Each regional server responsible for authorizing the calls has a 2 node Oracle RAC and it too handles far more transactions and queries than Twitter.
So, there you go... The excuse to use NoSQL was that it is quicker in some cases. It's not, time to move back t
Re: (Score:3, Insightful)
It is all about bang for the buck. I do not think that anyone has ever said that you can not scale a SQL server to handle a Twitter like load. The question is one of cost.
I am sure that you could handle the load with DB2 on a z Machine also but at what cost?
I am actually a big fan of SQL and find NoSQL to extremely cumbersome.
But then I really do not have a need to scale that big.
I just am not yet willing to write off NoSQL yet. I know that Google has used it for some things.
But when you are talking about T
Re: (Score:2)
That's a big part of it. Twitter is based on large numbers of very low value transactions. People often point to telco billing and airline reservation systems, but they simply aren't comparable. The first difference is in value, just as you say - a telco or an airline can afford mainframes or clusters of high-end servers; Twitter can't.
The second difference, however
Re: (Score:2)
True I had forgotten all about the query cost. I bet that it still may be possible to scale an SQL system to twitter volumes but that costs would be huge.
Each users update is in effect a query. Some of those could be very ugly when dealing with hash tags.
Yep your right just trying to think of all those query's and transactions makes my head hurt.
Re: (Score:2)
Sorry, no, you're just wrong.
I've built telco billing systems. I've built social networking systems. There are no special scaling issues in billing systems, just lots of data. Social networking, however, carries with it some seriously intractable scaling problems, and Twitter's model is probably the worst of all.
You can't fix this with RDBMSes. Been there. Migrated off that as fast as possible.
Re: (Score:1)
There's strict regulations for this kind of system:
- once the costumer charges credit for the phone, it must be available for use in less then 3 minutes;
- when a costumer makes a call (or sends an sms, mms, uses the data network etc), the amount of credit available must be validated in less than half a second, no calls should be dropped because of inability of the syste
Re: (Score:2)
Walmart does indeed have servers at each store. They cache a lot of stuff such as inventory, pricing, etc., and it's batch communicated on at least a daily basis.
Of course, payments must be realtime.
Re: (Score:3)
Start using real software, and start using real hardware. That's what other businesses do when they "grow up". If twitter is a viable business, it's time for them to grow up, too.
Well that's the problem right there. Many of these 'businesses' (Twitter, Foursquare, Gowalla, etc.) are not viable businesses. They are still bleeding their initial (or secondary, or tertiary) funding rounds dry with expenditures greatly outpacing their receivables (if any). Any hope of being around in a few years is if someone comes along and buys them out or dumps even more $$$$ into their black holes. Or get into advertising - where Google is the million tonne gorilla that you have to compete with.
Re: (Score:2)
Ahahahahaha!! Nice one.
Wait, you were serious?
Sorry, no, loads like Twitter's are absolute poison to RDBMSes. It doesn't matter how much money you pour down that rathole, it's not going to deliver. You can sustain the write load, that's not so much of an issue - less than 100 million records a day at the moment - but the read load is completely impossible.
The solution is left as an exercise for the reader.
Re: (Score:2)
Yeah that Google company hasn't been able to make it work at all. I mean I'm constantly having to wait _microseconds_ for my search results. That's just unacceptable.
Re: (Score:2)
Disclaimer: If storing data, i am using RDBMS only. I learned to DB2 on OS/400 and strong relational models, so that will always be where my heart is.
Twitter does not _need_ true RDBMS features. True integrity? Why? Strong relational ties? Why? etc pp.
They need to scale; massively at that. Sure, investing millions into a proper system and again as much for the backup system and then starting to invest more to spread geographically is nice for an established company with deep pockets.
Sure, you might hit maxi
Protocol overhead (Score:4, Interesting)
Is it that they are doing too much processing on the data, wasting too much bandwidth or is their database causing trouble? Since its twitter obviously any bandwidth used is a waste, but you know what I mean
Re: (Score:1)
I use XML over HTTP all the time. How is it hilarious?
Google as alternative? (Score:1)
Correct me if I am wrong.
this explains it (Score:2)
yep this explains why i can no longer use bitlbee or twirssi to follow twitter timeline
Twitter has jumped the shark (Score:3)
The limit is back up to 350 (Score:2)
AFAICT the limit is now back up to 350/hour, and has been for a day at least. This is in the UK, in case it's turned regional.
It was up at 350? (Score:2)
I used to use Qwit [google.com] before I found Pino [appspot.com], and in its status bar it showed you how many requests you had remaining. I only ever remember seeing it show ~150 at the most (presumably it used a load up in its initial loading, then I missed the rest) and I don't think I ever got below 100, even when jumping back through all of the pages of results I'd missed.
Seriously, what are you doing that needs to be updated so frequently and urgently that you're needing the equivalent of a refresh per second?!? Even if you've
Perhaps it's a monetary issue... (Score:3)
Does Twitter make money? I'm not trolling, I'm serious. A quick search yields this article:
http://www.pcworld.com/businesscenter/article/200635/twitter_to_promote_marketers_special_offers.html [pcworld.com]
Even the author of my linked article has doubts. If I wasn't making money, I'd try to limit my expenditures (bandwidth costs, etc.) too. It's not surprising to me.
So how do they make money?
genius web2.0 marketyng idea (Score:2, Insightful)
Twitter is a fundamentally stupid idea. It is like trying to run all of the mailing lists in the world from one server (and by 'like' I mean exactly the same) The end result is half as useful and twice as shitty. Seriously, write a web2.0 listserv interface and you will amaze tweeters. You can tweet with email holy cow!
Yes, it's a mailing list, suprise!