Disastrous 'Pokemon Go' Event Leads To Mass Refunds (techcrunch.com) 196
thegarbz writes: A Pokemon Go Fest hosted in Chicago and attended by between 15-20,000 people has ended in disaster. The event was plagued by logistical issues resulting in 3+ hour long delays getting into Chicago's Grant Park... Those people who were lucky enough to get into the paid event were greeted with a completely overloaded cell network unable to cope with the number of people trying to get online at the same time. The occasional person who was able to connect experienced a never ending string of game-breaking bugs when attempting to catch the rare Pokemon created specifically for this event.
Gaming company Niantic finally just gave a rare Pokemon Go character to everyone who attended -- though one attendee still called it a "horrible, terrible day." The Kansas City Star reported some people had paid as much as $400 for their tickets -- which had sold out within minutes -- and that some attendees had even started lining up for the event at 6 a.m.
Gaming company Niantic finally just gave a rare Pokemon Go character to everyone who attended -- though one attendee still called it a "horrible, terrible day." The Kansas City Star reported some people had paid as much as $400 for their tickets -- which had sold out within minutes -- and that some attendees had even started lining up for the event at 6 a.m.
Capacity planning (Score:5, Insightful)
I wouldn't ever go to a Pokemon Go event, but if I did, I'd be pissed too.
Re:Capacity planning (Score:5, Insightful)
Yes, but they are a software company, not an ISP / cell carrier. They obviously did not consider the fact that the existing cellular networks could not handle that many customers in one small area accessing high bandwidth resources at the same time. Had they alerted the carriers, the carriers could have set up temporary mobile cells for the event, and / or Niantic could have set up their own free Wifi hotspots for the attendees.
Either way, this was a very, very expensive way for them to learn about these kinds of connectivity logistics.
Re:Capacity planning (Score:5, Insightful)
Are software companies somehow immune from hiring competent event planners?
Re: (Score:2)
Are software companies somehow immune from hiring competent event planners?
Who needs to hire experienced and competent planners? The cell companies let anyone use their bandwidth and as everyone has a cell phone already, why should we need to pay for someone to tell us that our customers can get on line for free?????
Re:Capacity planning (Score:4, Insightful)
An experienced event planner would likely have made the situation worse. At a typical event, maybe 10-20% are on their cell at any time, mostly using low bandwidth voice and text. At this event, maybe 80-90% were trying to use the network, and at much higher bandwidths. The normal heuristics would not have worked.
Re: (Score:3, Insightful)
An experienced event planner would have known that.
Re: (Score:2)
I would think an experienced *anyone* who has gone to a high-volume event (sports/entertainment/etc) would have known.
Re: (Score:3)
LAN parties? Hackatons?
Re: Capacity planning (Score:4, Insightful)
No to both. A LAN party may have a bunch of gamers who have cell phones, but the focus is on the LAN. If the LAN is only part of the action- if teams are playing online- then you can be sure the planner will at least have a high speed connection available. Even a hackathon will focus more on local data transfer.
This was truly unprecedented.
And also completely, 100% percent, predictable.
Again, in Pokemon Go, literally all you do is play on their server, through a cell network or a wifi.
You can supply a fucking wifi. You can supply enough wifi. If this is your goal, you NEED to.
Re: (Score:2)
I just gave two examples of events where bandwidth saturation is an issue, of course they use different kinds of networks.
My job partially covers technical support for events organized by Unnamed Big Corporation, usually I don't even have to raise the issue of bandwidth, it's the event owner that already knows.
I think here we just got a disastrous combination of incompetence and greed, incompetence in predicting an obvious technical challenge, and greed in not limiting (paid) attendance once the issues were
Re:Capacity planning (Score:5, Insightful)
An experienced event planner would likely have made the situation worse. At a typical event, maybe 10-20% are on their cell at any time, mostly using low bandwidth voice and text.
What is this 2008? Anyone involved with any sort of event in the last 5 years knows that 99% of the crowd will be on Snapchat, FB, Twitter, Whatsapp, WeChat, Telegram, Viber, Maps, Uber etc It's all data these days, has been for years.
Re: (Score:2)
Stadiums, for example, have already fought, lost, and retooled for this war. If you didn't investigate what stadiums did for the Super Bowl the last few years, then you're a failure as a planner for an event like this.
Re: (Score:3)
An experienced event planner would likely have made the situation worse. At a typical event,
An experienced event planner would have seen atypical events, that's what makes experience valuable.
An intelligent person would have asked cellphone companies to deploy temporary cells for the event, or even had some partner come in and blanket the park with free Wi-Fi. It could have been an opportunity. Instead, it was a disaster.
Re:Capacity planning (Score:4, Interesting)
The cell companies let anyone use their bandwidth and as everyone has a cell phone already, why should we need to pay for someone to tell us that our customers can get on line for free?????
..you only need to tell the cell companies. hell, you can even get them to sponsor and give _you_ strbaight up money.
what you need to do is just tell them that on this day blabla at this place blabala there will be 30 000 people and they will cart some cell-in-box vans over. thats what they do for festivals etc all the time.
what was more than likely was that the event was handled by some social media presentative/pr side of niantic who of course don't understand anything about technology
Re: (Score:2)
No, PR people know to get planners in. PR people know that they need to set everything "up right" so they don't make a mess of an event (which would be bad PR).
However, it's possible it was handled by a "social media rep" who thinks highly of social media and a disdain for "traditions". So they may think they're immune to the whole "planning" t
Re: (Score:3)
They hired the guys who planned the Fyre Festival.
Re: (Score:2, Insightful)
Are software companies somehow immune from hiring competent event planners?
Competent in what? Precisely what kind of event has taken place before where 20000 people simultaneously need to use data from the same area. It's easy from the IT sidelines to see the obvious, but given an event with these requirements hasn't happened before, just how were a bunch of "competent event planners" (I hear that's a uni degree now) supposed to know the what they don't know without precedence?
Re:Capacity planning (Score:5, Insightful)
Competent in...planning events?
Events need to have things. Bathrooms, for instance: Someone needs to figure out how many porta johns to rent. Figuring out the quantity of things like is the job of an event planner.
This event needed bandwidth and none was provided. That's a pretty gnarly failure, and being a "software company" instead of some other type of company does not excuse this guffaw.
tl;dr, this is like organizing an Oktoberfest event and forgetting to bring beer.
Re: (Score:3)
tl;dr, this is like organizing an Oktoberfest event and forgetting to bring beer.
Nope. It's like organising Oktoberfest in a brewery and assuming the brewery will have beer.
All the examples you gave are "standard" event management items. There's nothing "standard" about an event that is expected to have 20000 with active mobile connections. It's not something taught outside of IT, it's not something that is considered for any other event, it's not something that has been experienced in the past.
It's easy to criticize the things you know. Not so much the things you don't.
Re: (Score:2)
But these things are done.
49% of the attendees at the last Superbowl used the in-house Wifi, with over 27,000 connected simultaneously at one point.
It's not unprecedented, and it certainly was not unexpected.
Re: (Score:1)
Re:Capacity planning (Score:5, Insightful)
Competent in what? Precisely what kind of event has taken place before where 20000 people simultaneously need to use data from the same area. It's easy from the IT sidelines to see the obvious, but given an event with these requirements hasn't happened before
Ok just stop. Have you heard of the Olympics, or any of the thousands of sporting or music events that happen every week somewhere in the world that have more than 20000 people all snapchatting, facebooking, twittering etc? I was doing this in 2010 and 3G (at the time) was on the same list as power, water, toilets and transport. This is a massive fail for a company that's primary product relies on a cellular network to function.
Re: (Score:2)
Ok just stop. Have you heard of the Olympics
Yes I have. Last I heard none of the sports required an active internet connection and none of the spectators could not watch the sports without those internet connections.
I also remember that internet didn't work well, we had stories about it, but in general there were no major complaints from people because there were also no expectations, and certainly no refunds because someone couldn't tweet something.
This is a massive fail for a company that's primary product relies on a cellular network to function.
Agreed. What I disagree with is that it was "obvious" to anyone who is an event planner.
Re: (Score:1)
Well, perhaps we are in agreement then, that not everyone practicing a trade is competent nor should they sell their services to others.
Re: (Score:2)
What I disagree with is that it was "obvious" to anyone who is an event planner.
Based on the numerous comments in here it seems obvious. But I'm not sure what the qualification is for event planning, is it to not know anything about planning events?
Re:Capacity planning (Score:5, Informative)
Practically every event ever nowadays. A stadium can hold 20,000+ people easily, and people love to tweet and snapchat and facebook and everything else. And there will be at least 10% of the people who are streaming video and audio commentary during the game, while everything else is bursty.
In fact, we had several festivals already. Not big ones, but since it was aimed at the millennial crowd, there were CoWs (Cells on Wheels) set up around the perimeter. These CoWs are miniature cell towers meant to cover a small area that's likely to have a high density of cellphones. They typically backhaul onto either a dedicated link (if one was put in during construction - so sites like stadiums and arenas and parks will have them) or via a microwave link to a regular cell tower.
All it takes is an event planner calling it in and the big carriers will truck in a CoW with advance notice and set it up. You tell them how many people are there and they'll estimate the number of units and type to bring in (they may bring in more than one and backhaul onto the one with a dedicated link).
This is not new - it's been around a few years and you'll often see them at stadiums until they set up local repeaters inside.
Re: (Score:2)
and people love to tweet
Yes because being able to tweet is in exactly the same league as the primary event being unable to take place. Last I remembered when I had no coverage at a sport event I just kept on doing what I paid for which was watching the sport.
Re: (Score:2)
Yes but it's completely transparent to the user. A phone that's been in use will have an up to date database, but that just means if you're searching for a cell signal, an updated database means you can acquire it really fast and have service in a few seconds.
If you haven't powered your phone up recently, it may have an outdated database that should still work, so it'll try to use it.
If the database doesn't work at all, then it wi
Re: (Score:1)
Yes, as far as I can tell.
Re: Capacity planning (Score:2, Insightful)
Who cares what the cell companies do?
Niantic should have set up a bunch of professional grade wifi networks, all tied to a fast fuckjng backbone. This was their event, and their event required a shit ton of network connections. Fix it with temporary wifi. It costs a shit lot less than like a hundred bucks a head to set that up.
$100 in game cash does not = $100 real cash (Score:2)
$100 in game cash does not = $100 real cash
Re: (Score:2)
You, OTOH, just no. Stop
Re: (Score:3)
Poor network coverage is a very common theme at large scale events. As an attendee, this is something that I expect and I plan for it by having everything I need available offline and planning meetups the old fashioned way.
Anyone who have already planned for large scale events should know that too and tackle the problem. This is simple incompetence, either from Niantic or from the event planners they hired.
Re: (Score:2)
Sure, but since their software is utterly dependent on connectivity, you might want them to know at least enough to realize they needed to consult with someone for a big event.
Re: (Score:2)
Yes, but they are a software company, not an ISP / cell carrier. They obviously did not consider the fact that the existing cellular networks could not handle that many customers in one small area accessing high bandwidth resources at the same time.
They are clearly stupid then. I've been involved with similar sorts of events and the first items on the list for any sort of event is power, water, toilets and telecoms.
This company relies on cellular networks for its business and not one person thought to check on capacity requirements?
a ticket Lottery does a lot to stop scalpers /fair (Score:2)
a ticket Lottery does a lot to stop scalpers / make it fair to over over meany different time zones.
Re: (Score:2)
And no backhaul.
Yes, this is a thing if the carrier has spare frequency spectrum in that area (They usually don't), a microwave point they can relay to (Probably not, unless pre-planned) or a fibre point that they can connect that cell up to the rest of the cell network (Only for multi-use sites).
If it's in the middle of the country side where they don't have a lot of frequency re-use, then sure, it's one problem less. If it's a frequently used site (At least once a year) then they'll possibly even run fibr
Re: (Score:2)
The community organises the events, Niantic gives a liason.
Re: (Score:3)
Do EA or Activision care when they release new games and people can't connect to play? They just feed you some BS and wait it out until people go play the next big thing
Re: (Score:2, Interesting)
The problem is us. We continue to pay for this shit. Companies release games that don't work and people continue to pre-order the things. If people would stop paying for these things, then they'd have to care about whether or not they work.
But as long as people are willing to pay for games that don't work, companies will happily continue to provide games that don't work at premium prices.
Re:Capacity planning (Score:5, Interesting)
In my humble opinion, they are a bunch of shits.
I tried to help some nuns by getting Niantic to delist some Pokemon Go places on the convent's private grounds (at the request of the nuns).
There was construction going on at the time. Despite uploading photos of the construction, showing that the area was dangerous, despite uploading photos of signs and fencing indicating that the area was private, Niantic would not remove the stops from the convent grounds.
Anyone who gets hurt playing Pokemon Go should sue Niantic, because there is a likelihood that Niantic has already been told that the area is dangerous.
Re: (Score:3)
I think they're just overwhelmed (Score:2)
Wow... (Score:2)
Get a demand letter from a lawyer. If it persists, go to court. They will settle pretty much immediately unless they are idiots--both because of the risk of bad press and because of the cost. It's just easier to delist the places.
(Lawsuits are a bad way to resolve things except for when they're not.)
Re: (Score:2)
Perfectly normal warning, and anyone who goes against it is a stupid shit that shouldn't be playing in the first place. Take responsibility for your own actions.
Re: (Score:2)
> That would be more acceptable if Niantic didn't place rare rewards in dangerous places
They really don't. Not in Pokemon Go at least, I don't know about Ingress.
In Pokemon Go, the world is filled with pokemon spawn points, the majority of which are totally reachable from a road or by walking through a park. Rare pokemon come up rarely.
A decent number of players do use a third party map (like on a website) that is built by using thousands of fake accounts which exist only to report the positions of mon
Re: (Score:3)
I'm not sure how you derive that. I certainly can't say for all, but over at the gated parks in my area, which closes at night, y
Re: (Score:2)
they'd learn a thing or two about capacity planning
What capacity? The launch issues were due to server capacity. These issues were not really within control of the company and had everything to do with the mobile network falling over.
Though they could have co-ordinates with Verizon, AT&T and T-Mobile to try and get a shitload of microcells set up. But in general what prior experience was there? When has an event with 20000 people who specifically require mobile data coverage at the same time ever been done before?
Re: (Score:2)
Besides sports events? Maybe the upcoming eclipse, reported here earlier. https://science.slashdot.org/story/17/06/27/202250/august-solar-eclipse-could-disrupt-roads-and-cellular-networks [slashdot.org]
Re: (Score:2)
When has an event with 20000 people who specifically require mobile data coverage at the same time ever been done before?
Not so sure about "specifically require mobile data", but Chicago is very experienced at large events, The Chigacgo Ribfest [festivalnet.com], with 50,000 attendance happens every June, also in Grant Park. The Air and Water show [wikipedia.org] (1,000,000 attendance per day) happens every August, about a mile north. It is pretty common to see multiple Mobile Cell Towers [wikipedia.org] at these and other large events in Chicago. I'm confident that if all the air and water COWs were brought to Grant Park, there would be more than enough coverage.
I'm g
Re: (Score:2)
Hell don't forget the Taste of Chicago every July, that draws in almost 2 million over 5 days. There's absolutely 20K people there on the network at the same time. Or the 225K who went to the nfl draft. THis was an event that needs planning, but Chicago puts on 2 orders of magnitude bigger events multiple times a year.
Re: (Score:2)
While I certainly agree with you, there's not much they can do about cellular capacity. I had the same issue (in the Chicago area, ironically enough) at a music festival a few weekends ago. It's depressing to see full bars and no throughput
I mean I guess they could have probably tried to arrange to get the carriers to install some picocells for the event, but given the number of carriers, that's a logistical nightmare.
Personally, I would have tried to get a temporary internet circuit out there, and hit up o
Re: (Score:2)
Re: (Score:2)
Using FCC assigned frequency at burning man probably won't get you into trouble (Unless someone complains), using it in the middle of chicago, probably will.
Re: (Score:2)
A Pokémon Go festival may seem trivial, but this is the kind of stress test that augmented reality needs if we're about to start using it in everyday life. What's going to happen in another summer or two when thousands of tourists pouring into Chicago fire up this fall's new AR extensions to iOS to find bathrooms and ATMs and restaurants?
Re: (Score:2)
Re: Why get together for an online game? (Score:1)
This is a GPS oriented online game.
Wow (Score:4, Insightful)
Wow, I cant believe people still play Pokemon Go.
Re: (Score:2)
Why? The game has been continuously developed to the point where you now need to form teams and have group battles. They keep making changes to the gameplay in an attempt to keep it fresh, and judging by the fact that pretty much every event is a sellout in minutes it's working quite well.
I can't believe people still play Counter Strike.
Re: (Score:3, Insightful)
Because it was the ultimate fad game. When it first came out I'd see countless people walking around outside waving their phones around. A dozen or so restuarants in my area (and i live in the burbs) even gave discounts to various factions as a means of bringing in business. Fast forward to today and my teenage neice and nephew dont have a single friend who still plays and literally every person who i knew who played no longer does.
Re: (Score:3)
When it first came out I'd see countless people walking around outside waving their phones around.
Of course. The initial start was an incredible fad. Now it just settled back down to the usual fan base who play the most popular current game in a 20 year old franchise. It has a healthy user base just like any other popular game on the market you don't see people play.
Fast forward to today and my teenage neice and nephew dont have a single friend who still plays and literally every person who i knew who played no longer does.
Congratulations. I also don't know anyone who plays WoW so that must not be a thing anymore either right?
You should take public transport sometime. I see people literally everywhere, even when I was in the USA 3 weeks ago, so it's not just a
Re: (Score:2)
You "see people literally everywhere"? Good for you, so do I.
Re: (Score:1)
Re: (Score:1)
I'll think about the countless lives I've lived, and worlds I've seen.
Re: (Score:2)
At the end of your life you can think about all the useful stuff you've done in your life.
I'll think about the countless lives I've lived, and worlds I've seen.
Wow are you implying that humans are capable of only doing one thing and it's an either or choice? Man your brain is limited.
There is another kind? (Score:3)
Sigh. (Score:1, Troll)
Moreso than the organisation of an event (which is hard), I am much more disappointed in tens of thousands of people turning up to chase virtual characters around a park on their phones AND PAYING FOR THE PRIVILEGE.
Honestly, that's much more in the "what the fuck has the world come to" area than someone who couldn't organise a piss-up in a brewery.
Re:Sigh. (Score:5, Insightful)
Must be pretty easy to see them all from your high horse.
How is this event any different from, say, Comic-Con or any other fan fest? Just because it's a video game you don't happen to like? Newsflash: People like enjoying their hobbies and sometimes that involves paying money to do things related to it.
Re:Sigh. (Score:4, Insightful)
Comic-Con or any other fan fest
Or professional sporting event, theater, concert, etc.
Re:Sigh. (Score:4, Funny)
Must be pretty easy to see them all from your high horse.
How is this event any different from, say, Comic-Con or any other fan fest?
Nothing, they are all equally stupid. Now running around a field with a ball, that is the real deal...
Re: (Score:1)
"Guys? Didn't you hear me? Stop having fun!"
Re: (Score:1)
A bunch of grown men, women, and all of the other genders dressing up as cartoon characters or aliens is pretty pathetic, too. This goes beyond just being a hobby. It many cases such behavior appears to be a symptom of an underlying mental illness, for example. Such people should seek medical assistance.
Yep. Exactly the same goes for baseball, football and any other team sport where people dress up and do stupid things together for no good reason.
Assuming you are consistent.
Which you are not, of course.
TL;DR: You are a fucking hypocrite.
Re: (Score:3)
As opposed to what?
Thousands of people turning up to watch a group kick a ball around?
Thousands of people going to a shopping centre on sale day?
Frankly I'm more disappointed in the 100s of thousands of people who are vegetating in front of the TV or "disappointed" in their fellow people Slashdot.
Re: (Score:1)
Re: (Score:1)
Re: (Score:2)
They don't vote because they no longer believe in democracy [journalofdemocracy.org]. Especially millennials.
Re: (Score:2)
But when it is time to vote.... screw that, let's stay home.
Are you surprised? Why not describe the two events:
a) a group of devote fans have the opportunity to participate in a rare event with like minded people to do something in their hobby that is effectively limited to this one opportunity (there was apparently all sorts of special things you get at this event). On a weekend no less.
vs.
b) a group of people who don't give a shit about politics get to cast a vote about which turd smells the least shitty with the wonderful promise that regardless of what they do,
Meaning of "disaster" (Score:5, Insightful)
I'm very disappointed in this story. A flaming airship crashing into a crowd of 10,000 people is a disaster, not the failure to catch a rare pokemon.
To be honest, I was hoping for the former.
Re: (Score:2)
A flaming airship crashing into 10,000 people who have paid to play Pokemon Go.
I would pay to see that.
Re: (Score:2)
not the failure to catch a rare pokemon.
If that's the "disaster" you got out of this story then you need reading comprehension skills. How about a $400000 event ending in not a profit, but rather a $200000 additional loss as everyone needed to be refunded and also compensated?
Re: (Score:3)
Still not a disaster. Nobody died. Company won't go out of business because of this. And even if it did the company makes a game.
Re: (Score:3)
Thanks. I'm glad we got your clear definition of disaster meaning someone dies or an entire company goes out of business.
Every other dictionary and definition in the world didn't make it clearly as black and white as you did. We have you to thank for clearing up the entire english language. I am in awe just to even be speaking to you.
Re: (Score:1)
As a systemd evangelist, I'd have expected you to have a much better idea of what really constitutes a disaster.
Re: (Score:2)
As a systemd evangelist, I'd have expected you to have a much better idea of what really constitutes a disaster.
Interesting. You don't know how to accurately define disaster ... or the meaning of evangelist. Maybe English isn't for you. Have you tried a less complicated language with shorter words?
Re: (Score:2)
Maybe some pokemons lost life?
Re: (Score:2)
Or maybe an accidental oversight caused hundreds of thousands of dollars in losses for Niantic.
I can think of far lesser things that qualify for the title of disaster.
Re: (Score:2)
Re: (Score:2)
Here's a photo from yesterday's Pokemon Go Fest in Chicago. I don't see any Japanese people.
https://cdn.vox-cdn.com/thumbo... [vox-cdn.com]
Organized by Ja Rule? (Score:2)
Seriously though, this many people in one location, can the cell towers handle that, as well as any kind of cross interference?
Dead? (Score:2)
Not bad for a supposedly dead game.
Should we be surprised? (Score:4)
It is a cell phone based game. I play in the suburbs which has reasonable coverage, but I have period network issues with the game. There cannot be that many people in my area that are impacting the network so it has to be the game.
Add in REAL network congestion, of course this is going to be disaster. The part I don't believe is that people actually PAID to go to this!
Did they hire Ja Rule [wikipedia.org] to run this or what?
Re: (Score:2)
This week it's definitely the game / server load.
Normally though, I don't hear about much of a problem (I don't play but everyone else in this house does).
Duh? (Score:1)
Clearly none of these people have ever tried to use their cell phones at an outdoor festival before. You can't do networking events at public parks because you can't add your own wireless infrastructure. I'm surprised that this didn't occur to anybody when planning this.
Re: (Score:3)
You can't do networking events at public parks because you can't add your own wireless infrastructure.
Rental of portable wifi access points specifically designed for events are available from half a dozen different vendors I found with a very cursory Google search. You most definitely can add your own wireless infrastructure, and I bet deployment happens in a matter of hours, just like the cellular tower trucks mentioned by the other poster. Might even be cheaper, given the phone company's penchant for overcharging for every byte.
Re: (Score:2)
And your uplink to the internet is going to be what exactly?
I'm assuming they use point to point wireless for backhaul, which runs at up to 866 Mbits/s, commercial off the shelf. That plus a brick of batteries in the base allows 100% unwired deployment, which is faster, safer, cheaper, and easier than running wires for either power or data. Neither wifi access points nor the point to point wireless data systems are high power devices, so batteries are perfectly acceptable.
If I were designing such a station, it would have a cylindrical steel base with a locking acc
These people need almond flavored Kool-Aid (Score:1)
Maybe we should host a Pokemon Go event near the edge of the Grand Canyon. Place a bunch of legendary birds just in arms reach over the edge and let Darwinism continue the thinning of the Go cult.
Really? (Score:2)
Stupid companies with stupid products attract stupid people and act stupidly.
How is that news for nerds? Or even stuff that matters?
Cell Network???? (Score:2)
Those people who were lucky enough to get into the paid event were greeted with a completely overloaded cell network
They were hoping to use the existing CELL network to handle this mass of people/smartphones??? What idiots. I can't believe they didn't arrange for some local wifi hotspots to be setup for an event like this.
Re: (Score:2)
And nothing of value was gained.
Re: (Score:2)
attended by between 15-20,000 people
Niantic gave a rare Pokemon Go character to everyone who attended
Doesn't sound very rare to me.