Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!


Forgot your password?
Cellphones Privacy

Cell Phones That Learn the Sounds of Your Life 121

An anonymous reader writes "Researchers at Dartmouth College have developed new software that uses the microphone on the iPhone to track and interpret a user's everyday activities using sound. The software, called SoundSense, picks up sounds and tries to classify them into certain categories. SoundSense can recognize completely unfamiliar sounds and runs entirely on the phone. It automatically classifies sounds as 'voice,' 'music,' or 'ambient noise.' If a sound is repeated often enough or for long enough, SoundSense gives it a high 'sound rank' and asks the user to confirm that it is a significant sound and offers the option to label the sound. In testing, the SoundSense software was able to correctly determine when the user was in a particular coffee shop, walking outside, brushing her teeth, cycling, and driving in the car. It also picked up the noise of an ATM and a fan in a particular room. The results [PDF] of the experiments were recently presented at the MobiSys 2009 conference."
This discussion has been archived. No new comments can be posted.

Cell Phones That Learn the Sounds of Your Life

Comments Filter:
  • Privacy (Score:5, Funny)

    by sopssa ( 1498795 ) * <sopssa@email.com> on Friday July 10, 2009 @04:13PM (#28654361) Journal

    For those of us who dont want to pay for it, dont worry! Next year the goverment will install it to your phone for free!

    • well if you live in the UK at least...
    • No kidding. Yet another way to track people's whereabouts. What is this, a way to close the gap when there's no GPS reception? Are we going to have to take the battery out of our phones when we're not actively using them? Perhaps install a hardware switch on the phone to disconnect the microphone? WTF?
    • Re: (Score:3, Interesting)

      by timeOday ( 582209 )
      But in contrast to previous technologies, at least this runs standalone on the device, rather than as a web service.

      Also, it doesn't record everything verbatim, but rather just tries to find characteristics of different environments, and classify them. (I don't have direct knowledge of this, but it would be very resource-intensive and pointless to record all the ambient noise used to recognize you're in the office, for example).

      I think it is a sensible idea. Obviously humans use their senses to be awa

      • Re: (Score:3, Insightful)

        by Helios1182 ( 629010 )

        GPS can generally tell you were you are, but only up to a point. Are you in the coffee shop, bathroom, or conference room in that giant building located at 455 N. Example?

        • Re: (Score:3, Informative)

          by sexconker ( 1179573 )

          GPS can tell you where you are within a few inches.

          Your mapping software is what lacks in precision.

          • Uh...I'm not sure that civilian GPS systems are that precise.

            Now, I don't have a way fancy GPS receiver--it's a Garmin Edge 205 [garmin.com]. It has a little read-out for accuracy, though, and in my normal travels, the read-out says 10-15 feet.

            Now, conceptually, I would agree. But between "detuning" and clock accuracy, I don't think you'll ever find accuracy measured in inches on any civilian GPS device.

          • by eltaco ( 1311561 )
            gps only maps 2d. it can't tell you if you're at the bottom or top of a building. (assuming you get a gps signal, you're probably at the top.)
            • Re: (Score:1, Informative)

              by Anonymous Coward

              That's actually entirely incorrect, GPS is able to track Latitude and Longitude as well as Altitude.

              Check the following link for more information:


              • by simaul ( 594771 )
                Here's a good chance for me to be pendantic... GPSs track altitude as "height above ellipsoid" (HAE). The ellipsoid being a mathematically perfect surface that is close to, but not the same as, mean sea level (MSL). MSL varies up and down with respect to HAE based on things like the density of the underlying rocks, the presence of mountains, and trenches. Converting between the two requires consulting a database of points and interpolating. This makes the (MSL) altitude reported by most GPSs not so accura
            • GPS will give you an altitude.

          • Re:Privacy (Score:5, Informative)

            by MartinSchou ( 1360093 ) on Friday July 10, 2009 @07:50PM (#28656261)

            Not only did you get it wrong, but most of the people replying to your post got it wrong as well.

            GPS comes in several varieties. The common one used in cheap handheld units are using the C/A signal [wikipedia.org]. This gives you an accuracy down to about 5 meters (due to multipath and atmospheric interference). The accuracy is determined by the precision of the built in clock which in term determines the size of your unit. C/A gets you down to about 5 meters accuracy.

            Add in DGPS [wikipedia.org] and you more or less eliminate atmospheric interference which can get you down to about half a meter. Technically 20 inches can count as a few inches when compared to the 200 otherwise.

            P-code (the military) gives you down to about 2 meters accuracy by comparison. Not sure how much better they get with DGPS, but I'd suspect it'd get them to 1/10th just like the C/A does.

            For the best accuracy you won't be relying on L1 and L2 directly (decoding the signals), but will be looking at the carrier phase change [wikipedia.org] which requires bigger and better antennas as well as a much more precise clock which is why most of these are big, bulky and used for surveying more than anything.

            If you're moving around (airplanes pictures and very likely road surveying as well) you can get down to about half a centimeter, but expect from 5 mm to 10 cm). When not moving for a significant amount of time, you can get down to 2 mm to about 3 cm)

            As for one of the replies claiming that GPS only gives you a 2D location, this is rubbish. You need a clear view of a minimum of four satellites to get a proper height as well as longitude and latitude. The accuracy of each of these coordinates varies as well . However the biggest inaccuracy you're likely to face when dealing with GPS is using the wrong datum [wikipedia.org] for your map. It's fairly easy to end up with coordinates several hundred meters from the correct one, merely by forgetting to switch datum when moving into a new area.

            If you're not using the US GPS but instead rely on Glonass [wikipedia.org] (not done yet) you get some advantages. Since each satellite is running its own discrete frequency, you're essentially able to rule out atmospheric errors. This adds to the cost of the unit though, as it now needs a much more advanced radio receiver. I can't remember if Galileo [wikipedia.org] uses one or multiple frequencies. I think it's supposed to use two, but it's not that important.

            Granted, it's been a while since I worked heavily with GPS theory, so feel free to correct me (if you can cite properly). I'm using Wikipedia because I doubt most of you are able to read Danish which is the language my text books on the subject.

            • But of course none of this matters if you can't get a lock on multiple satellites, which is case when inside large buildings.

            • So you took a copy and paste trip to Wikipedia?

              Sorry dude, but you're 100% wrong.

              GPS can get you within inches.
              Fractions of an inch if you use multiple, linked receivers.
              Surveyors use it all the fucking time.

              The issue is NOT GPS, but how shitty or how good your GPS receiver is. The P(Y) code is not for military use only, by the way. There are many civilian-class receivers that use it.

              • Using multiple receivers in a phone or regular handheld GPS (which is what the argument is about) works how?

                GPS can get you within inches.

                I put 2 mm as the lower boundary for GPS accuracy with carrier phase change. That's not only within inches, it's 1/12th of an inch.

                If you mean with regular code receivers, I doubt you'll get more accurate than half a meter with those, but you're more than welcome to source that claim which you haven't done out in your post at all. I'm not saying you're wrong, I'm saying "

                • The argument was mine, stating that GPS gets you within inches.

                  You being in a building, you having a shitty receiver, etc. is not the fault of GPS.

          • by SJ2000 ( 1128057 )
            Everyone forgets about the chipset. http://slashdot.org/comments.pl?sid=1297287&cid=28638245 [slashdot.org]
          • Find me a GPS that gives that level of precision inside a shopping mall. GPS needs to see the sky, large buildings have this inconvenient thing called a roof.

      • Re: (Score:3, Insightful)

        by PopeRatzo ( 965947 ) *

        the SoundSense software was able to correctly determine when the user was in a particular coffee shop, walking outside, brushing her teeth, cycling, and driving in the car

        at least this runs standalone on the device, rather than as a web service

        This sounds more like proof-of-concept than anything practical...for now. I'm not so worried about the "government" using this technology as I am concerned that it is a marketer's dream.

        It's curious that there is so much concern about the government having some sort

    • The whole government? If so, I'm pretty sure they'd never be able to agree on how to get to my house.

      What useless fearmongering. When cell phones first became pervasive, people had this same kind of hysteria: Oh no, we're all carrying microphones attached to a phone network! The government can work with cell phone companies to eavesdrop on our lives! 1984 was just a decade late!

      After all the repetition of not outlawing technology that has legitimate alternative uses (i.e. p2p software),

      • Yeah, the government could never data mine our e-mail or divert major percentages of our data transactions through the NSA's equipment, and then they could never get the congress to issue blanket immunity to the telcom companies who colluded with the illegal actions ... oh wait, they already did! Some people are sheep with blinders on, I swear. Talk about crazy!

        http://www.eff.org/cases/att [eff.org]

    • Re:Privacy (Score:4, Funny)

      by davester666 ( 731373 ) on Friday July 10, 2009 @05:58PM (#28655405) Journal

      That was the sound of you shooting someone. For a monthly charge of $500 on your cell-phone bill, the phone won't call the police. Accept the charge?

      Yes No

  • Have the phone tell everyone what you're doing at every moment in time. Someone get on that. Then when it happens, we'll have a YRO article about how this could be an invasion of privacy!
    • Re: (Score:3, Interesting)

      by jammindice ( 786569 )
      Actually this sounds like the echo location batman used in the Dark Night at the end except for being self contained on the phone versus using just the mic from all phones and processing centrally
  • by ilblissli ( 1480165 ) on Friday July 10, 2009 @04:18PM (#28654453) Homepage Journal
    so in other words this will effectively let me tweet when i'm pooping without having to type it up on my phone. it may also be handy finding out if people have washed their hands after such an event ;)
    • If you're tweeting when you poop, then I'd say you should go talk to your doctor about a referral to a proctologist.
  • by synthesizerpatel ( 1210598 ) on Friday July 10, 2009 @04:18PM (#28654457)

    Lets be honest people. If a device is capable, someone will write the software to enable it. This shouldn't be surprising or shocking. When 'wearable computers' started getting buzz it was because people were walking around with web-cams attached to their heads seeing everything they could see and slashdot thought it while amazingly geeky, was cool. This isn't that different except there's no soldering required.

    To be honest, we haven't even seen the worst of it yet. Considering the deluge of FPGA and EEPROM powered embedded devices out there you'd best be scared of the things that are _hard_ to reprogram, not the ones with complete IDEs and API documentation available.

    I'm more concerned about someone snooping on me from my Jura Capresso than I am from my cell phone.

    • > To be honest, we haven't even seen the worst of it yet. Considering the deluge of FPGA
      > and EEPROM powered embedded devices out there you'd best be scared of the things that
      > are _hard_ to reprogram, not the ones with complete IDEs and API documentation available.

      The latter are the ones that someone can reprogram remotely without your knowledge.

      • I would posit that the 'hard' to program devices will be the ones that will go unnoticed for longer as there's no introspection or prosumer/developer tools available to identify aberrant behavior. If your DVR stutters occasionally when playing back it could be a bad MPEG-TS packet, or because the packet sniffer stomped the decoding threads priority.. How would you ever know?

  • Like I really my phone to tell me my wife is bitching at me, or the kids are whining...

    • Re:Oh great.... (Score:5, Insightful)

      by CastrTroy ( 595695 ) on Friday July 10, 2009 @04:28PM (#28654549) Homepage
      Actually not a bad idea. A lot of time I would like to wear my iPod while doing house work, but I don't because I can't hear the baby cry. If I had an iPod that could recognize when the baby was crying, and play it over the headphones in place of my music, so that I knew the baby was crying, then I would really appreciate this feature. Same goes for somebody calling out my name. Even if there was a 1 second delay, it would be awesome.
      • by serutan ( 259622 )

        Now That's some creative thinking. I've been trying to figure out a practical use for this technology other than the article's lame example of turning off the ringer when you're in a meeting. Recognizing sounds that might be important -- baby crying, doorbell, other phone, barking dog -- so you can listen to something else. Nice idea.

      • by xaxa ( 988988 )

        Just get some speakers!

      • Until then, put in only one earbud and leave the other open. Low tech solutions FTW.

        • Re: (Score:3, Interesting)

          by orkybash ( 1013349 )
          At least until the hard panning starts driving you slowly insane... some people can wear one ear all day, while I can't last ten minutes like that.
        • Actually, I do this with voice podcasts. It works great. However, with music, it's often the case that a lot of the music is only on the left or right channel, and therefore I miss out on a lot of the music. If only my iPod had a balance setting.
      • For the nokia there is an application that calls a phone number at a certain decibel level. You can then talk to your baby over the phone (speakerphone) while you are getting there. There could be similar programs for other smartphones/iphones

    • Re: (Score:3, Funny)

      by Green Salad ( 705185 )

      Like I really my phone to tell me my wife is bitching at me...

      That's better than telling her you believe she's bitching at you!

      Just point to the phone's labeling, smile and shrug helplessly.

    • by Maniacal ( 12626 )
      Actually, this could work for me. I could have it kick off a macro that plays my ringtone. She starts bitching and "Oh baby, sorry to interrupt. It's work. Probably have a server down. I have to take this."
  • With this amazing new software, your smart phone can notify you when you are in a coffee shop, and when you are at home relaxing. Will wonders never cease?

    I can see some use for the stuff mentioned at the end, where the software would classify periods of time to make them easier to search through, but I don't have any particular desire to have a continuous recording of my life.

  • ob. Clippy (Score:4, Funny)

    by RobertB-DC ( 622190 ) * on Friday July 10, 2009 @04:24PM (#28654513) Homepage Journal

    "It sounds like you're going pee."

    Then, if male: "Please, remember that there are ladies in the house, and put the seat down when you're done."
    Else, if female: "If you're out of toilet paper again, I can call your sister to bring it to you."

  • by Ambitwistor ( 1041236 ) on Friday July 10, 2009 @04:27PM (#28654535)

    From TFA:

    "The SoundSense software was able to correctly determine when the user was [...] brushing her teeth [...] Choudhury says that enabling the software to learn to recognize new sounds will be essential for practical applications. 'A system that can recognize sounds in a person's life can be used to search for others who have the same preferences'"

    That sounds like great functionality. " If you like brushing your teeth, you may be interested to know that 21 out of 23 people on your Contacts list also enjoy brushing their teeth."

    • by jefu ( 53450 )

      It would probably go more like :
      Phone says : I hear that you're brushing your teeth.
      Pause for connection.
      Crelm toothpaste with the miracle ingredient, Fraudulin! Choose Crelm toothpaste!
      To order Crelm toothpaste with the miracle ingredient, Fraudulin, press the "Buy" button now.

  • by al0ha ( 1262684 ) on Friday July 10, 2009 @04:27PM (#28654541) Journal
    could be to anonymously feed thousands of opt-in users current sound input into a public algorithmic service which provides data facilitating the creation of truely random numbers. Similar to the use of atmospheric noise by sites like Random.org; but this could be better because the sound input devices are decentralized and always moving.

    There are the obvious problems to overcome with attempts to game this sort of system, but I think it an interesting idea.
    • by brkello ( 642429 )
      Except most people follow routines so maybe it wouldn't be all that random?
    • Re: (Score:3, Interesting)

      by evanbd ( 210358 )

      Or you could just use the quantum resistor noise from the input impedance of your sound card [digital-scurf.org]. Seriously, people seem to think getting good quality quantum entropy is hard. It isn't. The hard part is turning that entropy into bits: ensuring you have enough entropy to be generating as many bits as you are, and making sure that those bits are whitened and debiased properly, and that the entropy is properly distributed among them so that the non-random stuff in your input stream doesn't leak through. Not to

    • How would this be an easy way to get random numbers? Why not just sample an analog noise source? Admittedly, I wasn't even aware of this apparent need for random numbers and the difficulty of getting them.
  • Privacy concerns? (Score:1, Informative)

    by Anonymous Coward

    FTA: To address privacy concerns, they designed SoundSense so that information is not removed from the device for processing. Additionally, the program itself doesn't store raw audio clips. A user can also tell the software to ignore any sounds deemed off limits.

    I can think of lots of fun and interesting uses for this kind of monitoring. And I can't think of many reasons to limit what sounds my phone can hear. Total capture and processing of the sounds of my life could have remarkable implications which I

  • Now my phone will ask why it's 4:20 all the time ;)

  • OK, so jokes aside, could not this be used to build profiles of areas where you might be trying to hold a coversation and apply noise cancellation activily to the phone? blocking out all the background noise for the person you are talking too, perhaps even making it a bit eaiser for you to hear in the process?
  • by Anonymous Coward
    Finally, an app that can automatically record how many times a day we were rejected by women. It should be called iGiveUp and have an icon of a geek with a gun in his mouth.
    • Most of us don't encounter women at a range close enough to get rejected, you insensitive clod!
  • The application in question categories sounds and thus tries
    to label them. This is typically done using AI algorithms like
    a combo of a neural net and fuzzy logic. That is pretty impressive to have it running on a mobile
    phone of all things.

        I think it's very cool
    that we have come to the point that we basically have a device
    in our pocket that resembles a combination of the stuff mentioned
    in books like the Hitchhiker's Guide and the Ender saga.

  • How long before it learns all of the curse words in your life?
  • I wonder how many of these SoundSense users will end up having the "awooaaaa" sound of a booting Mac get classified as a "significant sound".

    • by n30na ( 1525807 )
      Not as many as those that end up with those weird-ass windows bootup sounds, i'm willing to bet.
  • What would be the point of identifying the sounds? Would the phone do anything useful with the info? And, why would anyone put useless software on a phone? OK, I know most of the software people use is useless, but this sounds unbelievably useless.
    • by Leebert ( 1694 ) *

      What would be the point of identifying the sounds?

      It would be kind of cool if, say, my phone said: "Ahh, Lee is at church! I'll go into silent/vibrate mode. Ahh, Lee is in the bathroom! I'll go into silent/vibrate mode. Ahh, Lee is at a rock concert! I'll go full volume ring and vibrate mode. Ahh, Lee is on the motorcycle! I'll just give up even trying to alert him of a call."

      There are plenty of nifty applications for this. And quite a few nefarious ones, at that. Don't poo poo it just because you can't think of an idea off the top of your head. :

  • And it rings you to ask if it's a significant sound?

    After 30 minutes...I'll get a buzz saying:
      "Do you want me to remember the fart sound I keep picking up? And by the way, you better get some fresh air"

  • After looking at the charts I would have thought that the pink asterisk would have muted out the red dot... Maybe the red dot includes the TV?
  • by MaXintosh ( 159753 ) on Friday July 10, 2009 @04:52PM (#28654805)
    ... though I suppose most of the Slashdotters would say "Someone really is out to get me!" Maybe. Let's ignore that for a moment, and address a question that arose multiple times in the comments. What possible use was this developed for?

    Research. That's right. Research. Actually finding out what people are doing/eating/etc. is actually really hard. People's recollection is full of holes. Think about it: What were you doing exactly 1 hour ago? Most people will be able to name the task, but not what specific activity in that task they were doing. Where you talking? Where you typing. Where you taking a two minute mental break and staring off into space? Yeah. It's hard to recall these inane details. But things like how many people you have contact with, how often you're in public-public (and not in an office, etc.) are hard to get at. How many times have I been to the café down the street in the last week? Hell if I know. But if I volunteer for a study - let's say, time spent in a restaurant by number of colds people get (totally made up on the spot for the purpose of illustration), this way the researcher doesn't have to trust a potentially erroneous recollection... they can get another, good estimate. It's made to be opt-in, and for research of this, that or the other thing.

    Is there a potential for abuse? Sure. There's lots of things with abuse potential. But I hate to break it to you, but they tracked Pablo Escobar in 93 from his phone. And it's only got easier since. If you want perfect privacy, don't get a cellphone.
    • by Renraku ( 518261 )

      Perfect privacy is impossible unless you never come into civilization. Detective work, for the longest time, has been about defeating privacy.

      Example: You go AWOL because you didn't want to get sent to (insert next big US deployment) to die in the miserable conditions against an enemy that you don't have permission to shoot. You know they'll come and haul you to jail for many years, so you simply don't get a phone. The military calls your house, and of course, your roomie/parents/etc say that they haven'

  • by Joe The Dragon ( 967727 ) on Friday July 10, 2009 @04:55PM (#28654853)

    how much battery power will this suck down?

  • We had Furbies doing this several years ago and nobody freak'd out
  • People don't care about privacy, they just don't. Everyone here on /. upset about govenment civilian spy programs and such, and then everyone else gives up everything about themselves, willingly and deliberately, on sites such as Facebook.

    Let me say that again, just so it's clear - most people just /don't care/ about privacy.

    And /they/ are the ones that we need the laws to protect, those not smart or competent enough to take care of themselves.

    Bringing the point back to the article - as soon as some flashy

    • The thing is that I can control what information about myself I put up on Facebook. I can project a nice clean-cut professional image there and just not talk about what I do with my wife in bed.

      It's about control. If I can control the information about me, then it's not a problem. If someone is peeping in my windows and posting about it, then it is a problem.

  • by bobetov ( 448774 ) on Friday July 10, 2009 @04:57PM (#28654865) Homepage

    I see a lot of tags/comments asking what this is useful for. There are a few uber-nerd things like recording your life and whatnot that I'm not going to get into, but the big one is determining location.

    There are a TON of sweet things you can do with accurate location information, but the one that I'm most yearning for is to control my bluetooth, wifi, ringer volume, etc based on where I am during the day.

    I'm an Android user, and there's a very nice applet called Locale that attempts to do this, but it proves to be pretty useless. The reason is that you're either using GPS (drains battery, doesn't work indoors) or wifi (drains lots of battery, and is the primary thing you want to control) to figure out where you are. If using the microphone and cpu is cheaper in energy, then this will be a big win.

    Beyond the energy use argument, one of the main things you want to control is bluetooth - again, it drains batteries when on, and is not generally useful. But it's EXTRAORDINARILY useful in the car if you have a hands-free setup. Again, figuring out when you're in a car is hard via GPS or wifi, but this technique would seem to knock that one out of the park.

    So, in summary, having your phone know where you are in your daily routine allows it to be more intelligent about what services and functionality it enables, and thus makes your cell phone that much smarter and more valuable.

    • I was actually in the process of trying to figure out how to write a Locale plugin for doing a very similar thing. Although my idea was to use accelerometer data to tell if I am in the car and consequently switch to louder ringtone/notifications.
  • by Locke2005 ( 849178 ) on Friday July 10, 2009 @05:00PM (#28654903)
    Finally, a phone that can automatically disable the ringer when the phone's owner is having sex (or snoring).
  • There is no way in hell an app constantly running and listening to the mic is going to last 8 hours.

    I'm on my second iPhone, both the original and the 3G, there is no way in hell they have the battery life to do this.

    If by 'everyday life' the author means sitting in front of your PC with the USB cable attached, then okay, not my definition of everyday life, but okay. Otherwise, I call bullshit.

  • Now I can make use of my extended battery that I bought off of woot today. Seriously though, it would be a pointless app to drain and deteriorate the already poor life of the battery. But as others have posted, the application would be better suited for something other than the iPhone.
  • Clippy! (Score:4, Funny)

    by Anonymous Coward on Friday July 10, 2009 @05:47PM (#28655323)

    It sounds like you are having trouble reaching orgasm. Would you like me to
    1) Moan real loud
    2) Show erotic video
    3) or, just vibrate

  • by merreborn ( 853723 ) on Friday July 10, 2009 @06:34PM (#28655675) Journal

    Arbitron is already using something somewhat like this.

    They have their "test families", or whatever they're called, carry small devices they call "portable people meters [wikipedia.org]". Television programming includes sonic markers outside of the audible range, which these devices pick up via a small mic.

    One would think that being able to identify television/radio programming without pre-inserting said inaudible watermarks could simplify their process.

  • Beano Ads? (Score:3, Funny)

    by c0d3r ( 156687 ) on Friday July 10, 2009 @06:49PM (#28655829) Homepage Journal

    So if you fart a lot you'd get beano ads on your iPhone?

  • Your personal black box

  • Well, I hope for me it can recognize the sound of hot lingerie models having screaming orgasms! Heh heh heh!

    No, not really. :-( Pretty much just silence here.

  • iPhone: That clicky sequence you made just now, I hear that a couple dozen times a day, what is it?
    user: That's my password.
    iPhone: Thanks, let me just back that up for future reference.
    user: <BLINK> "Where's the battery on this thing?"

Profanity is the one language all programmers know best.