Facebook Eavesdropping Patent and Congress Questions

Why does Facebook have patents to listen (eavesdrop) through the microphone for ambient sound if they are only using it for uploading your video? My live feed of the Congress hearings cut out right near the end, but did anyone ask these questions?

• Mr Zuckerberg Facebook submitted a patent 7 years ago to listen to background noise (ambient waveforms) for the purposes of delivering content. Is that technology now active?
• Mr Zuckerberg, Facebook has a Shazam-like service for music identification that is used to compile Top Trends lists. So why did you say that the only audio Facebook has access to is uploaded videos from phones, and Messenger?
• Mr Zuckerberg, Facebook has had a patent for 5 or 6 years to identify where person is via voice identification in ambient environment. How is that currently being used?
• Blah blah. The senators got their day in court with Zuck but they didn’t seem very well informed. Maybe they need, I dunno, an intern to do a few minutes research eg. Google jockeys. [Quel irony!]

Facebook Eavesdropping Patent #1

Periodic ambient waveform analysis for enhanced social functions is a patent to listen to “ambient waveforms” e.g. stuff around you

In 2014 ambient patent (filed 2011)

“Client devices periodically capture ambient audio waveforms, generate waveform fingerprints, and upload the fingerprints to a server for analysis. The server compares the waveforms to a database of stored waveform fingerprints, and upon finding a match, pushes content or other information to the client device. ” . Facebook Eavesdropping Patent 2014  8706499

(ambient means “sound around you” eg what you are watching on TV, if you are in the movies etc)

And if your friends are around you (the term voice waveform fingerprint seems to be voice mapping)

“Thus a client’s location may be enhanced if the location of users whose fingerprints match the client’s is known.” 8706499

yeah turn off location tracking, go to the pub with friends, and presto, your location is tracked.

Facebook Eavesdropping Patent #2

Periodic ambient waveform analysis for dynamic device configuration is a patent for modifying your phone

Particular embodiments of the special purpose client application hosted on a mobile device of a user may be configured to continuously or periodically capture ambient audio waveform data using a microphone or sensor subsystem in the mobile device. In particular embodiments, the mobile device may perform waveform analysis on the ambient audio, and adjust the device settings, such as the ringer volume or vibrate mode, based upon the ambient waveform data. In particular embodiments, the client device may automatically match the ringer volume to match or slightly exceed the ambient audio level, thereby increasing the probability that the ringer is audible in noisy environments and appropriate in quiet environments. In particular embodiments, the self-determined location of the mobile device may be utilized to adjust the device settings. For example, if a user’s GPS indicates he is in a movie theatre, the device may automatically turn off its ringer.

Useful yes. Bloody ringtones in a movie theatre. But did Zuckerberg lie when he said the only time Faceybookz connects to phones audio is when you are uploading a video? Imma gonna go with “yes”.

“For example, GPS devices often do not function indoors. Thus, a user’s last reported location may be in the vicinity of, but not in, a movie theatre. If the reported location of a user device is near a movie theatre, and the user’s waveform fingerprint matches the soundtrack of a particular movie, the social networking system may access an online database of the show times for the theatre in the user’s vicinity. Based off the timestamp from the uploaded waveform fingerprint, the matched movie, and the GPS location, the social network may determine that the user is watching, for example, the 5:30 PM showing of “True Grit” at AMC 30. Thus the device may automatically turn off its ringer.”

What would the oncology ward of a hospital sound like and could health insurance (3rd party access)  record that data? Asking for a friend.

” Facebook sent a doctor on a secret mission to ask hospitals to share patient data
Facebook has asked several major U.S. hospitals to share anonymized data about their patients, such as illnesses and prescription info, for a proposed research project. Facebook was intending to match it up with user data it had collected, and help the hospitals figure out which patients might need special care or treatment.” (CNN)

Before you waffle on about anonymized data the non- de-hashing of profiles is a gentleman’s handshake between developers. ala Cambridge Analytica.

Optimizing video conferencing using contextual information

2016 – Video eavesdropping

I often talk about the Deep Face algorithm in my classes and Facebook’s use of object recognition in videos. E.g. identifying Sydney Harbour Bridge in the background to better target your content to Australians. But this patent goes further.

“In one or more embodiments, when capturing and sending a video conference data stream, the video conferencing system causes the client device to analyze captured video content to determine social context of the captured video. For example, the video conference system can analyze the captured video content and identify that the video content includes a conference room with ten participants to determine that the video content indicates a company meeting.”

and then there’s this:

“For example, systems and methods described herein perform a contextual analysis to identify one or more portions of interest within a video conference data stream.”

Yeah Facebook totally pwns industrial espionage. I mean, facial recognition is one thing but how it’s used is a whole other thing. You agreed to upload the video so it’s not eavesdropping. But it sets the stage for further development.

Group Ambient Patent – Cluster algorithms

Some stuff is hidden in techy speak which I don’t understand.

In particular embodiments, multiple ambient-location readings from the location service of a mobile device may be filtered and grouped to suppress location data noise and reduce the amount of location data that is stored. In particular embodiments, ambient-location readings may be clustered using a spatial-clustering algorithm, such as for example k-means algorithm. The spatial-clustering algorithm clusters a large number of spatial readings into a smaller number of spatial centers based on the distance between the location of the ambient-location reading and the location of the nearest cluster.

Basically, I think it’s saying if it hears say, the latest movie in the background, the algorithm maps clusters nearby? So a big cluster nearby is probably lots of phones in a movie house? Maybe your friends’ phones who you go to the movies with if you have friends, which I don’t but its probably not the time to have that*sob* discussion. Or random strangers sitting in a movie theatre over 28 days? I mean info is gathered over 28 days, not that they sit there for 28 days, silly. But none nearby is probably a trailer playing on the TV or YouTube trailer?

Don’t ask me, I don’t even work here 😛

I’m reading through the other patents now (I read them before but didn’t understand the implications quite so much) but I want to address something important. Lots of companies claim they take out patents to stop competitors from doing so. Just to hang on to them. IBM annoying their competitors for gits and shiggles. They have no intention of doing anything with those patents. But we are finding out more and more that Facebook DOES use those patents. Facebook of course does deals with big data companies to grab your supermarket shopping list, to profit from your purchases in the last 28 days, of baby nappies and cigarettes. Writing #socialmedia on the whiteboard behind your livestream video will have it seen by more people who are into social media. Facial recognition in photos and videos and not just tagging, is a thing. Facebook has music Shazam-type services to listen through your microphone. Deep Text predicts when you will need a taxi.

If you think these patents are blocking patents or whatever they are called (patents to block others from using it) you are dreaming. Dreaming. Asleep and dreaming. Time to wake up.