What Snap’s Patent Activity Tells Us About Their Efforts In Facial Recognition, Payments, Wearables, And More

We analyzed Snap's patents as the startup roams further into wearables, payments, computer vision, and video streaming. 

Media and consumer attention is refocused on Snap Inc. — formerly known as Snapchat — after the launch of Spectacles, camera-laden sunglasses for capturing video. Another catalyst for buzz is the company’s expected 2017 IPO for a valuation of as much as $25B.

Snap is a notoriously secretive tech startup where even teams within the company rarely know what others are working on.

But when studying the patent applications and grants of the Santa Monica-based unicorn, you get forward-looking insight into its product roadmap and product innovation strategies, including the company’s plans as it roams further into consumer electronics, payments, AR/VR, and streaming.

Snap’s patent filing trends

Snap has been active in seeking intellectual property over the years, in various areas with an emphasis on user experience and user interface, object recognition, and automated photo and video curation:

As can be seen in the chart above, activity peaked in 2014 and 2015, years in which we identified 18 unique US patent applications for Snap. There also appear to be signs of a change in emphasis, away from a focus on UX/UI and curation technologies, and toward wearables (Spectacles), as well as facial and object recognition tech.

This aligns with an increasing tech industry emphasis on computer vision, which is a form of artificial intelligence in which computers use algorithms and machine learning techniques to analyze and classify images and video. Publicly traded Facebook and Google have invested a great deal in developing similar tech, and it makes sense for Snap to push forward in this area so as to not be left behind.

What Snap’s working on

Some of the more recent applications and grants reveal possible areas of future product direction for Snap, including payments, media, and advertising.

1. Video calls with commerce/payments

One notable patent grant (#9083770, granted 7/14/2015) indicates plans for a “real time communications” video streaming platform that could connect people like a Skype or FaceTime, but also incorporates a payments platform. The patent list 4 inventors, all of whom were engineers or co-founders of AddLive, which Snap acquired in April 2014.

(The below suggest a video call with an attorney that charges $50.)

The patent mentions low-latency audio as a focus, and gives some example graphical interfaces (below) that introduce concepts of a payment platform bolted on to this “real-time communications” feature. Snap has experimented with a peer-to-peer payments platform known as Snapcash, and already provides video calls in the style of FaceTime or Skype. But doesn’t currently have a built-in payment or commerce mechanism on the platform.

The patent goes into the system languages, APIs, and packet flow for this real-time communication (RTC) tool, and goes on to describe the commercial applications:

In some embodiment, the RTC application can also be used for commercial purposes. For example, service providers, such as attorneys, can providing [sic] counseling to their clients using the RTC application. The service providers and the client may view and hear each other using the RTC application.

2. Audio/acoustic fingerprinting

As a way to manage galleries of Snap videos taken at live events, Snap has applied for a patent (#20160182875) on its audio fingerprinting process. Acoustic fingerprinting is a common method for copyright compliance, licensing, and monetization. In this process, computers can scan video and audio for certain audio clues (inaudible to human ears) that effectively serve as a flag to mark certain content (e.g. content that is owned by a certain copyright holder).

With these audio fingerprints, videos of a live event could be grouped together in their proper order based on audio analysis to see if there is match, or perhaps to prevent showing redundant clips of the same moment:

The audio fingerprints are matched to known audio fingerprints to establish matched audio fingerprints. A determination is made whether the matched audio fingerprints correspond to a designated gallery constructed to receive a sequence of videos set to an audio time line. The matched audio fingerprints and corresponding video content are added to the audio time line. The operations are repeated until the audio time line is populated with corresponding video content to form a completed gallery with video segments set to audio segments that constitute a complete audio time line.

Conceivably, this means a process figuring out if a stream of Snap Live Stories (video and photo streams tied to events) are happening concurrently at a given concert or event, in order to automate the process of building the Stories that Snap curates and broadcasts as a Live Story to its users. For example, it may offer a Live Story of images and videos from a popular music festival.

But acoustic fingerprint technology could also suggest a monetization scheme for clips and copyrighted content in a manner similar to YouTube’s “ContentID” system for flagging copyrighted material and making sure revenue produced by advertising against that content goes to the copyright owner (e.g., a music label or movie studio).

3. Object recognition

In order to categorize Snaps (ephemeral photos and videos) and Snap Stories (photos and videos that last for 24 hours) into funnels like music festival-themed event-tied videos or city-based streams, Snap is analyzing the contents of messages. Snap has also applied for patents in object recognition, with an application that suggests tailored photo filters depending on objects contained in the video message.

Snap has also imagined the technology having a possible commercial application with advertisers, as imagined here with coffee brands, which could be a possible direction for sponsored content (e.g. a brand could pay Snap to include filters tied to its brand that trigger when a relevant certain object is detected):

4. Facial recognition & privacy protection

Granted just in July of 2016, Snap designed a method (#9396354) for running facial recognition on images and then applying privacy rules to the image according to user preferences. Simply put, Snap could amass a database of faces and blur out faces in public video/photo messages for those users who have opted for privacy in their settings.

The patent language cites ephemeral galleries (Live Stories) as possible applications of the process:

A method executed by a computer, comprising: receiving an image from a client device; executing a facial recognition technique against an individual face within the image to obtain a recognized face; applying privacy rules to the image, wherein the privacy rules are associated with privacy settings for a user associated with the recognized face; and distributing a privacy protected version of the image, wherein the privacy protected version of the image has an altered image feature, wherein the privacy protected version of the image is distributed to an ephemeral gallery

Basically, if you’re a private opted-out user who find their way into a Snap story that gets picked up by the Live Stories stream (i.e. someone at a music festival films you in the crowd), the event stream would blur out or otherwise stop the image from being in the public broadcast.

The technology is in a similar vein to Facebook’s DeepFace, a facial recognition technology for auto-tagging friends in photos with 97.3% accuracy (you can opt out of this). Facebook is currently facing lawsuits for its handling of biometric data with DeepFace, so Snap would need to tread cautiously in this area.

Snap’s M&A for patents

Aside from the streaming payments platform mentioned above (that was ostensibly a product of the AddLive acquisition), we rounded up a few more patents originating from Snap’s M&A. Snap has been one of the most aggressive startups in terms of acquiring other startups, with 8 such acquisitions through August of this year. Patent data reveals how M&A can help a company build its product strategy by acquiring outside innovation.

1. Recognition technology for 3D selfies

Snap quietly bought Seene in June 2016, and Seene’s parent company Obvious Engineeringapplied (#20160284123) for patents covering 3D facial recognition that also have an AR/VR application. Combined with Looksery‘s selfie filters (an image layer of fun special effects that is placed atop the original image), which Snap bought in September 2015, Seene’s software could add to the depth and flexibility of Snap’s arsenal of selfie filters. More accurate face mapping and 3D imaging would allow selfies and pictures to be taken in 3D (meaning, for example, that a three-dimensional selfie could be used to create a cartoon-like image of a Snap user).

Check out the patent drawings for its mobile app, featuring room capture & recreation and facial mapping (featuring a bearded user):

Seene’s product allowed you capture 3D objects (such as a statue or sneaker) and explore them within your smartphone.

2. ‘Eyewear having linkage assembly,’ a.k.a. Spectacles

Snap has two major filings related to its Spectacles eyewear, and both patents (which were granted November 1st 2016) list former Vergence Labs engineer David Meisenholder as an inventor. Vergence Labs was stealthily bought by Snap for $11M in cash and $4M in stock in December 2014.

Vergence Labs’ designs for camera laden glasses were fashionable and, interestingly, its camera housing (right) was very inconspicuous compared to Snap’s final Spectacles design (left). 

Snap’s patents describe the special in-case charging mechanism on the Spectacles that latches to the eyewear in a manner similar to Apple’s magnetic charger on the MacBook. The design was incorporated into the Spectacles case.

And the patent drawing details exactly where on the frame hinges the charging ports are available: 

Existing tech

While these patents have already been incorporated into the app, they still shed light on how the company has innovated in automation, security, privacy, and user experience/personalization.

1. Machine-driven content curation

Snap filed for IP related to automating the curation of Events-related galleries, which the company calls Live Stories (and was granted patent #9385983 in July 2016).

Where traditional media needed a centralized editing room to parse and piece together disparate camera angles and source into a coherent narrative, Snap has laid down IP for its vision for automating this curation process (likely working in its parsing of images and video with tech like object, facial, and audio recognition).

The language of the patent says:

A determination is made that the message parameter corresponds to a selected gallery, where the selected gallery includes a sequence of photographs or videos. The message is posted to the selected gallery in response to the determination. The selected gallery is supplied in response to a request.

The patent drawings show a familiar UI that suggests Event galleries for users to select.

Here’s a look at the patent specifics:

2. Internet Protocol-based intrusion detection

Granted in September 2015, the company’s patent (#9148424) relates to a method for preventing logins from unauthorized users. The patent describes a method to analyze the IP address of computers accessing a certain account to determine if it links up with the history of the authorized user.

The language describes an analysis method to prevent malicious account takeovers:

A login history comprising login request data for the server computer is analyzed to identify a plurality of usernames, wherein each username of the plurality of usernames is associated with a corresponding login request from the first IP address within a threshold time period of the first request time. In response to determining a login success ratio is below a threshold login success ratio and a number of unique usernames in the analyzed data is above the unique username threshold, the system automatically performs a security action.

3. Content delivery network for ephemeral objects

Snap’s secret sauce in the early days was its “self-destructing” messages. Unsurprisingly, the company sought several patents around its ephemeral messaging technology.

Grant #8914752, which was first applied for in 2013, gives a pretty simple schematic:

4. Social network profile access

Snap was granted a patent (#9026943) in May 2015, so that user settings could dictate how much profile information they give out to others. The patent states that the idea comes from concerns about social networks having “hard to find” privacy settings that make it difficult to know what’s visible to other users:

And the patent describes a method for implementing a way for users to manipulate their settings in one touch:

5. Permanent and semi-permanent messages

Snap applied for patents in 2014 that suggest something similar to what became the photo-management module known as Snapchat Memories, which allows users to browse photos and stories at a later time.

In part the patents focus on messaging replay (a feature that still exists, allowing a 1-time replay of disappearing messages). But they also break from Snap’s famous ephemeral messaging and allows users to save the content, showing the team at Snapchat was thinking about different types of permanence early on. Opting to have certain messages stay permanent could encompass message replays, but also Stories, and Snap’s latest big focus with Memories.

6. Hybridization of voice notes and calling

Snap got into the voice-calling realm with a late 2015 application for a calling and voice-messaging hybrid (#20160147435). Although it’s not a well-known feature, voice & video calling has been in the app for some time.

7. Ghost barcodes

In February 2016, Snap was granted a patent for its process with its ghost barcode reading (#9111164), that first recognizes the famous ghost shape before reading and decoding the QR code information. Initially the barcodes were quick method to add friends to your lists (each user has a unique barcode), but the patent drawings also imagined a commercial use case.

Below is a drawing with an advertising use case:

8. Geo-location based ‘pictographs’ aka. Geofilters

In late 2015, Snap was granted its first of several applications related to Geofilter technologies. Based on your location, Snap messages will have the option of filters that display related pictograms (often a fun sign of a user’s city location) superimposed on the message.

Other schematics in an application (#20160085773) show an emoji selection that changes based on the location of the user (they called them geomojis). Snap also went on to buy Bitstrips, the startup behind popular avatar maker Bitmoji, in March 2016 and has since incorporated the technology into messaging.

9. Personalized UI

Getting its start as a successful mobile app, Snap pursued lots of patents related to its widely-discussed UI/UX, which is notoriously difficult for anyone over 25 years old to use. The company’s focus has shifted to making the UX more personalized according to user data, location, favorite friends, and the like.

This patent below was granted August 2016 (#9430783), which employs geo-location to prioritize messages based on your user data (in this case, seeing where you are and suggesting a related event gallery to add a Story):