In recent years social media interactions have increasingly moved in the direction of visual, and it has become increasingly important for social media platforms to implement methods to detect and classify visual content – both images and video. Effective image recognition not only allows more accurate search results to be presented, but it can also have a role in recognizing potentially offensive content without a need for users to report it first.
Facebook has made many advances in its platform’s image recognition capabilities in recent years, including the ability to automatically categorize images based on an analysis of their content. This allows facebook users to search for photos and receive relevant results, even if the uploaded photo isn’t tagged with the search phrase.
The latest improvement to the Facebook image recognition capability is the facility to recognize and extract text on an image. This is particularly useful in allowing Facebook to identify and categorize memes, which often take the form of text overlaid on an image.
The new Facebook Rosetta text-in-images image recognition system is designed to recognize and extract text in any image, not just memes. This could be useful in categorizing other types of images, for example, photographs of shops or public buildings which have signage on them.
Facebook’s Rosetta system is already in use, extracting text in real time from more than a billion public images on both Facebook and Instagram. Perhaps more impressive is the fact that Rosetta doesn’t just analyze and extract from still images, but also from video frames, and has been configured to recognize multiple languages – not only English.
This advanced image recognition tool has obvious advantages in categorizing content to improve search and discovery of relevant visual content, but it also has the potential to provide greater context for visually impaired users on social media.
The system could also be useful to social media marketers. This type of advanced Facebook image recognition could, for example, allow brands to search for images based on their brand or product names, with Rosetta recognizing users wearing clothes with your branding on, or items visible in the background of images. This, in turn, could allow marketers to reach out to those users with related special offers. This has the advantage that you know you are reaching out to customers who already use and – whether deliberately or inadvertently – promote your products online, and who are likely to do so again in the future.
There is also the potential for Facebook to provide insight tools, giving perspective on your target audience by cross-referencing product ownership – extrapolated from image recognition– with other demographic user data.
It remains to be seen how successfully Rosetta or any future iterations of Facebook image recognition tech will be at recognizing text in videos – the functionality exists at present, but the sheer volume of video content uploaded to social media platforms these days makes the scalability of the endeavor challenging.
In the meantime, however, the ability to search for text in images across billions of posts on both Facebook and Instagram presents some intriguing possibilities.