SuperVision: A Smart App to Help People with Low Vision Search Their Surroundings

SuperVision Search, created by a team at Schepens Eye Research Institute of Massachusetts Eye and Ear, uses artificial intelligence to identify words in real-world environments.

Have you ever found yourself staring at a shelf looking for one particular product that you can’t seem to find? This can be a struggle for anyone, but for people with low-vision, this task can be downright impossible.

Many people with low vision carry around magnifiers or use smartphone applications to zoom in on text, but these methods are imperfect, especially in cluttered environments. To address these gaps, a team of researchers at Schepens Eye Research Institute of Mass. Eye and Ear has developed a mobile app called SuperVision Search that acts as a “smart” magnifier, enabling users to search their surroundings for a keyword that the phone’s camera will spot and zoom in on.

“People with low vision know what they’re looking for. The challenge is finding the item in a crowded scene, and that’s where this app comes in,” lead researcher Gang Luo, PhD, an associate scientist at Mass. Eye and Ear, explains to Focus. “By using artificial intelligence (AI) to make the traditional magnifier smarter, this app enables people to turn their device into a tool to assist with their search for information.”

An estimated 1.5M Americans over the age of 45 have severe visual impairments caused by a variety of conditions, including glaucoma and cataracts. People with low vision often have difficulty reading and discerning fine details.

The goal behind the SuperVision app is to provide advanced navigation services to people with visual impairments, in order to increase their independence and improve quality of life. Dr. Luo and his team described how they developed the app in a study published in the IEEE Journal of Translational Engineering in Health and Medicine.

How SuperVision Search works

To begin, a user enters a keyword by voice or by text. For example, if a person is allergic to “almonds,” he or she can use the app to find out if it’s listed as an ingredient in a product. The app uses AI tools to scan the surroundings and look for this particular word. The smartphone camera then zooms in on the keyword in the captured images, which enables the user to acquire this information, and avoid purchasing the item (see image right).

In the new study, adults without vision impairment wore blurry-filtered glasses and were asked to perform 50 easy and difficult keyword searches, using both a standard magnifier and the SuperVision app. An easy task was finding a word on a restaurant menu, while a difficult task was finding a date related to a particular transaction on a bank statement. Although there was no difference in the search times between the two methods, for the easier tasks the app was about 28 percent faster than the magnifier for the difficult tasks.

“Even people with good vision may benefit from this technology,” says Dr. Luo, who recently used the app when he was having a hard time shopping for his child’s school supplies (watch video above).

This research was supported by a grant from the Research to Prevent Blindness and an anonymous donor. Follow-up studies will include having people with visual impairments testing the app.

Future versions to focus on object finding

The researchers realize when people with low vision are out, there isn’t always a word in the environment to zoom in on. Now, the team is working to improve the app’s ability to search everyday objects. Recently, Dr. Luo’s team received an AI for Accessibility grant from Microsoft to enhance the SuperVision Search app, and use Microsoft’s AI tools to spot objects in the real world, specifically a bus stop sign.

The SuperVision apps are free to the public and have been downloaded more than 800,000 times to date. Other apps the team has released include Supervision Goggles and Supervision Magnifier. Dr. Luo and his team routinely update the apps to improve functionality and performance. For more information, visit the SuperVision website. You can also download the apps directly by clicking on the following links:

• Download the SuperVision Search app on iTunes and Google Play.
• Download the SuperVision Goggles app on iTunes.
• Download the SuperVision Magnifier app on iTunes and Google Play.

In addition to Dr. Luo, co-authors of the research at Mass. Eye and Ear include Shrinivas Pundlik, Anikait Singh, Gautam Baghel and Vilte Baliutaviciute.