Harvard smartphone app guides the blind faster, with fewer collisions

If you have standard human vision (even if assisted by glasses), chances are you’re adept at finding things, people, and paths to them. But if you have impaired vision, or none at all, countless tasks that are simple for fully sighted people become difficult, dangerous, or even impossible. Fortunately, plenty of new high-tech tools offer assistance, including robot guide dogs, a cane-bot (looking a bit like a small R2D2 with giant wheels), multiple types of smart glasses with descriptive audio, and even a haptic vest for blind dogs.Adding to such technological advances is Mobilio, a Harvard-University-developed AI smartphone app employing sensors and customized beeps that offers real-time navigation towards desired locations while steering users away from obstacles and dangers.In their Nature Biomedical Engineering paper, PhD student Raymond Liu and assistant professor of bioengineering Patrick Slade explain how 14 test subjects using the Mobilio app with a cane could reach their destinations 13% faster – and with 41% fewer indoor collisions – than those who used a cane with Google Maps (Google Maps isn't detailed enough for safe use by the visually impaired).Mobilio’s many advantages arise first from the researchers’ sociological needs- and experience-assessment of more than 100 people with limited or no sight, and whose responses demonstrated three core needs: continuous guidance along straight paths; accurate, real-time, turn-by-turn navigation; and dependable detection of obstacles. Personalization Experiments for Users with Blindness or Low Vision The app’s semantic segmentation datasets use images from pedestrians, not from drivers, and thus are more accurate and relevant to people walking on sidewalks and crosswalks instead of driving on roads. As Liu explains, “There’s a need for more data that comes from the perspective of humans walking along sidewalks, or walking outdoors.”Because his own older brother is blind, Liu is strongly aware of the need to consult those whom his technology seeks to help. “Having actual people test out your device is absolutely crucial, because it’s impossible to predict how somebody is going to react to your device,” says Liu. “Especially within blindness and visual impairment, which is a huge, diverse set of impairments."Another advantage of Mobilio is that, by harnessing the widespread adoption of smartphones, the app bypasses the need for users to purchase expensive or difficult-to-find equipment. Instead, it employs the smartphone’s camera, GPS, inertia-detection, and (when present) LiDAR. To this combination, Liu and Slade added an innovative computer-vision model analyzing the camera’s live video to offer a real-time safe path along an unfolding route by prompting users with customizable audio cues attuned to the user’s actual walking gait. Demo: Navigation Algorithms and Spatial Audio Feedback “Our system uses the sensors in a smartphone, GPS information, and the actual motion of the person to create what’s essentially a small autonomous vehicle plan on how to navigate them from where they are to where they want to go,” says Slade. Liu notes that a key breakthrough for Mobilio was its performance on a standard smartphone, rather than relying expensive and inaccessible specialty devices with faster processors and greater memory.The team at Harvard's John A. Paulson School of Engineering and Applied Sciences (SEAS) plan to test Mobilio in more real-world settings to assess how well it can adapt to users’ daily routines, habits, and preferences.Source: Harvard University