TARS Mobile App with Deep Fingertip Detector for the Visually Impaired. Tactile graphics with an Audio Response System (TARS) allows visually impaired people to learn maps, diagrams, and figures by synthesizing audio descriptions of tactile graphics. The current TARS is cumbersome since it relies on a computer and touch panel to detect fingertip coordinates. Here, we improve portability by implementing a TARS mobile app endowed with a camerabased deep fingertip detector. We implemented the hand keypoint detector proposed in Simon et al. 2017, and restricted our attention on the fingertips. Performance benchmarking was done over different tactile graphics types, participants and experimental conditions (i.e. darklight environment, partial occlusion). In normal conditions, accuracy is 98% on average, but in the case of a partially occluded hand, the detection accuracy of the thumb decreases. These results indicate that the proposed mobile TARS app would perform similar to the standard TARS while being more portable and intuitive.

An assistive lowvision platform that augments spatial cognition through proprioceptive guidance: PointtoTellandTouch Spatial cognition, as gained through the sense of vision, is one of the most important capabilities of human beings. However, for the visually impaired (VI), lack of this perceptual capability poses great challenges in their life. Therefore, we have designed PointtoTellandTouch, a wearable system with an ergonomic humanmachine interface, for assisting the VI with active environmental exploration, with a particular focus on spatial intelligence and navigation to objects of interest in an alien environment. Our key idea is to link visual signals, as decoded synthetically, to the VIxe2x80x99s proprioception for more intelligible guidance, in addition to visiontoaudio assistance, i.e., finger pose, as indicated by pointing, is used as xe2x80x9cproprioceptive laser pointerxe2x80x9d to target an object in that line of sight. The whole system consists of two features, PointtoTell and PointtoTouch, both of which can work independently or cooperatively. The PointtoTell feature contains a camera with a novel onestage neural network tailored for blindcentered object detection and recognition, and a headphone telling the VI the semantic label and distance from the pointed object. the PointtoTouch, the second feature, leverages a vibrating wrist band to create a haptic feedback tool that supplements the initial vectorial guidance provided by the first stage (hand pose being direction and the distance being the extent, offered through audio cues). Both platform features utilize proprioception or joint position sense. Through hand pose, the VI end user knows where he or she is pointing relative to their egocentric coordinate system and we are able to use this foundation to build spatial intelligence. Our successful indoor experiments demonstrate the proposed system to be effective and reliable in helping the VI gain spatial cognition and explore the world in a more intuitive way.