The Spoon Is in the Sink: Assisting Visually Impaired People in the Kitchen (2021)

Abstract: Visual Question Answering (VQA) systems are increasingly adept at a variety of tasks, and this technology can be used to assist blind and partially sighted people. To do this, the system’s responses must not only be accurate, but usable. It is also vital for assistive technologies to be designed with a focus on: (1) privacy, as the camera may capture a user’s mail, medication bottles, or other sensitive information; (2) transparency, so that the system’s behaviour can be explained and trusted by users; and (3) controllability, to tailor the system for a particular domain or user group. We have therefore extended a conversational VQA framework, called Aye-saac, with these objectives in mind. Specifically, we gave Aye-saac the ability to answer visual questions in the kitchen, a particularly challenging area for visually impaired people. Our system can now answer questions about quantity, positioning, and system confidence in regards to 299 kitchen objects. Questions about the spatial relations between these objects are particularly helpful to visually impaired people, and our system output more usable answers than other state of the art end-to-end VQA systems.

## introduction visual impairment can lead to seemingly unrelated health issues. specifically, malnutrition has been associated with visual impairment because of the difficulties encountered when shopping for, preparing, and eating food @xcite . one major issue is that preparing a meal involves various situations where visually impaired people feel unsafe. a lack of spatial awareness and depth perception, especially when using a knife or preparing hot meals, contributes to concerns about getting injured. another common concern is hygiene, e.g. the inability to read expiry dates, see dirt on vegetables, recognise mold on food or spot that meats are thoroughly cooked. as a result, most visually impaired people only prepare meals with the help from family members or carers -or simply do not prepare hot meals at all @xcite . in this study, we explore the use of a visual question answering (vqa) system to alleviate some of the issues that visually impaired people encounter in their kitchen. we describe aye-saac, a voice assistant that can locate objects commonly found in a kitchen. the underlying architecture for aye-saac was developed in 2020 by students at heriot-watt university and was designed to be both transparent and controllable, aligning with our needs. we extended the object detection capabilities from 30 to 299 types of kitchen objects. further added functionality includes: we investigate how well aye-saac handles the newly added user-intents, how well it detects the newly added kitchen objects, and compare the object detection capabilities against two dedicated end-to-end (e2e) vqa systems. we show that aye-saac outperforms these two e2e systems when queried about object positioning by returning more descriptive and usable positioning information. ## aye-saac aye-saac is a modular and extensible conversational vqa framework that is implemented as a collection of independent microservices. isolating each service allows aye-saac to utilise concurrency when analysing image data; ensuring more intensive operations do not hinder the system from responding to other requests. figure 2 shows the flow of data between all the individual services, using rabbitmq to control their communication. ## conclusion blind and partially sighted people face many challenges in the kitchen. we presented our version of aye-saac, a conversational vqa framework that aims to start tackling some of these challenges. while our system can still be improved, we have shown that aye-saac: (1) provides more usable responses than pythia and hiecoatt when asked for an object's location; (2) is transparent in design and also when detailing its own confidence of a previous response; (3) can have a multi-turn interaction; (4) still has an accurate nlu intent classifier with the added functionality and plural handling; and (5) can answer questions about 299 kitchen objects. we had planned to run a user evaluation with visually impaired people in an accessible kitchen. unfortunately due to covid, we had to cancel this. in future work we could train a more accurate kitchen-specific object detection model on the ek dataset, integrate common-sense knowledge using kgs, and more. once complete, we hope that a human evaluation could take place.