Bridging the Digital Divide: Empowering Elderly Smartphone Users with Intelligent and Human-Centered Design in Agemate (2025)

Abstract: As mobile devices become central to modern life, elderly users often struggle with their complexity, leading to digital divide. This paper explores how the integration of Human-Computer Interaction (HCI) principles and Natural Language Processing (NLP) techniques can enhance the way elderly users learn to use smartphones. To demonstrate this approach, we present AgeMate, a prototype mobile agent designed to support seniors in acquiring smartphone skills more intuitively and effectively. Specifically, we investigate how personalizedfeedback generated by large language models (LLMs), appropriate granularity in instructional content, and mechanisms for preventing and correcting user errors can contribute to more adaptive and user-friendly learning experiences for elderly users. Rather than focusing solely on system performance, our study emphasizes the instructional value of NLP-enhanced interaction: enabling step-by-step, conversational teaching tailored to users’ real-time context. By analyzing usage patterns and interaction challenges, we propose design strategies that bridge the gap between accessibility and intelligent guidance to better support elderly users in digital environments.

## introduction recent advancements in llms have demonstrated strong reasoning abilities, enabling them to perform complex tasks such as recommendation and hci @xcite . the growing availability of llm has inspired research into mobile agents that operate directly on mobile devices, offering potential for practical ai applications in real-world scenarios @xcite . previous work on mobile agents primarily focused on automated task execution through predefined rules or access to system metadata such as xml layouts. while these approaches have enabled end-to-end execution, they often lack interactivity, flexibility, and adaptability to user needs @xcite . real-time, as discussed in prior work @xcite , signifies the system's ability to provide timely and responsive assistance integrates smoothly with ongoing user interactions, minimizing perceived delays to ensure a seamless experience-a capability that has received relatively little attention in agent-user collaboration, such as assisting users through educational guidance or real-time feedback during mobile app usage @xcite . to address this gap, we explore the intersection of llms, natural language processing (nlp), and human-computer interaction to investigate how a mobile agent can support both autonomous execution and user-guided interaction. our central research question is: how can a mobile agent be designed to help users-especially elderly or less tech-savvy ones-learn and perform tasks on mobile devices? to explore this, we first conduct a brief literature review on mobile agents and llm-powered interaction systems. we then develop a prototype system, agemate, that leverages vision language model (vlm) to interpret ui screenshots, plan actions, and interact with users through either direct execution or tutorial-style guidance. agemate is capable of parsing multi-round llm responses and dynamically adapting to user feedback. our contributions are threefold: (1) we analyze prior work at the intersection of nlp and hci in the context of mobile agents; (2) we develop agemate, a dual-mode (text and speech io) agent system for mobile interaction and education; and ## related works in recent years, the development of mobile agents has advanced significantly, but most existing methods have focused on rule-based automation and limited interaction with users. for instance, some systems leverage pre-defined rules for task execution, relying on access to system-specific metadata like xml layouts @xcite . these approaches, while functional, often struggle with providing dynamic and personalized experiences, especially when it comes to handling complex tasks that require real-time adaptation to user behavior. one notable area that remains underexplored is the integration of more interactive mobile agents that can collaborate with users. in particular, research has highlighted the potential of mobile agents that provide educational guidance or realtime feedback to enhance user experience during mobile app usage @xcite . such collaboration could bridge the gap between pure automation and user-centered assistance, allowing for a more personalized and adaptive approach to mobile task execution. recent advancements in large language models (llms) have opened new avenues for enhancing mobile agents with greater flexibility and adaptability. some studies have shown the potential of llms in processing and interpreting complex information from various sources, offering a more natural form of communication between agents and users @xcite . however, integrating llms into mobile agents remains challenging due to the limitations in ensuring consistent task execution across diverse mobile environments. this gap in the literature underscores the importance of exploring ways to combine llms with task execution in mobile agents, as well as improving agent adaptability to dynamic, user-driven scenarios @xcite . thus, while current research has made strides in automating mobile tasks, there is a need for further exploration of more adaptable, interactive, and usercentric mobile agents that leverage advanced ai models. ## system system overview agemate is an llm-powered mobile assistant designed to address smartphone operation challenges for older adults. it uses gpt-4o as its underlying llm. agemate features two distinct interaction modalities: users can choose the mode that best meets their needs-either fast, assisted execution or detailed tutorials with varying levels of detail. auto-execute mode when users want agemate to execute a command on their behalf, they simply enter a question into the user interface. agemate then initiates a multi-round execution process. in each round, the following steps are performed: first, agemate captures the screenshot and xml structure of the current phone page and stores them locally. using the xml file, agemate overlays a semi-transparent numbered label on each element. once labeling is complete, the labeled screenshot is converted to base64 encoding. next, agemate concatenates a predefined prompt with the xml structure and the encoded image, and sends this composite input to the vlm. the response is structured into sections: observation, thought, action, and summary @xcite . observation is a brief, simple description of the current screen, outlining the app/page, main content, key interactive areas, and possible user actions. thought is the agent's reasoning process, explaining the plan to address the next task step based on the screen's observation and why a specific action is chosen, including sensitivity assessment. action specifies one of the following operations: back, tap, text, long press, or swipe, applied to a particular element or coordinate context. summary is a brief internal note for the agemate system to record all past actions and reflect progress toward completing the task; it is not shown to the user. upon receiving the response, agemate parses it to determine the next step, such as tapping a specific coordinate or entering text into an input box. as the rounds progress, agemate dynamically adjusts its decisions based on the current page content to fully address the original query. auto-execute mode, users begin by entering a question into the input box, and agemate performs a multi-round execution. however, the process diverges once agemate parses the llm response. ## conclusion we developed agemate, a mobile application that explores how llm agents can effectively teach elderly users. agemate not only automates execution of user queries but also offers three levels of tutorial granularity, allowing users to choose tutorials based on their understanding and willingness to explore. to address potential errors from the llm and those made by users, we incorporate both preventative and bidirectional error correction methods through advanced llm techniques and refined user interaction strategies, thereby enhancing robustness. for potentially hazardous actions, agemate issues explicit warnings to ensure user safety. through these approaches, agemate delivers accurate and user-friendly tutorials that support effective teaching and learning for the elderly. in addition, we discuss the use of multimodal interaction, multi-agent collaboration within mobile agents, and the advantages of browser-based forward and backward navigation for integrating tutorials with third-party applications. altogether, agemate exemplifies a promising direction for integrating recent advances in llm agents into realworld applications. we envision closer collaboration between the nlp and hci communities to further explore how such agents can assist elderly users and help bridge the digital divide. in addition, we aim for agemate to support forward and backward navigation similar to a web browser. however, through our research and technical validation, we found that it is relatively easy to realize backward in the interface of third-party applications on phones (meaning applications other than agemate), but it is difficult to realize forward accurately because the system doesn't provide apis similar to the "forward stack", and some applications are based on security considerations such as progressing the user to modify their code, etc. in future research, we hope to explore stable and reliable ways of forwarding and backwarding for thirdparty applications, based on which older adults can conveniently go back and forth to view tutorials at each step of the interaction and can roll back the interface to the correct position if they make an error.