C hat GPT _ P owered_ T ourist_ A id_ A pplications__ P roficient_in_ H indi__ Y et_ T o_ M aster_ T elugu_and_ K annada (2023)

Abstract: This research investigates the effectiveness of Chat- GPT, an AI language model by OpenAI, in translating English into Hindi, Telugu, and Kannada languages, aimed at assisting tourists in India’s linguistically diverse environment. To measure the translation quality, a test set of 50 questions from diverse fields such as general knowledge, food, and travel was used. These were assessed by five volunteers for accuracy and fluency, and the scores were subsequently converted into a BLEU score. The BLEU score evaluates the closeness of a machine-generated translation to a human translation, with a higher score indicating better translation quality. The Hindi translations outperformed others, showcasing superior accuracy and fluency, whereas Telugu translations lagged behind. Human evaluators rated both the accuracy and fluency of translations, offering a comprehensive perspective on the language model’s performance.

## introduction language barriers often hinder effective communication and can lead to misunderstanding. e.g., a tourist may ask for directions to get to a specific location, but the local person may understand differently. such communication can lead to getting lost or taking longer. though english is widely spoken and understood, there could be significant communication issues in understanding the accent and language. for instance, visitors may have a hard time understanding similarsounding words like "bazaar" (market) and "bizarre" (strange). accurate and fluent translations play a crucial role in overcoming these challenges and ensuring smooth communication, enhancing visitor satisfaction, and fostering cultural understanding. so, developing efficient communication tools to bridge language barriers has become highly fundamental @xcite , @xcite , @xcite . generative ai technology learns using supervised and unsupervised algorithms to analyze, synthesize, summarize and transform language data @xcite , @xcite . such technology provides many solutions, including language translation, to develop tourist-aided applications. chatgpt is one of the generative algorithms which can do several tasks, such as summarization, planning, and translation, based on prompts. chatgpt is a transformer model trained as instructgpt by openai @xcite . the objective of this study is twofold: 1. to evaluate how well chatgpt, an ai tool, can translate english into indian languages like kannada, hindi, and telugu for international visitors looking for information about indian cuisine and travel. 2. to conduct an error analysis and provide valuable insights to improve these models for the tourist domain, so that such ai language models can be applied to enhance communication and provide better experiences for travelers in foreign countries @xcite . ## data collection and categorization the initial data for this study was collected using a multi-faceted approach, combining insights from informal interviews with a diverse set of frequent travelers, online searches, and interactions with chatgpt. participants were asked questions about their experiences with language barriers, communication difficulties, and their expectations regarding translation services. this ensured a comprehensive understanding of travelers' challenges faced when 1 seeking local information, coupled with online travel websites and blogs. to further help the data collection process, queries were inputted to chatgpt simulating travelers' interactions. overall, around sixty questions were finalized for further analysis. all these sixty questions were then analyzed by two independent volunteers to identify the key themes. a third volunteer was asked to resolve any conflicts and discrepancies. from the original set of sixty questions, fifty relevant questions were shortlisted and were categorized into 3 themes; general, food, and travel. the theme-based categorization allowed for a comprehensive analysis that aligned with research objectives. you can find the list of 50 questions used for this experiment in the appendix a. ## experiment methodology the experiment utilizes the openai language model (llm) model, specifically the "gpt-3.5turbo" variant, for translating english text to the target language. the experiment involves a conversation-style interaction between the user and the system, where the system acts as a helpful assistant providing translations @xcite . the prompt used to generate the data consists of two parts: a system role and a user role. the system's role is to introduce the assistant's purpose, which is to assist with english to target language translation. the user's role is to instruct the assistant to translate a specific english text to the target language. the provided source text serves as the input for the translation task(lai et al., 2023). the prompt used to generate the translation data: ## evaluation methodology in this work, we use a subjective and an objective evaluation methodology. subjective evaluation asks native speakers of a language to rate the accuracy and fluency of the translation (white and o'connell, 1993). to assess translation quality, participants were asked specific questions covering several characteristics of translation accuracy and fluency. for example, one question asked participants to assess the translated text on a scale of 1 to 5, indicating how correctly it conveyed the original text's meaning. another question focused on fluency, asking participants to judge the smoothness and naturalness of the translated text. we aimed to cover multiple aspects of translation quality by incorporating a wide set of questions, ensuring a thorough evaluation procedure. for accuracy evaluation, "on a scale of 1 to 5, how accurately does the translated text convey the meaning of the original text?" for fluency evaluation, "on a scale of 1 to 5, how fluent is the translated text?", where 1-bad, 2 -poor, 3 -fair, 4 -good, 5 -excellent. collecting scores on accuracy and fluency provides a comprehensive evaluation of the translation quality. it helps this study understand how accurately the ai model performs in translating the meaning of the original text while maintaining fluency in the target language. ## results and analysis the average scores for hindi translations in the general theme indicate high accuracy (4.8) and fluency (4.6) indicating effective translations. kannada translations received relatively lower average scores for accuracy (3.7) and fluency (3.5) compared to hindi. however, they moderately demonstrated decent accuracy and fluency. telugu translations received the lowest average scores for accuracy (2.5) and fluency (2.1), suggesting there is room for development in accurately and fluently interpreting the meaning. ## objective evaluation ## qualitative evaluation the evaluators appreciated the comprehensive methodology used for data collection. they commended the use of a multi-faceted approach, combining insights from informal interviews, online searches, and interactions with chatgpt. below are some of the comments expressed in the participants' own words: for hindi translation: "i was amazed with how well chat gpt has translated some of the questions, few words were translated beautifully. but some did not make much sense and had opposite meaning. but a good start :) " 100 "i'm pleasantly pleased with some of the translations; they incorporate rich vocabulary and demonstrate good grammar." "the majority of the translated text shows promising improvements and correctness. with some additional refinement, it will become even more polished and accurate." for kannada translation: "quite impressed with some of the translations, using some rich words which do seem grammatically correct. there is room for improvement which i'm sure it'll happen soon. the colloquial and spoken kannada is quite different from the dramatized version and it does have work to do there." "harder questions translations and fluency is really good. simpler questions are off. some questions have literary translation gives a different meaning altogether." "some of the translations are apt and very good. majority of them seems to do literal translation instead of intelligent meaningful translation similar to spoken language. overall work in progress and would need more usage based inputs to correct the responses to make it very close to original intention of the sentence." "most of the translated text has half correct translation. it needs lot of refinement." for telugu translation: "the translation accuracy is not usable. regional variations or idiomatic expressions are not in the current translation." "while some parts of the translated text are correct, a significant portion requires further refinement to achieve better accuracy." "the translation is ambiguous and does not reflect the true meaning in several cases." ## limitations it is important to acknowledge the limitations of this study. firstly, the sample size, although diverse, may not fully represent the entire population of indian tourists and visitors. secondly, the volunteer ratings used for evaluating the translations introduce a potential source of subjectivity. while efforts were made to ensure consistency and reliability in the ratings, individual preferences and biases may have influenced the results. additionally, the study focused on three specific language pairs (english to hindi, english to kannada, and english to telugu), and the findings may not be directly applicable to other language pairs. furthermore, a substantial limitation lies in the ambiguity surrounding the employed prompting process. the lack of transparency regarding this critical aspect raises questions about the potential impact of alternative prompts, such as those focused on identifying correct and incorrect translations, or the utilization of different prompts altogether. also, this study did not benchmark the performance of chatgpt against other consumer tools supporting real-time language translation, such as apple and google. future studies should consider expanding the sample size, involving a wider range of participant demographics, and utilizing objective evaluation measures to complement the volunteer ratings. ## conclusion this research evaluated chatgpt's efficacy as a digital companion for tourists in india, particularly in translating english to hindi, kannada, and telugu. hindi outshone the others in accuracy and fluency, whereas telugu lagged. the study underscores the importance of effective translation tools in facilitating communication amidst india's linguistic diversity and surging foreign tourism. utilizing 50 diverse questions, participants assessed the translations for accuracy and fluency. the findings showed consistent superior performance by hindi, moderate results in kannada, and considerable improvement opportunities in telugu translations. @xcite the research utilized the bleu score, a recognized yardstick for machine translation quality, for assessment. results showed hindi translations aligning closely with reference translations, while kannada had a moderate 101 correspondence and telugu fell behind, needing considerable refinement. based on this, the study advised enhancing telugu translations through exploring advanced models, diversifying training data, or employing techniques like transfer learning. the high performance of hindi should be preserved through ongoing evaluation and training with refreshed datasets. improvement strategies for kannada include expanding training data, fine-tuning methods, or leveraging sophisticated translation models. continuous evaluation of the translation models and incorporating human review and feedback are recommended practices to ensure ongoing improvements in translation quality. by addressing these recommendations, the virtual tourist companion powered by chat-gpt can provide better experiences for indian tourists and visitors, enhancing communication and fostering cultural understanding.