Introductory hands-on courses such as our smartphone-based coding courses, SuaCode [1, 2] require a lot of support for students to accomplish learning goals. Offering assistance becomes even more challenging in an online course environment which has become important recently because of COVID-19 with students struggling to get answers to questions. Hence, offering quick and accurate answers could improve the learning experience of students. However, it is difficult to scale this support with humans when the class size is huge -hundreds of thousands -since students tend to ask questions whose answers could be found in course materials and also repetitively.
There has been some work to develop virtual teaching assistants (TA) such as Jill Watson [3, 4] , Rexy [5] , and a physics course TA [6] . All of these TAs have focused on logistical questions, and none have been developed and evaluated using coding courses in particular. Also, they have used one language (e.g. English).
Given the multilingual context of our students -learners across 38 African countries -in this work, we developed an Artificial Intelligence (AI) TA -Kwame -that provides answers to students' coding questions from our SuaCode courses in English and French. Kwame is named after Dr. Kwame Nkrumah the first President of Ghana and a Pan Africanist. Kwame is a Sentence-BERT-based questionanswering (QA) system that is trained using the SuaCode course material [7] and evaluated using accuracy and time to provide answers. It finds the paragraph most semantically similar to the question via cosine similarity. We compared Kwame with other approaches and performed a real-time implementation.
The rest of this paper is organized as follows: In Section 2, we talk about related work. In Section 3, we describe our data and preprocessing. In Section 4, we describe the QA system and its real-time implementation. In Section 5, we describe our experiments and evaluation. In Section 6 we present and discuss the results. In Section 7, we describe the limitations of this work and future work. We conclude in Section 8.
In this section, we describe related work on virtual teaching assistants and the QA sentence selection task.
The first work on virtual TAs was done by Professor Ashok Goel at the Georgia Institute of Technology in 2016. His team built Jill Watson, an IBM Watson-powered virtual TA to answer questions on course logistics in an online version of an Artificial Intelligence course for master's students [3] . It used question-answer pairs from past course forums. Given a question, the system finds the closest related question and returns its answer if the confidence level is above a certain threshold. Since then, various versions of Jill Watson have been developed to perform various functions: Jill Watson Q & A (revision of the original Jill Watson) now answers questions using the class' syllabi rather than QA pairs. Jill Watson SA gives personalized welcome messages to students when they introduce themselves in the course forum after joining and helps create online communities; Agent Smith aids to create coursespecific Jills using the course' syllabus [4] . Jill's revision now arXiv:2010.11387v1 [cs.CL] 22 Oct 2020 uses a 2-step process. The first step uses commercial machine learning classifiers such as Watson and AutoML to classify a sentence into general categories. Then the next step uses their own proprietary knowledge-based classifier to extract specific details and gives an answer from Jill's knowledge base using an ontological representation of the class syllabi which they developed. Also, the responses pass through a personality module that makes it sound more human-like. Jill has answered thousands of questions, in one online and various blended classes with over 4,000 students over the years.
Similar work has been done by other researchers who built a platform, Rexy for creating virtual TAs for various courses also built on top of IBM Watson [5] . The authors described Rexy, an application they built which can be used to build virtual teaching assistants for different courses and also presented a preliminary evaluation of one such virtual teaching assistant in a user study. The system has a messaging component (such as Slack), an application layer for processing the request, a database for retrieving the answers, and an NLP component that is built on top of IBM Watson Assistant which is trained with question-answer pairs. They use intents (the task students want to be done) and entities (the context of the interaction). The intents (71 of those, e.g., exam date) are defined in Rexy but the entities have to be specified by instructors as they are course-specific. For each question, a confidence score is determined and if it is below a threshold, the question is sent to a human TA to answer and the answer is forwarded to the student. The authors implemented an application using Rexy and deployed it in an in-person course (about recommender systems) with 107 students. After the course, the authors reviewed the conversation log identified two areas of requests (1) course logistics (e.g., date and time of exams) and (2) course content (e.g. definitions and examples about lecture topics).
In the work by Zylich et al. [6] , the authors developed a question-answering method based on neural networks whose goal was to answer logistical questions in an online introductory physics course. Their approach entailed the retrieval of relevant text paragraphs from course materials using TF-IDF and extracting the answer from it using an RNN to answer 172 logistical questions from a past physics online course forum. They also applied their system to answering 18 factual course questions using document retrieval without extracting an answer.
These works do not focus on answering questions about the course content but rather, course logistics such as the format for submitting assignments, goals of the course, etc. Our work will bridge that gap by focusing on providing answers to questions about course content like "how do I draw a circle at the center of my screen?". Additionally, the technical details of some of the systems and their evaluations both offline and online are not available (e.g. ( [3, 4, 5] and hence they are not that easy to build upon and compare with. Our work provided these details along with our systematic evaluation to allow other researchers to replicate our work and also build upon it to create similar systems for their context. Also, all the previous systems only work for one language (e.g. English) whereas our system works for English and French. Additionally, none of those TAs have been developed for coding courses in particular. Coding courses have specific vocabulary which become more important when answers to content questions have to be provided. They pose unique challenges that need to be addressed to have systems that work well. Our work addresses these.
Within the domain of question answering, there are two paradigms: Machine reading and sentence selection. Machine reading entails selecting a span of text in a paragraph that directly answers the question. To extract the answer span, either the paragraph is provided (e.g., in the SQuAD question answering challenge [8] , or the relevant paragraph or set of paragraphs have to be retrieved from various documents first (e.g., as used in the physics QA system described in the previous section [6] ). TF-IDF is mostly used for the paragraph(s) or document retrieval [9] . For answer extraction, RNNs have been used [6, 9] but BERT-based models are the current state-of-the-art [10] .
The task of sentence selection entails predicting which sentence among a group of sentences correctly answers a question. Two examples of public challenges within this domain are the WikiQA and Natural Questions. The WikiQA dataset contains 3K questions from Bing queries with possible answers selected from Wikipedia pages [11] . Natural Question contains 307K questions based on Google searches [12] . Various approaches have been used to tackle these challenges such as TF-IDF and unigram count, and word2vec with CNN [11] and BERT and RoBERTa [13] . BERT based models are currently state-of-the-art for natural language processing tasks and could be used in our work.
In this work, we use the sentence selection paradigm as opposed to the machine reading one. We do so because for our context of answering questions in online courses, especially content questions, short answers are generally not adequate but rather, whole contexts that could span multiple continuous sentences. Hence, extracting an answer span is not necessary and could even be inadequate and more error-prone. This point is also made by Zylich et al. [6] after assessing the misclassification in their evaluation using machine reading to answer physics course questions. We as a result focus on selecting and providing whole paragraphs as answers.
We used the course materials from our "Introduction to Programming" SuaCode course written in English and French. Each document contains text organized by paragraphs that ex- In this work, we used lesson 1, "Basic Concepts in the Processing Language" to create 2 types of question-answer pairs (1) quiz-based (n=20) using the course' quiz questions and (2) student-based (n=12) using real-world questions from students in past cohorts along with the corresponding paragraph answers in the course materials. There were 39 paragraphs and hence a random baseline of 2.6% for answer accuracy.
Our QA system's model is Sentence-BERT (SBERT), a modification of the BERT architecture with siamese and triplet network structures for generating sentence embeddings such that semantically similar sentences are close in vector space [14] . The SBERT network was shown to outperform state-ofthe-art sentence embedding methods such as BERT and Universal Sentence Encoder for semantically similarity tasks. We used the multilingual version for the French data.
We also created a basic real-time implementation of Kwame using the SBERT model fine-tuned with the student data. A user types a question, Kwame detects the language automatically and then searches through the bank of answers corresponding to that language, retrieves, and displays the top answer (Figure 1 ).
We performed various experiments to evaluate the accuracy of our proposed models and the duration to provide answers. We used 3 models. The first model -SBERT (regular) -is the normal SBERT model which has already been fine-tuned on various semantic similarity tasks. Hence, it has no coursespecific customization.
For the second model -SBERT (trained) -, we trained the SBERT model using weakly-labeled triplet sentences from the course materials in a similar manner as Reimers et al. [14] to learn the semantics of the course' text. For each paragraph, we used each sentence as an anchor, the next sentence after it in that paragraph as a positive example, and a random sentence in a random paragraph in the document as a negative example. We created a train-test split (75% : 25%) and trained the model the triplet objective in Reimers et al. [14] .
For the third model, we explored fine-tuning the SBERT model separately using the quiz QA pairs -SBERT (finetuned with Quiz) -and student QA pairs -SBERT (finetuned with Student) -using the same triplet objective to enable finding paragraphs that are semantically similar to questions.
We compared these models with TF-IDF and bi-grams as the baseline and Universal Sentence Encoder as a competitive alternative. The models are evaluated separately with the quiz and student QA pairs. To evaluate, we extracted each question's embedding and then computed the cosine similarity between the question's embedding and all the possible answers' embeddings, and returned the answer with the biggest similarity score. We then computed the accuracy of the predictions and the average duration per question. We precomputed and saved the embeddings to ensure the performance is quick.
The duration and accuracy results are shown in Table 1 . The duration results show that TF-IDF is the fastest method, followed by SBERT regular, USE, SBERT trained, and SBERT fine-tuned taking the most time of 6 seconds which is not very long nonetheless (Table 1) . These results are consistent with the result of Rana [15] who found a similar trend.
For the quiz data, TF-IDF has the worst performance of 30% and 45% for English and French respectively (but better than the random baseline of 2.6%) with USE and SBERT (Regular) having better performance.
Our SBERT model that we trained using the weaklylabeled data from the course materials did not perform better than the SBERT regular. This result might suggest that using weakly-labeled data with the triplet objective might not be adequate to improve the results. It could also be due to the small number of samples.
The SBERT models that were fine-tuned on the QA task had the highest accuracies as expected. Overall, the models performed better for the quiz data than the student data. This result is expected since the quiz's questions were created such that specific paragraphs in the text could answer them. The students' questions however were from the real-world and did not have specific answers in mind. Also, the questions were noisy with various phrases and sentences present which were not semantically related to the course text like "Any idea please?", "Good day class" etc. This realization highlights some key challenges and points to a need to develop various approaches to automatically clean the text of real-world questions before performing the semantic search.
A look at some of the questions our models got wrong provided interesting insights. The model's misclassifications sometimes happened when the model retrieved an answer that could either partially or completely also answer the question but was not the designated correct answer. In the real world, these answers by our model will be sufficient to address the questions of students. Our current evaluation approach assumes that there is only one answer and also that only one paragraph is needed to answer a question. Hence, relaxing this assumption in future evaluations would improve the recognition results. Further investigation is also needed on the best ways to combine partial answers in different paragraphs to provide one answer.
The closest work that we can compare our accuracy results to is the work of Zylich et al. [6] that we described in the related work section. For their work, they attempted to correctly answer 18 factual physics questions which is similar to our coding content questions (20 quiz type and 12 student type). Their document retrieval approach which is analogous to our QA system had a 44.4% accuracy. Our best system has an average of about 60% for the quiz and student type. Hence, our results are better than the work of Zylich et al. [6] Another work to compare with is by Rana [15] . They implemented an approach for semantic search to answer questions that individuals asked about a university, based on information on the university's website. One of their approaches is similar to ours and it involved document retrieval and paragraph selection using BERT. Their approach achieved 56% accuracy. Our results are better than that work.
In this work, we did not evaluate Kwame in a real-world context of an online course. We will do this in the future and evaluate the real-world performance using a platform like Piazza. We will do the evaluation quantitatively (accuracy, upvotes or downvotes, response rate) and qualitatively (students' feedback on Kwame).
Additionally, we used a small sample of QA pairs and sentences in this work -only the text from the first lesson.
Future work will use all lessons and compare performance across lessons.
We will explore various approaches that could potentially increase the accuracy and reduce the time to retrieve the answers by using tags as filters (e.g. #lesson1) to retrieve only the possible answers whose tag(s) match the tag of the question before performing comparison via cosine similarity.
In this work, we developed a QA system -Kwame -to answer students' questions from our online introductory coding course, SuaCode. Kwame is an SBERT-based questionanswering (QA) system that we trained and evaluated using question-answer pairs created from the course's quizzes and students' questions in past cohorts. We compared the system to TF-IDF and Universal Sentence Encoder. Our results showed that SBERT performed the worst for the duration of 6 secs per question but the best for accuracy, and fine-tuning on our course data improved the result. Nonetheless, key challenges remain such as having a higher accuracy for real-world questions.
Our long-term vision is to leverage the power of AI to democratize coding education across Africa using smartphones. Specifically, we aim to use AI to make learning more personalized for our student through providing accurate and timely answers to questions in our course cohorts currently in English and French and in the future, in various African languages such as Twi (Ghana), Yoruba (Nigeria), Swahili (Kenya, Tanzania), Zulu (South Africa). Kwame is an important part of achieving this vision by making it easy for students to get quick and accurate answers to questions in our SuaCode courses.
We are grateful to Professor Elloitt Ash for his guidance in this work.