Assessing the Helpfulness of Learning Materials with Inference-Based Learner-Like Agent (2020)

Abstract: Many English-as-a-second language learners have trouble using near-synonym words (e.g., small vs.little; briefly vs.shortly) correctly, and often look for example sentences to learn how two nearly synonymous terms differ. Prior work uses hand-crafted scores to recommend sentences but has difficulty in adopting such scores to all the near-synonyms as near-synonyms differ in various ways. We notice that the helpfulness of the learning material would reflect on the learners’ performance. Thus, we propose the inference-based learner-like agent to mimic learner behavior and identify good learning materials by examining the agent’s performance. To enable the agent to behave like a learner, we leverage entailment modeling’s capability of inferring answers from the provided materials. Experimental results show that the proposed agent is equipped with good learner-like behavior to achieve the best performance in both fill-in-the-blank (FITB) and good example sentence selection tasks. We further conduct a classroom user study with college ESL learners. The results of the user study show that the proposed agent can find out example sentences that help students learn more easily and efficiently. Compared to other models, the proposed agent improves the score of more than 17% of students after learning.

## introduction many english-as-a-second-language (esl) learners have trouble using near-synonyms correctly @xcite . "nearsynonym" refers to a word whose meaning is similar but not identical to that of another word, for instance, establish and construct. an experience common to many esl learners is looking for example sentences to learn how two nearly synonymous words differ @xcite . to facilitate the learner's learning process, our focus figure 1 : the learner-like agent mimics learners' behavior of performing well when learning from good material and vice versa. we utilize such a behavior to find out helpful learning materials. in previous work, researchers develop linguistic search engines, such as linggle @xcite and netspeak foot_0 , to allow users to query english words in terms of n-gram frequency. however, these tools can only help people investigate the difference, where learners are required to make assumptions toward the subtlety and verify them with the tools, but can not tell the difference proactively. other work attempts to automatically retrieve example sentences for dictionary entries @xcite ; however, finding clarifying examples for near-synonyms is not the goal of such work. in a rare exception, @xcite retrieve useful examples for near-synonyms by defining a clarification score for a given english sentence and using it to recommend sentences. however, the sentence selection process depends on handcrafted scoring functions that are unlikely to work well for all nearsynonym sets. for example, the difference between refuse and reject is their grammatical usages where we would use "refuse to verb" but not "reject to verb"; such a rule, yet, is not applicable for delay and postpone as they differ in sentiment where delay expresses more negative feeling. though @xcite propose two different models to handle these two cases respectively, there is no clear way to automatically detect which model we should use for an arbitrary near-synonym set. in the search for a better solution, we noted that esl learners learn better with useful learning materials-as evidenced by their exam scoreswhereas bad materials cause confusion. such behavior can be used to assess the usefulness of example sentences as shown in figure 1 . therefore, we propose a learner-like agent which mimics human learning behavior to enable the ability to select good example sentences. this task concerns the ability to answer questions according to the example sentences for learning. as such, we transform this research problem to an entailment problem, where the model needs to decide whether the provided example sentence can entail the question or not. moreover, to encourage learner-like behavior, we propose perturbing instances for model training by swapping the target confusing word to its nearsynonyms. we conduct a lexical choice experiment to show that the proposed entailment modeling can distinguish the difference of near-synonyms. a behavior check experiment is used to illustrate that perturbing instances do encourage learner-like behavior, that is inferring answers from the provided materials. in addition, we conduct a sentence selection experiment to show that such learner-like behavior can be used for identifying helpfulness materials. last, we conduct a user study to analyze near-synonym learning effectiveness when deploying the proposed agent on students. our contributions are three-fold. we (i) propose a learner-like agent which perturbs instances to effectively model learner behavior, (ii) use inferencebased entailment modeling instead of context modeling to discern nuances between near-synonyms, and (iii) construct the first dataset of helpful example sentences for esl learners. 2 2 related works this task is related to (i) learning material generation, (ii) near-synonyms disambiguation, and (iii) natural language inference. near-synonyms disambiguation. unlike the language modeling task that aims at predicting the next word given the context, near-synonyms disambiguation focuses on differentiating the subtlety of the near-synonyms. @xcite first introduced a lexical co-occurrence network with secondorder co-occurrence for near-synonym disambiguation. edmonds also suggested a fill-in-the-blank (fitb) task, providing a benchmark for evaluating lexical choice performance on near-synonyms. islam and inkpen (2010) used the google 5-gram dataset to distinguish near-synonyms using language modeling techniques. @xcite encoded words into vectors in latent semantic space and applied a machine learning model to learn the difference. @xcite applied bilstm and gmm models to learn the subtle context distribution. recently, bert @xcite brought a big success in nearly all the natural language processing tasks. though bert is not designed to differentiate near-synonyms, its powerful learning capability could be used to understand the subtlety lies in the near-synonyms. in this paper, our models are all designed on top of the pre-trained bert model. ## method in this paper, we use learner-like agent to refer to a model that answers questions given examples. the goal of the learner-like agent is to answer fill-in-theblank questions on near-synonyms selection. however, instead of answering the question from the agent's prior knowledge, the agent needs to answer the question using the information from the given examples. that is, if the given examples provide incorrect information, the agent should then come up with the wrong answer. this process is to simulate the learner behavior illustrated in the figure 1 . since the model is required to infer the answer, we further formulate it as an entailment modeling problem to enable model's capability of inference. in this section, we (i) define the proposed learner-like agent, (ii) describe how to formulate it as an entailment modeling problem, and (iii) introduce the perturbed instances to further enhance the agent's learner behavior. ## experiments we conducted three experiments: lexical choice, behavior check, and sentence selection. the lexical choice task assesses whether the model differentiates confusing words, the behavior check measures whether the model responds to the quality of learning material as learners do, and sentence selection evaluates the model's ability to explore useful example sentences. ## learner study we conducted a user study to see the effect of learning on example sentences selected by emla, cmla, and a random baseline. in this learner study, a total of 29 chinese-speaking college freshmen majored in english were recruited. all the participants were aged between 18 and 19. a proficiency test @xcite was given before the study to identify their english level for further analysis. ## conclusion we introduce the learner-like agent, in particular emla, which differentiates the helpfulness of learning materials using inference. entailment modeling, unlike common context-based nearsynonymous word disambiguation, makes inferences to learn the relationship between the example sentences and the question, similar to human behavior. context modeling in the learnerlike agent relies upon additional perturbed examples to mimic human behavior, whereas emla already has this ability. the agent can be used to evaluate the helpfulness of learning materials, or-more interestingly-to select the best materials from a large candidate pool. we select good example sentences in practice, which confirms the usefulness of modeling learner behavior. using the emla learner-like agent, we find more helpful learning material for learners, as demonstrated by the learner study. these demonstrate the usefulness of modeling learner behavior using an inference approach. in the future, we would like to explore if the learner-like agent can be extended to materials and data beyond the example sentences for near-synonyms.