Meeting the 2020 D uolingo Challenge on a Shoestring (2020)

Abstract: What is given below is a brief description of the two systems, called gFCONV and c-VAE, which we built in a response to the 2020 Duolingo Challenge. Both are neural models that aim at disrupting a sentence representation the encoder generates with an eye on increasing the diversity of sentences that emerge out of the process. Importantly, we decided not to turn to external sources for extra ammunition, curious to know how far we can go while confining ourselves to the data released by Duolingo. gFCONV works by taking over a pre-trained sequence model, and intercepting the output its encoder produces on its way to the decoder. c-VAE is a conditional variational auto-encoder, seeking the diversity by blurring the representation that the encoder derives. Experiments on a corpus constructed out of the public dataset from Duolingo, containing some 4 million pairs of sentences, found that gFCONV is a consistent winner over c-VAE though both suffered heavily from a low recall.

## introduction a major driver for our participating in the challenge was the curiosity to see whether recent approaches to sentence encoding with the variational auto-encoder (vae) have any relevance to the generation of diverse sentences. @xcite were the first to explore the use of vae in language generation. the work demonstrated that vae provides a continuous code space for sentences, where any randomly picked data point in the space can be decoded to yield a coherent sentence, which is significant given that the conventional rnns do not provide such a capability. the problem with vae however, is that it has no mechanism to ensure that the meaning of the source sentence is passed over to the output, which often causes a sentence to be altered, or deformed beyond recognition. while vae is a popular approach people turn to as a way to diversify sentences the model generates, no definitive answer has been found on how to control or tame what it spews out. a typical solution is to fuse a vae code with the output of a regular sentence encoder, in order to encourage the decoder to output a sentence that retains some semantic features present in the source sentence @xcite . also noteworthy is a recent work by @xcite , who building on an idea similar to vae, talk about modeling the distribution of cosine similarities between word vectors for the input and target. @xcite is something of an odd ball in the pursuit of the diversity in sentence generation. the authors argued that we could achieve the diversity by discouraging the decoder to select candidates that are similar to the input. a clear advantage they have over others is that their scheme does not involve any learning and is straightforward to implement. the idea that one can view a latent representation as a sample drawn from some probabilistic distribution inspired people to explore its potential in a wide range of tasks and domains. @xcite , while working on document modeling, suggested that we use vae as a way to get a compact representation for a document. @xcite argued for using a sample based distribution over gaussian distribution for a latent code to better express the holistic property of the source sentence. in this work, we focus on two approaches, both based on vae: one that attempts to achieve the diversity by generalizing the sentence representation produced by the encoder; and another which randomly perturbs the encoder's output during the sentence generation. we report here their respective performance on a test corpus we carved out of the official training data. for the final submission, we went along with the latter approach. 2 translation as paraphrase our effort revolved around two questions: (1) how best to incorporate likelihood scores of target translations that were provided as part of the training data, and (2) how not to rely on an external resource while building a solution. we wanted to know how far we can go using only the data made available to us at the competition, and nothing more. our answer to the first question takes advantage of the fact that a set of translations associated with each english prompt are considered an equivalence class in the sense that if we take any pair from the set, we can substitute one for the other without significantly affecting its meaning. we may take the likelihood that a human picks a particular sentence (call it x) as a good translation for some prompt (p) as the probability of its being a paraphrase of some other sentence (say y) from a group of possible translations of which x is part. the intuition here is that if x is more typical as a translation of p, it is more likely to serve as a paraphrase of whatever other way we may have to express p in the target language. following this idea, we created training data by randomly sampling a pair of sentences (both in the same language) that appear as alternate translations for a given prompt in accordance with their popular rating. for each prompt, we sampled 2,000 pairs of translations (which may include pairs consisting of identical sentences), resulting in 4,601,000 training instances (which amount to 2,300 prompts plus those provided in the development and test set) @xcite . 1 we set aside 100 ## results and discussion results are provided in as with gfconv, we ran c-vae on the test set 100 times, obtaining 100 distinct pools of candidate translations for each prompt. 6 we report in ## conclusions we discussed two approaches as a way to tackle the duolingo challenge. one is gfconv, which takes over a pre-trained sequence model, intercepts and perturbs the output its encoder produces on its way to the decoder. another is c-vae, a conditional variational auto-encoder, which seeks the diversity by blurring the representation that the encoder derives. either approach, it was found, outperformed the vanilla fconv. we also noted a large discrepancy between micro and macro f1, suggesting that the models' performance is not even and fluctuates wildly from item to item. moreover, there were some test prompts for which the models were not able to find any translations. we recognize that this is an area we need to scrutinize to further improve the performance. in the long run, it would be interesting to see if we can bring to the task recent developments in vae such as @xcite .