Multilingual Social Media Text Generation and Evaluation with Few-Shot Prompting (2022)

Abstract: This work adapts large language models to generate multilingual social media text that meets several objectives simultaneously: topic relevance, author style consistency, and reply validity. Leveraging existing online information behavior simulators, which currently only forecast activities but not content, our approach comprised of generalizable prompt formation and efficient evaluation to produce a believable, personalized, and responsive synthetic social network. According to some preliminary experiments, our multi-objective prompt formation and automatic evaluation/selection methods are able to yield a significant number of high-quality synthetic texts according to both standardized and trained metrics.

## evaluation and metrics ground truth in our tests, the ground truth used for evaluation contains real social media texts with specific attributes. there are three groups of ground truth: one group of texts manually annotated with topic labels (the test set of the manual annotation process in section 5.2.3), a group of texts by specific authors (the test set of the data used to train the authorship verification model in section 5.2.1), and a group of valid and invalid statement-response pairs (the test set of the dataset in section5.2.2). we apply the author fine-tuned language model to each generated text, and measure the cosine similarity of the vector of each text to the centroid of the vectors for the ground truth texts by the intended author. the cosine similarity is used as an indicator of the degree to which the synthetic text matches the style of the intended author. classification for the sentence-pair classification task, we utilize the simple transformers library (rajapakse, 2020). our final configuration is set up for a maximum sequence length of 512 to be trained over 2 epochs with an adam optimizer and learning rate of 4e-5. our trained model is evaluated on our evaluation dataset and returns a f1 score of .81, accuracy of .79, auroc of .84 and auprc of .77. we test several model types including bert @xcite , roberta @xcite ## limitation due to the shortage of time and computing resource, we didn't finish running some of the heavy metrics such as nubia @xcite for measuring faithfulness and bertscore @xcite for better semantic measurement. we also didn't measure the entire pipeline, e.g., comparing generated text and real text in the same simulation time frame and measuring the impact of generated and filtered text in simulation. we will address some of these limitations in our next steps. we will also try to evaluate the filtering step with manual review of filtered texts, and by testing the impact of synthetic texts on social media simulations. ## conclusions and future work we show that it is feasible to generate and evaluate synthetic social media texts which not only focus on a desired topic, but also mimic an author style and properly respond to figure 3: reply validity scores by language 10 426 figure 4: statement-reply language counts company. #pakistan#polarity #uk-indian union of petroleum, air and chem" e synthetic text examples in a user style real user bio: "news. views. analysis. conversations. india's no.1 digital news destination, world's largest-selling english newspaper -a times internet limited product" real tweet from user: "pakistan pm imran khan claims pm modi apologised for lockdown, pak media corrects him url: https://t.co/m4vaitpotghdhatzakxaig " synthetic tweets: 1. "india's prime minister narendra modi apologises to pakistan pm imran khan after a video clip of him being filmed during the crisis. the video has gained support from international organisations url: https://t ..." 2. "the government of pakistan has made efforts, i am pleased to say, to ensure that the indian government did not see the rise of pakistans internal tensions," foreign minister shah mehmood qureshi tells the new york times. "the two nations have a very good relationship." he also claims pakistan has never asked for india's "aid and support" read: https://t.co/emhxc7lr1mdnt3fjl "