Formalizing content creation and evaluation methods for AI -generated social media content (2023)

Abstract: This study explores the use of large language models (LLMs), such as ChatGPT and GPT-4, in creating high-quality text-based social media content for businesses on LinkedIn. We introduce a novel architecture incorporating external knowledge bases and a multi-step writing approach, which extracts facts from company websites to form a knowledge graph. Our method’s efficacy is assessed using the “Long-LinkedIn” evaluation dataset designed for long-form post generation. Results indicate that our iterative refinement significantly improves content quality. However, knowledge-enhanced prompts occasionally reduced quality due to potential formulation issues. LLM-based evaluations, particularly using ChatGPT, showcased potential as a less resource-intensive alternative to human assessments, with a notable alignment between the two evaluation techniques.

## introduction marketing is a crucial but resource-intensive facet of running a thriving business. recent advances in natural language processing have revolutionized this field by automating various marketing functions. several businesses now offer aiassisted copywriting services that leverage chat-gpt, gpt-4, and related models @xcite @xcite . despite the widespread adoption of such tools by millions of users worldwide, the academic literature on their efficacy and impact on content quality remains scarce. in light of this gap, we seek to address the following question: how can large language models (llms) be utilized to produce high-quality social media content? to narrow our research scope, we focus on textbased social media posts by businesses, excluding images, videos, polls, and other non-text content. with this focus, the social media linkedin 1 serves 1 www.linkedin.com as a suitable platform due to its highly text-based nature. in this work we introduce a novel architecture for social media content generation using large language models, external knowledge bases and a multi-step writing approach. our approach involves automatically extracting facts from a company website and constructing a knowledge graph (see section 2) the primary contributions of this work include: we apply our architecture to the long-linkedin task and assess its effectiveness using an ablation study and chatgpt (openai, 2022) (see section 4). to evaluate our approach, we summarize the topics of publicly available posts and generate new posts on the same subject for the respective companies, using models with varying levels of ablation. human evaluators then rank the generated posts based on several factors, allowing us to gauge the impact of our architecture on the quality of social media content. ## architecture our architecture requires five inputs to craft a social media post for a company: we use the brief description for basic context, then enhance it with pertinent facts based on the post topic. this tailored approach ensures content is engaging and informative. the writing process undergoes multiple phases to boost quality. we'll now delve into our architectural design in three stages: baseline, knowledge graph enriched, and multi-step, with each stage building upon the last ## long-linkedin task to assess our approach, we use public linkedin posts for human comparison. different postcomparisons pose challenges due to differences in subject matter, audience, and authorship. the long-linkedin task curtails these disparities by producing synthetic posts on identical topics from the same company as reference human posts. we auto-generate topics for scraped posts using prompt a.4, which guides post-generation. to ensure consistency, we factor in the original post's word count and source company descriptions from their linkedin pages. utilizing human posts as the foundation for generating artificial posts presents several advantages. firstly, it ensures that the generated content is written on topics that are relevant to the company's focus and brand. secondly, it simplifies the comparison process between original and generated posts. however, this approach also imposes limitations on the source post's length. if the original post is too short, the 1-2 topic sentences may encompass most of the post's nuances, essentially leaking the original post in the prompt and rendering the writing task redundant. there's concern that lms trained on vast internet data might unintentionally reference test or development sets during training @xcite . to counter this, we'll gather posts from: a. smes and startups, likely not part of the llm training set. b. large corporations like google and amazon, probably in the training set. this dual approach aims to: for this project, we deploy the chatgpt api (ope-nai, 2022), a product of the gpt-3. ## company knowledge graphs several open-source knowledge graphs, such as dbpedia, freebase, and wikidata @xcite @xcite , are available for use. however, these knowledge bases are not suitable for our purposes, as the companies in set a are not wellrepresented in them. as a result, we will generate custom knowledge graphs for each of these companies using information from their official websites. as a result, every company in our dataset has a website, ensuring a leveled playing field. that being said, we do hypothesize, that companies in set b, which tend to be older and more financially established, may have more informative websites, potentially affecting the quality of the generated knowledge graphs. the impact of the knowledge graphs' comprehensiveness on content quality is further discussed in section 8. ## evaluation automated metrics like bleu, meteor, and bertscore evaluate the quality of language generation systems by measuring similarity to a reference text @xcite @xcite . a higher score indicates that our system closely matches the original post. yet, solely using these metrics isn't apt for our purpose since multiple versions can convey the same message with equal quality. hence, we've incorporated human feedback for evaluation. however, human evaluations come with challenges like inconsistency, high costs, and slow results @xcite . a recent study, chiang and yi lee (2023), suggests that llms might offer a quicker and more reliable evaluation method. based on their findings, we're integrating chatgpt into our evaluation to see how its results align with human evaluations and the mentioned study. ## results the kruskal-wallis test confirms the overall statistical significance for both set a and b separately with @xmath0.006. the dunn posthoc tests confirm all the pairwise differences in set b with @xmath1.005 except for the comparisons between human and multistep, as well as multistep+kg and baselinethese represent the two best-performing and worstperforming conditions, respectively. in set a the only significant pairwise differences are between multistep and multistep + kg and multipstep and baseline with @xmath2.02. for a comprehensive overview of the dunn posthoc test results, please refer to appendix d. significance in itself however is not particularly informative about how much better one system is compared to another. this information is rather captured by effect size estimates (van der @xcite . as trueskill models each condition's skill value as n µ, σ 2 , this allows us to get a sense of the effect size through cohen's d @xcite ). cohen's d expresses the magnitude of the differ-ence between two groups and is calculated as foot_2 : in in comparison, we calculated kendall's τ between the authors using a random subset of 50 posts, which resulted in an iaa of 0.25 and a p-value of less than 0.005. this indicates a moderate correlation that is statistically significant. the reasonable iaa score among the authors suggests that the annotation guidelines were clear and well-defined, and increase the likelihood that our observations can be generalized to a larger population. multistep + kg multistep baseline human multistep + kg 0 3.83 0.36 4.07 multistep 0 4.20 0.27 baseline 0 4.45 human 0 ## discussion and error analysis in line with our hypothesis, we noticed a marked improvement in the multistep writing approach, with an effect size of around four standard deviations compared to the baseline, as detailed in section 3. however, integrating more knowledge led to a performance dip. in appendix f, we provide an example where adding knowledge adversely affects the post's content. the example indicates a tendency for the content to focus on new facts at the expense of the original topic, suggesting an over-reliance on potentially misleading facts. we think this largely accounts for the performance variation. to address this, two strategies emerge: 1. reducing unrelated facts in the prompts, and 2. improving the model's capability to screen out irrelevant information. the prevalence of unrelated facts might be due to our search method or the limited size of our company knowledge graph, which had roughly 100 triples. a more extensive knowledge graph might enhance quality, but that was beyond our project's scope. the prompts might also fail to signal that the facts aren't always relevant. we anticipate that a more advanced llm, like gpt-4, could overcome some of these challenges due to better reasoning and task adherence (openai, 2023). regarding our second hypothesis -that llm posts would outshine human ones -results are inconclusive. human posts in set b seem to compete more effectively against our methods than those in set a. we suspect that increased sampling might reveal a significant difference, mainly because set b's content, produced by well-established companies, seems superior. our final hypothesis, about the negligible impact of knowledge graph enrichment on set b performance, remains unconfirmed. although figure 2 suggests the baseline method performed better on set a, this difference could be due to a drop in human post performance. using likert scales @xcite might have offered clearer insights, but we chose not to, due to potential interpretation disparities. chatgpt's evaluation supports our initial hypothesis, showing an effect size difference of 7.38 between the multistep and baseline approaches. however, the preference of chatgpt for generated over human content isn't clear-cut. chiang and yi lee (2023) tested chatgpt's evaluations, but it's uncertain if chatgpt shows bias towards its output. interestingly, chatgpt favored the multistep + kg approach more than human evaluators did. it's plausible that human evaluators were more critical, possibly because they factored in topic relevance more. this could explain why chatgpt rated both multistep methods similarly. ## future work further research is needed to assess the components of our multistep approach and their impact on quality. the benefits of multistep prompting lead to questions about performance gains from other prompting types. further exploration into advanced prompting for creative writing is necessary. a recent paper intro-duced the tree of thoughts framework @xcite , generalizing the chain of thought method for language model prompting. this allows for the study of coherent text units used in problemsolving. investigating high-level planning for creative writing, such as social media posts, is a promising area. the llm could potentially explore and generate posts based on selected outlines. pairing this with the critic framework @xcite ) might enable the model to query a knowledge base during outlining, guiding it towards topics with rich, accurate information. ## conclusion in conclusion, we presented a novel architecture for social media content creation that utilizes large language models, external knowledge bases, and a multi-phase writing process. this method generates content by extracting data from a company's website, forming a knowledge graph, and creating detailed prompts for language models through iterative refinement. to evaluate, we initiated the long-linkedin task for long-form posts, targeting content similar to actual linkedin posts in topic and style. human evaluators compared our system's content to genuine human posts and two ablated versions. the results confirmed our iterative method enhanced content quality, with significant effect sizes. however, it was unclear if our posts outperformed human references. sometimes, knowledgeenhanced prompts reduced quality due to knowledge graph and prompt formulation issues. in sets a and b, quality variations in human posts were observed, but the impact of knowledgeenhanced prompts was indeterminate. using chatgpt for evaluation, we found a mild correlation with human assessments but with high statistical significance, suggesting chatgpt's potential as a cost-effective evaluation tool. notably, chatgpt found human posts comparable to our multi-step approach.