Generating Slogans with Linguistic Features using Sequence-to-Sequence Transformer (2021)

Abstract: Previous work generating slogans depended on templates or summaries of company descriptions, making it difficult to generate slogans with linguistic features. We present LexPOS, a sequence-to-sequence transformer model that generates slogans given phonetic and structural information. Our model searches for phonetically similar words given user keywords. Both the sound-alike words and user keywords become lexical constraints for generation. For structural repetition, we use POS constraints. Users can specify any repeated phrase structure by POS tags. Our model-generated slogans are more relevant to the original slogans than those of baseline models. They also show phonetic and structural repetition during inference, representative features of memorable slogans.

## introduction advertising slogans share many linguistic features, such as phonetic or structural repetition @xcite ). these factors make slogans more memorable @xcite ). however, most previous works on slogan generation depended on templates or summaries of company descriptions, making it difficult to generate slogans with linguistic features. we present lexpos, a sequence-to-sequence (seq2seq) transformer model with an additional pos encoder. it models the phonetic and structural repetition in slogans, using lexical and pos constraints. when given keywords and pos tags of the desired output structure as input, the model finds words that sound and mean similar to the user keywords. the model-generated slogans include both the user keywords and one sound-alike word. they also reflect the pos constraints. for instance, if a user inputs the word 'cake' and ['verb', 'det', 'noun', 'punct', 'verb', 'det', 'noun', 'punct'], the output could be 'bake a cake, bake a smile'. it includes the word 'cake' and its sound-alike word 'bake' and has repeated verb phrases. the source code, pretrained weights, and data are available online foot_0 . this paper primarily makes the following contributions: ## previous work most of the previous work in slogan generation focused on modifying templates. brainsup, proposed by @xcite , is the first study to generate customized slogans with lexical, emotional, and domain constraints. brainsup utilizes morpho-syntactic patterns mined from corpus as templates. it first selects the most compatible template and fills the empty slots in the template according to user specifications. before returning the results, it evaluates the candidate slogans with various metrics, including phonetic repetition. however, it can only determine whether the same phonetic features were used or not. ## model our model first forms the lexical constraints. during training, it uses the given lexical constraints as it is. during inference, it searches for soundalike words of user keywords. we use the phonetic vector representation proposed by @xcite . the phonetic vector uses interleaved phonetic feature bigrams extracted from phonetic transcriptions and it covers all the words in cmu pronouncing dictionary 2 . the model also considers the semantic similarity of sound-alike words, to improve the naturalness of the outputs. we use pretrained glove embeddings @xcite ) for semantic similarity. after we compute cosine similarity to select the top 100 phonetically similar words, we sort them in semantic similarity. we exclude words that are not present in glove embeddings @xcite or bart tokenizer vocabulary, not to use 2 http://www.speech.cs.cmu.edu/cgibin/cmudict unfamiliar words. we select the first three words to each form the lexical constraints, together with user keywords. unlike lexical constraints, pos constraints don't need further processing during training and inference. pos constraints during inference can be manually specified or popular pos structures from data would be recommended. after processing the lexical constraints, the transformer architecture @xcite comes in. the transformer architecture has achieved state-of-the-art results on various natural language processing tasks. we apply a transformer-based sequence-to-sequence model because we need encoders for constraints and decoders for generation. to leverage the power of pretrained transformers, we utilized the pretrained weights of bart and t5 @xcite released by huggingface foot_1 . we choose bart and t5 because both models were pretrained by denoising consecutive spans of corrupted tokens, meaning they can generate natural sentences using lexical constraints. the only architectural difference is that our model has an additional encoder. one encoder encodes the lexical constraints, and the other encodes the pos constraints. the weights of the pos encoder are randomly initialized, and the vocabulary size of the pos encoder is limited to 20. the vocabulary includes the spacy foot_2 pos tags @xmath0, 
, <@xmath1. to incorporate the pos constraints into the rest of the model, the last hidden states @xmath2. we choose the last hidden state @xmath3. these two hidden states are summed and given to the decoder. then, the decoder generates slogans with the given lexical and pos constraints. figure 1 presents the architecture of our proposed model. ## data our training objective is to implement lexical and pos constraints. the desired model-generated slogans should follow the lexical constraints and the pos structural constraints. we crawl 30,759 unique slogans from online slogan databases such as textart.ru foot_3 , slogans hub foot_4 , slogan list 7 , think slogans 8 , and slogans point 9 . unlike previous works focusing on commercial slogans, our dataset covers both commercial and public slogans. public slogans include slogans for health, women's rights, the environment, and more. 45.43% of our slogans are commercial, 54.56% are public. company names in commercial slogans are delexicalized using a custom special @xmath4, following @xcite . we reserve 20% of the data for validation. the lexical inputs are lexical constraints surrounded @xmath5. just like the pretraining method of bart and t5, our model predicts consecutive spans @xmath6. the lexical constraints are limited to verbs, nouns, proper nouns, and adjectives. we extract them from original slogans using spacy. then, we randomly delete lexical constraints, when there are 5 or more of them to keep the average ratio of the number of lexical constraints to the total number of words in the original slogan below 50%. the average ratio is 41.90%. unlike the pretraining method of bart and t5, we shuffle the lexical constraints to make the model predict natural ordering. if we don't shuffle them, we need to permutate lexical constraints during inference. for instance, if the user keyword is 'cake' and its sound-alike word is 'bake', we need both '<@xmath7. the number of permutations would increase dramatically as the number of user keywords increase. to address this issue, we shuffle the lexical constraints. the pos inputs are pos constraints themselves. we use spacy pos tagging results of the original slogan as pos inputs. ## experiments following previous work, we conduct a quantitative evaluation using rouge @xcite ) f1 scores and compare our model with the original sequence-to-sequence model baselines. we also compute the included lexical constraints rates and pos f1 scores to check how well the given constraints are applied. the included lexical constraints rates are the rates of the lexical constraints included in model-generated slogans. pos f1 scores are computed by comparing the pos inputs and pos tagging results of modelgenerated slogans. the model-generated slogans include both the user keywords and one selected sound-alike word, fully reflecting users' intentions. the results also show phonetic and structural repetition, representative features of memorable slogans. ## conclusion in this work, we generate slogans with phonetic and structural repetition using lexpos model, a transformer-based sequence-to-sequence model with an additional pos encoder. it generates slogans using sound-alike words given user keywords. the model-generated slogans also follow structural constraints thanks to the pos encoder. to our knowledge, it is the first model to generate slogans without templates, taking linguistic features into account. future work should implement other linguistic features shown in slogans.