All You Need is Attention: Lightweight Attention-based Data Augmentation for Text Classification (2024)

Abstract: This paper introduces LADAM, a novel method for enhancing the performance of text classification tasks. LADAM employs attention mechanisms to exchange semantically similar words between sentences. This approach generates a greater diversity of synthetic sentences compared to simpler operations like random insertions, while maintaining the context of the original sentences. Additionally, LADAM is an easy-to-use, lightweight technique that does not require external datasets or large language models. Our experimental results across five datasets demonstrate that LADAM consistently outperforms baseline methods across diverse text classification conditions.

## introduction text classification is a prominent research area in natural language processing (nlp), where performance heavily relies on dataset quality. with the recent emergence of large language models (llms), there has naturally been an increasing need for substantial datasets. for example, gpt-3 @xcite utilizes 175 billion parameters, emphasizing the crucial requirement for extensive dataset training. in this context, data augmentation (da) plays a critical role in improving text classification tasks. previous studies on da have employed simple noising operations such as random insertions and deletions of words @xcite . these techniques often struggle to generate diverse contexts in augmented texts and can inadvertently alter the original meaning due to their random nature. alternatively, model-based methods @xcite have been employed to replace words with synonyms, aiming to maintain the original context of sentences. however, the effectiveness of this approach heavily relies on the quality of the pre-trained dataset used * corresponding author for the model and may be challenging to implement due to dependencies on external resources like heavy language models. interpolation is another da method @xcite @xcite . however, when applied directly to raw data such as words, this approach tends to produce nonsensical sentences that can alter labels or meanings of the original sentences @xcite . to address these limitations, we propose ladam (lightweight attention-based data augmentation method), which utilizes attention mechanisms to identify synonyms. ladam generates diverse new sentences by replacing words with synonyms while preserving the original meaning. it is lightweight, as it operates without external datasets or language models, ensuring ease of use. our evaluation results demonstrate it' s effectiveness in text classification tasks, and we have made ladam publicly available foot_0 . ## ladam ladam generates synthetic texts while preserving the original sentence's meaning. key differences from existing methods include: 1 minimizing the scope of word replacements, 2 directly applying to raw data, and 3 being the first approach to use attention scores to preserve the original sentence's context. figure 1 illustrates the overall architecture of ladam, which consists of two phases: contextbased sentence selection and attention-based word selection. in the context-based sentence selection phase, a random target sentence t is selected for augmentation, and another random sentence a with the same label l is chosen as the assistant sentence. the two sentences are tokenized into words, and each word is vectorized using bert word embedding @xcite employs the attention mechanism in transformer models @xcite , deriving attention scores using scaled dot-product for each word in sentences t and a. the attention scores are extracted via sof tmax( qk t √ d k ## results we compared ladam with baseline methods under various conditions, repeating each experiment five times and reporting the average results. all experiments were conducted on a geforce rtx 3060 gpu with 12 gb of memory. ## conclusion we propose ladam, a lightweight text data augmentation approach designed to generate diverse augmented data while preserving the original sentence context. our experiments show that ladam outperforms baselines on both original and biased datasets, demonstrating it's effectiveness in text classification tasks. future work will explore whether alternative attention scoring functions, such as concat and bahdanau, can enhance ladam's performance.