Start Simple: Progressive Difficulty Multitask Learning (2024)

Abstract: The opaque nature of neural networks, often described as black boxes, poses significant challenges in understanding their learning mechanisms, which limit our ability to fully optimize and trust these models.Inspired by how humans learn, this paper proposes a novel neural network training strategy that employs multitask learning with progressive difficulty subtasks, which we believe can potentially shed light on the internal learning mechanisms of neural networks.We implemented this strategy across a range of NLP tasks, data sets, and neural network architectures and observed notable improvements in model performance.This suggests that neural networks may be able to extract common features and internalize shared representations across similar subtasks that differ in their difficulty.Analyzing this strategy could lead us to more interpretable and robust neural networks, enhancing both their performance and our understanding of their nature.

## introduction how do neural networks learn? this question remains a complex and intriguing area in the field. despite the substantial advancements in the application and performance of neural networks nowadays, a comprehensive understanding of the logical connection between their internal configurations and external behaviors is still developing (a.o.: @xcite @xcite . what we might want to seek intuition from, however, is how humans learn. one of the key observations in this heavily researched field (e.g., @xcite is that people's learning process can be facilitated by starting from understanding simple notions or from solving toy problems. inspired by this observation, we propose a multitask learning (mtl) strategy @xcite that trains a neural network using subtasks of progressive difficulty. we apply this strategy to train neural networks across different nlp tasks: sentiment analysis, text classification, unit segmentation, and syllogistic reasoning. we also experiment with training different types of neural networks using this strategy, including a generative pretrained transformer (gpt) in the sense of zhao et al. 2023 as we recognize the growing interest in large language models (llms). we expect that progressive difficulty mtl will enhance the performance of neural networks. by proposing and testing this mtl strategy, we hope to better understand the behavior of neural networks and establish links between their internal learning mechanisms and those of humans. ## background in his review of previous work on mtl, @xcite introduced mtl as "an inductive transfer mechanism" that "improves generalization by leveraging the domain-specific information contained in the training signals of related tasks" (page 41). the motivation behind mtl is to divide and conquer: we break large problems into small ones (cf. @xcite . subsequent work on mtl (a.o.: @xcite @xcite also showed that similar tasks trained simultaneously can benefit from each other in terms of convergence time and overall accuracy. in particular, @xcite applied mtl to sentiment analysis and effectively improved the overall accuracy of variational autoencoders. in this paper, we use progressive difficulty subtasks for mtl, and we broaden the empirical ground of previous work beyond sentiment analysis to encompass other nlp tasks like text classification, unit segmentation, and syllogistic reasoning. as for backbone models, among many others, @xcite and @xcite applied neural networks to hierarchical text classification. the former offered a locally connected network approach, in which the prediction scores for the classi-fication of the current label level are used as input to the classification of the next label level. the latter offered a convolutional neural network (cnn) approach, in which hierarchical dependencies among the labels are provided to a classifier with recursive regularization @xcite . in this paper, we extend the experiments to additional types of neural networks, including the fully connected neural network (fcnn), the long short-term memory network (lstm), and transformers. additionally, @xcite employed deep cnns for text classification, adopting ideas from vgg @xcite and resnet @xcite , using a small size of kernels for convolutional filters and adding residual connections to address degradation problems. with a similar motivation, @xcite employed deep lstms, which proved to be empirically remarkable on feature extraction. in this paper, as an extension of our study, we implement their deep neural networks and test how adding more layers to the model may affect the performance of progressive difficulty mtl. as another extension, we test how @xcite transfer learning (tl) may affect the strategy we propose. finally, in the field of llms, numerous recent studies (a.o.: @xcite @xcite @xcite ) discussed a chain-of-though (cot) strategy, which involves prompting the model to generate intermediate steps before arriving at a final answer. this strategy substantially enhanced model performance, and we believe that this strategy aligns perfectly with the strategy we propose in this paper: they "start simple." in this paper, we apply progressive cot to syllogistic reasoning. specifically, from an information flow perspective, we let the model be informed about the information of the main task using manually designed progressive difficulty subtasks. ## methods we conduct two experiments. the first experiment works on sentiment analysis, text classification, and unit segmentation. the second experiment works on syllogistic reasoning. ## results in experiment 1, we used a batch size of 64 for coronavirus tweets and amazon product reviews, 128 for dbpedia, and 4 for argumentative essays. we used a mini-batch sgd optimizer with a momentum of .9 and a fixed learning rate of .01. the models converged in around 15 epochs for coronavirus tweets and dbpedia, 30 epochs for amazon product reviews, and 5 epochs for argumentative essays. in the meantime, baseline models were trained without mtl. as an extension to experiment 1, we tried to stack more layers inside the feature extractor used in the sentiment analysis and text classification tasks and see if they can boost the performance of mtl with progressive difficulty subtasks. as mentioned in the background section, for cnn, we implemented the deep neural network discussed in @xcite , which contained 17 convolutional layers followed by max pooling and fully connected layers. additionally, we implemented the deep lstm discussed in @xcite with 8 lstm layers, which added residual connections between every two layers in the middle six layers. last but not least, we conducted experiment 2 using gpt 3.5. the results are in ## discussion as can be seen in results in ## conclusion inspired by how humans learn, we proposed an mtl strategy using progressive difficulty subtasks and discovered that this strategy improved the performance of various neural networks on various nlp tasks. we also found out that our strategy worked better with neural networks having a larger field of view and with subtasks sharing a common, focused goal. we stipulate that the internal learning mechanisms of neural networks are akin to human learning in the sense that it can apply its knowledge from simpler tasks to more complex tasks.