A c K nowledge: Acquired Knowledge Representation by Small Language Model Without Pre-training (2024)

Abstract: Large language models (LLMs) are pre-trained on enormous amounts of text data and show acclaimed success in knowledge representation. However, there are two bottlenecks with this approach. (1) Pre-training data cannot be regularly updated once the models are deployed, and it is not very fruitful if the model cannot represent updated knowledge. (2) The consistently increasing size and computational resources make it difficult for non-commercial and individual researchers to fine-tune and scale these language models. Major LLMs with external knowledge are also proprietary. In this paper, we propose AcKnowledge, a framework wrapped around a small, non-pre-trained language model for an open-domain question-answering (QA) experiment. AcKnowledge learns relevant knowledge from the internet via meta-learning based on user questions, and re-learns from user feedback if knowledge is misrepresented. Our efficient knowledge representation framework avoids pre-training overhead while enabling updated information. Benchmarking shows competitive performance against similarly sized state-of-the-art (SoTA) LLMs on gold standard QA datasets, demonstrating the potential of integrating internet search and user feedback for improved performance and generalizability.

## introduction the excellent performance of large language models (llms) in various natural language processing (nlp) tasks can be mainly attributed to their ability to capture and represent knowledge from extensive pre-training on massive text corpora @xcite . however, the outdated nature of data for pre-trained knowledge can limit their adaptability to new information or recent events @xcite . traditional methods for updating llms, such as continuous pretraining on the latest data or fine-tuning, are computationally expensive and time-consuming @xcite . to address these limitations, we propose ac-knowledge, a novel framework that equips a small language model (slm) with the dynamic capability to acquire and represent knowledge without conventional pre-training. our approach exploits real-time web search and meta-learning @xcite to enable an slm to learn new information efficiently. upon receiving a user's question, the topics are extracted using latent dirichlet allocation (lda) @xcite . these topics are then transmitted to the language model (lm), which uses these topics as keywords to perform a targeted online search and uses meta-learning @xcite ) to acquire relevant knowledge. the acquired knowledge is then summarized @xcite and presented to the user as a concise answer. acknowledge also integrates a user feedback mechanism to ensure authenticity and reliability for generated answers. negative feedback triggers a new search iteration to find more accurate information, while positive feedback fortifies the learning. user feedback plays a vital role in acknowledge's learning loop in augmenting the acquired knowledge for increasingly more correct and factual answers. we evaluate acknowledge's performance by benchmarking it against similarly sized lms in open-domain qa tasks, demonstrating competitive results despite the absence of traditional pretraining. the main contributions of this paper are: ## relevant works knowledge representation is indispensable for nlp systems to understand meaning and perform reasoning. the statistical approaches in the early last decade like word embeddings @xcite learned vector representations but lacked explicit knowledge modeling. further advances in integrated neural networks with symbolic knowledge graphs and ontologies through techniques like graph convolutional networks @xcite . hybrid neuro-symbolic methods show promise in injecting knowledge into large pre-trained language models like roberta @xcite to improve common sense reasoning @xcite and factual grounding @xcite . multimodal learning from transformer architecture has also been in research focus @xcite . key challenges in representing knowledge often include effective representation and contextsensitivity to the core topic @xcite , performing reasoning over learned representations @xcite , and generating logical forms @xcite . promising directions also involve meta-learning for fast knowledge adaptation @xcite and graph embedding methods for knowledge representation @xcite . ## system framework our fundamental objective is to develop acknowledge with the ability to dynamically retrieve and adapt relevant knowledge seamlessly from the internet. the proposed system comprises several key components that work in tandem to facilitate this process. the overview of our system is illustrated in figure 2 . the primary components of the framework are discussed in the following sections. ## experiments to evaluate acknowledge's performance on goldstandard corpora, we use two widely-used opendomain qa datasets; stanford question answering dataset (squad 2.0) @xcite and natural questions (nq) @xcite , squad 2.0 integrates approximately 50,000 adversarial-designed unanswerable questions to mimic responsive questions. for good performance on squad 2.0, ideally, systems should recognize when the text does not support a response and refrain from responding. nq is a large-scale dataset with more than 300,000 question-answer pairs based on real-world google search queries. it includes a diverse range of topics and question types, with an average of 16.5 tokens per question and 77 tokens per answer. we preprocess the datasets by tokenizing the text using the wordpiece tokenizer @xcite and converting the tokens to their corresponding embeddings using word2vec @xcite . the preprocessed datasets are then split into training, validation, and test sets with a ratio of 80%, 10%, and 10%, respectively. ## conclusion we introduced acknowledge, a framework that can search the internet for updated answers to user questions, learn from the search results using metalearning, and assimilate user feedback to improve performance. our proposed approach outperforms various sota and baseline models in standard qa evaluation metrics. our approach has several potentials for language model applications. firstly, it demonstrates the benefit of integrating internet search and meta-learning in language models to improve their answering ability. secondly, it can also answer complex questions that require access to up-to-date and diverse information sources. third, it can be used to develop scalable language models that can learn from user feedback to improve their performance and adapt to user preferences. there are prominent future research directions for our work. we aim to explore the prospect of scalability and robustness in larger and real-world deployable scenarios. this experiment can pave the way for developing more knowledgeable language models that can assist users in various tasks and scenarios. dimensionality of the hidden states and embeddings used in the model. the total number of parameters in the model is calculated to be around 124.6 million, or approximately 125 million. in addition, we implement layer normalization instead of batch normalization for our model.