A Gentle Introduction to Deep Nets and Opportunities for the Future (2022)

Abstract: The first half of this tutorial will make deep nets more accessible to a broader audience, following “Deep Nets for Poets” and “A Gentle Introduction to Fine-Tuning.” We will also introduce GFT (general fine tuning), a little language for fine tuning deep nets with short (one line) programs that are as easy to code as regression in statistics packages such as R using glm (general linear models). Based on the success of these methods on a number of benchmarks, one might come away with the impression that deep nets are all we need. However, we believe the glass is half-full: while there is much that can be done with deep nets, there is always more to do. The second half of this tutorial will discuss some of these opportunities.

## introduction this tutorial is split into two parts: a glass is half-full: deep nets can do much b glass is half-empty: there is always more to do part a will make deep nets more accessible to a broader audience @xcite by introducing gft (general fine-tuning), a new "little language" 1 for deep nets that is similar to glm (general linear models) in the statistics package r. 2 gft code will be posted on the tutorial website. 3 2 part a: glass is half-full ## part b: opportunities for improvement language models (lms) are based on @xcite : "you shall know a word by the company it keeps" and zellig harris's (1954) "distributional hypothesis." by construction, this approach learns many aspects of language, some more desirable (fluency, collocations, word patterns) and some less desirable (biases @xcite ). however, there are many aspects that are not learned: truth (logical form, temporal/spatial logic and possible worlds), meaning, purpose (planning @xcite , discourse structure) and commonsense knowledge (time and space). these topics have been studied for decades in ai and knowledge representation and for centuries in linguistics and philosophy. ## conclusions: some paths forward some of these opportunities can be addressed by relatively easy patches to firth-based methods. for example, biases can be mitigated in the short term by vetting the training corpus @xcite . similarly, penalty terms can be added to the objective function to discourage hallucinations @xcite . fine-tuning on a corpus of commonsense knowledge can help with violations of commonsense @xcite . in the long term, it may be helpful to consider more radical alternatives @xcite . part a described some recent advances that have been remarkably successful, though to make long term advances beyond that, it may be necessary to take advantage of more diverse interdisciplinary approaches that include firth-based methods, as well as decades of work on knowledge representation in ai, and centuries of work in linguistics and philosophy.