On Writing a Textbook on Natural Language Processing (2021)

Abstract: There are thousands of papers about natural language processing and computational linguistics, but very few textbooks. I describe the motivation and process for writing a college textbook on natural language processing, and offer advice and encouragement for readers who may be interested in writing a textbook of their own.

## introduction as natural language processing reaches evergreater heights of popularity, its students can learn from blogs and tutorials, videos and online courses, podcasts, social media, open source software projects, competitions, and more. in this environment, is there still any room for textbooks? this paper describes why you might write a textbook about natural language processing, how to do it, and what i learned from writing one. summary of the book. this paper will not focus on the details of my textbook @xcite , but i offer a brief summary for context. my main goal was to create a text with a formal and coherent mathematical foundation in machine learning, which would explain a broad range of techniques and applications in natural language processing. the first section of the book builds up the mathematical foundation from linear classification though neural networks and unsupervised learning. the second section extends this foundation to structure prediction, with classical algorithms for search and marginalization in sequences and trees, while also introducing some ideas from morphology and syntax. the third section treats the special problem of semantics, which distinguishes natural language processing from other applications of machine learning. this section is more methodologically diverse, ranging from logical to distributional semantics. the final section treats three of the primary application areas: machine translation, information extraction, and text generation. altogether this comprises nineteen chapters, which is more than could be taught in a single semester. rather, the teacher or student can select subsets of chapters depending on whether they wish to emphasize machine learning, linguistics, or applications. the preface sketches out a few paths through the book for various types of courses. ## motivation and related work in this section, i offer some reasons for writing a textbook, compare textbooks with alternative educational formats, and provide a few words of encouragement for prospective authors. ## methodology publishers seem keenly aware of the need for more textbooks in natural language processing and in ai more generally, and i found several editors that were eager to talk at conferences. i selected mit press because of their track record in publishing some of my favorite computer science textbooks. other factors that you may wish to consider are the length of the review and production process, and the publisher's position towards open access. i was lucky to get feedback on the contract from another editor and from colleagues who have written books in other fields, but i did not think of negotiating with regard to electronic editions and translations. fortunately the publisher was generous on these points, as they turned out to be a significant fraction of the revenue for the book. in any case, in the current environment of high demand for ai expertise, the financial compensation is not competitive with other uses of the same amount of time. you may find that it makes more sense to negotiate on aspects of the book and publishing process, such as length, open access, and support. the publisher requires four main inputs from the author: a proposal, a complete draft for review, a "finished version" for copy editing and composition, and markup of page proofs. in the rest of the section, i'll describe how i approached each of these inputs. a timeline is given in ## evaluation what worked. it is difficult to be objective about a project of this scope. i am always happy to learn that the book is being used in a course or for selfstudy, and was thrilled about translations into chinese and korean. the text seems best suited to classes that are similar to mine, where the primary goal is to train future researchers, who need a mathematical foundation in the discipline. i have been told that the exercises are particularly helpful, and i have received many requests for solutions, which i am happy to provide to teachers. there have been fewer reports of errors than i had expected, which i attribute to the careful reading of several classes of students while i was teaching from the unpublished notes. foot_4 offering the pdf online seems to have been an essential factor in the adoption of the textbook, especially given that the most popular alternative (j&m) is also freely available. what could have been better. by the time the book appeared in print, there had been a number of significant changes in both the theory and practice of natural language processing. while this was expected, it is nonetheless hard not to be disappointed not to have put more emphasis on the topics that increased in importance -i'll say a bit more on this in the final section. some readers feel that the term "introduction" in the title is misleading with regard to the amount of mathematical background that is expected. while the text assumes only multivariate calculus (and attempts to be clear about this expectation), the pace of the opening chapters is difficult for students who are out of practice. my editor was probably correct that adoption would be greater if i provided slides that professors could teach from, but i couldn't bring myself to make time for this tedious task after finishing the book. ## future work as the field of natural language processing continues to progress, it is tempting to update the textbook with the latest research developments. for example, multilinguality and multimodality would deserve significantly more emphasis in a second edition, and a revision would have to reflect the maturation of applications such as question answering and dialog. but while some changes could be addressed by adding or modifying a few chapters, others -particularly the shift from conventional supervised learning to more complex methodologies like pretraining, multi-task learning, distilla-tion, and prompt-based learning -seem to require a more fundamental rethinking of the book's underlying structure, particularly in a textbook that emphasizes a coherent mathematical foundation. any such revisions would have to grow out of classroom teaching experience, which did so much to determine the shape of the first edition.