Building a ML Transformer in a Spreadsheet
Tags: machinelearning, tutorial, neuralnetworks, beginners
Published: 2022-03-13 14:47:24

---
title: Building a ML Transformer in a Spreadsheet 
published: true
description: To better understand Transformer models and how self-attention works, I built one in a spreadsheet. You can too!
tags: machinelearning, tutorial, neuralnetworks, beginners
---

[Attention Is All You Need](https://arxiv.org/pdf/1706.03762.pdf) introduced Transformer models, which have been wildly effective in solving various Machine Learning problems. However, the 10 page paper is incredibly dense. There are so many details, it was difficult for me to gain high-level insights about how they work and why they are effective.

After several months of reading other blog posts about them, I understood them well enough to [create a Transformer in a spreadsheet and made a video walking through it](https://www.youtube.com/watch?v=S9eKuRVigjY).

![Diagram showing how Transformers have alternating "data converter" sublayers and "pattern finder" sublayers. Zooming in on the "data converter" sublayer shows how Query and Key matrices combine to form a self-attention matrix, which focuses on and remembers important parts of the value matrix](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/scd93u4f0yg36j52su06.png)

At a high level, Transformers are effective because they convert the data in a way that can make it easier to find patterns. They build on ideas from Convolutional Neural Networks and Recurrent Neural Networks (Focus and Memory), combining them in something called self-attention.

[The video](https://www.youtube.com/watch?v=S9eKuRVigjY) covers these ideas in more details and [this is the link to the spreadsheet with the implemented Transformer](https://bit.ly/TransformerSheet2). Skip to the "Appendix" sheet if you want to see a layer with all the bells and whistles, including multi-headed attention and residual connections.

Implementing the Transformer really helped me understand all the components. I'm especially proud of my metaphor of "scoring points" for explaining self-attention. 

Other resources I found useful when researching transformers:
 - <https://nlp.seas.harvard.edu/2018/04/03/attention.html>
 - <https://jalammar.github.io/illustrated-transformer/>
 - <https://towardsdatascience.com/illustrated-guide-to-transformers-step-by-step-explanation-f74876522bc0>
 - [The Narrated Transformer Language Model](https://youtu.be/-QH8fRhqFHM)
 - [Attention Model Intuition](https://youtu.be/SysgYptB198)
 - [Attention Mechanisms](https://youtu.be/3qdQAtfq1jU)
 - [LSTM is dead. Long Live Transformers!](https://youtu.be/S27pHKBEp30)
 - [What exactly are keys, queries, and values?](https://stats.stackexchange.com/a/424127)
 - <https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/>
 - <https://medium.com/deeper-learning/glossary-of-deep-learning-word-embedding-f90c3cec34ca>

More of my work:
 - Building a traditional neural network in a spreadsheet to learn AND and XOR ([Part 1](https://www.youtube.com/watch?v=fjfZZ6S1ad4), [Part 2](https://www.youtube.com/watch?v=1zwnPt73pow))
 - Building a convolutional neural network in a spreadsheet to recognize letters ([Part 1](https://www.youtube.com/watch?v=8QfFtUcuYCU), [Part 2](https://www.youtube.com/watch?v=pH0s0e62GT8))
 - [Building a recurrent neural network in a spreadsheet to emulate autocomplete](https://www.youtube.com/watch?v=a4JLw4Du8YM)