Spaces:

hesamation
/

primer-llm-embedding

Running

File size: 44,194 Bytes

<!DOCTYPE html>
<html>

<head>
    <script src="distill.bundle.js" type="module" fetchpriority="high" blocking></script>
    <script src="main.bundle.js" type="module" fetchpriority="low" defer></script>
    <script src="https://cdn.plot.ly/plotly-3.0.0.min.js" charset="utf-8"></script>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta charset="utf8">
    <base target="_blank">
    <title>LLM Embeddings Explained: A Visual and Intuitive Guide</title>
    <link rel="stylesheet" href="style.css">
</head>

<body>
    <d-front-matter>
        <script id='distill-front-matter' type="text/json">{
        "title": "LLM Embeddings Explained: A Visual and Intuitive Guide",
        "description": "How Language Models Turn Text into Meaning, From Traditional Embeddings to Modern LLMs",
        "published": "Mar 18, 2025",
        "affiliation": {
            "name": "University of Bologna"
        },
        "authors": [
        {
            "author":"Hesam Sheikh Hassani",
            "authorURL":"https://huggingface.co/hesamation"
        }
        ],
        "katex": {
        "delimiters": [
            {"left": "$$", "right": "$$", "display": false}
        ]
        } 
    }
    </script>
    </d-front-matter>
    <d-title>
        <h1 class="l-page" style="text-align: center;">LLM Embeddings Explained:<br>A Visual and Intuitive Guide</h1>
        <div id="title-plot" class="main-plot-container l-screen" style="overflow-x: hidden; width: 100%; text-align: center;">
            <div style="display: flex; justify-content: center; position: relative;">
                {{{fragment-banner}}}
            </div>
            <p style="text-align: cekter; font-style: italic; margin-top: 10px; max-width: 900px; margin-left: auto; margin-right: auto;">
                The embedding atlas of 50 random words and their closest tokens in the embedding space of `deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B`.
            </p>
        </div>
    </d-title>
    <d-byline></d-byline>
      <d-article>
        <d-contents>
        </d-contents>
        
        <p>Embeddings are the semantic backbone of LLMs, the gate at which raw text is transformed into vectors of numbers that are understandable by the model. When you prompt an LLM to help you debug your code, your words and tokens are transformed into a high-dimensional vector space where semantic relationships become mathematical relationships.</p>
        <aside>Reading time: 12-15 minutes. This blog post is recommended for desktop users.</aside>

        <p>In this article we go through the fundamentals of embeddings. We will cover what embeddings are, how they evolved over time from statistical methods to modern techniques, check out how they're implemented in practice, look at some of the most important embedding techniques, and how the embeddings of an LLM (DeepSeek-R1-Distill-Qwen-1.5B) look like as a graph representation.</p>

        <p>This article includes interactive visualization and hands-on code examples. It also avoids verbosity and focuses on the core concepts for a fast-paced read to get straight to the point. The full code is available on my <a href="https://github.com/hesamsheikh/llm-mechanics">LLM Mechanics GitHub repository</a>.</p>
        <aside>To contribute to the article, point out any mistakes, or suggest improvements, please check out <a href="https://huggingface.co/spaces/hesamation/primer-llm-embedding/discussions">Community</a>.</aside>

        <div class="audio-container">
            <audio controls>
                <source src="assets/audio/podcast.wav" type="audio/wav">
                Your browser does not support the audio element.
              </audio>
              <div class="figure-legend">
              <p>If your ears are more important than your eyes, you can listen to the podcast version of this article generated by <a href="https://www.notebooklm.com/">NotebookLM</a>.</p>
            </div>
        </div>


        <h2>What Are Embeddings?</h2>
        
        <p>Processing text for NLP tasks requires a numeric representation of each word. Most embedding methods come down to turning a word or token into a vector. What makes embedding techniques different from each other, is how they approach this word → vector conversion.</p>

        <p>Embedding is not just for text, they can be applied to images, audio, or even graph data. In a general sense, embedding is the process of converting data [of any type] into vectors. Of course, the embedding methods of each modality is different and unique. In this article, when we talk about "embeddings", we are referring to the text embeddings.</p>

        <p>You might have heard embeddings in the context of large language models, but embeddings actually have a much longer history. Here is an overview of various embedding techniques:</p>

        <figure class="fullscreen">
            <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
              {{{fragment-word-emb-hierarchy}}}
            </div>
            <figcaption>Figure 1: Overview of different word embedding techniques.</figcaption>
        </figure>

        <p>When reading about embeddings you may come across static vs. dynamic (or contextualized) embeddings. It's important to distinguish between token embeddings—which are the fixed vectors assigned to input tokens at the very beginning of an LLM—and the contextual representations produced by the deeper layers of the model. While both are technically embeddings, token embeddings are static, whereas the intermediate hidden states evolve as they pass through each layer, capturing the full context of the input. In some literature, these contextual outputs are also referred to as "embeddings," which can be confusing.</p>
        <aside>In some literature, vector embeddings in the form of hidden states are also referred to as "embeddings," which can be confusing.</aside>
          
        <h2>What Makes a Good Embedding?</h2>

        <p>When it comes to LLMs, embeddings can be thought of as the dictionary of their language. Better embeddings allow these models to understand the human language and communicate with us. But what makes an embedding technique good? In other words, what makes an embedding ideal? Here are two major properties of an embedding technique:</p>

        <h3>Semantic Representation</h3>

        <p>Some types of embeddings capture the semantic relationship between words. This means that words with closer meanings or relationships are closer in the vector space than words that are less related. For example, the vectors of "cat" and "dog" must be more similar than "dog" and "strawberry".</p>

        <h3>Dimensionality</h3>

        <p>What should be the size of an embedding vector, 15, 50, 300? Striking the right balance is key. Smaller vectors (lower dimensions) are more efficient to keep in memory or to process, while bigger vectors (higher dimensions) can capture intricate relationships, but are prone to overfitting. For reference, GPT-2 model family has an embedding size of at least 768.</p>
        <aside>For reference, DeepSeek-V3 and R1 have an embedding size of 7168.</aside>

        <h2>Traditional Embedding Techniques</h2>

        <p>Almost every embedding technique relies on a large corpus of text data to extract the relationship of the word. Previously, embedding methods relied on statistic methods based on the occurence or co-occurence of words in a text. This was based on the assumption that if a pair of words often appear together then they must have a closer relationship. These are simple methods that are not as computation-heavy as other techniques. One of such methods is:</p>

        <h3>TF-IDF (Term Frequency-Inverse Document Frequency)</h3>

        <p>The idea of TF-IDF is to calculate the importance of a word in a document by considering two factors<d-cite bibtex-key="vardhan2024embeddings"></d-cite>:</p>

        <ol>
            <li><strong>Term-Frequency (TF)</strong>: How frequent a term appears in a document. A higher TF shows that a term is more important to the document.</li>
            <li><strong>Inverse Document Frequency (IDF)</strong>: How rare a term is across the documents. This is based on the assumption that terms appearing in multiple documents are less important than terms that are unique to fewer documents.</li>
        </ol>
        
        <p>The formula for TF-IDF consists of two parts. First, the term frequency (TF) is calculated as:</p>

        <d-math>
            tf(t,d) = \frac{\text{count of term }t\text{ in document }d}{\text{total number of terms in document }d}
        </d-math>

        <p>For example, if a document has 100 words and the word "cat" appears 5 times, the term frequency for "cat" would be 5/100 = 0.05. This gives us a simple numerical representation of how prevalent that term is in the document.</p>

        <p>Then, the inverse document frequency (IDF) is calculated as:</p>

        <d-math>
            idf(t) = \log\left(\frac{\text{total number of documents}}{\text{number of documents containing term }t}\right)
        </d-math>

        <p>This component gives higher weight to terms that appear in fewer documents. Common words that appear in many documents (like "the", "a", "is") will have a lower IDF, while rare, more informative words will have a higher IDF.</p>

        <p>Finally, the TF-IDF score is calculated by multiplying these two components:</p>

        <d-math>
            tfidf(t,d) = tf(t,d) \times idf(t)
        </d-math>

        <p>Let's look at a concrete example:</p>

        <p>Suppose we have a corpus of 10 documents, and the word "cat" appears in only 2 of these documents. The IDF for "cat" would be:</p>

        <d-math>
            idf(\text{"cat"}) = \log\left(\frac{10}{2}\right) = \log(5) \approx 1.61
        </d-math>

        <p>If in one particular document, "cat" appears 5 times out of 100 total words, its TF would be 0.05. Therefore, the final TF-IDF score for "cat" in this document would be:</p>

        <d-math>
            tfidf(\text{"cat"}) = 0.05 \times 1.61 \approx 0.08
        </d-math>

        <p>This score tells us how important the word "cat" is to this specific document relative to the entire corpus. A higher score indicates that the term is both frequent in this document and relatively rare across all documents, making it potentially more meaningful for characterizing the document's content.</p>

        <p>Let's use TF-IDF on the <a href="https://github.com/karpathy/char-rnn/blob/master/data/tinyshakespeare/input.txt">TinyShakespeare</a> dataset. To simulate multiple documents, we chop off the document into ten chunks.</p>
        
        <details style="background: #f6f8fa; border: 1px solid #d0d7de; border-radius: 6px; margin: 1em 0;">
            <summary style="padding: 12px; cursor: pointer; user-select: none; background: #f3f4f6; border-bottom: 1px solid #d0d7de;">
                TF-IDF example using TinyShakespeare
            </summary>
                <div class="code-embed-container" style="margin: 0; border-radius: 0; overflow-x: scroll; width: max-content; min-width: 100%; font-size: 8px;"></div>
                <script src="https://gist.github.com/hesamsheikh/951ba078734a66d19a6c963edfd8bc3c.js"></script>
                </div>
        </details>

        <p>This gives us a 10 dimensional embedding, each for a document we have. Now to get a better idea of the TF-IDF embeddings, we use PCA to map the 10d space to 2d space so we can visualize it better.</p>
        
        <figure class="fullscreen">
            <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
              {{{fragment-tfidf-emb}}}
            </div>
            <figcaption>TF-IDF example showing word embeddings plotted in 2D space after dimensionality reduction</figcaption>
        </figure>

        <p>There are two things noticeable about this embedding space:</p>

        <ol>
            <li>The majority of the words are concentrated into one particular area. This means that the embedding of most words are similar in this approach. It signals a lack of expressiveness and uniqueness about these embeddings.</li>
            <li>The embeddings have no semantic connection. The distance between words has nothing to do with their meaning.</li>
        </ol>

        <p>Because TF-IDF is based on the occurrence frequency of terms in the document, words that are semantically close (such as numbers) have no relation in the vector space. The simplicity of TF-IDF and similar statistical methods is what makes them useful in applications such as information retrieval, keyword extraction, and basic text analysis. You can read about some of these methods in <d-cite bibtex-key="turing2022embeddings"></d-cite>.</p>

        <h2>word2vec</h2>

        <p>Originally proposed in <d-cite bibtex-key="Mikolov_Chen_Corrado_Dean_2013"></d-cite>, is a more modern technique than TF-IDF. As can be assumed by the name, it is a network that aims to convert words into embedding vectors. It achieves this by defining a side goal, something to optimize the network for. For example, in CBOW (continuous bag of words), the word2vec network is trained to predict a missing word when its given the neighbors of that word as input. The intuition is that you can infer the embeddings of a word given the words around it.</p>

        <p>The word2vec architecture is pretty simple: one hidden layer that we extract the embeddings from, and one output layer which predicts the probabilities of all words in the vocabulary. On the surface, the network is trained to predict the right missing word given its neighbors, but in reality, this is an excuse to train the hidden layer of the network and find the right embeddings for each word. After the network is trained, the last layer can be tossed out the window because figuring out the embeddings is the real goal of the network.</p>

        <figure class="fullscreen">
            <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
                <img src="assets/images/word2vec.png" alt="Diagram showing word2vec architecture with input layer, hidden layer (embeddings), and output layer. Input words are encoded as one-hot vectors, passed through a hidden layer to generate embeddings, and finally through an output layer that predicts probabilities for each word in vocabulary.">
            </div>
            <figcaption>word2vec architecture showing the input layer, hidden layer (embeddings), and output layer</figcaption>
        </figure>

        <p>Aside from CBOW, another variant is Skipgram which works completely the opposite: it aims to predict the neighbors, given a particular word as input.</p>

        <p>Here's how CBOW word2vec works step by step:</p>
        <ol>
            <li>Choose a context window (e.g. 2 in the image above)</li>
            <li>Take the two words before and two words after a particular word as input</li>
            <li>Encode these four context words as one-hot vectors</li>
            <li>Pass the encoded vectors through the hidden layer, which has a linear activation function that outputs the input unchanged</li>
            <li>Aggregate the outputs of the hidden layer (e.g. using a lambda mean function)</li>
            <li>Feed the aggregated output to the final layer which uses Softmax to predict probabilities for each possible word</li>
            <li>Select the token with the highest probability as the final output of the network</li>
        </ol>

        <p>The hidden layer is where the embeddings are stored. It has a shape of <i>Vocabulary size x Embedding size</i> and as we give a one-hot vector (a vector that is all zeros except for one element set to 1) of a word to the network, that specific <code>1</code> triggers the embeddings of that word to be passed to the next layers. You can see a cool and simple implementation of the word2vec network in <d-cite bibtex-key="sarkar2018cbow"></d-cite>.</p>

        <figure class="fullscreen">
            <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
                <img src="assets/images/word2vec2.png" alt="Visualization of word2vec embeddings in 2D space, showing how semantically similar words cluster together. Words with related meanings appear closer to each other in the embedding space.">
            </div>
            <figcaption>word2vec embeddings visualization</figcaption>
        </figure>
        
        <p>Since the network relies on the relationship between words in a context, and not on the occurrence or co-occurrence of words as in TF-IDF, it is able to capture <strong>Semantics Relationships</strong> between the words.</p>
 
        <p>You can download the pretrained version from Google's official page <d-cite bibtex-key="word2vec"></d-cite>. Here's the code to use word2vec in practice:</p>

        <details style="background: #f6f8fa; border: 1px solid #d0d7de; border-radius: 6px; margin: 1em 0;">
            <summary style="padding: 12px; cursor: pointer; user-select: none; background: #f3f4f6; border-bottom: 1px solid #d0d7de;">
                word2vec example
            </summary>
                <div class="code-embed-container" style="margin: 0; border-radius: 0; overflow-x: scroll; width: max-content; min-width: 100%; font-size: 8px;"></div>
                <script src="https://gist.github.com/hesamsheikh/bba3b97d0ba6dee8e45fcb2ec3ead1de.js"></script>
                </div>
        </details>

        <p>To train word2vec efficiently, especially with large vocabularies, an optimization technique called <strong>negative sampling</strong> is used. Instead of computing the full softmax over the entire vocabulary (which is computationally expensive), negative sampling simplifies the task by updating only a small number of negative examples (i.e., randomly selected words not related to the context) along with the positive ones. This makes training faster and more scalable.</p>

        <p>The semantic relationship is a fun topic to explore and word2vec is a simple setup for your experiments. You can explore the biases of society or the data, or explore how words have evolved over time by studying the embeddings of older manuscripts.</p>

        <p>You can actually visualize and play with word2vec embeddings with <a href="https://projector.tensorflow.org/">Tensorflow Embedding Projector</a>.</p>

        <figure class="fullscreen">
            <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
                <img src="assets/images/word2vec_tfep.jpg" alt="Screenshot of TensorFlow Embedding Projector showing interactive 3D visualization of word2vec embeddings, allowing exploration of semantic relationships between words in the embedding space.">
            </div>
            <figcaption>TensorFlow Embedding Projector's interactive exploration of word2vec embeddings in 3D/2D space.</figcaption>
        </figure>

        <h2>BERT (Bidirectional encoder representations from transformers)</h2>

        <p>Wherever you look in the world of NLP, you will see BERT. It's a good idea to do yourself a favor and learn about BERT once and for all, as it is the source of many ideas and techniques when it comes to LLMs.</p>
        <aside>Here's a good video to get started on BERT. <d-cite bibtex-key="codeemporium2020bert"></d-cite></aside>

        <p>In summary, BERT is an encoder-only transformer model consisting of 4 main parts:</p>

        <ol>
            <li>Tokenizer: chops up texts into sequences of integers.</li>
            <li>Embedding: the module that converts discrete tokens into vectors.</li>
            <li>Encoder: a stack of transformer blocks with self-attention.</li>
            <li>Task head: when encoder is finished with the representations, this task-specific head handles them for token generation or classification tasks.</li>
        </ol>

        <p>BERT inspired from the Transformer architecture introduced in "Attention is all you need", to become an encoder-only transformer that can produce meaningful representations and understand language. The idea was that depending on specific problems to solve, BERT is fine-tuned to learn about that task. These specific tasks can be Q&A (question + passage -> answer), text summarization, classification, etc.</p>

        <p>In the pretraining phase, BERT is trained to learn two tasks simultaneously:</p>

        <ol>
            <li><strong>Masked Language Modeling</strong>: is to predict masked words in a sentence (I [MASKED] this book before -> read)</li>
            <li><strong>Next Sentence Prediction</strong>: given two sentences, predict if A came before B or not. The special [SEP] token separates the two sentences and the task is similar to binary classification.</li>
        </ol>

        <p>Note the other special token, [CLS]. This special token helps with classification tasks. As the model processes input layer by layer, [CLS] becomes an aggregation of all the input tokens, which can later be used for classification purposes.</p>

        <figure class="fullscreen">
            <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
                <img src="assets/images/BERT.png" alt="BERT architecture diagram showing the model's components: input layer with tokenization, embedding layer, multiple transformer encoder blocks with self-attention mechanisms, and task-specific output heads. Special tokens [CLS] and [SEP] are highlighted.">
            </div>
            <figcaption>BERT architecture overview. (Image source: <d-cite bibtex-key="devlin2018bert"></d-cite>)</figcaption>
        </figure>

        <p><strong>So why is BERT important?</strong></p>

        <p>BERT is among the first instances of Transformer-based <strong>contextualized, dynamic embeddings</strong>. When given a sentence as input, the layers of the BERT model use self-attention and feed-forward mechanisms to update and incorporate context from all other tokens in the sentence. The final output of each Transformer layer is a contextualized representation of the word.</p>

        <h2>Embeddings in Modern LLMs</h2>

        <p>Embeddings are a foundational component in large language models and also a broad term. For the purpose of this article, we focus on "embeddings" as the module that transforms tokens into vector representations as opposed to the latent space in the hiddent layers.</p>

        <h3>Where does the embedding fit into LLMs?</h3>

        <p>In transformer-based models, the term "embedding" can refer to both static embeddings and dynamic contextual representations:</p>

        <ol>
            <li><strong>Static Embeddings</strong> generated in the first layer and combine token embeddings (vectors representing tokens) with positional embeddings (vectors encoding a token's position in the sequence).</li>
            <li><strong>Dynamic Contextual Representations</strong>. As input tokens pass through the self-attention and feed-forward layers, their embeddings are updated to become contextual. These dynamic representations capture the meaning of tokens based on their surrounding context. For example, the word "bank" appears both as "river bank" and "bank robbery", and while the <strong>token embedding</strong> of the word bank is the same in both cases, the transformations it goes through in the layers of the network take into account the context of which the word "bank" appears in.</li>
        </ol>

        <figure class="fullscreen">
            <div id="image-as-graph" style="display: grid; position: relative; justify-content: center; width: 70%;">
                <img src="assets/images/llm_embedding_overview.png" alt="Diagram showing how embeddings fit into LLM architecture: input text is tokenized, converted to token embeddings, combined with positional embeddings, and processed through transformer layers to generate contextual representations.">
            </div>
            <figcaption>Overview of how embeddings fit into the LLM architecture</figcaption>
        </figure>

        <h3>LLM Embeddings are Trained</h3>

        <p>LLM embeddings are optimized during the training process. Borrowing from Sebastian Raschka's <strong>Build a Large Language Model (From Scratch)</strong><d-cite bibtex-key="manning2024llm"></d-cite>, <em>" While we can use pretrained models such as Word2Vec to generate embeddings for machine learning models, LLMs commonly produce their own embeddings that are part of the input layer and are updated during training. The advantage of optimizing the embeddings as part of the LLM training instead of using Word2Vec is that the embeddings are optimized to the specific task and data at hand."</em></p>

        <h3>torch.nn.Embedding</h3>

        <p>The embedding layer in LLMs works as a look-up table. Given a list of indices (token ids) it returns their embeddings. Build a Large Language Model (From Scratch) <d-cite bibtex-key="manning2024llm"></d-cite> shows this concept comperhensively.</p>

        <figure class="fullscreen">
            <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
                <img src="assets/images/embedding_layer.png" alt="Illustration of embedding layer as a lookup table: input tokens are mapped to their corresponding embedding vectors through a matrix multiplication operation. Shows how token IDs index into rows of the embedding matrix.">
            </div>
            <figcaption>Visualization of how the embedding layer works as a lookup table. (Image source: <d-cite bibtex-key="manning2024llm"></d-cite>)</figcaption>
        </figure>

        <p>The code implementation of an embedding layer in PyTorch is done using <code>torch.nn.Embedding</code> which acts as a simple look-up table. There is nothing more special about this layer than a simple Linear layer, other than the fact that it can work with indices as input rather than one-hot encoding inputs. The Embedding layer is simply a Linear layer that works with indices.</p>

        <figure class="fullscreen">
            <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
                <img src="assets/images/emb_lookup.jpg" alt="Detailed visualization of embedding lookup process: shows how token IDs are used to select specific rows from the embedding weight matrix, resulting in embedding vectors for each input token.">
            </div>
            <figcaption>Visualization of the embedding lookup process. (Image source: <d-cite bibtex-key="manning2024llm"></d-cite>)</figcaption>
        </figure>

        <p>This notebook by Sebastian Raschka explains the Embedding layer in depth <d-cite bibtex-key="rasbt2024llm"></d-cite>.</p>

        <p>Now let's work with the embedding of a model and see some visuals!</p>

        <h2>Embeddings in Action (DeepSeek-R1-Distill-Qwen-1.5B)</h2>

        <p>How does the embedding layer in a large language model look like?</p>

        <p>Let's dissect the embeddings of the distilled version of DeepSeek-R1 in the Qwen model. Some parts of the following code is inspired by <d-cite bibtex-key="chrishayuk2024embeddings"></d-cite>. Let's get our hands dirty.</p>

        <p>We begin by loading the <code>deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B</code> model from <a href="https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B">Hugging Face</a> and saving the embeddings.</p>

        <details style="background: #f6f8fa; border: 1px solid #d0d7de; border-radius: 6px; margin: 1em 0;">
            <summary style="padding: 12px; cursor: pointer; user-select: none; background: #f3f4f6; border-bottom: 1px solid #d0d7de;">
                Load the model and save the embeddings
            </summary>
                <div class="code-embed-container" style="margin: 0; border-radius: 0; overflow-x: scroll; width: max-content; min-width: 100%; font-size: 8px;"></div>
                <script src="https://gist.github.com/hesamsheikh/6d62d890604078580342fb919e3d2798.js"></script>
                </div>
        </details>

        <p>Now let's load the embedding layer and work with it. The goal of separating the embedding from the other parts of the model, saving, and loading it is to get the embeddings of an input much faster and efficiently rather than doing a complete forward pass of the model.</p>

        <details style="background: #f6f8fa; border: 1px solid #d0d7de; border-radius: 6px; margin: 1em 0;">
            <summary style="padding: 12px; cursor: pointer; user-select: none; background: #f3f4f6; border-bottom: 1px solid #d0d7de;">
                Load the model embeddings
            </summary>
                <div class="code-embed-container" style="margin: 0; border-radius: 0; overflow-x: scroll; width: max-content; min-width: 100%; font-size: 8px;"></div>
                <script src="https://gist.github.com/hesamsheikh/6fb5bfbeeb69e62a94c8eeae09a03cfc.js"></script>
                </div>
        </details>

        <p>Now let's see how a sentence is tokenized and then converted to embeddings.</p>

        <details style="background: #f6f8fa; border: 1px solid #d0d7de; border-radius: 6px; margin: 1em 0;">
            <summary style="padding: 12px; cursor: pointer; user-select: none; background: #f3f4f6; border-bottom: 1px solid #d0d7de;">
                Convert prompt to embeddings
            </summary>
                <div class="code-embed-container" style="margin: 0; border-radius: 0; overflow-x: scroll; width: max-content; min-width: 100%; font-size: 8px;"></div>
                <script src="https://gist.github.com/hesamsheikh/f7918916235222ee67e5c337600ac292.js"></script>
                </div>
        </details> 

        <p>In the above code, we tokenize the sentence and print the embeddings of the tokens. The embeddings are 1536-dimensional vectors. Here is a simple example with the sentence: "HTML coders are not considered programmers":</p>
        <figure>
            <div style="overflow-x: auto;">
                <table style="width: 100%; border-collapse: collapse; font-family: monospace; font-size: 14px;">
                    <thead style="background-color: #f3f4f6;">
                        <tr>
                            <th style="padding: 8px; border: 1px solid #d0d7de;">token_id</th>
                            <th style="padding: 8px; border: 1px solid #d0d7de;">token</th>
                            <th style="padding: 8px; border: 1px solid #d0d7de;">Embedding Vector (1536 dimensions)</th>
                        </tr>
                    </thead>
                    <tbody>
                        <tr>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">151646</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">&nbsp;</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de; font-size: 12px;">-0.027466, 0.002899, -0.005188 ... 0.021606</td>
                        </tr>
                        <tr>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">5835</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">HTML</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de; font-size: 12px;">-0.018555, 0.000912, 0.010986 ... -0.015991</td>
                        </tr>
                        <tr>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">20329</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">#cod</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de; font-size: 12px;">-0.026978, -0.012939, 0.021362 ... 0.042725</td>
                        </tr>
                        <tr>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">388</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">ers</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de; font-size: 12px;">-0.012085, 0.001244, -0.069336 ... -0.001213</td>
                        </tr>
                        <tr>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">525</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">#are</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de; font-size: 12px;">-0.001785, -0.008789, 0.006195 ... -0.016235</td>
                        </tr>
                        <tr>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">537</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">#not</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de; font-size: 12px;">0.016357, -0.039062, 0.045898 ... 0.001686</td>
                        </tr>
                        <tr>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">6509</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">#considered</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de; font-size: 12px;">-0.000721, -0.021118, 0.027710 ... -0.051270</td>
                        </tr>
                        <tr>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">54846</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de;">#programmers</td>
                            <td style="padding: 8px; border: 1px solid #d0d7de; font-size: 12px;">-0.047852, 0.057861, -0.069336 ... 0.005280</td>
                        </tr>
                    </tbody>
                </table>
            </div>
            <figcaption>Example of token embeddings for the sentence "HTML coders are not considered programmers"</figcaption>
        </figure>

        <p>Finally, let's see how we can find the most similar embeddings to a particular word. As embeddings are vectors, we can use cosine similarity to find the most similar embeddings to a particular word. Then, we can use the <code>torch.topk</code> function to find the top k most similar embeddings.</p>
        
        <details style="background: #f6f8fa; border: 1px solid #d0d7de; border-radius: 6px; margin: 1em 0;">
            <summary style="padding: 12px; cursor: pointer; user-select: none; background: #f3f4f6; border-bottom: 1px solid #d0d7de;">
                Find similar embeddings
            </summary>
                <div class="code-embed-container" style="margin: 0; border-radius: 0; overflow-x: scroll; width: max-content; min-width: 100%; font-size: 8px;"></div>
                <script src="https://gist.github.com/hesamsheikh/63bb68c69d094e7f905ba7358e2c0771.js"></script>
                </div>
        </details> 

        <h2>Embeddings as Graphs: A Network Analysis</h2>

        <p>How can we view the embeddings? One method is to look at the embedding layer as a network, in which tokens are the nodes; if two token vectors are close then we assume their nodes are connected via an edge.</p>

        <p>As an example, if we take the sentence <em>"AI agents will be the most hot topic of artificial intelligence in 2025."</em>, tokenize it, convert the tokens to embeddings, find the 20 most similar embeddings to each of the ones we had, the following will be the embedding graph:</p>

        <figure class="fullscreen">
            <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
              {{{fragment-token-visualization-sentence}}}
            </div>
            <figcaption>Figure 2: Visualization of the embedding graph for the sentence "AI agents will be the most hot topic of artificial intelligence in 2025."</figcaption>
        </figure>

        <p>You can actually see a more comprehensive example at the beginning of the article in which 50 tokens and their closest tokens are mapped out.</p>

        <p>In the graph examples, we have ignored the "token variations". A token or word may have many different variations, each with their own embeddings. For example, the token "list" may have many different variations with their own embeddings, such as "_list", "List", and many more. These variations often have an embedding that is very similar to the original token. The following is a graph of the token "list" and its close neighbors, including its variations.</p>

        <figure class="fullscreen">
            <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
              {{{fragment-token-visualization-list}}}
            </div>
            <figcaption>Figure 3: The word “list” and some of the closest tokens to it including its variants.</figcaption>
        </figure>


        <h2>Let's Wrap Up</h2>

        <p>Embeddings remain as one of the fundamental parts in natural language processing and modern large language models. While the research in machine learning and LLMs discover new methods and techniques every day, embeddings haven't seen much change in large language models (and that has to mean something). They are essential, easy to understand, and easy to work with.</p>
        
        <aside>This blog post's template is inspired by the <a href="https://distill.pub/">distill.pub</a>, if you want to use it for your own blog post, you can find the template <a href="https://huggingface.co/spaces/lvwerra/distill-blog-template">here</a>.</aside>
        
        <p>In this blog post we went through the basics of what you need to know about embeddings, and their evolution from traditional statistical methods into their use case in today's LLMs. My hope is that this has been a comprehensive jump-start to help you gain an intuitive understanding of the word embeddings and what they represent.</p>

        <p><strong>Thank you for reading through this article.</strong> If you found this useful, consider following me on <a href="https://x.com/Hesamation">X (Twitter)</a> and <a href="https://huggingface.co/hesamation">Hugging Face</a> to be notified about my next projects. Also, don't forget to star ⭐ this space!</p>

        <p>If you have any questions or feedback, please feel free to write in the <a href="https://huggingface.co/spaces/hesamation/primer-llm-embedding/discussions">community</a>.</p>

        <p>You can also check out my <a href="https://github.com/hesamsheikh/ml-retreat">ml-retreat repository</a> for my notes and resources on fundamental NLP, LLM, or machine learning topics.</p>

    </d-article>

    <d-appendix>
        <d-bibliography src="bibliography.bib"></d-bibliography>
        <style>
            d-appendix .citation {
                font-size: 11px;
                line-height: 15px;
                border-left: 1px solid rgba(0, 0, 0, 0.1);
                padding-left: 18px;
                border: 1px solid rgba(0, 0, 0, 0.1);
                background: rgba(0, 0, 0, 0.02);
                padding: 10px 18px;
                border-radius: 3px;
                color: rgba(150, 150, 150, 1);
                overflow: hidden;
                margin-top: -12px;
                white-space: pre-wrap;
                word-wrap: break-word;
            }
        </style>

        <h3 id="citation">Citation</h3>
        <p>For attribution in academic contexts, please cite this work as</p>
        <pre
            class="citation short">"LLM Embeddings Explained: A Visual and Intuitive Guide", 2025.</pre>
        <p>BibTeX citation</p>
        <pre class="citation long">@misc{llm_embeddings_explained,
        title={LLM Embeddings Explained: A Visual and Intuitive Guide},
        author={Hesam Sheikh Hessani},
        year={2025},
        }</pre>
    </d-appendix>
    <script>
      function toggleTOC() {
          const content = document.querySelector('.toc-content');
          const icon = document.querySelector('.toggle-icon');
          
          content.classList.toggle('collapsed');
          icon.classList.toggle('collapsed');
      }
    </script>

    <script>
        const article = document.querySelector('d-article');
        const toc = document.querySelector('d-contents');
        if (toc) {
            const headings = article.querySelectorAll('h2, h3, h4');
            // let ToC = `<nav role="navigation" class="l-text figcaption"><h3>Table of contents</h3>`;
            let ToC = `<nav role="navigation" class="l-text figcaption"><div class="toc-header" onclick="toggleTOC()">
            <span class="toc-title">Table of Contents</span>
            <span class="toggle-icon">▼</span>
            </div><div class="toc-content">`;
            let prevLevel = 0;

            for (const el of headings) {
                // should element be included in TOC?
                const isInTitle = el.parentElement.tagName == 'D-TITLE';
                const isException = el.getAttribute('no-toc');
                if (isInTitle || isException) continue;
                el.setAttribute('id', el.textContent.toLowerCase().replaceAll(" ", "_"))
                const link = '<a target="_self" href="' + '#' + el.getAttribute('id') + '">' + el.textContent + '</a>';

                const level = el.tagName === 'H2' ? 0 : (el.tagName === 'H3' ? 1 : 2);
                while (prevLevel < level) {
                    ToC += '<ul>'
                    prevLevel++;
                }
                while (prevLevel > level) {
                    ToC += '</ul>'
                    prevLevel--;
                }
                if (level === 0)
                    ToC += '<div>' + link + '</div>';
                else
                // else if (level === 1)
                    ToC += '<li>' + link + '</li>';
            }

            while (prevLevel > 0) {
                ToC += '</ul>'
                prevLevel--;
            }
            ToC += '</div></nav>';
            toc.innerHTML = ToC;
            toc.setAttribute('prerendered', 'true');
            const toc_links = document.querySelectorAll('d-contents > nav div a');

            window.addEventListener('scroll', (_event) => {
                if (typeof (headings) != 'undefined' && headings != null && typeof (toc_links) != 'undefined' && toc_links != null) {
                    find_active: {
                        for (let i = headings.length - 1; i >= 0; i--) {
                            const heading = headings[i];
                            // Skip headings that shouldn't be in TOC
                            if (heading.parentElement.tagName == 'D-TITLE' || heading.getAttribute('no-toc')) {
                                continue;
                            }
                            
                            if (heading.getBoundingClientRect().top - 50 <= 0) {
                                // Find matching TOC link by href
                                const headingId = heading.getAttribute('id');
                                const activeLink = Array.from(toc_links).find(link => 
                                    link.getAttribute('href') === '#' + headingId
                                );
                                
                                if (activeLink && !activeLink.classList.contains("active")) {
                                    toc_links.forEach(link => link.classList.remove("active"));
                                    activeLink.classList.add('active');
                                }
                                break find_active;
                            }
                        }
                        toc_links.forEach(link => link.classList.remove("active"));
                    }
                }
            });
        }
    </script>

</body>

</html>