Repository: pytorch
Issue: LazyEmbedding, an embedding layer with a dynamically sized vocabulary
Labels: feature,module: nn,triaged,module: embedding

## 🚀 Feature

It'd be AMAZING to have a lazy embedding layer that grows to accommodate new tokens. So, for example, the interface would look like this:

```python
import torch

hidden_size = 16
embed_token = torch.nn.LazyEmbedding(hidden_size)

embedding = embed_token(["a", "b", "c"])
assert embedding.shape == (3, hidden_size)
assert embed_token.vocab_size == 3

embedding = embed_token(["d", "e"])
assert embedding.shape == (2, hidden_size)
assert embed_token.vocab_size == 5

embed_token = embed_token.eval()
embedding = embed_token(["f"])  # ERROR: Token 'f' not found.

```

And, for example, here is a basic, inefficient, implementation:

```python
import typing

import torch

class LazyEmbedding(torch.nn.Module):
    def __init__(self, max_num_embeddings: int, *args, **kwargs):
        super().__init__()
        self.embedding = torch.nn.Embedding(num_embeddings=max_num_embeddings, *args, **kwargs)
        self.vocab: typing.Dict[str, int] = {}

    def forward(self, tokens: typing.List) -> torch.Tensor:
        indicies = []
        for token in tokens:
            if token not in self.vocab and self.training:
                self.vocab[token] = len(self.vocab)
            indicies.append(self.vocab[token])
        return self.embedding(torch.tensor(indicies))


embedding = LazyEmbedding(10, embedding_dim=16)

tensor = embedding(["a", "b", "c"])
print(tensor.shape) # torch.Size([3, 16])
print(embedding.vocab)  # {'a': 0, 'b': 1, 'c': 2}

tensor = embedding(["d", "e", "f"])
print(tensor.shape) # torch.Size([3, 16])
print(embedding.vocab)  # {'a': 0, 'b': 1, 'c': 2, 'd': 3, 'e': 4, 'f': 5}

try:
    embedding.eval()(["g"])
except KeyError:
    print("KeyError")
```

_There is also a bit of work that can be done with modularity. For example, `PyTorch` could be responsible for providing a `LazyEmbedding` that supports only `torch.Tensor`. `torchtext` could provide a wrapper that adds the vocabulary._

## Motivation

This feature has been on my mind a lot because it'd dramatically simplify my NLP training pipelines. 

Without a `LazyEmbedding`, typically, I need to use `torchnlp.encoders` or `torchtext.vocab`. I'll need to initialize these objects by looping through my entire dataset, in order to determine all the tokens, that I might need. Afterward, I'll need to use the vocabulary with my `DataLoader` in order to encode training examples. Lastly, I need to store this object in the related checkpoints, so that other people can use the same encoding. 

With a module like this, I wouldn't need to use `torchnlp.encoders` and I wouldn't need to use `torchtext.vocab`. Also, we could:
- We could remove a lot of boilerplate, like, we could remove the "Prepare data processing pipelines" section of this tutorial:  https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html#prepare-data-processing-pipelines
- It'd better support a pipeline where the vocab size grows, over time. For example, in really large datasets, it is inefficient to read the entire dataset and tokenize the entire dataset, in order to extract the vocabulary. 

Furthermore, a `LazyEmbedding` could also be extended with a tokenizer, padding tokens, eos tokens, unknown tokens, pre-trained word vectors, etc. 

cc @albanD @mruberry @jbschlosser