YAML Metadata Warning:The pipeline tag "text2text-generation" is not in the official list: text-classification, token-classification, table-question-answering, question-answering, zero-shot-classification, translation, summarization, feature-extraction, text-generation, fill-mask, sentence-similarity, text-to-speech, text-to-audio, automatic-speech-recognition, audio-to-audio, audio-classification, audio-text-to-text, voice-activity-detection, depth-estimation, image-classification, object-detection, image-segmentation, text-to-image, image-to-text, image-to-image, image-to-video, unconditional-image-generation, video-classification, reinforcement-learning, robotics, tabular-classification, tabular-regression, tabular-to-text, table-to-text, multiple-choice, text-ranking, text-retrieval, time-series-forecasting, text-to-video, image-text-to-text, image-text-to-image, image-text-to-video, visual-question-answering, document-question-answering, zero-shot-image-classification, graph-ml, mask-generation, zero-shot-object-detection, text-to-3d, image-to-3d, image-feature-extraction, video-text-to-text, keypoint-detection, visual-document-retrieval, any-to-any, video-to-video, other
Model Card for Model ID
A new way to create embeddings
Model Details
Model Description
Auto-regressivly generates thought vectors (embeddings) for an input. This means fewer elements for the core model to process and thereby less compute to use. Additionally it can decode a thought vector back into a (vaugly) similar meaning in text. It doesn't focus on exact wording, but rather capturing the full meaning of the input.
As of now this is a prototype, not ready for full use. It proves the concept works, runs really fast, and vaugly grasps some concepts in english.
- Developed by: nochinator
- Model type: embeddings
- Language(s) (NLP): English
- License: AGPL-3.0
Model Sources
- Repository: https://huggingface.co/nochiantor/ThoughtVectors
- Paper: Comming soon?
- Demo: Comming soon?
Uses
Specifically built for use in chatbots, but the embeddings should apply for any NLP system that doesn't rely on percise wordings (eg. classification) then sentence level embeddings.
Direct Use
Comming soon.
Bias, Risks, and Limitations
Adds a small amount of compute to the front (and if using decoder, back) of the overall system compared to other embedding mechanisms. However, if the core model is larger then it should end up saving compute.
How to Get Started with the Model
This "model" is actually a collection of models that don't work without each other (excepting SentencePiece). A library is included in the files to manage it for you.
Training Data
https://huggingface.co/datasets/sentence-transformers/stsb https://huggingface.co/datasets/stanfordnlp/snli/tree/main/plain_text
Training Procedure
Trained by taking a sentence, tokenizing, passing through an encoder to get thought vectors then passing the vectors through the decoder to back tokens and comparing with original tokens, then backpropagating the error through both encoder and decoder. Slightly punishes for longer sets of vectors to encurage fewer vectors.
Training Hyperparameters
group_data="train.csv", test_data="val.csv", num_epochs=1000, batch_size=256, accum_steps=1, learning_rate=1e-4, weight_decay=2e-5, length_penalty=0.001, single_vector_prob=0.1, save_path="thought_vectors_prototype_0.2.tar", spm_model_prefix="spm", vocab_size=8192, d_model=512, encoder_nhead=8, decoder_nhead=8, encoder_layers=4, decoder_layers=4, max_thoughts=16, dropout=0.1, max_len=256, termination_threshold=0.8, patience=10
Stopped prematurely on stsb, around 5 epochs. Stopped after 1 epoch on snli.
Evaluation
Comming soon
Testing Data
https://huggingface.co/datasets/sentence-transformers/stsb
Model Architecture and Objective
Strings -> SentencePiece - Tokens -> Encoder -> Thought Vectors -> Thinker (your processing system) -> Thought Vectors (transformed, or just raw output) -> Decoder -> Tokens -> SentencePiece -> String
Glossary
"Thought Vector" is the name I have given to this type of embedding - a vector that represents thoughts rather than words or tokens. A "Thought" is a complete collection of thought vectors representing a full thought