Sorry everyone this got sort of popular but it doesnt generate understandable text - I think there's a way to make this generate good results w/ relatively little compute I'll experiment a bit later

LMLab Mistral 1B Untrained

This is an untrained base model modified from Mistral-7B-Instruct. It has 1.13 billion parameters.

Untrained

This model is untrained. This means it will not generate comprehensible text.

Model Details

Model Description

Developed by: LMLab
License: Apache 2.0
Parameters: 1.13 billion (1,134,596,096)
Modified from model: mistralai/Mistral-7B-v0.1

Model Architecture

LMLab Mistral 1B is a transformer model, with the following architecture choices:

Grouped-Query Attention
Sliding-Window Attention
Byte-fallback BPE tokenizer

Usage

Use MistralForCausalLM.

from transformers import MistralForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('lmlab/lmlab-mistral-1b-untrained')
model = MistralForCausalLM.from_pretrained('lmlab/lmlab-mistral-1b-untrained')
text = "Once upon a time"
encoded_input = tokenizer(text, return_tensors='pt')
output = model.generate(**encoded_input)
print(tokenizer.decode(output[0]))

Notice

This model does not have any moderation systems.