Hugging Face's logo

maciek-pioro
/
Mixtral-8x7B-v0.1-pl

Feature Extraction
Transformers
Safetensors
Polish
English
mixtral
MoE
Mixtral
text-generation-inference
Inference Endpoints
Model card Files Community
2
Mixtral-8x7B-v0.1-pl / README.md
maciek-pioro's picture
maciek-pioro
Update README.md
97518ad verified
preview code
|
raw
history blame
6.53 kB
metadata
library_name: transformers
tags:
  - MoE
  - Mixtral
license: apache-2.0
datasets:
  - togethercomputer/RedPajama-Data-1T
  - SpeakLeash
language:
  - pl
  - en

Mixtral-8x7B-v0.1-pl

Mixtral-8x7B-v0.1-pl is a Mixtral 8x7b model fine-tuned using 2.2B Polish tokens selected from the SpeakLeash. This is, to our knowledge, the first open-weights MoE model fine-tuned on Polish data. In order to preserve English capabilities, we include about 600M tokens from the RedPajama dataset.

The training was made possible thanks to TPU Research Cloud program. The model was trained on a TPUv3-256.

Our codebase is based on the Focused Transformer repo and EasyLM.

Example output

Poni偶szy artyku艂 opisuje histori臋 Polski. Polska to kraj, kt贸ry w ci膮gu wiek贸w by艂 艣wiadkiem wielu wa偶nych wydarze艅, kt贸re mia艂y wp艂yw na jej rozw贸j. W artykule przedstawiono najwa偶niejsze wydarzenia z historii Polski, od czas贸w staro偶ytnych do wsp贸艂czesno艣ci. Historia Polski - staro偶ytno艣膰 Polska to kraj, kt贸ry ma d艂ug膮 i bogat膮 histori臋. Pierwsze wzmianki o Polsce pochodz膮 z czas贸w staro偶ytnych. W 966 roku Mieszko I przyj膮艂 chrzest, co zapocz膮tkowa艂o proces ...

Model Details

Model Description

This is the model card of a 馃 transformers model that has been pushed on the Hub. This model card has been automatically generated.

  • Developed by: [More Information Needed]
  • Funded by [optional]: [More Information Needed]
  • Shared by [optional]: [More Information Needed]
  • Model type: [More Information Needed]
  • Language(s) (NLP): [More Information Needed]
  • License: [More Information Needed]
  • Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

  • Repository: [More Information Needed]
  • Paper [optional]: [More Information Needed]
  • Demo [optional]: [More Information Needed]

Uses

Direct Use

[More Information Needed]

Downstream Use [optional]

[More Information Needed]

Out-of-Scope Use

[More Information Needed]

Bias, Risks, and Limitations

[More Information Needed]

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Use the code below to get started with the model.

[More Information Needed]

Training Details

Training Data

[More Information Needed]

Training Procedure

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

  • Training regime: [More Information Needed]

Speeds, Sizes, Times [optional]

[More Information Needed]

Evaluation

Testing Data, Factors & Metrics

Testing Data

[More Information Needed]

Factors

[More Information Needed]

Metrics

[More Information Needed]

Results

[More Information Needed]

Summary

Model Examination [optional]

[More Information Needed]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

  • Hardware Type: [More Information Needed]
  • Hours used: [More Information Needed]
  • Cloud Provider: [More Information Needed]
  • Compute Region: [More Information Needed]
  • Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

[More Information Needed]

Citation [optional]

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

maciej.pioro at ideas-ncbr.pl