README.md

---
license: apache-2.0
thumbnail: https://i.ibb.co/28dVbkB/alfred-mini-1.png
datasets:
- Anthropic/hh-rlhf
- OpenAssistant/oasst1
- databricks/databricks-dolly-15k
language:
- en
- fr
- de
- es
- it
tags:
- falcon-40b
- rlhf
- falcon
---
# Model Card for Alfred-40B-0723

![a witty and elegant butler with a falcon on his shoulder, smile, flat illustration, simple shapes, colorful, lo-fi aesthetics](https://i.ibb.co/28dVbkB/alfred-mini-1.png)

`Alfred-40B-0723` is a finetuned version of [Falcon-40B](https://huggingface.co/tiiuae/falcon-40b), obtained with Reinforcement Learning from Human Feedback (RLHF).
Finetuning was performed in July 2023. It is the first of a series of RLHF models based on Falcon-40B that will be regularly released. It is made available under the Apache 2.0 License.

## Model Details

### Model Description

- **Developed by:** [LightOn](https://www.lighton.ai/) -  [Axel Marmet](https://huggingface.co/WeightsnWizardry) (lead), [Oskar Hallström](https://huggingface.co/ohallstrom) (reward models), [Clément Thiriet](https://huggingface.co/cthiriet) (data infrastructure), [Julien Seailles](https://huggingface.co/Jseailleslighton), [Othman Hicheur](https://huggingface.co/othmanlighton), [Amélie Chatelain](https://huggingface.co/ameliechatelain) (data collection)
- **Model type:** Causal decoder-only;
- **Language(s) (NLP):** English, German, Spanish, French (and limited capabilities in Italian, Portuguese, Polish, Dutch, Romanian, Czech, Swedish);
- **License:** Apache 2.0 license.
- **Finetuned from model:** [Falcon-40B](https://huggingface.co/tiiuae/falcon-40b)
- **Training date:** July 2023 (`0723`).

## Uses

### Direct Use

`Alfred-40B-0723` can be used as an instruct or chat model. We encourage its usage for research on large language models finetuned with RLHF as well.

The prefix to use Alfred in chat mode is:

```
Alfred is a large language model trained by LightOn. Knowledge cutoff: November 2022. Current date: 31 July, 2023

User: {user query}
Alfred:
```

The stop word `User:` should be used.

### Out-of-Scope Use

Production use without adequate assessment of risks and mitigation; any use cases which may be considered irresponsible or harmful. 

## Bias, Risks, and Limitations

`Alfred-40B-0723` is a finetune of Falcon-40B. As such, it is trained mostly on English, German, Spanish, French, with limited capabilities also in in Italian, Portuguese, Polish, Dutch, Romanian, Czech, Swedish. It will not generalize appropriately to other languages. Furthermore, as it is trained on a large-scale corpora representative of the web, it will carry the stereotypes and biases commonly encountered online.

### Recommendations

We recommend users of `Alfred-40B-0723` to implement appropriate guardrails and precautions in any production use.

### Observed failure modes
From internal testing, the following failure modes have been observed:
* The model has a tendency to respond in Spanish to very short prompts in English, such as shorter greetings (e.g. "Hello", "Hi");
* At times, the model encloses its response in quotes;
* A times, the model adds a sentiment in brackets to its output (e.g. "[sadly] *model response*")

 These are mainly due to certain patterns prevalent in the open source datasets used, and will be adressed in future iterations of Alfred.

 If you encounter any other recurring failure modes, please open a community discussion, or contact us.

## How to Get Started with the Model

Use the code below to get started with the model.

```
from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model = "lightonai/alfred-40b-0723"
tokenizer = AutoTokenizer.from_pretrained(model)

pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)


sequences = pipeline(
   "Write a short text to announce that the new transformer model Alfred is available in open-source on Huggingface, include emojis.",
    max_length=200,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")
```

## Training Details

### Training Data

Alfred-40B-0723 was trained on a mixture of publicly available and in-house curated datasets. 

| **Data source**    |
|--------------------|
| [oasst1](https://huggingface.co/datasets/OpenAssistant/oasst1) | 
| [hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf) |
| [dolly](https://huggingface.co/datasets/databricks/databricks-dolly-15k)  |
| [NatInstV2](https://github.com/allenai/natural-instructions) |
| momentum-internal |

`momentum-internal` is a collection of prompts rated as gold quality from the staff of LightOn in their daily workflow.

### Training Procedure 

`Alfred-40B-0723` was trained on 128 A100 40GB GPUs, using a 3D parallelism strategy (TP=8, PP=4, DP=4) combined with ZeRO. The value model is initialized from the reward model and does not have any shared parameters with the policy network.

#### Preprocessing

Samples from each of the datasets have been programmatically formatted to chat, instructions and few-shot promtps.

#### Training Hyperparameters

##### Policy and Value Optimizer Config

| **Hyperparameter** | **Value**  | **Comment**                               |
|--------------------|------------|-------------------------------------------|
| Precision          | `bfloat16` |                                           |
| Optimizer          | AdamW      |                                                            |
| Learning rate      | 1.85e-6    | 10 warm-up steps, cosine decay over a 100 steps to 1.85e-7 |

##### Trainer config
| **Hyperparameter** | **Value**  |
|--------------------|------------|
| Num Rollouts       | 1024       | 
| Policy Epochs      | 1          | 
| Value Epochs       | 1          | 
| KL Coef            | 0.01       | 
| Gamma              | 1.0        | 
| GAE Lambda         | 0.95       | 
| Clip Range Policy  | 0.2        | 
| Clip Range Value   | 0.2        | 
| Whiten Advantages  | `true`     | 
| Whiten Rewards     | `false`    | 
| Score on EOD       | `true`     | 
| Max Steps          | 200        | 
| PPO steps/epoch    | 1          | 
| Value steps/epoch  | 8          | 

##### Trajectory data config
| **Hyperparameter**   | **Value**  |
|----------------------|------------|
| Continuation Max Len | 1024       |
| Continuation Min Len | 0          |
|                Top P | 1.0        |
|          Temperature | 1.0        |

##### Of interest to the community
The following hyper parameters have not been extensively explored and should not be taken as a gold standard:
- learning rate
- number of rollouts
- number of epochs
- steps per epoch 

## Evaluation

![aggregated evaluation of RAW vs SFT vs PPO - including random baseline - PPO suffers in arithmetic due to effects on calibration](https://i.ibb.co/9yQFJ40/aggregated.png "aggregated evaluation of RAW vs SFT vs PPO - including random baseline")

Initial evaluation results derived from the EleutherAI harness are as follows:

- Arithmetic capabilities exhibit a significant decline.
- Common Sense, Paraphrase, Reasoning, and Reading Comprehension remain relatively stable.
- Natural Language Inference (NLI) demonstrates improvement while Question Answering (QA) shows deterioration.

These outcomes align with existing literature expectations. It is worth noting that benchmark metrics do not necessarily align with human preferences. Moreover, all these metrics employ a Select methodology which penalizes RLHF models due to their sub-standard calibration compared to raw LLMs.

**Human evaluation is currently underway.**

### Compute Infrastructure

#### Hardware

Alfred-40B-0723 was trained on AWS SageMaker, on 128 A100 40GB GPUs in P4d instances.

#### Software

Alfred-40B-0723 was trained with a custom RLHF codebase. Training leverages a 3D parallelism approach combined with ZeRO, as well as high-performance kernels such as FlashAttention.

## Model Card Contact

Please open a Community Discussion for any support request related to using Alfred with HuggingFace transformers.

For any other inquiry: contact@lighton.ai