Update README.md
Browse files
README.md
CHANGED
|
@@ -87,6 +87,9 @@ python /opt/NeMo-Aligner/examples/nlp/gpt/serve_reward_model.py \
|
|
| 87 |
|
| 88 |
2. Annotate data files using the served reward model. As an example, this can be the Open Assistant train/val files. Then follow the next step to train a SteerLM model based on [SteerLM training user guide](https://docs.nvidia.com/nemo-framework/user-guide/latest/modelalignment/steerlm.html#step-5-train-the-attribute-conditioned-sft-model) .
|
| 89 |
|
|
|
|
|
|
|
|
|
|
| 90 |
```python
|
| 91 |
python /opt/NeMo-Aligner/examples/nlp/data/steerlm/preprocess_openassistant_data.py --output_directory=data/oasst
|
| 92 |
|
|
|
|
| 87 |
|
| 88 |
2. Annotate data files using the served reward model. As an example, this can be the Open Assistant train/val files. Then follow the next step to train a SteerLM model based on [SteerLM training user guide](https://docs.nvidia.com/nemo-framework/user-guide/latest/modelalignment/steerlm.html#step-5-train-the-attribute-conditioned-sft-model) .
|
| 89 |
|
| 90 |
+
Please note that this script rounds the predicted floats to the nearest int (between 0 and 4 inclusive), as it's meant for SteerLM training.
|
| 91 |
+
For other use cases (e.g. reward bench measurement, response filtering/ranking), we recommend using the floats directly, which can be done by commenting out [two lines of code in NeMo-Aligner](https://github.com/NVIDIA/NeMo-Aligner/blob/main/examples/nlp/data/steerlm/attribute_annotate.py#L139-140)
|
| 92 |
+
|
| 93 |
```python
|
| 94 |
python /opt/NeMo-Aligner/examples/nlp/data/steerlm/preprocess_openassistant_data.py --output_directory=data/oasst
|
| 95 |
|