bf16_vs_fp8 / docs /commands /conv_release.md
zjasper666's picture
Upload folder using huggingface_hub
8655a4b verified
|
raw
history blame
552 Bytes

Chatbot Arena Conversations

  1. Gather battles
python3 clean_battle_data.py --max-num 10 --mode conv_release
  1. Tag OpenAI moderation
python3 tag_openai_moderation.py --in clean_battle_conv_20230814.json
  1. Clean PII

  2. Filter additional blocked words

python3 filter_bad_conv.py --in clean_battle_conv_20230630_tagged_v1_pii.json
  1. Add additional toxicity tag

All Conversations

  1. Gather chats
python3 clean_chat_data.py
  1. Sample
python3 conv_release_scripts/sample.py

Prompt distribution