File size: 5,925 Bytes
1d43d47 2c715b9 7a5dfbe 2c715b9 1d43d47 3471544 1d43d47 3471544 7a5dfbe 1d43d47 2171f3d 1d43d47 7a5dfbe 1d43d47 7a5dfbe 1d43d47 7a5dfbe 1d43d47 7a5dfbe 1d43d47 7a5dfbe 1d43d47 7a5dfbe |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 |
---
language:
- en
license: apache-2.0
datasets:
- teknium/OpenHermes-2.5
base_model: mistral-community/Mixtral-8x22B-v0.1
---
# Model Card for Fireworks Mixtral 8x22b Instruct OH
Fireworks Mixtral 8x22b Instruct OH is an instruct version of the latest MoE model from [mistralai](https://huggingface.co/mistralai) - [8x22b](https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1). This
model was finetuned on ~10K entries from [OpenHermes](https://huggingface.co/datasets/teknium/OpenHermes-2.5) dataset by [NousResearch](https://huggingface.co/NousResearch). Huge
shoutout to [Teknium](https://huggingface.co/teknium) and the NousResearch team for this high-quality SFT dataset.
## How to Get Started with the Model
Visit our [Playground](https://fireworks.ai/models/fireworks/mixtral-8x22b-instruct-preview) to try the model on a serverless platform, or access it via [API](https://readme.fireworks.ai/reference/createchatcompletion). Fine-tune it using our [guide](https://readme.fireworks.ai/docs/fine-tuning-models).
The model uses Vicuna chat template:
```
USER: ...
ASSISTANT: ...
```
or:
```
SYSTEM: ...
USER: ...
ASSISTANT: ...
```
### Training Data
To save training time, we sample & format 10K samples from [OpenHermes 2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5) dataset. We perform reformatting so as to
suit the vicuna chat template. Apart from that dataset is used as is.
#### Training Setup
The model was trained on 8xA100 [80G] using Deepspeed Zero 3. We had to fix an existing bug with deepspeed zero 3 where interaction between zero 3 & MoE causes
training to get stuck.
## Model Examination
<details Sample 1>
<summary>Itinerary Question</summary>
<code>
SYSTEM: You are a helpful assistant. Please be kind, patient and truthful in answering the questions.
USER: Create a travel itinerary for the bahamas. Spare no detail.
ASSISTANT: Sure, I'd be happy to help you plan a trip to the Bahamas! Here's a detailed itinerary for your consideration:
Day 1: Arrival in Nassau
- Arrive at Lynden Pindling International Airport in Nassau, the capital city of the Bahamas.
- Check into your hotel or resort.
- Spend the afternoon exploring the city, visiting places like the Queen's Staircase, Fort Fincastle, and the Straw Market.
- Enjoy a delicious Bahamian dinner at one of the local restaurants.
Day 2: Nassau City Tour and Atlantis Resort
- Start your day with a guided city tour of Nassau, visiting historical sites like the Government House, Parliament Square, and the Pirates of Nassau Museum.
- In the afternoon, head to the famous Atlantis Resort on Paradise Island. Explore the resort's attractions, such as the Aquaventure Water Park, the Marine Habitat, and the casino.
- End the day with a relaxing dinner at one of the resort's restaurants.
Day 3: Exuma Cays Day Trip
- Take a day trip to the Exuma Cays, a group of over 365 islands and cays.
- Visit popular spots like the Swimming Pigs at Big Major Cay, the Thunderball Grotto (featured in the James Bond movie "Thunderball"), and the stunning sandbars.
- Enjoy a picnic lunch on one of the pristine beaches.
- Return to Nassau in the evening.
Day 4: Blue Lagoon Island and Beach Day
- Spend the morning at Blue Lagoon Island, a private island just a short boat ride from Nassau.
- Enjoy activities like swimming, snorkeling, kayaking, or simply relaxing on the beach.
- In the afternoon, head back to Nassau and spend some time at one of the city's beautiful beaches, such as Cable Beach or Junkanoo Beach.
- Have dinner at a local seafood restaurant.
Day 5: Departure
- Depending on your flight schedule, you may have some time to explore Nassau further or do some last-minute shopping before heading to the airport for your departure.
I hope this itinerary helps you plan your trip to the Bahamas! Let me know if you have any other questions or need further assistance.
</code>
</details>
<details Sample 2>
<summary>QnA Question</summary>
<code>
SYSTEM: You are a helpful assistant. Please be kind, patient and truthful in answering the questions.
USER: What is the medical term for strong headache on one side of the head?
ASSISTANT: The medical term for a strong headache on one side of the head is called a migraine.
</code>
</details>
<details Sample 3>
<summary>Knowledge Question</summary>
<code>
SYSTEM: You are a helpful assistant. Please be kind, patient and truthful in answering the questions.
USER: How can I get stock price of stripe?
ASSISTANT: Stripe is a private company and does not have publicly traded stock. Therefore, you cannot get the stock price for Stripe.
</code>
</details>
|