Discussion
Can you tell me exactly how the datasets were used?
of course: wikimedia/wikipedia, used 8k from simple English, and some custom chit-chat examples (30~, good at those prompts - medium at other prompts, gonna fix that), i will try to make it better, as i know this is the best thing i got.
I will try to use an Q&A dataset (still choosing) and some custom generated 500~ chit examples.
but wikipedia doesent have chat? Isnt it used for pretraining
idk, i used wikipedia and chit chat, like wikipedia is q&a and chit chat learns how to really use this
i tried using alpaca cleaned only and it was better in some scenarios.
idk, i used wikipedia and chit chat, like wikipedia is q&a and chit chat learns how to really use this
Can you give me a link to chit chat
i created it, i used examples like:
Prompt: Hello
Assistant: Hello! How can i help you today?
i created it, i used examples like:
Prompt: Hello
Assistant: Hello! How can i help you today?
So its your own
yes, but you can generate some via llms, or i can generate 500~ examples for you?
yes, but you can generate some via llms, or i can generate 500~ examples for you?
Yes I wanted to do that but they say "that won't work because of the response limit" how do you get around that
Im going to try smol-smoltalk.
what response limit? i used claude / qwen for that
what response limit? i used claude / qwen for that
There is none, but i don't know why they still say it like here's a example from gemini
Generating 500 full text examples in a single response exceeds output token limits.
Which is wrong
huh? maybe you need to type a 500 examples dataset? actually idk.
huh? maybe you need to type a 500 examples dataset? actually idk.
its fine smol-smoltalk is very good dataset
is the model you fine tuned (the bananamind 2 mini as i know) good?
is the model you fine tuned (the bananamind 2 mini as i know) good?
i havnt yet im finetuning bananamind 2 medium on smol-smoltalk right now before the full release
okay, right know im fine tuning bananamind 2 mini on smol talk:
800 6.987823
850 7.030435
900 7.065323
950 7.052130
kinda bad, but i will try my best to make small models chattable.
okay, right know im fine tuning bananamind 2 mini on smol talk:
800 6.987823
850 7.030435
900 7.065323
950 7.052130
kinda bad, but i will try my best to make small models chattable.
make sure to use https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk not https://huggingface.co/datasets/HuggingFaceTB/smoltalk
because HuggingFaceTB/smoltalk made for 1B+ models
i know, the model is trying its best but hallucination rate is so big.
also know, Medium is releasing already in 3 hours
would be good to try your dataset you used for this on medium
3 epochs can be not enough or too much, will try 1 and then 5.
bananamind 2 medium is gonna be great on smol-smoltalk or on my custom mini dataset, also your fine tuned bananamind 2 nano on my q&a dataset (very simple one made by me) became actually usable, 70% of the time its right.
bananamind 2 medium is gonna be great on smol-smoltalk or my custom mini dataset, also your fine tuned bananamind 2 nano on q&a dataset (very simple one made by me) became actually usable, 70% of the time its right.
would really like if you release your custom datasets
i mean, they are not perfect but i will try my best to make them really unique, as i know, base 100m< models need something very simple, like hello > hello how can i help you today or what is the capital of france? > the capital of france is paris.
I'm finetuning medium on smol-smoltalk
step=000001 epoch=1 loss=1.8031 lr=2.000e-06 grad=0.759 tokens=98,304 assistant_tokens=74,964 assistant_ratio=76.3% tok/s=3,221
step=000010 epoch=1 loss=1.7261 lr=2.000e-05 grad=0.482 tokens=983,040 assistant_tokens=761,785 assistant_ratio=80.1% tok/s=111,768
step=000020 epoch=1 loss=1.6812 lr=4.000e-05 grad=0.362 tokens=1,966,080 assistant_tokens=1,516,886 assistant_ratio=74.4% tok/s=111,540
step=000030 epoch=1 loss=1.7243 lr=6.000e-05 grad=0.227 tokens=2,949,120 assistant_tokens=2,271,102 assistant_ratio=78.4% tok/s=110,526
step=000040 epoch=1 loss=1.6986 lr=8.000e-05 grad=0.199 tokens=3,932,160 assistant_tokens=3,026,718 assistant_ratio=71.4% tok/s=107,894
step=000050 epoch=1 loss=1.5920 lr=1.000e-04 grad=0.153 tokens=4,915,200 assistant_tokens=3,796,871 assistant_ratio=79.1% tok/s=108,610
step=000060 epoch=1 loss=1.6586 lr=1.200e-04 grad=0.165 tokens=5,898,240 assistant_tokens=4,556,800 assistant_ratio=73.6% tok/s=107,760
step=000070 epoch=1 loss=1.6753 lr=1.400e-04 grad=0.151 tokens=6,881,280 assistant_tokens=5,313,174 assistant_ratio=78.2% tok/s=107,804
step=000080 epoch=1 loss=1.6463 lr=1.600e-04 grad=0.155 tokens=7,864,320 assistant_tokens=6,054,607 assistant_ratio=76.7% tok/s=107,256
step=000090 epoch=1 loss=1.6679 lr=1.800e-04 grad=0.168 tokens=8,847,360 assistant_tokens=6,810,351 assistant_ratio=76.3% tok/s=107,640
I think you didnt mask the loss to response only?
my bad i forgot to do that
way better:
480 1.635868
490 1.537375
500 1.561093
510 1.638555
520 1.599631
oh my god, how hugging face translated this
way better:
480 1.635868
490 1.537375
500 1.561093
510 1.638555
520 1.599631
to this:
The Commission has also adopted a number of proposals for the establishment of a European Community framework for the implementation of the common agricultural policy.
oh my god, how hugging face translated this
way better:
480 1.635868
490 1.537375
500 1.561093
510 1.638555
520 1.599631
to this:
The Commission has also adopted a number of proposals for the establishment of a European Community framework for the implementation of the common agricultural policy.
facebook/nllb-200-distilled-600M sucks haha
Wait what?
My FInetuned Medium CAN WRITE PYTHON
It wrote this fully working:
def square_number(n):
return n ** 2
when will you release both?
when will you release both?
1PM Us Eastern time thats in 2 hours and 30 min
okay thanks
do i need to join the company?
do i need to join the company?
yes its private model then you can access
base is better than the chat one, base passed on few shot examples while chat didnt pass normal instruction following (but code quality is excellent)
base is better than the chat one, base passed on few shot examples while chat didnt pass normal instruction following (but code quality is excellent)
Can you give me some examples where chat fails
here:
[ TEST: Simple QA ]
The sky on a clear day is often referred to as the "cool" or "green" sky. This is because the light reflects off the clouds and into the atmosphere, creating a warm and vibrant atmosphere that is rich in light and color.
In a clear sky, you can expect to see a wide range of colors, from light blue and orange to deep green and indigo. The sky on a clear day is often the result of a combination of both natural light and artificial
[ TEST: Basic Math ]
When working with the absolute value of a number, we need to consider the magnitude of the number when taking its absolute value.
The absolute value of a number is a measure of how far it's "squared" (i.e., how much it's scaled up or down). In this case, it's positive because we're squaring the number, and the magnitude of the number is 1.
In other words, the result of the absolute value calculation is
[ TEST: Text Completion ]
A Clever Plan for a Dreamy Breakfast
In this sentence, the verb "complete" indicates that the cat is already in the plan, but the word "things" and "behind" indicate that the plan is a part of the breakfast.
[ TEST: Simple Code ]
def is_number_of_fruits():
fruits = ['apple', 'banana', 'cherry', 'date']
return fruits != ['apple', 'banana', 'cherry', 'date']
print(is_number_of_fruits("apple")) # True
print(is_number_of_fruits("banana")) # False
[ TEST: One Word Answer ]
The opposite of hot is hot, also known as fuzzy or fuzzy, or "fuzzy" in the context of the sentence, "The new policy has been implemented, all employees are required to do so."
In this sentence, "fuzzy" refers to a character that is not fully understood by the audience, but still conveys the intended meaning. It's a more subtle and specific way of expressing the idea.
By using "fuzzy
prompts:
test_cases = [
{
"name": "Simple QA",
"messages": [
{"role": "user", "content": "What color is the sky on a clear day?"}
]
},
{
"name": "Basic Math",
"messages": [
{"role": "user", "content": "What is 2 + 3?"}
]
},
{
"name": "Text Completion",
"messages": [
{"role": "user", "content": "Complete this sentence: The cat sat on the"}
]
},
{
"name": "Simple Code",
"messages": [
{"role": "user", "content": "Write a Python function that returns True."}
]
},
{
"name": "One Word Answer",
"messages": [
{"role": "user", "content": "What is the opposite of hot? Answer in one word."}
]
}
]
yeah code is perfect sometimes not gonna lie.
Are you using the correct chat template? And try repeat penalty 1.1 and temperature 0
yes:
[ TEST: Simple QA ]
The sky on a clear day is often referred to as "the sky on a clear day." This is because the sky is composed of several layers, including the atmosphere, which is composed of gases like oxygen and nitrogen. The atmosphere helps regulate the temperature and humidity levels in the air, making it ideal for photosynthesis.
In the case of the sky on a clear day, the atmosphere is composed of gases such as carbon dioxide, water vapor, and methane, which
[ TEST: Basic Math ]
To solve this problem, we need to isolate the variable x. We can do this by subtracting 3 from both sides of the equation:
2 + 3 = 4 - 3
2 = 1
Now that we have the value of x, we can substitute it back into either side of the equation to find the value of y:
2 + 3 = 4 - 1
2 = 1
So, the solution to the equation
[ TEST: Text Completion ]
the ground floor of a building.
[ TEST: Simple Code ]
def is_prime(n):
if n < 2:
return False
for i in range(2, int(n**0.5) + 1):
if n % i == 0:
return False
return True
[ TEST: One Word Answer ]
The opposite of hot is cold.
so it improved a little bit
i recommend experimenting with datasets in marimo (i do the same thing).
BananaMind Instruction Bench (Internal)
Overall:
BananaMind 2 Medium Chat 38%
Supra 1.5 50M Instruct* 21%
Supra 1.0 50M Instruct* 15%
General:
BananaMind 2 Medium Chat 22.5%
Supra 1.5 50M Instruct* 27.5%
Supra 1.0 50M Instruct* 27.5%
Multi-turn:
BananaMind 2 Medium Chat 56%
Supra 1.5 50M Instruct* 20%
Supra 1.0 50M Instruct* 0.0%
System Prompts:
BananaMind 2 Medium Chat 25%
Supra 1.5 50M Instruct* 5%
Supra 1.0 50M Instruct* 10%
Context Recall:
BananaMind 2 Medium Chat 60%
Supra 1.5 50M Instruct* 30%
Supra 1.0 50M Instruct* 20%
Code:
BananaMind 2 Medium Chat 80%
Supra 1.5 50M Instruct* 20%
Supra 1.0 50M Instruct* 0.0%
*BananaMind Instruction Bench (Internal) contains 100 examples total: 40 General, 25 Multi-turn, 20 System Prompts, 10 Context Recall, and
5 Code. Supra 1.5 50M Instruct received conversation context through its Supra 1.0 50M Instruct used 3.0×
linear RoPE scaling, extending its context from 1,024 to 3,072 tokens without additional long-context training to fit the context of the benchmark.
your model is obviously better than other models, but i mean, i asked it: Which color is the sky? > Some essay about atmosphere
but still on other questions its good
yes i know