thenlper/gte-small
Sentence Similarity
•
Updated
•
1.08M
•
•
155
Vlad Iliescu
vladi
AI & ML interests
None yet
Recent Activity
liked
a model
27 days ago
thenlper/gte-small
liked
a model
27 days ago
mixedbread-ai/mxbai-embed-large-v1
liked
a model
27 days ago
thenlper/gte-large
Organizations
None yet
vladi's activity
liked
4
models
27 days ago
liked
a
model
28 days ago
liked
a
model
29 days ago
liked
2
Spaces
about 1 month ago
liked
a
model
about 1 month ago
liked
a
model
2 months ago
liked
a
model
12 months ago
reacted to
MoritzLaurer's
post with 👍
about 1 year ago
Post
Prompts are hyperparameters. Every time you test a different prompt on your data, you become less sure if the LLM actually generalizes to unseen data.
Issues of overfitting to a test set seem like concepts from boring times when people still fine-tuned models, but it's just as important for "zeroshot prompting". Using a separate validation split to tune the main hyperparameter of LLMs (the prompt) is just as important as train-val-test splitting for fine-tuning. The only difference is that you don't have a training dataset anymore and it somehow feels different because there is no training / no parameter updates.
Its easy to trick yourself into believing that an LLM performs well on your task, while you've actually overfit the prompt on your data. Every good "zeroshot" paper should clarify that they used a validation split for finding their prompt before final testing.
Issues of overfitting to a test set seem like concepts from boring times when people still fine-tuned models, but it's just as important for "zeroshot prompting". Using a separate validation split to tune the main hyperparameter of LLMs (the prompt) is just as important as train-val-test splitting for fine-tuning. The only difference is that you don't have a training dataset anymore and it somehow feels different because there is no training / no parameter updates.
Its easy to trick yourself into believing that an LLM performs well on your task, while you've actually overfit the prompt on your data. Every good "zeroshot" paper should clarify that they used a validation split for finding their prompt before final testing.
- 1 reply
liked
2
models
about 1 year ago
liked
a
model
over 1 year ago
liked
3
models
almost 2 years ago
