Clearly something is wrong in debarta models, fails the simplest example!
Running the following "default" example input:
"Paris is the [MASK] of France."
It gives the following output:
"[
{
"score": 0.0018945893971249461,
"token": 6502,
"token_str": "idd",
"sequence": "Paris is theidd of France."
},
{
"score": 0.0012056089472025633,
"token": 12326,
"token_str": "race",
"sequence": "Paris is therace of France."
},
{
"score": 0.0011407239362597466,
"token": 35884,
"token_str": "ulpt",
"sequence": "Paris is theulpt of France."
},
{
"score": 0.0010635541984811425,
"token": 19605,
"token_str": "angled",
"sequence": "Paris is theangled of France."
},
{
"score": 0.0010488962288945913,
"token": 9056,
"token_str": "raw",
"sequence": "Paris is theraw of France."
}
]"
The issue seems to be the same regardless of the model size!
I was hoping this outperform the roberta model from facebook AI, but clearly not!
Perhaps something related to the model config on Huggingface!