Post
So, I have this idea to (potentially) improve uncertainty quantification for LLM hallucination detection.
The premise is that not all output tokens of a generated response share the same importance. Hallucinations are more dangerous in the form of a noun, date, number, etc.
The idea is to have a "token selection" layer that filters the output token probabilities sequence. Then, we use only the probabilities of the relevant tokens to calculate uncertainty quantification metrics.
The big question is how we know which tokens are the relevant ones. 🤔
My idea is to use the output sequence (decoded one) and use an NLP model (it doesn't need to be a fancy one) to do entity recognition and part-of-speech tagging to the output sequence and then do uncertainty quantification only on the entities that we have set as relevant (nouns, dates, numbers, etc).
What are your thoughts? Have you seen anyone try this before?
Curious to see if anyone has tried this before and know if this would have an impact on the correlation with human-annotated evaluations.
The premise is that not all output tokens of a generated response share the same importance. Hallucinations are more dangerous in the form of a noun, date, number, etc.
The idea is to have a "token selection" layer that filters the output token probabilities sequence. Then, we use only the probabilities of the relevant tokens to calculate uncertainty quantification metrics.
The big question is how we know which tokens are the relevant ones. 🤔
My idea is to use the output sequence (decoded one) and use an NLP model (it doesn't need to be a fancy one) to do entity recognition and part-of-speech tagging to the output sequence and then do uncertainty quantification only on the entities that we have set as relevant (nouns, dates, numbers, etc).
What are your thoughts? Have you seen anyone try this before?
Curious to see if anyone has tried this before and know if this would have an impact on the correlation with human-annotated evaluations.
