Getting cut-off responses with Mixtral 8x7B-Instruct-v0.1 mostly in Date of Birth years
#182
by
keskival
- opened
We are using Mixtral for structured data extraction from unstructured sources.
We used to have the problem described here with missing spaces in the chat template: https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/discussions/176
The problem described there describes a similar problem with cutting off the responses prematurely by emitting a spurious stop token:
https://github.com/vllm-project/vllm/issues/2464
After fixing the chat template, the spurious cut-offs disappeared almost completely, but not totally. We still see spurious cut-offs sometimes, and typically they are well reproducible with the same prompt even with high temperature and top_P values.
It seems to always cut off in date of birth years or other years. See also similar experience here: https://github.com/vllm-project/vllm/issues/2464#issuecomment-1895321708
Are there cut-off JSONs in the Mistral training corpus?
Here is a prompt which reproduces the problem even with high temperature values (the content is fully synthetic, and doesn't correspond to real people, note that the triple backticks inside the prompt are written as three tildes to show it here nicely, HuggingFace forum Markdown doesn't seem to support escaping triple backticks):
You are a helpful doctor's assistant and a data entry expert filling in forms based on patient documents to make sure all information is stored correctly to allow for proper healthcare.
You are meticulous and thorough and understand how patients' well-being is based on correct information.
You will receive task instructions, a document chunk marked as
document-chunk and a JSON schema marked as first-consult-form-schema
defining a first consult preparation form.
You must perform this task in steps, as laid out in the following instructions, where each
step will be surrounded by the respective XML tags:
1. Facts
2. Relevance
3. Preparation form
Your response must to be in the form of:
<response>
<facts>
[free-form JSON]
</facts>
<relevance>
[free-form JSON]
</relevance>
<preparation-form>
[JSON compliant with the first-consult-form-schema]
</preparation-form>
</response>
Don't skip any step. Don't stop until you have finally closed the response tag, and do not write
anything after closing the response tag.
Start your response by writing a tag <response>.
The available documents which you will process one by one are as follows:
['referral letter - processed\n', 'MRI report - processed\n', 'CT scan report - processed\n', 'ultrasound imaging report - processed\n', 'pathology report - processing\n'].
The current document id: pathology report
The next document chunk to use to extract the facts, answer the questions and
fill up the triage form JSON is:
<document-chunk>
Pathology Report
Patient Name: John Doe
Patient ID: 123-45-6789
Date of Birth: January 1, 1970
Gender: Male
Referring Doctor: Dr. R.O. Bert
Referring Hospital: General Hospital Amsterdam
Referred to Doctor: Dr. De Heer Jansen
Referred to Hospital: NKI-AvL Cancer Hospital
Clinical History:
The patient, John Doe, was diagnosed with cancer on January 1, 2022. He started chemotherapy on February 1, 2022. On May 1, 2022, he was found to have metastasis to the head and neck region. Due to this finding, he has been referred to Dr. De Heer Jansen at NKI-AvL Cancer Hospital for further evaluation and management.
Pathology Findings:
The pathology report reveals the presence of squamous cell carcinoma in the head and neck region. The tumor is moderately differentiated and shows invasion into the surrounding tissues. Lymphovascular invasion is also noted. The tumor is positive for p16 immunostaining, indicating human papillomavirus (HPV) infection. The presence of HPV suggests a better prognosis for the patient.
Conclusion:
Based on the pathology findings, the patient, John Doe, has squamous cell carcinoma with metastasis to the head and neck region. The presence of HPV infection indicates a more favorable prognosis. Further evaluation and treatment are recommended to Dr. De Heer Jansen at NKI-AvL Cancer Hospital.
</document-chunk>
If you have some facts in the above partial answers for some fields you have in
the JSON schema, please, duplicate these answers to the JSON Schema fields
you are filling. It is more important to have these answers in the defined JSON Schema
fields than in free-form fields added outside the schema. Be conscientious in filling
up each of the fields defined in the JSON Schema, because answers from these go to the
final form to be submitted.
Let's start with the facts section:
Write <facts>.
First, let's make sure we see and understand every separate fact in the document-chunk.
Write a verbose and complete JSON which contains *every* fact from
the document-chunk separated into separate properties.
Be thorough here, by "every fact" I really mean every fact, irrespective
of importance or length. Don't try to be terse here, but don't write the whole chunk here either.
Extract facts only from the document-chunk, not from elsewhere in this text.
Your response here is a free-form JSON, do not follow the given JSON Schema with this.
The JSON Schema given later is for the preparation-form part later.
This is for your support to make later steps easier. Be sure to include every fact,
so that you have the best possible basis to perform the next steps!
A fact is a value of something, for example: "allergy: penicillin".
Feel free to use any convenient syntax, but for example this basic structure would be fine:
{"allergy": "penicillin"}
Then write </facts> to end this section and move on to the next step.
Write <relevance>, and then write a JSON which matches the JSON you produced in the
previous step, but now includes more information.
Do this in English, and translate values and facts to English where necessary.
This is a free-form JSON, do not follow the given JSON Schema with this.
In this JSON, for each fact mark two things:
1. How surprising this fact will be to a doctor from least surprising 1 to most surprising 5.
2. How much relevance will this fact have to the first consult meeting with the specialist,
or to the eventual carepath from least relevance 1 to most relevance 5.
Try to use every value from 1 to 5, because you want to determine relative, not
absolute importance of items. Relevance and surprise will determine the most important items
to highlight to the specialist, to the treating doctor eventually, because the most important
and the most surprising items need to be made sure to be seen.
Feel free to use any convenient syntax, but for example this basic structure would be fine:
[{"fact": "allergy: penicillin", "surprise": 4, "relevance": 5}]
Then write </relevance> to end this section.
Now, let's integrate this information to a backend system which provides all the case details
to the doctor in a standard form.
Write <preparation-form> to start this part.
Each property defined in the first-consult-form-schema
defines a form field. The first consult preparation form is sent to the treating doctor
for the first consult and this form is the basis the doctor uses to treat the patient.
Make sure it is perfectly filled!
Your task is to fill out this form to the best of your ability, without missing any clinically
relevant details. Missing details or summarizing too heavily will endanger the patient.
Make sure you include everything in the form!
You will need to consider each of the first-consult-form-schema form fields and
specifically their descriptions in order.
First consider the question or the instruction in the JSON schema property "description"
for the form field.
Consider whether the answer to this can be found in the chunk of the document.
The patient well-being and the quality of care is of utmost importance, so be
meticulous and thorough, the patient's life may depend on it!
When you provide an answer, designate the part where this information can be found with an excerpt
which matches only to that part of the document
in the property "sourceReferenceExcerpt". This excerpt must be the text excerpt mentioning
the fact verbatim. These references will be graphically presented to the user,
so that the part which excerpt matches to will be highlighted like a quote.
Hence, don't refer to the entirety of the document-chunk but only to the specific part
where the answer to the form field can be found by crafting a excerpt which matches
to that part exactly.
Mark the document from which the excerpt is from to the "sourceReferenceDocumentId"
property.
This is important because doctors and nurses will use these properties
to verify the information.
Then consider how this part answers the first-consult-form-schema field, and write
your thoughts into the "notes" property.
Use the provided notes property in the schema to reason step by step
about your answer before you fill in the answer property.
And after that, consider the document-chunk, the part of it where the answer was found,
your thoughts in the "notes" property and then write the answer to that form field
to the "answer" property.
Then continue with the subsequent form field and its description. You can work on the form
in this sort of a sequential order conveniently.
After having filled up the form, you should go back and check if there is
still something clinically relevant in the document-chunk which wasn't already
filled up to the form by you.
Leaving clinically relevant information out can be very dangerous to the patient.
Go through the fields in this form one by one, check the descriptions of each property
for instructions on how to fill them. Check the document-chunk if there is
an answer for the field there, and if so, write the notes, source and the answer to the field.
<first-consult-form-schema>
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"medications": {
"title": "Medications",
"description": "If the document mentions medications the patient is taking or has taken, list them in the field. Don't list allergies or such here, only medications which the patient is on. Note that this is an array where 'items' defines the items inside a plain array. Answers are plain strings. Put each family history fact in a separate item in this array.",
"type": [
"array",
"null"
],
"items": {
"type": "object",
"properties": {
"sourceReferenceDocumentId": {
"description": "Designate the document id from which document the information was found in and which document the below excerpt is from.",
"type": [
"string",
"null"
]
},
"sourceReferenceExcerpt": {
"description": "Use this field to reference a specific part of document text where you found the answer. This must be a verbatim excerpt which only matches to the part of the document where the information is extracted from. This will be used to find the part of the document with the answer in order to verify your answer. You must fill this if you fill in an answer to the field, because an answer without a source is worthless. Don't just write e.g. 'referral letter' here but instead repeat the part of the text where it was found here, verbatim. The subsequent 'answer' field needs to have the correct answer to the form field which doesn't necessarily exist in the referral letter in the same exact form.",
"type": [
"string",
"null"
]
},
"notes": {
"description": "Use this field to explain your answer and thinking before filling in the answer. Do also add the same information to the subsequent answer property as well.",
"type": [
"string",
"null"
]
},
"answer": {
"type": "string"
}
}
}
},
"Development feedback": {
"type": "object",
"properties": {
"notes": {
"description": "Use this field to explain your answer and thinking before filling in the answer. Do also add the same information to the subsequent answer property.",
"type": [
"string",
"null"
]
},
"answer": {
"description": "This is a compulsory item. Be as strict as Dr. Gregory House and be very critical of the instructions and the process with the aim of improving the process. As Dr. Gregory House, review the process: Fill here your critiques about the documents and this JSON Schema. For the referral letter, and other documents, focus on realism of the document, and whether the case is realistic and realistically described. For the JSON Schema, focus on whether it is valid JSON Schema or not, point out all the typos, mistakes and unclear expressions. Also check your own output so far whether it is correct against the schema, especially in relation to arrays. Check that all fields for which there is information available are actually filled, and point out if that doesn't seem to be the case. This won't go to the doctor but to the developers of this application as a feedback. Be very thorough here as well! Point out any confusing parts as well, and offer improvement suggestions! Consider this from the point of view of patient safety and data consistency as well. You can put here a long multi-line document. Any worries, development suggestions, general feelings go here as well. Note that this is for the feedback from you, the assistant filling up the form, not feedback given in the patient referral letter. Consider carefully that each form item was filled to the best of your ability. You might be held liable if the patient care was compromised if some relevant information was lost and not included in the First Consultation Form!",
"type": [
"string",
"null"
]
}
}
}
},
"additionalProperties": false
}
</first-consult-form-schema>
Be careful to produce exactly valid JSON with all the curly braces matched,
and no "..." for pretty printing or stylished quotes or anything like that.
Fill up the array type fields correctly, as per JSON Schema definition.
Note especially that there are no
"sourceReferenceExcerpt", "answer", or "notes" properties
for the whole array fields, but only for the leaf items contained within.
The type of the "answer" property is ALWAYS a plain string.
Array types in JSON Schema work as follows. The following example JSON Schema excerpt
designates an array of strings and nulls:
~~~
{
"type": "array",
"items": {
"type": ["string", "null"]
}
}
~~~
For example the following example array would be compliant with that example JSON Schema:
~~~
["2025-05-01", null, "cat"]
~~~
Now let's look a more complicated example.
In the schema you'll have things like this:
~~~
"pathologyResults": {
"type": [
"array",
"null"
],
"items": {
"type": "object",
"properties": {
"location": {
"type": "object",
"properties": {
"sourceReferenceDocumentId": {
"type": [
"string",
"null"
]
},
"sourceReferenceExcerpt": {
"type": [
"string",
"null"
]
},
"notes": {
"type": [
"string",
"null"
]
},
"answer": {
"type": [
"string",
"null"
]
}
}
},
"date": {
"type": "object",
"properties": {
"sourceReferenceDocumentId": {
"type": [
"string",
"null"
]
},
"sourceReferenceExcerpt": {
"type": [
"string",
"null"
]
},
"notes": {
"type": [
"string",
"null"
]
},
"answer": {
"type": [
"string",
"null"
],
"format": "date"
}
}
},
"tNumber": {
"type": "object",
"properties": {
"sourceReferenceDocumentId": {
"type": [
"string",
"null"
]
},
"sourceReferenceExcerpt": {
"type": [
"string",
"null"
]
},
"notes": {
"type": [
"string",
"null"
]
},
"answer": {
"type": [
"string",
"null"
]
}
}
},
"result": {
"type": "object",
"properties": {
"sourceReferenceDocumentId": {
"type": [
"string",
"null"
]
},
"sourceReferenceExcerpt": {
"type": [
"string",
"null"
]
},
"notes": {
"type": [
"string",
"null"
]
},
"answer": {
"type": [
"string",
"null"
]
}
}
}
}
}
}
~~~
The following JSON is compliant with the above example JSON Schema excerpt:
~~~
"pathologyResults": [
{
"location": {
"sourceReferenceDocumentId": "[put a real source document id here]",
"sourceReferenceExcerpt": "[put a real source excerpt here]",
"notes": "[put real notes here]",
"answer": "[put real answer here]"
},
"date": {
"sourceReferenceDocumentId": "[put a real source document id here]",
"sourceReferenceExcerpt": "[put a real source excerpt here]",
"notes": "[put real notes here]",
"answer": "[put real answer here]"
},
"tNumber": {
"sourceReferenceDocumentId": "[put a real source document id here]",
"sourceReferenceExcerpt": "[put a real source excerpt here]",
"notes": "[put real notes here]",
"answer": "[put real answer here]"
},
"result": {
"sourceReferenceDocumentId": "[put a real source document id here]",
"sourceReferenceExcerpt": "[put a real source excerpt here]",
"notes": "[put real notes here]",
"answer": "[put real answer here]"
}
},
{
"location": {
"sourceReferenceDocumentId": "[put a real source document id here]",
"sourceReferenceExcerpt": "[put a real source excerpt here]",
"notes": "[put real notes here]",
"answer": "[put real answer here]"
},
"date": {
"sourceReferenceDocumentId": "[put a real source document id here]",
"sourceReferenceExcerpt": "[put a real source excerpt here]",
"notes": "[put real notes here]",
"answer": "[put real answer here]"
},
"tNumber": {
"sourceReferenceDocumentId": "[put a real source document id here]",
"sourceReferenceExcerpt": "[put a real source excerpt here]",
"notes": "[put real notes here]",
"answer": "[put real answer here]"
},
"result": {
"sourceReferenceDocumentId": "[put a real source document id here]",
"sourceReferenceExcerpt": "[put a real source excerpt here]",
"notes": "[put real notes here]",
"answer": "[put real answer here]"
}
}
]
~~~
Note that in the above structure the top level array contains complex objects, and those complex
objects themselves contain four properties and the final answers are put to the respective "answer"
properties along with the sources and notes for that answer.
The "answer" property is the most important one. It is the contents of the form item
you are filling. If you don't have anything for the answer, leave out the whole item.
The final form will only have the "answer" property value visible, the other properties
are for supporting functionality only.
If there are no applicable test results or history, just output an empty array for such case,
like this:
~~~
"Pathology results": []
~~~
Similarly for example for Co-morbidities you might have a JSON Schema:
~~~
"coMorbidities": {
"type": [
"array",
"null"
],
"items": {
"type": "object",
"properties": {
"sourceReferenceDocumentId": {
"type": [
"string",
"null"
]
},
"sourceReferenceExcerpt": {
"type": [
"string",
"null"
]
},
"notes": {
"type": [
"string",
"null"
]
},
"answer": {
"type": "string"
}
}
}
}
~~~
For this schema, the following JSON would be compliant:
~~~
"coMorbidities": [
{
"sourceReferenceDocumentId": "[put a real source document id here]",
"sourceReferenceExcerpt": "[put a real source excerpt here]",
"notes": "[put real notes here]",
"answer": "[put real answer here]"
}
]
~~~
Note how that's a bit simpler, because inside the array you only have the form answer
objects themselves. Regardless, the form answer objects are a bit complex as they
have the source references and notes in addition to the answer which is the text
you actually write to the form field.
The schema you will need to respect will include array types in it, be sure to fill those
up properly. Typically you will encounter arrays where items are complex objects, not
just primitive types. It's complicated, but I know you can do it without any mistakes!
Now, let's make sure the patient will receive the best possible care by filling up
the first consult form perfectly so that the treating doctor will have all the information
they need to treat this patient!
Your response must be only valid JSON and must conform exactly to
the triage-form-schema JSON Schema. Look at the "description" fields in the schema
for instructions for filling the respective field. Make sure all fields are filled according
to information given in the document-chunk. Be conscientious to add ALL clinically relevant
information from the document and the extracted facts, so that the treating doctor will get
everything they need from the form!
Do not use references to the answers from the previous chunks using constructs like:
`...answersFromPreviousChunks`
The JSON parser doesn't support this; instead you need to repeat the answers verbatim.
Note that the root level of the above schema is an object, not an array. Look closely
if you need to close the braces whether they are supposed to be curly braces or brackets;
obviously braces must match the starting braces in all JSON.
After having written the whole JSON, write </preparation-form></response>.
Do not stop your response before closing the response tag; your response is complete
only after closing the preparation-form and response tags.
This must be the final thing you write, write nothing after it.
Do not worry about the length of the answer, by necessity you will have to write a really long
document as an answer.
The response for the single user chat message above is:
<response>
<facts>
{
"Patient Name": "John Doe",
"Patient ID": "123-45-6789",
"Date of Birth": "January 1, 1970",
"Gender": "Male",
"Referring Doctor": "Dr. R.O. Bert",
"Referring Hospital": "General Hospital Amsterdam",
"Referred to Doctor": "Dr. De Heer Jansen",
"Referred to Hospital": "NKI-AvL Cancer Hospital",
"Clinical History": "The patient, John Doe, was diagnosed with cancer on January 1, 2022. He started chemotherapy on February 1, 2022. On May 1, 2022, he was found to have metastasis to the head and neck region.",
"Pathology Findings": "The pathology report reveals the presence of squamous cell carcinoma in the head and neck region. The tumor is moderately differentiated and shows invasion into the surrounding tissues. Lymphovascular invasion is also noted. The tumor is positive for p16 immunostaining, indicating human papillomavirus (HPV) infection.",
"Conclusion": "Based on the pathology findings, the patient, John Doe, has squamous cell carcinoma with metastasis to the head and neck region. The presence of HPV infection suggests a better prognosis for the patient."
}
</facts>
<relevance>
[
{
"fact": "Patient Name: John Doe",
"surprise": 1,
"relevance": 5
},
{
"fact": "Patient ID: 123-45-6789",
"surprise": 1,
"relevance": 5
},
{
"fact": "Date of Birth: January 1, 197
We are somewhat capable of mitigating this by adding the following instruction:
I know you have a huge temptation to end the response in the middle of a birth date, but that
is not a normal thing to do, and you absolutely should produce a complete JSON and your answer
shouldn't stop before you have closed off the task XML tags.
The comment above does indeed mitigate the issue but not completely. Are there specific temperature/parameter settings you are using?
See the underlying problem here: https://github.com/mistralai/mistral-common/issues/3