Request for Merlin test-set report-generation and evaluation script
Thank you for releasing Jolia, and for the significant effort that went into building and sharing this work!
I am interested in reproducing and comparing the report-generation results reported on the Merlin-Abdominal-CT test set.
Would it be possible to share the following for the Merlin test cases:
- The model-generated reports used for the reported evaluation
- The evaluation script or code used to compute the reported metrics (including any text preprocessing, normalization, or sectioning steps applied before scoring)
Having the exact predictions, reference formatting, and evaluation pipeline would greatly simplify reproducibility and enable direct comparison with other report-generation models.
Thank you again for your time and for making this work available to the community!
Dear Jonathan,
Thank you for your interest in Jolia. We're glad to share the model-generated reports at the following link:
https://drive.google.com/file/d/1U7BtSi8YcoDcUyzCzzgc6g8OC2ggqY4h
You'll find one .txt file per Merlin-Abdominal-CT study, each named by its study ID and containing the corresponding Jolia prediction.
Regarding evaluation, all reported metrics were computed with RadEval (https://github.com/jbdel/RadEval).
Best,
Charles
Dear Charles,
Thank you very much for sharing the predictions and for the quick reply! I was able to download them, but I still have a follow-up question about the study IDs.
The prediction filenames use 10-character hexadecimal IDs, such as '000cb1205b', whereas the Merlin test set available to me uses study IDs such as 'AC4213645'. I was unable to match these identifiers directly or using standard hashing methods.
Could you please clarify how the prediction filenames map to the Merlin study IDs? Maybe there was a hashing step, or do you possibly have a different CSV containing these identifiers?
Best,
Jonathan
Oh yes indeed.
I have regenerated the ZIP with the original Merlin study IDs and updated the previous link.
Should be good now 😄