Love it!
Fantastic model! I've been playing with it for the last week. I was wondering, is the "summarize, simplify, and contextualize: " instruction the only instruction that works with this model, or could you just do something line "summarize and contextualize:", or "contextualize and correct punctuation:"? Is there a list of all possible instructions that you can provide to flan?
Hi, thanks for your attention and comment!
This is our baseline model. The only reason we put the instruction "summarize, simplify, and contextualize: " before an abstract is for a fair comparison to our full model. We don't need to prepend an instruction/task prefix if there is only one task. From t5 page :
According to this forum post, task prefixes matter when (1) doing multi-task training (2) your task is similar or related to one of the supervised tasks used in T5’s pre-training mixture (see Appendix D of the paper for the task prefixes used).
In our full model, we paired each instruction verb with a corpus
- "simplify: " with wikiauto
- "summarize: " with cnn_dm
- "contextualize: " with a new corpus from Science, where the context part of an editor abstract is grafted onto the corresponding paper's abstract for the purpose of recontextualization.
- "summarize, simplify, and contextualize: " with PNAS SAS corpus.
We show that the performance of main task (SAS) is boosted when adding more relevant tasks prepended with natural language instruction.
We haven't tested the instruction of "summarize and contextualize: ", but I like your idea! In theory, the model should generate shorter but contextualized summary with roughly the same readability. I think modularizing tasks with human language instruction is a fantastic idea and should be concerned by neural/data science researchers. We also haven' tested the "contextualize and correct punctuation:" instruction. But I think this is possible: with the recontextualization corpus from Science and randomly add/delete/substitute punctation, you can force the model to predict the corresponding recontextualized, clean version. If the goal is to test how well models like flan t5 can understand instructions and perform task mixtures (or even for zero-shot generalization), those ideas are all terrific and perhaps understudied. For now, we concern the last mile of scientific understanding.
I hope I made myself clear. And we will release our full model shortly, stay tuned!