initial push

.gitattributes

@@ -29,3 +29,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/PromptSource[[:space:]]ACL[[:space:]]Demo[[:space:]]Figure.png filter=lfs diff=lfs merge=lfs -text

API_DOCUMENTATION.md

@@ -0,0 +1,40 @@
+# Manipulating prompts
+PromptSource implements 4 classes to store, manipulate and use prompts and their metadata: `Template`, `Metadata`, `DatasetTemplates` and `TemplateCollection`. All of them are implemented in [`templates.py`](promptsource/templates.py)
+
+## Class `Template` and `Metadata`
+`Template` is a class that wraps a prompt, its associated metadata, and implements the helper functions to use the prompt.
+
+Instances of `Template` have the following main methods that will come handy:
+* `apply(example, truncate=True, highlight_variables=False)`: Create a prompted example by applying the template to the given example
+  - `example` (Dict): the dataset example to create a prompt for
+  - `truncate` (Bool, default to `True`): if True, example fields will be truncated to `TEXT_VAR_LENGTH` chars
+  - `highlight_variables`(Bool, default to `False`): highlight the added variables (internal use for the app rendering)
+* `get_id()`: Get the uuid of the prompt
+* `get_name()`: Get the name of the prompt
+* `get_reference()`: Get any additional information about the prompt (such as bibliographic reference)
+* `get_answer_choices_list(example)`: If applicable, returns a list of answer choices for a given example.
+
+Each `Template` also has a `metadata` attribute, an instance of the class `Metadata` that encapsulates the following 3 attributes:
+* `original_task`: If True, this prompt asks a model to perform the original task designed for this dataset.
+* `choices_in_prompt`: If True, the answer choices are included in the templates such that models see those choices in the input. Only applicable to classification tasks.
+* `metrics`: List of strings denoting metrics to use for evaluation
+
+## Class `DatasetTemplates`
+`DatasetTemplates` is a class that wraps all the prompts (each of them are instances of `Template`) for a specific dataset/subset and implements all the helper functions necessary to read/write to the YAML file in which the prompts are saved.
+
+You will likely mainly be interested in getting the existing prompts and their names for a given dataset. You can do that with the following instantiation:
+```python
+>>> template_key = f"{dataset_name}/{subset_name}" if subset_name is not None else dataset_name
+>>> prompts = DatasetTemplates(template_key)
+>>> len(prompts) # Returns the number of prompts for the given dataset
+>>> prompts.all_template_names # Returns a sorted list of all templates names for this dataset
+```
+
+## Class `TemplateCollection`
+`TemplateCollection` is a class that encapsulates all the prompts available under PromptSource by wrapping the `DatasetTemplates` class. It initializes the `DatasetTemplates` for all existing template folders, gives access to each `DatasetTemplates`, and provides aggregated counts overall `DatasetTemplates`.
+
+The main methods are:
+* `get_dataset(dataset_name, subset_name)`: Return the DatasetTemplates object corresponding to the dataset name
+  - `dataset_name` (Str): name of the dataset to get
+  - `subset_name` (Str, default to None): name of the subset
+* `get_templates_count()`: Return the overall number count over all datasets. NB: we don't breakdown datasets into subsets for the count, i.e subsets count are included into the dataset count

CITATION.cff

@@ -0,0 +1,118 @@
+cff-version: "0.2.2"
+date-released: 2022-02
+message: "If you use this software, please cite it using these metadata."
+title: "PromptSource"
+url: "https://github.com/bigscience-workshop/promptsource"
+authors:
+  - family-names: Bach
+    given-names: "Stephen H."
+  - family-names: Sanh
+    given-names: Victor
+  - family-names: Yong
+    given-names: Zheng-Xin
+  - family-names: Webson
+    given-names: Albert
+  - family-names: Raffel
+    given-names: Colin
+  - family-names: Nayak
+    given-names: "Nihal V."
+  - family-names: Sharma
+    given-names: Abheesht
+  - family-names: Kim
+    given-names: Taewoon
+  - family-names: Bari
+    given-names: "M Saiful"
+  - family-names: Fevry
+    given-names: Thibault
+  - family-names: Alyafeaiu
+    given-names: Zaid
+  - family-names: Dey
+    given-names: Manan
+  - family-names: Santilli
+    given-names: Andrea
+  - family-names: Sun
+    given-names: Zhiqing
+  - family-names: Ben-David
+    given-names: Srulik
+  - family-names: Xu
+    given-names: Canwen
+  - family-names: Chhablani
+    given-names: Gunjan
+  - family-names: Wang
+    given-names: Han
+  - family-names: Fries
+    given-names: "Jason Alan"
+  - family-names: Al-shaibani
+    given-names: "Maged S."
+  - family-names: Sharma
+    given-names: Shanya
+  - family-names: Thakker
+    given-names: Urmish
+  - family-names: Almubarak
+    given-names: Khalid
+  - family-names: Tang
+    given-names: Xiangru
+  - family-names: Tian-Jian
+    given-names: Mike
+  - family-names: Rush
+    given-names: "Alexander M."
+preferred-citation:
+  type: article
+  authors:
+  - family-names: Bach
+    given-names: "Stephen H."
+  - family-names: Sanh
+    given-names: Victor
+  - family-names: Yong
+    given-names: Zheng-Xin
+  - family-names: Webson
+    given-names: Albert
+  - family-names: Raffel
+    given-names: Colin
+  - family-names: Nayak
+    given-names: "Nihal V."
+  - family-names: Sharma
+    given-names: Abheesht
+  - family-names: Kim
+    given-names: Taewoon
+  - family-names: Bari
+    given-names: "M Saiful"
+  - family-names: Fevry
+    given-names: Thibault
+  - family-names: Alyafeaiu
+    given-names: Zaid
+  - family-names: Dey
+    given-names: Manan
+  - family-names: Santilli
+    given-names: Andrea
+  - family-names: Sun
+    given-names: Zhiqing
+  - family-names: Ben-David
+    given-names: Srulik
+  - family-names: Xu
+    given-names: Canwen
+  - family-names: Chhablani
+    given-names: Gunjan
+  - family-names: Wang
+    given-names: Han
+  - family-names: Fries
+    given-names: "Jason Alan"
+  - family-names: Al-shaibani
+    given-names: "Maged S."
+  - family-names: Sharma
+    given-names: Shanya
+  - family-names: Thakker
+    given-names: Urmish
+  - family-names: Almubarak
+    given-names: Khalid
+  - family-names: Tang
+    given-names: Xiangru
+  - family-names: Tian-Jian
+    given-names: Mike
+  - family-names: Rush
+    given-names: "Alexander M."
+  title: "PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts"
+  year: 2022
+  publisher: "arXiv"
+  url: "https://arxiv.org/abs/2202.01279"
+  address: "Online"

CODEOWNERS

@@ -0,0 +1 @@
+@bigscience-workshop/promptsource-codeowners

CONTRIBUTING.md

@@ -0,0 +1,321 @@
+# Contributing
+
+The best way to contribute growing P3 is by writing prompts for new datasets!
+
+### What are Prompts?
+
+A prompt consists of a template: input template and target template, along with collection of associated metadata. A template is a piece of code written in a templating language called
+[Jinja](https://jinja.palletsprojects.com/en/3.0.x/). A template defines
+a function that maps an example from a dataset in the
+[Hugging Face datasets library](https://huggingface.co/datasets) to two strings of
+text. The first is called the _input_ which provides all information that
+will be available to solve a task, such as the instruction and the context.
+The second piece is called the _target_, which is the desired response to the
+prompt.
+
+### Quick-Start Guide to Writing Prompts
+
+1. **Set up the app.** Fork the app and set up using the
+[README](https://github.com/bigscience-workshop/promptsource/blob/main/README.md).
+1. **Examine the dataset.** In the "Sourcing" mode, select or type the dataset into the dropdown.
+If the dataset has subsets (subsets are not the same as splits), you can select
+which one to work on. Note that prompts are subset-specific. You can find
+out background information on the dataset by reading the information in the
+app. The dataset is a collection of examples, and each example is a Python
+dictionary. The sidebar will tell you the schema that each example has.
+1. **Start a new prompt**. Enter a name for your first prompt and hit "Create."
+You can always update the name later. If you want to cancel the prompt, select
+"Delete Prompt."
+1. **Write the prompt**. In the box labeled "Template," enter a Jinja expression.
+See the [getting started guide](#getting-started-using-jinja-to-write-prompts)
+and [cookbook](#jinja-cookbook) for details on how to write templates.
+1. **Fill in metadata**. Fill in the metadata for the current prompt: reference, original task, choices in templates, metrics, languages, and answer choices.
+See [Metadata](#metadata) for more details about these fields.
+1. **Save the prompt**. Hit the "Save" button. The output of the prompt
+applied to the current example will appear in the right sidebar.
+1. **Verify the prompt**. Check that you didn't miss any case by scrolling
+through a handful of examples of the prompted dataset using the
+"Prompted dataset viewer" mode.
+1. **Write between 5 and 10 prompts**. Repeat the steps 4 to 9 to create between 5
+and 10 (more if you want!) prompts per dataset/subset. Feel free to introduce
+a mix of formats, some that follow the templates listed in the [best practices](#best-practices)
+and some that are more diverse in the format and the formulation.
+1. **Duplicate the prompts(s).** If the dataset you have chosen bear the same
+format as other datasets (for instance, `MNLI` and `SNLI` have identical formats),
+you can simply duplicate the prompts you have written to these additional datasets.
+1. **Upload the template(s).** Submit a PR using the instructions
+[here](#uploading-prompts).
+
+## Getting Started Using Jinja to Write Prompts
+
+Here is a quick crash course on using [Jinja](https://jinja.palletsprojects.com/en/3.0.x/)
+to write templates. More advanced usage is in the [cookbook](#jinja-cookbook).
+
+Generally, in a template, you'll want to use a mix of hard-coded data that is
+task-specific and stays the same across examples, and commands that tailor the
+input and target to a specific example.
+
+To write text that should be rendered as written, just write it normally. The
+following "template" will produce the same text every time:
+```jinja2
+This is just literal text that will be printed the same way every time.
+```
+
+To make your template do something more interesting, you'll need to use Jinja
+expressions. Jinja expressions are surrounded by curly braces `{` and `}`.
+One common thing you'll want to do is access information in the dataset example.
+When applied to an example, you can access any value in the example dictionary
+via its key. If you just want to print that value surround it in double curly
+braces. For example, if you want to print a value with the key `text`, use this:
+```jinja2
+The text in this example is {{ text }}.
+```
+
+You can also use information from the example to control behavior. For example,
+suppose we have a label with the key `label` in our example, which either has a
+value of 0 or 1. That's not very "natural" language, so maybe we want to decide
+which label name to use based on the example. We can do this by creating a list
+and indexing it with the example key:
+```jinja2
+The label for this example is {{ ["Label A", "Label B"][label] }}.
+```
+We can also use dictionaries for the same thing:
+```jinja2
+The label for this example is {{
+{"a": "Label A",
+ "b": "Label B"
+}[label]
+}}.
+```
+
+Note that some things in a template are particular to the task, and should not be
+modified by downstream steps that try to increase the diversity of the prompts.
+A common example is listing label names in the prompt to provide choices. Anything
+that should not be modified by data augmentation should be surrounded by double
+curly braces and quoted. For example:
+```jinja2
+The choices are {{"a"}}, {{"b"}}, and {{"c"}}.
+```
+You can leave binary options like yes/no, true/false, etc. unprotected.
+
+Finally, remember that a template must produce two strings: an input and a target.
+To separate these two pieces, use three vertical bars `|||`.
+So, a complete template for Squad could be:
+```jinja2
+I'm working on the final exam for my class and am trying to figure out the answer
+to the question "{{question}}" I found the following info on Wikipedia and I think
+it has the answer. Can you tell me the answer?
+{{context}}
+|||
+{{answers["text"][0]}}'
+```
+
+## Metadata
+In addition to the template itself, you need to fill out several other fields.
+These metadata facilitate finding and using the prompts.
+* **Prompt Reference.** If your template was inspired by a paper, note the
+reference in the "Prompt Reference" section. You can also add a description of
+what your template does.
+* **Original Task?** The checkbox should be checked if the template requires solving a
+task that the underlying dataset is used to study. For example, a template that asks a
+question from a question answering dataset would be an original task template, but one that asks
+to generate a question for a given answer would not.
+* **Choices in Template?** The checkbox should be checked if the input explicitly indicates
+the options for the possible outputs (regardless of whether `answer_choices` is used).
+* **Metrics.** Use the multiselect widget to select all metrics commonly used to evaluate
+this task. Choose “Other” if there is one that is not included in the list.
+* **Languages.** Use the multiselect widget to select all languages used in the prompt. This is independent of what languages are used in the underlying dataset. For example, you could have an English prompt for a Spanish dataset.
+* **Answer Choices.**  If the prompt has a small set of possible outputs (e.g., Yes/No,
+class labels, entailment judgements, etc.), then the prompt should define and use answer
+choices as follows. This allows evaluation to consider just the possible targets for
+scoring model outputs. The answer choices field is a Jinja expression that should produce
+a `|||` separated list of all possible targets. If the choices don't change from example
+to example, then you can just list them. For example, AG News is
+```jinja2
+World News ||| Sports ||| Business ||| Science and Technology
+```
+Note that whitespace is stripped from the ends of the choices. If answer choices are set,
+then they are also available to Jinja in the prompt itself in the form of a list called
+`answer_choices`. You should use this list in both input and target templates so that the
+resulting inputs and targets match the answer choices field exactly. For example, a prompt
+for AG News could use `answer_choices` like this:
+```jinja2
+{{text}} Which of the following sections of a newspaper would
+this article likely appear in? {{answer_choices[0]}}, {{answer_choices[1]}},
+{{answer_choices[2]}}, or {{answer_choices[3]}}?
+|||
+{{ answer_choices[label] }}
+```
+Since Answer Choices is a Jinja expression that has access to the example, it can also be used
+to extract example-specific choices from the underlying data. For example, in AI2 ARC, we could
+use
+```jinja2
+{{choices.text | join("|||")}}
+```
+
+## Best Practices
+
+* **Writing target templates.** The target template should only contain the answer to the task.
+It should not contain any extra text such as “The answer is…” (unless that extra text is also in
+`answer_choices`). If `answer_choices` is populated, the output should only contain the values
+in `answer_choices`.
+* **Formatting multple-choice questions.** If the target should match the name of the choice
+(e.g., “World News”), then it should list the choices either as part of a grammatical question
+or a list with the marker for each (e.g, dashes). If the target should indicate the choice from
+the list (e.g., “A,” “Explanation 1,” etc.), then it should list the choices with the indicator
+before each one.
+* **Choosing input and target pairs.** Lots of datasets have multiple columns that can be
+combined to form different (input, target) pairs i.e. different "tasks". Don't hesitate to
+introduce some diversity by prompting a given dataset into multiple tasks and provide some
+description in the "Template Reference" text box. An example is given
+in the already prompted `movie_rationales`.
+* **Filtering prompts.** If a prompt is applied to an example and produces an
+empty string, that prompt/example pair will be skipped.
+You can therefore create prompts that only apply to a subset of the examples by
+wrapping them in Jinja if statements. For example, in the `TREC` dataset, there
+are fine-grained categories that are only applicable to certain coarse-grained categories.
+We can capture this with the following prompt:
+```jinja2
+{% if label_coarse == 0 %}
+Is this question asking for a {{"definition"}}, a {{"description"}}, a {{"manner of action"}}, or a {{"reason"}}?
+{{text}}
+|||
+{{ {0: "Manner", 7: "Defintion", 9: "Reason", 12: "Description"}[label_fine] }}
+{% endif %}
+```
+For datasets that have splits with no labels (for instance test split without ground truth labels), you can wrap the conditional statement on the target side.
+For instance for `super_glue/boolq`, the following prompt would return an empty target on the test split, but not an empty prompted example:
+```jinja2
+{{ passage }}
+Question: {{ question }}
+Answer:
+|||
+{% if label != -1 %}
+{{ answer_choices[label] }}
+{% endif %}
+```
+* **Conditional generation format.** Always specify the target and separate it from the prompt
+by indicating the vertical bars `|||`. The target will be generated by a generative model
+conditioned on the input you wrote. You can always transform an "infix" prompt format
+```jinja2
+Given that {{premise}}, it {{ ["must be true", "might be true", "must be false"][label] }} that {{hypothesis}}
+```
+into a conditional generation format
+```jinja2
+Given that {{premise}}, it {{ "must be true, might be true, or must be false" }} that {{hypothesis}}?|||
+{{ ["must be true", "might be true", "must be false"][label] }}
+```
+* **Pre-defined formats.** The goal is to collect a diverse set of prompts with diverse formats, but
+we also want to include a few less diverse prompts that follow the following two structures:
+1) A question-answer pair with optional multiple choices like:
+```
+[Context]                         # optional depending on the task
+[Question]
+[Label1], [Label2], [Label3]      # optional depending on the task
+```
+So for SNLI it will look like:
+```jinja2
+{{premise}}
+Is it the case that {{hypothesis}}?
+{{ "Yes" }}, {{ "No" }}, {{ "Maybe" }} ||| {{ ["Yes", "No", "Maybe"][label] }}
+```
+
+2) Task description followed by the input. So for SNLI it will look like:
+```jinja2
+Determine the relation between the following two sentences. The relations are entailment, contradiction, or neutral.
+{{premise}}
+{{hypothesis}} ||| {{label}}
+```
+* **Setting variables.** You might want to use the Jinja expression `{% set %}` to define a variable. If you do,
+do it at the beginning of the prompt, outside any conditional statements, so that the automatic prompt checks
+recognize that the variable is defined.
+
+## More Examples
+
+Here are a few interesting examples of prompts with explanations.
+
+Here's one for `hellaswag`:
+```jinja2
+First, {{ ctx_a.lower() }} Then, {{ ctx_b.lower() }}...
+
+Complete the above description with a chosen ending:
+
+(a) {{ answer_choices[0] }}
+
+(b) {{ answer_choices[1] }}
+
+(c) {{ answer_choices[2] }}
+
+(d) {{ answer_choices[3] }}
+
+||| {{ answer_choices[label | int()] }}
+```
+Notice how it uses functions to consistently capitalize the information and provides lots
+of context (referring explicitly to "description" and "chosen ending.")
+
+Here's one for `head_qa`:
+```jinja2
+Given this list of statements about {{category}}: {{ answers | map(attribute="atext")
+| map("lower") | map("trim", ".") | join(", ") }}.
+Which one is the most appropriate answer/completion for the paragraph that follows?
+{{qtext}}
+|||
+{% for answer in answers if answer["aid"]==ra -%}
+{{answer["atext"]}}
+{%- endfor %}
+```
+Like above, it uses functions to present the choices in a readable way. Also, it
+uses a for loop with conditions to handle the more intricate dataset schema.
+
+Here's one for `paws`:
+```jinja2
+Sentence 1: {{sentence1}}
+Sentence 2: {{sentence2}}
+Question: Does Sentence 1 paraphrase Sentence 2? Yes or No?
+|||
+{{answer_choices[label]}}
+```
+Notice that the choices `Yes or No` are not escaped. Yes/no, true/false
+are choices that do not need to be escaped (unlike categories).
+
+## Uploading Prompts
+
+Once you save or modify a template, the corresponding file inside the `templates`
+directory in the repo will be modified. To upload it, follow these steps:
+1. Run `make style` and `make quality`.
+2. Commit the modified template files (anything under `templates`) to git.
+3. Push to your fork on GitHub.
+4. Open a pull request against `main` on the PromptSource repo.
+
+
+## Jinja Cookbook
+
+- Accessing nested attributes of a dict
+```jinja
+{{ answers_spans.spans }}
+```
+
+- Joining list
+```jinja=
+{{ spans_list | join(", ") }}
+```
+
+- If conditions
+```jinja
+{% if label==0 %}
+do_something
+{% elif condition %}
+do_something_else
+{% endif %}
+```
+- Using `zip()` to zip multiple lists
+```jinja
+{% for a, b in zip(list_A, list_B) %}
+do_something_with_a_and_b
+{% endfor %}
+```
+
+
+Jinja includes lots of complex features but for most instances you likely only
+need to use the methods above. If there's something you're not sure how to do,
+just open an issue. We'll collect other frequent patterns here.

LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

Makefile

@@ -0,0 +1,16 @@
+.PHONY: quality style
+
+check_dirs := promptsource
+
+# Check that source code meets quality standards
+
+quality:
+	black --check --line-length 119 --target-version py38 $(check_dirs)
+	isort --check-only $(check_dirs)
+	flake8 $(check_dirs) --max-line-length 119
+
+# Format source code automatically
+
+style:
+	black --line-length 119 --target-version py38 $(check_dirs)
+	isort $(check_dirs)

README.md

@@ -4,9 +4,13 @@ emoji: 👁
 colorFrom: red
 colorTo: blue
 sdk: streamlit
-sdk_version: 1.10.0
-app_file: app.py
+sdk_version: 0.82
+app_file: promptsource/app.py
 pinned: false
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+PromptSource is a toolkit for creating, sharing and using natural language prompts. This Space is a hosted demo of Promptsource and allows you to browse through existing prompts.
+
+More information about Promptsource and how to use it is available on the [Github repository](https://github.com/bigscience-workshop/promptsource).
+
+NB: As of now, this Space is not synched with the Github repository automatically and captures the state of the repository on October 21, 2022.

README_promptsource.md

@@ -0,0 +1,140 @@
+# PromptSource
+**PromptSource is a toolkit for creating, sharing and using natural language prompts.**
+
+Recent work has shown that large language models exhibit the ability to perform reasonable zero-shot generalization to new tasks. For instance, [GPT-3](https://arxiv.org/abs/2005.14165) demonstrated that large language models have strong zero- and few-shot abilities. [FLAN](https://arxiv.org/abs/2109.01652) and [T0](https://arxiv.org/abs/2110.08207) then demonstrated that pre-trained language models fine-tuned in a massively multitask fashion yield even stronger zero-shot performance. A common denominator in these works is the use of prompts which have gathered of interest among NLP researchers and engineers. This emphasizes the need for new tools to create, share and use natural language prompts.
+
+Prompts are functions that map an example from a dataset to a natural language input and target output PromptSource contains a growing collection of prompts (which we call **P3**: **P**ublic **P**ool of **P**rompts). As of January 20, 2022, there are ~2'000 English prompts for 170+ English datasets in [P3](https://huggingface.co/datasets/bigscience/P3).
+
+<p align="center">
+  <img src="assets/PromptSource ACL Demo Figure.png" width="800"/>
+</p>
+
+PromptSource provides the tools to create, and share natural language prompts (see [How to create prompts](#how-to-create-prompts), and then use the thousands of existing and newly created prompts through a simple API (see [How to use prompts](#how-to-use-prompts)). Prompts are saved in standalone structured files and are written in a simple templating language called Jinja. An example of prompt availabe in PromptSource for [SNLI](https://huggingface.co/datasets/snli) is:
+```jinja2
+{{premise}}
+
+Question: Does this imply that "{{hypothesis}}"? Yes, no, or maybe? ||| {{answer_choices[label]}}
+```
+
+**You can browse through existing prompts on the [hosted version of PromptSource](https://bigscience.huggingface.co/promptsource).**
+
+## Setup
+If you do not intend to modify prompts, you can simply run:
+```bash
+pip install promptsource
+```
+
+Otherwise, you need to install the repo locally:
+1. Download the repo
+1. Navigate to the root directory of the repo
+1. Run `pip install -e .` to install the `promptsource` module
+
+*Note: for stability reasons, you will currently need a Python 3.7 environment to run the last step. However, if you only intend to use the prompts, and not create new prompts through the interface, you can remove this constraint in the [`setup.py`](setup.py) and install the package locally.*
+
+## How to use prompts
+You can apply prompts to examples from datasets of the [Hugging Face Datasets library](https://github.com/huggingface/datasets).
+```python
+# Load an example from the datasets ag_news
+>>> from datasets import load_dataset
+>>> dataset = load_dataset("ag_news", split="train")
+>>> example = dataset[1]
+
+# Load prompts for this dataset
+>>> from promptsource.templates import DatasetTemplates
+>>> ag_news_prompts = DatasetTemplates('ag_news')
+
+# Print all the prompts available for this dataset. The keys of the dict are the uuids the uniquely identify each of the prompt, and the values are instances of `Template` which wraps prompts
+>>> print(ag_news_prompts.templates)
+{'24e44a81-a18a-42dd-a71c-5b31b2d2cb39': <promptsource.templates.Template object at 0x7fa7aeb20350>, '8fdc1056-1029-41a1-9c67-354fc2b8ceaf': <promptsource.templates.Template object at 0x7fa7aeb17c10>, '918267e0-af68-4117-892d-2dbe66a58ce9': <promptsource.templates.Template object at 0x7fa7ac7a2310>, '9345df33-4f23-4944-a33c-eef94e626862': <promptsource.templates.Template object at 0x7fa7ac7a2050>, '98534347-fff7-4c39-a795-4e69a44791f7': <promptsource.templates.Template object at 0x7fa7ac7a1310>, 'b401b0ee-6ffe-4a91-8e15-77ee073cd858': <promptsource.templates.Template object at 0x7fa7ac7a12d0>, 'cb355f33-7e8c-4455-a72b-48d315bd4f60': <promptsource.templates.Template object at 0x7fa7ac7a1110>}
+
+# Select a prompt by its name
+>>> prompt = ag_news_prompts["classify_question_first"]
+
+# Apply the prompt to the example
+>>> result = prompt.apply(example)
+>>> print("INPUT: ", result[0])
+INPUT:  What label best describes this news article?
+Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\which has a reputation for making well-timed and occasionally\controversial plays in the defense industry, has quietly placed\its bets on another part of the market.
+>>> print("TARGET: ", result[1])
+TARGET:  Business
+```
+
+In the case that you are looking for the prompts available for a particular subset of a dataset, you should use the following syntax:
+```python
+dataset_name, subset_name = "super_glue", "rte"
+
+dataset = load_dataset(f"{dataset_name}/{subset_name}", split="train")
+example = dataset[0]
+
+prompts = DatasetTemplates(f"{dataset_name}/{subset_name}")
+```
+
+You can also collect all the available prompts for their associated datasets:
+
+```python
+>>> from promptsource.templates import TemplateCollection
+
+# Get all the prompts available in PromptSource
+>>> collection = TemplateCollection()
+
+# Print a dict where the key is the pair (dataset name, subset name)
+# and the value is an instance of DatasetTemplates
+>>> print(collection.datasets_templates)
+{('poem_sentiment', None): <promptsource.templates.DatasetTemplates object at 0x7fa7ac7939d0>, ('common_gen', None): <promptsource.templates.DatasetTemplates object at 0x7fa7ac795410>, ('anli', None): <promptsource.templates.DatasetTemplates object at 0x7fa7ac794590>, ('cc_news', None): <promptsource.templates.DatasetTemplates object at 0x7fa7ac798a90>, ('craigslist_bargains', None): <promptsource.templates.DatasetTemplates object at 0x7fa7ac7a2c10>,...}
+```
+
+You can learn more about PromptSource's API to store, manipulate and use prompts in the [documentation](API_DOCUMENTATION.md).
+
+## How to create prompts
+PromptSource provides a Web-based GUI that enables developers to write prompts in a templating language and immediately view their outputs on different examples.
+
+There are 3 modes in the app:
+- **Sourcing**: create and write new prompts
+- **Prompted dataset viewer**: check the prompts you wrote (or the existing ones) on the entire dataset
+- **Helicopter view**: aggregate high-level metrics on the current state of P3
+
+<p align="center">
+  <img src="assets/promptsource_app.png" width="800"/>
+</p>
+
+To launch the app locally, please first make sure you have followed the steps in [Setup](#setup), and from the root directory of the repo, run:
+```bash
+streamlit run promptsource/app.py
+```
+
+You can also browse through existing prompts on the [hosted version of PromptSource](https://bigscience.huggingface.co/promptsource). Note the hosted version disables the Sourcing mode (`streamlit run promptsource/app.py -- --read-only`).
+
+### Writing prompts
+Before creating new prompts, you should read the [contribution guidelines](CONTRIBUTING.md) which give an step-by-step description of how to contribute to the collection of prompts.
+
+### Datasets that require manual downloads
+Some datasets are not handled automatically by `datasets` and require users to download the dataset manually (`story_cloze` for instance ).
+
+To handle those datasets as well, we require users to download the dataset and put it in `~/.cache/promptsource`. This is the root directory containing all manually downloaded datasets.
+
+You can override this default path using `PROMPTSOURCE_MANUAL_DATASET_DIR` environment variable. This should point to the root directory.
+
+## Development structure
+PromptSource and P3 were originally developed as part of the [BigScience project for open research 🌸](https://bigscience.huggingface.co/), a year-long initiative targeting the study of large models and datasets. The goal of the project is to research language models in a public environment outside large technology companies. The project has 600 researchers from 50 countries and more than 250 institutions.
+
+In particular, PromptSource and P3 were the first steps for the paper [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://arxiv.org/abs/2110.08207).
+
+**You will find the official repository to reproduce the results of the paper here: https://github.com/bigscience-workshop/t-zero.** We also released T0* (pronounce "T Zero"), a series of models trained on [P3](https://huggingface.co/datasets/bigscience/P3) and presented in the paper. Checkpoints are available [here](https://huggingface.co/bigscience/T0pp).
+
+## Known Issues
+**Warning or Error about Darwin on OS X:** Try downgrading PyArrow to 3.0.0.
+
+**ConnectionRefusedError: [Errno 61] Connection refused:** Happens occasionally. Try restarting the app.
+
+## Citation
+If you find P3 or PromptSource useful, please cite the following reference:
+```bibtex
+@misc{bach2022promptsource,
+      title={PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts},
+      author={Stephen H. Bach and Victor Sanh and Zheng-Xin Yong and Albert Webson and Colin Raffel and Nihal V. Nayak and Abheesht Sharma and Taewoon Kim and M Saiful Bari and Thibault Fevry and Zaid Alyafeai and Manan Dey and Andrea Santilli and Zhiqing Sun and Srulik Ben-David and Canwen Xu and Gunjan Chhablani and Han Wang and Jason Alan Fries and Maged S. Al-shaibani and Shanya Sharma and Urmish Thakker and Khalid Almubarak and Xiangru Tang and Xiangru Tang and Mike Tian-Jian Jiang and Alexander M. Rush},
+      year={2022},
+      eprint={2202.01279},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG}
+}
+```

promptsource/__init__.py

@@ -0,0 +1,4 @@
+from pathlib import Path
+
+
+DEFAULT_PROMPTSOURCE_CACHE_HOME = str(Path("~/.cache/promptsource").expanduser())

promptsource/app.py

@@ -0,0 +1,663 @@
+import argparse
+import functools
+import multiprocessing
+import os
+import textwrap
+from hashlib import sha256
+from multiprocessing import Manager, Pool
+
+import pandas as pd
+import plotly.express as px
+import streamlit as st
+from datasets import get_dataset_infos
+from datasets.info import DatasetInfosDict
+from pygments import highlight
+from pygments.formatters import HtmlFormatter
+from pygments.lexers import DjangoLexer
+
+from promptsource import DEFAULT_PROMPTSOURCE_CACHE_HOME
+from promptsource.session import _get_state
+from promptsource.templates import INCLUDED_USERS, LANGUAGES, METRICS, DatasetTemplates, Template, TemplateCollection
+from promptsource.utils import (
+    get_dataset,
+    get_dataset_confs,
+    list_datasets,
+    removeHyphen,
+    renameDatasetColumn,
+    render_features,
+)
+
+
+DATASET_INFOS_CACHE_DIR = os.path.join(DEFAULT_PROMPTSOURCE_CACHE_HOME, "DATASET_INFOS")
+os.makedirs(DATASET_INFOS_CACHE_DIR, exist_ok=True)
+
+# Python 3.8 switched the default start method from fork to spawn. OS X also has
+# some issues related to fork, eee, e.g., https://github.com/bigscience-workshop/promptsource/issues/572
+# so we make sure we always use spawn for consistency
+multiprocessing.set_start_method("spawn", force=True)
+
+
+def get_infos(all_infos, d_name):
+    """
+    Wrapper for mutliprocess-loading of dataset infos
+
+    :param all_infos: multiprocess-safe dictionary
+    :param d_name: dataset name
+    """
+    d_name_bytes = d_name.encode("utf-8")
+    d_name_hash = sha256(d_name_bytes)
+    foldername = os.path.join(DATASET_INFOS_CACHE_DIR, d_name_hash.hexdigest())
+    if os.path.isdir(foldername):
+        infos_dict = DatasetInfosDict.from_directory(foldername)
+    else:
+        infos = get_dataset_infos(d_name)
+        infos_dict = DatasetInfosDict(infos)
+        os.makedirs(foldername)
+        infos_dict.write_to_directory(foldername)
+    all_infos[d_name] = infos_dict
+
+
+def format_language(tag):
+    """
+    Formats a language tag for display in the UI.
+
+    For example, if the tag is "en", then the function returns "en (English)"
+    :param tag: language tag
+    :return: formatted language name
+    """
+    return tag + " (" + LANGUAGES[tag] + ")"
+
+
+# add an argument for read-only
+# At the moment, streamlit does not handle python script arguments gracefully.
+# Thus, for read-only mode, you have to type one of the below two:
+# streamlit run promptsource/app.py -- -r
+# streamlit run promptsource/app.py -- --read-only
+# Check https://github.com/streamlit/streamlit/issues/337 for more information.
+parser = argparse.ArgumentParser(description="run app.py with args")
+parser.add_argument("-r", "--read-only", action="store_true", help="whether to run it as read-only mode", default=True)
+
+args = parser.parse_args()
+if args.read_only:
+    select_options = ["Helicopter view", "Prompted dataset viewer"]
+    side_bar_title_prefix = "Promptsource (Read only)"
+else:
+    select_options = ["Helicopter view", "Prompted dataset viewer", "Sourcing"]
+    side_bar_title_prefix = "Promptsource"
+
+#
+# Cache functions
+#
+get_dataset = st.cache(allow_output_mutation=True)(get_dataset)
+get_dataset_confs = st.cache(get_dataset_confs)
+list_datasets = st.cache(list_datasets)
+
+
+def run_app():
+    #
+    # Loads session state
+    #
+    state = _get_state()
+
+    def reset_template_state():
+        state.template_name = None
+        state.jinja = None
+        state.reference = None
+
+    #
+    # Initial page setup
+    #
+    st.set_page_config(page_title="Promptsource", layout="wide")
+    st.sidebar.markdown(
+        "<center><a href='https://github.com/bigscience-workshop/promptsource'>💻Github - Promptsource\n\n</a></center>",
+        unsafe_allow_html=True,
+    )
+    mode = st.sidebar.selectbox(
+        label="Choose a mode",
+        options=select_options,
+        index=0,
+        key="mode_select",
+    )
+    st.sidebar.title(f"{side_bar_title_prefix} 🌸 - {mode}")
+
+    #
+    # Adds pygments styles to the page.
+    #
+    st.markdown(
+        "<style>" + HtmlFormatter(style="friendly").get_style_defs(".highlight") + "</style>", unsafe_allow_html=True
+    )
+
+    WIDTH = 140
+
+    def show_jinja(t, width=WIDTH):
+        def replace_linebreaks(t):
+            """
+            st.write does not handle double breaklines very well. When it encounters `\n\n`, it exit the curent <div> block.
+            Explicitely replacing all `\n` with their html equivalent to bypass this issue.
+            Also stripping the trailing `\n` first.
+            """
+            return t.strip("\n").replace("\n", "<br/>")
+
+        wrap = textwrap.fill(t, width=width, replace_whitespace=False)
+        out = highlight(wrap, DjangoLexer(), HtmlFormatter())
+        out = replace_linebreaks(out)
+        st.write(out, unsafe_allow_html=True)
+
+    def show_text(t, width=WIDTH, with_markdown=False):
+        wrap = [textwrap.fill(subt, width=width, replace_whitespace=False) for subt in t.split("\n")]
+        wrap = "\n".join(wrap)
+        if with_markdown:
+            st.write(wrap, unsafe_allow_html=True)
+        else:
+            st.text(wrap)
+
+    if mode == "Helicopter view":
+        st.title("High level metrics")
+        st.write("This will take a minute to collect.")
+        st.write(
+            "If you want to contribute, please refer to the instructions in "
+            + "[Contributing](https://github.com/bigscience-workshop/promptsource/blob/main/CONTRIBUTING.md)."
+        )
+
+        #
+        # Loads template data
+        #
+        try:
+            template_collection = TemplateCollection()
+        except FileNotFoundError:
+            st.error(
+                "Unable to find the prompt folder!\n\n"
+                "We expect the folder to be in the working directory. "
+                "You might need to restart the app in the root directory of the repo."
+            )
+            st.stop()
+
+        #
+        # Global metrics
+        #
+        counts = template_collection.get_templates_count()
+        nb_prompted_datasets = len(counts)
+        st.write(f"## Number of *prompted datasets*: `{nb_prompted_datasets}`")
+        nb_prompts = sum(counts.values())
+        st.write(f"## Number of *prompts*: `{nb_prompts}`")
+
+        #
+        # Metrics per dataset/subset
+        #
+        # Download dataset infos (multiprocessing download)
+        manager = Manager()
+        all_infos = manager.dict()
+        all_datasets = list(set([t[0] for t in template_collection.keys]))
+
+        pool = Pool(processes=multiprocessing.cpu_count())
+        pool.map(functools.partial(get_infos, all_infos), all_datasets)
+        pool.close()
+        pool.join()
+
+        results = []
+        for (dataset_name, subset_name) in template_collection.keys:
+            # Collect split sizes (train, validation and test)
+            if dataset_name not in all_infos:
+                infos = get_dataset_infos(dataset_name)
+                all_infos[dataset_name] = infos
+            else:
+                infos = all_infos[dataset_name]
+            if infos:
+                if subset_name is None:
+                    subset_infos = infos[list(infos.keys())[0]]
+                else:
+                    subset_infos = infos[subset_name]
+
+                try:
+                    split_sizes = {k: v.num_examples for k, v in subset_infos.splits.items()}
+                except Exception:
+                    # Fixing bug in some community datasets.
+                    # For simplicity, just filling `split_sizes` with nothing, so the displayed split sizes will be 0.
+                    split_sizes = {}
+            else:
+                split_sizes = {}
+
+            # Collect template counts, original task counts and names
+            dataset_templates = template_collection.get_dataset(dataset_name, subset_name)
+            results.append(
+                {
+                    "Dataset name": dataset_name,
+                    "Subset name": "∅" if subset_name is None else subset_name,
+                    "Train size": split_sizes["train"] if "train" in split_sizes else 0,
+                    "Validation size": split_sizes["validation"] if "validation" in split_sizes else 0,
+                    "Test size": split_sizes["test"] if "test" in split_sizes else 0,
+                    "Number of prompts": len(dataset_templates),
+                    "Number of original task prompts": sum(
+                        [bool(t.metadata.original_task) for t in dataset_templates.templates.values()]
+                    ),
+                    "Prompt names": [t.name for t in dataset_templates.templates.values()],
+                }
+            )
+        results_df = pd.DataFrame(results)
+        results_df.sort_values(["Number of prompts"], inplace=True, ascending=False)
+        results_df.reset_index(drop=True, inplace=True)
+
+        nb_training_instances = results_df["Train size"].sum()
+        st.write(f"## Number of *training instances*: `{nb_training_instances}`")
+
+        plot_df = results_df[["Dataset name", "Subset name", "Train size", "Number of prompts"]].copy()
+        plot_df["Name"] = plot_df["Dataset name"] + " - " + plot_df["Subset name"]
+        plot_df.sort_values(["Train size"], inplace=True, ascending=False)
+        fig = px.bar(
+            plot_df,
+            x="Name",
+            y="Train size",
+            hover_data=["Dataset name", "Subset name", "Number of prompts"],
+            log_y=True,
+            title="Number of training instances per data(sub)set - y-axis is in logscale",
+        )
+        fig.update_xaxes(visible=False, showticklabels=False)
+        st.plotly_chart(fig, use_container_width=True)
+        st.write(
+            f"- Top 3 training subsets account for `{100 * plot_df[:3]['Train size'].sum() / nb_training_instances:.2f}%` of the training instances."
+        )
+        biggest_training_subset = plot_df.iloc[0]
+        st.write(
+            f"- Biggest training subset is *{biggest_training_subset['Name']}* with `{biggest_training_subset['Train size']}` instances"
+        )
+        smallest_training_subset = plot_df[plot_df["Train size"] > 0].iloc[-1]
+        st.write(
+            f"- Smallest training subset is *{smallest_training_subset['Name']}* with `{smallest_training_subset['Train size']}` instances"
+        )
+
+        st.markdown("***")
+        st.write("Details per dataset")
+        st.table(results_df)
+
+    else:
+        # Combining mode `Prompted dataset viewer` and `Sourcing` since the
+        # backbone of the interfaces is the same
+        assert mode in ["Prompted dataset viewer", "Sourcing"], ValueError(
+            f"`mode` ({mode}) should be in `[Helicopter view, Prompted dataset viewer, Sourcing]`"
+        )
+
+        #
+        # Loads dataset information
+        #
+
+        dataset_list = list_datasets()
+        ag_news_index = dataset_list.index("ag_news")
+
+        #
+        # Select a dataset - starts with ag_news
+        #
+        dataset_key = st.sidebar.selectbox(
+            "Dataset",
+            dataset_list,
+            key="dataset_select",
+            index=ag_news_index,
+            help="Select the dataset to work on.",
+        )
+
+        #
+        # If a particular dataset is selected, loads dataset and template information
+        #
+        if dataset_key is not None:
+
+            #
+            # Check for subconfigurations (i.e. subsets)
+            #
+            configs = get_dataset_confs(dataset_key)
+            conf_option = None
+            if len(configs) > 0:
+                conf_option = st.sidebar.selectbox("Subset", configs, index=0, format_func=lambda a: a.name)
+
+            subset_name = str(conf_option.name) if conf_option else None
+            try:
+                dataset = get_dataset(dataset_key, subset_name)
+            except OSError as e:
+                st.error(
+                    f"Some datasets are not handled automatically by `datasets` and require users to download the "
+                    f"dataset manually. This applies to {dataset_key}{f'/{subset_name}' if subset_name is not None else ''}. "
+                    f"\n\nPlease download the raw dataset to `~/.cache/promptsource/{dataset_key}{f'/{subset_name}' if subset_name is not None else ''}`. "
+                    f"\n\nYou can choose another cache directory by overriding `PROMPTSOURCE_MANUAL_DATASET_DIR` environment "
+                    f"variable and downloading raw dataset to `$PROMPTSOURCE_MANUAL_DATASET_DIR/{dataset_key}{f'/{subset_name}' if subset_name is not None else ''}`"
+                    f"\n\nOriginal error:\n{str(e)}"
+                )
+                st.stop()
+
+            splits = list(dataset.keys())
+            index = 0
+            if "train" in splits:
+                index = splits.index("train")
+            split = st.sidebar.selectbox("Split", splits, key="split_select", index=index)
+            dataset = dataset[split]
+            dataset = renameDatasetColumn(dataset)
+
+            #
+            # Loads template data
+            #
+            try:
+                dataset_templates = DatasetTemplates(dataset_key, conf_option.name if conf_option else None)
+            except FileNotFoundError:
+                st.error(
+                    "Unable to find the prompt folder!\n\n"
+                    "We expect the folder to be in the working directory. "
+                    "You might need to restart the app in the root directory of the repo."
+                )
+                st.stop()
+
+            template_list = dataset_templates.all_template_names
+            num_templates = len(template_list)
+            st.sidebar.write(
+                "No of prompts created for "
+                + f"`{dataset_key + (('/' + conf_option.name) if conf_option else '')}`"
+                + f": **{str(num_templates)}**"
+            )
+
+            if mode == "Prompted dataset viewer":
+                if num_templates > 0:
+                    template_name = st.sidebar.selectbox(
+                        "Prompt name",
+                        template_list,
+                        key="template_select",
+                        index=0,
+                        help="Select the prompt to visualize.",
+                    )
+
+                step = 50
+                example_index = st.sidebar.number_input(
+                    f"Select the example index (Size = {len(dataset)})",
+                    min_value=0,
+                    max_value=len(dataset) - step,
+                    value=0,
+                    step=step,
+                    key="example_index_number_input",
+                    help="Offset = 50.",
+                )
+            else:  # mode = Sourcing
+                st.sidebar.subheader("Select Example")
+                example_index = st.sidebar.slider("Select the example index", 0, len(dataset) - 1)
+
+                example = dataset[example_index]
+                example = removeHyphen(example)
+
+                st.sidebar.write(example)
+
+            st.sidebar.subheader("Dataset Schema")
+            rendered_features = render_features(dataset.features)
+            st.sidebar.write(rendered_features)
+
+            #
+            # Display dataset information
+            #
+            st.header("Dataset: " + dataset_key + " " + (("/ " + conf_option.name) if conf_option else ""))
+
+            # If we have a custom dataset change the source link to the hub
+            split_dataset_key = dataset_key.split("/")
+            possible_user = split_dataset_key[0]
+            if len(split_dataset_key) > 1 and possible_user in INCLUDED_USERS:
+                source_link = "https://huggingface.co/datasets/%s/blob/main/%s.py" % (
+                    dataset_key,
+                    split_dataset_key[-1],
+                )
+            else:
+                source_link = "https://github.com/huggingface/datasets/blob/master/datasets/%s/%s.py" % (
+                    dataset_key,
+                    dataset_key,
+                )
+
+            st.markdown("*Homepage*: " + dataset.info.homepage + "\n\n*Dataset*: " + source_link)
+
+            md = """
+            %s
+            """ % (
+                dataset.info.description.replace("\\", "") if dataset_key else ""
+            )
+            st.markdown(md)
+
+            #
+            # Body of the app: display prompted examples in mode `Prompted dataset viewer`
+            # or text boxes to create new prompts in mode `Sourcing`
+            #
+            if mode == "Prompted dataset viewer":
+                #
+                # Display template information
+                #
+                if num_templates > 0:
+                    template = dataset_templates[template_name]
+                    st.subheader("Prompt")
+                    st.markdown("##### Name")
+                    st.text(template.name)
+                    st.markdown("##### Reference")
+                    st.text(template.reference)
+                    st.markdown("##### Original Task? ")
+                    st.text(template.metadata.original_task)
+                    st.markdown("##### Choices in template? ")
+                    st.text(template.metadata.choices_in_prompt)
+                    st.markdown("##### Metrics")
+                    st.text(", ".join(template.metadata.metrics) if template.metadata.metrics else None)
+                    st.markdown("##### Prompt Languages")
+                    if template.metadata.languages:
+                        st.text(", ".join([format_language(tag) for tag in template.metadata.languages]))
+                    else:
+                        st.text(None)
+                    st.markdown("##### Answer Choices")
+                    if template.get_answer_choices_expr() is not None:
+                        show_jinja(template.get_answer_choices_expr())
+                    else:
+                        st.text(None)
+                    st.markdown("##### Jinja template")
+                    splitted_template = template.jinja.split("|||")
+                    st.markdown("###### Input template")
+                    show_jinja(splitted_template[0].strip())
+                    if len(splitted_template) > 1:
+                        st.markdown("###### Target template")
+                        show_jinja(splitted_template[1].strip())
+                    st.markdown("***")
+
+                #
+                # Display a couple (steps) examples
+                #
+                for ex_idx in range(example_index, example_index + step):
+                    if ex_idx >= len(dataset):
+                        continue
+                    example = dataset[ex_idx]
+                    example = removeHyphen(example)
+                    col1, _, col2 = st.beta_columns([12, 1, 12])
+                    with col1:
+                        st.write(example)
+                    if num_templates > 0:
+                        with col2:
+                            prompt = template.apply(example, highlight_variables=False)
+                            if prompt == [""]:
+                                st.write("∅∅∅ *Blank result*")
+                            else:
+                                st.write("Input")
+                                show_text(prompt[0])
+                                if len(prompt) > 1:
+                                    st.write("Target")
+                                    show_text(prompt[1])
+                    st.markdown("***")
+            else:  # mode = Sourcing
+                st.markdown("## Prompt Creator")
+
+                #
+                # Create a new template or select an existing one
+                #
+                col1a, col1b, _, col2 = st.beta_columns([9, 9, 1, 6])
+
+                # current_templates_key and state.templates_key are keys for the templates object
+                current_templates_key = (dataset_key, conf_option.name if conf_option else None)
+
+                # Resets state if there has been a change in templates_key
+                if state.templates_key != current_templates_key:
+                    state.templates_key = current_templates_key
+                    reset_template_state()
+
+                with col1a, st.form("new_template_form"):
+                    new_template_name = st.text_input(
+                        "Create a New Prompt",
+                        key="new_template",
+                        value="",
+                        help="Enter name and hit enter to create a new prompt.",
+                    )
+                    new_template_submitted = st.form_submit_button("Create")
+                    if new_template_submitted:
+                        if new_template_name in dataset_templates.all_template_names:
+                            st.error(
+                                f"A prompt with the name {new_template_name} already exists "
+                                f"for dataset {state.templates_key}."
+                            )
+                        elif new_template_name == "":
+                            st.error("Need to provide a prompt name.")
+                        else:
+                            template = Template(new_template_name, "", "")
+                            dataset_templates.add_template(template)
+                            reset_template_state()
+                            state.template_name = new_template_name
+                    else:
+                        state.new_template_name = None
+
+                with col1b, st.beta_expander("or Select Prompt", expanded=True):
+                    template_list = dataset_templates.all_template_names
+                    if state.template_name:
+                        index = template_list.index(state.template_name)
+                    else:
+                        index = 0
+                    state.template_name = st.selectbox(
+                        "", template_list, key="template_select", index=index, help="Select the prompt to work on."
+                    )
+
+                    if st.button("Delete Prompt", key="delete_prompt"):
+                        dataset_templates.remove_template(state.template_name)
+                        reset_template_state()
+
+                variety_guideline = """
+                :heavy_exclamation_mark::question:Creating a diverse set of prompts whose differences go beyond surface wordings (i.e. marginally changing 2 or 3 words) is highly encouraged.
+                Ultimately, the hope is that exposing the model to such a diversity will have a non-trivial impact on the model's robustness to the prompt formulation.
+                \r**To get various prompts, you can try moving the cursor along theses axes**:
+                \n- **Interrogative vs affirmative form**: Ask a question about an attribute of the inputs or tell the model to decide something about the input.
+                \n- **Task description localization**: where is the task description blended with the inputs? In the beginning, in the middle, at the end?
+                \n- **Implicit situation or contextualization**: how explicit is the query? For instance, *Given this review, would you buy this product?* is an indirect way to ask whether the review is positive.
+                """
+
+                col1, _, _ = st.beta_columns([18, 1, 6])
+                with col1:
+                    if state.template_name is not None:
+                        show_text(variety_guideline, with_markdown=True)
+
+                #
+                # Edit the created or selected template
+                #
+                col1, _, col2 = st.beta_columns([18, 1, 6])
+                with col1:
+                    if state.template_name is not None:
+                        template = dataset_templates[state.template_name]
+                        #
+                        # If template is selected, displays template editor
+                        #
+                        with st.form("edit_template_form"):
+                            updated_template_name = st.text_input("Name", value=template.name)
+                            state.reference = st.text_input(
+                                "Prompt Reference",
+                                help="Short description of the prompt and/or paper reference for the prompt.",
+                                value=template.reference,
+                            )
+
+                            # Metadata
+                            state.metadata = template.metadata
+                            state.metadata.original_task = st.checkbox(
+                                "Original Task?",
+                                value=template.metadata.original_task,
+                                help="Prompt asks model to perform the original task designed for this dataset.",
+                            )
+                            state.metadata.choices_in_prompt = st.checkbox(
+                                "Choices in Template?",
+                                value=template.metadata.choices_in_prompt,
+                                help="Prompt explicitly lists choices in the template for the output.",
+                            )
+
+                            state.metadata.metrics = st.multiselect(
+                                "Metrics",
+                                sorted(METRICS),
+                                default=template.metadata.metrics,
+                                help="Select all metrics that are commonly used (or should "
+                                "be used if a new task) to evaluate this prompt.",
+                            )
+
+                            state.metadata.languages = st.multiselect(
+                                "Prompt Languages",
+                                sorted(LANGUAGES.keys()),
+                                default=template.metadata.languages,
+                                format_func=format_language,
+                                help="Select all languages used in this prompt. "
+                                "This annotation is independent from the language(s) "
+                                "of the dataset.",
+                            )
+
+                            # Answer choices
+                            if template.get_answer_choices_expr() is not None:
+                                answer_choices = template.get_answer_choices_expr()
+                            else:
+                                answer_choices = ""
+                            state.answer_choices = st.text_input(
+                                "Answer Choices",
+                                value=answer_choices,
+                                help="A Jinja expression for computing answer choices. "
+                                "Separate choices with a triple bar (|||).",
+                            )
+
+                            # Jinja
+                            state.jinja = st.text_area("Template", height=40, value=template.jinja)
+
+                            # Submit form
+                            if st.form_submit_button("Save"):
+                                if (
+                                    updated_template_name in dataset_templates.all_template_names
+                                    and updated_template_name != state.template_name
+                                ):
+                                    st.error(
+                                        f"A prompt with the name {updated_template_name} already exists "
+                                        f"for dataset {state.templates_key}."
+                                    )
+                                elif updated_template_name == "":
+                                    st.error("Need to provide a prompt name.")
+                                else:
+                                    # Parses state.answer_choices
+                                    if state.answer_choices == "":
+                                        updated_answer_choices = None
+                                    else:
+                                        updated_answer_choices = state.answer_choices
+
+                                    dataset_templates.update_template(
+                                        state.template_name,
+                                        updated_template_name,
+                                        state.jinja,
+                                        state.reference,
+                                        state.metadata,
+                                        updated_answer_choices,
+                                    )
+                                    # Update the state as well
+                                    state.template_name = updated_template_name
+                #
+                # Displays template output on current example if a template is selected
+                # (in second column)
+                #
+                with col2:
+                    if state.template_name is not None:
+                        st.empty()
+                        template = dataset_templates[state.template_name]
+                        prompt = template.apply(example)
+                        if prompt == [""]:
+                            st.write("∅∅∅ *Blank result*")
+                        else:
+                            st.write("Input")
+                            show_text(prompt[0], width=40)
+                            if len(prompt) > 1:
+                                st.write("Target")
+                                show_text(prompt[1], width=40)
+
+    #
+    # Must sync state at end
+    #
+    state.sync()
+
+
+if __name__ == "__main__":
+    run_app()

promptsource/session.py

@@ -0,0 +1,89 @@
+#
+# Code for managing session state, which is needed for multi-input forms
+# See https://github.com/streamlit/streamlit/issues/1557
+#
+# This code is taken from
+# https://gist.github.com/okld/0aba4869ba6fdc8d49132e6974e2e662
+#
+from streamlit.hashing import _CodeHasher
+from streamlit.report_thread import get_report_ctx
+from streamlit.server.server import Server
+
+
+class _SessionState:
+    def __init__(self, session, hash_funcs):
+        """Initialize SessionState instance."""
+        self.__dict__["_state"] = {
+            "data": {},
+            "hash": None,
+            "hasher": _CodeHasher(hash_funcs),
+            "is_rerun": False,
+            "session": session,
+        }
+
+    def __call__(self, **kwargs):
+        """Initialize state data once."""
+        for item, value in kwargs.items():
+            if item not in self._state["data"]:
+                self._state["data"][item] = value
+
+    def __getitem__(self, item):
+        """Return a saved state value, None if item is undefined."""
+        return self._state["data"].get(item, None)
+
+    def __getattr__(self, item):
+        """Return a saved state value, None if item is undefined."""
+        return self._state["data"].get(item, None)
+
+    def __setitem__(self, item, value):
+        """Set state value."""
+        self._state["data"][item] = value
+
+    def __setattr__(self, item, value):
+        """Set state value."""
+        self._state["data"][item] = value
+
+    def clear(self):
+        """Clear session state and request a rerun."""
+        self._state["data"].clear()
+        self._state["session"].request_rerun(None)
+
+    def sync(self):
+        """
+        Rerun the app with all state values up to date from the beginning to
+        fix rollbacks.
+        """
+        data_to_bytes = self._state["hasher"].to_bytes(self._state["data"], None)
+
+        # Ensure to rerun only once to avoid infinite loops
+        # caused by a constantly changing state value at each run.
+        #
+        # Example: state.value += 1
+        if self._state["is_rerun"]:
+            self._state["is_rerun"] = False
+
+        elif self._state["hash"] is not None:
+            if self._state["hash"] != data_to_bytes:
+                self._state["is_rerun"] = True
+                self._state["session"].request_rerun(None)
+
+        self._state["hash"] = data_to_bytes
+
+
+def _get_session():
+    session_id = get_report_ctx().session_id
+    session_info = Server.get_current()._get_session_info(session_id)
+
+    if session_info is None:
+        raise RuntimeError("Couldn't get your Streamlit Session object.")
+
+    return session_info.session
+
+
+def _get_state(hash_funcs=None):
+    session = _get_session()
+
+    if not hasattr(session, "_custom_session_state"):
+        session._custom_session_state = _SessionState(session, hash_funcs)
+
+    return session._custom_session_state

promptsource/templates.py

@@ -0,0 +1,731 @@
+import logging
+import os
+import random
+import uuid
+from collections import Counter, defaultdict
+from shutil import rmtree
+from typing import Dict, List, Optional, Tuple
+
+import pandas as pd
+import pkg_resources
+import yaml
+from jinja2 import BaseLoader, Environment, meta
+
+
+# Truncation of jinja template variables
+# 1710 = 300 words x 4.7 avg characters per word + 300 spaces
+TEXT_VAR_LENGTH = 2048
+
+# Local path to the folder containing the templates
+TEMPLATES_FOLDER_PATH = pkg_resources.resource_filename(__name__, "templates")
+
+env = Environment(loader=BaseLoader)
+
+# Allow the python function zip()
+env.globals.update(zip=zip)
+
+# These are users whose datasets should be included in the results returned by
+# filter_english_datasets (regardless of their metadata)
+INCLUDED_USERS = {"Zaid", "craffel"}
+
+# These are the metrics with which templates can be tagged
+METRICS = {
+    "BLEU",
+    "ROUGE",
+    "Squad",
+    "Trivia QA",
+    "Accuracy",
+    "Pearson Correlation",
+    "Spearman Correlation",
+    "MultiRC",
+    "AUC",
+    "COQA F1",
+    "Edit Distance",
+    "Mean Reciprocal Rank",
+    "Other",
+}
+
+# These are the languages with which templates can be tagged. Keys are ISO 639-1
+# tags, which are the actual tags we use. Values are English names shown in the
+# UI for convenience.
+LANGUAGES = {
+    "ab": "Abkhazian",
+    "aa": "Afar",
+    "af": "Afrikaans",
+    "ak": "Akan",
+    "sq": "Albanian",
+    "am": "Amharic",
+    "ar": "Arabic",
+    "an": "Aragonese",
+    "hy": "Armenian",
+    "as": "Assamese",
+    "av": "Avaric",
+    "ae": "Avestan",
+    "ay": "Aymara",
+    "az": "Azerbaijani",
+    "bm": "Bambara",
+    "ba": "Bashkir",
+    "eu": "Basque",
+    "be": "Belarusian",
+    "bn": "Bengali",
+    "bi": "Bislama",
+    "bs": "Bosnian",
+    "br": "Breton",
+    "bg": "Bulgarian",
+    "my": "Burmese",
+    "ca": "Catalan, Valencian",
+    "ch": "Chamorro",
+    "ce": "Chechen",
+    "ny": "Chichewa, Chewa, Nyanja",
+    "zh": "Chinese",
+    "cu": "Church Slavic, Old Slavonic, Church Slavonic, Old Bulgarian, Old Church Slavonic",
+    "cv": "Chuvash",
+    "kw": "Cornish",
+    "co": "Corsican",
+    "cr": "Cree",
+    "hr": "Croatian",
+    "cs": "Czech",
+    "da": "Danish",
+    "dv": "Divehi, Dhivehi, Maldivian",
+    "nl": "Dutch, Flemish",
+    "dz": "Dzongkha",
+    "en": "English",
+    "eo": "Esperanto",
+    "et": "Estonian",
+    "ee": "Ewe",
+    "fo": "Faroese",
+    "fj": "Fijian",
+    "fi": "Finnish",
+    "fr": "French",
+    "fy": "Western Frisian",
+    "ff": "Fulah",
+    "gd": "Gaelic, Scottish Gaelic",
+    "gl": "Galician",
+    "lg": "Ganda",
+    "ka": "Georgian",
+    "de": "German",
+    "el": "Greek, Modern (1453–)",
+    "kl": "Kalaallisut, Greenlandic",
+    "gn": "Guarani",
+    "gu": "Gujarati",
+    "ht": "Haitian, Haitian Creole",
+    "ha": "Hausa",
+    "he": "Hebrew",
+    "hz": "Herero",
+    "hi": "Hindi",
+    "ho": "Hiri Motu",
+    "hu": "Hungarian",
+    "is": "Icelandic",
+    "io": "Ido",
+    "ig": "Igbo",
+    "id": "Indonesian",
+    "ia": "Interlingua (International Auxiliary Language Association)",
+    "ie": "Interlingue, Occidental",
+    "iu": "Inuktitut",
+    "ik": "Inupiaq",
+    "ga": "Irish",
+    "it": "Italian",
+    "ja": "Japanese",
+    "jv": "Javanese",
+    "kn": "Kannada",
+    "kr": "Kanuri",
+    "ks": "Kashmiri",
+    "kk": "Kazakh",
+    "km": "Central Khmer",
+    "ki": "Kikuyu, Gikuyu",
+    "rw": "Kinyarwanda",
+    "ky": "Kirghiz, Kyrgyz",
+    "kv": "Komi",
+    "kg": "Kongo",
+    "ko": "Korean",
+    "kj": "Kuanyama, Kwanyama",
+    "ku": "Kurdish",
+    "lo": "Lao",
+    "la": "Latin",
+    "lv": "Latvian",
+    "li": "Limburgan, Limburger, Limburgish",
+    "ln": "Lingala",
+    "lt": "Lithuanian",
+    "lu": "Luba-Katanga",
+    "lb": "Luxembourgish, Letzeburgesch",
+    "mk": "Macedonian",
+    "mg": "Malagasy",
+    "ms": "Malay",
+    "ml": "Malayalam",
+    "mt": "Maltese",
+    "gv": "Manx",
+    "mi": "Maori",
+    "mr": "Marathi",
+    "mh": "Marshallese",
+    "mn": "Mongolian",
+    "na": "Nauru",
+    "nv": "Navajo, Navaho",
+    "nd": "North Ndebele",
+    "nr": "South Ndebele",
+    "ng": "Ndonga",
+    "ne": "Nepali",
+    "no": "Norwegian",
+    "nb": "Norwegian Bokmål",
+    "nn": "Norwegian Nynorsk",
+    "ii": "Sichuan Yi, Nuosu",
+    "oc": "Occitan",
+    "oj": "Ojibwa",
+    "or": "Oriya",
+    "om": "Oromo",
+    "os": "Ossetian, Ossetic",
+    "pi": "Pali",
+    "ps": "Pashto, Pushto",
+    "fa": "Persian",
+    "pl": "Polish",
+    "pt": "Portuguese",
+    "pa": "Punjabi, Panjabi",
+    "qu": "Quechua",
+    "ro": "Romanian, Moldavian, Moldovan",
+    "rm": "Romansh",
+    "rn": "Rundi",
+    "ru": "Russian",
+    "se": "Northern Sami",
+    "sm": "Samoan",
+    "sg": "Sango",
+    "sa": "Sanskrit",
+    "sc": "Sardinian",
+    "sr": "Serbian",
+    "sn": "Shona",
+    "sd": "Sindhi",
+    "si": "Sinhala, Sinhalese",
+    "sk": "Slovak",
+    "sl": "Slovenian",
+    "so": "Somali",
+    "st": "Southern Sotho",
+    "es": "Spanish, Castilian",
+    "su": "Sundanese",
+    "sw": "Swahili",
+    "ss": "Swati",
+    "sv": "Swedish",
+    "tl": "Tagalog",
+    "ty": "Tahitian",
+    "tg": "Tajik",
+    "ta": "Tamil",
+    "tt": "Tatar",
+    "te": "Telugu",
+    "th": "Thai",
+    "bo": "Tibetan",
+    "ti": "Tigrinya",
+    "to": "Tonga (Tonga Islands)",
+    "ts": "Tsonga",
+    "tn": "Tswana",
+    "tr": "Turkish",
+    "tk": "Turkmen",
+    "tw": "Twi",
+    "ug": "Uighur, Uyghur",
+    "uk": "Ukrainian",
+    "ur": "Urdu",
+    "uz": "Uzbek",
+    "ve": "Venda",
+    "vi": "Vietnamese",
+    "vo": "Volapük",
+    "wa": "Walloon",
+    "cy": "Welsh",
+    "wo": "Wolof",
+    "xh": "Xhosa",
+    "yi": "Yiddish",
+    "yo": "Yoruba",
+    "za": "Zhuang, Chuang",
+    "zu": "Zulu",
+}
+
+
+def highlight(input):
+    return "<span style='color: #F08080'>" + input + "</span>"
+
+
+def choice(choices):
+    return random.choice(choices)
+
+
+def most_frequent(items):
+    """Returns the set of items which appear most frequently in the input"""
+    if not items:
+        return
+    item_counts = Counter(items).most_common()
+    max_freq = item_counts[0][1]
+    most_frequent_items = [c[0] for c in item_counts if c[1] == max_freq]
+    return most_frequent_items
+
+
+env.filters["highlight"] = highlight
+env.filters["choice"] = choice
+env.filters["most_frequent"] = most_frequent
+
+
+class Template(yaml.YAMLObject):
+    """
+    A prompt template.
+    """
+
+    yaml_tag = "!Template"
+
+    def __init__(self, name, jinja, reference, metadata=None, answer_choices=None):
+        """
+        Creates a prompt template.
+
+        A prompt template is expressed in Jinja. It is rendered using an example
+        from the corresponding Hugging Face datasets library (a dictionary). The
+        separator ||| should appear once to divide the template into prompt and
+        output. Generally, the prompt should provide information on the desired
+        behavior, e.g., text passage and instructions, and the output should be
+        a desired response.
+
+        :param name: unique name (per dataset) for template
+        :param jinja: template expressed in Jinja
+        :param reference: string describing author or paper reference for template
+        :param metadata: a Metadata object with template annotations
+        :param answer_choices: Jinja expression for answer choices. Should produce
+                               a ||| delimited string of choices that enumerates
+                               the possible completions for templates that should
+                               be evaluated as ranked completions. If None, then
+                               the template is open-ended. This list is accessible
+                               from within Jinja as the variable `answer_choices`.
+        """
+        self.id = str(uuid.uuid4())
+        self.name = name
+        self.jinja = jinja
+        self.reference = reference
+        self.metadata = metadata if metadata is not None else Template.Metadata()
+        self.answer_choices = answer_choices
+
+    def get_id(self):
+        """
+        Returns the id of the template
+
+        :return: unique id for template
+        """
+        return self.id
+
+    def get_name(self):
+        """
+        Returns the name of the template
+
+        :return: unique (per dataset) name for template
+        """
+        return self.name
+
+    def get_reference(self):
+        """
+        Returns the bibliographic reference (or author) for the template
+
+        :return: reference as a string
+        """
+        return self.reference
+
+    def get_answer_choices_expr(self):
+        """
+        Returns a Jinja expression for computing the answer choices from an example.
+
+        :return: String, or None if no answer choices
+        """
+        return self.answer_choices
+
+    def get_answer_choices_list(self, example):
+        """
+        Returns a list of answer choices for a given example
+
+        :return: list of strings, or None if get_answer_choices_expr is None
+        """
+        jinja = self.get_answer_choices_expr()
+        if jinja is None:
+            return None
+
+        rtemplate = env.from_string(jinja)
+        protected_example = self._escape_pipe(example)
+        rendered_choices = rtemplate.render(**protected_example)
+        return [self._unescape_pipe(answer_choice.strip()) for answer_choice in rendered_choices.split("|||")]
+
+    def get_fixed_answer_choices_list(self):
+        """
+        Returns a list of answer choices that is static across examples, if possible
+        :return: list of strings, or None if no static list exists
+        """
+        jinja = self.get_answer_choices_expr()
+        if jinja is None:
+            return None
+
+        parse = env.parse(jinja)
+        variables = meta.find_undeclared_variables(parse)
+        if len(variables) == 0:
+            rtemplate = env.from_string(jinja)
+            rendered_choices = rtemplate.render()
+            return [answer_choice.strip() for answer_choice in rendered_choices.split("|||")]
+        else:
+            return None
+
+    def apply(self, example, truncate=True, highlight_variables=False):
+        """
+        Creates a prompt by applying this template to an example
+
+        :param example: the dataset example to create a prompt for
+        :param truncate: if True, example fields will be truncated to TEXT_VAR_LENGTH chars
+        :param highlight_variables: highlight the added variables
+        :return: tuple of 2 strings, for prompt and output
+        """
+        jinja = self.jinja
+
+        # Truncates the prompt if needed
+        if truncate:
+            trunc_command = (
+                f" | string | truncate({TEXT_VAR_LENGTH}) }}}}"  # Escaping curly braces requires doubling them
+            )
+            jinja = jinja.replace("}}", trunc_command)
+
+        # Highlights text that was substituted for variables, if requested
+        if highlight_variables:
+            jinja = jinja.replace("}}", " | highlight }}")
+        rtemplate = env.from_string(jinja)
+
+        protected_example = self._escape_pipe(example)
+
+        # Adds in answer_choices variable
+        if "answer_choices" in protected_example:
+            raise ValueError("Example contains the restricted key 'answer_choices'.")
+
+        protected_example["answer_choices"] = self.get_answer_choices_list(example)
+
+        # Renders the Jinja template
+        rendered_example = rtemplate.render(**protected_example)
+
+        # Splits on the separator, and then replaces back any occurrences of the
+        # separator in the original example
+        return [self._unescape_pipe(part).strip() for part in rendered_example.split("|||")]
+
+    pipe_protector = "3ed2dface8203c4c9dfb1a5dc58e41e0"
+
+    @classmethod
+    def _escape_pipe(cls, example):
+        # Replaces any occurrences of the "|||" separator in the example, which
+        # which will be replaced back after splitting
+        protected_example = {
+            key: value.replace("|||", cls.pipe_protector) if isinstance(value, str) else value
+            for key, value in example.items()
+        }
+        return protected_example
+
+    @classmethod
+    def _unescape_pipe(cls, string):
+        # replaces back any occurrences of the separator in a string
+        return string.replace(cls.pipe_protector, "|||")
+
+    class Metadata(yaml.YAMLObject):
+        """
+        Metadata for a prompt template.
+        """
+
+        yaml_tag = "!TemplateMetadata"
+
+        def __init__(
+            self,
+            original_task: Optional[bool] = None,
+            choices_in_prompt: Optional[bool] = None,
+            metrics: Optional[List[str]] = None,
+            languages: Optional[List[str]] = None,
+        ):
+            """
+            Initializes template metadata.
+
+            In the following, trivial choices are defined as Yes/No, True/False,
+            etc. and nontrivial choices are other types of choices denoted in
+            the answer_choices field.
+
+            :param original_task: If True, this prompt asks a model to perform the original task designed for
+                this dataset.
+            :param choices_in_prompt: If True, the answer choices are included in the templates such that models
+                see those choices in the input. Only applicable to classification tasks.
+            :param metrics: List of strings denoting metrics to use for evaluation
+            :param metrics: List of strings denoting languages used in the prompt (not the associated dataset!)
+            """
+            self.original_task = original_task
+            self.choices_in_prompt = choices_in_prompt
+            self.metrics = metrics
+            self.languages = languages
+
+
+class TemplateCollection:
+    """
+    This helper class wraps the DatasetTemplates class
+    - Initialized the DatasetTemplates for all existing template folder
+    - Give access to each DatasetTemplates
+    - Provides aggregated counts over all DatasetTemplates
+    """
+
+    def __init__(self):
+
+        # Dict of all the DatasetTemplates, key is the tuple (dataset_name, subset_name)
+        self.datasets_templates: Dict[(str, Optional[str]), DatasetTemplates] = self._collect_datasets()
+
+    @property
+    def keys(self):
+        return list(self.datasets_templates.keys())
+
+    def __len__(self) -> int:
+        return len(self.datasets_templates)
+
+    def remove(self, dataset_name: str, subset_name: Optional[str] = None) -> None:
+        del self.datasets_templates[dataset_name, subset_name]
+
+    def _collect_datasets(self) -> Dict[Tuple[str, str], "DatasetTemplates"]:
+        """
+        Initialize a DatasetTemplates object for each templates.yaml detected in the templates folder
+
+        Returns: a dict with key=(dataset_name, subset_name)
+        """
+        dataset_folders = os.listdir(TEMPLATES_FOLDER_PATH)
+        dataset_folders = [folder for folder in dataset_folders if not folder.startswith(".")]
+
+        output = {}  # format is {(dataset_name, subset_name): DatasetsTemplates}
+        for dataset in dataset_folders:
+            if dataset in INCLUDED_USERS:
+                for filename in os.listdir(os.path.join(TEMPLATES_FOLDER_PATH, dataset)):
+                    output = {**output, **self._collect_dataset(dataset + "/" + filename)}
+            else:
+                output = {**output, **self._collect_dataset(dataset)}
+        return output
+
+    def _collect_dataset(self, dataset):
+        output = {}  # format is {(dataset_name, subset_name): DatasetsTemplates}
+        for filename in os.listdir(os.path.join(TEMPLATES_FOLDER_PATH, dataset)):
+            if filename.endswith(".yaml"):
+                # If there is no sub-folder, there is no subset for this dataset
+                output[(dataset, None)] = DatasetTemplates(dataset)
+            else:
+                # This is a subfolder, and its name corresponds to the subset name
+                output[(dataset, filename)] = DatasetTemplates(dataset_name=dataset, subset_name=filename)
+        return output
+
+    def get_dataset(self, dataset_name: str, subset_name: Optional[str] = None) -> "DatasetTemplates":
+        """
+        Return the DatasetTemplates object corresponding to the dataset name
+
+        :param dataset_name: name of the dataset to get
+        :param subset_name: name of the subset
+        """
+        # if the dataset does not exist, we add it
+        if dataset_name not in self.keys:
+            self.datasets_templates[(dataset_name, subset_name)] = DatasetTemplates(dataset_name, subset_name)
+
+        return self.datasets_templates[(dataset_name, subset_name)]
+
+    def get_templates_count(self) -> Dict:
+        """
+        Return the overall number count over all datasets
+
+        NB: we don't breakdown datasets into subsets for the count, i.e subsets count are included
+        into the dataset count
+        """
+
+        count_dict = defaultdict(int)
+        for k, v in self.datasets_templates.items():
+            # Subsets count towards dataset count
+            count_dict[k[0]] += len(v)
+        # converting to regular dict
+        return dict(count_dict)
+
+
+class DatasetTemplates:
+    """
+    Class that wraps all templates for a specific dataset/subset and implements all the helper
+    functions necessary to read/write to the yaml file
+    """
+
+    TEMPLATES_KEY = "templates"
+    DATASET_KEY = "dataset"
+    SUBSET_KEY = "subset"
+    TEMPLATE_FILENAME = "templates.yaml"
+
+    def __init__(self, dataset_name: str, subset_name: str = None):
+        self.dataset_name: str = dataset_name
+        self.subset_name: str = subset_name
+        # dictionary is keyed by template name.
+        self.templates: Dict = self.read_from_file()
+
+        # Mapping from template name to template id
+        self.name_to_id_mapping = {}
+        self.sync_mapping()
+
+    def sync_mapping(self) -> None:
+        """
+        Re-compute the name_to_id_mapping to ensure it is in sync with self.templates
+        """
+        self.name_to_id_mapping = {template.name: template.id for template in self.templates.values()}
+
+    @property
+    def all_template_names(self) -> List[str]:
+        """
+        Sorted list of all templates names for this dataset
+        """
+        return sorted([template.name for template in self.templates.values()])
+
+    @property
+    def folder_path(self) -> str:
+        if self.subset_name:
+            return os.path.join(TEMPLATES_FOLDER_PATH, self.dataset_name, self.subset_name)
+        else:
+            return os.path.join(TEMPLATES_FOLDER_PATH, self.dataset_name)
+
+    @property
+    def yaml_path(self) -> str:
+        return os.path.join(self.folder_path, self.TEMPLATE_FILENAME)
+
+    def format_for_dump(self) -> Dict:
+        """
+        Create a formatted dictionary for the class attributes
+        """
+        formatted_dict = {self.DATASET_KEY: self.dataset_name, self.TEMPLATES_KEY: self.templates}
+        if self.subset_name:
+            formatted_dict[self.SUBSET_KEY] = self.subset_name
+        return formatted_dict
+
+    def read_from_file(self) -> Dict:
+        """
+        Reads a file containing a prompt collection.
+        """
+
+        if not os.path.exists(self.yaml_path):
+            dataset_name = f"{self.dataset_name} {self.subset_name}" if self.subset_name else self.dataset_name
+            logging.warning(
+                f"Tried instantiating `DatasetTemplates` for {dataset_name}, but no prompts found. "
+                "Please ignore this warning if you are creating new prompts for this dataset."
+            )
+            return {}
+        yaml_dict = yaml.load(open(self.yaml_path, "r"), Loader=yaml.FullLoader)
+        return yaml_dict[self.TEMPLATES_KEY]
+
+    def write_to_file(self) -> None:
+        """
+        Writes to a file with the current prompt collection.
+        """
+        # Sync the mapping
+        self.sync_mapping()
+
+        # We only create the folder if a template is written
+        if not os.path.exists(self.folder_path):
+            os.makedirs(self.folder_path)
+        yaml.dump(self.format_for_dump(), open(self.yaml_path, "w"))
+
+    def add_template(self, template: "Template") -> None:
+        """
+        Adds a new template for the dataset
+
+        :param template: template
+        """
+        self.templates[template.get_id()] = template
+
+        self.write_to_file()
+
+    def remove_template(self, template_name: str) -> None:
+        """
+        Deletes a template
+
+        :param template_name: name of template to remove
+        """
+
+        # Even if we have an ID, we want to check for duplicate names
+        if template_name not in self.all_template_names:
+            raise ValueError(f"No template with name {template_name} for dataset {self.dataset_name} exists.")
+
+        del self.templates[self.name_to_id_mapping[template_name]]
+
+        if len(self.templates) == 0:
+            # There is no remaining template, we can remove the entire folder
+            self.delete_folder()
+        else:
+            # We just update the file
+            self.write_to_file()
+
+    def update_template(
+        self,
+        current_template_name: str,
+        new_template_name: str,
+        jinja: str,
+        reference: str,
+        metadata: Template.Metadata,
+        answer_choices: str,
+    ) -> None:
+        """
+        Updates a pre-existing template and writes changes
+
+        :param current_template_name: current name of the template stored in self.templates
+        :param new_template_name: new name for the template
+        :param jinja: new jinja entry
+        :param reference: new reference entry
+        :param metadata: a Metadata object with template annotations
+        :param answer_choices: new answer_choices string
+        """
+        template_id = self.name_to_id_mapping[current_template_name]
+        self.templates[template_id].name = new_template_name
+        self.templates[template_id].jinja = jinja
+        self.templates[template_id].reference = reference
+        self.templates[template_id].metadata = metadata
+        self.templates[template_id].answer_choices = answer_choices
+
+        self.write_to_file()
+
+    def delete_folder(self) -> None:
+        """
+        Delete the folder corresponding to self.folder_path
+        """
+        self.sync_mapping()
+
+        rmtree(self.folder_path)
+
+        # If it is a subset, we have to check whether to remove the dataset folder
+        if self.subset_name:
+            # have to check for other folders
+            base_dataset_folder = os.path.join(TEMPLATES_FOLDER_PATH, self.dataset_name)
+            if len(os.listdir(base_dataset_folder)) == 0:
+                rmtree(base_dataset_folder)
+
+    def __getitem__(self, template_key: str) -> "Template":
+        return self.templates[self.name_to_id_mapping[template_key]]
+
+    def __len__(self) -> int:
+        return len(self.templates)
+
+
+def get_templates_data_frame():
+    """
+    Gathers all template information into a Pandas DataFrame.
+
+    :return: Pandas DataFrame
+    """
+    data = {
+        "id": [],
+        "dataset": [],
+        "subset": [],
+        "name": [],
+        "reference": [],
+        "original_task": [],
+        "choices_in_prompt": [],
+        "metrics": [],
+        "languages": [],
+        "answer_choices": [],
+        "jinja": [],
+    }
+
+    template_collection = TemplateCollection()
+
+    for key in template_collection.keys:
+        templates = template_collection.get_dataset(key[0], key[1])
+        for template_name in templates.all_template_names:
+            template = templates[template_name]
+            data["id"].append(template.get_id())
+            data["dataset"].append(key[0])
+            data["subset"].append(key[1])
+            data["name"].append(template.get_name())
+            data["reference"].append(template.get_reference())
+            data["original_task"].append(template.metadata.original_task)
+            data["choices_in_prompt"].append(template.metadata.choices_in_prompt)
+            data["metrics"].append(template.metadata.metrics)
+            data["languages"].append(template.metadata.languages)
+            data["answer_choices"].append(template.get_answer_choices_expr())
+            data["jinja"].append(template.jinja)
+
+    return pd.DataFrame(data)

promptsource/templates/Zaid/coqa_expanded/templates.yaml

@@ -0,0 +1,130 @@
+dataset: Zaid/coqa_expanded
+templates:
+  12ad4331-d063-4b56-b0f6-76f59c690717: !Template
+    answer_choices: null
+    id: 12ad4331-d063-4b56-b0f6-76f59c690717
+    jinja: "Below is a passage, followed by a series of questions and answers about\
+      \ the passage. Answer the last question based on the information contained in\
+      \ the passage. If there is no answer in the passage, say \"unknown\".\n\nPassage:\
+      \ {{story}}\n\nQ: {{question}} \nA: ||| {% if answer[\"answer_start\"] != -1\
+      \ %}\n{{answer[\"input_text\"]}}\n{% else %}\nunknown\n{% endif %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: Verbose instructions
+    reference: 'Metric: variant of SQuAD (Section 6.1 of the paper)'
+  2f9fb20d-f4c9-4371-9cd4-db47607cb7a3: !Template
+    answer_choices: null
+    id: 2f9fb20d-f4c9-4371-9cd4-db47607cb7a3
+    jinja: "What is the answer to the last question in the dialogue below? If there\
+      \ is no answer in the passage, say \"unknown\".\n\nPassage: {{story}}\n\nQ:\
+      \ {{question}} \nA: ||| {% if answer[\"answer_start\"] != -1 %}\n{{answer[\"\
+      input_text\"]}}\n{% else %}\nunknown\n{% endif %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: What is the answer
+    reference: 'Metric: variant of SQuAD (Section 6.1 of the paper)'
+  9aff8967-d41c-4d79-8ef4-fc3650773735: !Template
+    answer_choices: null
+    id: 9aff8967-d41c-4d79-8ef4-fc3650773735
+    jinja: "Complete the dialogue based on the information contained in the passage.\
+      \ If there is no answer in the passage, say \"unknown\".\n\nPassage: {{story}}\n\
+      \nQ: {{question}} \nA: ||| {% if answer[\"answer_start\"] != -1 %}\n{{answer[\"\
+      input_text\"]}}\n{% else %}\nunknown\n{% endif %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: Complete the dialogue
+    reference: 'Metric: variant of SQuAD (Section 6.1 of the paper)'
+  9bc32f2e-eee6-4006-bce3-74a79403d33e: !Template
+    answer_choices: null
+    id: 9bc32f2e-eee6-4006-bce3-74a79403d33e
+    jinja: "Answer the last question based on the information contained in the passage.\
+      \ If there is no answer in the passage, say \"unknown\".\n\nPassage: {{story}}\n\
+      \nQ: {{question}} \nA: ||| {% if answer[\"answer_start\"] != -1 %}\n{{answer[\"\
+      input_text\"]}}\n{% else %}\nunknown\n{% endif %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: Answer the last question
+    reference: 'Metric: variant of SQuAD (Section 6.1 of the paper)'
+  bacb6534-e607-4afc-a412-ccfcd9fe38e2: !Template
+    answer_choices: null
+    id: bacb6534-e607-4afc-a412-ccfcd9fe38e2
+    jinja: 'In the passage below, extract the part which answers the last question.
+      If there is no answer in the passage, say "unknown".
+
+
+      Passage: {{story}}
+
+
+      Q: {{question}}
+
+      A: |||
+
+      {% if answer["answer_start"] != -1 %}
+
+      {{story[answer["answer_start"] : answer["answer_end"] ]}}
+
+      {% else %}
+
+      unknown
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: false
+    name: extract_answer
+    reference: ''
+  be39974f-aa86-4076-b444-bd3c2732b17b: !Template
+    answer_choices: null
+    id: be39974f-aa86-4076-b444-bd3c2732b17b
+    jinja: "Help me complete the dialogue about this passage. If there is no answer\
+      \ in the passage, say \"unknown\".\n\nPassage: {{story}}\n\nQ: {{question}}\
+      \ \nA: ||| {% if answer[\"answer_start\"] != -1 %}\n{{answer[\"input_text\"\
+      ]}}\n{% else %}\nunknown\n{% endif %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: Help me
+    reference: 'Metric: variant of SQuAD (Section 6.1 of the paper)'
+  d95440ce-d538-40f8-ae09-664e05852ca8: !Template
+    answer_choices: null
+    id: d95440ce-d538-40f8-ae09-664e05852ca8
+    jinja: "{{story}}\n\nQ: {{question}} \nA: ||| {% if answer[\"answer_start\"] !=\
+      \ -1 %}\n{{answer[\"input_text\"]}}\n{% else %}\nunknown\n{% endif %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: GPT-3 Style
+    reference: 'Brown et al. NeurIPS 2020. Metric: variant of SQuAD (Section 6.1 of
+      the paper)'

promptsource/templates/Zaid/quac_expanded/templates.yaml

@@ -0,0 +1,91 @@
+dataset: Zaid/quac_expanded
+templates:
+  01d8c949-89a7-4a44-9a39-6cf2ac3e0a7b: !Template
+    answer_choices: null
+    id: 01d8c949-89a7-4a44-9a39-6cf2ac3e0a7b
+    jinja: "What is the answer to the last question in the dialogue below? If there\
+      \ is no answer in the passage, say \"unknown\".\n\nPassage: {{context}}\n\n\
+      Q: {{question}} \nA: ||| {{answer[\"texts\"][0]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: What is the answer
+    reference: 'Metric: F1'
+  1484c6e6-bf42-47ca-9ea7-c3c552a24de1: !Template
+    answer_choices: null
+    id: 1484c6e6-bf42-47ca-9ea7-c3c552a24de1
+    jinja: "{{context}}\n\nQ: {{question}} \nA: ||| {{answer[\"texts\"][0]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: GPT-3 Style
+    reference: 'Brown et al. NeurIPS 2020. Metric: F1'
+  2bca0532-01a3-4a64-a228-a57ae0965719: !Template
+    answer_choices: null
+    id: 2bca0532-01a3-4a64-a228-a57ae0965719
+    jinja: "Below is a passage, followed by a series of questions and answers about\
+      \ the passage. Answer the last question based on the information contained in\
+      \ the passage. If there is no answer in the passage, say \"unknown\".\n\nPassage:\
+      \ {{context}}\n\nQ: {{question}} \nA: ||| {{answer[\"texts\"][0]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: Verbose instructions
+    reference: 'Metric: F1'
+  4abd0379-dbc0-4f71-901b-dd0af3581157: !Template
+    answer_choices: null
+    id: 4abd0379-dbc0-4f71-901b-dd0af3581157
+    jinja: "Answer the last question based on the information contained in the passage.\
+      \ If there is no answer in the passage, say \"unknown\".\n\nPassage: {{context}}\n\
+      \nQ: {{question}} \nA: ||| {{answer[\"texts\"][0]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: Answer the last question
+    reference: 'Metric: F1'
+  8ebbd098-b40c-4e69-8cbb-0ffecf0fe2a6: !Template
+    answer_choices: null
+    id: 8ebbd098-b40c-4e69-8cbb-0ffecf0fe2a6
+    jinja: "Complete the dialogue based on the information contained in the passage.\
+      \ If there is no answer in the passage, say \"unknown\".\n\nPassage: {{context}}\n\
+      \nQ: {{question}} \nA: ||| {{answer[\"texts\"][0]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: Complete the dialogue
+    reference: 'Metric: F1'
+  e624695b-5d26-47cc-bdb4-ac2bee4ddaea: !Template
+    answer_choices: null
+    id: e624695b-5d26-47cc-bdb4-ac2bee4ddaea
+    jinja: "Help me complete the dialogue about this passage. If there is no answer\
+      \ in the passage, say \"unknown\".\n\nPassage: {{context}}\n\nQ: {{question}}\
+      \ \nA: ||| {{answer[\"texts\"][0]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: Help me
+    reference: 'Metric: F1'

promptsource/templates/acronym_identification/templates.yaml

@@ -0,0 +1,248 @@
+dataset: acronym_identification
+templates:
+  64f438f2-9968-459f-82d2-24bad632b358: !Template
+    answer_choices: null
+    id: 64f438f2-9968-459f-82d2-24bad632b358
+    jinja: "{% set random_abbr = '' %}\n{% set _dummy = none %}\n{% set abbr_exp_dict\
+      \ = namespace(value = {}) %}\n{% set abbr_string=namespace(value='') %}\n{%\
+      \ set exp_string=namespace(value='')%}\n \n{% for label_idx in range(labels|length)\
+      \ %}\n  {% if labels[label_idx] == 0 %}{# Long Beginning #}\n    {% set exp_string.value\
+      \ = tokens[label_idx] %}{# Create new long string #}\n  {% elif labels[label_idx]\
+      \ == 1 %}{# Short Beginning #}\n    {% if abbr_string.value!='' and abbr_string.value\
+      \ not in abbr_exp_dict.value.keys()%}{# Some string already present #}\n   \
+      \   {% set _dummy = abbr_exp_dict.value.update({abbr_string.value:''}) %}{#\
+      \ Discard this string as a new short string is coming #}\n    {% endif %}\n\
+      \    {% set abbr_string.value = tokens[label_idx] %}{# Create new short string\
+      \ #}\n  {% elif labels[label_idx] == 2 %}{# Long Intermediate #}\n    {% set\
+      \ exp_string.value = exp_string.value+' '+tokens[label_idx] %}{# Update existing\
+      \ string #}\n  {% elif labels[label_idx] == 3 %}{# Short Intermediate #}\n \
+      \   {% set abbr_string.value = abbr_string.value+tokens[label_idx] %}{# Update\
+      \ existing string #}\n  {% else %}{# Other #}\n    {# Both non-empty, and first\
+      \ characters match #}\n    {% if abbr_string.value!='' and exp_string.value!=''\
+      \ and exp_string.value.split()[0][0]|lower in abbr_string.value|lower and exp_string.value.split()[-1][0]|lower\
+      \ in abbr_string.value|lower%}\n      {# Update both the dictionaries #}\n \
+      \     {% set _dummy = abbr_exp_dict.value.update({abbr_string.value:exp_string.value})\
+      \ %}\n      {# Empty both the strings #}\n      {% set abbr_string.value= ''\
+      \ %}\n      {% set exp_string.value= '' %}\n    {% endif %}\n  {% endif %}\n\
+      {% endfor %}\n{# Both non-empty, and first characters match #}\n{% if abbr_string.value!=''\
+      \ and exp_string.value!='' %}\n  {% if exp_string.value.split()[0][0]|lower\
+      \ in abbr_string.value|lower and exp_string.value.split()[-1][0]|lower in abbr_string.value|lower\
+      \ %}\n    {# Update both the dictionaries #}\n    {% set _dummy = abbr_exp_dict.value.update({abbr_string.value:exp_string.value})\
+      \ %}\n  {% elif abbr_exp_dict.value.items()|length==0 %}\n    {% set _dummy\
+      \ = abbr_exp_dict.value.update({abbr_string.value:exp_string.value}) %}\n  {%\
+      \ endif %}\n{% else %}\n  {% if abbr_string.value!=''%}\n    {% if abbr_string.value\
+      \ not in abbr_exp_dict.value.keys() %}\n      {% set _dummy = abbr_exp_dict.value.update({abbr_string.value:''})\
+      \ %}\n    {% endif %}\n  {% endif %}\n{% endif %}\n{% if abbr_exp_dict.value\
+      \ %}\n{% set random_abbr = abbr_exp_dict.value.keys()|list|choice %}\nGiven\
+      \ the tokens below, find the expansion (acronym meaning) of \"{{random_abbr}}\"\
+      . Return {{\"\\\"Unclear\\\"\"}} if the expansion can't be found.\n \nTokens:\
+      \ {{tokens|join(' ')}}\nExpansion: |||\n{% if random_abbr in abbr_exp_dict.value.keys()\
+      \ and abbr_exp_dict.value[random_abbr]!='' %}\n{{abbr_exp_dict.value[random_abbr]}}\n\
+      {% else %}\nUnclear\n{% endif %}\n{% endif %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: find_acronym_meaning
+    reference: 'Given the tokens, find the expansion of an abbreviation in the tokens.
+      Metrics: Precision, Recall, F1'
+  81babc83-18cd-4eed-a343-8ede56b21df5: !Template
+    answer_choices: null
+    id: 81babc83-18cd-4eed-a343-8ede56b21df5
+    jinja: "Specification for BIO tags:  \"{{\"B-short\"}}\" and \"{{\"I-short\"}}\"\
+      \ represent respectively the beginning and intermediate tokens for abbreviations\
+      \ (acronyms).\"{{\"B-long\"}}\" and \"{{\"I-long\"}}\" represent respectively\
+      \ the beginning and intermediate tokens for expansions of abbreviations (acronyms\
+      \ meaning). All other tokens are represented by \"{{\"O\"}}\". \n\nGiven the\
+      \ space-separated tokens below,  write down for each token the corresponding\
+      \ BIO tag. Use a space to separate tags in the answer.\n\nTokens: {{tokens|join('\
+      \ ')}}\nBIO tags:|||{% for label in labels %}{{[\"B-long\", \"B-short\", \"\
+      I-long\",  \"I-short\", \"O\"][label]}}{% if not loop.last %} {%endif %}{% endfor\
+      \ %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: acronyms_and_expansions_bio_encode
+    reference: 'Given the comma separated tokens, generate BIO encoding for abbreviations.
+      Metrics: Precision, Recall, F1'
+  8832e5f7-7c45-46da-b85f-71fcb444f264: !Template
+    answer_choices: null
+    id: 8832e5f7-7c45-46da-b85f-71fcb444f264
+    jinja: 'List all the expansions (meanings) of the acronyms present in the following
+      space-separated tokens. Return {{"\"No expansions found\""}} if the expansions
+      can''t be found.
+
+
+      Tokens: {{tokens|join('' '')}}
+
+      |||
+
+      {% set abbr_string=namespace(value='''') %}
+
+      {% set answer_list=namespace(value=[]) %}
+
+      {% for label_idx in range(labels|length) %}
+
+      {% if labels[label_idx] == 0 %}
+
+      {% set abbr_string.value = tokens[label_idx] %}
+
+      {% elif abbr_string.value!='''' and labels[label_idx]==2%}
+
+      {% set abbr_string.value = abbr_string.value+'' ''+tokens[label_idx] %}
+
+      {% elif abbr_string.value!='''' and labels[label_idx]!=2%}
+
+      {% set answer_list.value = answer_list.value +[abbr_string.value] %}
+
+      {% set abbr_string.value = '''' %}
+
+      {% endif %}
+
+      {% if loop.last and abbr_string.value!='''' %}
+
+      {% set answer_list.value = answer_list.value +[abbr_string.value] %}
+
+      {% endif %}
+
+      {% endfor %}
+
+      {% if answer_list.value|length!=0 %}
+
+      {{ answer_list.value|join('', '') }}
+
+      {% else %}
+
+      No expansions found
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: list_expansions
+    reference: 'Given the tokens, list the expansion tokens. Metrics: Precision, Recall,
+      F1'
+  cae58242-cde9-472d-ae9e-56fc7e79c0d1: !Template
+    answer_choices: null
+    id: cae58242-cde9-472d-ae9e-56fc7e79c0d1
+    jinja: "List all the acryonyms in the following space-separated tokens: \n\n{{tokens|join('\
+      \ ')}}\n|||\n{% set abbr_string=namespace(value='') %}\n{% set answer_list=namespace(value=[])\
+      \ %}\n{% for label_idx in range(labels|length) %}\n{% if labels[label_idx] ==\
+      \ 1 %}\n{% set abbr_string.value = tokens[label_idx] %}\n{% elif abbr_string.value!=''\
+      \ and labels[label_idx]==3%}\n{% set abbr_string.value = abbr_string.value+tokens[label_idx]\
+      \ %}\n{% elif abbr_string.value!='' and labels[label_idx]!=3 %}\n{% set answer_list.value\
+      \ = answer_list.value +[abbr_string.value] %}\n{% set abbr_string.value = ''\
+      \ %}\n{% endif %}\n{% if loop.last and abbr_string.value!='' %}\n{% set answer_list.value\
+      \ = answer_list.value +[abbr_string.value] %}\n{% endif %}\n{% endfor %}\n{{\
+      \ answer_list.value|join(', ') }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: list_abbreviations
+    reference: 'Given the tokens, list the abbreviations. Metrics: Precision, Recall,
+      F1'
+  e4e42433-0e37-4aa5-bbce-7f336ecac6a3: !Template
+    answer_choices: null
+    id: e4e42433-0e37-4aa5-bbce-7f336ecac6a3
+    jinja: "{% set _dummy = none %}\n{% set abbr_exp_dict = namespace(value = {})\
+      \ %}\n{% set abbr_string=namespace(value='') %}\n{% set exp_string=namespace(value='')%}\n\
+      \ \n{% for label_idx in range(labels|length) %}\n  {% if labels[label_idx] ==\
+      \ 0 %}{# Long Beginning #}\n    {% set exp_string.value = tokens[label_idx]\
+      \ %}{# Create new long string #}\n  {% elif labels[label_idx] == 1 %}{# Short\
+      \ Beginning #}\n    {% if abbr_string.value!='' and abbr_string.value not in\
+      \ abbr_exp_dict.value.keys()%}{# Some string already present #}\n      {% set\
+      \ _dummy = abbr_exp_dict.value.update({abbr_string.value:''}) %}{# Discard this\
+      \ string as a new short string is coming #}\n    {% endif %}\n    {% set abbr_string.value\
+      \ = tokens[label_idx] %}{# Create new short string #}\n  {% elif labels[label_idx]\
+      \ == 2 %}{# Long Intermediate #}\n    {% set exp_string.value = exp_string.value+'\
+      \ '+tokens[label_idx] %}{# Update existing string #}\n  {% elif labels[label_idx]\
+      \ == 3 %}{# Short Intermediate #}\n    {% set abbr_string.value = abbr_string.value+tokens[label_idx]\
+      \ %}{# Update existing string #}\n  {% else %}{# Other #}\n    {# Both non-empty,\
+      \ and first characters match #}\n    {% if abbr_string.value!='' and exp_string.value!=''\
+      \ and exp_string.value.split()[0][0]|lower in abbr_string.value|lower and exp_string.value.split()[-1][0]|lower\
+      \ in abbr_string.value|lower%}\n      {# Update both the dictionaries #}\n \
+      \     {% set _dummy = abbr_exp_dict.value.update({abbr_string.value:exp_string.value})\
+      \ %}\n      {# Empty both the strings #}\n      {% set abbr_string.value= ''\
+      \ %}\n      {% set exp_string.value= '' %}\n    {% endif %}\n  {% endif %}\n\
+      {% endfor %}\n{# Both non-empty, and first characters match #}\n{% if abbr_string.value!=''\
+      \ and exp_string.value!='' %}\n  {% if exp_string.value.split()[0][0]|lower\
+      \ in abbr_string.value|lower and exp_string.value.split()[-1][0]|lower in abbr_string.value|lower\
+      \ %}\n    {# Update both the dictionaries #}\n    {% set _dummy = abbr_exp_dict.value.update({abbr_string.value:exp_string.value})\
+      \ %}\n  {% elif abbr_exp_dict.value.items()|length==0 %}\n    {% set _dummy\
+      \ = abbr_exp_dict.value.update({abbr_string.value:exp_string.value}) %}\n  {%\
+      \ endif %}\n{% else %}\n  {% if abbr_string.value!=''%}\n    {% if abbr_string.value\
+      \ not in abbr_exp_dict.value.keys() %}\n      {% set _dummy = abbr_exp_dict.value.update({abbr_string.value:''})\
+      \ %}\n    {% endif %}\n  {% endif %}\n{% endif %}\n \nGiven the following tokens,\
+      \ find the abbreviations (acronyms) and their expansions (acronyms meaning).\
+      \ Return {{\"\\\"Unclear\\\"\"}} if the expansion can't be found.\n \nTokens:\
+      \ {{tokens|join(' ')}}\n|||\n{% for item, value in abbr_exp_dict.value.items()\
+      \ %}\n{{item}} : {% if value!='' %}{{value}}{% else %}Unclear{% endif %}\n{%endfor%}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: find_acronyms_and_expansions
+    reference: 'Given the tokens, find the abbreviation mapping. Metrics: Precision,
+      Recall, F1'
+  eed32ee4-ebc3-499f-ba61-e91461f56ccb: !Template
+    answer_choices: null
+    id: eed32ee4-ebc3-499f-ba61-e91461f56ccb
+    jinja: "{% set random_exp = '' %}{% set _dummy = none %}{% set exp_abbr_dict =\
+      \ namespace(value = {}) %}{% set abbr_string=namespace(value='') %}{% set exp_string=namespace(value='')%}{%\
+      \ for label_idx in range(labels|length) %}{% if labels[label_idx] == 0 %}{#\
+      \ Long Beginning #}{% if exp_string.value!='' and exp_string.value not in exp_abbr_dict.value.keys()\
+      \ %}{# Some string already present #}{% set _dummy = exp_abbr_dict.value.update({exp_string.value:''})\
+      \ %}{# Discard this string as a new long string is coming #} {% endif %}{% set\
+      \ exp_string.value = tokens[label_idx] %}{# Create new long string #}{% elif\
+      \ labels[label_idx] == 1 %}{# Short Beginning #}{% set abbr_string.value = tokens[label_idx]\
+      \ %}{# Create new short string #}{% elif labels[label_idx] == 2 %}{# Long Intermediate\
+      \ #}{% set exp_string.value = exp_string.value+' '+tokens[label_idx] %}{# Update\
+      \ existing string #}{% elif labels[label_idx] == 3 %}{# Short Intermediate #}{%\
+      \ set abbr_string.value = abbr_string.value+tokens[label_idx] %}{# Update existing\
+      \ string #}{% else %}{# Other #}{# Both non-empty, and first characters match\
+      \ #}{% if abbr_string.value!='' and exp_string.value!='' and exp_string.value.split()[0][0]|lower\
+      \ in abbr_string.value|lower and exp_string.value.split()[-1][0]|lower in abbr_string.value|lower%}{#\
+      \ Update both the dictionaries #}{% set _dummy = exp_abbr_dict.value.update({exp_string.value:abbr_string.value})\
+      \ %}{# Empty both the strings #}{% set abbr_string.value= '' %}{% set exp_string.value=\
+      \ '' %}{% endif %}{% endif %}{% endfor %}{# Both non-empty, and first characters\
+      \ match #}{% if abbr_string.value!='' and exp_string.value!='' %}{% if exp_string.value.split()[0][0]|lower\
+      \ in abbr_string.value|lower and exp_string.value.split()[-1][0]|lower in abbr_string.value|lower\
+      \ %}{# Update the dictionary #}{% set _dummy = exp_abbr_dict.value.update({exp_string.value:abbr_string.value})\
+      \ %}{% elif exp_abbr_dict.value.items()|length==0 %}{% set _dummy = exp_abbr_dict.value.update({exp_string.value:abbr_string.value})\
+      \ %}{% endif %}{% else %}{% if exp_string.value!='' %}{% if exp_string.value\
+      \ not in exp_abbr_dict.value.keys() %}{% set _dummy = exp_abbr_dict.value.update({exp_string.value:''})\
+      \ %}{% endif %}{% endif %}{% endif %}{% if exp_abbr_dict.value.items()|length!=0\
+      \ %}{% set random_exp = exp_abbr_dict.value.keys()|list|choice %}Given the tokens\
+      \ below, find the abbreviation (acronym) for: \"{{random_exp}}\". Return {{\"\
+      \\\"Unclear\\\"\"}} if the abbreviation can't be found.\n \nTokens: {{tokens|join('\
+      \ ')}}\nAcronyms: |||{% if random_exp in exp_abbr_dict.value.keys() and exp_abbr_dict.value[random_exp]!=''\
+      \ %}{{exp_abbr_dict.value[random_exp]}}{% else %}Unclear{% endif %}{% endif\
+      \ %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: find_acronym
+    reference: 'Given the tokens, find the abbreviation for an expansion. Metrics:
+      Precision, Recall, F1'

promptsource/templates/ade_corpus_v2/Ade_corpus_v2_classification/templates.yaml

@@ -0,0 +1,50 @@
+dataset: ade_corpus_v2
+subset: Ade_corpus_v2_classification
+templates:
+  56bd12a8-b8ee-464e-98cc-5f586ba9f74d: !Template
+    answer_choices: No ||| Yes
+    id: 56bd12a8-b8ee-464e-98cc-5f586ba9f74d
+    jinja: 'Please answer the below Yes / No question.
+
+
+      Is "{{text}}" related to adverse drug effect (ADE)? ||| {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: binary-classification
+    reference: ''
+  78c4ce65-dd66-46ed-878d-11f4eca5e544: !Template
+    answer_choices: No ||| Yes
+    id: 78c4ce65-dd66-46ed-878d-11f4eca5e544
+    jinja: "Read the below text and answer the question.\n\nText: {{text}} \n\nQuestion:\
+      \ Is the above text related to adverse drug effect (ADE)? Your answer should\
+      \ be either \"Yes\" or \"No\".\n\n|||\n{{answer_choices[label]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: verbose-binary-classification
+    reference: ''
+  dabc0337-5bd3-4150-98b3-794a15ce1a3a: !Template
+    answer_choices: null
+    id: dabc0337-5bd3-4150-98b3-794a15ce1a3a
+    jinja: "{% if label==1 %}\nPlease write a short medical report that is related\
+      \ to adverse drug effect (ADE). \n{% else %}\nWrite a medical report that is\
+      \ not related to adverse drug effect (ADE). \n{% endif %}\n|||\n{{text}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: label-to-text
+    reference: ''

promptsource/templates/ade_corpus_v2/Ade_corpus_v2_drug_ade_relation/templates.yaml

@@ -0,0 +1,125 @@
+dataset: ade_corpus_v2
+subset: Ade_corpus_v2_drug_ade_relation
+templates:
+  0ec35408-652d-4ebc-9478-5a0d330c24c8: !Template
+    answer_choices: null
+    id: 0ec35408-652d-4ebc-9478-5a0d330c24c8
+    jinja: 'Read the below text and answer the question.
+
+
+      Text: {{text}}
+
+
+      Question: What drug has an effect of {{effect}}?
+
+      |||
+
+      {{drug}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: find-drug
+    reference: ''
+  2682a789-a435-4976-b34f-f376991c842a: !Template
+    answer_choices: null
+    id: 2682a789-a435-4976-b34f-f376991c842a
+    jinja: '{{drug}} has an effect of {{effect}}. Please write a short medical report
+      about this.
+
+      |||
+
+      {{text}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - ROUGE
+      - BLEU
+      original_task: false
+    name: drug-and-effect-to-text
+    reference: ''
+  61ba3622-72bc-4fd8-acfc-826bc2a93aa5: !Template
+    answer_choices: null
+    id: 61ba3622-72bc-4fd8-acfc-826bc2a93aa5
+    jinja: 'Read the below text and answer the question.
+
+
+      Text: {{text}}
+
+
+      Question: What effect does {{drug}} have?
+
+      |||
+
+      {{effect}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: find-effect
+    reference: ''
+  6acf3588-baa1-4ff6-87c4-4c2356855464: !Template
+    answer_choices: null
+    id: 6acf3588-baa1-4ff6-87c4-4c2356855464
+    jinja: 'Read the below text and answer the question.
+
+
+      Text: {{text}}
+
+
+      Question: What are the drug and its effect of the above text?
+
+
+      You should answer in the "drug" and "effect" format (e.g., alcohol and high
+      blood pressure)
+
+      |||
+
+      {{drug}} and {{effect}}.'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: find-drug-and-effect
+    reference: ''
+  db68e609-ba92-40ae-b161-8b7710124142: !Template
+    answer_choices: null
+    id: db68e609-ba92-40ae-b161-8b7710124142
+    jinja: 'Read the below text and answer the two following questions.
+
+
+      Text: {{text}}
+
+
+      Question 1: What is the drug in the above text?
+
+
+      Question 2: What is the effect of it?
+
+
+      You should answer in the "drug" and "effect" format (e.g., alcohol and high
+      blood pressure)
+
+      |||
+
+      {{drug}} and {{effect}}.'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: find-drug-and-effect-two-questions
+    reference: ''

promptsource/templates/ade_corpus_v2/Ade_corpus_v2_drug_dosage_relation/templates.yaml

@@ -0,0 +1,114 @@
+dataset: ade_corpus_v2
+subset: Ade_corpus_v2_drug_dosage_relation
+templates:
+  1de6d411-ed0a-4d48-806e-cad009f07a65: !Template
+    answer_choices: null
+    id: 1de6d411-ed0a-4d48-806e-cad009f07a65
+    jinja: 'Read the below text and answer the question.
+
+
+      Text: {{text}}
+
+
+      Question: What drug has a dosage of {{dosage}}?
+
+      |||
+
+      {{drug}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: find-drug
+    reference: ''
+  1e719388-59c9-4b0a-9ed9-dd02b6ddd0a6: !Template
+    answer_choices: null
+    id: 1e719388-59c9-4b0a-9ed9-dd02b6ddd0a6
+    jinja: '{{dosage}} of {{drug}} was given to a patient. Please write a short medical
+      report about this.
+
+      |||
+
+      {{text}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: drug-and-dosage-to-text
+    reference: ''
+  2bed0f04-8249-4248-86ea-e3a1971b2e1b: !Template
+    answer_choices: null
+    id: 2bed0f04-8249-4248-86ea-e3a1971b2e1b
+    jinja: 'Read the below text and answer the two following questions.
+
+
+      Text: {{text}}
+
+
+
+      Question 1: What is the drug in the above text?
+
+
+      Question 2: What is the dosage of it?
+
+
+      You should answer in the "drug" and "dosage" format (e.g., Aspirin and 500mg)
+
+      |||
+
+      {{drug}} and {{dosage}}.'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: find-drug-and-dosage-two-questions
+    reference: ''
+  ca175bed-d046-40e7-9dbb-1e50fde7e603: !Template
+    answer_choices: null
+    id: ca175bed-d046-40e7-9dbb-1e50fde7e603
+    jinja: 'Read the below text and answer the question.
+
+
+      Text: {{text}}
+
+
+      Question: What is the dosage of {{drug}}?
+
+      |||
+
+      {{dosage}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: find-dosage
+    reference: ''
+  ce5208ac-6b4c-4a35-8738-e20232df1917: !Template
+    answer_choices: null
+    id: ce5208ac-6b4c-4a35-8738-e20232df1917
+    jinja: "Read the below text and answer the question.\n\nText: {{text}}\n\nQuestion:\
+      \ What are the drug and its dosage of the above text? \n\nYou should answer\
+      \ in the \"drug\" and \"dosage\" format (e.g., Aspirin and 500mg)\n|||\n{{drug}}\
+      \ and {{dosage}}."
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: find-drug-and-dosage
+    reference: ''

promptsource/templates/adversarial_qa/adversarialQA/templates.yaml

@@ -0,0 +1,120 @@
+dataset: adversarial_qa
+subset: adversarialQA
+templates:
+  00755780-f3c0-44b4-b159-8f3873cdb16c: !Template
+    answer_choices: null
+    id: 00755780-f3c0-44b4-b159-8f3873cdb16c
+    jinja: 'I want to test the ability of students to read a passage and answer questions
+      about it. Could you please come up with a good question for the passage "{{context}}"?
+      |||
+
+      {{question}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: generate_question
+    reference: 'Input: Context, Output: Question (generate a question)'
+  3b2459cc-6600-443c-abf8-8f60c34cd998: !Template
+    answer_choices: null
+    id: 3b2459cc-6600-443c-abf8-8f60c34cd998
+    jinja: '{% if metadata.split != "test" %}
+
+      I know that the answer to the question "{{question}}" is in "{{context}}". Can
+      you tell me what it is? |||
+
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: tell_what_it_is
+    reference: 'Input: QC, Output: A (rephrase)'
+  5bdb1815-5c6f-49a3-ad1d-367344420701: !Template
+    answer_choices: null
+    id: 5bdb1815-5c6f-49a3-ad1d-367344420701
+    jinja: '{% if metadata.split != "test" %}
+
+      Question: "{{question}}"
+
+
+      Context: "{{context}}"
+
+
+      Answer:
+
+      |||
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: question_context_answer
+    reference: 'Input: QC, Output: Answer (short form)'
+  a0872cde-2f19-4ae6-919a-868da47bfbcb: !Template
+    answer_choices: null
+    id: a0872cde-2f19-4ae6-919a-868da47bfbcb
+    jinja: '{% if metadata.split != "test" %}
+
+      Extract the answer to the question from the following context.
+
+      Question: {{question}}
+
+      Context: {{context}}|||
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: based_on
+    reference: ''
+  a64d5a15-68e2-4d1c-b30a-ca8250c860f9: !Template
+    answer_choices: null
+    id: a64d5a15-68e2-4d1c-b30a-ca8250c860f9
+    jinja: '{% if metadata.split != "test" %}
+
+      Given the following passage
+
+
+      "{{context}}",
+
+
+      answer the following question. Note that the answer is present within the text.
+
+
+      Question: {{question}} |||
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: answer_the_following_q
+    reference: 'Input: QC, Output: Answer'

promptsource/templates/adversarial_qa/dbert/templates.yaml

@@ -0,0 +1,120 @@
+dataset: adversarial_qa
+subset: dbert
+templates:
+  00755780-f3c0-44b4-b159-8f3873cdb16a: !Template
+    answer_choices: null
+    id: 00755780-f3c0-44b4-b159-8f3873cdb16a
+    jinja: 'I want to test the ability of students to read a passage and answer questions
+      about it. Could you please come up with a good question for the passage "{{context}}"?
+      |||
+
+      {{question}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: generate_question
+    reference: 'Input: Context, Output: Question (generate a question)'
+  3b2459cc-6600-443c-abf8-8f60c34cd99a: !Template
+    answer_choices: null
+    id: 3b2459cc-6600-443c-abf8-8f60c34cd99a
+    jinja: '{% if metadata.split != "test" %}
+
+      I know that the answer to the question "{{question}}" is in "{{context}}". Can
+      you tell me what it is? |||
+
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: tell_what_it_is
+    reference: 'Input: QC, Output: A (rephrase)'
+  5bdb1815-5c6f-49a3-ad1d-36734442070a: !Template
+    answer_choices: null
+    id: 5bdb1815-5c6f-49a3-ad1d-36734442070a
+    jinja: '{% if metadata.split != "test" %}
+
+      Question: "{{question}}"
+
+
+      Context: "{{context}}"
+
+
+      Answer:
+
+      |||
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: question_context_answer
+    reference: 'Input: QC, Output: Answer (short form)'
+  a0872cde-2f19-4ae6-919a-868da47bfbca: !Template
+    answer_choices: null
+    id: a0872cde-2f19-4ae6-919a-868da47bfbca
+    jinja: '{% if metadata.split != "test" %}
+
+      Extract the answer to the question from the following context.
+
+      Question: {{question}}
+
+      Context: {{context}}|||
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: based_on
+    reference: ''
+  a64d5a15-68e2-4d1c-b30a-ca8250c860fa: !Template
+    answer_choices: null
+    id: a64d5a15-68e2-4d1c-b30a-ca8250c860fa
+    jinja: '{% if metadata.split != "test" %}
+
+      Given the following passage
+
+
+      "{{context}}",
+
+
+      answer the following question. Note that the answer is present within the text.
+
+
+      Question: {{question}} |||
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: answer_the_following_q
+    reference: 'Input: QC, Output: Answer'

promptsource/templates/adversarial_qa/dbidaf/templates.yaml

@@ -0,0 +1,120 @@
+dataset: adversarial_qa
+subset: dbidaf
+templates:
+  41f28b31-d0fc-4f20-a0a2-ff21813e298e: !Template
+    answer_choices: null
+    id: 41f28b31-d0fc-4f20-a0a2-ff21813e298e
+    jinja: '{% if metadata.split != "test" %}
+
+      Extract the answer to the question from the following context.
+
+      Question: {{question}}
+
+      Context: {{context}}|||
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: based_on
+    reference: ''
+  a64d5a15-68e2-4d1c-b30a-ca8250c860d9: !Template
+    answer_choices: null
+    id: a64d5a15-68e2-4d1c-b30a-ca8250c860d9
+    jinja: '{% if metadata.split != "test" %}
+
+      Given the following passage
+
+
+      "{{context}}",
+
+
+      answer the following question. Note that the answer is present within the text.
+
+
+      Question: {{question}} |||
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: answer_the_following_q
+    reference: 'Input: QC, Output: Answer'
+  c7a80603-d610-4999-98a7-815b2f84592d: !Template
+    answer_choices: null
+    id: c7a80603-d610-4999-98a7-815b2f84592d
+    jinja: 'I want to test the ability of students to read a passage and answer questions
+      about it. Could you please come up with a good question for the passage "{{context}}"?
+      |||
+
+      {{question}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: generate_question
+    reference: 'Input: Context, Output: Question (generate a question)'
+  ce9bc00a-567b-4c4e-aad7-df6f5d5d57bb: !Template
+    answer_choices: null
+    id: ce9bc00a-567b-4c4e-aad7-df6f5d5d57bb
+    jinja: '{% if metadata.split != "test" %}
+
+      I know that the answer to the question "{{question}}" is in "{{context}}". Can
+      you tell me what it is? |||
+
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: tell_what_it_is
+    reference: 'Input: QC, Output: A (rephrase)'
+  fa185424-6ebe-49b8-b4ed-7632ca33c361: !Template
+    answer_choices: null
+    id: fa185424-6ebe-49b8-b4ed-7632ca33c361
+    jinja: '{% if metadata.split != "test" %}
+
+      Question: "{{question}}"
+
+
+      Context: "{{context}}"
+
+
+      Answer:
+
+      |||
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: question_context_answer
+    reference: 'Input: QC, Output: Answer (short form)'

promptsource/templates/adversarial_qa/droberta/templates.yaml

@@ -0,0 +1,120 @@
+dataset: adversarial_qa
+subset: droberta
+templates:
+  00755780-f3c0-44b4-b159-8f3873cdb163: !Template
+    answer_choices: null
+    id: 00755780-f3c0-44b4-b159-8f3873cdb163
+    jinja: 'I want to test the ability of students to read a passage and answer questions
+      about it. Could you please come up with a good question for the passage "{{context}}"?
+      |||
+
+      {{question}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: generate_question
+    reference: 'Input: Context, Output: Question (generate a question)'
+  3b2459cc-6600-443c-abf8-8f60c34cd993: !Template
+    answer_choices: null
+    id: 3b2459cc-6600-443c-abf8-8f60c34cd993
+    jinja: '{% if metadata.split != "test" %}
+
+      I know that the answer to the question "{{question}}" is in "{{context}}". Can
+      you tell me what it is? |||
+
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: tell_what_it_is
+    reference: 'Input: QC, Output: A (rephrase)'
+  5bdb1815-5c6f-49a3-ad1d-367344420703: !Template
+    answer_choices: null
+    id: 5bdb1815-5c6f-49a3-ad1d-367344420703
+    jinja: '{% if metadata.split != "test" %}
+
+      Question: "{{question}}"
+
+
+      Context: "{{context}}"
+
+
+      Answer:
+
+      |||
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: question_context_answer
+    reference: 'Input: QC, Output: Answer (short form)'
+  a0872cde-2f19-4ae6-919a-868da47bfbc3: !Template
+    answer_choices: null
+    id: a0872cde-2f19-4ae6-919a-868da47bfbc3
+    jinja: '{% if metadata.split != "test" %}
+
+      Extract the answer to the question from the following context.
+
+      Question: {{question}}
+
+      Context: {{context}}|||
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: based_on
+    reference: ''
+  a64d5a15-68e2-4d1c-b30a-ca8250c860f3: !Template
+    answer_choices: null
+    id: a64d5a15-68e2-4d1c-b30a-ca8250c860f3
+    jinja: '{% if metadata.split != "test" %}
+
+      Given the following passage
+
+
+      "{{context}}",
+
+
+      answer the following question. Note that the answer is present within the text.
+
+
+      Question: {{question}} |||
+
+      {{answers.text | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Squad
+      original_task: true
+    name: answer_the_following_q
+    reference: 'Input: QC, Output: Answer'

promptsource/templates/aeslc/templates.yaml

@@ -0,0 +1,163 @@
+dataset: aeslc
+templates:
+  0bef38b8-6d0b-440b-8a3d-db034aaf5a15: !Template
+    answer_choices: null
+    id: 0bef38b8-6d0b-440b-8a3d-db034aaf5a15
+    jinja: '{{ email_body }}
+
+
+      What is this email about? |||
+
+
+      {{ subject_line }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - ROUGE
+      - Other
+      original_task: true
+    name: what_is_this_email_about
+    reference: Ask a question from a context
+  11de8b2c-8016-4b98-b5f2-c1a7e5c0e433: !Template
+    answer_choices: null
+    id: 11de8b2c-8016-4b98-b5f2-c1a7e5c0e433
+    jinja: 'What is the subject of this email:
+
+
+      {{ email_body }} |||
+
+
+      {{ subject_line }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - ROUGE
+      - Other
+      original_task: true
+    name: what_is_the_subject_of_this_email
+    reference: Ask a question from a context
+  12616e45-1d61-4924-8ce4-fe3efd061e7a: !Template
+    answer_choices: null
+    id: 12616e45-1d61-4924-8ce4-fe3efd061e7a
+    jinja: 'The text below is the content of an email. What is the topic of this email?
+
+
+      {{ email_body }} |||
+
+
+      {{ subject_line }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - ROUGE
+      - Other
+      original_task: true
+    name: the_text_below
+    reference: ''
+  25179c66-5638-4de5-bdce-d6dccec64c65: !Template
+    answer_choices: null
+    id: 25179c66-5638-4de5-bdce-d6dccec64c65
+    jinja: 'Generate a subject line for the email body below:
+
+
+      {{ email_body }} |||
+
+
+      {{ subject_line }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - ROUGE
+      - Other
+      original_task: true
+    name: generate_subject_line
+    reference: Instruct to generate
+  8917d7f0-5f72-418f-a2d9-98d4a8da13b0: !Template
+    answer_choices: null
+    id: 8917d7f0-5f72-418f-a2d9-98d4a8da13b0
+    jinja: 'What is this email about:
+
+
+      {{ email_body }} |||
+
+
+      {{ subject_line }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - ROUGE
+      - Other
+      original_task: true
+    name: what_about
+    reference: Ask a question from a context
+  d1c5da3f-f1e4-4891-abcb-79463b30a616: !Template
+    answer_choices: null
+    id: d1c5da3f-f1e4-4891-abcb-79463b30a616
+    jinja: '{{ email_body }}
+
+
+      What is the subject of this email? |||
+
+
+      {{ subject_line }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - ROUGE
+      - Other
+      original_task: true
+    name: what_subject_of_email
+    reference: Ask a question from a context
+  d9dd8e72-acb4-4aad-aeb7-a877bacbb402: !Template
+    answer_choices: null
+    id: d9dd8e72-acb4-4aad-aeb7-a877bacbb402
+    jinja: '{{ email_body }}
+
+
+      Generate a subject line for the email body above. |||
+
+
+      {{ subject_line }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - ROUGE
+      - Other
+      original_task: true
+    name: generate_subject
+    reference: Instruct to generate
+  dca29ebb-2372-423f-b93c-21d99eddf455: !Template
+    answer_choices: null
+    id: dca29ebb-2372-423f-b93c-21d99eddf455
+    jinja: '{{ email_body }}
+
+
+      The above text is the content of an email. What is the topic of this email?
+      |||
+
+
+      {{ subject_line }} '
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - ROUGE
+      - Other
+      original_task: true
+    name: what_topic
+    reference: ''

promptsource/templates/ag_news/templates.yaml

@@ -0,0 +1,108 @@
+dataset: ag_news
+templates:
+  24e44a81-a18a-42dd-a71c-5b31b2d2cb39: !Template
+    answer_choices: World politics ||| Sports ||| Business ||| Science and technology
+    id: 24e44a81-a18a-42dd-a71c-5b31b2d2cb39
+    jinja: "What label best describes this news article?\n{{text}} ||| \n{{answer_choices[label]\
+      \ }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: classify_question_first
+    reference: ''
+  8fdc1056-1029-41a1-9c67-354fc2b8ceaf: !Template
+    answer_choices: World politics ||| Sports ||| Business ||| Science and technology
+    id: 8fdc1056-1029-41a1-9c67-354fc2b8ceaf
+    jinja: "Is this a piece of news regarding {{\"world politics, sports, business,\
+      \ or science and technology\"}}?\n{{text}} \n||| \n{{answer_choices[label] }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: classify_with_choices_question_first
+    reference: ''
+  918267e0-af68-4117-892d-2dbe66a58ce9: !Template
+    answer_choices: Politician ||| Athlete ||| Business executive ||| Scientist
+    id: 918267e0-af68-4117-892d-2dbe66a58ce9
+    jinja: 'Would you recommend the following article to a {{"politician"}}, an {{"athlete"}},
+      a {{"business executive"}}, or a {{"scientist"}}?
+
+
+      {{ text }}
+
+      |||
+
+      {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: recommend
+    reference: ''
+  9345df33-4f23-4944-a33c-eef94e626862: !Template
+    answer_choices: World News ||| Sports ||| Business ||| Science and Technology
+    id: 9345df33-4f23-4944-a33c-eef94e626862
+    jinja: "{{text}} \n\nWhich of the following sections of a newspaper would this\
+      \ article likely appear in? {{\"World News\"}}, {{\"Sports\"}}, {{\"Business\"\
+      }}, or {{\"Science and Technology\"}}? ||| \n{{answer_choices[label] }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: which_section_choices
+    reference: ''
+  98534347-fff7-4c39-a795-4e69a44791f7: !Template
+    answer_choices: World News ||| Sports ||| Business ||| Science and Technology
+    id: 98534347-fff7-4c39-a795-4e69a44791f7
+    jinja: "{{text}} \n\nWhich section of a newspaper would this article likely appear\
+      \ in? ||| \n{{answer_choices[label] }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: which_section
+    reference: ''
+  b401b0ee-6ffe-4a91-8e15-77ee073cd858: !Template
+    answer_choices: World politics ||| Sports ||| Business ||| Science and technology
+    id: b401b0ee-6ffe-4a91-8e15-77ee073cd858
+    jinja: "{{text}} \nIs this a piece of news regarding {{\"world politics, sports,\
+      \ business, or science and technology\"}}? ||| \n{{answer_choices[label] }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: classify_with_choices
+    reference: ''
+  cb355f33-7e8c-4455-a72b-48d315bd4f60: !Template
+    answer_choices: World politics ||| Sports ||| Business ||| Science and technology
+    id: cb355f33-7e8c-4455-a72b-48d315bd4f60
+    jinja: "{{text}} \nWhat label best describes this news article? ||| \n{{answer_choices[label]\
+      \ }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: classify
+    reference: ''

promptsource/templates/ai2_arc/ARC-Challenge/templates.yaml

@@ -0,0 +1,142 @@
+dataset: ai2_arc
+subset: ARC-Challenge
+templates:
+  32f7eb4d-dd38-4503-b67d-a8a96ab40449: !Template
+    answer_choices: null
+    id: 32f7eb4d-dd38-4503-b67d-a8a96ab40449
+    jinja: 'Pick and copy all the incorrect options for the following question:
+
+
+      {{question}}
+
+
+      Options:
+
+      - {{choices["text"] | join("\n- ")}}|||
+
+      {% for i in range(choices["label"]|length) %}
+
+      {% if i != choices["label"].index(answerKey) %}
+
+      - {{choices["text"][i]}}
+
+      {% endif %}
+
+      {% endfor %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - Other
+      original_task: false
+    name: pick_false_options
+    reference: ''
+  540ebc31-2ea6-4feb-a6fd-67b6e71cf20a: !Template
+    answer_choices: '{{choices.label | join("|||")}}'
+    id: 540ebc31-2ea6-4feb-a6fd-67b6e71cf20a
+    jinja: "Here's a problem to solve: {{question}}\n\nAmong the 4 following options,\
+      \ which is the correct answer?\n{% for letter, t in zip(answer_choices, choices.text)\
+      \ %}\n- {{letter}}: {{t}}\n {% endfor %}|||{{answerKey}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: heres_a_problem
+    reference: ''
+  5ec2b8ca-e4c0-444e-b097-89ccce811550: !Template
+    answer_choices: '{{choices.text | join("|||")}}'
+    id: 5ec2b8ca-e4c0-444e-b097-89ccce811550
+    jinja: '{{question}}
+
+
+      Options:
+
+      - {{answer_choices | join("\n- ")}}|||
+
+      {{answer_choices[choices["label"].index(answerKey)]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: qa_options
+    reference: ''
+  5ff84886-9d5f-40d1-80d7-2a39b7c16ec6: !Template
+    answer_choices: '{{choices.text | join("|||")}}'
+    id: 5ff84886-9d5f-40d1-80d7-2a39b7c16ec6
+    jinja: 'I am hesitating between 4 options to answer the following question, which
+      option should I choose?
+
+      Question: {{question}}
+
+      Possibilities:
+
+      - {{answer_choices | join("\n- ")}}|||
+
+      {{answer_choices[choices["label"].index(answerKey)]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: i_am_hesitating
+    reference: ''
+  ced2b33b-b590-4522-b041-51d7dd669561: !Template
+    answer_choices: '{{choices.text | join("|||")}}'
+    id: ced2b33b-b590-4522-b041-51d7dd669561
+    jinja: 'I gave my students this multiple choice question: {{question}}
+
+
+      Only one answer is correct among these 4 choices:
+
+      - {{answer_choices | join("\n- ")}}
+
+
+      Could you tell me which one is correct?|||
+
+      {{answer_choices[choices["label"].index(answerKey)]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: multiple_choice
+    reference: ''
+  e371fc1a-8edb-477b-b345-9d73e97ffade: !Template
+    answer_choices: '{{choices.label | join("|||")}}'
+    id: e371fc1a-8edb-477b-b345-9d73e97ffade
+    jinja: 'Pick the most correct option to answer the following question.
+
+
+      {{question}}
+
+
+      Options:
+
+      {% for letter, t in zip(answer_choices, choices.text) %}
+
+      - {{letter}}: {{t}}
+
+      {% endfor %} |||
+
+      {{answerKey}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: pick_the_most_correct_option
+    reference: ''

promptsource/templates/ai2_arc/ARC-Easy/templates.yaml

@@ -0,0 +1,142 @@
+dataset: ai2_arc
+subset: ARC-Easy
+templates:
+  033498ca-3d9a-47e3-b631-d881ab53b5ad: !Template
+    answer_choices: '{{choices.label | join("|||")}}'
+    id: 033498ca-3d9a-47e3-b631-d881ab53b5ad
+    jinja: 'Pick the most correct option to answer the following question.
+
+
+      {{question}}
+
+
+      Options:
+
+      {% for letter, t in zip(answer_choices, choices.text) %}
+
+      - {{letter}}: {{t}}
+
+      {% endfor %} |||
+
+      {{answerKey}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: pick_the_most_correct_option
+    reference: ''
+  252aa566-9482-4e81-aad9-664a9bebd8e8: !Template
+    answer_choices: '{{choices.text | join("|||")}}'
+    id: 252aa566-9482-4e81-aad9-664a9bebd8e8
+    jinja: '{{question}}
+
+
+      Options:
+
+      - {{answer_choices | join("\n- ")}}|||
+
+      {{answer_choices[choices["label"].index(answerKey)]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: qa_options
+    reference: ''
+  4fb13ac1-f770-45ea-b5d5-91ac50b0d609: !Template
+    answer_choices: '{{choices.text | join("|||")}}'
+    id: 4fb13ac1-f770-45ea-b5d5-91ac50b0d609
+    jinja: 'I am hesitating between 4 options to answer the following question, which
+      option should I choose?
+
+      Question: {{question}}
+
+      Possibilities:
+
+      - {{answer_choices | join("\n- ")}}|||
+
+      {{answer_choices[choices["label"].index(answerKey)]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: i_am_hesitating
+    reference: ''
+  8c689423-880d-402b-8c7d-a1a98c7589e8: !Template
+    answer_choices: '{{choices.text | join("|||")}}'
+    id: 8c689423-880d-402b-8c7d-a1a98c7589e8
+    jinja: 'I gave my students this multiple choice question: {{question}}
+
+
+      Only one answer is correct among these 4 choices:
+
+      - {{answer_choices | join("\n- ")}}
+
+
+      Could you tell me which one is correct?|||
+
+      {{answer_choices[choices["label"].index(answerKey)]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: multiple_choice
+    reference: ''
+  c988ee30-a523-457b-af21-87353349b543: !Template
+    answer_choices: null
+    id: c988ee30-a523-457b-af21-87353349b543
+    jinja: 'Pick and copy all the incorrect options for the following question:
+
+
+      {{question}}
+
+
+      Options:
+
+      - {{choices["text"] | join("\n- ")}}|||
+
+      {% for i in range(choices["label"]|length) %}
+
+      {% if i != choices["label"].index(answerKey) %}
+
+      - {{choices["text"][i]}}
+
+      {% endif %}
+
+      {% endfor %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - Other
+      original_task: false
+    name: pick_false_options
+    reference: ''
+  d90da519-0e2c-4f9b-a546-7cba82824eb2: !Template
+    answer_choices: '{{choices.label | join("|||")}}'
+    id: d90da519-0e2c-4f9b-a546-7cba82824eb2
+    jinja: "Here's a problem to solve: {{question}}\n\nAmong the 4 following options,\
+      \ which is the correct answer?\n{% for letter, t in zip(answer_choices, choices.text)\
+      \ %}\n- {{letter}}: {{t}}\n {% endfor %}|||{{answerKey}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: heres_a_problem
+    reference: ''

promptsource/templates/amazon_polarity/templates.yaml

@@ -0,0 +1,192 @@
+dataset: amazon_polarity
+templates:
+  1e90a24a-1182-43dd-9445-22f2e56e5761: !Template
+    answer_choices: Negative ||| Positive
+    id: 1e90a24a-1182-43dd-9445-22f2e56e5761
+    jinja: 'Title: {{title}}
+
+      Review: {{content}}
+
+      Is the review positive or negative? |||
+
+      {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Is_this_review
+    reference: ''
+  3a48f287-6a4b-4df0-ab2d-2eaf6cb8e53d: !Template
+    answer_choices: No ||| Yes
+    id: 3a48f287-6a4b-4df0-ab2d-2eaf6cb8e53d
+    jinja: 'Based on this review, would the user recommend this product?
+
+      ===
+
+      Review: {{content}}
+
+      Answer: |||
+
+      {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: User_recommend_this_product
+    reference: 'Reformulation equivalent to sent analysis: would the user recommend
+      this product?'
+  592caf8f-f8ff-426a-a61b-b7e95ed510b6: !Template
+    answer_choices: No ||| Yes
+    id: 592caf8f-f8ff-426a-a61b-b7e95ed510b6
+    jinja: 'Is this product review positive?
+
+      Title: {{title}}
+
+      Review: {{content}}
+
+      Answer: |||
+
+      {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Is_this_product_review_positive
+    reference: ''
+  745b9c05-10df-4a7e-81ad-1b88cefcb166: !Template
+    answer_choices: Yes ||| No
+    id: 745b9c05-10df-4a7e-81ad-1b88cefcb166
+    jinja: 'Title: {{title}}
+
+      Review: {{content}}
+
+      Is this product review negative?|||
+
+      {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Is_this_review_negative
+    reference: ''
+  8abb5377-5dd3-4402-92a5-0d81adb6a325: !Template
+    answer_choices: Negative ||| Positive
+    id: 8abb5377-5dd3-4402-92a5-0d81adb6a325
+    jinja: 'Title: {{title}}
+
+      Review: {{content}}
+
+      Does this product review convey a negative or positive sentiment?|||
+
+      {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: convey_negative_or_positive_sentiment
+    reference: ''
+  9df70cdf-f8ed-4e79-8e2f-b4668058d637: !Template
+    answer_choices: Negative ||| Positive
+    id: 9df70cdf-f8ed-4e79-8e2f-b4668058d637
+    jinja: 'Is there a negative or positive tone to this product review?
+
+      ===
+
+      Title: {{title}}
+
+      Review: {{content}}
+
+      Answer: |||
+
+      {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: negative_or_positive_tone
+    reference: ''
+  b13369e8-0500-4e93-90d4-8e6814bfb97b: !Template
+    answer_choices: dissatisfied ||| satisfied
+    id: b13369e8-0500-4e93-90d4-8e6814bfb97b
+    jinja: 'Here is a review left by a customer on a product. Would you say he was
+      {{answer_choices[1]}} or {{answer_choices[0]}}?
+
+      Title: {{title}}
+
+      Review: {{content}}
+
+      |||
+
+      {{answer_choices[label]}} '
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: user_satisfied
+    reference: ''
+  b13369e8-0500-4e93-90d4-8e6814bfb98b: !Template
+    answer_choices: decrease ||| increase
+    id: b13369e8-0500-4e93-90d4-8e6814bfb98b
+    jinja: 'You are considering whether to buy a product. You look at the reviews.
+      Would the following review {{answer_choices[0]}} or {{answer_choices[1]}} the
+      chances of you buying the product?
+
+      Review title: {{title}}
+
+      Product review: {{content}}
+
+      |||
+
+      {{answer_choices[label]}} '
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: would_you_buy
+    reference: ''
+  b13369e8-0500-4e93-90d4-8e6814bfb99b: !Template
+    answer_choices: unflattering ||| flattering
+    id: b13369e8-0500-4e93-90d4-8e6814bfb99b
+    jinja: 'Title: {{title}}
+
+      Product review: {{content}}
+
+      Would you say this review depicts the product in a {{answer_choices[1]}} or
+      {{answer_choices[0]}} light?
+
+      |||
+
+      {{answer_choices[label]}} '
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: flattering_or_not
+    reference: ''

promptsource/templates/amazon_reviews_multi/en/templates.yaml

@@ -0,0 +1,147 @@
+dataset: amazon_reviews_multi
+subset: en
+templates:
+  073dfd34-5aef-461a-81d9-bdb8e00f12c9: !Template
+    answer_choices: null
+    id: 073dfd34-5aef-461a-81d9-bdb8e00f12c9
+    jinja: 'Write a title for the review below:
+
+      ===
+
+      {{review_body}} |||
+
+      {{review_title}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: generate_title
+    reference: Review Title based on Review body
+  0f5b005b-c6bc-4fe0-bde4-0917cdba39e8: !Template
+    answer_choices: 1|||2|||3|||4|||5
+    id: 0f5b005b-c6bc-4fe0-bde4-0917cdba39e8
+    jinja: 'Rate the product by the number of stars based on the review title below:
+      (1 being the lowest and 5 the highest)
+
+      ===
+
+      {{review_title}} |||
+
+      {{answer_choices[stars-1]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - Other
+      original_task: false
+    name: prompt_title_to_star
+    reference: Rating based on review title
+  199ad6de-5bcc-421e-90e2-4b6edada6a01: !Template
+    answer_choices: 1|||2|||3|||4|||5
+    id: 199ad6de-5bcc-421e-90e2-4b6edada6a01
+    jinja: 'Rate the product by the number of stars based on the review body below:
+      (1 being the lowest and 5 the highest)
+
+      ===
+
+      {{review_body}} |||
+
+      {{answer_choices[stars-1]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - Other
+      original_task: true
+    name: prompt_review_to_star
+    reference: Rating based on review body
+  37806754-58f7-4383-961a-fe2c88109fcd: !Template
+    answer_choices: 1|||2|||3|||4|||5
+    id: 37806754-58f7-4383-961a-fe2c88109fcd
+    jinja: 'Rate the product by the number of stars based on the review below: (1
+      being the lowest and 5 the highest)
+
+      ===
+
+      {{review_title}}. {{review_body}} |||
+
+      {{answer_choices[stars-1]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - Other
+      original_task: true
+    name: prompt_body_title_to_star
+    reference: Rating based on review body,title
+  7ecaf718-c85d-47f4-83cb-f14c58f2911f: !Template
+    answer_choices: null
+    id: 7ecaf718-c85d-47f4-83cb-f14c58f2911f
+    jinja: 'Guess the product category from the following review:
+
+      ===
+
+      {{review_body}} |||
+
+      {{product_category}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - Other
+      original_task: false
+    name: prompt_review_to_category
+    reference: Product category based on review body
+  8e8973f6-431f-4e78-b83a-a86c04655882: !Template
+    answer_choices: 1|||2|||3|||4|||5
+    id: 8e8973f6-431f-4e78-b83a-a86c04655882
+    jinja: 'Rate the product by the number of stars based on the review below: (1
+      being the lowest and 5 the highest)
+
+      ===
+
+      {{review_title}}. {{review_body}} Product category: {{product_category}}|||
+
+      {{answer_choices[stars-1]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - Other
+      original_task: true
+    name: prompt_body_title_category_to_star
+    reference: Rating based on review body, title, category
+  c4717e75-4d3e-4b79-9737-167155f51513: !Template
+    answer_choices: null
+    id: c4717e75-4d3e-4b79-9737-167155f51513
+    jinja: 'Guess the product category from the review title below:
+
+      ===
+
+      {{review_title}} |||
+
+      {{product_category}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - Other
+      original_task: false
+    name: prompt_title_to_product_category
+    reference: Product category from review title

promptsource/templates/amazon_us_reviews/Wireless_v1_00/templates.yaml

@@ -0,0 +1,79 @@
+dataset: amazon_us_reviews
+subset: Wireless_v1_00
+templates:
+  5feaa0d7-e4e0-46cc-8517-e00bfa7fd00e: !Template
+    answer_choices: null
+    id: 5feaa0d7-e4e0-46cc-8517-e00bfa7fd00e
+    jinja: "Give a short sentence describing the following product review:\n{{review_body}}\
+      \ \n|||\n{{review_headline}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - ROUGE
+      - BLEU
+      original_task: false
+    name: Generate review headline based on review body
+    reference: Generate review headline based on review body
+  9588a967-d698-4a33-9b96-a5254df9d260: !Template
+    answer_choices: null
+    id: 9588a967-d698-4a33-9b96-a5254df9d260
+    jinja: Generate a {{star_rating}}-star review (1 being lowest and 5 being highest)
+      about this product {{product_title}}.        |||        {{review_body}}
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: Generate review based on rating and category
+    reference: Generate review based on rating and category
+  9a8b953d-2c68-4046-a7b7-8fd5f7469d10: !Template
+    answer_choices: '1 ||| 2 ||| 3 ||| 4 ||| 5 '
+    id: 9a8b953d-2c68-4046-a7b7-8fd5f7469d10
+    jinja: "Given the following review headline \n{{review_headline}}\npredict the\
+      \ the associated rating from the following choices\n- {{ answer_choices | join('\\\
+      n- ') }} \n(1 being lowest and 5 being highest)\n|||\n{{answer_choices[star_rating-1]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Given the review headline return a categorical rating
+    reference: 'Given the review headline, return a categorical rating. '
+  e40e4a53-ca5d-4fc8-a7c3-be9adfe0dbec: !Template
+    answer_choices: null
+    id: e40e4a53-ca5d-4fc8-a7c3-be9adfe0dbec
+    jinja: "Generate a {{star_rating}}-star review headline (1 being lowest and 5\
+      \ being highest) about this product: \n{{product_title}}        \n|||      \
+      \  \n{{review_headline}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: Generate review headline based on rating
+    reference: 'Generate review headline based on rating. '
+  e6a1bbde-715d-4dad-9178-e2bcfaf5c646: !Template
+    answer_choices: 1 ||| 2 ||| 3 ||| 4 ||| 5
+    id: e6a1bbde-715d-4dad-9178-e2bcfaf5c646
+    jinja: "Given the following review:\n{{review_body}}\npredict the associated rating\
+      \ from the following choices (1 being lowest and 5 being highest)\n- {{ answer_choices\
+      \ | join('\\n- ') }} \n|||\n{{answer_choices[star_rating-1]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Given the review body return a categorical rating
+    reference: 'Given the review body, return a categorical rating. '

promptsource/templates/ambig_qa/light/templates.yaml

@@ -0,0 +1,128 @@
+dataset: ambig_qa
+subset: light
+templates:
+  050b1534-b53f-4341-b42c-6e689ef8911b: !Template
+    answer_choices: null
+    id: 050b1534-b53f-4341-b42c-6e689ef8911b
+    jinja: "{# Assignement in if clause breaks test, we need to declare variables\
+      \ in global scope first: https://github.com/pallets/jinja/issues/1314 #}\n{%\
+      \ set selected_question = \"\" %}\n{% set selected_answer = \"\" %}\n{% set\
+      \ random_question_id = -1 %}\n{% if annotations.type[0] == \"multipleQAs\" %}\n\
+      \   {% set random_question_id = range(0, annotations.qaPairs[0].question | length)\
+      \ | choice%}\n   {% set selected_question = annotations.qaPairs[0].question[random_question_id]%}\n\
+      \   {% set selected_answer = annotations.qaPairs[0].answer[random_question_id]\
+      \ | choice%}\n{% else %}\n   {% set selected_question = question %}\n   {% set\
+      \ selected_answer = annotations.answer[0] | choice %}\n{% endif %}\n\nHere's\
+      \ a question-answer pair: {{question}} {{selected_answer}}.\nIs the question\
+      \ ambiguous? If so, generate a better question suitable for the answer. Otherwise,\
+      \ output the same question.\n|||\n{{selected_question}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - Edit Distance
+      original_task: false
+    name: is_question_ambiguous
+    reference: ''
+  09880e1a-0fcc-49dc-8462-b6603e15d691: !Template
+    answer_choices: null
+    id: 09880e1a-0fcc-49dc-8462-b6603e15d691
+    jinja: "What are the possible answers to the question \"{{question}}\"? Use semi-colons\
+      \ to separate your answers if you have multiple answers.\n\n|||\n\n{# Assignement\
+      \ in if clause breaks test, we need to declare variables in global scope first:\
+      \ https://github.com/pallets/jinja/issues/1314 #}\n{% set random_answer = \"\
+      \" %}\n{% set random_answer_form = \"\" %}\n{% if annotations.type[0] == \"\
+      singleAnswer\" %}\n    {% set random_answer_form = [] %}\n    {% for possible_answer\
+      \ in annotations.answer[0] %}\n    {{ random_answer_form.append(possible_answer\
+      \ ) or \"\"}}\n    {% endfor %}\n{% else %}\n    {% set random_answer_form =\
+      \ [] %}\n    {% for possible_answers in annotations.qaPairs[0].answer %}\n \
+      \   {% for possible_answer in possible_answers %}\n    {{ random_answer_form.append(possible_answer\
+      \ ) or \"\"}}\n    {% endfor %}\n    {% endfor %}\n{% endif %}\n\n{{random_answer_form\
+      \ | join(\"; \")}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: answer_prediction_all_answers_interrogative
+    reference: ''
+  45b20de4-a3c1-4e76-ad79-06d7c8c66009: !Template
+    answer_choices: null
+    id: 45b20de4-a3c1-4e76-ad79-06d7c8c66009
+    jinja: "Given the question \"{{question}}\", generate all the possible answers,\
+      \ separated by semi-colon.\n\n|||\n\n{# Assignement in if clause breaks test,\
+      \ we need to declare variables in global scope first: https://github.com/pallets/jinja/issues/1314\
+      \ #}\n{% set random_answer = \"\" %}\n{% set random_answer_form = \"\" %}\n\
+      {% if annotations.type[0] == \"singleAnswer\" %}\n    {% set random_answer_form\
+      \ = [] %}\n    {% for possible_answer in annotations.answer[0] %}\n    {{ random_answer_form.append(possible_answer\
+      \ ) or \"\"}}\n    {% endfor %}\n{% else %}\n    {% set random_answer_form =\
+      \ [] %}\n    {% for possible_answers in annotations.qaPairs[0].answer %}\n \
+      \   {% for possible_answer in possible_answers %}\n    {{ random_answer_form.append(possible_answer\
+      \ ) or \"\"}}\n    {% endfor %}\n    {% endfor %}\n{% endif %}\n\n{{random_answer_form\
+      \ | join(\"; \")}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: answer_prediction_all_answers_affirmative
+    reference: ''
+  72bf511b-44ce-4b9f-a2d0-5ed6334f0e07: !Template
+    answer_choices: Yes ||| No
+    id: 72bf511b-44ce-4b9f-a2d0-5ed6334f0e07
+    jinja: "{# Assignement in if clause breaks test, we need to declare variables\
+      \ in global scope first: https://github.com/pallets/jinja/issues/1314 #}\n{%\
+      \ set random_question_id = -1 %}\n{% set random_answer_id = -1 %}\n{% set selected_question\
+      \ = \"\" %}\n{% set selected_answer = \"\" %}\n{% if annotations.type[0] ==\
+      \ \"multipleQAs\" %}\n   {% set random_question_id = range(0, annotations.qaPairs[0].question\
+      \ | length) | choice%}\n   {% set random_answer_id = range(0, annotations.qaPairs[0].answer\
+      \ | length) | choice%}\n   {% set selected_question = annotations.qaPairs[0].question[random_question_id]\
+      \ %}\n   {% set selected_answer = annotations.qaPairs[0].answer[random_answer_id]\
+      \ | choice%}\n{% else %}\n   {% set random_question_id = 0 %}\n   {% set random_answer_id\
+      \ = 0 %}\n   {% set selected_question = question %}\n   {% set selected_answer\
+      \ = annotations.answer[0] | choice %}\n{% endif %}\n\nIs \"{{selected_answer}}\"\
+      \ an acceptable answer to \"{{selected_question}}\"? {{answer_choices[0]}} or\
+      \ {{answer_choices[1].lower()}}?\n\n|||\n\n{% if random_answer_id == random_question_id\
+      \ %} {{answer_choices[0]}} {% else %} {{answer_choices[1]}} {% endif %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: answer_prediction_yes_or_no
+    reference: Classify if the given answer if correct compared to the chosen question
+  bb089312-23cb-475d-93b5-952781bc6be4: !Template
+    answer_choices: null
+    id: bb089312-23cb-475d-93b5-952781bc6be4
+    jinja: "{# Assignement in if clause breaks test, we need to declare variables\
+      \ in global scope first: https://github.com/pallets/jinja/issues/1314 #}\n{%\
+      \ set selected_question = \"\" %}\n{% set selected_answer = \"\" %}\n{% set\
+      \ random_question_id = -1 %}\n{% if annotations.type[0] == \"multipleQAs\" %}\n\
+      \   {% set random_question_id = range(0, annotations.qaPairs[0].question | length)\
+      \ | choice%}\n   {% set selected_question = annotations.qaPairs[0].question[random_question_id]%}\n\
+      \   {% set selected_answer = annotations.qaPairs[0].answer[random_question_id]\
+      \ | choice%}\n{% else %}\n   {% set selected_question = question %}\n   {% set\
+      \ selected_answer = annotations.answer[0] | choice %}\n{% endif %}\n\nQuestion:\
+      \ {{question}}\nAnswer: {{selected_answer}}\n\nKnowing that the question can\
+      \ be ambiguous, can you perform question disambiguation by generating a question\
+      \ such that \"{{selected_answer}}\" is a more suitable answer? If you deem that\
+      \ the question is not ambiguous, generate the same question given above.\n|||\n\
+      {{selected_question}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: perform_question_disambiguation
+    reference: ''

promptsource/templates/anli/templates.yaml

@@ -0,0 +1,221 @@
+dataset: anli
+templates:
+  0cc3ae39-3997-4686-8c93-5d51457efa1f: !Template
+    answer_choices: Correct ||| Inconclusive ||| Incorrect
+    id: 0cc3ae39-3997-4686-8c93-5d51457efa1f
+    jinja: '{{premise}} Using only the above description and what you know about the
+      world, "{{hypothesis}}" is definitely correct, incorrect, or inconclusive? |||
+      {{ answer_choices[label] }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: MNLI crowdsource
+    reference: Adapted from Williams et al. 2018's instructions to crowdsourcing workers.
+  179eb863-3ece-4e6f-af0f-fcb46d997306: !Template
+    answer_choices: Yes ||| Maybe ||| No
+    id: 179eb863-3ece-4e6f-af0f-fcb46d997306
+    jinja: 'Given {{premise}} Should we assume that "{{hypothesis}}" is true? Yes,
+      no, or maybe? ||| {{ answer_choices[label] }} '
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: should assume
+    reference: Webson & Pavlick 2021
+  5459237b-97de-4340-bf7b-2939c3f7ca19: !Template
+    answer_choices: Yes ||| Maybe ||| No
+    id: 5459237b-97de-4340-bf7b-2939c3f7ca19
+    jinja: Given that {{premise}} Does it follow that {{hypothesis}} Yes, no, or maybe?
+      ||| {{ answer_choices[label] }}
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: does it follow that
+    reference: Sanh et al. 2021
+  620aa3fc-d5eb-46f5-a1ee-4c754527aa97: !Template
+    answer_choices: True ||| Neither ||| False
+    id: 620aa3fc-d5eb-46f5-a1ee-4c754527aa97
+    jinja: '{{premise}}
+
+      Question: {{hypothesis}} True, False, or Neither? ||| {{ answer_choices[label]
+      }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: GPT-3 style
+    reference: 'Same as reported in Figure G7 of the GPT-3 paper, except that there
+      is no task identifying tokens like "anli R1: ".'
+  9b613182-c6ab-4427-9221-3d68f6d62765: !Template
+    answer_choices: Yes ||| Maybe ||| No
+    id: 9b613182-c6ab-4427-9221-3d68f6d62765
+    jinja: '{{premise}} Based on the previous passage, is it true that "{{hypothesis}}"?
+      Yes, no, or maybe? ||| {{ answer_choices[label] }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: based on the previous passage
+    reference: "Adapted from the BoolQ prompts in Schick & Sch\xFCtze 2021."
+  a850110d-f1a3-49b4-949a-d3bfe9f81344: !Template
+    answer_choices: Yes ||| Maybe ||| No
+    id: a850110d-f1a3-49b4-949a-d3bfe9f81344
+    jinja: '{{premise}} Are we justified in saying that "{{hypothesis}}"? Yes, no,
+      or maybe? ||| {{ answer_choices[label] }} '
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: justified in saying
+    reference: Webson & Pavlick 2021
+  bab86d5a-4f9c-40db-b619-a7b7d5cae681: !Template
+    answer_choices: True ||| Inconclusive ||| False
+    id: bab86d5a-4f9c-40db-b619-a7b7d5cae681
+    jinja: 'Take the following as truth: {{premise}}
+
+      Then the following statement: "{{hypothesis}}" is {{"true"}}, {{"false"}}, or
+      {{"inconclusive"}}? ||| {{ answer_choices[label] }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: take the following as truth
+    reference: Sanh et al. 2021
+  bcd90047-3a2b-426b-b065-8a418f1317b8: !Template
+    answer_choices: Yes ||| Maybe ||| No
+    id: bcd90047-3a2b-426b-b065-8a418f1317b8
+    jinja: 'Given that {{premise}} Therefore, it must be true that "{{hypothesis}}"?
+      Yes, no, or maybe? ||| {{ answer_choices[label] }} '
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: must be true
+    reference: Sanh et al. 2021
+  c4ed37ae-d7d7-4197-a725-ef2152fa3b1f: !Template
+    answer_choices: Yes ||| Maybe ||| No
+    id: c4ed37ae-d7d7-4197-a725-ef2152fa3b1f
+    jinja: 'Suppose {{premise}} Can we infer that "{{hypothesis}}"? Yes, no, or maybe?
+      ||| {{ answer_choices[label] }} '
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: can we infer
+    reference: Webson & Pavlick 2021
+  ca24b93a-6265-462f-b140-e329c03d94fa: !Template
+    answer_choices: Guaranteed ||| Possible ||| Impossible
+    id: ca24b93a-6265-462f-b140-e329c03d94fa
+    jinja: "Assume it is true that {{premise}} \n\nTherefore, \"{{hypothesis}}\" is\
+      \ {{\"guaranteed\"}}, {{\"possible\"}}, or {{\"impossible\"}}? ||| {{ answer_choices[label]\
+      \ }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: guaranteed/possible/impossible
+    reference: Sanh et al. 2021
+  dbc68425-5c42-43ae-9748-70ce8c5a167e: !Template
+    answer_choices: Always ||| Sometimes ||| Never
+    id: dbc68425-5c42-43ae-9748-70ce8c5a167e
+    jinja: Suppose it's true that {{premise}} Then, is "{{hypothesis}}" {{"always"}},
+      {{"sometimes"}}, or {{"never"}} true? ||| {{ answer_choices[label] }}
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: always/sometimes/never
+    reference: Sanh et al. 2021
+  e5b7fdd7-fdff-4630-889b-3c7a052e5da0: !Template
+    answer_choices: Yes ||| Maybe ||| No
+    id: e5b7fdd7-fdff-4630-889b-3c7a052e5da0
+    jinja: "{{premise}} \n\nQuestion: Does this imply that \"{{hypothesis}}\"? Yes,\
+      \ no, or maybe? ||| {{answer_choices[label]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: does this imply
+    reference: Sanh et al. 2021
+  e6f32b9c-7e0b-474a-a0d2-e84d20c22aba: !Template
+    answer_choices: Always ||| Sometimes ||| Never
+    id: e6f32b9c-7e0b-474a-a0d2-e84d20c22aba
+    jinja: "{{premise}} \n\nKeeping in mind the above text, consider: {{hypothesis}}\
+      \ Is this {{\"always\"}}, {{\"sometimes\"}}, or {{\"never\"}} correct? ||| {{\
+      \ answer_choices[label] }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: consider always/sometimes/never
+    reference: Sanh et al. 2021
+  ec249357-e672-4e7d-b8b6-d97ed7d090c5: !Template
+    answer_choices: True ||| Inconclusive ||| False
+    id: ec249357-e672-4e7d-b8b6-d97ed7d090c5
+    jinja: '{{premise}} Based on that information, is the claim: "{{hypothesis}}"
+      {{"true"}}, {{"false"}}, or {{"inconclusive"}}? ||| {{ answer_choices[label]
+      }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: claim true/false/inconclusive
+    reference: Sanh et al. 2021
+  ffa0a6f0-7186-4ccb-bb35-8b1affb747a0: !Template
+    answer_choices: Yes ||| Maybe ||| No
+    id: ffa0a6f0-7186-4ccb-bb35-8b1affb747a0
+    jinja: 'Given {{premise}} Is it guaranteed true that "{{hypothesis}}"? Yes, no,
+      or maybe? ||| {{ answer_choices[label] }} '
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: guaranteed true
+    reference: Webson & Pavlick 2021

promptsource/templates/app_reviews/templates.yaml

@@ -0,0 +1,78 @@
+dataset: app_reviews
+templates:
+  2da8f134-58db-4f9d-b3b0-8c6b50693ab5: !Template
+    answer_choices: Not at all ||| No ||| Maybe ||| Yes ||| Definitely
+    id: 2da8f134-58db-4f9d-b3b0-8c6b50693ab5
+    jinja: 'Given this review: "{{review}}"
+
+      Would you recommend this app to a friend? {{answer_choices[0]}}, {{answer_choices[1]}},
+      {{answer_choices[2]}}, {{answer_choices[3]}}, or {{answer_choices[4]}}?
+
+      |||
+
+      {{answer_choices[star-1]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - Spearman Correlation
+      original_task: false
+    name: categorize_rating_using_review
+    reference: Given the review, return a categorical answer.
+  8086b434-a75e-45a4-87fb-4364601e2e05: !Template
+    answer_choices: null
+    id: 8086b434-a75e-45a4-87fb-4364601e2e05
+    jinja: 'Generate a {{star}}-star review (1 being lowest and 5 being highest) about
+      an app with package {{package_name}}.
+
+      |||
+
+      {{review}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: null
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - Spearman Correlation
+      original_task: false
+    name: generate_review
+    reference: Generate a review from the rating.
+  9746ce4b-ac58-4dfb-9783-d77c95cb62cf: !Template
+    answer_choices: "\u2605 ||| \u2605\u2605 ||| \u2605\u2605\u2605 ||| \u2605\u2605\
+      \u2605\u2605 ||| \u2605\u2605\u2605\u2605\u2605"
+    id: 9746ce4b-ac58-4dfb-9783-d77c95cb62cf
+    jinja: "What would be the \u2605-rating of this review (\u2605 being the lowest\
+      \ and \u2605\u2605\u2605\u2605\u2605 being the highest)? \"{{review}}\"\n|||\n\
+      {{answer_choices[star-1]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - Spearman Correlation
+      original_task: false
+    name: convert_to_star_rating
+    reference: Given the review, generate a star rating.
+  d34e1413-2699-4701-baa2-05d931d012ba: !Template
+    answer_choices: null
+    id: d34e1413-2699-4701-baa2-05d931d012ba
+    jinja: 'On a scale of 1-5 (with 1 being least favorable and 5 being most favorable),
+      how would you rate this review? "{{review}}"
+
+      |||
+
+      {{star}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - Spearman Correlation
+      original_task: false
+    name: convert_to_rating
+    reference: Convert review to rating

promptsource/templates/aqua_rat/raw/templates.yaml

@@ -0,0 +1,131 @@
+dataset: aqua_rat
+subset: raw
+templates:
+  13bd5099-33fa-4383-a441-33a7d2e1746f: !Template
+    answer_choices: A ||| B ||| C ||| D ||| E
+    id: 13bd5099-33fa-4383-a441-33a7d2e1746f
+    jinja: "Given the problem:\n{{question}}\n\nand the options:\n{% for i in range(options|length)\
+      \ %}\n{{options[i].replace(')', ') ')}}\n{% endfor %}\n\nThe correct answer\
+      \ is\n |||\n{{correct}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: select_the_best_option
+    reference: ''
+  58a6aa2b-ca26-473d-9bf8-385dd1a743cd: !Template
+    answer_choices: A ||| B ||| C ||| D ||| E
+    id: 58a6aa2b-ca26-473d-9bf8-385dd1a743cd
+    jinja: 'You will now be given a question and a set of options. Choose the correct
+      option and provide a rationale for the same.
+
+
+      Question:
+
+      {{question}}
+
+
+      Options:
+
+      {% for i in range(options|length) %}
+
+      - {{options[i].replace('')'', '') '')}}
+
+      {% endfor %}
+
+
+      |||
+
+      {{correct}}
+
+
+      {{rationale}}
+
+      '
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: generate_rational_and_correct_choice
+    reference: ''
+  5acfaa48-e1b6-44df-8e92-c58b94bff595: !Template
+    answer_choices: null
+    id: 5acfaa48-e1b6-44df-8e92-c58b94bff595
+    jinja: "Answer the given question by providing the correct rationale:\n\n{{question}}\n\
+      {% for i in range(options|length) %}\n   {{options[i].replace(')', ') ')}}\n\
+      {%endfor%}\n|||\n{{rationale}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: generate_rationale
+    reference: ''
+  815acaf5-2e59-4f81-8190-ae75dc237cf1: !Template
+    answer_choices: A ||| B ||| C ||| D ||| E
+    id: 815acaf5-2e59-4f81-8190-ae75dc237cf1
+    jinja: '{{question}}
+
+
+      The above question was asked in a Math test. Given the following options, can
+      you choose the correct one?
+
+
+      {% for i in range(options|length) %}
+
+      - {{options[i].replace('')'', '') '')}}
+
+      {% endfor %}
+
+      |||
+
+      {{correct}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: answer_quiz
+    reference: ''
+  c0403841-68b0-4c08-8c3b-a00a81272d05: !Template
+    answer_choices: A ||| B ||| C ||| D ||| E
+    id: c0403841-68b0-4c08-8c3b-a00a81272d05
+    jinja: "Solve the following question and choose the correct option.\n\n{{question}}\
+      \ \n{% for i in range(options|length) %}\n- {{options[i].replace(')', ') ')}}\n\
+      {%endfor%}\n||| \n{{correct}}\n\n"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: Answer questions from options
+    reference: ''
+  c9352c6c-074b-4beb-8489-c151adeeedcb: !Template
+    answer_choices: null
+    id: c9352c6c-074b-4beb-8489-c151adeeedcb
+    jinja: "Question: \n{{question}}\n\nOptions: \n{% for i in range(options|length)\
+      \ %}\n- {{options[i].replace(')', ') ')}}\n{% endfor %}\n\nThis is how I solved\
+      \ the above question:\n|||\n{{rationale}}\n"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: answer_question_with_rationale
+    reference: ''

promptsource/templates/art/templates.yaml

@@ -0,0 +1,133 @@
+dataset: art
+templates:
+  151d0e97-d7d2-47f2-86b4-6777587b16f2: !Template
+    answer_choices: '{{hypothesis_1 | trim(''.?!'') }} ||| {{hypothesis_2 | trim(''.?!'')
+      }}'
+    id: 151d0e97-d7d2-47f2-86b4-6777587b16f2
+    jinja: "We know that:\n\n{{ observation_1 }},\n\nand:\n\n{{ observation_2 }} \n\
+      \nWhich one is more likely?\n\nThe first option: \n\n{{ answer_choices[0] }},\
+      \ \n\nor the second option:\n\n{{ answer_choices[1] }}?\n|||\n{{ answer_choices[label-1]\
+      \ }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: choose_hypothesis_options
+    reference: ''
+  a090e019-1b98-4863-ab5d-ff9772f682d6: !Template
+    answer_choices: '{{hypothesis_1| trim(''.?!'') }} ||| {{hypothesis_2| trim(''.?!'')
+      }}'
+    id: a090e019-1b98-4863-ab5d-ff9772f682d6
+    jinja: 'You know the following:
+
+
+      {{ observation_1 }} {{ observation_2 }}
+
+
+      Which one is more believable?
+
+
+      - {{ answer_choices[0] }}
+
+      - {{ answer_choices[1] }}
+
+
+      |||
+
+
+      {{ answer_choices[label-1] }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: choose_hypothesis_believable
+    reference: ''
+  bf8a5b8a-70cb-4b27-82db-8ca4fbd2318d: !Template
+    answer_choices: '{{hypothesis_1| trim(''.?!'') }} ||| {{hypothesis_2| trim(''.?!'')
+      }}'
+    id: bf8a5b8a-70cb-4b27-82db-8ca4fbd2318d
+    jinja: '{{ observation_1 }} {{ observation_2 }}
+
+
+      Would you rather believe that:
+
+
+      {{ answer_choices[0] }},
+
+
+      or:
+
+
+      {{ answer_choices[1] }}?
+
+      |||
+
+      {{ answer_choices[label-1] }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: choose_hypothesis
+    reference: ''
+  d418b574-9d0a-4d29-a518-7d9a5f5a4a3d: !Template
+    answer_choices: '{{hypothesis_1| trim(''.?!'') }} ||| {{hypothesis_2| trim(''.?!'')
+      }}'
+    id: d418b574-9d0a-4d29-a518-7d9a5f5a4a3d
+    jinja: "Which of the following better fits the description?\n\nIs it that: \n\n\
+      {{ answer_choices[0] }},\n\nor rather: \n\n{{ answer_choices[1] }}?\n\nDescription:\
+      \ \n\n{{ observation_1 }} {{ observation_2 }}\n|||\n{{ answer_choices[label-1]\
+      \ }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: choose_hypothesis_desc
+    reference: ''
+  eb0baa43-3c79-4d1d-973a-37e0055bbfec: !Template
+    answer_choices: '{{hypothesis_1| trim(''.?!'') }} ||| {{hypothesis_2| trim(''.?!'')
+      }}'
+    id: eb0baa43-3c79-4d1d-973a-37e0055bbfec
+    jinja: 'Which version is more likely?
+
+
+      The first one:
+
+
+      {{ answer_choices[0] }},
+
+
+      or the second one:
+
+
+      {{ answer_choices[1] }}?
+
+
+      Assuming that:
+
+
+      {{ observation_1 }} {{ observation_2 }}
+
+      |||
+
+      {{ answer_choices[label-1] }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: choose_hypothesis_likely
+    reference: ''

promptsource/templates/asnq/templates.yaml

@@ -0,0 +1,211 @@
+dataset: asnq
+templates:
+  0e06d340-6d2c-44f7-b977-604925773f0b: !Template
+    answer_choices: No ||| Yes
+    id: 0e06d340-6d2c-44f7-b977-604925773f0b
+    jinja: "Question: {{question}} \nSentence: {{sentence}} \nAre the question and\
+      \ the sentence positive pairs where positive pairs means that the sentence answers\
+      \ the question? ||| {{answer_choices[label]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: positive_pairs
+    reference: ''
+  55f386ba-9a86-405e-a805-152e254a4205: !Template
+    answer_choices: null
+    id: 55f386ba-9a86-405e-a805-152e254a4205
+    jinja: "{% if label == 1 %}\n\nWhat is a question that someone might ask that\
+      \ the following sentence can answer?\n\n {{sentence}}\n\n|||\n\n{{question}}\n\
+      {% endif %}\n"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: question_from_sentence
+    reference: ''
+  5b6abb0a-1b4f-4338-aab6-430465669164: !Template
+    answer_choices: null
+    id: 5b6abb0a-1b4f-4338-aab6-430465669164
+    jinja: '{% if label == 1 %}
+
+
+      Write a question based on this sentence: {{sentence}}
+
+
+      |||
+
+
+      {{question}}
+
+      {% endif %}
+
+      '
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: write_question
+    reference: ''
+  684aea91-34c4-47de-a61f-7cc9a182b657: !Template
+    answer_choices: No ||| Yes
+    id: 684aea91-34c4-47de-a61f-7cc9a182b657
+    jinja: Can the answer "{{sentence}}" be inferred from the question "{{question}}"
+      ? ||| {{answer_choices[label]}}
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: answer_infer_question
+    reference: ''
+  719306b9-5dc8-46c7-b693-9b2edc2e09f2: !Template
+    answer_choices: No ||| Yes
+    id: 719306b9-5dc8-46c7-b693-9b2edc2e09f2
+    jinja: Does this sentence "{{sentence}}"  answer this question "{{question}}"
+      ? ||| {{answer_choices[label]}}
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Does_sentence_answer_question
+    reference: ''
+  859ec580-957b-42da-be1b-c3ccb8b52d24: !Template
+    answer_choices: null
+    id: 859ec580-957b-42da-be1b-c3ccb8b52d24
+    jinja: '{% if label == 1 %}
+
+
+      Generate a one-sentence answer to the following question: {{question}}?
+
+
+      |||
+
+
+      {{sentence}}
+
+      {% endif %}
+
+      '
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: answer question with a sentence
+    reference: ''
+  85da6666-9e50-4122-84c8-d00b90967475: !Template
+    answer_choices: null
+    id: 85da6666-9e50-4122-84c8-d00b90967475
+    jinja: '{% if label == 1 %}
+
+
+      Given the following question: {{question}}? Can you give me a full sentence
+      answer?
+
+
+      |||
+
+
+      {{sentence}}
+
+      {% endif %}
+
+      '
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: give me a full sentence answer
+    reference: ''
+  85fe8aaa-83c5-41ec-ada5-0e6d60bab1f9: !Template
+    answer_choices: null
+    id: 85fe8aaa-83c5-41ec-ada5-0e6d60bab1f9
+    jinja: '{% if label == 1 %}
+
+      Answer this question as a full sentence: {{question}}?
+
+      |||
+
+      {{sentence}}
+
+      {% endif %}
+
+      '
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: answer question as a sentence
+    reference: ''
+  95e39e1d-a830-4b6c-bd2a-10fe51552427: !Template
+    answer_choices: No ||| Yes
+    id: 95e39e1d-a830-4b6c-bd2a-10fe51552427
+    jinja: 'Can this question: "{{question}}" be answered as follow: "{{sentence}}"
+      \ \ please answer yes or no.  ||| {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: yes_vs_no
+    reference: ''
+  a36d6152-72c4-4278-8266-d27b28667f61: !Template
+    answer_choices: null
+    id: a36d6152-72c4-4278-8266-d27b28667f61
+    jinja: "{% if label == 1 %}\n\nHere is a sentence:\n\n {{sentence}}\n\nWrite a\
+      \ question to which this sentence is an answer.\n\n|||\n\n{{question}}\n{% endif\
+      \ %}\n"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: write_a_question
+    reference: ''
+  a7927e90-1a9b-49e2-a2f8-5ac9e6d286cb: !Template
+    answer_choices: No ||| Yes
+    id: a7927e90-1a9b-49e2-a2f8-5ac9e6d286cb
+    jinja: 'Does the following sentence "{{sentence}}" seem a right answer for the
+      following question : {{question}} ||| {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: right_answer
+    reference: ''

promptsource/templates/asset/ratings/templates.yaml

@@ -0,0 +1,119 @@
+dataset: asset
+subset: ratings
+templates:
+  09b2a13b-cba6-4473-8a46-3fa24be71ce2: !Template
+    answer_choices: No ||| Yes
+    id: 09b2a13b-cba6-4473-8a46-3fa24be71ce2
+    jinja: "{% set label = None %}\n{% set questions = None %}\n{% if rating > 50\
+      \ %}\n{% set label = 1 %}\n{% else %}\n{% set label = 0 %}\n{% endif %}\n{%\
+      \ set questions= [ \"Does the second sentence better convey the information?\"\
+      ,  \"Is the second sentence more fluent?\", \"Is the second sentence simpler?\"\
+      ] %}\n\nFirst sentence: {{original}}\n\nSecond sentence: {{simplification}}\n\
+      \n{{questions[aspect]}}. Please answer Yes or No. \n|||\n{{answer_choices[label]}}\n"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: rate-binary
+    reference: Taking questions from the original paper, we use rating to establish
+      a binary classification problem
+  47142040-4121-4144-98b9-61cb5cbb1313: !Template
+    answer_choices: null
+    id: 47142040-4121-4144-98b9-61cb5cbb1313
+    jinja: 'First sentence: {{original}}
+
+
+      Second sentence: {{simplification}}
+
+
+      I am scoring these simplification exercises. How easier to read is the second
+      sentence on a scale from 0 (harder to read) to 100 (easier to read)?
+
+
+      |||
+
+
+      {{rating}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: rate-regression-simplicity
+    reference: Prompt model to rate how simplified the sentence is in the general
+      sense, instead of an particular aspect. This is a regression task whose range
+      is from 0 to 100.
+  7dd6e8b6-eae0-40c5-aa5e-1cc24357d85d: !Template
+    answer_choices: null
+    id: 7dd6e8b6-eae0-40c5-aa5e-1cc24357d85d
+    jinja: '{% set label = None %}
+
+      {% set questions = None %}
+
+      {% if rating > 50 %}
+
+      {% set label = 1 %}
+
+      {% else %}
+
+      {% set label = 0 %}
+
+      {% endif %}
+
+      {% if label == 1 %}
+
+      {% set questions= [ "Rewrite the following sentence so that it conveys the information
+      better.",  "Rewrite the following sentence so that it is more fluent.", "Rewrite
+      the following sentence so that it is simpler."] %}
+
+      {% else %}
+
+      {% set questions= [ "Rewrite the following sentence so that it conveys the information
+      more poorly.",  "Rewrite the following sentence so that it is less fluent.",
+      "Rewrite the following sentence so that it is more complicated."] %}
+
+      {% endif %}
+
+      {{questions[aspect]}}
+
+
+      {{original}}
+
+      |||
+
+      {{simplification}}
+
+      '
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: generate-text-based-on-rating
+    reference: ''
+  d2bed959-29ab-4962-a106-dc91c00f3f03: !Template
+    answer_choices: null
+    id: d2bed959-29ab-4962-a106-dc91c00f3f03
+    jinja: "{% set statements= [ \"the second sentence expresses the underlying meaning\
+      \ the best.\",  \"the second sentence is more fluent.\", \"the second sentence\
+      \ is simpler.\"] %}\n\nFirst sentence: {{original}}\n\nSecond sentence: {{simplification}}\n\
+      \nRate the following statement from 0 (strongly disagree) to 100 (strongly agree):\
+      \ {{statements[aspect]}} \n\n|||\n{{rating}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Other
+      original_task: true
+    name: rate-regression
+    reference: Require the model to output the rating. This is a regression task whose
+      range is from 0 to 100.

promptsource/templates/asset/simplification/templates.yaml

@@ -0,0 +1,168 @@
+dataset: asset
+subset: simplification
+templates:
+  0f0e55f9-28b4-4844-b65d-b9544a0918eb: !Template
+    answer_choices: null
+    id: 0f0e55f9-28b4-4844-b65d-b9544a0918eb
+    jinja: "{% set real_simplifications = [] %}{% for text in simplifications %}{%\
+      \ if text|length < original|length %}{{real_simplifications.append(text) | default(\"\
+      \", True)}}{% endif %}{% endfor %}\n{% if real_simplifications %}\nText: {{original}}\n\
+      \nHow would I simplify this? \n\n|||\n\n{{real_simplifications | choice}}\n\
+      {% endif %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: true
+    name: verbose-to-simplification
+    reference: Rewrite text using one random simplification
+  3cbfbc1c-6876-4dd7-b7db-45fb3233a667: !Template
+    answer_choices: null
+    id: 3cbfbc1c-6876-4dd7-b7db-45fb3233a667
+    jinja: '{% set real_simplifications = [] %}{% for text in simplifications %}{%
+      if text|length < original|length %}{{real_simplifications.append(text) | default("",
+      True)}}{% endif %}{% endfor %}
+
+      {% if real_simplifications %}
+
+      Make the below sentence more verbose:
+
+
+      {{real_simplifications | choice}}
+
+
+      |||
+
+
+      {{original}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: simplification-to-verbose
+    reference: Make the simplified text more verbose
+  41d32553-433c-44fb-9eda-0fce51bf9e14: !Template
+    answer_choices: A ||| B
+    id: 41d32553-433c-44fb-9eda-0fce51bf9e14
+    jinja: '{% set rand_num = range(0,2) | choice  %}
+
+      {% set real_simplifications = [] %}{% for text in simplifications %}{% if text|length
+      < original|length %}{{real_simplifications.append(text) | default("", True)}}{%
+      endif %}{% endfor %}
+
+      {% if real_simplifications %}
+
+      One of the following two sentences is more verbose than the other. Which one
+      is it?
+
+      {% if rand_num %}
+
+      A: {{real_simplifications | choice}}
+
+
+      B: {{original}}
+
+      {% else %}
+
+      A: {{original}}
+
+
+      B: {{real_simplifications | choice}}
+
+      {% endif %}
+
+      |||
+
+      {{ answer_choices[rand_num] }}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: choose-verbose
+    reference: ''
+  5c2f56b9-5bd4-4455-9d68-0729bfdb9c84: !Template
+    answer_choices: A ||| B
+    id: 5c2f56b9-5bd4-4455-9d68-0729bfdb9c84
+    jinja: '{% set rand_num = range(0,2) | choice %}
+
+      {% set real_simplifications = [] %}{% for text in simplifications %}{% if text|length
+      < original|length %}{{real_simplifications.append(text) | default("", True)}}{%
+      endif %}{% endfor %}
+
+      {% if real_simplifications %}
+
+      One of the following two sentences is more simple than the other. Which one
+      is it?
+
+      {% if rand_num %}
+
+      A: {{real_simplifications | choice}}
+
+
+      B: {{original}}
+
+      {% else %}
+
+      A: {{original}}
+
+
+      B: {{real_simplifications | choice}}
+
+      {% endif %}
+
+      |||
+
+      {{ answer_choices[1-rand_num] }}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: choose-simplification
+    reference: ''
+  d528d74b-bbc2-4888-ae21-db0ab37304df: !Template
+    answer_choices: null
+    id: d528d74b-bbc2-4888-ae21-db0ab37304df
+    jinja: '{% set real_simplifications = [] %}{% for text in simplifications %}{%
+      if text|length < original|length %}{{real_simplifications.append(text) | default("",
+      True)}}{% endif %}{% endfor %}
+
+      {% if real_simplifications %}
+
+      I''d like to explain to my child "{{original}}". How would I do so?
+
+
+      |||
+
+
+      {{real_simplifications | choice}}
+
+      {% endif %}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: true
+    name: verbose-to-simplification-implicit
+    reference: Implicit simplification request

promptsource/templates/banking77/templates.yaml

@@ -0,0 +1,288 @@
+dataset: banking77
+templates:
+  0dba8abc-248a-44db-bb86-20492ffc17f6: !Template
+    answer_choices: activate my card|||age limit|||apple pay or google pay|||atm support|||automatic
+      top up|||balance not updated after bank transfer|||balance not updated after
+      cheque or cash deposit|||beneficiary not allowed|||cancel transfer|||card about
+      to expire|||card acceptance|||card arrival|||card delivery estimate|||card linking|||card
+      not working|||card payment fee charged|||card payment not recognised|||card
+      payment wrong exchange rate|||card swallowed|||cash withdrawal charge|||cash
+      withdrawal not recognised|||change pin|||compromised card|||contactless not
+      working|||country support|||declined card payment|||declined cash withdrawal|||declined
+      transfer|||direct debit payment not recognised|||disposable card limits|||edit
+      personal details|||exchange charge|||exchange rate|||exchange via app|||extra
+      charge on statement|||failed transfer|||fiat currency support|||get disposable
+      virtual card|||get physical card|||getting spare card|||getting virtual card|||lost
+      or stolen card|||lost or stolen phone|||order physical card|||passcode forgotten|||pending
+      card payment|||pending cash withdrawal|||pending top up|||pending transfer|||pin
+      blocked|||receiving money|||Refund not showing up|||request refund|||reverted
+      card payment?|||supported cards and currencies|||terminate account|||top up
+      by bank transfer charge|||top up by card charge|||top up by cash or cheque|||top
+      up failed|||top up limits|||top up reverted|||topping up by card|||transaction
+      charged twice|||transfer fee charged|||transfer into account|||transfer not
+      received by recipient|||transfer timing|||unable to verify identity|||verify
+      my identity|||verify source of funds|||verify top up|||virtual card not working|||visa
+      or mastercard|||why verify identity|||wrong amount of cash received|||wrong
+      exchange rate for cash withdrawal
+    id: 0dba8abc-248a-44db-bb86-20492ffc17f6
+    jinja: 'Which help page can be provided to provide information regarding this
+      query?
+
+      Query: {{text}} |||
+
+      {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: help_page_topic
+    reference: ''
+  2520f6d0-fcdf-44b6-abb3-a76e44948047: !Template
+    answer_choices: activate my card|||age limit|||apple pay or google pay|||atm support|||automatic
+      top up|||balance not updated after bank transfer|||balance not updated after
+      cheque or cash deposit|||beneficiary not allowed|||cancel transfer|||card about
+      to expire|||card acceptance|||card arrival|||card delivery estimate|||card linking|||card
+      not working|||card payment fee charged|||card payment not recognised|||card
+      payment wrong exchange rate|||card swallowed|||cash withdrawal charge|||cash
+      withdrawal not recognised|||change pin|||compromised card|||contactless not
+      working|||country support|||declined card payment|||declined cash withdrawal|||declined
+      transfer|||direct debit payment not recognised|||disposable card limits|||edit
+      personal details|||exchange charge|||exchange rate|||exchange via app|||extra
+      charge on statement|||failed transfer|||fiat currency support|||get disposable
+      virtual card|||get physical card|||getting spare card|||getting virtual card|||lost
+      or stolen card|||lost or stolen phone|||order physical card|||passcode forgotten|||pending
+      card payment|||pending cash withdrawal|||pending top up|||pending transfer|||pin
+      blocked|||receiving money|||Refund not showing up|||request refund|||reverted
+      card payment?|||supported cards and currencies|||terminate account|||top up
+      by bank transfer charge|||top up by card charge|||top up by cash or cheque|||top
+      up failed|||top up limits|||top up reverted|||topping up by card|||transaction
+      charged twice|||transfer fee charged|||transfer into account|||transfer not
+      received by recipient|||transfer timing|||unable to verify identity|||verify
+      my identity|||verify source of funds|||verify top up|||virtual card not working|||visa
+      or mastercard|||why verify identity|||wrong amount of cash received|||wrong
+      exchange rate for cash withdrawal
+    id: 2520f6d0-fcdf-44b6-abb3-a76e44948047
+    jinja: 'To which department in the bank can this query be directed?
+
+      Query: {{text}}
+
+      ||| {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: direct_to_which_department
+    reference: ''
+  9482bce0-f201-451b-9384-af588d707629: !Template
+    answer_choices: activate my card|||age limit|||apple pay or google pay|||atm support|||automatic
+      top up|||balance not updated after bank transfer|||balance not updated after
+      cheque or cash deposit|||beneficiary not allowed|||cancel transfer|||card about
+      to expire|||card acceptance|||card arrival|||card delivery estimate|||card linking|||card
+      not working|||card payment fee charged|||card payment not recognised|||card
+      payment wrong exchange rate|||card swallowed|||cash withdrawal charge|||cash
+      withdrawal not recognised|||change pin|||compromised card|||contactless not
+      working|||country support|||declined card payment|||declined cash withdrawal|||declined
+      transfer|||direct debit payment not recognised|||disposable card limits|||edit
+      personal details|||exchange charge|||exchange rate|||exchange via app|||extra
+      charge on statement|||failed transfer|||fiat currency support|||get disposable
+      virtual card|||get physical card|||getting spare card|||getting virtual card|||lost
+      or stolen card|||lost or stolen phone|||order physical card|||passcode forgotten|||pending
+      card payment|||pending cash withdrawal|||pending top up|||pending transfer|||pin
+      blocked|||receiving money|||Refund not showing up|||request refund|||reverted
+      card payment?|||supported cards and currencies|||terminate account|||top up
+      by bank transfer charge|||top up by card charge|||top up by cash or cheque|||top
+      up failed|||top up limits|||top up reverted|||topping up by card|||transaction
+      charged twice|||transfer fee charged|||transfer into account|||transfer not
+      received by recipient|||transfer timing|||unable to verify identity|||verify
+      my identity|||verify source of funds|||verify top up|||virtual card not working|||visa
+      or mastercard|||why verify identity|||wrong amount of cash received|||wrong
+      exchange rate for cash withdrawal
+    id: 9482bce0-f201-451b-9384-af588d707629
+    jinja: 'To which of the following departments in the bank can the given query
+      be directed?
+
+      Query: {{text}} Departments:
+
+      {% for intent in answer_choices %}
+
+      - {{intent}} {% endfor %}
+
+      |||
+
+      {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: choose the correct department
+    reference: ''
+  e629d77c-46f9-4e00-b23a-c522d07a9943: !Template
+    answer_choices: activate my card|||age limit|||apple pay or google pay|||atm support|||automatic
+      top up|||balance not updated after bank transfer|||balance not updated after
+      cheque or cash deposit|||beneficiary not allowed|||cancel transfer|||card about
+      to expire|||card acceptance|||card arrival|||card delivery estimate|||card linking|||card
+      not working|||card payment fee charged|||card payment not recognised|||card
+      payment wrong exchange rate|||card swallowed|||cash withdrawal charge|||cash
+      withdrawal not recognised|||change pin|||compromised card|||contactless not
+      working|||country support|||declined card payment|||declined cash withdrawal|||declined
+      transfer|||direct debit payment not recognised|||disposable card limits|||edit
+      personal details|||exchange charge|||exchange rate|||exchange via app|||extra
+      charge on statement|||failed transfer|||fiat currency support|||get disposable
+      virtual card|||get physical card|||getting spare card|||getting virtual card|||lost
+      or stolen card|||lost or stolen phone|||order physical card|||passcode forgotten|||pending
+      card payment|||pending cash withdrawal|||pending top up|||pending transfer|||pin
+      blocked|||receiving money|||Refund not showing up|||request refund|||reverted
+      card payment?|||supported cards and currencies|||terminate account|||top up
+      by bank transfer charge|||top up by card charge|||top up by cash or cheque|||top
+      up failed|||top up limits|||top up reverted|||topping up by card|||transaction
+      charged twice|||transfer fee charged|||transfer into account|||transfer not
+      received by recipient|||transfer timing|||unable to verify identity|||verify
+      my identity|||verify source of funds|||verify top up|||virtual card not working|||visa
+      or mastercard|||why verify identity|||wrong amount of cash received|||wrong
+      exchange rate for cash withdrawal
+    id: e629d77c-46f9-4e00-b23a-c522d07a9943
+    jinja: "Summarise the following query in the form of key banking terms: \n{{text}}\n\
+      |||\n{{answer_choices[label]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: rephrase_as_banking_term
+    reference: ''
+  edd67883-0386-4496-af7f-37a44c41293f: !Template
+    answer_choices: activate my card|||age limit|||apple pay or google pay|||atm support|||automatic
+      top up|||balance not updated after bank transfer|||balance not updated after
+      cheque or cash deposit|||beneficiary not allowed|||cancel transfer|||card about
+      to expire|||card acceptance|||card arrival|||card delivery estimate|||card linking|||card
+      not working|||card payment fee charged|||card payment not recognised|||card
+      payment wrong exchange rate|||card swallowed|||cash withdrawal charge|||cash
+      withdrawal not recognised|||change pin|||compromised card|||contactless not
+      working|||country support|||declined card payment|||declined cash withdrawal|||declined
+      transfer|||direct debit payment not recognised|||disposable card limits|||edit
+      personal details|||exchange charge|||exchange rate|||exchange via app|||extra
+      charge on statement|||failed transfer|||fiat currency support|||get disposable
+      virtual card|||get physical card|||getting spare card|||getting virtual card|||lost
+      or stolen card|||lost or stolen phone|||order physical card|||passcode forgotten|||pending
+      card payment|||pending cash withdrawal|||pending top up|||pending transfer|||pin
+      blocked|||receiving money|||Refund not showing up|||request refund|||reverted
+      card payment?|||supported cards and currencies|||terminate account|||top up
+      by bank transfer charge|||top up by card charge|||top up by cash or cheque|||top
+      up failed|||top up limits|||top up reverted|||topping up by card|||transaction
+      charged twice|||transfer fee charged|||transfer into account|||transfer not
+      received by recipient|||transfer timing|||unable to verify identity|||verify
+      my identity|||verify source of funds|||verify top up|||virtual card not working|||visa
+      or mastercard|||why verify identity|||wrong amount of cash received|||wrong
+      exchange rate for cash withdrawal
+    id: edd67883-0386-4496-af7f-37a44c41293f
+    jinja: 'Which of the following intents best represents this banking query?
+
+      Text: {{text}}
+
+      Intents:
+
+      {% for intent in answer_choices %}
+
+      - {{intent}} {% endfor %}
+
+      |||
+
+      {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: choose_the_correct_intent
+    reference: ''
+  eee2366a-8f0c-4ac3-b9cc-aa038e40f8cb: !Template
+    answer_choices: activate my card|||age limit|||apple pay or google pay|||atm support|||automatic
+      top up|||balance not updated after bank transfer|||balance not updated after
+      cheque or cash deposit|||beneficiary not allowed|||cancel transfer|||card about
+      to expire|||card acceptance|||card arrival|||card delivery estimate|||card linking|||card
+      not working|||card payment fee charged|||card payment not recognised|||card
+      payment wrong exchange rate|||card swallowed|||cash withdrawal charge|||cash
+      withdrawal not recognised|||change pin|||compromised card|||contactless not
+      working|||country support|||declined card payment|||declined cash withdrawal|||declined
+      transfer|||direct debit payment not recognised|||disposable card limits|||edit
+      personal details|||exchange charge|||exchange rate|||exchange via app|||extra
+      charge on statement|||failed transfer|||fiat currency support|||get disposable
+      virtual card|||get physical card|||getting spare card|||getting virtual card|||lost
+      or stolen card|||lost or stolen phone|||order physical card|||passcode forgotten|||pending
+      card payment|||pending cash withdrawal|||pending top up|||pending transfer|||pin
+      blocked|||receiving money|||Refund not showing up|||request refund|||reverted
+      card payment?|||supported cards and currencies|||terminate account|||top up
+      by bank transfer charge|||top up by card charge|||top up by cash or cheque|||top
+      up failed|||top up limits|||top up reverted|||topping up by card|||transaction
+      charged twice|||transfer fee charged|||transfer into account|||transfer not
+      received by recipient|||transfer timing|||unable to verify identity|||verify
+      my identity|||verify source of funds|||verify top up|||virtual card not working|||visa
+      or mastercard|||why verify identity|||wrong amount of cash received|||wrong
+      exchange rate for cash withdrawal
+    id: eee2366a-8f0c-4ac3-b9cc-aa038e40f8cb
+    jinja: 'What is the intent of this banking query?
+
+      {{text}} |||
+
+      {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: what_is_intent
+    reference: ''
+  f4e80455-1523-4b91-aacc-249d8c6f0f2a: !Template
+    answer_choices: activate my card|||age limit|||apple pay or google pay|||atm support|||automatic
+      top up|||balance not updated after bank transfer|||balance not updated after
+      cheque or cash deposit|||beneficiary not allowed|||cancel transfer|||card about
+      to expire|||card acceptance|||card arrival|||card delivery estimate|||card linking|||card
+      not working|||card payment fee charged|||card payment not recognised|||card
+      payment wrong exchange rate|||card swallowed|||cash withdrawal charge|||cash
+      withdrawal not recognised|||change pin|||compromised card|||contactless not
+      working|||country support|||declined card payment|||declined cash withdrawal|||declined
+      transfer|||direct debit payment not recognised|||disposable card limits|||edit
+      personal details|||exchange charge|||exchange rate|||exchange via app|||extra
+      charge on statement|||failed transfer|||fiat currency support|||get disposable
+      virtual card|||get physical card|||getting spare card|||getting virtual card|||lost
+      or stolen card|||lost or stolen phone|||order physical card|||passcode forgotten|||pending
+      card payment|||pending cash withdrawal|||pending top up|||pending transfer|||pin
+      blocked|||receiving money|||Refund not showing up|||request refund|||reverted
+      card payment?|||supported cards and currencies|||terminate account|||top up
+      by bank transfer charge|||top up by card charge|||top up by cash or cheque|||top
+      up failed|||top up limits|||top up reverted|||topping up by card|||transaction
+      charged twice|||transfer fee charged|||transfer into account|||transfer not
+      received by recipient|||transfer timing|||unable to verify identity|||verify
+      my identity|||verify source of funds|||verify top up|||virtual card not working|||visa
+      or mastercard|||why verify identity|||wrong amount of cash received|||wrong
+      exchange rate for cash withdrawal
+    id: f4e80455-1523-4b91-aacc-249d8c6f0f2a
+    jinja: 'Generate the subject for an email containing the following text:
+
+      {{text}} |||
+
+      {{answer_choices[label]}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: generate_subject_for_text
+    reference: ''

promptsource/templates/billsum/templates.yaml

@@ -0,0 +1,153 @@
+dataset: billsum
+templates:
+  0938c6e4-dbaf-43d8-8d8f-4bc62489ae74: !Template
+    answer_choices: null
+    id: 0938c6e4-dbaf-43d8-8d8f-4bc62489ae74
+    jinja: 'Given the title: "{{title}}" and the summary of a bill: {{summary}}.
+
+      Write this bill based on the title and summary.
+
+      |||
+
+      {{text}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: 'Write a bill: (title, summary->text)'
+    reference: ''
+  3c790ac3-0557-47a9-9b71-1cb435f15629: !Template
+    answer_choices: null
+    id: 3c790ac3-0557-47a9-9b71-1cb435f15629
+    jinja: "Given a state bill: {{text}}. \nPlease write the title of this bill in\
+      \ one sentence.\n|||\n{{title}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: 'Summarize this bill in one sentence: (text-> title)'
+    reference: ''
+  438192e5-d67a-4098-9d82-a9fe892f6be2: !Template
+    answer_choices: null
+    id: 438192e5-d67a-4098-9d82-a9fe892f6be2
+    jinja: 'Given a summary of a bill: {{summary}}.
+
+      Write this bill.
+
+      |||
+
+      {{text}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: 'Write a bill: (summary-> text)'
+    reference: ''
+  4891a8e7-258c-41e2-80d3-0c1a054acb07: !Template
+    answer_choices: null
+    id: 4891a8e7-258c-41e2-80d3-0c1a054acb07
+    jinja: 'Given a title: "{{title}}" of a bill.
+
+      Write this bill based on this title.
+
+      |||
+
+      {{text}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: 'Write a bill: (title-> text)'
+    reference: ''
+  550fa161-af4e-4430-9844-ce7dad587733: !Template
+    answer_choices: null
+    id: 550fa161-af4e-4430-9844-ce7dad587733
+    jinja: 'Given this bill: {{text}}.
+
+      Write a summary of this bill.
+
+      |||
+
+      {{summary}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: true
+    name: 'Summarize this bill: (text-> summary)'
+    reference: ''
+  5d2404b9-63ff-406e-977d-eda6afb5c689: !Template
+    answer_choices: null
+    id: 5d2404b9-63ff-406e-977d-eda6afb5c689
+    jinja: 'Given a summary: {{summary}}, we want to generate a title from this summary.
+
+      |||
+
+      {{title}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: Generate title from summary
+    reference: ''
+  6a439a80-4924-49e9-b5ae-f661683b399f: !Template
+    answer_choices: null
+    id: 6a439a80-4924-49e9-b5ae-f661683b399f
+    jinja: 'Summarize this US bill: {{text}}
+
+      |||
+
+      {{summary}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: true
+    name: 'Summarize: (text -> summary )'
+    reference: ''
+  ea9f0376-6cec-450c-b258-89f479cb9f6d: !Template
+    answer_choices: null
+    id: ea9f0376-6cec-450c-b258-89f479cb9f6d
+    jinja: 'Given a summary of a bill: {{summary}}.
+
+      Please write the title of this summary.
+
+      |||
+
+      {{title}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: 'Summarize: (summary -> title)'
+    reference: ''

promptsource/templates/bing_coronavirus_query_set/templates.yaml

@@ -0,0 +1,77 @@
+dataset: bing_coronavirus_query_set
+templates:
+  43332782-9e92-4bb2-94bf-28759f3fe181: !Template
+    answer_choices: null
+    id: 43332782-9e92-4bb2-94bf-28759f3fe181
+    jinja: "This search query talks about the coronavirus and was published on {{Date}}.\
+      \ In what country was it issued? \n{{Query}}\n|||\n{{Country}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: 'what_country '
+    reference: ''
+  68f9c063-1907-4866-ab1b-756cc57e5695: !Template
+    answer_choices: implicit ||| explicit
+    id: 68f9c063-1907-4866-ab1b-756cc57e5695
+    jinja: "The user is searching for coronavirus results on Bing.com. Is the intent\
+      \ implicit or explicit? \n{{Query}}\n|||\n{% if IsImplicitIntent == \"True\"\
+      \ %}\n{{answer_choices[0] }}\n{% else %}\n{{answer_choices[1] }}\n{% endif %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: 'is_implicit_or_explicit '
+    reference: ''
+  992d541f-9e0c-466d-b4c4-92e9e236f863: !Template
+    answer_choices: implicit ||| explicit
+    id: 992d541f-9e0c-466d-b4c4-92e9e236f863
+    jinja: "This search query about coronavirus was issued in {{Country}} on {{Date}}.\
+      \ Is the intent implicit or explicit? \n{{Query}}\n|||\n{% if IsImplicitIntent\
+      \ == \"True\" %}\n{{answer_choices[0] }}\n{% else %}\n{{answer_choices[1] }}\n\
+      {% endif %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: 'is_explicit_country_date '
+    reference: ''
+  df53652c-36dc-45fe-a015-d0781e32cd33: !Template
+    answer_choices: Yes ||| No
+    id: df53652c-36dc-45fe-a015-d0781e32cd33
+    jinja: "Does this search engine query have an indirect relation to Covid-19? \n\
+      {{Query}}\n|||\n{% if IsImplicitIntent == \"True\" %}\n{{answer_choices[0] }}\n\
+      {% else %}\n{{answer_choices[1] }}\n{% endif %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: is_implicit_query
+    reference: ''
+  df7bc2ee-686c-4826-ad84-3a056a2da4d4: !Template
+    answer_choices: No ||| Yes
+    id: df7bc2ee-686c-4826-ad84-3a056a2da4d4
+    jinja: "Does this search query on Bing.com talk about the coronavirus explicitly?\
+      \ \n{{Query}}\n|||\n{% if IsImplicitIntent == \"True\" %}\n{{answer_choices[0]\
+      \ }}\n{% else %}\n{{answer_choices[1] }}\n{% endif %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: is_explicit_query
+    reference: ''

promptsource/templates/biosses/templates.yaml

@@ -0,0 +1,186 @@
+dataset: biosses
+templates:
+  084e20ea-689d-4813-9db0-04735016aa0b: !Template
+    answer_choices: null
+    id: 084e20ea-689d-4813-9db0-04735016aa0b
+    jinja: 'How similar are the following two sentences? {{sentence1}} {{sentence2}}
+
+
+      Give the answer on a scale from 0 - 4, where 0 is "not similar at all" and 4
+      is "means the same thing". |||
+
+
+      {{(((5*score)|round)/5)}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Pearson Correlation
+      original_task: true
+    name: similarity with question first
+    reference: stsb template from FLAN
+  2aa62df9-5905-4f50-baff-c11986670122: !Template
+    answer_choices: null
+    id: 2aa62df9-5905-4f50-baff-c11986670122
+    jinja: On a scale from 0 to 4, where 0 is "not similar" and 4 is "very similar",
+      how similar is the sentence "{{sentence1}}" to the sentence {{sentence2}}"?
+      ||| {{(((5*score)|round)/5)}}
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Pearson Correlation
+      original_task: true
+    name: compare one sentence to another
+    reference: stsb template from FLAN
+  2ec48b7b-c2c8-4253-9c0f-b57814ba0027: !Template
+    answer_choices: null
+    id: 2ec48b7b-c2c8-4253-9c0f-b57814ba0027
+    jinja: "Sentence 1: {{sentence1}} \nSentence 2: {{sentence2}}\n\nFrom 0 to 4 (0\
+      \ = \"no meaning overlap\" and 4 = \"means the same thing\"), how similar are\
+      \ the two sentences? |||\n\n{{(((5*score)|round)/5)}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Pearson Correlation
+      original_task: true
+    name: similarity with sentences first
+    reference: stsb template from FLAN
+  400dcb4c-8654-44aa-acec-4dbe108e34a6: !Template
+    answer_choices: null
+    id: 400dcb4c-8654-44aa-acec-4dbe108e34a6
+    jinja: '{{sentence1}} {{sentence2}}
+
+
+      On a scale from 0 to 4, where 0 is "no meaning overlap" and 4 is "means the
+      same thing", how closely does the first sentence resemble the second one? |||
+
+
+      {{(((5*score)|round)/5)}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Pearson Correlation
+      original_task: true
+    name: resemblance
+    reference: stsb template from FLAN
+  5a6bc1a2-8d73-4c57-baa1-cc4b5c4dfacc: !Template
+    answer_choices: null
+    id: 5a6bc1a2-8d73-4c57-baa1-cc4b5c4dfacc
+    jinja: 'Do the following sentences say the same thing? {{sentence1}} {{sentence2}}
+
+
+      Return your answer on a scale from 0 to 4, where 0 is "not similar" and 5 is
+      "very similar". |||
+
+
+      {{(((5*score)|round)/5)}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Pearson Correlation
+      original_task: true
+    name: same thing scoring
+    reference: stsb template from FLAN
+  5c53ce9b-45f6-41ab-9da7-9c24f0f6f56d: !Template
+    answer_choices: no ||| yes
+    id: 5c53ce9b-45f6-41ab-9da7-9c24f0f6f56d
+    jinja: "(1) {{sentence1}} \n(2) {{sentence2}}\n\nDo these two sentences convey\
+      \ the same information? |||\n\n{{answer_choices[0 if score < 2.5 else 1]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: same info binary
+    reference: paws_wiki from FLAN
+  c1b48040-b083-4501-a7ef-a21b65800eb6: !Template
+    answer_choices: null
+    id: c1b48040-b083-4501-a7ef-a21b65800eb6
+    jinja: '{{sentence1}} {{sentence2}}
+
+
+      Rate the textual similarity of these two sentences on a scale of {{"0.0"}} and
+      {{"4.0"}}, where 0 is "no relation" and 4 is "equivalent". |||
+
+
+      {{(((5*score)|round)/5)}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Pearson Correlation
+      original_task: true
+    name: rate with sentences first
+    reference: stsb template from FLAN
+  d52895b8-71bb-4b87-a20f-e8eae53ede92: !Template
+    answer_choices: no ||| yes
+    id: d52895b8-71bb-4b87-a20f-e8eae53ede92
+    jinja: Please check if these have the same meaning. Answer "yes" if they do, otherwise
+      "no". {{sentence1}} {{sentence2}} ||| {{answer_choices[0 if score < 2.5 else
+      1]}}
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: same meaning binary
+    reference: paws_wiki from FLAN
+  e22d8c63-3184-40df-84c2-6800960496a7: !Template
+    answer_choices: no ||| yes
+    id: e22d8c63-3184-40df-84c2-6800960496a7
+    jinja: Do "{{sentence1}}" and "{{sentence2}}" seem similar to you ? ||| {{answer_choices[0
+      if score < 2.5 else 1]}}
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: similarity binary
+    reference: stsb_multi_mt
+  f2b20779-4ac9-41d9-9660-b9c5223fe9c1: !Template
+    answer_choices: null
+    id: f2b20779-4ac9-41d9-9660-b9c5223fe9c1
+    jinja: 'Rate the similarity of these two sentences: ({{"0.0"}} being the lowest
+      and {{"4.0"}} the highest) "{{sentence1}}" and "{{sentence2}}" |||
+
+
+      {{(((5*score)|round)/5)}}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Pearson Correlation
+      original_task: true
+    name: rate with question first
+    reference: stsb_multi_mt
+  fc22748c-72c0-4727-bc4e-53aae4449bef: !Template
+    answer_choices: no ||| yes
+    id: fc22748c-72c0-4727-bc4e-53aae4449bef
+    jinja: Do you think "{{sentence1}}" and "{{sentence2}}" express the same thing?
+      ||| {{answer_choices[0 if score < 2.5 else 1]}}
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: same thing binary
+    reference: stsb_multi_mt

promptsource/templates/blbooksgenre/title_genre_classifiction/templates.yaml

@@ -0,0 +1,63 @@
+dataset: blbooksgenre
+subset: title_genre_classifiction
+templates:
+  0c3e83f4-7f4d-4eca-8f80-6b6bdd8eeedd: !Template
+    answer_choices: Fiction ||| Non-fiction
+    id: 0c3e83f4-7f4d-4eca-8f80-6b6bdd8eeedd
+    jinja: "Given the title: {{title}}, which of the following genres is the book?\n\
+      (a) {{ answer_choices[0] }}\n(b) {{ answer_choices[1] }}\n|||\n {{  answer_choices[label]\
+      \ }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - AUC
+      original_task: true
+    name: multi-choice
+    reference: ''
+  5564acb9-c911-4d71-ba4d-add444aaf1e3: !Template
+    answer_choices: True ||| False
+    id: 5564acb9-c911-4d71-ba4d-add444aaf1e3
+    jinja: "{{title}} is the title of a fictional book, True or False?\nAnswer: \n\
+      |||\n{{ answer_choices[label] }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - AUC
+      original_task: true
+    name: premise_context_first
+    reference: ''
+  afc18daa-999d-495f-908a-d99477f6f5ac: !Template
+    answer_choices: True ||| False
+    id: afc18daa-999d-495f-908a-d99477f6f5ac
+    jinja: "The following is the title of a fictional book, True or False?\n{{title}}\n\
+      Answer: \n|||\n{{ answer_choices[label] }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - AUC
+      original_task: true
+    name: premise
+    reference: ''
+  cf4b6ce0-ff87-4c7a-9b9e-ec7c4cf741d8: !Template
+    answer_choices: Fiction ||| Non-fiction
+    id: cf4b6ce0-ff87-4c7a-9b9e-ec7c4cf741d8
+    jinja: The genre of the book "{{title}}" is |||  {{  answer_choices[label] }}
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      - AUC
+      original_task: true
+    name: classify
+    reference: ''

promptsource/templates/blended_skill_talk/templates.yaml

@@ -0,0 +1,57 @@
+dataset: blended_skill_talk
+templates:
+  54f785e9-453a-4ffe-8181-28095e3f2b80: !Template
+    answer_choices: null
+    id: 54f785e9-453a-4ffe-8181-28095e3f2b80
+    jinja: "Given the below conversation between two people, what would the listener\
+      \ say?\n\nA: {{previous_utterance[0]}}\n\nB: {{previous_utterance[1]}}\n{% for\
+      \ message_f, message_g in zip(free_messages[:-1], guided_messages[:-1]) %}\n\
+      A: {{message_f}}\n\nB: {{message_g}}\n{% endfor %} \nA: {{free_messages[-1]}}\n\
+      \nB: \n|||\n{{guided_messages[-1]}}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: guess-last-utterance
+    reference: ''
+  58f4e068-26fa-4843-a1d6-54bde324e780: !Template
+    answer_choices: Yes ||| No
+    id: 58f4e068-26fa-4843-a1d6-54bde324e780
+    jinja: "Two people are having a conversation. Are the utterances in the correct\
+      \ order? \n\nYour answer should be either \"Yes\" or \"No\".\n{% if range(0,\
+      \ 2) | choice %}\nA: {{previous_utterance[0]}}\n\nB: {{previous_utterance[1]}}\n\
+      {% for message_f, message_g in zip(free_messages, guided_messages) %}\nA: {{message_f}}\n\
+      \nB: {{message_g}}\n{% endfor %} \n\n|||\nYes.\n{% else %}\nA: {{previous_utterance[1]}}\n\
+      \nB: {{previous_utterance[0]}}\n{% for message_f, message_g in zip(guided_messages,\
+      \ free_messages) %}\nA: {{message_f}}\n\nB: {{message_g}}\n{% endfor %} \n\n\
+      |||\nNo.\n{% endif %}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: false
+    name: guess-correct-order
+    reference: ''
+  8792b63e-7217-40fe-8130-7392baca3519: !Template
+    answer_choices: null
+    id: 8792b63e-7217-40fe-8130-7392baca3519
+    jinja: "Two people are talking to each other.  What do you think Person A said\
+      \ in the beginning?\n\nPerson B: {{previous_utterance[1]}}\n{% for message_f,\
+      \ message_g in zip(free_messages, guided_messages) %}\nPerson A: {{message_f}}\n\
+      \nPerson B: {{message_g}}\n{% endfor %} \n|||\n{{previous_utterance[0]}}\n"
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: guess-first-utterance
+    reference: ''

promptsource/templates/cbt/CN/templates.yaml

@@ -0,0 +1,147 @@
+dataset: cbt
+subset: CN
+templates:
+  08820238-5bb3-4c7c-98bb-ec3d81e432e7: !Template
+    answer_choices: null
+    id: 08820238-5bb3-4c7c-98bb-ec3d81e432e7
+    jinja: '{{sentences | join('' '')}}
+
+
+      Write the next sentence of this story.
+
+      |||
+
+      {{ question.replace("XXXXX", answer) }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: Generate Next Sentence
+    reference: Generate the next sentence given the story.
+  1f8cad96-4c0f-435a-9a6f-653fcf158dd0: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: 1f8cad96-4c0f-435a-9a6f-653fcf158dd0
+    jinja: '{{sentences | join ('' '')}} {{question}}
+
+
+      Replace {{"XXXXX"}} with the correct option from:
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - Replace
+    reference: Fill the blank given the options.
+  556ee207-18c9-4c6c-860a-8ea09b93505c: !Template
+    answer_choices: '{{options|join(''|||'')}}'
+    id: 556ee207-18c9-4c6c-860a-8ea09b93505c
+    jinja: "{{sentences | join (' ')}}\n\nIn this following sentence: \n\"{{question}}\"\
+      , \naptly substitute the {{\"XXXXX\"}} with one of the following options:\n\
+      {{answer_choices|join(\", \")}}\n|||\n{{ answer }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - In the following
+    reference: Fill in the blanks given options.
+  63bfa7b6-b566-4693-848c-e05cd7a12a03: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: 63bfa7b6-b566-4693-848c-e05cd7a12a03
+    jinja: '{{ sentences | join('' '') }} {{question}}
+
+
+      Fill in the {{"XXXXX"}}.
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank without Options
+    reference: Fill in the blank without options.
+  a2e38459-90d9-4292-9d96-491ad7d4e3db: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: a2e38459-90d9-4292-9d96-491ad7d4e3db
+    jinja: '{{sentences | join ('' '')}} {{question}}
+
+
+      Which of the following options replaces {{"XXXXX"}} the best in the above story?
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blanks with Options - above story
+    reference: Given the sentences, fill the blanks using the options.
+  a6fa37d5-899c-4ad0-b888-fab04cc8e423: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: a6fa37d5-899c-4ad0-b888-fab04cc8e423
+    jinja: '{{sentences | join ('' '')}}
+
+
+      Which of the following options replaces {{"XXXXX"}} in "{{question}}"?
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - Which
+    reference: Fill Blank given options.
+  a8b67815-1927-4ef3-8d04-8d3f95525ef5: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: a8b67815-1927-4ef3-8d04-8d3f95525ef5
+    jinja: '{{sentences | join ('' '')}} {{question}}
+
+
+      Fill in the {{"XXXXX"}} from the following choices:
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - Fill in
+    reference: Fill in the blank given options

promptsource/templates/cbt/NE/templates.yaml

@@ -0,0 +1,147 @@
+dataset: cbt
+subset: NE
+templates:
+  08820238-5bb3-4c7c-98bb-ec3d81e432bb: !Template
+    answer_choices: null
+    id: 08820238-5bb3-4c7c-98bb-ec3d81e432bb
+    jinja: '{{sentences | join('' '')}}
+
+
+      Write the next sentence of this story.
+
+      |||
+
+      {{ question.replace("XXXXX", answer) }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: Generate Next Sentence
+    reference: Generate the next sentence given the story.
+  1f8cad96-4c0f-435a-9a6f-653fcf158dbb: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: 1f8cad96-4c0f-435a-9a6f-653fcf158dbb
+    jinja: '{{sentences | join ('' '')}} {{question}}
+
+
+      Replace {{"XXXXX"}} with the correct option from:
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - Replace
+    reference: Fill the blank given the options.
+  556ee207-18c9-4c6c-860a-8ea09b9350bb: !Template
+    answer_choices: '{{options|join(''|||'')}}'
+    id: 556ee207-18c9-4c6c-860a-8ea09b9350bb
+    jinja: "{{sentences | join (' ')}}\n\nIn this following sentence: \n\"{{question}}\"\
+      , \naptly substitute the {{\"XXXXX\"}} with one of the following options:\n\
+      {{answer_choices|join(\", \")}}\n|||\n{{ answer }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - In the following
+    reference: Fill in the blanks given options.
+  63bfa7b6-b566-4693-848c-e05cd7a12abb: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: 63bfa7b6-b566-4693-848c-e05cd7a12abb
+    jinja: '{{ sentences | join('' '') }} {{question}}
+
+
+      Fill in the {{"XXXXX"}}.
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank without Options
+    reference: Fill in the blank without options.
+  a2e38459-90d9-4292-9d96-491ad7d4e3bb: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: a2e38459-90d9-4292-9d96-491ad7d4e3bb
+    jinja: '{{sentences | join ('' '')}} {{question}}
+
+
+      Which of the following options replaces {{"XXXXX"}} the best in the above story?
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blanks with Options - above story
+    reference: Given the sentences, fill the blanks using the options.
+  a6fa37d5-899c-4ad0-b888-fab04cc8e4bb: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: a6fa37d5-899c-4ad0-b888-fab04cc8e4bb
+    jinja: '{{sentences | join ('' '')}}
+
+
+      Which of the following options replaces {{"XXXXX"}} in "{{question}}"?
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - Which
+    reference: Fill Blank given options.
+  a8b67815-1927-4ef3-8d04-8d3f95525ebb: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: a8b67815-1927-4ef3-8d04-8d3f95525ebb
+    jinja: '{{sentences | join ('' '')}} {{question}}
+
+
+      Fill in the {{"XXXXX"}} from the following choices:
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - Fill in
+    reference: Fill in the blank given options

promptsource/templates/cbt/P/templates.yaml

@@ -0,0 +1,147 @@
+dataset: cbt
+subset: P
+templates:
+  08820238-5bb3-4c7c-98bb-ec3d81e432ea: !Template
+    answer_choices: null
+    id: 08820238-5bb3-4c7c-98bb-ec3d81e432ea
+    jinja: '{{sentences | join('' '')}}
+
+
+      Write the next sentence of this story.
+
+      |||
+
+      {{ question.replace("XXXXX", answer) }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: Generate Next Sentence
+    reference: Generate the next sentence given the story.
+  1f8cad96-4c0f-435a-9a6f-653fcf158dda: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: 1f8cad96-4c0f-435a-9a6f-653fcf158dda
+    jinja: '{{sentences | join ('' '')}} {{question}}
+
+
+      Replace {{"XXXXX"}} with the correct option from:
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - Replace
+    reference: Fill the blank given the options.
+  556ee207-18c9-4c6c-860a-8ea09b93505a: !Template
+    answer_choices: '{{options|join(''|||'')}}'
+    id: 556ee207-18c9-4c6c-860a-8ea09b93505a
+    jinja: "{{sentences | join (' ')}}\n\nIn this following sentence: \n\"{{question}}\"\
+      , \naptly substitute the {{\"XXXXX\"}} with one of the following options:\n\
+      {{answer_choices|join(\", \")}}\n|||\n{{ answer }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - In the following
+    reference: Fill in the blanks given options.
+  63bfa7b6-b566-4693-848c-e05cd7a12a0a: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: 63bfa7b6-b566-4693-848c-e05cd7a12a0a
+    jinja: '{{ sentences | join('' '') }} {{question}}
+
+
+      Fill in the {{"XXXXX"}}.
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank without Options
+    reference: Fill in the blank without options.
+  a2e38459-90d9-4292-9d96-491ad7d4e3da: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: a2e38459-90d9-4292-9d96-491ad7d4e3da
+    jinja: '{{sentences | join ('' '')}} {{question}}
+
+
+      Which of the following options replaces {{"XXXXX"}} the best in the above story?
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blanks with Options - above story
+    reference: Given the sentences, fill the blanks using the options.
+  a6fa37d5-899c-4ad0-b888-fab04cc8e42a: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: a6fa37d5-899c-4ad0-b888-fab04cc8e42a
+    jinja: '{{sentences | join ('' '')}}
+
+
+      Which of the following options replaces {{"XXXXX"}} in "{{question}}"?
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - Which
+    reference: Fill Blank given options.
+  a8b67815-1927-4ef3-8d04-8d3f95525efa: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: a8b67815-1927-4ef3-8d04-8d3f95525efa
+    jinja: '{{sentences | join ('' '')}} {{question}}
+
+
+      Fill in the {{"XXXXX"}} from the following choices:
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - Fill in
+    reference: Fill in the blank given options

promptsource/templates/cbt/V/templates.yaml

@@ -0,0 +1,147 @@
+dataset: cbt
+subset: V
+templates:
+  08820238-5bb3-4c7c-98bb-ec3d81e432cc: !Template
+    answer_choices: null
+    id: 08820238-5bb3-4c7c-98bb-ec3d81e432cc
+    jinja: '{{sentences | join('' '')}}
+
+
+      Write the next sentence of this story.
+
+      |||
+
+      {{ question.replace("XXXXX", answer) }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - BLEU
+      - ROUGE
+      original_task: false
+    name: Generate Next Sentence
+    reference: Generate the next sentence given the story.
+  1f8cad96-4c0f-435a-9a6f-653fcf158dcc: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: 1f8cad96-4c0f-435a-9a6f-653fcf158dcc
+    jinja: '{{sentences | join ('' '')}} {{question}}
+
+
+      Replace {{"XXXXX"}} with the correct option from:
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - Replace
+    reference: Fill the blank given the options.
+  556ee207-18c9-4c6c-860a-8ea09b9350cc: !Template
+    answer_choices: '{{options|join(''|||'')}}'
+    id: 556ee207-18c9-4c6c-860a-8ea09b9350cc
+    jinja: "{{sentences | join (' ')}}\n\nIn this following sentence: \n\"{{question}}\"\
+      , \naptly substitute the {{\"XXXXX\"}} with one of the following options:\n\
+      {{answer_choices|join(\", \")}}\n|||\n{{ answer }}"
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - In the following
+    reference: Fill in the blanks given options.
+  63bfa7b6-b566-4693-848c-e05cd7a12acc: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: 63bfa7b6-b566-4693-848c-e05cd7a12acc
+    jinja: '{{ sentences | join('' '') }} {{question}}
+
+
+      Fill in the {{"XXXXX"}}.
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: false
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank without Options
+    reference: Fill in the blank without options.
+  a2e38459-90d9-4292-9d96-491ad7d4e3cc: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: a2e38459-90d9-4292-9d96-491ad7d4e3cc
+    jinja: '{{sentences | join ('' '')}} {{question}}
+
+
+      Which of the following options replaces {{"XXXXX"}} the best in the above story?
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blanks with Options - above story
+    reference: Given the sentences, fill the blanks using the options.
+  a6fa37d5-899c-4ad0-b888-fab04cc8e4cc: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: a6fa37d5-899c-4ad0-b888-fab04cc8e4cc
+    jinja: '{{sentences | join ('' '')}}
+
+
+      Which of the following options replaces {{"XXXXX"}} in "{{question}}"?
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - Which
+    reference: Fill Blank given options.
+  a8b67815-1927-4ef3-8d04-8d3f95525ecc: !Template
+    answer_choices: '{{options|join("|||")}}'
+    id: a8b67815-1927-4ef3-8d04-8d3f95525ecc
+    jinja: '{{sentences | join ('' '')}} {{question}}
+
+
+      Fill in the {{"XXXXX"}} from the following choices:
+
+      {{answer_choices|join(", ")}}
+
+      |||
+
+      {{ answer }}'
+    metadata: !TemplateMetadata
+      choices_in_prompt: true
+      languages:
+      - en
+      metrics:
+      - Accuracy
+      original_task: true
+    name: Fill Blank with Options - Fill in
+    reference: Fill in the blank given options