Update content.py
Browse files- content.py +9 -9
content.py
CHANGED
|
@@ -1,16 +1,16 @@
|
|
| 1 |
-
TITLE = """<h1 align="center" id="space-title">
|
| 2 |
|
| 3 |
INTRODUCTION_TEXT = """
|
| 4 |
-
|
| 5 |
|
| 6 |
## Data
|
| 7 |
-
In
|
| 8 |
For constraints, from the perspective of real world applications, we design three types of them: Environment Constraint, Commonsense Constraint, and Hard Constraint.
|
| 9 |
-
|
| 10 |
|
| 11 |
-
|
| 12 |
|
| 13 |
-
## Submission Guidelines for
|
| 14 |
Participants are invited to submit results for both validation and testing phases. The submissions will be evaluated based on several metrics: delivery rate, commonsense constraint pass rate (micro/macro), hard constraint pass rate (micro/macro), and the final pass rate.
|
| 15 |
|
| 16 |
### Format of Submission:
|
|
@@ -39,14 +39,14 @@ Format: Use "Name, City" to specify the chosen restaurant and its location. If a
|
|
| 39 |
Description: Information about attractions visited.
|
| 40 |
Format: List attractions as "Name, City". If visiting multiple attractions, separate them with a semicolon ";". If no attraction is planned, use "-".
|
| 41 |
|
| 42 |
-
Please refer to [this](https://huggingface.co/datasets/osunlp/
|
| 43 |
|
| 44 |
-
Submission made by our team are labelled "
|
| 45 |
|
| 46 |
"""
|
| 47 |
|
| 48 |
CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results"
|
| 49 |
-
CITATION_BUTTON_TEXT = r"""@misc{
|
| 50 |
title={},
|
| 51 |
author={},
|
| 52 |
year={2024},
|
|
|
|
| 1 |
+
TITLE = """<h1 align="center" id="space-title">TravelPlanner Leaderboard</h1>"""
|
| 2 |
|
| 3 |
INTRODUCTION_TEXT = """
|
| 4 |
+
TravelPlanner is a benchmark crafted for evaluating language agents in tool-use and complex planning within multiple constraints. (See our [paper](https://arxiv.org/abs/2311.12983) for more details.)
|
| 5 |
|
| 6 |
## Data
|
| 7 |
+
In TravelPlanner, for a given query, language agents are expected to formulate a comprehensive plan that includes transportation, daily meals, attractions, and accommodation for each day.
|
| 8 |
For constraints, from the perspective of real world applications, we design three types of them: Environment Constraint, Commonsense Constraint, and Hard Constraint.
|
| 9 |
+
TravelPlanner comprises 1,225 queries in total. The number of days and hard constraints are designed to test agents' abilities across both the breadth and depth of complex planning.
|
| 10 |
|
| 11 |
+
TravelPlanner data can be found in [this dataset](https://huggingface.co/datasets/osunlp/TravelPlanner).
|
| 12 |
|
| 13 |
+
## Submission Guidelines for TravelPlanner
|
| 14 |
Participants are invited to submit results for both validation and testing phases. The submissions will be evaluated based on several metrics: delivery rate, commonsense constraint pass rate (micro/macro), hard constraint pass rate (micro/macro), and the final pass rate.
|
| 15 |
|
| 16 |
### Format of Submission:
|
|
|
|
| 39 |
Description: Information about attractions visited.
|
| 40 |
Format: List attractions as "Name, City". If visiting multiple attractions, separate them with a semicolon ";". If no attraction is planned, use "-".
|
| 41 |
|
| 42 |
+
Please refer to [this](https://huggingface.co/datasets/osunlp/TravelPlanner/resolve/main/example_submission.jsonl?download=true) for example submission file.
|
| 43 |
|
| 44 |
+
Submission made by our team are labelled "TravelPlanner Team". Each submission will be automatically evaluated and scored based on the predefined metrics. The scores and rankings will be updated and displayed on the leaderboard.
|
| 45 |
|
| 46 |
"""
|
| 47 |
|
| 48 |
CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results"
|
| 49 |
+
CITATION_BUTTON_TEXT = r"""@misc{Xie2024TravelPlanner,
|
| 50 |
title={},
|
| 51 |
author={},
|
| 52 |
year={2024},
|