Paper: OKR-Agent: An Object and Key Results Driven Agent System with Hierarchical Self-Collaboration and Self-Evaluation (ICLR-2024)

Reviewer comment (Novelty):
One of my major concerns is the limited technical novelty. Apart from the cute prompting, I did not see many method improvements from both technical and theoretical perspectives.

Author response:
We want to emphasize that the proposed method is not a simple prompt engineering. In fact,  the complexity of the tasks we undertake exceeds the inference capabilities of existing Large Language Models (LLM). To avoid bottlenecks in LLM's performance, we have design the OKR based method, which allows the LLM to focus both on the global objectives and the details of the task. This approach enables the LLM to autonomously decompose tasks and evaluate the results of the decomposition, ensuring the accurate execution of target outcomes within the LLM's performance range.

Reviewer comment (Evaluation):
More analysis is needed on the agent objectives, key results, and evaluation criteria to provide insights into what is being learned.

Author response:
We offer additional ablation studies.

Reviewer comment (Reproducibility):
Agent coordination details are lacking - how are dependencies managed?

Author response:
The tasks for agents are automatically assigned by the Large Language Model (LLM) without any manual explicit assignment. The LLM autonomously decomposes the tasks based on their type and content, and allocates them to the appropriate agents.

Reviewer comment (Theory):
Safety mechanisms to prevent unbounded generation of objectives/agents need to be addressed.

Author response:
The generation of objectives and agents is bounded by the evaluation, where large number of objectives could lead to a low score in the evaluation. We also set the overall hierarchical levels to further prevent the possible unbounded generation.

Reviewer comment (Experiments):
How does the performance scale with increasing levels of hierarchy? Is there a sweet spot?

Author response:
Yes. Through ablation experiments, we compared the generation effects of 1-3 levels of hierarchy. Subjectively, three layers are currently the most suitable, with more than three levels showing no significant improvement in effectiveness.

Reviewer comment (Experiments):
Can the approach work for very open-ended creative tasks without clear sub-objectives?

Author response:
There are two methods that can have an impact: 1) By using the Few-Shot approach to inform the LLM of the required use cases. 2) By fine-tuning the LLM.

Reviewer comment (Experiments):
Has the approach been tested on goal-driven tasks like process planning vs just content generation?

Author response:
In our paper, "Multi-day Travel Planning" is an example of goal-driven tasks. Due to the length constraints of the paper, we did not include another task: "Wedding Planning." If permitted, we will supplement this in the accompanying materials.

Reviewer comment (Experiments):
What techniques are used to elicit useful evaluation criteria from the agents?

Author response:
We enable the LLM to customize assessment methods based on task objectives and assign them to different agents, thereby selecting the best inference results. The effectiveness would be further enhanced with the introduction of human interactive evaluation.

Reviewer comment (Experiments):
How sensitive is performance to the quality of the initial user prompt?

Author response:
We have adopted a method that allows the LLM to independently understand and decompose user input tasks (for example, providing the LLM with some cases to guide it in imitating methods that can successfully decompose objectives). Therefore, our algorithm is currently not sensitive to user input; the LLM will automatically analyze user objectives. Of course, more detailed descriptions lead to more effective results.

Reviewer comment (Reproducibility):
How difficult is it to construct such initial prompts from ambiguous and general tasks?

Author response:
We have adopted a method that allows the LLM to independently understand and decompose user input tasks (for example, providing the LLM with some cases to guide it in imitating methods that can successfully decompose objectives). Therefore, our algorithm is currently not sensitive to user input; the LLM will automatically analyze user objectives. Of course, more detailed descriptions lead to more effective results.

Reviewer comment (Experiments):
Limited Comparative Benchmarking: The paper's comparison with existing methods seems constrained. A more extensive benchmark, including a variety of state-of-the-art systems, would better position the OKR-Agent's performance relative to the current landscape.

Author response:
While we acknowledge the limited objective evaluations in our current work focusing on creative content generation, the assessment primarily relies on the subjective judgment of industry experts, which is a valid approach for this context. To align more closely with expert expectations, we have employed various engineering methods to enhance inference outcomes. These include using Few-Shot learning to improve the initial user input to OKR information conversion success rate, employing multi-round generation and assessment voting for optimal results, and integrating global and local information to enhance current inference outcomes. Additionally, our ablation studies on OKR generation outcomes further validate the effectiveness of our methods.

Reviewer comment (Evaluation):
Heavy Reliance on Subjective Evaluation: The paper predominantly uses subjective evaluations for performance assessment. Objective metrics and statistical validation could provide a more balanced and reproducible evaluation framework.

Author response:
While we acknowledge the limited objective evaluations in our current work focusing on creative content generation, the assessment primarily relies on the subjective judgment of industry experts, which is a valid approach for this context. To align more closely with expert expectations, we have employed various engineering methods to enhance inference outcomes. These include using Few-Shot learning to improve the initial user input to OKR information conversion success rate, employing multi-round generation and assessment voting for optimal results, and integrating global and local information to enhance current inference outcomes. Additionally, our ablation studies on OKR generation outcomes further validate the effectiveness of our methods.

Reviewer comment (Evaluation):
Weakness in Evaluation Depth and Diversity: The paper lacks a diverse set of evaluation metrics. It primarily relies on subjective assessments, which, while valuable, may not capture the full performance spectrum of the OKR-Agent. Objective metrics like precision, recall, F1-score, or BLEU score for language tasks could provide a more comprehensive picture. The absence of these metrics is a significant weakness, as they are critical for substantiating the model's effectiveness across different scenarios and for different use cases.

Author response:
While we acknowledge the limited objective evaluations in our current work focusing on creative content generation, the assessment primarily relies on the subjective judgment of industry experts, which is a valid approach for this context. To align more closely with expert expectations, we have employed various engineering methods to enhance inference outcomes. These include using Few-Shot learning to improve the initial user input to OKR information conversion success rate, employing multi-round generation and assessment voting for optimal results, and integrating global and local information to enhance current inference outcomes. Additionally, our ablation studies on OKR generation outcomes further validate the effectiveness of our methods.

Reviewer comment (Experiments):
Insufficient Ablation Studies: The paper does not present in-depth ablation studies. Ablation studies are critical for understanding which components of the proposed method contribute most significantly to its performance. By systematically removing or altering parts of the OKR-Agent, researchers can gain insights into the importance of each feature. This omission means that the reader has limited understanding of why the OKR-Agent works and which aspects are essential for its success.

Author response:
We have conducted several ablation studies,  we will include them in main text or  the supplementary materials based on the page limits.

Reviewer comment (Evaluation):
No Discussion on Model Robustness and Generalizability: The evaluation does not robustly test the generalizability of the OKR-Agent. It's unclear how well the agent would perform on tasks that were not part of the initial evaluation set, which is a significant weakness for claims of versatility and adaptability.

Author response:
Due to the length constraints, we experimented two different types of experiments, including creative writing and goal planning in the submission. We will add more varied types of experiments if space allows.

Reviewer comment (Evaluation):
If I understand correctly, for each key result the LLM has to be invoked at least multiple times during solution proposal and review. So overall this system seems costly and slow from an inference perspective. It will be great if the authors can compare with other methods from this perspective.

Author response:
Yes, there is always a trade-off between the cost of inference and the overall quality of solutions. Our inference process is divided into two stages: OKR generation and Agent working stage. In the OKR generation phase, well-designed prompts enable the LLM to effectively improve the OKR inference results, generating reasonable keywords. This process relies on the model's inference capability rather than content generation capacity, so it does not require excessive Token consumption. In the Agent working stage, the overall cost is directly proportional to the number of tokens. Here, we validated several different LLM base models (including gpt-3.5-turbo, gpt-4, flan-alpaca-xl-3B, etc.) under the constraints of OKRs, all capable of producing satisfactory content generation results. This observation also helps in reducing generation costs. Based on GPT-API's pricing, the cost for a single creative writing generation is about \$0.5 to \$0.6, and even lower, less than \$0.1, if using local models.

Reviewer comment (Experiments):
The proposed system will decompose a task into many hierarchies. Who is deciding the number of hierarchies needed? From algorithm 1 it looks like a hyper parameter.

Author response:
We set the number of hierarchies empirically. Based on our experiments, 3 layers are sufficient for the tasks we tested and there is no significant improvement with more than 3 layers. We will provide this ablation in the revision.

Reviewer comment (Theory):
Since the task will be broken into many objectives, are there causal or temporal links between different objectives? For example, how objective 2 can be solved should depend on objective 1's solution.

Author response:
There is no explicit link between objects although there could be shared components of task for different objects. Both the tasks and agents are automatically assigned by the Large Language Model (LLM) without any manual assignment.

Reviewer comment (Theory):
If there are such links, how can the different agents coordinate to solve the objectives in the correct order? From algorithm 2 it seems that each agent is just independently updating a portion of the final answer.

Author response:
There is no explicit link between objects although there could be shared components of task for different objects. Both the tasks and agents are automatically assigned by the Large Language Model (LLM) without any manual assignment.

Reviewer comment (Evaluation):
What if one agent cannot give a satisfying answer? What will happen to the whole task?

Author response:
Once OKRs are generated, they will still fixed. We use an assessment and adjustment process during the OKRs creation stage to ensure the OKRs are correct and necessary. After an agent completes the execution of OKRs, the outcomes produced will be "evaluated" and "revised" again according to the OKR objectives.

Reviewer comment (Experiments):
Are there any "replannings" where certain objectives should be removed/reconsidered?

Author response:
Once OKRs are generated, they will still fixed. We use an assessment and adjustment process during the OKRs creation stage to ensure the OKRs are correct and necessary. After an agent completes the execution of OKRs, the outcomes produced will be "evaluated" and "revised" again according to the OKR objectives.