initial: self-discover framework

README.md

@@ -0,0 +1,107 @@
+## SELF-DISCOVER FRAMEWORK
+
+## Paper Overview [link](https://arxiv.org/pdf/2402.03620.pdf)
+This project implements the paper titled "Self-Discover: Large Language Models Self-Compose Reasoning Structures," submitted on February 6, 2024, by Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Heng-Tze Cheng, Quoc V. Le, Ed H. Chi, Denny Zhou, Swaroop Mishra, and Huaixiu Steven Zheng. The paper introduces SELF-DISCOVER, a framework designed to enhance the performance of Large Language Models (LLMs) on complex reasoning tasks by enabling them to self-discover task-intrinsic reasoning structures.
+
+
+## Functionality (as given in paper)
+- **Self-Discovery Process:** The system engages in a self-discovery process where it selects atomic reasoning modules and composes them into an explicit reasoning structure.
+- **Performance Improvement:** SELF-DISCOVER significantly enhances the performance of LLMs on challenging reasoning benchmarks such as BigBench-Hard, grounded agent reasoning, and MATH, achieving up to a 32% improvement compared to conventional prompting methods like Chain of Thought (CoT).
+- **Efficiency:** Despite its effectiveness, SELF-DISCOVER requires 10-40 times fewer inference computations compared to inference-intensive methods like CoT-Self-Consistency.
+- **Universality:** The self-discovered reasoning structures are found to be universally applicable across different LLM model families, indicating commonalities with human reasoning patterns.
+
+
+
+##  Project Overview
+
+This project consists of a Python script (`self_discover.py`) along with associated modules and prompts. It allows users to input a specific task, and then it guides them through the process of selecting, adapting, and implementing reasoning modules to tackle that task effectively.
+
+## Implementation Details
+
+- **Model Used:** The implementation  Large Language Model (LLM) "gemini-pro" or "gpt-3.5-turbo"
+- **Tasks:** The system is capable of handling various task to generate reasoning structure
+- **Actions:** The system performs three main actions: SELECT, ADAPT, and IMPLEMENT.
+  - **SELECT:** This action involves selecting several reasoning modules crucial for solving the given task.
+  - **ADAPT:** The selected reasoning modules are rephrased and specified to better suit the task at hand.
+  - **IMPLEMENT:** The reasoning modules are operationalized into a step-by-step reasoning plan in JSON format, providing a structured approach for solving the task.
+
+
+## Prerequisites
+
+- Python 3.10
+- Libraries: google-generativeai, openai, dotenv
+- Input the task you want to generate a reasoning structure in task_example.py
+
+## Installation
+
+1. Clone this repository:
+
+   ```bash
+   git clone https://github.com/kailashsp/SelfDiscover.git
+   ```
+
+2. Install the required libraries:
+
+   ```bash
+   pip install -r requirements.txt
+   ```
+3. create a .env file
+
+4. Open the `.env` file in a text editor.
+
+5. Add the following line to the `.env` file:
+
+   ```
+   GOOGLE_API_KEY=your_google_api_key_here
+   ```
+
+   Replace `your_google_api_key_here` with your actual Google API key obtained from [google makersuite](https://makersuite.google.com/app/apikey).
+   Your can also use OPENAI_API_KEY as well
+
+
+## Usage
+
+1. Initialize a `SelfDiscover` object with a task:
+    
+   ```python
+   from self_disover import SelfDiscover
+   from task_example import task1
+
+   result = SelfDiscover(task=task1)
+   ```
+
+2. Call the `SelfDiscover` object:
+
+   ```python
+   result()
+   ```
+
+3. Access the selected and adapted modules also implemented reasoning structure:
+
+   ```python
+   print(f"SELECTED_MODULES : {result.selected_modules}")
+   print(f"ADAPTED_MODULES : {result.adapted_modules}")
+   print(f"REASONING_STRUCTURE : {result.reasoning_structure}")
+   ```
+
+## Customization
+
+- Modify the `reasoning_modules` variable in `prompts.py` to add, remove, or modify reasoning modules.
+- Adjust the prompts in `prompts.py` to customize the user interaction flow.
+
+## How to use the reasoning JSON structure
+
+- As mentioned in the paper 
+```markdown
+For Stage 2, where we use the self-discovered structure to solve the task instances, we start with the prompt: “Follow the
+step-by-step reasoning plan in JSON to correctly solve the task. Fill in the values following the keys by reasoning specifically 
+about the task given. Do not simply rephrase the keys.”, followed by the reasoning structure, and finally the task instance.
+```
+You can now give the task with the reasoning structure with the above prompt
+
+## Contributing
+
+Contributions are welcome! Feel free to open issues or pull requests with any improvements or suggestions.
+
+---
+

llm.py

@@ -0,0 +1,55 @@
+import os
+import google.generativeai as genai
+from openai import OpenAI
+from dotenv import load_dotenv
+
+load_dotenv()
+
+generation_config = {
+  "temperature": 0,
+  "top_k": 1,
+  "max_output_tokens": 4000,
+}
+
+
+class LLM:
+    def __init__(self, model_name) -> None:
+        self.model_name = model_name
+        self.model = self.create_model(model_name)
+
+    def create_model(self, model_name):
+        match model_name:
+            case "gemini-pro-vision":
+                genai.configure(api_key=os.environ.get("GOOGLE_API_KEY"))
+                return genai.GenerativeModel(model_name)
+            case "gemini-pro":
+                genai.configure(api_key=os.environ.get("GOOGLE_API_KEY"))
+                return genai.GenerativeModel(
+                    model_name,generation_config=generation_config)
+            case "OpenAI":
+                return OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
+            case _:
+                print("Not Implemented")
+          
+    def __call__(self, prompt, image=None):
+        if self.model_name == 'gemini-pro-vision':
+            response = self.model.generate_content(
+                [image, prompt]
+            )
+        elif self.model_name == "gemini-pro":
+            response = self.model.generate_content(
+                prompt)
+            return response.text
+        
+        elif self.model_name == 'openai':
+            res = self.model.chat.completions.create(
+                model="gpt-3.5-turbo-1106",
+                response_format={"type": "json_object"},
+                messages=[
+                    # {"role": "system", "content": "You are a helpful assistant."},
+                    {"role": "user", "content": f"{prompt}"},
+                ],
+                # seed=10,
+                temperature=0
+            )
+            return res.choices[0].message.content

prompts.py

@@ -0,0 +1,105 @@
+reasoning_modules = """
+1 How could I devise an experiment to help solve that problem?
+2 Make a list of ideas for solving this problem, and apply them one by one to the problem to see if any progress can be made.
+3 How could I measure progress on this problem?
+4 How can I simplify the problem so that it is easier to solve?
+5 What are the key assumptions underlying this problem?
+6 What are the potential risks and drawbacks of each solution?
+7 What are the alternative perspectives or viewpoints on this problem?
+8 What are the long-term implications of this problem and its solutions?
+9 How can I break down this problem into smaller, more manageable parts?
+10 Critical Thinking: This style involves analyzing the problem from different perspectives, questioning assumptions, and evaluating
+the evidence or information available. It focuses on logical reasoning, evidence-based decision-making, and identifying
+potential biases or flaws in thinking.
+11 Try creative thinking, generate innovative and out-of-the-box ideas to solve the problem. Explore unconventional solutions,
+thinking beyond traditional boundaries, and encouraging imagination and originality.
+12 Seek input and collaboration from others to solve the problem. Emphasize teamwork, open communication, and leveraging the
+diverse perspectives and expertise of a group to come up with effective solutions.
+13 Use systems thinking: Consider the problem as part of a larger system and understanding the interconnectedness of various elements.
+Focuses on identifying the underlying causes, feedback loops, and interdependencies that influence the problem, and developing holistic
+solutions that address the system as a whole.
+14 Use Risk Analysis: Evaluate potential risks, uncertainties, and tradeoffs associated with different solutions or approaches to a
+problem. Emphasize assessing the potential consequences and likelihood of success or failure, and making informed decisions based
+on a balanced analysis of risks and benefits.
+15 Use Reflective Thinking: Step back from the problem, take the time for introspection and self-reflection. Examine personal biases,
+assumptions, and mental models that may influence problem-solving, and being open to learning from past experiences to improve
+future approaches.
+16 What is the core issue or problem that needs to be addressed?
+17 What are the underlying causes or factors contributing to the problem?
+18 Are there any potential solutions or strategies that have been tried before? If yes, what were the outcomes and lessons learned?
+19 What are the potential obstacles or challenges that might arise in solving this problem?
+20 Are there any relevant data or information that can provide insights into the problem? If yes, what data sources are available,
+and how can they be analyzed?
+21 Are there any stakeholders or individuals who are directly affected by the problem? What are their perspectives and needs?
+22 What resources (financial, human, technological, etc.) are needed to tackle the problem effectively?
+23 How can progress or success in solving the problem be measured or evaluated?
+24 What indicators or metrics can be used?
+25 Is the problem a technical or practical one that requires a specific expertise or skill set? Or is it more of a conceptual or
+theoretical problem?
+26 Does the problem involve a physical constraint, such as limited resources, infrastructure, or space?
+27 Is the problem related to human behavior, such as a social, cultural, or psychological issue?
+28 Does the problem involve decision-making or planning, where choices need to be made under uncertainty or with competing
+objectives?
+29 Is the problem an analytical one that requires data analysis, modeling, or optimization techniques?
+30 Is the problem a design challenge that requires creative solutions and innovation?
+31 Does the problem require addressing systemic or structural issues rather than just individual instances?
+32 Is the problem time-sensitive or urgent, requiring immediate attention and action?
+33 What kinds of solution typically are produced for this kind of problem specification?
+34 Given the problem specification and the current best solution, have a guess about other possible solutions.
+35 Let's imagine the current best solution is totally wrong, what other ways are there to think about the problem specification?
+36 What is the best way to modify this current best solution, given what you know about these kinds of problem specification?
+37 Ignoring the current best solution, create an entirely new solution to the problem.
+38 Let's think step by step.
+39 Let's make a step by step plan and implement it with good notion and explanation"""
+
+
+select_prompt = """
+Select several reasoning modules that are crucial to utilize in order solve the given task:
+All reasoning module description
+{resonining_modules}
+{Task}
+Select several modules that are crucial for solving the tasks above
+"""
+
+adapt_prompt = """
+Rephrase and specify each reasoning module so that it better helps solving the task:
+SELECTED module descriptions:
+{selected_modules}
+{Task}
+Adapt each reasoning module description to better solve the tasks:
+"""
+
+implement_prompt = """
+Operationalize the reasoning modules into a step-by-step reasoning plan in JSON format:
+
+Paired IMPLEMENT step Demonstration
+Example:
+This SVG path element <path d="M 55.57,80.69 L 57.38,65.80 M 57.38,65.80 L 48.90,57.46 M 48.90,57.46 L
+45.58,47.78 M 45.58,47.78 L 53.25,36.07 L 66.29,48.90 L 78.69,61.09 L 55.57,80.69"/> draws a:
+(A) circle (B) heptagon (C) hexagon (D) kite (E) line (F) octagon (G) pentagon(H) rectangle (I) sector (J) triangle
+{ "Simplify SVG Path": ...
+"Breakdown of Path Commands": {
+"Move to Command (M)": "Sets the starting point for the next
+command without drawing anything.",
+"Line to Command (L) steps":
+{"Start and end coordinates of each line segment":
+"M 55.57,80.69 L 57.38,65.80: From point (55.57, 80.69) to (57.38,
+65.80)"}, … and finally closing the shape at (55.57, 80.69)"}
+"Critical Thinking Analysis": {
+"Logical Reasoning": {
+"Analysis of path continuity": "The path includes
+multiple line segments that connect distinct points. The path ends by
+connecting back to the starting point, indicating a closed shape.",
+"Identification of closed shapes": "The final line
+segment connects the last point back to the first point, which is
+characteristic of a closed shape."},
+…
+"Final Reasoning and Decision": "With 7 distinct points all
+connected in a closed path, the shape formed is a heptagon.",
+"Final Answer": "B) heptagon}
+
+ADAPTED module descriptions:
+{adapted_modules}
+{Task}
+Implement a reasoning structure for solvers to follow step-by-dtep and arrive at correct answers
+"""

requirements.txt

@@ -0,0 +1,3 @@
+openai==1.12.0
+python-dotenv==1.0.1
+google-generativeai==0.3.2

self_discover.py

@@ -0,0 +1,44 @@
+from prompts import(
+    select_prompt,
+    reasoning_modules,
+    adapt_prompt,
+    implement_prompt
+)
+from llm import LLM
+from task_example import task1
+
+class SelfDiscover:
+    def __init__(self, task) -> None:
+        self.llm = LLM(model_name="gemini-pro")
+        self.actions = ["SELECT", "ADAPT", "IMPLEMENT"]
+        self.task = task
+
+    def __call__(self):
+        for action in self.actions:
+            print(action)
+            if action == "SELECT":
+                print("yes")
+                prompt = select_prompt.replace("{Task}",self.task)
+                prompt = prompt.replace("{resonining_modules}", reasoning_modules)
+                print(prompt)
+                self.selected_modules = self.llm(prompt)
+
+            elif action == "ADAPT":
+                prompt = adapt_prompt.replace("{Task}",self.task)
+                prompt = prompt.replace("{selected_modules}",self.selected_modules)
+                print(prompt)
+                self.adapted_modules = self.llm(prompt)
+
+            elif action == "IMPLEMENT":
+                prompt = implement_prompt.replace("{Task}",self.task)
+                prompt = prompt.replace("{adapted_modules}", self.adapted_modules)
+                print(prompt)
+                self.reasoning_structure = self.llm(prompt)
+
+
+if __name__=="__main__":
+    result = SelfDiscover(task=task1)
+    result()
+    print(f"SELECTED_MODULES : {result.selected_modules}")
+    print(f"ADAPTED_MODULES : {result.adapted_modules}")
+    print(f"REASONING_STRUCTURE : {result.reasoning_structure}")

task_example.py

@@ -0,0 +1,42 @@
+task1 =""""
+You will be provided with unstructured data and your task is to accurately extract  property answers to a set of questions 
+from within the context.
+The context is from a  report which provides  with details of subject property as key value pairs
+
+<context>
+
+Distance from Subject: 0.20
+Comp 2
+Address: Kite street,207, CA
+Owner: JISHNU S
+Sale Price: $455,000
+Living Area: 2100
+APN: 04-05664-05660-05505
+Lot Area: 5000
+Total Rooms: 4
+Bedrooms: 2
+Sale Date: 14/10/2021
+Year Built: 1823
+
+</context>
+
+<questions> 
+
+0. what is the address of subject property
+1. which city is the subject property located
+2. which county is the subject property located
+3. what is the owner name of the subject property
+4. what id the loan number of the subject property
+5. wht is the Total assessed value of subject property
+6. what is the assessed value of improvements to the subject property
+7. what is the assessed value of Land of the subject property
+8. what is the estimated value of subject property
+9. what is the date of the estimated value of the subject property
+10. what is the estimated value range of the subject property
+11. what is the processed date of the subject property
+12. What is the confidence score of the subject property
+13. what is the forecast standard deviation of the subject property
+14. which state is the property located in of the subject property
+
+ </questions>
+"""