Initial commit: Add ChatGPT interface, UI elements, sample prompts

.gitattributes

@@ -0,0 +1,34 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,42 @@
+---
+title: ChatGPT Robotics
+emoji: 🤖
+colorFrom: blue
+colorTo: red
+sdk: gradio
+sdk_version: 3.19.1
+app_file: app.py
+pinned: false
+license: mit
+---
+
+**ChatGPT + Robotics Gradio demo.**
+
+This is an accompaniment to our work on using ChatGPT for robotics applications. This demonstration space allows you to
+interact with ChatGPT using prompts and high-level APIs specified to robotics scenarios, and get the model to generate code
+that solves robotics problems. 
+
+For details, please check out our blog post: https://aka.ms/ChatGPT-Robotics   
+and our paper: https://www.microsoft.com/en-us/research/uploads/prod/2023/02/ChatGPT___Robotics.pdf
+
+We encourage you to check out our main repo to access a full robot simulator with ChatGPT integration:  
+https://github.com/microsoft/PromptCraft-Robotics
+
+**Setup**
+
+1. Make an OpenAI ChatGPT account at chat.openai.com
+2. Navigate to https://chat.openai.com/api/auth/session and copy the `accessToken`.
+
+**Usage**
+
+1. Paste the access token in the ChatGPT Login textbox and click Login.
+2. Choose a sample prompt to start with. We provide the following prompts
+
+airsim: Drone navigation scenario in the AirSim simulator.
+embodied_agent: Embodied agent scenario with discrete actions (left, right, forward), and perception API for object detection.
+embodied_agent_closed_loop: Embodied agent scenario but instead of interacting with ChatGPT normally, the user is expected to provide observations of the scene (converted into text) at every step, such as the locations of objects around.
+manipulation: Robot arm scenario that is equipped with a suction pump that can pick up/release objects.
+real_drone_sample: Drone navigation scenario for the DJI Tello drone as seen in the paper.
+
+3. "Display Prompt" can be optionally used if you wish to see the prompt.
+4. Click on Initialize, and then continue the conversation in the following chat box.

app.py

@@ -0,0 +1,122 @@
+'''
+ChatGPT + Robotics Gradio demo.
+Author: Sai Vemprala
+
+For details, please check out our blog post: https://aka.ms/ChatGPT-Robotics, and our paper:
+https://www.microsoft.com/en-us/research/uploads/prod/2023/02/ChatGPT___Robotics.pdf
+
+In this demo, we provide a quick way to interact with ChatGPT in robotics settings using some custom prompts.
+As seen in our paper, we provide prompts for several scenarios: robot manipulation, drone navigation 
+(in a simulated setting (airsim) as well as real life), and embodied AI. embodied_agent_closed_loop is an
+experimental setting where observations from a scene can be described to ChatGPT as text.
+
+Parts of the code were inspired by https://huggingface.co/spaces/VladislavMotkov/chatgpt_webui/
+
+'''
+
+import gradio as gr
+from revChatGPT.V1 import Chatbot
+import glob, os
+
+access_token = None
+
+def parse_text(text):
+    lines = text.split("\n")
+    for i,line in enumerate(lines):
+        if "```" in line:
+            items = line.split('`')
+            if items[-1]:
+                lines[i] = f'<pre><code class="{items[-1]}">'
+            else:
+                lines[i] = f'</code></pre>'
+        else:
+            if i>0:
+                lines[i] = '<br/>'+line.replace(" ", "&nbsp;")
+    return "".join(lines)
+
+def configure_chatgpt(info): 
+    access_token = info
+    config = {}
+    config.update({"access_token": access_token})
+
+    global chatgpt
+    chatgpt = Chatbot(config=config)
+
+def ask(prompt):
+    message = ""
+    for data in chatgpt.ask(prompt):
+        message = data["message"]
+    return parse_text(message)
+    
+def query_chatgpt(inputs, history, message):
+    history = history or []
+    output = ask(inputs)
+    history.append((inputs, output))
+    return history, history, ''
+
+def initialize_prompt(prompt_type, history):
+    history = history or []
+    
+    if prompt_type:
+        prompt_file = './prompts/' + str(prompt_type) + '.txt'
+
+        with open(prompt_file, "r") as f:
+            prompt = f.read()
+        output = ask(prompt)
+        history.append(("<ORIGINAL PROMPT>", output))
+        
+    return history, history  
+
+def display_prompt(show, prompt_type):
+    if not prompt_type:
+        show = False
+        return 'Error - prompt not selected'
+
+    else:
+        if show:
+            prompt_file = './prompts/' + str(prompt_type) + '.txt'
+
+            with open(prompt_file, "r") as f:
+                prompt = f.read()
+
+            return prompt
+        else:
+            return ''
+
+with gr.Blocks() as demo:
+    gr.Markdown("""<h3><center>ChatGPT + Robotics</center></h3>""")
+    gr.Markdown(
+        "This is a companion app to the work [ChatGPT for Robotics: Design Principles and Model Abilities](https://aka.ms/ChatGPT-Robotics)")
+
+    if not access_token:
+        gr.Markdown("""<h4>Login to ChatGPT</h4>""")
+        with gr.Row():
+            with gr.Group():
+                info = gr.Textbox(placeholder="Enter access token here", label="ChatGPT Login")
+                with gr.Row():
+                    login = gr.Button("Login")
+                    login.click(configure_chatgpt, inputs=[info])
+                    
+    l=os.listdir('./prompts')
+    li=[x.split('.')[0] for x in l]
+
+    gr.Markdown("""<h4>Initial Prompt (based on scenario)</h4>""")
+    prompt_type = gr.components.Dropdown(li, label="Select sample prompt", value=None) 
+    
+    show_prompt = gr.Checkbox(label="Display prompt")
+    prompt_display = gr.Textbox(interactive=False, label="Prompt")
+    show_prompt.change(fn=display_prompt, inputs=[show_prompt, prompt_type], outputs=prompt_display)
+    
+    initialize = gr.Button(value="Initialize")
+
+    gr.Markdown("""<h4>Conversation</h4>""")
+    chatgpt_robot = gr.Chatbot()
+    message = gr.Textbox(placeholder="Enter query", label="")
+    
+    state = gr.State()
+    
+    initialize.click(fn=initialize_prompt, inputs=[prompt_type, state], outputs=[chatgpt_robot, state])
+    
+    message.submit(query_chatgpt, inputs=[message, state], outputs=[chatgpt_robot, state, message])
+
+    demo.launch()

prompts/airsim.txt

@@ -0,0 +1,32 @@
+I would like you to help me work with the AirSim simulator for drones. When I ask you to do something, please give me Python code that is needed to achieve that task using AirSim and then an explanation of what that code does. 
+Do not use any of the normal AirSim functions, you should only use the following functions that I have defined for you. You are also not to use any hypothetical functions that you think might exist. You should only use the functions that I have defined for you.
+You can use simple Python functions from libraries such as math and numpy.
+
+aw.takeoff() - takes off the drone.
+aw.land() - lands the drone.
+aw.get_drone_position() - returns the current position of the drone as a list of 3 floats corresponding to XYZ coordinates.
+aw.fly_to([x, y, z]) - flies the drone to the position specified as a list of three arguments corresponding to X, Y, Z coordinates.
+aw.fly_path(points) - flies the drone along the path specified by the list of points. Each point is again a list of 3 floats corresponding to X, Y, Z coordinates.
+aw.set_yaw(yaw) - sets the yaw of the drone to the specified value in degrees.
+aw.get_yaw() - returns the current yaw of the drone in degrees.
+aw.get_position(object_name): Takes a string as input indicating the name of an object of interest, and returns a list of 3 floats indicating its X,Y,Z coordinates.
+
+A few useful things: 
+Instead of moveToPositionAsync() or moveToZAsync(), you should use the function fly_to() that I have defined for you.
+If you are uncertain about something, you can ask me a clarification question, as long as you specifically identify it saying "Question".
+Here is an example scenario that illustrates how you can ask clarification questions. Let us assume a scene contains two spheres.
+
+Me: Fly to the sphere.
+You: Question - there are two spheres. Which one do you want me to fly to?
+Me: Sphere 1, please.
+
+The following objects are in the scene, and you are to refer to them using these exact names:
+
+turbine1, turbine2, solarpanels, car, crowd, tower1, tower2, tower3.
+
+None of the objects except for the drone itself are movable. Remember that there are two turbines, and three towers. When there are multiple objects of a same type,
+and if I don't specify explicitly which object I am referring to, you should always ask me for clarification. Never make assumptions.
+
+In terms of axis conventions, forward means positive X axis. Right means positive Y axis. Up means positive Z axis.
+
+Are you ready?

prompts/embodied_agent.txt

@@ -0,0 +1,18 @@
+Imagine you are helping me interact with the AirSim simulator. We are controlling an embodied agent. At any given point of time, you have the following abilities. 
+You are also required to output code for some of the requests.
+Question - Ask me a clarification question 
+Reason - Explain why you did something the way you did it. 
+Code - Output a code command that achieves the desired goal.
+
+The scene consists of several objects. We have access to the following functions, please use only these functions as much as possible:
+
+Perception:
+get_image() : Renders an image from the front facing camera of the agent 
+detect_objects(img): Runs an object detection model on an image img, and returns two variables - obj_list, which is a list of the names of objects detected in the scene. obj_locs, a list of bounding box coordinates in the image for each object.
+
+Action:
+forward(): Move forward by 0.1 meters. 
+turn_left(): Turn left by 90 degrees. 
+turn_right(): Turn right by 90 degrees.
+
+You are not to use any other hypothetical functions. You can use functions from Python libraries such as math, numpy etc. Are you ready?

prompts/embodied_agent_closed_loop.txt

@@ -0,0 +1,10 @@
+Imagine I am a robot equipped with a camera and a depth sensor. 
+I am trying to perform a task, and you should help me by sending me commands. You are only allowed to give me the following commands:
+
+turn(angle): turn the robot by a given number of degrees
+move(distance): moves the robot straight forward by a given distance in meters.
+
+On each step, I will provide you with the objects in the scene as a list of <object name, distance, angle in degrees>. 
+You should reply with only one command at a time. The distance is in meters, and the direction an angle in degrees with respect to the robot's orientation. 
+Negative angles are to the left and positive angles are to the right. If a command is not valid, I will ignore it and ask you for another command. 
+If there is no relevant information in the scene, use the available commands to explore the environment.

prompts/manipulation.txt

@@ -0,0 +1,21 @@
+Imagine we are working with a manipulator robot. This is a robotic arm with 6 degrees of freedom that has a suction pump attached to its end effector. I would like you to assist me in sending commands to this robot given a scene and a task.
+
+At any point, you have access to the following functions:
+
+grab(): Turn on the suction pump to grab an object
+release(): Turns off the suction pump to release an object
+get_position(object): Given a string of an object name, returns the coordinates and orientation of the vacuum pump to touch the top of the object [X, Y, Z, Yaw, Pitch, Roll]
+move_to(position): It moves the suction pump to a given position [X, Y, Z, Yaw, Pitch, Roll].
+
+You are allowed to create new functions using these, but you are not allowed to use any other hypothetical functions.
+
+Keep the solutions simple and clear. The positions are given in mm and the angles in degrees. You can also ask clarification questions using the tag "Question - ". Here is an example scenario that illustrates how you can ask clarification questions.
+
+Let's assume a scene contains two spheres.
+Me: pick up the sphere. 
+You: Question - there are two spheres. Which one do you want me to pick up? 
+Me: Sphere 1, please.
+
+Use Python code to express your solution.
+
+Are you ready?

prompts/real_drone_sample.txt

@@ -0,0 +1,34 @@
+Imagine you are helping me interact with a real drone flying in an indoor space. At any given point of time, you have the following abilities, each identified by a unique tag. You are also required to output code for some of the requests.
+
+Question: You can ask me a clarification question, as long as you specifically identify it saying "Question".
+Code: Output a code command that achieves the desired goal.
+Reason: After you output code, you should provide an explanation why you did what you did.
+
+The simulator contains a drone, along with several objects. Apart from the drone, none of the objects are movable. Within the code, we have the following commands available to us. You are not to use any other hypothetical functions.
+
+get_position(object_name): Takes a string as input indicating the name of an object of interest, and returns a vector of 4 floats indicating its X,Y,Z,Angle coordinates.
+self.tello.fly_to(position): Takes a vector of 4 floats as input indicating X,Y,Z,Angle coordinates and commands the drone to fly there and look at that angle
+self.tello.fly_path(positions): Takes a list of X,Y,Z,Angle positions indicating waypoints along a path and flies the drone along that path
+self.tello.look_at(angle): Takes an angle as input indicating the yaw angle the drone should look at, and rotates the drone towards that angle
+
+Here is an example scenario that illustrates how you can ask clarification questions. Let us assume a scene contains two spheres?
+
+Me: Fly to the sphere.
+You: Question - there are two spheres. Which one do you want me to fly to?
+Me: Sphere 1, please.
+
+You also have access to a Python dictionary whose keys are object names, and values are the X,Y,Z,Angle coordinates for each object:
+
+self.dict_of_objects = {'origin': [0.0, 0.0, 0.0, 0],
+'mirror': [1.25, -0.15, 1.2, 0],
+'chair 1': [0.9, 1.15, 1.1, np.pi/2],
+'orchid': [0.9, 1.65, 1.1, np.pi/2],
+'lamp': [1.6, 0.9, 1.2, np.pi/2],
+'baby ducks': [0.1, 0.8, 0.8, np.pi/2],
+'sanitizer wipes': [-0.3, 1.75, 0.9, 0],
+'coconut water': [-0.6, 0.0, 0.8, -np.pi],
+'shelf': [0.95, -0.9, 1.2, np.pi/2],
+'diet coke can': [1.0, -0.9, 1.55, np.pi/2],
+'regular coke can': [1.3, -0.9, 1.55, np.pi/2]}
+
+Are you ready?

requirements.txt

@@ -0,0 +1 @@
+revchatGPT