Update README.md

README.md

@@ -1,3 +1,73 @@
----
-license: mit
----
+---
+license: mit
+pipeline_tag: robotics
+---
+# Octo Base
+
+See https://github.com/octo-models/octo for instructions for using this model.
+
+Octo Base is trained with a window size of 2, predicting 7-dimensional actions 4 steps into the future using a diffusion policy. The model is a Transformer with 93M parameters (equivalent to a ViT-B). Images are tokenized by preprocessing with a lightweight convolutional encoder, then grouped into 16x16 patches. Language is tokenized by applying the T5 tokenizer, and then applying the T5-Base language encoder. 
+
+Observations and tasks conform to the following spec:
+
+Observations: 
+
+```
+{
+    image_primary: ('batch', 'history_window', 256, 256, 3),
+    image_wrist: ('batch', 'history_window', 128, 128, 3),
+}
+```
+
+Tasks: 
+```
+{
+    image_primary: ('batch', 256, 256, 3),
+    image_wrist: ('batch', 128, 128, 3),
+    language_instruction: {
+        attention_mask: ('batch', 16),
+        input_ids: ('batch', 16),
+    },
+}
+```
+
+At inference, you may pass in any subset of these observation and task keys, with a history window up to 2 timesteps.
+
+
+This model was trained on a mix of datasets from the Open X-Embodiment dataset.
+
+| Dataset                                                    | Proportion of batch |
+|------------------------------------------------------------|---------------------|
+| Fractal (Brohan et al, 2022)                               | 17.0\%              |
+| Kuka (Kalashnikov et al, 2018)                             | 17.0\%              |
+| Bridge (Walke et al, 2023)                         | 17.0\%              |
+| BC-Z (Jang et al, 2022)                                    | 9.1\%               |
+| Stanford Hydra Dataset (Belkhale et al, 2023)          | 6.0\%               |
+| Language Table~ (Lynch et al, 2023)                | 5.9\%               |
+| Taco Play (Rosete-Beas et al, 2022, Mees et al., 2023)   | 3.6\%               |
+| Furniture Bench Dataset (Heo et al, 2023)      | 3.3\%               |
+| UTAustin Mutex (Shah et al, 2023)                       | 3.0\%               |
+| Austin Sailor Dataset (Nasiriany et al, 2022)          | 2.9\%               |
+| Roboturk (Mandlekar et al, 2018)         | 2.8\%               |
+| Toto (Zhou et al, 2023)                                 | 2.4\%               |
+| Austin Sirius Dataset (Liu et al, 2023)                 | 2.3\%               |
+| Berkeley Autolab UR5 (Chen et al)            | 1.5\%               |
+| IAMLab CMU Pickup Insert (Saxena et al, 2023) | 1.2\%               |
+| Viola (Zhu et al, 2023)                                 | 1.2\%               |
+| Berkeley Fanuc Manipulation (Zhu et al, 2023) | 1.0\%               |
+| NYU Franka Play Dataset (Cui et al, 2022)                | 0.9\%               |
+| UCSD Kitchen Dataset (Ge Yan and Wang, 2023)                 | <0.1\%              |
+| Jaco Play (Dass et al, 2023)                         | 0.6\%               |
+| Berkeley Cable Routing (Luo et al, 2023)           | 0.3\%               |
+| Austin Buds Dataset (Zhu et al, 2022)                  | 0.3\%               |
+| CMU Stretch (Mendonca et al, 2023)                 | 0.2\%               |
+| NYU Door Opening (Pari et al, 2021)                | 0.1\%               |
+| DLR EDAN Shared Control (Quere et al, 2020)          | 0.1\%               |
+
+# Updates for Version 1.5
+- Language task tokens are now repeated at every timestep in the context window.
+- Augmented the language instructions in the data with rephrasings from GPT-3.5.
+- Bug fixes:
+  - Turned off dropout in the diffusion head due to incompatibility with layer norm.
+  - Fixed an off-by-one error with the attention mask.
+  - Fixed an issue where different image augmentations did not get fresh random seeds.