Repository: simpler-env/SimplerEnv
Issue: Replicating results with octo-small
Root cause: Validation Gap

Hi,  thank you very much for the great work, I really appreciate the resources you provide!

As part of my project, I'm comparing various VLAs in simulation and, later on, a real robot arm. To facilitate this, I'm running both **SimpleEnv** and **Octo** in separate containers, communicating via ROS messages. The goal is to make it easy to **"plug and play"** different policies across both simulated and real-world setups.

When I compare the behavior of the Google robot in SimpleEnv using the **Octo inference class** you provided with the behavior of my own implementation of the Octo model, I observe significant differences. I’ve replicated the inference logic from your class, using the same parameters (`observations`, `rng_key`, `pad_mask`, etc.) in my model’s inference call, but the `norm_raw_actions` differ substantially.

Here is a comparison of a single inference step using both implementations (same observation, same rgb-key, same task prompt, same data set ID and same Octo Model in both cases):

|                    | Octo Model (Provided)                                                                                  | My Implementation                                                                                         |
|--------------------|--------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|
| **norm_raw_actions** | `[[[ 0.18295571,  0.10927399, -0.04399001,  0.5336809,   0.01472024,  0.08168381,  1.0123447 ],`<br>` [ 1.1213822,   0.7550302,  -0.09065438,  1.3423952,  -0.7036034,  0.12773165,  1.0055703 ],`<br>` [ 1.7296495,   1.1819491,  -0.04365452,  1.346617,   -0.8032606,  0.14698559,  1.0084409 ],`<br>` [ 1.9198289,   1.1065358,   0.14024378,  1.2324716,  -0.7146998, -0.2564179,   1.0052774 ]]]` | `[[[ 1.9387648,  -0.6417607,  -0.93725055,  1.6864867,  -1.2197884,  -0.5053983,   1.0149632 ],`<br>` [ 2.183039,   -0.595375,  -0.7117496,  2.4544923,  -1.6508219, -0.6470888,   0.99405676 ],`<br>` [ 2.0241492,  -0.5420692,  -0.6493971,  2.274405,  -1.2239971,  -0.51823604,  1.0139754 ],`<br>` [ 1.7702967,  -0.43534094, -0.6890749,  2.0151796,  -1.034934,  -0.2530603,   0.9973863 ]]]` |
| **rgb_key**        | `[2336434339, 995697925]`                                                                              | `[2336434339, 995697925]`                                                                                 |


So, I have a couple of questions:  
1. Did you perform any fine-tuning on the Octo model included in the repository?  
2.1. If so, what would you consider the **minimal** amount of fine-tuning necessary to achieve reasonable results?
2.2 If not, do you have any suggestions what else could have caused the differences? What should I look into?

_(I’m asking because my initial plan was to evaluate the policies in a zero-shot setting. However, since I'm new to VLAs and this is my first project in the area, I’m still trying to understand the minimal requirements for meaningful comparisons.)_

Thanks in advance for any advice you can share!






I didn't perform any finetuning. I used Octo 1.0 model off-the-shelf. I didn't use the later Octo 1.5 model.