isYes/act-labelnet-mycobot280

ACT + LabelNet — observation-conditioned action decoding on myCobot 280

Eight Action Chunking Transformer (ACT) checkpoints from a controlled ablation on isYes/mycobot280_amr_pickplace, plus eight checkpoints from a separate frame-rate / chunk-size sweep.

The short version of what we found: giving the policy auxiliary heads that infer item shape and pose from its own observation, and feeding that inference into the action decoder, cuts open-loop action error roughly in half. Splitting the CVAE latent into label-dependent and label-invariant halves with an adversarial loss — the mechanism we originally expected to matter — contributes nothing measurable, and is actively harmful when used without the observation heads. The training logs say why: the CVAE posterior collapses to the prior, so there was never anything in the latent to disentangle. We publish the negative result in full.

📄 Read the full report (PDF) — method, statistics, limitations.

Authors

• Daye Lee (Addinedu) — @isYes · github.com/Daye-Lee18
• Gyujeong Song (Addinedu) — @songgyujeong

Task and hardware

A 7-DoF myCobot 280 picks up a casting and places it into a bin carried by a small AMR. Three synchronised RGB cameras (overhead, side, wrist), 480×640, 3 Hz. Observation and action are both 7-dim (J1–J6 in degrees + gripper 0–100); the action is the follower arm's joint position one step ahead.

Method

The base policy is ACT (Zhao et al., 2023) — a CVAE whose decoder predicts a chunk of k future actions from the current observation and a latent z. At training time z is encoded from the true future action chunk; at test time z = 0, so the decoder must work from the observation alone. That detail turns out to explain the main result.

Two mechanisms are added on top:

1. Latent split with gradient reversal. The 32-d latent is split into z_var (16-d) and z_inv (16-d). Linear heads decode item shape/pose from z_var under cross-entropy (w_var); another pair decodes the same labels from z_inv through a gradient-reversal layer (w_inv), pushing the encoder to make the labels undecodable there.

2. Observation-conditioned decoding. Two heads predict shape and pose from pooled camera features + joint state (w_obs). Their argmax indexes learnable shape/pose embeddings that are added to the decoder's latent input. This is the path that survives to test time — since z = 0 at inference, the latent split has no route to the action, while the observation heads do.

Checkpoints

Ablation (labelnetFull_*) — 8 checkpoints

ID	use_label_cvae	w_var	w_inv	w_obs	w_shape	w_pose	seed
A1_vanilla_seed0	✗	—	—	—	—	—	0
A2_vanilla_seed1	✗	—	—	—	—	—	1
B1_obsonly_seed0	✓	0.0	0.0	1.0	1.0	1.0	0
B2_disonly_seed0	✓	1.0	0.3	0.0	1.0	1.0	0
B3_v2full_seed0	✓	1.0	0.3	1.0	1.0	1.0	0
B3p_v2full_seed1	✓	1.0	0.3	1.0	1.0	1.0	1
C1_shapeonly_seed0	✓	1.0	0.3	1.0	1.0	0.0	0
C2_poseonly_seed0	✓	1.0	0.3	1.0	0.0	1.0	0

The vanilla baselines have the label module absent (use_label_cvae: False); the weight columns are inapplicable rather than zero — their stored config.pkl still contains default w_* values that are never read. You can verify this from the weights themselves: the vanilla state dicts hold 544 tensors, the label variants 570 — the 26 extra are the z_var/z_inv classifiers and the two observation heads.

Frame-rate / chunk sweep (ckpt_act_final_VA_*) — 8 checkpoints

chunk20_fps1, chunk20_fps3, chunk20_fps6, chunk20_fps15, chunk20_fps30, chunk52_fps1, chunk150_fps15, chunk300_fps30 (all seed 0).

⚠️ Provided for reuse; no conclusions drawn. These were trained on an earlier, unreleased dataset (final_VA_HDF5), not on the dataset released alongside this repo, so their numbers cannot be compared with anything in the Results section. Their final val_l1 also falls monotonically with frame rate (0.278 at 1 Hz → 0.114 at 30 Hz), but that mostly reflects the shrinking real-time horizon a fixed 20-step chunk covers, not a quality difference — so it should not be read as "30 Hz is better".

Common hyperparameters

ResNet-18 backbone per camera · hidden 512 · 4 encoder / 7 decoder layers · 8 heads · FFN 3200 · chunk k=20 · KL weight 10 · latent_var_dim 16 · latent_inv_dim 16 · grl_lambda 1.0 · lr 1e-5 · 5000 steps · best-on-validation selection. ≈106M parameters per checkpoint (~406 MB fp32).

Each folder holds policy_best.ckpt, config.pkl and dataset_stats.pkl. (policy_last.ckpt, policy_eval_temp.ckpt and optimizer state are not published — they are redundant with policy_best or are scratch artifacts.)

Results

Open-loop action prediction error on the 39 held-out episodes, normalised per joint. Every checkpoint is evaluated on the same episodes, so all comparisons are paired.

Checkpoint	Action MSE ↓	95% CI	shape acc	pose acc
A1 vanilla (s0)	1.041	[0.903, 1.178]	—	—
A2 vanilla (s1)	0.796	[0.682, 0.911]	—	—
B1 obs-only	0.504	[0.361, 0.647]	1.00	1.00
B2 dis-only	1.065	[0.921, 1.208]	0.35	0.26
B3 full (s0)	0.506	[0.364, 0.648]	1.00	1.00
B3′ full (s1)	0.665	[0.526, 0.803]	1.00	1.00
C1 shape-only	0.672	[0.539, 0.805]	1.00	0.18
C2 pose-only	0.503	[0.385, 0.620]	0.49	1.00

Paired comparisons (Wilcoxon signed-rank, n = 39):

Comparison	Δ MSE	95% CI	p
A1 vanilla − B3 full	+0.534	[+0.416, +0.653]	<1e-4
A2 vanilla − B3 full	+0.290	[+0.183, +0.397]	<1e-4
B1 obs-only − B3 full	−0.003	[−0.036, +0.031]	0.62 (n.s.)
B2 dis-only − A1 vanilla	+0.024	[+0.011, +0.036]	1e-4
C1 shape-only − B3 full	+0.166	[+0.111, +0.220]	<1e-4
C2 pose-only − B3 full	−0.004	[−0.087, +0.080]	0.92 (n.s.)

1. Observation-conditioning is the whole effect. ~51% error reduction against A1.

2. Latent disentanglement contributes nothing. B1 (no latent split at all) vs B3 (full) is Δ = −0.003 with a tight CI straddling zero. That is evidence for equivalence, not just a failed test. A linear probe agrees: shape/pose decode from z_var and z_inv at 0.28–0.42 against chance 0.33 / 0.25, with no systematic ordering.

Why it failed — the latent collapsed. The training logs locate the failure exactly. KL falls from ≈8.6 nats at init to 0.038 (train) / 0.0042 (val) across a 32-d latent, identically in all eight runs — textbook posterior collapse at kl_weight = 10.

Of the three heads reading the shape label during training: the supervised head on z_var never beats chance (final CE 1.048 vs ln 3 = 1.099; best-ever 0.946), the adversarial head on z_inv also sits at chance (1.139) — it "wins" only because there is nothing to find — and the observation head reaches CE 1e-4. So the disentanglement objective did not lose to the adversary; its easy, supervised half never trained at all. The pattern repeats in every label-module run: ce_var_shape ∈ [1.03, 1.11], ce_var_pose ∈ [1.36, 1.50], against chance 1.099/1.386.

This makes the negative result conditional: on this configuration the latent is inert, so anything built on it is decoration. It is not a general claim that disentanglement cannot help — testing that would require first fixing the collapse (KL annealing, free bits, or a lower kl_weight), which we did not do.

3. Disentanglement alone is worse than nothing. B2 (w_obs = 0) is the worst configuration. With the observation heads untrained but still read at inference, the decoder is conditioned on essentially random label embeddings. Its head accuracies (0.35 / 0.26, against chance 0.33 / 0.25) confirm the mechanism directly.

Use B1_obsonly_seed0. It matches the full method and is simpler.

Limitations

Read these before citing any number above.

1. Seed variance is comparable to several of the effects. Two seeds of the identical full config (B3, B3′) differ by 0.159 (p = 0.001); the two vanilla seeds differ by 0.245. The C1-vs-B3 gap behind "pose matters more than shape" is 0.166 — the same magnitude. With one seed per ablation cell that ranking is not separable from seed noise. The two headline results are safe (one is much larger than the seed spread, the other much smaller); the finer rankings are not.
2. Sparse evaluation on the rarest conditions. Per-episode error averages 5 sampled timesteps for 28 of 39 held-out episodes but only 1 for the other 11 — and those 11 are disproportionately the tilted/vertical episodes, i.e. the conditions we call hardest are measured most noisily.
3. Open-loop only. No success rates are reported. Preliminary real-robot rollouts succeeded under unseen combinations of item pose, interpolated AMR position and extrapolated heading, but the same condition sometimes succeeded and sometimes failed across repeats, and those rollouts used an earlier checkpoint series. A formal closed-loop comparison needs robot access we do not currently have.
4. far is confounded. No tilted/vertical episodes exist at far AMR positions, so a far slice is not a clean pose-generalisation test.
5. AMR pose is never conditioned on — it is continuous, so it is used only as an OOD evaluation axis.
6. The disentanglement result is conditional on the collapse (see above) — it is a finding about this configuration, not about latent disentanglement in general.
7. Narrow domain. One robot, one workcell, one task, three object shapes. Nothing here has been shown to transfer beyond that.

Usage

These are ACT checkpoints from a fork of act-plus-plus, not lerobot policies — load them with the training repo:

git clone https://github.com/Daye-Lee18/VLA_exp
hf download isYes/act-labelnet-mycobot280 \
    --include "labelnetFull_B1_obsonly_seed0/*" --local-dir ./ckpt

import pickle, torch

cfg = pickle.load(open("ckpt/labelnetFull_B1_obsonly_seed0/config.pkl", "rb"))
stats = pickle.load(open("ckpt/labelnetFull_B1_obsonly_seed0/dataset_stats.pkl", "rb"))
sd = torch.load("ckpt/labelnetFull_B1_obsonly_seed0/policy_best.ckpt", map_location="cpu")

print(cfg["policy_config"])   # architecture + loss weights

Reproducing the evaluation:

python scripts/data/offline_eval.py        # per-episode action MSE + head accuracy
python scripts/data/slide3_probe.py        # latent linear probe
python scripts/data/offline_ood_breakdown.py

Citation

@misc{lee2026obscond,
  title  = {Observation-conditioned action decoding beats latent disentanglement
            for out-of-distribution manipulation},
  author = {Lee, Daye and Song, Gyujeong},
  institution = {Addinedu},
  year   = {2026},
  url    = {https://huggingface.co/isYes/act-labelnet-mycobot280}
}

License

Apache 2.0. Built on ACT / act-plus-plus (MIT, © 2023 Tony Z. Zhao).