Overview

Vision-based manipulation policies are sensitive to new objects, changes in illumination and distractors in the scene. This is particularly critical for assembly tasks such as LEGO stacking, where the policy judges from pixel-level cues whether a brick is aligned, and a different brick shape, color or lighting alters these cues. The policies from Learning Where it Matters, which this project builds on, are each trained for a single object and act on a cropped wrist-camera image. They remain largely robust to scene-level distractors but fail when a distractor appears close to the object within that crop. In practice they also proved sensitive to changes in lighting. Real-world demonstrations that cover such variation are expensive and hard to collect.

This project reuses the recorded robot trajectories to enable the transfer to other objects and scenes. Replaying them in a photorealistic simulation with other bricks, stacks, colors, lights and backgrounds and co-training the vision network of the policy on the resulting images teaches the policy visual cues that carry over to different bricks and colors. Demonstrations recorded with a single 2x2 brick thus provide training data for many objects and scenes without additional robot runs. A policy trained only on a 2x2 brick and co-trained on ten rendered LEGO scenes places new bricks, even ones absent from all training data, without losing performance on the original brick. In simulated evaluations in Isaac Lab, success on bricks that appear only in the synthetic images rises from 36–50 to 97–100 of 100 attempts, and co-training with half of the recorded episodes gives the same success rates.

Simulation setup

The pipeline is a CAD-to-simulation workflow that parses LEGO model files from common design formats (LDraw, LXFML, BrickLink Studio) and turns them into Isaac Sim scenes. It has three modules, a rendering module that creates photorealistic images of LEGO scenes, a real-to-sim module that replays real-world trajectories in LEGO scenes in simulation and a scene exporter for physics simulation in Isaac Lab. Further modifications extend this into a prompt-to-simulation workflow, which will follow in a future update. Each scene consists of a LEGO model on a table and a Franka Emika FR3 robot arm with a wrist camera. To vary the appearance, the pipeline includes a fully configurable scene randomizer that changes the lighting, the HDRI background, the table and the brick colors.

LEGO scene rendering from the robot’s wrist camera (left) and a third-person view (right)

Training data

Training data for LEGO stacking policies are generated in the physics simulation by an automated script from Learning Where it Matters (see future updates for multi-brick stacking and planning). It autonomously generates a dataset of successful and failed stacking demonstrations with randomized trajectories. The resulting dataset consists of 300 push-down demonstrations (241 successes/59 failures), in which the robot holds a brick a few millimeters above its target and aligns it using only the wrist camera. The demonstrations are recorded in simulation with real-time rendering, as a stand-in for real robot recordings.

Recorded wrist-camera trajectories of the dataset.

Every successful episode is replayed in Isaac Sim in ten scenes with different bricks and randomized appearance, turning 21,630 recorded frames into 216,300 photorealistic path-traced training frames without additional robot runs, where the number of scenes is limited only by computing time.

One recorded episode (left) and its re-rendered versions for ten different scenes (right)

Real camera vs. simulation

To assess how closely the simulation matches the real world, it was compared with real robot recordings of a 2x4 brick. After a rough manual alignment, the camera extrinsics and the gripper height were fitted to the edges of the fingers and bricks in the real image. The camera model and the look of the scene (lighting and materials) were tuned by minimizing the DINOv2 feature distance between rendered and real frames. For each episode, the position of the held brick in the gripper is estimated from its silhouette. For each frame, a small translational shift of the robot is fitted based on the lower-brick silhouette and the baseplate stud grid to correct gripper pose errors in the recordings.

Real wrist-camera recording (left) and its re-render (right)

Method

Pipeline

1 · Record
Wrist-camera frame of a push-down demonstration with a yellow 2x2 brick

2x2 push-down demos

→
2 · Real-to-Sim
The same frame re-rendered in ten brick scenes

Re-rendered trajectories in 10 randomized scenes

→
3 · Co-training
Vision
Policy

Train the vision encoder, action policy frozen

→
4 · Deploy
Robot after placing a brick on a tower of five

Deploy on seen and unseen bricks

Co-training

Demo frameThe same frame in ten Real2Sim scenes
→
Vision + bridge DINOv2 + LoRA, pooler, 16 features
→
Policy + controller feedback
→
Loss Action loss MSE to demo action from demo frame + 2 of 10 renders + 0.01 × variance loss std of each pooled image feature across the batch

Architecture

Wrist-camera view
→
DINOv2 + LoRA image tokens
→
Pooler tokens → vector
→
Bridge 16 features
→
controller feedback↓
16 features8
→
Policy BC actor, trained on 2x2 only
→
Actionx/y end-effector motion
trained in co-trainingfrozen

Results

All evaluations run in Isaac Lab with a different renderer mode than the co-training renders. Every scene is evaluated with 100 rollouts using the same policy, random seed and physics version.

Comparing the behavior cloning (BC) and soft actor-critic (SAC) baselines, which are both trained on the 2x2 brick only, the co-trained policies match or exceed them on the training brick (BC 99 % vs. 98 %, SAC 100 % vs. 91 %). On bricks that appear only in re-renders, the co-trained policies clearly outperform the baselines, with 97–100 % compared to 16–50 %. The baselines perform reasonably only on the 2x3 brick (BC 93 %, SAC 61 %), which is similar in size to the 2x2. On the unseen 1x6 and the tower of five, the baselines fail (11–14 %), while the co-trained BC policy reaches 92–97 %. This suggests that co-training on the re-rendered trajectories lets the vision encoder learn cues that transfer to new brick shapes and colors, without losing performance on the original brick. Comparing the co-trained BC and SAC policies indicates that the BC policy is more robust on unseen bricks, most clearly on the tower of five (92 % vs. 42 %), but this difference was not investigated further within the scope of this project. In the following sections, the co-trained BC policy is used for all evaluations.

Success rate with the full dataset over 100 rollouts per bar, with 95 % Wilson confidence intervals.

Data size

Even though rendering does not require additional robot runs, it still requires computing time. This motivates assessing how much of the dataset is needed to achieve the same performance. Using the same co-training procedure, the policy was trained on 100 %, 50 %, 25 % and 10 % of the 241 successful episodes. With half of the 241 episodes, it still places every brick in 95 to 100 of 100 rollouts and the 2x2 stack in 98, while the distractor scene drops to 86. With 25 %, the long 1x6 brick drops to 75 % and the distractor scene to 82 %, while the stack stays at 97 %. With 10 %, the 1x6 (17 %), the distractor scene (27 %), the stack (39 %) and the tower (61 %) break down, while the other seven bricks stay at 89 to 99 %. The scenes that differ most from the training data are therefore the first to fail when the data is reduced.

Success of the co-trained, frozen BC actor vs. share of the 241 training episodes (21,630 frames, each rendered in 10 scenes), 100 rollouts per point and one training run per data size.

Take
Side view
Wrist camera (what the policy sees)

Out-of-distribution evaluation

In this evaluation, the policy faces ten completely random stacks of 3 to 5 bricks and plates in random colors, none of them in this combination in any training data. Over five rollouts each, 32 of 50 placements succeed. Four stacks work every time and two always fail, a 1x4 that covers only half of the 2x4 below and a 2x2 on a 2x2 on a plate.

Ten randomly generated stacks of 3 to 5 bricks and plates in random colors, none of them in any training data. Co-trained BC actor with the full dataset.

Take
Side view
Wrist camera (what the policy sees)

Key takeaways

  • The co-trained policy matches or exceeds the baselines on the training brick and generalizes to new bricks and colors, even when they don’t appear in the training data.
  • The co-trained policy still places the brick in 90 % of rollouts with a second brick of the same color one stud beside the target.
  • Half of the demonstrations give the same results, while the scenes that differ most from the training data are the first to fail with less data.

All projects