# Savonia Article Pro: From operator demonstrations to autonomous excavator control in simulation

##### Savonia Article Pro is a collection of multidisciplinary Savonia expertise on various topics.

11.11.2026 [3D](https://www.savonia.fi/en/category/3d)[Articles Pro](https://www.savonia.fi/en/category/articles-pro)[Mechanical Engineering](https://www.savonia.fi/en/category/mechanical-engineering)[RDI](https://www.savonia.fi/en/category/rdi)

Getting started with simulation can be challenging, especially without prior experience. In my first simulation project, I developed a workflow for collecting operator demonstrations, training and testing a Vision-Language-Action (VLA) model to control a simulated excavator. The project showed me how simulation can support machine development and testing before experimenting with physical equipment.

Simulation enables rapid prototyping and testing without risking damage to physical equipment. It also allows developers to explore demanding operating conditions, such as severe weather or hazardous environments.

## The task and motivation

The main goal was to develop a workflow for collecting operator demonstrations, preparing training data, training and running VLA model in simulation. The project uses SmolVLA, a lightweight robotics model that combines camera images, machine state, and language instruction to predict a sequence of actions. [1]

SmolVLA is trained and deployed using Hugging Face’s LeRobot library. The library allows developers to create custom robot integrations and deploy models through a Python interface. It also provides tools for recording, storing, and sharing datasets.

I could not find an existing project that combined ROS 2 communication between the policy and simulation, action scheduling based on Isaac Sim’s simulation clock, and support for Real-Time Chunking (RTC). The closest match I found was LeIsaac, which supports imitation learning in simulation and is built on Isaac Lab. [2] However, it did not provide the combination of features needed for this project, so I developed a modified LeRobot adapter.

The task used for data collection, training, and evaluation was to pick up a wooden log from the ground using the excavator’s gripper and place it in a container. The specified task name was “Pick up the log and place it in the blue container”.

## Modelling the excavator and its operating environment

The simulation’s visual background was reconstructed using Gaussian splatting from the environment used to develop the physical miniature excavator in Savonia’s Ai-MaSi project. The scene also includes a 3D model of the excavator, a container, and a wooden log for the manipulation task. Two simulated cameras provide observations: one mounted above the cabin and the other attached to the gripper head.

During operator demonstrations, the excavator is operated using an Xbox controller. The joysticks control the excavator’s slew, boom, arm and bucket movements, while the A button toggles the gripper between open and closed. An Isaac Sim action graph reads the controller inputs and applies them to the simulated excavator.

Using its ROS 2 Bridge extension, Isaac Sim publishes camera images, machine state, joystick commands, and simulation time on ROS 2 topics. [3] These data are used to record operator demonstrations and prepare datasets for policy training.

## Collecting consistent operator demonstrations in simulation

During operator demonstrations, a modified LeRobot recorder collects camera images, machine state, and joystick commands from the ROS 2 topics published by Isaac Sim. Camera images and machine state are saved as observations, while the applied joystick commands and the gripper open/close command are saved as actions.

The rate at which simulation time advances relative to wall-clock time can vary with computational load. Recording at fixed wall-clock intervals could therefore produce uneven spacing between samples in simulation time. To maintain consistent sampling intervals, the recorder follows the simulation’s internal clock, published on the ROS2 /clock topic. It schedules samples at regular intervals in simulation time and uses source timestamps to match the corresponding observations and actions.

The recorder accepts camera images and combined machine-state, and joystick values whose simulation timestamps are within ±2 milliseconds of the scheduled sampling time. If the required data are still missing more than 0.5 seconds of simulation time after the scheduled sampling time, the recorder raises an exception identifying the missing data and discards the incomplete episode.

Figure 1 illustrates this recording process. At a sampling rate of 30 Hz, the recorder schedules 30 samples per simulated second, regardless of how quickly the simulation advances relative to wall-clock time. Each topic carries timestamp within.

Figure 1. Recording in simulation time.

## Fine-tuning SmolVLA with operator demonstrations

LeRobot’s standard training command is used to fine-tune the pretrained SmolVLA model for the excavator and the log-handling task using the collected operator demonstrations. An example dataset is available on Hugging Face. [4]

## Running the excavator autonomously

During autonomous operation, the model receives synchronized observations from ROS 2 topics published by Isaac Sim and predicts an action chunk, a sequence of control commands. The chunk is associated with the scheduled sampling time of the observation used for inference.

Once inference finishes, the modified rollout uses the latest simulation time to determine the next scheduled execution time and select the corresponding action from the chunk. Actions whose scheduled execution times have already passed are skipped. The selected action is timestamped with its scheduled execution time and published in a ROS 2 message. Figure 2 illustrates this process.

Figure 2. VLA rollout in simulation time.

Commands pass through a ROS 2 control bridge before reaching Isaac Sim. The bridge validates the commands and checks whether policy control is permitted. It immediately forwards accepted timestamped commands to Isaac Sim, which schedules their execution. Within the simulation, normalized joystick commands determine the rate of change of joint-position targets for the simulated excavator. Figure 3 illustrates the command execution process.

Figure 3. Timestamped command execution.

If the predicted chunk contains no valid action for the next scheduled execution time, the rollout sends zero movement commands while retaining the gripper command. The simulation’s physics continues to run.

## A working demonstration workflow and priorities for further testing

The developed environment provides a practical way to collect operator demonstrations and evaluate VLA models supported by the LeRobot library. The project’s source code and dataset are linked in the references below. [5]

The dataset described earlier serves as proof of concept, illustrating how demonstrations are collected and used for model training. It contains only a few minutes of similar demonstrations, which limits its usefulness for evaluating performance across varied conditions and creates risk of overfitting during training. The next step is to collect more varied demonstrations and compare policy performance using different datasets and model checkpoints.

While evaluating the policy’s task specific performance, I also explored ways to correct unsuccessful trajectories. Human-in-the-loop data collection approach allows a human operator to supervise the policy and take control when necessary. The resulting corrective demonstrations can then be used for further training to improve the policy. [6]

Figure 4. Cabin and gripper camera views published on ROS 2 topics.

#### Author

Olli Kangasniemi, Test Engineer at T.A.I.T.O, Savonia University of Applied Sciences.

This article was prepared with the help of AI.

#### References:

Hugging Face. n.d. [SmolVLA](https://huggingface.co/docs/lerobot/smolvla). Accessed 2 November 2026. [1]

Hugging Face. n.d. [LeIsaac × LeRobot EnvHub](https://huggingface.co/docs/lerobot/en/envhub_leisaac). Accessed 5 November 2026. [2]

NVIDIA. n.d. [[isaacsim.ros2.bridge] ROS 2 Bridge](https://docs.isaacsim.omniverse.nvidia.com/6.1.0/py/source/extensions/isaacsim.ros2.bridge/docs/index.html). Accessed 5 November 2026. [3]

Kangasniemi, O. 2026. [isaac_excavator](https://huggingface.co/datasets/KolliOlli318/isaac_excavator). Dataset. Hugging Face. Accessed 6 November 2026. [4]

Kangasniemi, O. 2026. [isaac_lerobot_excavator](https://github.com/OlliKolli318/isaac_lerobot_excavator). GitHub repository. Source code for the project described in this article. Accessed 11 November 2026. [5]

Hugging Face. n.d. [Human-In-the-Loop Data Collection](https://huggingface.co/docs/lerobot/en/hil_data_collection). Accessed 9 November 2026. [6]