NiiL

Robot dataset analysis and VLA inputs/outputs

English translation

Learning objectives

Read public demonstration data and establish evaluation criteria, then check the meanings of inputs and outputs connected to vision, language, and action models.

Execution conditions

Small-scale data analysis requires requests, pandas, pyarrow, and scikit-learn. Read only one public Parquet file; an internet connection is required.

VLA model inference and training use separate hardware and model-specific environments.

If the required package is missing, run %pip install pyarrow==23.0.1 in a Colab or Jupyter code cell.

Applications

From demonstration data to a pick-and-place policy

Inspect public grasping and movement demonstrations to check the meaning of actions and the evaluation split. Extend a small offline baseline model toward policies using vision and language.

Observation and state: synchronized camera images, robot state, and task instructions. Check dataset features and timing.

Action: Dataset-defined joint and gripper actions, and action chunks. First map their representation and normalization.

Reward and constraints: Separate imitation-learning prediction error from actual task success. Record joint and input ranges and failure conditions.

Comparison baseline: Offline action prediction with state-only Ridge. Evaluate the closed-loop policy separately.

Metrics to check: MAE by component with an episode-based split. Add success rate, safety violations, and latency when running on real hardware or in simulation.

Analyze states and actions in a robot dataset

Starting conditions

Read the first Parquet shard of LeRobot svla_so100_pickplace. Do not download videos or model weights.

Procedure

  1. Use the provided code to check repository revision, FPS, episode count, and state/action shapes.
  2. Plot Episode 0 state and action against time, and compare the components and units of both arrays with the data card.
  3. State which metadata and features correspond to the camera, robot state, and natural-language instructions.

Results to retain

"""Read one public LeRobot data shard, without downloading videos or model weights."""
import io,json
import numpy as np
import pandas as pd
import requests
repo='lerobot/svla_so100_pickplace'
def read_json(url):
    response=requests.get(url,timeout=30);response.raise_for_status();return response.json()
manifest=read_json('https://huggingface.co/api/datasets/'+repo)
revision=manifest['sha']
base='https://huggingface.co/datasets/'+repo+'/resolve/'+revision+'/'
metadata=read_json(base+'meta/info.json')
shards=sorted(x['rfilename'] for x in manifest['siblings'] if x['rfilename'].startswith('data/') and x['rfilename'].endswith('.parquet'))
assert shards,'Dataset layout changed; inspect the repository data tree.'
response=requests.get(base+shards[0],timeout=60);response.raise_for_status()
frame=pd.read_parquet(io.BytesIO(response.content))
print('Repository / revision / shard:',repo,revision,shards[0])
print('Format / FPS / episodes:',metadata.get('codebase_version'),metadata['fps'],metadata['total_episodes'])
print('Features:',metadata['features'])
print('Loaded rows / episodes:',len(frame),frame['episode_index'].nunique())
state=np.stack(frame['observation.state'].to_numpy())
action=np.stack(frame['action'].to_numpy())
print('State / action shape:',state.shape,action.shape)
# Task text and video paths are described in the repository metadata.
# No robot commands, model execution, remote uploads, or video downloads occur here.

Record results and self-check

Item:

Feature·metadata:

Shape and units:

Verified meaning:

Points to check

Episode-level evaluation and information leakage

Starting conditions

Split the loaded episodes into the first 80% and last 20%. Build a Ridge baseline model that predicts actions from states.

Procedure

  1. Sort the unique values of episode_index, use the first 80% of complete episodes for training and the remaining complete episodes for testing. Keep frames from the same episode from appearing in both sets, and record the episode IDs and frame counts for train/test.
  2. Train Ridge(alpha=1.) on the training set's state and action. Calculate component-wise MAE between test predictions action and ground truth using multioutput="raw_values".
  3. Also train and evaluate the same model with a random 80/20 split of frame indices from the same complete dataset. Tabulate MAE for each action component by split, and plot the true and predicted values of one component in the corresponding test order.
  4. Using the results from both splits, explain the difference between placing adjacent frames in both training and test sets and evaluating new episodes.

Results to retain

from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error
episodes=np.sort(frame['episode_index'].unique());cut=max(1,int(.8*len(episodes)))
train=np.isin(frame['episode_index'],episodes[:cut]);test=~train
assert test.any(),'Select a shard containing at least two episodes.'
model=Ridge(alpha=1.).fit(state[train],action[train])
predicted=model.predict(state[test])
print('Episode split MAE:',mean_absolute_error(action[test],predicted,multioutput='raw_values'))
# TODO: Compare with a frame-based split of the same data.

Record results and self-check

Split:

Number of training episodes:

Number of evaluation episodes:

MAE by component:

Potential leakage:

Points to check

Inputs needed to extend to VLA

Starting conditions

Use the official SmolVLA explanation and the current dataset's features. Complete the input/output and evaluation design instead of training the entire model.

Procedure

  1. Tabulate the mapping from camera, current state, and instructions to action chunks.
  2. Identify where to check action normalization, coordinate systems, control periods, and delays during chunk execution.
  3. Design one additional observation, instruction, or evaluation episode needed for your research. When a GPU is ready, extend to pretrained policy inference using the official example.

Results to retain

Record results and self-check

Inputs and outputs:

Dataset mapping:

Normalization and timing:

Evaluation method:

Points to check

Explore further

  1. When using a VLM to interpret high-level goals, test image and instruction → structured goal conversion with vLLM's multimodal input examples.
  2. Measure inference latency and output validity, and separately validate coordinates and ranges before connecting to low-level robot actions.

Official examples and data