Robot dataset analysis and VLA inputs/outputs

Read public demonstration data and establish evaluation criteria, then check the meanings of inputs and outputs connected to vision, language, and action models.

Small-scale data analysis requires requests, pandas, pyarrow, and scikit-learn. Read only one public Parquet file; an internet connection is required.

VLA model inference and training use separate hardware and model-specific environments.

If the required package is missing, run %pip install pyarrow==23.0.1 in a Colab or Jupyter code cell.

From demonstration data to a pick-and-place policy

Inspect public grasping and movement demonstrations to check the meaning of actions and the evaluation split. Extend a small offline baseline model toward policies using vision and language.

Observation·State

Synchronized camera images, robot states, and task instructions. Check the dataset features and time basis.

Action

Dataset-defined joint and gripper actions, and action chunks. First map their representation and normalization.

Reward·Constraint

Separate imitation-learning prediction error from actual task success. Record joint and input ranges and failure conditions.

Comparison baseline

Offline action prediction with state-only Ridge. Evaluate the closed-loop policy separately.

Metrics to check

Component-wise MAE for episode-level splits. Add success rate, safety violations, and latency for real or simulated execution.

Starting conditions

Read the first Parquet shard of LeRobot svla_so100_pickplace. Do not download videos or model weights.

Procedure

1. Use the provided code to check repository revision, FPS, episode count, and state/action shapes.
2. Plot Episode 0 state and action against time, and compare the components and units of both arrays with the data card.
3. State which metadata and features correspond to the camera, robot state, and natural-language instructions.

Results to retain

• A data structure table and Episode 0 state/action plots.

Starting conditions

Split the loaded episodes into the first 80% and last 20%. Build a Ridge baseline model that predicts actions from states.

Procedure

1. Sort the unique values of episode_index, use the first 80% of complete episodes for training and the remaining complete episodes for testing. Keep frames from the same episode from appearing in both sets, and record the episode IDs and frame counts for train/test.
2. Train Ridge(alpha=1.) on the training set's state and action. Calculate component-wise MAE between test predictions action and ground truth using multioutput="raw_values".
3. Also train and evaluate the same model with a random 80/20 split of frame indices from the same complete dataset. Tabulate MAE for each action component by split, and plot the true and predicted values of one component in the corresponding test order.
4. Using the results from both splits, explain the difference between placing adjacent frames in both training and test sets and evaluating new episodes.

Results to retain

• Error table by split and a prediction plot for one action component.

Starting conditions

Use the official SmolVLA explanation and the current dataset's features. Complete the input/output and evaluation design instead of training the entire model.

Procedure

1. Tabulate the mapping from camera, current state, and instructions to action chunks.
2. Identify where to check action normalization, coordinate systems, control periods, and delays during chunk execution.
3. Design one additional observation, instruction, or evaluation episode needed for your research. When a GPU is ready, extend to pretrained policy inference using the official example.

Results to retain

• VLA input/output table and research application plan.

1. When using a VLM to interpret high-level goals, test image and instruction → structured goal conversion with vLLM's multimodal input examples.
2. Measure inference latency and output validity, and separately validate coordinates and ranges before connecting to low-level robot actions.

• SmolVLA · Architecture and usage
• Public demonstration dataset
• Episode visualization
• vLLM · multimodal inputs