ManiMux

Install

Run commands from the repository root. Python 3.11/3.12 are supported.

uv sync --dev
uv run manimux-robogui --robot yam --demo --host 127.0.0.1 --port 8086

Open http://127.0.0.1:8086. This demo does not connect hardware. Use --robot piper or --robot aloha for the corresponding asset preview.

For an existing hardware environment, install the current checkout explicitly:

uv pip install --python envs/yam/.venv/bin/python -e '.[replay,realsense,xpolicylab]'

This does not install the robot SDK or create the model environment. Follow the station setup reference and the selected model runbook. Do not run root uv sync against an independent hardware/model environment; it can remove dependencies absent from the root lockfile.

Configuration

One experiment YAML selects the deployment. Keep common settings in their owning component YAML and reference them with config:.

Station setup: copy yam_example.yaml to station.yaml only if a private station file does not already exist. Fill actual CAN interfaces, camera serials, endpoints and checkpoint roots. Preserve component names.

config: paths resolve relative to the containing YAML; mappings merge and lists replace. Experiment run.output_dir is relative to the launch directory. Station-local paths resolve relative to the station file. Formal studies use collection recording.root; station output overrides are rejected.

Command limits and receive-only validation

PiPER's configured station joint_limits (radians) and normalized gripper range [0, 1] are shared automatically with the runtime. If executor.command_safety also defines position bounds, its existing 0.001 coordinate-unit tolerance is applied before intersecting with the strict station bounds and the executor's optional symmetric position cap. An empty intersection is an error. Other controllers can declare bounds through the same interface; undeclared bounds continue to use the existing configured limits.

Smooth uses these per-coordinate bounds together with its own velocity and acceleration limits and any stricter command-safety rates. It brakes before a boundary, retains the velocity of the emitted command, and rejects an infeasible state instead of clipping it into range. Inference-gap holds also respect these limits. Raw policy predictions remain available unchanged in the recording. Direct and MPC commands pass the same final validation; this does not add smoothing to those executors.

With robot.options.execute: false, command shape, configured safety limits and the controller's pure command validation still apply, without command transmission. This does not test motor readiness or successful physical execution. The resolved shared bounds, scalar cap, command-safety rates and control period are saved as meta.json: command_limits.

Use the same --local PATH on entry points that support station binding. The common camera, policy and runtime entry points support it; RoboGUI network options remain separate. Advanced model-specific launchers may instead take a model recipe as --config. See entry-point scope.

Before launch, inspect the resolved config without connecting hardware:

envs/yam/.venv/bin/python - <<'PY'
from manimux.collections import load_collection
catalog = load_collection('manimux/configs/lists/pi05-latest300ep-step30000.yaml')
cfg = catalog.choices['coin_rtc'].config
for key in ('run', 'policy', 'inference', 'executor', 'evaluation'):
    print(key, cfg[key])
PY

Edit YAML before loading the experiment. Restart the affected process after changes; editing a file does not hot-reload a running deployment.

Launch

Agent-assisted managed deployment

On an installed station, start a fresh agent conversation in the checkout and ask to deploy a specific supported model/checkpoint and task, open RoboGUI, and leave preparation and execution to you. The deployment skill resolves existing recipes and station bindings. Missing physical camera mappings or an ambiguous checkpoint still require your input. This is a new conversation on a configured station; SDK installation, weights and calibration are separate setup work.

The managed path supports Linux + tmux, one runtime config, local XPolicyLab WebSocket serving and a camera-server recipe. It reuses existing service entry points and isolated Python environments. Config-list studies, remote model hosts and other serving frameworks continue to use the manual launch commands below.

# Read-only checks; no devices opened and no model weights loaded.
uv run manimux doctor \
  --config manimux/configs/experiments/pi05-latest300ep-step30000/coin/serial32.yaml \
  --local manimux/configs/local/station.yaml \
  --runtime-python envs/yam/.venv/bin/python \
  --policy-python XPolicyLab/policy/Pi_05/openpi/.venv/bin/python --json

# Start camera, RoboGUI, model and interactive runtime. Wait for manual Prepare.
uv run manimux up \
  --config manimux/configs/experiments/pi05-latest300ep-step30000/coin/serial32.yaml \
  --local manimux/configs/local/station.yaml \
  --runtime-python envs/yam/.venv/bin/python \
  --policy-python XPolicyLab/policy/Pi_05/openpi/.venv/bin/python --open --json

uv run manimux status --json
uv run manimux attach --service runtime  # detach: Ctrl-B, then D
uv run manimux down --timeout 30 --json

Substitute your recipe and installed interpreters. Camera defaults to the runtime interpreter; use --camera-python when needed. Use --output-dir to select a separate trial root. up --timeout is a readiness deadline per service. JSON results stay on stdout and progress goes to stderr. --open opens the default browser; agents can instead open the returned gui_url in their browser panel.

Private state, resolved plans and logs live under .local/deployment by default. Use the same --deployment-dir for all commands when overriding it. Repeated up reuses an unchanged healthy deployment owned by this tool. Changed configurations, partial deployments and external port owners are reported explicitly; no external process is adopted or stopped. Configuration source hashes are rechecked during startup. Restart after editing running service code. Preserve the state directory until its services have stopped.

Shutdown: down signals the owned runtime first and waits for its existing cleanup. If it remains alive, dependencies and the operator terminal stay available; the command reports shutdown_blocked. Attach to the terminal and inspect RoboGUI. There is no force-kill or automatic RELEASE. Repeated down waits without sending another interrupt. When runtime has exited, policy, camera and RoboGUI stop in order.

Rehearse and record the deployment demo

# Synthetic preview: no camera, policy service, runtime or robot connection.
uv run manimux up --demo --robot yam --gui-port 18086 \
  --deployment-dir .local/demo --open --json
uv run manimux status --deployment-dir .local/demo --json
uv run manimux down --deployment-dir .local/demo --json

# Terminal footage (requires VHS); run from the repository root.
mkdir -p recordings
vhs scripts/media/deployment-demo.tape

# Full RoboGUI window recording on an X11 desktop.
scripts/media/record-robogui.sh --duration 30

Demo uses isolated bridge/control ports and visibly labels the panel Synthetic preview · no hardware. It does not simulate a successful policy deployment or task. For the public film, capture a fresh conversation, real deployment progress, opening RoboGUI, manual Prepare/Warmup/Start, a continuous external robot view, and the resulting recording/replay. Keep the full take and label accelerated model-loading waits. Suggested 90-second edit: request 0–10s, deployment 10–25s, RoboGUI 25–40s, operator handoff 40–55s, robot execution 55–80s, recording/replay 80–90s. These are edit durations, not deployment latency guarantees.

Remotion's official skills can compose real footage, captions and transitions; VHS renders repeatable terminal clips. Neither is a runtime dependency. Keep generated video in the ignored recordings directory and use task results only when supported by the take.

Choose the workflow first: manage experiments or try once. An agent should ask when this is unclear, and reuse an existing answer for ongoing experiments.

To try an existing experiment variant, pass its YAML directly to run --config; no duplicate configuration or config list is needed. Use the same --config for camera and generic policy services. The GUI provides Prepare Run only. Use --config-list when formal scoring and progress tracking are wanted. The public serve command is removed. Camera/model servers.* remain separate services.

Current Pi05 / YAM study. Requires installed hardware/model environments, the checkpoint selected by the recipe, and a configured station. Run from the repository root in four terminals, using the same config list in cameras, model and runtime.

YAM fault shutdown: a camera/model failure stops task execution and requests zero Home only when robot connection and control health checks pass. Measured arrival is checked before opening grippers or closing motor control. If Home or feedback validation fails, the runtime retains control ownership and displays a persistent Hardware recovery panel in RoboGUI. Check Home path is clear and click Retry Home and finish, or physically support all arms and secure payloads, check the support confirmation and click Confirm support and release motor control. Release does not Home. There is no terminal input. Disconnection, timeout and stale requests never authorize release; without a GUI recovery channel, the process remains blocked. Do not kill that process or start another runtime while it owns the arms. This requires hardware validation and does not guarantee support after a drive fault or power loss.

YAM Home keeps the Home target after interpolation and waits without an arrival deadline. Position error and speed must stay within the release limits continuously for home_settle_duration_s: 0.25, configured in configs/embodiment/arm/yam.yaml. While waiting, logs report joint error and speed every five seconds. Hardware feedback checks remain active; loss of control health stops motion rather than being ignored. The retired home_settle_timeout_s setting in old snapshots is ignored with a warning. RoboGUI displays Returning Home, Saving rollout or Actuator release blocked while cleanup owns the robot; its cleanup heartbeat does not certify motor health. Runtime and dashboard changes require restarting both processes after safe release; a browser refresh alone does not load Python changes.

1 · Camera service

envs/yam/.venv/bin/python -m manimux.servers.camera.server \
  --config-list manimux/configs/lists/pi05-latest300ep-step30000.yaml

2 · RoboGUI

envs/yam/.venv/bin/python -m manimux.robogui.dashboard \
  --robot yam --host 127.0.0.1 --port 8086

3 · Pi05 model service

XPolicyLab/policy/Pi_05/openpi/.venv/bin/python -m manimux.servers.policy \
  --config-list manimux/configs/lists/pi05-latest300ep-step30000.yaml

4 · Runtime

envs/yam/.venv/bin/python -m manimux run \
  --config-list manimux/configs/lists/pi05-latest300ep-step30000.yaml

Open http://127.0.0.1:8086. run waits for GUI preparation; it does not launch the other three services. Prepare can connect hardware and move to the configured start pose. Use Prepare → adjust layout → Activate Warmup → Start rollout → Finish. After Finish, Prepare becomes available when the runtime is ready and evaluation is saved or skipped. A submitted Prepare stays pending until the runtime acknowledges it; repeated clicks do not create extra rollouts.

Recovery controls follow the connected runtime's declared capabilities and arm selections. The ordinary Finish & Home button explicitly requests Home before the runtime closes hardware, even if a legacy configuration has home_on_close: false. A separate idle Home capability must not suppress this request. Return Home keeps an active rollout open. Explicit Drag and Release transitions finish without homing and remain subject to the embodiment's release checks; they are not the ordinary Finish action. If actuator release is blocked, keep the owning process alive: do not release control or kill it while the arms are unsupported. Tianji Release offers continuous or five-second cooperative hand-guiding when declared; loss of the recovery lease cancels it.

Pack-plate deployments select tianji_taccap_pack_plate.yaml for the saved pack_plate_start Home. Update older bound configurations and restart the runtime to load a changed assembly.

Warmup is explicit when run.warmup_before_start: true; otherwise go directly to Start. Follow the selected Finish/Home behavior before stopping runtime, then stop model, camera and GUI processes.

For another compatible method, select it in RoboGUI. For another collection/model, replace terminal 3 with that model's environment and launcher; do not reuse Pi05's interpreter.

# XR1: matching standalone model recipe
"$MM_MODEL_PY" -m manimux.servers.xr1 --config "$MM_RECIPE"

# LingBot: shared XPolicyLab server, matching model recipe
"$MM_MODEL_PY" XPolicyLab/setup_policy_server.py --config-path "$MM_RECIPE"

# DP: recipe selected from the experiment
"$MM_MODEL_PY" -m manimux.servers.dp --experiment "$MM_EXPERIMENT"

# OpenWAM: matching standalone model recipe
"$MM_MODEL_PY" -m manimux.servers.openwam --config "$MM_RECIPE"

Shared parameters

Use this guide to configure Serial, asynchronous chunking, RTC, ACT temporal ensembling, AAC, PAINT, AutoHorizon and backward-only BID, including their supported combinations. Start from an experiment that already works with your robot and policy; keep its checkpoint, cameras, action representation and executor settings.

Configure an experiment

Set inference.algorithm and inference.inference_schedule explicitly. The examples below replace the experiment's entire inference section unless described as overrides. They assume a model horizon H = 50; adjust the example windows for your model's actual H. Do not shorten policy.horizon_policy_steps to change how long a chunk executes.

You can instead load an inference preset and override its settings:

inference:
  config: ../../../inference/yam_act_serial.yaml
  algorithm: act_temporal_ensemble
  chunk_policy_steps: 8

This relative path is for an experiment under manimux/configs/experiments/<task>/<model>/. config paths are relative to the experiment YAML. Nested mappings merge; omitted fields remain inherited. Each section loads one preset, without recursive config references. When switching methods, remove incompatible inherited limits, triggers and blending. Some presets contain only parameters, so select the algorithm explicitly.

The examples use policy.action_decoding: inline (the default). Verify that the deployment's adapter and action format meet the compatibility requirements below before enabling a combination. See configuration and deployment recipes for the rest of the experiment and launch procedure.

Request schedules and triggers

Serial waits for scheduled execution time, not measured arrival at the target. Reaching policy.timeout_s alone does not release a single-inflight request; the client must return a response or error. It also does not cancel a remote model call.

With multi_inflight, the server runs one model call at a time and replaces its waiting observation with the newest available one. It starts another call when a fresh observation is available; this is not parallel model execution. RTC/PAINT execution windows and ACT query intervals still limit submissions.

Hosted policy APIs: Dream-1

For a demo launched with manimux run --config ..., edit Task prompt directly in RoboGUI before Prepare Run. Prepare freezes the instruction for warmup and execution. After Finish and cleanup, edit it for the next rollout without restarting the runtime. Demo sessions hide layout and scoring controls, including Edit last evaluation. Formal --config-list studies retain their configured task instructions and evaluation workflow.

ManiMux can run a hosted API policy through its existing policy worker, asynchronous scheduler and executor, without a local model server or GPU. The first supported provider is policy.worker: dream_api. Other providers need an explicit observation/action adapter; arbitrary JSON responses are not robot commands.

Install the optional client with pip install -e '.[dream]'. Set DREAM_API_KEY in the runtime environment, never in configuration files. Dream-1 beta uses one current frame per camera (overhead and both wrists). Inference sends these images, state and task text to Dream Labs; the SDK's additional episode recording is disabled. A public 30 Hz serial example is included; see the README for startup commands. Tuned station-specific trial configurations are not distributed. Custom configurations use policy.worker: dream_api, the Dream YAM adapter and explicit playback, scheduling and execution parameters.

python -m manimux inspect --config /path/to/your/dream.yaml
python -m scripts.validation.dream_api_probe --config /path/to/your/dream-smooth.yaml --calls 3 --output /tmp/dream-api-probe.json

The probe uses synthetic observations and opens no robot or cameras. For operator-controlled deployment, use the same config with the camera server and manimux run --config ...; no policy_server is needed. The key must be present in the process that runs the policy worker. Every SDK response is saved before validation under policy.options.response_log_dir (default data/diagnostics/dream-api), including rejected responses. Each JSON record links raw actions to session/request, prompt, state, camera hashes and timing; errors identify the file and offending action values. Non-finite numbers use explicit strings and credentials are redacted. Finite gripper commands are clipped to [0,1], following the official Dream robot adapter. Raw values and affected rows remain in the diagnostic record; joint predictions are unchanged and non-finite output is still rejected. Only ordinary sampling is supported: RTC, PAINT and multi-inflight are not advertised. Smooth limits and filters commands but cannot fix stale predictions, network stalls or incorrect model trajectories. API and offline checks do not establish physical task success.

Time, rate and handoff parameters

Choose a refill threshold that leaves time for preparation, transport, inference and decoding. Outside refill mode, only the default 0.4 is tolerated and ignored; other explicit values, including null, are rejected. If a preset supplies another value, use a different preset or override it to 0.4 when changing triggers.

first_step_when_ready starts row zero when the result is accepted. Use it for the serial examples below. drop_infer_latency keeps the observation-based action clock and discards source rows whose times have passed before acceptance. Use it for the asynchronous examples. Neither setting changes the recorded observation timestamp. A late reply can have too few usable rows and be rejected.

Meaning of chunk_policy_steps

Step counts are model action rows, not control-loop ticks. The row interval dt is normally policy.action_dt_s. If policy.trajectory_duration_s is set, it overrides dt with trajectory_duration_s / (H - 1).

K rows span (K - 1) * dt between their first and last targets. Ordinary serial, serial ACT and BID additionally hold the last row for one interval, giving K * dt for K usable rows. AAC/AutoHorizon do not add that interval. Thus 12 rows at 30 Hz span about 0.367 s, or 0.4 s with the last-row hold; the control-loop rate does not change these times.

Use either chunk_policy_steps or its method-specific counterpart. Conflicting values are rejected; setting chunk_policy_steps: null does not clear an inherited counterpart. Remove/reset that field too. Do not carry max_chunk_policy_steps into a specialized method.

Related deployment controls

Algorithms

ACT here is temporal ensembling, not the ACT policy architecture. algorithm: manimux is removed; use async. Executors are configured separately.

Compatibility

“Yes” means a supported built-in runtime combination, subject to the requirements below; “—” means unsupported. ACT means temporal ensembling, not an ACT policy checkpoint.

The deadline resolutions also apply with ACT enabled. The final column adds ACT to the base method under each of its supported schedules; it does not enable other pairs such as RTC + PAINT.

Policy and decoding requirements

• Ordinary chunking and standalone ACT use the policy's default sampling mode. RTC, PAINT, AAC and AutoHorizon require the corresponding advertised sampling mode (rtc, paint, aac, autohorizon). BID also requires aac candidate sampling. Adding ACT does not change the base algorithm's sampling requirement. Check the model sampling matrix and the running server's capabilities; a YAML setting cannot enable a missing model hook.
• Process decoding supports ordinary async, serial, RTC and train-time RTC without ACT, using a compatible adapter and a non-streaming schedule. Serial preserves its complete selected prefix through inference and decoding. Multi-inflight requires inline decoding.
• Multi-inflight requires a streaming-capable client/server and adapter. The supplied path uses xpolicylab_ws, ManiMux's streaming server, and manimux.policy_adapter.joint:JointAdapter. An upstream server without the multi_inflight capability cannot run these configurations.
• BID, serial ACT, ACT added to another algorithm, and asynchronous AAC/AutoHorizon require complete absolute-joint trajectories with inline decoding. The supplied adapter is JointAdapter; do not pass delta-joint or end-effector actions directly to it. These combinations do not support dense IK or waypoint handoffs.
• Asynchronous AAC, AAC + ACT, and BID + AAC require policy.worker: xpolicylab_ws. Every use of AAC, including BID + AAC, needs an explicit robot.config and matching dual-arm end-effector increment statistics.
• Custom inference.strategy plugins do not support multi-inflight, continuous requests, serial ACT or ACT composition. A model recipe that needs its own history or decoding adapter cannot gain these combinations by replacing its adapter name.

Serial

inference:
  algorithm: serial
  inference_schedule: serial
  chunk_policy_steps: 12
  action_start_mode: first_step_when_ready
  blend_policy_steps: 0

Executes the selected prefix, including a full interval for its last row, before observing and requesting again. Omit chunk_policy_steps to execute the full chunk. algorithm: serial defaults to serial and requires that schedule.

Async

inference:
  algorithm: async
  inference_schedule: single_inflight
  request_trigger: refill
  refill_threshold_s: 0.4
  action_start_mode: drop_infer_latency
  blend_policy_steps: 0

This uses the full available prediction and refills while actions are still executing. Optionally add chunk_policy_steps to cap its source prefix. Change only inference_schedule to deadline for deadline admission, or use the multi-inflight example below. The defaults are async and deadline. With only inference_schedule: serial, the algorithm defaults to serial. The old algorithm: manimux selector is no longer accepted.

Replan after source progress

inference:
  algorithm: async
  inference_schedule: single_inflight
  request_trigger: progress
  chunk_policy_steps: 16
  action_start_mode: drop_infer_latency
  blend_policy_steps: 0

With progress, the shared chunk_policy_steps field is a replanning threshold, not a source-prefix cap. Retain the full model prediction; request a new chunk when source progress reaches 16 policy steps, including rows already dropped for observation-to-commit latency. This matches the RTC/PAINT source clock and does not promise 16 newly executed rows after each handoff.

For playback from the first returned action, use action_start_mode: first_step_when_ready. With an uncropped prediction, the 16-step threshold is counted from activation: at 10 Hz the next request starts after 1.6 seconds. The previous trajectory continues during inference. This submission threshold is separate from min_plan_hold_policy_steps, which only delays replacement by an already returned prediction.

The first request is immediate. Later requests wait for both the progress threshold and an available worker. Continue the old trajectory while inference runs, then trim the new result's expired prefix and switch on its action clock. Rejected results do not reset progress. The threshold must be smaller than the model horizon. Omit max_chunk_policy_steps and refill_threshold_s; no waypoint handoff or handoff skip is supported. Cross-chunk blending remains separate.

ACT temporal ensemble

inference:
  algorithm: act_temporal_ensemble
  inference_schedule: single_inflight
  chunk_policy_steps: 1
  action_start_mode: drop_infer_latency
  blend_policy_steps: 0
  temporal_ensemble:
    coefficient: 0.01

coefficient must be non-negative: zero weights overlapping predictions equally; positive values favor older contributing predictions. The default is 0.01. No temporal_ensemble.enabled flag is needed for this algorithm.

Standalone asynchronous ACT permits a positive blend_policy_steps, which adds a separate handoff blend. Keep it zero to use only temporal ensembling.

Serial ACT with a configurable execution horizon

inference:
  algorithm: act_temporal_ensemble
  inference_schedule: serial
  chunk_policy_steps: 5
  action_start_mode: first_step_when_ready
  blend_policy_steps: 0
  temporal_ensemble:
    coefficient: 0.01

Full predictions contribute at their actual execution times, including inference gaps. Set K below H to leave potential overlap; K = H or long gaps may leave only the current prediction. query_interval_policy_steps is unused in serial mode. Keep blending and handoff skip at zero.

RTC

inference:
  algorithm: rtc
  inference_schedule: single_inflight
  chunk_policy_steps: 12
  action_start_mode: drop_infer_latency
  blend_policy_steps: 0
  rtc:
    initial_delay_policy_steps: 4
    delay_buffer_size: 10
    beta: 5.0

The rtc values above are the defaults:

• initial_delay_policy_steps initializes latency in model steps; require 2 * initial_delay_policy_steps <= H.
• delay_buffer_size sets the number of recent samples whose maximum predicts delay.
• beta controls conditioning strength and must be finite and positive.
• condition_source defaults to canonical (decoded model rows). timeline samples the accepted, blended trajectory on the observation clock. executor forecasts Smooth commands from its current filter history on the same clock, including recent issued commands. It requires standalone RTC and a stateless continuous/curve gripper without feedback guards. Unavailable tail rows receive zero weight; a missing delay prefix disables conditioning for that request. Forecasts assume the current plan remains active and nominal control cadence; they do not predict measured robot motion. Row skipping is unsupported for timeline/executor conditions.
• initial_delay_policy_steps: null initializes from pre-Start warmup samples. To use that workflow, enable run.warmup_before_start: true, choose Activate Warmup after Prepare in RoboGUI, and use inline decoding. Without calibration samples, the initial estimate is zero. See the preset guide for warmup controls.

SAPolicy can separately select policy.adapter.ik_seed_source: previous_plan to seed IK from the accepted joint trajectory at the first decoded source row's timestamp. If that timestamp has no accepted reference, it uses the runtime seed. Subsequent rows continue from the previous IK solution. This leaves measured model observations unchanged and does not impose a joint-step limit or guarantee IK branch continuity.

Train-time RTC

Prefix-trained UMI-DP checkpoints use algorithm: train_time_rtc and inference_schedule: single_inflight. They advertise the separate train_time_rtc sampling capability; ordinary RTC guidance checkpoints do not support this mode.

The pack-plate recipe fixes five leading action rows with delay_estimator: fixed and fixed_prefix_steps: 5. The first request and unavailable committed prefixes are unconditioned. Fixed prefixes do not guarantee latency coverage. Dynamic decode_max and end_to_end_max estimators are also available.

Use action_start_mode: drop_infer_latency, blend_policy_steps: 0 and handoff_skip_steps: 0. Both inline and process decoding are supported. Select manimux/configs/experiments/umi_dp-tianji-pack_plate-train_time_rtc/pack_plate/train_time_rtc.yaml.

ABC-DiT 200k uses this strategy with chunk_policy_steps: 16, delay_estimator: fixed and fixed_prefix_steps: 7. Its model recipe retains rtc_prefix_length: 4: the backend conditions on the final four of the seven fixed rows, then preserves the complete seven-row lead in the returned trajectory. Select the existing ABC RTC experiment YAMLs; their strategy and wire mode are now train_time_rtc, not PAINT. The checkpoint, 30-step horizon, action frames and direct executor are unchanged.

See the timing contract and deployment reference.

PAINT

inference:
  algorithm: paint
  inference_schedule: single_inflight
  chunk_policy_steps: 12
  action_start_mode: drop_infer_latency
  blend_policy_steps: 0
  paint:
    initial_delay_policy_steps: 4
    delay_buffer_size: 10

Use a positive initial delay d and execution/query window K satisfying d <= K <= H - d. The defaults are d = 4, K = 10 and a delay buffer of 10 samples. Unlike RTC, PAINT does not accept null for its initial delay. Its delay estimate must cover observation age, queueing, transport, inference and decoding; higher observed delays can make the window infeasible. Select K and d for the actual model horizon and measured latency.

Warmup uses the selected configuration's scheduler, sampling algorithm, decoder, IK and executor on an isolated preview timeline. Serial waits for its configured prefix; asynchronous and conditioned methods use the same progress triggers as rollout. Computed actions are checked but never sent to the robot, which continues holding its measured pose. Real cameras and feedback are used; this is not a simulated task trial. RTC/PAINT Start waits for one accepted ordinary plan, two accepted conditioned plans, and backend reset acknowledgement. The first call per branch is displayed separately and excluded from latency calibration. Plan E2E includes decoding; Step IK/executor work runs separately on control ticks. Compilation that exceeds the configured plan deadline is shown as a rejected warmup plan and must be retried after warming.

AAC

inference:
  algorithm: aac
  inference_schedule: serial
  action_start_mode: first_step_when_ready
  blend_policy_steps: 0
  aac:
    num_samples: 20
    motion_threshold: 0.2
    ee_stats_path: manimux/policies/xpolicylab/norm_stats/yam_60ep_ee_increment.json
    chunk_id_selector: "0"
    backward_beta: 0.99

This example uses YAM statistics and a YAM preset threshold. Replace them for a different embodiment or dataset. Relative ee_stats_path values resolve from the launch working directory, unlike config; run from the repository root for this example or use an absolute path.

AAC's "backward" selector is distinct from algorithm: bid_backward.

AutoHorizon

inference:
  algorithm: autohorizon
  inference_schedule: serial
  action_start_mode: first_step_when_ready
  blend_policy_steps: 0

The policy chooses the execution length; omit fixed chunk limits. There is no runtime inference.autohorizon block. Optionally configure the selector in the experiment's existing policy_server section:

policy_server:
  # Keep the existing model recipe/config reference and add:
  autohorizon:
    hold_thr: 0.3
    run_len: 1
    max_entropy_q: 0.9

Omitted fields retain their defaults. Attention is always taken from the third denoising step; the model needs at least three denoising evaluations. These settings apply to serial and supported asynchronous AutoHorizon schedules.

Overrides require the ManiMux websocket wrapper and a supported Pi05 JAX, XR1, LingBot-VLA2 or OpenWAM AutoHorizon backend. Restart the policy server after changing them. Pi05 loads the experiment with python -m manimux.servers.pi05 --experiment path/to/experiment.yaml in its model environment. With a standalone server recipe, put autohorizon at the recipe's top level and launch through the ManiMux wrapper; direct XPolicyLab/setup_policy_server.py does not apply these overrides. See the model recipe and sampling matrix for backend constraints.

BID backward-only

inference:
  algorithm: bid_backward
  inference_schedule: serial
  chunk_policy_steps: 5
  action_start_mode: first_step_when_ready
  blend_policy_steps: 0
  bid:
    num_samples: 16
    rho: 0.9
    execution_horizon: fixed

Uses candidates from one policy and BID's backward criterion; no forward criterion or second policy is used.

• num_samples: integer at least 2, default 16.
• rho: backward score decay, in (0, 1], default 0.9. Smaller values emphasize earlier overlapping action times more strongly.
• execution_horizon: fixed (default), using K, or aac as shown below.
• Fixed K must be smaller than H. Keep handoff_skip_steps: 0.

Serial BID requires first_step_when_ready. Fixed-horizon asynchronous BID accepts either action-start mode; the examples use drop_infer_latency to discard expired rows. Asynchronous scoring compares overlapping future action times; its K counts future rows remaining after latency alignment, not already-expired source rows.

Supported combinations

Choose one base algorithm. Add ACT with temporal_ensemble.enabled, or select AAC's execution horizon through bid.execution_horizon. Merely adding an rtc, paint or aac parameter block does not activate a second algorithm.

What ensemble adds

Standalone algorithm: act_temporal_ensemble already performs temporal ensembling. The optional combination wraps another algorithm, such as RTC, with the same kind of output fusion. It uses predictions from different inference requests for the same future action time. It does not combine measured state with actions, call another model, or add another executor.

RTC only:
Scheduler → RTC request/condition → model → adapter → Timeline → executor

RTC + ACT:
Scheduler → RTC request/condition → model → adapter → ACT ensemble → Timeline → executor

BID + ACT:
model candidates → adapter decodes each → BID selects one → ACT ensemble → Timeline → executor

The factory creates EnsembleStrategy(base_strategy, config) only when the combination is enabled. The base strategy still constructs requests. The wrapper aligns and averages overlapping predictions before Timeline commit. Reset clears both the base strategy and ensemble history.

RTC + ACT keeps RTC conditioning and adds one output ensemble. It requires blend_policy_steps: 0 so handoff interpolation is not added as well. This is a supported combination, not evidence that it improves task success. The next RTC condition uses the accepted fused timeline. No arbitrary stack of fusion algorithms is configured here.

Add ACT to RTC, PAINT, BID, AAC or AutoHorizon

For example, RTC + ACT using a preset:

inference:
  config: ../../../inference/yam_rtc.yaml
  algorithm: rtc
  inference_schedule: single_inflight
  blend_policy_steps: 0
  temporal_ensemble:
    enabled: true
    coefficient: 0.01

Disable the added ensemble: keep the base algorithm and set the flag to false in the experiment YAML. These are overrides to the same RTC experiment:

inference:
  algorithm: rtc
  temporal_ensemble:
    enabled: false

The combination flag defaults to false. Explicit false also overrides an enabled referenced preset. After changing YAML, finish the current rollout and restart manimux run --config-list ... before preparing the next trial. The same model server can stay running for this flag change. For a controlled comparison, leave the other settings unchanged.

coefficient: 0 still averages predictions with equal weights; it does not disable ensemble. A positive coefficient follows ACT's oldest-to-newest exponential weighting and gives older overlapping predictions more weight. Changing enabled to false does not disable the standalone act_temporal_ensemble algorithm; select another base algorithm to remove it.

Apply the same temporal_ensemble block to PAINT, BID, AAC or AutoHorizon. Require inline complete absolute-joint decoding, blend_policy_steps: 0, handoff_skip_steps: 0, and the default handoff: blend. RTC/PAINT + ACT require drop_infer_latency; serial combinations use first_step_when_ready.

The base algorithm retains its query/execution timing; ACT's query_interval_policy_steps is unused in these combinations. Only overlapping action times contribute, so serial AAC/AutoHorizon may have no effective overlap.

Which prediction rows participate

Prediction length and execution length are different. A model may predict 50 steps while this request executes only 5. The base strategy declares ensemble_scope in code; this is a developer contract, not another operator YAML setting.

Only future, overlapping timestamps contribute in either mode. Cropping uses source-row indices; expired rows are still removed by Timeline. The ensemble wrapper reads this declaration rather than matching algorithm names. Keep the chosen scope fixed for a strategy instance.

For agents and strategy developers

Use the optional protocols in manimux/runtime/inference.py when extending the existing strategy path. Do not introduce another control loop or move model sampling into the executor.

Asynchronous adaptive horizons

To use AAC or AutoHorizon with single-inflight, keep the selected method's parameters and replace its serial settings with these overrides:

inference:
  inference_schedule: single_inflight
  request_trigger: refill
  refill_threshold_s: 0.4
  action_start_mode: drop_infer_latency
  blend_policy_steps: 0

Merge these overrides into the base example. They also work for BID and BID + AAC. Asynchronous AAC, AutoHorizon and BID + AAC require drop_infer_latency, zero blending/skip, and the decoding/client requirements above. ACT may also be added.

For an adaptive horizon of e source rows, if d rows expire before acceptance, only the remaining e−d rows are eligible. When d reaches e, the reply is rejected. The runtime does not extend the selected horizon to compensate for latency; short horizons and slow inference can therefore cause holds.

Multi-inflight and continuous requests

Ordinary asynchronous chunking with continuous observation uploads:

inference:
  algorithm: async
  inference_schedule: multi_inflight
  request_trigger: continuous
  observation_hz: 30.0
  action_start_mode: drop_infer_latency
  blend_policy_steps: 0

For BID, AAC or AutoHorizon, keep its algorithm and method parameters and apply the other settings in this example, including action_start_mode. For refill-based uploads, use request_trigger: refill and a positive refill_threshold_s instead. continuous can also be used with single_inflight or actual deadline scheduling.

RTC, PAINT and standalone ACT require request_trigger: algorithm. For example, RTC + ACT + multi-inflight:

inference:
  config: ../../../inference/yam_rtc_multi_inflight.yaml
  algorithm: rtc
  inference_schedule: multi_inflight
  request_trigger: algorithm
  observation_hz: 30.0
  temporal_ensemble:
    enabled: true
    coefficient: 0.01

For standalone RTC, PAINT or ACT, start from its own individual example and set inference_schedule: multi_inflight, observation_hz, and request_trigger: algorithm. Keep its algorithm and method parameters. Streaming PAINT requires drop_infer_latency and zero blending/skip. Add ACT to RTC/PAINT as shown above.

RTC/PAINT replies can be rejected when the plan they were conditioned on has already been replaced. PAINT also rejects replies that arrive beyond the usable conditioned prefix. A higher upload rate can increase such rejections; it does not remove model latency or guarantee uninterrupted execution.

Minimum plan hold time for continuous inference

Primary use: continuous multi-inflight inference, where completed predictions can replace the executing plan in quick succession. Continuous single-inflight can also benefit when inference completes rapidly. This advanced, opt-in setting limits how soon a plan may be replaced; it does not limit request frequency, change robot control frequency, or select the model prediction horizon.

For ordinary Serial or RTC workflows, keep this setting at zero unless you specifically need an additional replacement delay. Existing chunk_policy_steps and rtc.min_execute_policy_steps retain their algorithm-specific execution/request roles. Adding a hold to RTC may be redundant and can make a buffered response expire or invalidate its conditioned prefix.

By default, both single-inflight and multi-inflight replace the current plan as soon as a valid reply can be committed. To let each accepted plan execute for a minimum time while inference continues, set a positive integer inference.min_plan_hold_policy_steps. For example:

inference:
  algorithm: async
  inference_schedule: multi_inflight  # also supports continuous single_inflight
  request_trigger: continuous
  observation_hz: 30.0
  min_plan_hold_policy_steps: 4
  action_start_mode: drop_infer_latency
  blend_policy_steps: 0

The minimum starts at the accepted plan's execution start, including the start of a command_cubic blend when enabled. Four policy intervals of 20 ms give an 80 ms minimum; this does not change the model horizon or control rate. The wait ends early if the plan's available trajectory ends. The first plan starts without this wait. If no valid replacement is available when the minimum ends, the current plan continues until its remaining actions run out.

During the wait, the runtime retains the newest eligible completed reply and replaces it when a newer one arrives. Waiting does not occupy the single-inflight request slot or restart the minimum. Requests still follow the configured trigger and rate limit; continuous in this example keeps requesting throughout the wait. At handoff, decoding, candidate selection and latency trimming use the handoff time. With drop_infer_latency, rows that expired during the wait are discarded; first_step_when_ready deliberately starts row zero at acceptance.

This option requires policy.action_decoding: inline and a resolved single_inflight or multi_inflight schedule. It also works with their supported algorithm combinations; keep each method's required trigger and other constraints. Request deadlines and max_plan_age_s still apply, and Pause/Home clears the waiting reply. RTC/PAINT reject replies conditioned on a replaced plan; PAINT additionally checks that the handoff stays within its anchored prefix. Extra waiting can therefore cause rejections. Allow for it when choosing delay estimates and time limits.

This setting controls replacement timing. rtc.min_execute_policy_steps controls when RTC requests its next prediction, while chunk_policy_steps retains its algorithm-specific meaning. Leave the new setting at 0 to keep the existing behavior.

BID with AAC's execution horizon

Use the dedicated preset to avoid inheriting a fixed K:

inference:
  config: ../../../inference/yam_bid_aac.yaml
  algorithm: bid_backward
  inference_schedule: serial
  bid:
    execution_horizon: aac
    num_samples: 16
    rho: 0.9

The preset supplies serial scheduling, first_step_when_ready, zero blending, YAM EE statistics and AAC's motion threshold. Override aac.ee_stats_path and aac.motion_threshold for your deployment as described in the AAC section.

bid.num_samples sets the shared batch size. AAC's sample-count/selector settings and BID's fixed replan length do not control this combination; leave them at defaults. Omit chunk_policy_steps, or set it to null to clear an inherited fixed K. Apply the asynchronous overrides above for single-inflight or multi-inflight; deadline also works. To add ACT, use the earlier temporal_ensemble block.

Unsupported combinations

• RTC + PAINT, and RTC/PAINT with BID or AAC candidate selection.
• AutoHorizon with RTC, PAINT, BID or AAC sampling/horizon selection.

Further configuration

For model-specific sampler options, use the sampling matrix. For executor smoothing, limits and IK, use execution and IK. Algorithm sources and derivations are kept in the reproduction records; advanced decoder/waypoint configuration is covered by the runtime configuration reference.

Support

BID requires AAC candidate sampling plus complete absolute-joint decoding. Adding ACT or asynchronous adaptive horizons also requires the decoding and client constraints in the compatibility table; a model hook alone is insufficient.

This matrix covers the current YAM adapters, not every checkpoint or embodiment. “Hook” means an implementation path; the loaded server's HELLO.capabilities.sampling_modes decides availability. It does not establish task performance.

OpenWAM specialized modes require a single-stream, full-horizon backend and CFG scale 1. RTC permits compilation and can opt into DiT caching. Compiled RTC has synthetic CPU regression coverage; real-checkpoint CUDA validation is still required before claiming correctness or speed on a deployment. Existing eager experiments keep their recorded settings; compilation comparisons need distinct variants. PAINT/AutoHorizon currently require eager execution and remain uncached; AutoHorizon also requires the joint-attention driver. LingBot conditioned/attention modes require the native horizon; attention capture needs eager execution. Pi05 PyTorch and Pi05 EEF are outside this matrix. Detailed model contracts

Generate selecting configs from one existing deployment, without launching services:

PYTHONPATH=. envs/yam/.venv/bin/python scripts/experiments/build_inference_matrix.py \
  --base manimux/configs/experiments/pi05-yam-coin-joint-step50000/coin/rtc_joint_step50000.yaml \
  --output-dir data/configs/coin-pi05-comparison-v1 \
  --execution-steps 16 --paint-delay-steps 4 --async-trigger progress \
  --aac-stats manimux/policies/xpolicylab/norm_stats/yam_60ep_ee_increment.json \
  --methods serial async act rtc paint aac autohorizon

The generator preserves the base checkpoint, adapter and executor, skips unsupported DP methods, validates configs, and refuses to overwrite outputs. The AAC stats must exist and be suitable for the study; omitting AAC removes that requirement. Generated values are starting presets, not a frozen protocol.

--async-trigger progress uses K as a source-progress replanning threshold and retains the full prediction. --async-trigger refill retains the generator's older behavior: cap the source prefix to K and refill with K action intervals remaining. Choose explicitly when comparing execution windows; equal K does not imply equal timing. Progress requires the built-in Async strategy with single-inflight scheduling.

Use each selected experiment's actual policy_server recipe; changing only runtime algorithm cannot add a missing sampler hook. Historical runbook measurements apply to their recorded revisions, not automatically to a new checkpoint.

Config lists & task switching

A config list is the entry point for a study, including a single experiment. It selects reusable variants, the trial plan and one recording root. Camera, model and runtime commands use the same config list and station. Compatible task/method choices switch in RoboGUI after Finish, recovery and evaluation.

Configuration ownership

Published example: only experiments/pi05-latest300ep-step30000/ and its config list are tracked in this checkout. Other study directories and config lists are local and Git-ignored. Shared components and task definitions remain reusable. Update both allowlists in .gitignore explicitly when publishing another study.

manimux/configs/
  lists/<study>.yaml                 # entries, trials, recording.root
  tasks/<task>.yaml                        # prompt, layouts, rubric, optional pose
  embodiment/{robot,sensor,initial_poses}/  # shared hardware and named poses
  policy/  inference/  executor/  local/
  experiments/<model-checkpoint>/
    _base_.yaml                            # common choices, only when needed
    <task>/rtc32_beta10.yaml                # reusable variant and identity

_base_ accepts a file or list of files. Later bases override earlier ones, then the variant overrides its bases. Mappings merge; lists replace. References are relative to the file declaring them. Cycles and inheritance chains longer than four files are rejected. Prefer one shared base plus one task.

# lists/my-study.yaml
id: my-study
recording:
  root: /data/robot-experiments             # user chooses; relative paths use this file's directory
experiments:
  - id: coin_rtc
    config: ../experiments/my-policy/coin/rtc32_beta10.yaml
    label: RTC beta 10
    trials:
      layout_ids: ['01', '02']              # subset of the task's available layouts
      repeats: 3

# experiments/my-policy/coin/rtc32_beta10.yaml
_base_: [../_base_.yaml, ../../../tasks/coin.yaml]
experiment:
  task: coin
  variant: rtc32_beta10
inference:
  algorithm: rtc
  rtc:
    min_execute_policy_steps: 32
    beta: 10

The agent first establishes single-task versus multi-task scope, checkpoint/contract, observable success and optional partial scores, layouts/reset, stopping rules and storage. It writes the task, variants, config list and private station from the answers and validates them. Users do not need to fill YAML forms. Do not infer a different user's scoring weights, trial count or joint coordinates.

Portable paths and validation

Shared recipes use ${paths.checkpoints}/... for model artifacts and ${repo}/... for repository resources. Additional roots, such as paths.sapolicy_workspace, belong in the private station. Relative station roots resolve beside that file. Config list and standalone recipe readers use the same expansion; a missing root is an error, not a guessed path.

Unknown public configuration fields fail with their section and a spelling suggestion. Plugin-owned options remain extensible and are validated by their owner. Use python -m manimux inspect --config-list <list.yaml> --local <station.yaml> to inspect resolved settings and output paths without starting services. Unresolved historical deployments live under ignored configs/local/archived_experiments/. The matrix generator preserves model cache/compile settings and leaves recording roots to config lists.

Initial pose and Home

Named YAM poses live in embodiment/initial_poses/yam.yaml: home, raised_ready and table_ready. A task or variant selects initial_pose: {config: …, id: …}. Pi05's shared base selects Home; the shared tasks/pick_the_moving_cube.yaml selects raised_ready and enables startup motion for all current YAM model variants of that task. The joint values and 5-second transition are defined once in the pose library. SAPolicy deployment variants explicitly select Table ready; the policy does not choose the robot start.

The robot assembly declares its preset library with initial_poses: ../initial_poses/yam.yaml. RoboGUI reads validated presets from this library for both --config trials and --config-list studies; the menu is independent of the listed methods. A task's explicit initial_pose.config can select a different library. Only configured, dimension-checked poses are offered; another embodiment must provide its own verified coordinates.

Selecting Initial pose → Home changes the next Prepare configuration without moving hardware. Selection is available only after rollout cleanup, evaluation and recovery are complete. Within a task, changing methods preserves an operator-selected pose when available; changing tasks restores the new task's default. Nondefault poses record under <variant>__pose_<pose-id>, with separate progress and a frozen initial target. Selecting the default again returns to the original variant's records. No extra experiment YAML is required.

YAM startup motion requires robot.options.execute: true and move_to_start_on_connect: true. After all arms connect and pass health checks, both arms move to their selected initial targets concurrently over the configured duration, using the same peer-cancellation mechanism as Home. Device connection and gripper calibration may still happen sequentially before this motion begins. Do not copy these driver options to a different embodiment without a supported motion implementation. Return Home remains a robot-owned operation, separate from the experiment's starting pose. Records distinguish the configured initial target, measured state and actual Home target.

Recording identity and historical data

A session created before Prepare may contain only a manifest. Such empty startup sessions do not prevent configuration fixes; their manifests are retained unchanged. Once any rollout record exists, including a partial rollout, condition checks apply.

<recording.root>/<list.id>/<experiment.task>/<experiment.variant>/
  session-<timestamp>-<unique-id>/
    session-manifest.json
    rollout-001/...

Restarting the same config list resumes progress from the same destination. Selecting an entry in RoboGUI and using --experiment-id resolve to the same identity and directory. Each launch/selection creates a session; repeats create rollouts, not new YAML files. The same variant launched through run --config is an unscored trial. It does not load study history or consume repeat slots. Trial recordings default to data/trials/<task>/<variant>/session-…; --output-dir selects another trial root. For standalone configs without experiment identity, the directory uses the filename and a hash of its absolute path. If configured, run.output_dir replaces data as the parent of trials/.

Session manifests freeze the resolved configuration, component/base hashes and config list identity. Changed scientific conditions under an already recorded variant ID are rejected: create a clearly named variant. Labels and trial budgets can change without pretending that the model or algorithm changed.

Existing recordings keep their paths, raw contents and recorded configurations. A config list entry may declare history: [old-study/step32/coin/rtc_beta10], relative to recording.root. Only sessions with matching saved effective conditions count toward current progress. Canonical paths deduplicate aliases. Finalized, valid scored experiments count; partial, unlabelled and invalid attempts remain distinguishable. Historical 16-step data are preserved for analysis and never relabelled as 32-step trials.

inspect --config-list … --index refreshes the derived recordings-index-v2.json inside the config list's recording directory. It inventories real session directories without following aliases. It is a snapshot, not the score authority; labels are read from recordings. The earlier recordings-index.json and migration/deletion receipts remain historical records. Index generation never rewrites raw episode files.

Delete recordings and retest

At the bottom of RoboGUI, expand Experiment records to browse the selected Task/Method output as sessions, rollouts and files. Filter by layout or session, select completed rollouts, then choose Delete selected…. The confirmation shows the exact paths and asks for a reason. Deletion moves the entire rollout, including its evaluation, videos and trajectories, into _archive/deleted; it releases the scored repeat slots and refreshes progress. Undo last deletion restores the last batch without overwriting newer recordings.

Deletion is available only while the runtime is idle and evaluation is complete. Active, saving and partial recordings cannot be deleted here; imported history remains read-only. Session manifests stay in place. Small .deleted markers reserve rollout IDs so retests receive new IDs. Keep deletion receipts and exclude _archive from subsequent analyses. Changing a score uses Edit last evaluation; deletion is for explicitly discarded trials.

Current Pi05 study: five tasks, 32-step fixed windows

Use manimux/configs/lists/pi05-latest300ep-step30000.yaml. It contains five tasks × six methods; moving-cube entries use the shared raised starting pose. Pi05 fixed windows use 32/50, Direct and blend=0; RTC beta=10. AutoHorizon remains adaptive. Async uses request_trigger: progress and retains the full model horizon. Async/RTC/PAINT source progress includes latency-trimmed rows; this is not a guarantee of 32 physical commands.

For this comparison, LingBot uses 32/50 source steps and OpenWAM uses 20/32. Subsequent Xiaomi Robotics-1 trials use an explicitly selected 18/30 window (2026-10-20), replacing the previous rounded 19/30 window under separate recording identities. These are study choices, not universal defaults for new users. Xiaomi's chunk-IK and step-IK config lists both include Serial, Async, RTC and PAINT. FastWAM has no bound YAM config list.

For Xiaomi/YAM, policy.adapter.gripper_output: clip explicitly declares the adapter's gripper handling: reconstruct absolute opening from the observation and predicted delta, then clip to [0, 1] for both chunk and step IK. Non-finite inputs and reconstructed openings fail. Chunk metadata records gripper_clip_max_abs. Raw observations, predicted deltas, pose targets and IK constraints remain unchanged. Current local experiment variants use _grip_clip to separate earlier tolerance-based recordings.

cd /home/ubuntu/manimux
envs/yam/.venv/bin/python -m manimux inspect \
  --config-list manimux/configs/lists/pi05-latest300ep-step30000.yaml --index
envs/yam/.venv/bin/python scripts/experiments/check_step_ratio.py \
  --config-list manimux/configs/lists/pi05-latest300ep-step30000.yaml

Start each long-running command in its own terminal after checking existing services:

# Camera service
cd /home/ubuntu/manimux
envs/yam/.venv/bin/python -m manimux.servers.camera.server \
  --config-list manimux/configs/lists/pi05-latest300ep-step30000.yaml

# RoboGUI
cd /home/ubuntu/manimux
envs/yam/.venv/bin/python -m manimux.robogui.dashboard \
  --robot yam --host 127.0.0.1 --port 8086

# Pi05 model service
cd /home/ubuntu/manimux
XPolicyLab/policy/Pi_05/openpi/.venv/bin/python -m manimux.servers.policy \
  --config-list manimux/configs/lists/pi05-latest300ep-step30000.yaml

# Runtime: Task / Method / Layout selection
cd /home/ubuntu/manimux
envs/yam/.venv/bin/python -m manimux run \
  --config-list manimux/configs/lists/pi05-latest300ep-step30000.yaml

Open RoboGUI. Choose Task → Method → Layout → Prepare Run / Prepare Experiment → Activate Warmup → Start → Finish → save/skip evaluation. Prepare may move to the configured initial pose. The moving-cube task defaults to the measured Raised ready preset across the current Pi05, Xiaomi, LingBot and OpenWAM config lists. Other tasks retain their configured starts. Earlier Home-start Pi05 moving-cube variants remain available for historical reproduction but are excluded from the active list. Changed starts use distinct recorded variant IDs; existing recordings are not rewritten. General tasks may inherit an initial_pose: {config: …, id: …} from their task YAML; overrides belong to deliberate variants.

Other config lists and service selection

The following config lists are local examples, not included in a fresh clone. When present on your station, use lists/lingbot-vla2-bottles30k-all-tasks.yaml, openwam-bottles30k-all-tasks.yaml, openwam-bottles30k-cached-all-tasks.yaml, openwam-bottles30k-rtc-cache-all-tasks.yaml or xiaomi-xr1-bottles30k-ratio64-all-tasks.yaml as appropriate. These bottle-trained weights are being evaluated across tasks; inclusion is not evidence of training coverage or successful task performance.

The generic XPolicyLab launcher is python -m manimux.servers.policy --config-list …, using that model's installed Python environment. --check prints the resolved service recipe without loading weights. Camera --check resolves devices without opening them. Pi05's dedicated launcher additionally validates its artifact paths.

YAM IK position acceptance: Set options.pos_threshold in manimux/configs/embodiment/arm/yam.yaml: 0.0001 means 0.1 mm (solver default), 0.001 means 1 mm, and null disables final position-error acceptance. This applies to chunk and step IK across model adapters using that arm definition. Disabling acceptance retains position optimization, orientation acceptance, finite-value checks and joint limits; it does not guarantee faster solving or bound the remaining position error. Step IK retains its computation budget. Restart the runtime after finishing the current trial to load a changed setting.

LingBot-VLA2 RTC withdrawn (2026-10-18). RTC is removed from the model capability list and all LingBot experiment entries. Measured conditioned sampling averaged 961.32 ms, exceeding the 0.60 s remaining-action budget at H=50, K=32 and 30 Hz. Historical RTC-labelled records include unconditioned fallbacks and must not be treated as a validated RTC baseline. Normal inference, PAINT and AutoHorizon retain their separate paths. Running processes must be restarted normally to load this removal; editing files does not update an already loaded model.

This station's multi-task LingBot-VLA2 checkpoint uses manimux/configs/lists/lingbot-vla2-latest300ep-step30000.yaml, with envs/lingbot-vla2/.venv/bin/python for the model service. It binds yam_latest300_micro8_v5_step30000 on the external drive, checkpoint-matched normalization and the existing relative-joint deployment projection. The main list contains five tasks × four fixed-window methods with use_compile: true and 32/50 windows. Serial is the initial selection; no --experiment-id is needed. Compiled PAINT and AutoHorizon are selected by manimux/configs/lists/lingbot-vla2-latest300ep-step30000-compiled-samplers.yaml (five tasks, separate recording variants). AutoHorizon receives attention as an explicit tensor from the third model evaluation; selection runs outside compilation. The old eager list is retained for historical reproduction. Start the model service and runtime with the same selected list after stopping the previous run normally. Warm each algorithm separately and check its measured latency and numerical diagnostics; a compile flag alone is not a performance guarantee. Compiled variants have an _compile recording suffix, and use_compile is part of the model handshake identity. Configuration and live HELLO checks passed, followed by finite real-checkpoint CUDA ordinary/PAINT outputs on synthetic observations; this is not a hardware trial or a matched speedup benchmark. Compilation happens lazily during warmup. The first ordinary call in the probe took 86.52 seconds, so policy.options.warmup_request_timeout_s: 300 permits display-only warmup to finish while the ordinary request timeout remains 20 seconds. A started server alone does not establish a warmed sampler. Moving-cube starts at Raised ready, other tasks at Home. Records use data/experiments/lingbot-vla2-latest300ep-step30000/, separately from the bottle-trained checkpoint.

PAINT waiting behavior: PAINT can keep running when inference exceeds the old chunk's remaining duration. It conditions the next prediction on the available suffix followed by a repeated terminal target, including the gripper. The runtime holds the last command until a valid result arrives, then applies the usual latency trimming. It does not reduce the configured execution steps or switch algorithms. Recorded metadata includes forecast and elapsed terminal-hold policy steps. Real request failures retain the common recovery path; a delay consuming the entire new prediction horizon still fails explicitly. The terminal-hold extension is covered by offline tests, not a new hardware validation.

OpenWAM cached Serial/Async uses the original compiled DiT cache recipe. The RTC-cache config list uses eager execution and the declared cache modes. Specialized sampling and compilation support remain model-specific; retain these flags and measured latency when comparing methods.

Try one existing configuration

No study YAML is required. Reuse a concrete variant, or let the agent write one runtime configuration from your robot, model and task requirements. Layouts, repetition budgets and a scoring rubric are optional for trials. Keep the same config and station across services; use the model's installed environment for policy serving.

cd /home/ubuntu/manimux
# Validate only; opens no hardware and creates no recording directories.
envs/yam/.venv/bin/python -m manimux inspect \
  --config manimux/configs/experiments/pi05-latest300ep-step30000/coin/rtc32_beta10.yaml

# Camera: separate terminal; append --check for configuration-only validation.
envs/yam/.venv/bin/python -m manimux.servers.camera.server \
  --config manimux/configs/experiments/pi05-latest300ep-step30000/coin/rtc32_beta10.yaml

# Pi05 model: separate terminal; append --check to avoid loading weights.
XPolicyLab/policy/Pi_05/openpi/.venv/bin/python -m manimux.servers.policy \
  --config manimux/configs/experiments/pi05-latest300ep-step30000/coin/rtc32_beta10.yaml

# Runtime: separate terminal; reuse the RoboGUI command above.
envs/yam/.venv/bin/python -m manimux run \
  --config manimux/configs/experiments/pi05-latest300ep-step30000/coin/rtc32_beta10.yaml

The last command waits for Prepare Run. It does not start robot motion by itself. Optional --once connects immediately and exits after one rollout, so use it deliberately. A bare camera component recipe uses the advanced --camera-config flag; common --config always refers to a full runtime configuration.

Naming migration: configuration lists now live under manimux/configs/lists/ and use --config-list. The list YAML contents, IDs, recording roots and existing episode files are unchanged. Internal manimux.collections helpers and saved collection metadata retain their schema names for historical compatibility.

Switching boundaries

Current recorded conditions remain the authority. Configuration inspection does not validate model outputs or physical execution. Validate the first trials' actual switch timing before collecting a full cohort.

Experiments

Agree on the rule before collection

An agent setting up a custom scored study should ask whether partial credit is needed, unless already agreed. It should not invent weights or require scoring for free inference. Keep PRM optional. Record protocol choices separately from runtime configuration.

Define the task in one file

A new rubric version may add score options while keeping every existing option unchanged. The collection retains prior trial progress when execution settings match. Old recordings keep their original rubrics and scores; adding an option does not automatically regrade them or make scores across rubric versions interchangeable.

Write manimux/configs/tasks/<task>.yaml with the exact prompt, available layouts, inline scoring rules, run.max_control_steps and optional initial pose. The task owns the per-rollout control-tick limit; models and methods inherit it rather than duplicating it. A stop-budget-only change applies to subsequent sessions and retains study progress. Saved session budgets and condition fingerprints remain unchanged; compare results using those recorded stopping rules. Experiment variants inherit this task through _base_; do not create a separate scoring YAML. Set repeats and the recording root in the collection. The sample task values below are replaceable.

run:
  task_prompt: put_the_coin_into_the_bin2  # Exact instruction sent to the policy
  experiment_template:
    name: Coin layouts
    layout_ids: [near, far]
    require_reference: false
evaluation:
  kind: stages
  task_id: coin
  rubric_id: coin_stages_v2
  label: Coin transfer and deposit
  options:
    - {id: incomplete, label: No coin picked up, score: 0, success: false}
    - {id: picked_up, label: Coin picked up by one hand, score: 0.1, success: false}
    - {id: handover, label: Handed over, score: 0.3, success: false}
    - {id: above_bank, label: Coin held above piggy bank, score: 0.5, success: false}
    - {id: deposited, label: Deposited in target, score: 1, success: true}

Prepare Run / Prepare Experiment. Both buttons are available before a rollout. Prepare Run needs no layout or scoring. For Prepare Experiment, select Layout once; its reference is attached automatically and scoring is offered after Finish. The Images panel is expanded by default. Historical replay has a separate command below. Task, policy and executor are read-only. Keep experiment name, condition and model parameters in YAML. The Top gallery matches run.task_prompt ignoring case, punctuation and spaces versus underscores; store images in data/evaluation_layouts/<task>/<layout>.png.

Repeat numbers are suggested from finalized, scored trials across sessions under the same run.output_dir, matched by task, experiment name, condition, layout and reference hash. Successful and failed tasks count; invalid, skipped and unstarted attempts remain recorded but do not consume a repeat. When a layout is complete, select another. Use a separate output directory for each model/checkpoint/method or changed scoring rule. No repeat counter is stored only in the browser.

The task's run.experiment_template defines available layouts and reference requirements; collection trials selects a layout subset and repeats. Inline evaluation defines RoboGUI's mutually exclusive score choices. There is no UI template editor; edit the task file and reload the runtime before the next study condition. Neither file randomizes the trial order.

Write every task with kind: stages and explicit options. Each option declares its outcome label, total score and success. Use the same format for binary outcomes, item counts and intermediate milestones. RoboGUI displays single-choice buttons; it never sums selected milestones.

Stacking example — evaluation section in the task file:

evaluation:
  kind: stages
  task_id: stack_blocks
  rubric_id: stack_blocks_count2_v1
  label: Block stacking
  options:
    - {id: count_0, label: "Blocks stacked: 0/2", score: 0.0, success: false}
    - {id: count_1, label: "Blocks stacked: 1/2", score: 0.5, success: false}
    - {id: count_2, label: "Blocks stacked: 2/2", score: 1.0, success: true}

For a binary task, explicitly list Not completed (0.0, false) and Completed (1.0, true). IDs and labels must be unique. Scores must be finite in [0, 1]; success is true exactly for score 1. Bottles list every count and its score explicitly; preserve score precision in YAML and round only for display. Old recorded count/binary rubrics remain readable; no historical labels are rewritten.

Use one rubric across compared methods. Version rubric_id when criteria change. Each rollout saves a snapshot; later YAML edits do not change historical scores. Current coin task · Stacking task

Errors and panel width

Rollout errors identify the source: Model service, Camera / sensor, Robot / CAN, or Action decoding / IK. The persistent card shows the failure stage, known device or endpoint, reason, and next step. Enable Show original error (its state persists across error updates) for the underlying exception. Unknown errors remain explicitly unclassified; a missing CAN interface alone does not establish DFU mode or a broken motor. The top status retains the failure category while the runtime is idle. For example, a model connection refusal means check model loading and its endpoint; a missing robot interface means restore that interface before another Prepare. A camera service that fails before runtime connection must also be checked in its own terminal; camera startup and read errors include the failing camera name.

Operation failures appear in a persistent card with the operation, time and selectable error text. Live status updates do not overwrite it. Dismiss error message only hides the card; it does not reset a robot fault or send commands. Connecting to a new runtime service clears the previous service's error card; any failure reported by the new service is displayed independently. The operator panel uses native floating-panel resize handles. Drag a side edge to change its width; newer Viser versions dock it on the right automatically.

Operate each trial

Run controls stay in the same place; diagnostic Run details start collapsed. Switching method or initial pose keeps your selected layout for the same task when it remains available. Each task remembers its own selection during this GUI session.

Start/Pause show the requested state until the runtime confirms it. Prepare and Home show their current phase and elapsed time. A progress heartbeat means the reporter is alive, not that the robot has reached its target. If the heartbeat is lost, the UI marks hardware state unknown and keeps another Prepare blocked until the runtime reports completion or failure.

1. Use Prepare Run for a free trial. For a scored trial, reset the scene, select Layout, then click Prepare Experiment; its Top reference is attached automatically.
2. Hover over the Top layout reference and click its expand icon to fill the browser window. The image keeps its aspect ratio; press Escape or click the close button to return.
3. Prepare → adjust layout → Activate Warmup → Start rollout. Layout stays editable until first Start. The suggested repeat updates with the selected layout. Start validates the template and freezes layout, repeat and image hash; Pause/Resume keeps that selection locked. Warmup only previews predictions; it never executes them. Keep reset and stopping rules consistent.
4. Finish, select one score option, then Save evaluation. The task supplies every single-choice label and score explicitly. Choose one outcome; nothing is preselected.
5. For an unusable trial, mark invalid and state the reason. Skip evaluation leaves it unreviewed.
6. Start a fresh rollout for the next trial. Preserve failed attempts and separately identify retries.

For completion time, annotate seconds from formal Start to the success criterion, only for successful attempts. Do not substitute the full recording duration. Free rollouts have no scoring step. Optional operator notes and failure details are entered after Finish.

After Finish, camera previews remain visible and are marked Frozen. During a rollout, Stale means no new preview frame has arrived recently. Saving shows a persistent receipt. Edit last evaluation reopens the saved rubric and result; saving again preserves the previous label in revision history. Skipping an edit leaves the saved result unchanged.

If a finished experiment was skipped or the GUI was restarted, Trial setup shows Unscored experiment → Score selected experiment while the runtime is idle. It lists started, finalized, unlabelled experiments for the same task and method output directory. Select the session/rollout and score it in the existing form using its saved rubric; new YAML rules do not replace historical scoring rules. Free runs and partial attempts are excluded.

Records

<run.output_dir>/
└── session-.../
    ├── session-manifest.json
    └── rollout-.../
        ├── meta.json
        ├── result.json
        ├── events.jsonl
        ├── data.zarr/           # ticks + raw / strategy / committed plans
        ├── videos/             # camera video + index.json
        └── evaluation/
            └── human-label.json

What evaluation actually reads

For a validation or dispatch failure, inspect command_rejected in the aborted rollout's .partial/events.jsonl. It records the attempted command, measured state and timestamps, coordinate names/units, channel mapping, shared limits, plan/request identity, failure stage and exception causes. dispatch.submitted_channels lists batches that returned successfully; dispatch.attempting_channels may have been partly submitted before failure. dispatch.dispatch_attempted indicates an attempted dispatch, not confirmed execution. If optional channel or coordinate diagnostics cannot be read, diagnostic_error explains the missing fields while the core attempt is still saved. The failed attempt is excluded from successful tick data.

Read results first, traces second. Human labels hold task outcomes; metadata identifies the trial. Zarr holds numerical motion data. Events explain lifecycle changes, inference and handoffs; they are not the score sheet.

data.zarr/ticks/state is measured motion; scheduled is the timeline reference; command is the executor submission to the SDK. plans includes predicted future points that may never execute. JSON Unicode escapes such as \u4e22 are valid Chinese text encodings; new JSON records preserve readable UTF-8 text. Existing records and their hashes are unchanged.

Check the actual episode path in RoboGUI; station configuration can override the output root. Open a finalized episode in a separate, read-only replay process:

cd /home/ubuntu/manimux
envs/yam/.venv/bin/python -m manimux.robogui.replay \
  --robot yam --episode /path/to/session-xxx/rollout-001 \
  --source state --host 127.0.0.1 --port 8087

--source state shows measured state; scheduled shows the timeline reference; command shows executor output, not a hardware acknowledgment. For a custom display model, replace --robot yam with --config /path/to/robogui.yaml. The player starts paused and supports seeking and playback speed using recorded tick timestamps. It does not replay RGB videos or connect to hardware. No camera, policy or runtime service is needed. Partial episodes and records without the selected stream cannot be replayed. Recording contract

Reports

The report index is separate from the scoring template. Save it under data/evaluation_tasks/<task>/task.yaml and select exact episodes/settings.

schema: task-report-v1
task_id: coin
title: Coin study
language: zh-CN  # Omit for English.
output_root: ../../analysis
groups:
  - setting_id: pi05-coin-50k-rtc-v1
    model: Pi05
    method: RTC
    split: test
    target_attempts: 6
    expected:
      meta.task: put_the_coin_into_the_bin
      manifest.config.inference.algorithm: rtc
      meta.policy_backend.model.checkpoint_variant: pi05_yam_coin_step50000
    episodes: []  # Add actual episode_dir entries after collection.

Paths resolve relative to task.yaml. An empty list means pending, not zero success. Bind the recorded checkpoint and resolved setting; do not pool trials by folder name alone.

envs/yam/.venv/bin/python -m manimux.evaluation \
  --task-dir data/evaluation_tasks/coin

The command prints the HTML path. With the example output root:

data/analysis/coin/
├── latest.json
└── reports/<version>/
    ├── index.html
    ├── summary.json
    ├── episode-metrics.json
    ├── episodes.csv
    ├── cohort.jsonl
    ├── provenance.json
    ├── task.yaml
    └── figures/

Metrics and aggregation

The comparison table groups attempts by the configured model/method setting. Missing values stay missing; every metric has its own eligible sample count.

Which two points form a handoff jump?

A boundary must appear in executed ticks/plan_id transitions and match its committed plan and boundary event. A received prediction alone does not count.

• Continuous switch (typically RTC): at the incoming committed start time t, interpolate the outgoing committed reference at t and compare it with the incoming committed first point. That incoming point is after runtime trimming/blending. This is not a difference between two raw model rows or two adjacent control ticks.
• End-hold restart (typically Serial): compare the last verified constant held command with the incoming committed first point. Report this class separately from continuous switches.
• Apply the selected robot FK to both joint targets. The reported jump is 1000 × ||XYZ_new − XYZ_old|| in mm, separately for each arm. It describes reference continuity, not measured robot motion.

Mean / P95 / Max: within one rollout, compute the mean, linearly interpolated 95th percentile and maximum over eligible boundaries of the same arm/class. Across a setting, Mean averages rollout means, P95 averages rollout P95s, and Max is the largest boundary. Rollouts have equal weight; the displayed P95 is not the percentile of pooled boundaries.

10 / 20 / 30 steps: compare old and new committed trajectories on the same future time grid. Each window requires complete coverage and has its own sample set; no endpoint padding. Serial can have a restart jump without a future overlap score. Aggregate valid window values within each rollout, then average rollout means. Velocity cosine excludes stationary pairs.

Enable smoothness and plots

Add this task-wide profile to the report index. The FK assembly must match the recorded robot/tool; it runs offline.

smoothness:
  robot_config: ../../../manimux/configs/embodiment/robot/yam_dual.yaml
  groups: [left_arm, right_arm]
  sample_hz: 30
  max_gap_s: 0.1
  cutoff_hz: 10
  threshold: 0.05

NLDLJ ↓ uses ln(T³ × ∫||jerk||²dt / peak_speed²); jerk stencils crossing plan changes are excluded. NSPARC ↓ uses positive spectral arc length of speed and retains switches. Both use scheduled EE references, with verified constant commands replacing historical hold placeholders. Pause/reset, identity breaks and long gaps split segments. Duration-weight valid segments, then equally weight active channels and rollouts; this is not a score of measured tracking accuracy.

The report generates handoff, overlap and outcome/smoothness figures when data is available. Colored manipulation-area overlays require a manipulation_scope profile; inference coverage must use that method's recorded state points. Speed-distribution diagnostics have a separate entry point.

Each report is a new immutable version. latest.json points to its HTML and summary. The CLI leaves raw recordings and human labels unchanged; episode-metrics.json preserves exclusions and metric provenance.

Invalid and unreviewed counts stay visible; missing metrics remain missing. Keep different rubric snapshots separate. The report CLI computes reference seams and smoothness offline; it does not run a robot or PRM inference. Report schema and metric options · PRM workflow · Speed diagnostics

Agent references

This HTML is the maintained user guide. Technical contracts live in each skill's references/. docs/ is an ignored local archive. The team's private experiments.md retains its study decisions and result tables; it is not a second public guide.

Troubleshooting

Start here

Choose when IK runs

Pose-producing adapters support two modes. chunk is the default and solves the decoded chunk before Timeline commit. step keeps TCP poses in Timeline and solves only the current target inside each control tick. Joint-output adapters need no IK mode.

execution:
  ik:
    mode: step
    budget_ms: 5.0
run:
  control_timing: true

Merge these fields into an experiment with a direct executor and inline action decoding. The optional budget defaults to half the control period and must be shorter than that period. Reuse the example manimux/configs/experiments/openwam-custom/put_bottles/serial_step30000_step_ik.yaml. Existing experiments remain in chunk mode.

Step mode supports serial/async for IK adapters. OpenWAM YAM and XR1 YAM also support pose-space RTC/PAINT; Pi05 Tianji, XR1 Tianji and UMI-DP Tianji support pose-space RTC. Unsupported combinations fail at configuration validation. Smooth/MPC, joint candidate scoring, temporal ensembling, executor-based RTC forecasts, cubic joint blends, waypoint handoff and decoder diffIK substeps require chunk mode.

The current pose is interpolated with linear position/gripper and quaternion SLERP, then IK is seeded from the current measured joint state. Both arms share the budget; a failed or late solution holds both arms at the previous command. Watch the step_ik timing stage and step_ik_hold events. The budget is cooperative: native calls cannot always be interrupted, so verify control timing on the target machine. Pose-space interpolation can produce a different joint path from interpolating a pre-solved joint chunk.

Optional TCP target speed limit (off by default): set execution.ik.pose_speed_limit to a mapping with max_linear_m_s and/or max_angular_rad_s. Values must be positive and finite. Omit it or use null to disable. This setting applies to step IK only.

execution:
  ik:
    mode: step
    pose_speed_limit:
      max_linear_m_s: 0.25
      max_angular_rad_s: 0.6

The limiter bounds each target increment from the TCP pose of the last sent joint command, using one nominal control period. IK still starts from measured joints. Gripper commands keep their existing limits. Timeline and RTC/PAINT timestamps remain unchanged; sustained limiting can cause tracking lag or skipped intermediate targets. This limits the requested pose increment, not measured physical velocity. It does not implement waypoint-time stretching.

Execution ownership: Timeline owns chunk handoff and time-based reference sampling. The execution pipeline applies the optional pose limiter, resolves IK when needed, and calls the joint executor. The IK solver only solves; the pipeline handles failed solves by holding the last command. The main loop orchestrates these calls, records diagnostics, validates safety and sends the command. Neither the limiter nor IK rewrites the trajectory clock.

The runtime path is observation → policy → action adapter → inference strategy / Timeline → executor → robot. Model inference and hardware control run in separate processes. Model output, scheduled reference, executor command and measured state are distinct recorded stages.