OopsieVerse: A Safety Benchmark with Damage-Aware Simulation for Robot Manipulation

Abstract

What if your home robot finishes the job... but breaks your kitchen in the process?

Robots are getting better at manipulating everyday objects, but task success alone is not enough. A robot that picks up an egg while cracking it, or pours water while spilling half of it, is not ready for real homes. The missing piece is safety. And today's simulators barely measure it.

OopsieVerse tackles this problem head on. It introduces a unified, damage-aware simulation framework for household manipulation, where harm is no longer invisible. Instead of only rewarding task completion, OopsieVerse augments the standard decision-making setup with explicit damage signals, penalties, and user-defined safety preferences.

At the heart of the system is DamageSim, a simulator-agnostic layer that converts physical signals like contact forces, temperature changes, and liquid interactions into measurable mechanical, thermal, or fluid damage. Paired with a benchmark suite of household tasks, OopsieVerse cleanly separates "did the robot succeed?" from "did it do so safely?"

Integrated into both BEHAVIOR-1K (Omniverse) and RoboCasa (MuJoCo), OopsieVerse supports 1) safer data collection, 2) damage-aware imitation and 3) reinforcement learning, 4) safety benchmarking of Vision-Language-Action policies and more, for improved sim-to-real transfer of safer behaviors.

In short, OopsieVerse shifts the focus from performance alone to safer performance, helping move household robots closer to damage-free deployment in the real world.

DamageSim

DamageSim is our simulator-agnostic plugin that makes physical safety measurable by tracking object-centric “health.” It monitors simulator signals—such as contact forces, temperature, and liquid exposure—and converts them into mechanical (e.g., impact or compression), thermal, and fluid damage, which can be used as observations, rewards, or termination conditions. We instantiate it in RoboCasa (MuJoCo) and BEHAVIOR-1K (Omniverse) showcasing its consistency across simulators.

Mechanical Damage

Thermal Damage

Fluid Damage

Damage-Augmented POMDP Implementation with DamageSim. Conceptually, our DamageSim plugin (blue) extends an existing POMDP in simulation (green, dotted arrow to agent) by augmenting the state with health. The health state can in turn influence the observations and rewards, and/or provide access to new damage-aware terminal states. DamageSim implements the health state to augment the original POMDP state into a Damage Aware state (s^DA ∈ S^DA = S × S^h) as a per-object physical property, updated at every timestep by a set of damage evaluator models (DEM) that compute mechanical, thermal, and fluid damage (d_DEM) from simulator state and interactions, reducing per-entity health accordingly. The augmented POMDP provided by DamageSim approximates better the conditions of the real-world task.

DamageSim is simulator-agnostic. we instantiate it in BEHAVIOR-1k (Nvidia Omniverse) and RoboCasa (MuJoCo) to demonstrate consistent safety measurement across different physics backends.

RoboCasa

BEHAVIOR-1k

OopsieBench

OopsieBench is a suite of 32 household tasks in total (15 tasks shown in the grid below; hover a tile to see its name). The suite is designed to (i) expose policies to realistic, physically damaging failure modes in household manipulation, and (ii) make safety measurable by contrasting easy but risky strategies with safer ones that require more careful interaction (e.g., gentler contact, safer approaches, or avoiding hazards). The benchmark spans diverse scenes, objects, and damage modalities, is cross-platform (BEHAVIOR-1k and RoboCasa), and includes a dataset of safe and unsafe human teleop demonstrations for five tasks.

Pour Glass

Ignite Wood

Place Log

Place Bowl in Sink

Open Drawer

Nav to Table

Attach Camera

Pick Scrub

Shelve Item

Open Single Door

Lift Egg

Place Plate

Put Cup in Microwave

Turn on Microwave

Wipe Counter

Click a tile to enlarge.

How can OopsieVerse enable safer policy learning and evaluation?

OopsieVerse can provide real-time safety feedback during data collection, enabling data collectors to collect safer data. This feedback is provided in two ways: (1) damage-based coloration and (2) health bars of tracked objects. The health bars are particularly helpful when damage occurs outside the current camera view (e.g., the wineglass falling behind the counter).

Many real-world (or sim-collected) demonstration datasets mix safe and unsafe behavior. Using DamageSim, we can automatically flag trajectories that incur damage and filter them out to construct a dataset of only safe demonstrations. In the videos below, the top row shows an IL policy trained on the full dataset (including unsafe trajectories), while the bottom row shows a policy trained on the filtered safe-only demonstrations.

Pour Glass

Add Firewood

Lift Egg

Shelve Item

Wipe Countertop

DamageSim lets us turn physically-grounded damage signals into a shaping reward that penalizes harmful interactions. Combined with the task reward, this yields a learning signal that encourages agents to complete the task while minimizing damage. We evaluate this idea in three settings: (i) pure RL on the Place Plate task with and without the damage reward (PPO), (ii) behavior cloning on Move Glass of Water task followed by PPO finetuning with the damage reward, and (iii) fine-tuning a Shelve Item policy—initially trained on the full IL dataset—using an additional damage-reward using DSRL. The video below shows the training process for (iii), illustrating that over time the agent learns to reduce damage while maintaining high task performance.

We evaluate the safety of modern manipulation policies by benchmarking GR00T, a state-of-the-art Vision-Language-Action model from NVIDIA. Even when GR00T achieves high task success, it frequently causes damage, resulting in much lower safe success rates and degraded environment health. This underscores the limits of evaluating VLAs solely by task completion and highlights the need for benchmarks and learning signals that explicitly account for harmful interactions.

Attach Camera

Open Microwave Door

Pick up Scrubber

Ignite Wood

Open Single Door

Turn on Microwave

Counter to Microwave

Turn on Stove

Our ultimate goal is safe real-world robot behavior, so we test whether policies trained with OopsieVerse transfer safely to real world. Compared to a baseline IL policy trained on all data, a damage-aware IL policy trained on health-filtered episodes behaves more cautiously in the Pour Water and Shelve Cereal Box tasks. It goes futher away from the laptop when pouring water and learns to make space for the cereal box by pushing non-fragile objects like the crackers box instead of fragile objects like glass bottles. Our experiments suggest that explicit damage signals obtained via OopsieVerse can translate into improved safety on hardware.

Pour Water (Safe + Unsafe Data)

Pour Water (Safety-filtered Data)

Shelve Cereal Box (Safe + Unsafe Data)

Shelve Cereal Box (Safety-filtered Data)