Benchmarking code generation for physics-inspired animations

SimuScene
Training & Benchmarking Code Generation for Physics-Inspired Animations

Given a natural-language physical scenario, models generate executable Python code that renders a physics-inspired animation. Correctness is judged by whether the rendered qualitative dynamics are visually aligned with the description — verified by a vision-language model over verification questions.

Paper Code Download Dataset

Videos

Animation gallery

Mechanics

Within a 5-meter diameter, glass-walled circular room, a 2-meter diameter wooden carousel spins on a frictionless central axis at a constant rate of 0.5 revolutions per second. A small, 0.2 kg rubber ball is placed 1 meter from the axis on the smooth wooden surface. The problem is to determine the ball's trajectory and final position after 15 seconds, considering the Coriolis force and absence of friction.

Optics

In a 2m x 0.5m x 0.3m acrylic tank filled with water up to 0.25m, a collimated beam of white light enters the water at a 10-degree angle from a 75-watt incandescent bulb. At the water surface, a thin film of soap creates an interference pattern. The room is set to a consistent 21°C. The film causes additional dispersion of the light, forming a complex, iridescent rainbow on the tank's bottom. Over 20 seconds, as the soap film slowly thins due to evaporation, the rainbow patterns shift and change, displaying a rich interplay of colors that gradually fade.

Fluid Mechanics

Inside a rectangular aquarium with dimensions of 100 cm by 50 cm and filled to a height of 40 cm with a transparent gel, a small stainless steel disc with a radius of 7 cm is submerged and fixed horizontally at a depth of 20 cm. The disc is rotated at a constant speed of 22 revolutions per minute by an overhead motor. The rotation of the disc generates vortices that influence the movement of suspended fine glitter particles within the gel. Over the course of 14 seconds, the glitter particles are swept into helical paths by the vortices, gradually forming concentrated, spiral formations that trace the flow patterns within the gel, showcasing the interaction of rotational motion with a semi-solid medium.

Highlights

What SimuScene brings

A benchmark for physics-inspired animation generation

SimuScene is a novel task, dataset, and benchmark for physical-scenario understanding, qualitative dynamics, and animation-oriented code generation — spanning 52 concepts across five physics domains, with a rigorously human-verified 334-scenario test set and quality-controlled training data.

Frontier LLMs still struggle

We evaluate 10 contemporary LLMs and find the task challenging: even the strongest model reaches only ~21.5% Avg@8 accuracy, exposing the gap between generating video-producing code and producing animations that faithfully capture physical dynamics.

Code–Video–Judge reinforcement learning

An automated pipeline executes generated code into videos and uses a vision-language model to answer verification questions. The resulting visual reward trains text-only models via RL, improving text-to-code-to-video consistency with no evidence of reward hacking.

Benchmark

Frontier LLMs on the SimuScene benchmark

Rank	Model	Avg@8 (↑)	Pass@8 (↑)
—	Qwen3-32B	11.1%	30.5%
—	GPT-oss-20b	10.5%	32.0%
—	GPT-oss-120b	14.0%	37.4%
—	Gemini-2.5-pro	12.7%	37.4%
—	DeepSeek-V3.1	14.5%	40.7%
—	Qwen3-235B-A22B	15.1%	41.0%
—	GPT-o4-mini	17.2%	42.5%
—	GPT-o3	15.9%	45.2%
—	DeepSeek-R1-0528	21.5%	52.7%
—	GPT-5-medium	20.5%	59.9%

Pipeline

How evaluation works

SimuScene evaluates whether the generated animation is visually aligned with the intended physical dynamics.

Text → Code

The model writes executable Python code that renders the scenario from a natural-language description.

Code → Video

Run the program to render a short video of object trajectories and interactions.

Video → Judgement

A VLM answers verification questions to decide whether the dynamics match the description.

Resources

Downloads & links

Dataset

Direct download for the benchmark dataset.

Download Dataset

Project

Paper, code, models, and demo materials.

Paper Code Models

Citation

BibTeX

If you use SimuScene in your research, please cite:

@misc{simuscene2026,
          title={Training and Benchmarking Code Generation for Physics-Inspired Animations},
          author={Yanan Wang and Renxi Wang and Yongxin Wang and Xuezhi Liang and Fajri Koto and Timothy Baldwin and Xiaodan Liang and Haonan Li},
          year={2026},
          eprint={2602.10840},
          archivePrefix={arXiv},
          primaryClass={cs.LG},
          url={https://arxiv.org/abs/2602.10840}, 
    }

SimuScene Training & Benchmarking Code Generation for Physics-Inspired Animations