Recently we’ve targeted the question how to reliably evaluate a stochastic, temporal GenAI-surrogate. One of our excellent Bachelor students at TUM, Sebastian Pfister together with Benjamin, has created a stochastic chaotic system benchmark that combines robustness and ease-of-use, StocBench: https://tum-pbs.github.io/stocbench/
It targets the basic question: how well do generative models capture stochastic dynamics, and how many NFEs are really needed?
StocBench provides a systematic benchmark using turbulent fluid flows, combining:
- 300,000 training snapshots from 1,500 trajectories
- 240,000 ground-truth samples across 48 conditioning states
- 3,000 generated samples per condition
- 5,000 long-term tests with 50-step autoregressive rollouts
The evaluation reveals a really interesting efficiency–accuracy trade-off: a distilled model matches the conditional-variance accuracy of a 400-step Flow Matching (FM) sampler for an uncertain task. However, in deterministic (zero variance) settings, the picture changes and methods like FM yield better estimates. A central insight is that distinguishing model noise from learned distributional variation requires an evaluation in both settings. This is enabled by StocBench.
