Latent-space Physics source code

The source code for our latent space physics paper is online now:

It contains both the Navier-Stokes solver for data generation (based on mantaflow, and the keras code (for tensorflow for training the autoencoder and LSTM networks.

The preprint can be found here:

Paper Abstract

Our work explores methods for the data-driven inference of temporal evolutions of physical functions with deep learning techniques. More specifically, we target fluid flow problems, and we propose a novel network architecture to predict the changes of the pressure field over time. The central challenge in this context is the high dimensionality of Eulerian space-time data sets. Key for arriving at a feasible algorithm is a technique for dimensionality reduction based on convolutional neural networks, as well as a special architecture for temporal prediction. We demonstrate that dense 3D+time functions of physics system can be predicted with neural networks, and we arrive at a neural-network based simulation algorithm with practical speed-ups. We demonstrate the capabilities of our method with a series of complex liquid simulations, and with a set of single-phase simulations. Our method predicts pressure fields very efficiently. It is more than two orders of magnitudes faster than a regular solver. Additionally, we present and discuss a series of detailed evaluations for the different components of our algorithm.

Droplets with Neural Networks

The final version of our paper on learning droplet formation models with neural networks is online now. It will be presented at SCA 2018 in Paris.

Our paper proposes a new data-driven approach to model detailed splashes for liquid simulations with neural networks. Our model learns to generate small-scale splash detail for the fluid-implicit-particle method using training data acquired from physically parameterized, high resolution simulations. We use neural networks to model the regression of splash formation using a classifier together with a velocity modifier. For the velocity modification, we employ a heteroscedastic model. We evaluate our method for different spatial scales, simulation setups, and Navier-Stokes solvers. Our simulation results demonstrate that our model significantly improves visual fidelity with a large amount of realistic droplet formation and yields splash detail much more efficiently than finer discretizations.

Further information


Volumetric Fluid Flow from a Single Video

Our SCA 2018 paper on Coupled Fluid Density and Motion from Single Views is online now! You can check it out here. We’re reconstructing a dense fluid flow field from a single video stream using a strong Navier-Stokes prior.

To be presented at:

Full Abstract
We present a novel method to reconstruct a fluid’s 3D density and motion based on just a single sequence of images. This is rendered possible by using powerful physical priors for this strongly under-determined problem. More specifically, we propose a novel strategy to infer density updates strongly coupled to previous and current estimates of the flow motion. Additionally, we employ an accurate discretization and depth-based regularizers to compute stable solutions. Using only one view for the reconstruction reduces the complexity of the capturing setup drastically and could even allow for online video databases or smart-phone videos as inputs. The reconstructed 3D velocity can then be flexibly utilized, e.g., for re-simulation, domain modification or guiding purposes. We will demonstrate the capacity of our method with a series of synthetic test cases and the reconstruction of real smoke plumes captured with a Raspberry Pi camera.

New paper online: Deep Fluids – A Generative Network for Parameterized Fluid Simulations

Our paper “Deep Fluids: A Generative Network for Parameterized Fluid Simulations” in collaboration with the CGL of ETH Zurich is online now!

You can check it out here:

The goal is a novel generative model to synthesize fluid simulations from a set of reduced parameters. A convolutional neural network is trained on a collection of discrete, parameterizable fluid simulation velocity fields. We also demonstrate that we can handle complex parameterizations in reduced spaces, and advance simulations in time by integrating in the latent space with a second network.

Computer Game Laboratory, SS’18 up and running

The current instance of our computer game laboratory, a practical course for TUM games engineering students, is ongoing at the moment, you can check out the latest development of the four game projects on our wiki main page.

Look out for the final presentations and playable demos during the SS’18 demo day…

One interesting new development here is a first joint team between TUM and students from the Media Design Hochschule Muenchen (MDH).

You can also find a full list of previous games here.


Ferienakademie 2018 – Accelerating Physics Simulations with Deep Learning

Computer simulations are a powerful method to study physical and engineering systems such as fluids, collections of molecules, or social agents. Traditionally, differential equations such as the Navier-Stokes equations form the basics of performing simulations as they dictate the time evolution. A recent, promising development is to use machine learning for model-free prediction of a systems’ behavior and to identify the appearance of spatio-temporal patterns. Deep learning with neural networks is a particularly interesting and powerful machine learning method that can be employed for this task.

More info can be found here.

TEDx talk by Nils Thuerey

Nils recently gave a talk titled “Deep Learning Beyond Cats and Dogs” at a TEDx event organized at the Technical University of Munich.

Abstract: Deep learning, which is seemingly everywhere these days, is well-known for its capability to recognize cats and dogs in internet images, but it can and should be used for other things too. It can be used to figure out the complicated physics that dictate fluid behavior. Actually, simulating turbulence is not only a million dollar problem (really, google it!) but it can help us create more realistic virtual worlds. It can even help us understand medical and physiological behaviors like blood flowing through our body. Nils performs cutting-edge research and explains how neural networks are well on their way to becoming the fourth pillar of science.

Biography: Nils Thuerey’s work is in the field of computer graphics: he models physical behaviors of fluids such as water and smoke to enable computer created virtual effects to look like the real thing. These phenomena are very expensive to simulate computationally, so Nils’ research explores the use of deep learning methods to generate the effects more quickly and more realistically. Before assuming his assistant professor position at TUM, Nils studied in Erlangen, held a post-doc position in Zurich, and worked in the visual effects industry. He was awarded a technical Oscar for the development of an algorithm which aids in editing explosion and smoke effects for film.

This talk was given at a TEDx event using the TED conference format but independently organized by a local community. Learn more at , or visit the TEDxTUM website.


Recent research on physics-based deep learning

In the following we give an overview of our recent publications on physics-based deep learning methods. In particular, we focus on solving various aspects of fluid problems modeled with the Navier-Stokes (NS) equations. These topics are a central theme of the research work in our group. While, naturally, the long term goal would be to simply give the initial conditions of a problem as input to a neural network, and then rely on the network to infer the solution with appropriate accuracy, the complexity of the NS equations makes this an extremely challenging problem.
Thus, we typically consider constrained solution spaces for sub-problems of the NS equations, that we believe are nonetheless very interesting, and useful for their respective domains. In this way, we are also working towards improving the state of the art in order to tackle more and more general problems in the future.
Our publications have targeted different aspects of a typical simulation pipeline, and differ in terms of how deeply integrated they are into the Navier-Stokes solve. The following list is order from loose to tight couplings. E.g., the last entry completely replaces a regular solver.

mantaflow in Blender at BCon17

Sebastian Barschkis, CS-student at TUM, has just presented his latest progress regarding the integration of our mantaflow solver into Blender. You can check out his full presentation including insights about code structure as well as using the solver here:

There are admittedly still some rough edges, but mantaflow should give Blender users a significant step forward in terms of visual quality and performance.

Code for CNN-based flow descriptors is online

The full source code of our recent SIGGRAPH paper coupling fluid simulations with convolutional neural-networks is finally online now! You can check it out here:

The code uses our mantaflow framework for the Navier-Stokes simulation part, and Google’s tensorflow framework for the deep learning portion. You can find a short introduction / how-to on the github page above. If you give it a try, let us know how it works!

The corresponding paper is this one
“Data-Driven Synthesis of Smoke Flows with CNN-based Feature Descriptors”, by Rachel Chu and Nils Thuerey.