TecoGAN training code online

We have finally uploaded code for training new TecoGAN models in our github repository: https://github.com/thunil/TecoGAN

While inference mode and pre-trained models have been up for a while now, the new version also contains the code necessary to train new models, and a script to download a suitable amount of training data.

A properly trained TecoGAN model can generate fine details that persist over the course of long generated video sequences. The github repo contains a few samples, such as the mesh structures of the armor, the scale patterns of the lizard, and the dots on the back of the spider highlight the capabilities of our method. A spatio-temporal discriminator plays a key role to guide the generator network towards producing coherent detail.

Video: https://www.youtube.com/watch?v=pZXFXtfd-Ak , and
Paper: https://arxiv.org/pdf/1811.09393.pdf.

Multi-Pass GAN Paper to appear at SCA 2019 now online

Our “physics-based deep learning” paper on generating very high resolution fluid flows based on generative adversarial neural networks is online now. The key idea is to split the problem into multiple orthogonal passes, which nicely works in conjunction with progressive growing techniques. We demonstrate this for several Navier-Stokes flow problems.

You can check out the video here:

The arXiv preprint can be found here.

Abstract: We propose a novel method to up-sample volumetric functions with generative neural networks using several orthogonal passes. Our method decomposes generative problems on Cartesian field functions into multiple smaller sub-problems that can be learned more efficiently. Specifically, we utilize two separate generative adversarial networks: the first one up-scales slices which are parallel to the XY- plane, whereas the second one refines the whole volume along the Z- axis working on slices in the YZ- plane. In this way, we obtain full coverage for the 3D target function and can leverage spatio-temporal supervision with a set of discriminators. Additionally, we demonstrate that our method can be combined with curriculum learning and progressive growing approaches. We arrive at a first method that can up-sample volumes by a factor of eight along each dimension, i.e., increasing the number of degrees of freedom by 512. Large volumetric up-scaling factors such as this one have previously not been attainable as the required number of weights in the neural networks renders adversarial training runs prohibitively difficult. We demonstrate the generality of our trained networks with a series of comparisons to previous work, a variety of complex 3D results, and an analysis of the resulting performance.

Deformation-aware Neural Networks for Liquid Simulations at ICLR 2019

Lukas Prantl last week successfully presented our paper on deformation learning for capturing solution spaces of Navier-Stokes (liquids in particular) at the International Conference on Learning Representations (ICLR). The full paper and video can be found here.

Our proof-of-concept Android app is still available for free in the app store: https://play.google.com/store/apps/details?id=fluidsim.de.interactivedrop

Full abstract: We propose a novel approach for deformation-aware neural networks that learn the weighting and synthesis of dense volumetric deformation fields. Our method specifically targets the space-time representation of physical surfaces from liquid simulations. Liquids exhibit highly complex, non-linear behavior under changing simulation conditions such as different initial conditions. Our algorithm captures these complex phenomena in two stages: a first neural network computes a weighting function for a set of pre-computed deformations, while a second network directly generates a deformation field for refining the surface. Key for successful training runs in this setting is a suitable loss function that encodes the effect of the deformations, and a robust calculation of the corresponding gradients. To demonstrate the effectiveness of our approach, we showcase our method with several complex examples of flowing liquids with topology changes. Our representation makes it possible to rapidly generate the desired implicit surfaces. We have implemented a mobile application to demonstrate that real-time interactions with complex liquid effects are possible with our approach.

Code for deep-learning based subgrid flow online now

We’ve just (i.e. finally) released the code for our SCA 2018 paper on learning sub-grid detail for Navier-Stokes (liquid) simulations with a stochastic deep-learning model. Our approach learns to predict the probability and a Gaussian distribution for under-resolved splash formations. It’s a good example from the larger field of “physics-based deep learning” techniques to enhance physics simulations with the help of neural network techniques. The code comes with a data generator based on our mantaflow framework, and TensorFlow code to train the neural network predictor.

You can check it out on github:https://github.com/kiwonum/mlflip

The corresponding paper and video can be found here.

Full abstract: This paper proposes a new data-driven approach to model detailed splashes for liquid simulations with neural networks. Our model learns to generate small-scale splash detail for the fluid-implicit-particle method using training data acquired from physically parameterized, high resolution simulations. We use neural networks to model the regression of splash formation using a classifier together with a velocity modifier. For the velocity modification, we employ a heteroscedastic model. We evaluate our method for different spatial scales, simulation setups, and solvers. Our simulation results demonstrate that our model significantly improves visual fidelity with a large amount of realistic droplet formation and yields splash detail much more efficiently than finer discretizations.

New Results from our Spatio-temporal Super-Resolution GAN (TecoGAN)

We’ve been working on new examples with our deep-learning based video super-resolution method (TecoGAN) that employ a novel spatio-temporal discriminators. Enjoy! These examples nicely highlight the huge amount of coherent detail that our method generates via a GAN-based training of the generator. And we’re of course still working on publishing the source code and trained models, coming up soon…

 

If you’re interested in the details, you can read the full pre-print here: https://ge.in.tum.de/publications/2019-tecogan-chu/

Or you can check out the accompanying paper video here:

Latent-space Physics Paper Video Finally Online on YouTube

The video for our latent-space physics paper is finally online! It’s been a while, the first paper version was on online on arXiv in February 2018 🙂 The paper will now be presented at Eurographics 2019 in Genoa.

Abstract: Our work explores methods for the data-driven inference of temporal evolutions of physical functions with deep learning techniques. More specifically, we target Navier-Stokes / fluid flow problems, and we propose a novel network architecture to predict the changes of the pressure field over time. The central challenge in this context is the high dimensionality of Eulerian space-time data sets. Key for arriving at a feasible algorithm is a technique for dimensionality reduction based on convolutional neural networks, as well as a special architecture for temporal prediction. We demonstrate that dense 3D+time functions of physics system can be predicted with neural networks, and we arrive at a neural-network based simulation algorithm with practical speed-ups. We demonstrate the capabilities of our method with a series of complex liquid simulations, and with a set of single-phase simulations. Our method predicts pressure fields very efficiently. It is more than two orders of magnitudes faster than a regular solver. Additionally, we present and discuss a series of detailed evaluations for the different components of our algorithm.

More detailed infos can be found here

ERC Proof of Concept Grant for a Data-driven Fluid Flow Solving Platform

We’re very happy to report that the Thuerey research group has very recently been award a so-called “Proof of Concept” grant by the European Research Council (ERC).

We will leverage deep convolutional neural networks (CNNs) with physically-based architectures and loss functions for a first deep learning based flow solver. From the ERC Starting Grant realFlow a first algorithmic realization exists, which provides the core technology that will be taken to the next level within this PoC. Specifically, we plan to employ this technology for a prediction of Reynolds-averaged turbulence flows in order to achieve interactive runtimes for complex simulations that previously took long computing times. However, instead of aiming for general purpose solvers, we will target specific application areas with targeted trained models. This technology has the potential to fundamentally change the way designers and engineers can work with physics simulations to get feedback for their designs. It will also make these simulations available to smaller companies in the value chain that previously were not able to fund and maintain complex simulators.

In parallel, our goal is to establish an open platform for exchanging data and trained models for physics simulations. We believe that open standards will on the one hand support the adoption of the new technology, while at the same time providing publicity and marketing opportunities for products to be developed alongside this platform. In particular, the deep learning based turbulence solver will make use of the open data and model formats. In the long run, this will make it possible to incorporate the trained model into new applications, e.g., for solving inverse problems in the context of flow simulations.

Pre-publication: Deep Learning Methods for Reynolds-Averaged Navier-Stokes Simulations

Details: https://erc.europa.eu/news/proof-concept-erc-awards-60-grants-innovation

 

Magic Fluid Control Animation – A Classic…

This animation is more than 10 years old, and one of our first works on fluid control (back then using Lattice-Boltzmann and SPH to simulate the fluid with a free surface). It to does not include any deep-learning or conv-nets – despite this, it’s still fun and worth a look 🙂 Enjoy!

On YouTube:

 

Fluid Simulations and Deep Learning at ICLR’19

Our work on “Generating Liquid Simulations with Deformation-aware Neural Networks” has been conditionally accepted at the International Conference on Learning Representations (ICLR), and will be presented there in May.

It focuses on an approach to pre-compute solution spaces for free-surface Navier-Stokes with deformation learning. The first version of our work appeared in April 2017, so video, paper and the accompanying demo Android app can all be already found online. More information here.

Abstract
Liquids exhibit complex non-linear behavior under changing simulation conditions such as user interactions. We propose a method to map this complex behavior over a parameter range onto reduced representation based on space-time deformations. In order to represent the complexity of the full space of inputs, we leverage the power of generative neural networks to learn a reduced representation. We introduce a novel deformation-aware loss function, which enables optimization in the highly non-linear space of multiple deformations. To demonstrate the effectiveness of our approach, we showcase the method with several complex examples in two and four dimensions. Our representation makes it possible to generate implicit surfaces of liquids very efficiently, which makes it possible to display the scene from any angle, and to add secondary effects such as particle systems. We have implemented a mobile application for our full output pipeline to demonstrate that real-time interaction is possible with our approach.

Learning temporal predictions and reduced representations at EG’19

Our two papers on learning temporal predictions and reduced representations for fluids have been accepted to the CGF Journal and will be presented at Eurographics 2019 in Milano! Congratulations to Steffen, Moritz, Byungsoo and Vinicius!
Our work explores methods for the data-driven inference of temporal evolutions of physical functions with deep learning techniques. More specifically, we target fluid flow problems, and we propose a novel network architecture to predict the changes of the pressure field over time. The central challenge in this context is the high dimensionality of Eulerian space-time data sets. Key for arriving at a feasible algorithm is a technique for dimensionality reduction based on convolutional neural networks, as well as a special architecture for temporal prediction. We demonstrate that dense 3D+time functions of physics system can be predicted with neural networks, and we arrive at a neural-network based simulation algorithm with practical speed-ups. We demonstrate the capabilities of our method with a series of complex liquid simulations, and with a set of single-phase simulations. Our method predicts pressure fields very efficiently. It is more than two orders of magnitudes faster than a regular solver. Additionally, we present and discuss a series of detailed evaluations for the different components of our algorithm.
This paper presents a novel generative model to synthesize fluid simulations from a set of reduced parameters. A convolutional neural network is trained on a collection of discrete, parameterizable fluid simulation velocity fields. Due to the capability of deep learning architectures to learn representative features of the data, our generative model is able to accurately approximate the training data set, while providing plausible interpolated in-betweens. The proposed generative model is optimized for fluids by a novel loss function that guarantees divergence-free velocity fields at all times. In addition, we demonstrate that we can handle complex parameterizations in reduced spaces, and advance simulations in time by integrating in the latent space with a second network. Our method models a wide variety of fluid behaviors, thus enabling applications such as fast construction of simulations, interpolation of fluids with different parameters, time re-sampling, latent space simulations, and compression of fluid simulation data. Reconstructed velocity fields are generated up to 700x faster than traditional CPU solvers, while achieving compression rates of over 1300x.

Deep Learning for Graphics Course at SIGGRAPH Asia 2018

We yesterday held our course on Deep Learning for Graphics Course at SIGGRAPH Asia 2018 in Tokyo. The slides are now available online at:

http://geometry.cs.ucl.ac.uk/creativeai/

Abstract: In computer graphics, many traditional problems are now better handled by deep-learning based data-driven methods. In an increasing variety of problem settings, deep networks are state-of-the-art, beating dedicated hand-crafted methods by significant margins. This tutorial gives an organized overview of core theory, practice, and graphics-related applications of deep learning.

 

Video Super-resolution with Deep Learning

Our work on video super-resolution with GANs is online now as a preview. The main trick is a special discriminator CNN that learns to supervise in terms of detail as well as temporal coherence. In addition, we propose a novel set of metrics for quantifying temporal coherence in videos. Enjoy 🙂 !

Abstract: Adversarial training has been highly successful in the context of image super-resolution. It was demonstrated to yield realistic and highly detailed results. Despite this success, many state-of-the-art methods for video super-resolution still favor simpler norms such as L_2 over adversarial loss functions. This is caused by the fact that the averaging nature of direct vector norms as loss functions leads to temporal smoothness. The lack of spatial detail means temporal coherence is easily established. In our work, we instead propose an adversarial training for video super-resolution that leads to temporally coherent solutions without sacrificing spatial detail.

In our generator, we use a recurrent, residual framework that naturally encourages temporal consistency. For adversarial training, we propose a novel spatio-temporal discriminator in combination with motion compensation to guarantee photo-realistic and temporally coherent details in the results. We additionally identify a class of temporal artifacts in these recurrent networks, and propose a novel Ping-Pong loss to remove them. Quantifying the temporal coherence for image super-resolution tasks has also not been addressed previously. We propose a first set of metrics to evaluate the accuracy as well as the perceptual quality of the temporal evolution, and we demonstrate that our method outperforms previous work by yielding realistic and detailed images with natural temporal changes.

Physics-based Deep Learning at NIPS 2018

We will be presenting our recent works on physics-based deep learning for fluid flow at the NIPS 2018 workshop on “Modeling the Physical World: Learning, Perception, and Control“, organized by Jiajun Wu, Kelsey Allen, Kevin Smith, Jessica Hamrick, Emmanuel Dupoux, Marc Toussaint, and Joshua Tenenbaum.

NIPS Conference: https://nips.cc

NIPS 2018 Workshop “Modeling the Physical World: Learning, Perception, and Control”: https://nips.cc/Conferences/2018/Schedule?showEvent=10931

Workshop homepage: http://phys2018.csail.mit.edu/submission.html

In particular we will discuss our works on:

Detailed abstracts:

Latent-space Physics: Towards Learning the Temporal Evolution of Fluid Flow: Our work explores methods for the data-driven inference of temporal evolutions of physical functions with deep learning techniques. More specifically, we target fluid flow problems, and we propose a novel network architecture to predict the changes of the pressure field over time. The central challenge in this context is the high dimensionality of Eulerian space-time data sets. Key for arriving at a feasible algorithm is a technique for dimensionality reduction based on convolutional neural networks, as well as a special architecture for temporal prediction. We demonstrate that dense 3D+time functions of physics system can be predicted with neural networks, and we arrive at a neural-network based simulation algorithm with practical speed-ups. We demonstrate the capabilities of our method with a series of complex liquid simulations, and with a set of single-phase simulations. Our method predicts pressure fields very efficiently. It is more than two orders of magnitudes faster than a regular solver. Additionally, we present and discuss a series of detailed evaluations for the different components of our algorithm.

Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow: We propose a temporally coherent generative model addressing the super-resolution problem for fluid flows. Our work represents the first approach to synthesize four-dimensional physics fields with neural networks. Based on a conditional generative adversarial network that is designed for the inference of three-dimensional volumetric data, our model generates consistent and detailed results by using a novel temporal discriminator, in addition to the commonly used spatial one. Our experiments show that the generator is able to infer more realistic high-resolution details by using additional physical quantities, such as low-resolution velocities or vorticities. Besides improvements in the training process and in the generated outputs, these inputs offer means for artistic control as well. We additionally employ a physics-aware data augmentation step, which is crucial to avoid overfitting and to reduce memory requirements. In this way, our network learns to generate advected quantities with highly detailed, realistic, and temporally coherent features. Our method works instantaneously, using only a single time-step of low-resolution fluid data. We demonstrate the abilities of our method using a variety of complex inputs and applications in two and three dimensions.

Coupled Fluid Density and Motion from Single Views: We present a novel method to reconstruct a fluid’s 3D density and motion based on just a single sequence of images. This is rendered possible by using powerful physical priors for this strongly under-determined problem. More specifically, we propose a novel strategy to infer density updates strongly coupled to previous and current estimates of the flow motion. Additionally, we employ an accurate discretization and depth-based regularizers to compute stable solutions. Using only one view for the reconstruction reduces the complexity of the capturing setup drastically and could even allow for online video databases or smart-phone videos as inputs. The reconstructed 3D velocity can then be flexibly utilized, e.g., for re-simulation, domain modification or guiding purposes. We will demonstrate the capacity of our method with a series of synthetic test cases and the reconstruction of real smoke plumes captured with a Raspberry Pi camera.

 

Reynolds-averaged Navier Stokes Simulations with Deep Learning

Our paper on Reynolds-averaged Navier Stokes simulations with deep learning (i.e. convolutional neural networks) is online now. You can read it on arXiv:
https://arxiv.org/abs/1810.08217

The full source code and a first training data set with 6400 RANS simulations as ground truth data is available here:
https://github.com/thunil/Deep-Flow-Prediction

Deep Flow Prediction is a pytorch framework for fluid flow (Reynolds-averaged Navier Stokes) predictions with deep learning. It contains code for data generation, network training, and evaluation for the aforementioned paper. A brief manual for the ode can be found in the github repository.

Paper Details:
Title: Well, how accurate is it? A Study of Deep Learning Methods for Reynolds-Averaged Navier-Stokes Simulations

Authors: Nils Thuerey, Konstantin Weissenow, Harshit Mehrotra, Nischal Mainali, Lukas Prantl, Xiangyu Hu

Abstract: With this study we investigate the accuracy of deep learning models for the inference of Reynolds-Averaged Navier-Stokes solutions. We focus on a modernized U-net architecture, and evaluate a large number of trained neural networks with respect to their accuracy for the calculation of pressure and velocity distributions. In particular, we illustrate how training data size and the number of weights influence the accuracy of the solutions. With our best models we arrive at a mean relative pressure and velocity error of less than 3% across a range of previously unseen airfoil shapes. In addition all source code is publicly available in order to ensure reproducibility and to provide a starting point for researchers interested in deep learning methods for physics problems. While this work focuses on RANS solutions, the neural network architecture and learning setup are very generic, and applicable to a wide range of PDE boundary value problems on Cartesian grids.

Details

Real-time Fluid Simulations at the TUM Open House 2018

We’are currently (as of Oct. 13) demoing our deformation neural network app and real-time fluid solver at the TUM open house 2018.

If you have time today – stop by at Garching Forschungszentrum 🙂

Otherwise you can check out the corresponding papers:

You can of course also check out our “Neural Liquid Drop” App on the Android app store.

SIGGRAPH 2018 tempoGAN talk on YouTube

You Xie and Rachel Chu just successfully presented their paper “tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow” at SIGRGAPH 2018. You can enjoy their full presentation via the YouTube video below.

Presentation:

More info regarding the SIGGRAPH 2018 technical papers:

Technical Papers

 

 

tempoGAN paper abstract:

We propose a temporally coherent generative model addressing the super-resolution problem for fluid flows. Our work represents the first approach to synthesize four-dimensional physics fields with neural networks. Based on a conditional generative adversarial network that is designed for the inference of three-dimensional volumetric data, our model generates consistent and detailed results by using a novel temporal discriminator, in addition to the commonly used spatial one. Our experiments show that the generator is able to infer more realistic high-resolution details by using additional physical quantities, such as low-resolution velocities or vorticities. Besides improvements in the training process and in the generated outputs, these inputs offer means for artistic control as well. We additionally employ a physics-aware data augmentation step, which is crucial to avoid overfitting and to reduce memory requirements. In this way, our network learns to generate advected quantities with highly detailed, realistic, and temporally coherent features. Our method works instantaneously, using only a single time-step of low-resolution fluid data. We demonstrate the abilities of our method using a variety of complex inputs and applications in two and three dimensions.

 

tempoGAN source code now online

The source code for our tempoGAN project, which aims for super-resolution inference of Navier-Stokes transport phenomena (such as smoke clouds) is online now at: https://github.com/thunil/tempoGAN

It comes with a readme, data generation scripts, and should give an easy starting point for training generative adversarial nets for fluids. If you try it, let us know how it works!

Project page

Here’s again the full abstract of the paper: We propose a temporally coherent generative model addressing the super-resolution problem for fluid flows. Our work represents the first approach to synthesize four-dimensional physics fields with neural networks. Based on a conditional generative adversarial network that is designed for the inference of three-dimensional volumetric data, our model generates consistent and detailed results by using a novel temporal discriminator, in addition to the commonly used spatial one. Our experiments show that the generator is able to infer more realistic high-resolution details by using additional physical quantities, such as low-resolution velocities or vorticities. Besides improvements in the training process and in the generated outputs, these inputs offer means for artistic control as well. We additionally employ a physics-aware data augmentation step, which is crucial to avoid overfitting and to reduce memory requirements. In this way, our network learns to generate advected quantities with highly detailed, realistic, and temporally coherent features. Our method works instantaneously, using only a single time-step of low-resolution fluid data. We demonstrate the abilities of our method using a variety of complex inputs and applications in two and three dimensions.

Best-paper award for “Coupled Fluid Density and Motion from Single Views” at SCA 2018

Congratulations to Marie-Lena Eckert for winning the best-paper award at SCA 2018 for her submission “Coupled Fluid Density and Motion from Single Views”.

Her work aims for reconstructing fluid flow phenomena with strong Navier-Stokes priors. This makes it possible to compute dense flow fields based on only a monocular video, i.e., an image sequence from a single viewpoint.

Paper Abstract: We present a novel method to reconstruct a fluid’s 3D density and motion based on just a single sequence of images. This is rendered possible by using powerful physical priors for this strongly under-determined problem. More specifically, we propose a novel strategy to infer density updates strongly coupled to previous and current estimates of the flow motion. Additionally, we employ an accurate discretization and depth-based regularizers to compute stable solutions. Using only one view for the reconstruction reduces the complexity of the capturing setup drastically and could even allow for online video databases or smart-phone videos as inputs. The reconstructed 3D velocity can then be flexibly utilized, e.g., for re-simulation, domain modification or guiding purposes.

More information about the ACM SIGGRAPH / Eurographics Symposium on Computer Animation: (SCA): http://sca2018.inria.fr

Coupled Fluid Density and Motion from Single Views