Author: GE_at_TUM

Our Paper on deep learning algorithms interacting with differentiable PDE solvers was just successfully presented at NeurIPS. And just in time for the conference, we also finished uploading the last piece of the corresponding source code release.

An extended version of our CG Solver-in-the-Loop results from our NeurIPS’20 paper is finally online at: https://github.com/tum-pbs/CG-Solver-in-the-Loop

The main code for the paper is also available at: https://github.com/tum-pbs/Solver-in-the-Loop

This is the full abstract of the paper: Finding accurate solutions to partial differential equations (PDEs) is a crucial task in all scientific and engineering disciplines. It has recently been shown that machine learning methods can improve the solution accuracy by correcting for effects not captured by the discretized PDE. We target the problem of reducing numerical errors of iterative PDE solvers and compare different learning approaches for finding complex correction functions. We find that previously used learning approaches are significantly outperformed by methods that integrate the solver into the training loop and thereby allow the model to interact with the PDE during training. This provides the model with realistic input distributions that take previous corrections into account, yielding improvements in accuracy with stable rollouts of several hundred recurrent evaluation steps and surpassing even tailored supervised variants. We highlight the performance of the differentiable physics networks for a wide variety of PDEs, from non-linear advection-diffusion systems to three-dimensional Navier-Stokes flows.

Additional details can be found on the project page.

In addition to our paper at the NeurIPS 2020 main conference (which targets deep learning via differentiable PDE solvers for numerical error reduction) we are excited about contributions to the following four NeurIPS workshops. Details will follow over the course of the next weeks, but these workshops very nicely align with our goals to fuse deep learning, numerical methods and physical simulations as seamlessly as possible. E.g., we will present our work on shape optimizations for Navier-Stokes flows as well as our differentiable physics framework phiflow.

For now, we can highly recommend checking out the workshops themselves:

• Differentiable Vision, Graphics, and Physics in Machine Learning
  http://montrealrobotics.ca/diffcvgp/
  Organizers: Krishna Jatavallabhula , Kelsey Allen , Victoria Dean , Johanna Hansen , Shuran Song , Florian Shkurti , Liam Paull , Derek Nowrouzezahrai , Josh Tenenbaum
• Interpretable Inductive Biases and Physically Structured Learning
  https://inductive-biases.github.io/
  Organizers: Shirley Ho , Michael Lutter , Alexander Terenin , Lei Wang
• Machine Learning for Engineering Modeling, Simulation, and Design
  https://ml4eng.github.io/
  Organizers: Alex Beatson , Priya L. Donti , Amira Abdel-Rahman , Stephan Hoyer , Rose Yu , J. Zico Kolter , Ryan P. Adam
• Machine Learning and the Physical Sciences https://ml4physicalsciences.github.io/2020/
  Organizers: Atılım Güneş Baydin , Juan Felipe Carrasquilla , Adji Bousso Dieng , Karthik Kashinath , Gilles Louppe , Brian Nord , Michela Paganini , Savannah Thais

Our paper “Numerical investigation of minimum drag profiles in laminar flow using deep learning surrogates” is online now as a preprint. It targets optimizing shapes by using trained deep neural networks models that infer manifolds of Navier-Stokes solutions. We evaluate accuracy and performance of using pretrained models to minimize the drag for shapes immersed in a moving fluid in low Reynolds number regimes.

Preprint on arXiv
Project page

Full abstract: Efficiently predicting the flowfield and load in aerodynamic shape optimisation remains a highly challenging and relevant task. Deep learning methods have been of particular interest for such problems, due to their success for solving inverse problems in other fields. In the present study, U-net based deep neural network (DNN) models are trained with high-fidelity datasets to infer flow fields, and then employed as surrogate models to carry out the shape optimisation problem, i.e. to find a drag minimal profile with a fixed cross-section area subjected to a two-dimensional steady laminar flow. A level-set method as well as B{\’e}zier-curve method are used to parameterise the shape, while trained neural networks in conjunction with automatic differentiation are utilized to calculate the gradient flow in the optimisation framework. The optimised shapes and drag force values calculated from the flowfields predicted by DNN models agree well with reference data obtained via a Navier-Stokes solver and from the literature, which demonstrates that the DNN models are capable of predicting not only flowfield but also yield satisfactory aerodynamic forces. This is particularly promising as the DNNs were not specifically trained to infer aerodynamic forces. In conjunction with the fast runtime, the DNN-based optimisation framework shows promise for general aerodynamic design problems.

Our paper on improved and controllable latent-space physics predictions is set to be shown at SCA 2020. It builds on our previous work from EG’19 (Latent-space Physics: Towards Learning the Temporal Evolution of Fluid Flow, Deep Fluids: A Generative Network for Parameterized Fluid Simulations) to improve temporal predictions within the LSTM neural network. In addition, it controls the latent-space content to allow for modifications and improved long-term stability. It will be shown at SCA 2020 soon.

The pre-print and repository can be found at here.

Full abstract: We propose an end-to-end trained neural network architecture to robustly predict the complex dynamics of fluid flows with high temporal stability. We focus on single-phase smoke simulations in 2D and 3D based on the incompressible Navier-Stokes (NS) equations, which are relevant for a wide range of practical problems. To achieve stable predictions for long-term flow sequences, a convolutional neural network (CNN) is trained for spatial compression in combination with a temporal prediction network that consists of stacked Long Short-Term Memory (LSTM) layers. Our core contribution is a novel latent space subdivision (LSS) to separate the respective input quantities into individual parts of the encoded latent space domain. This allows to distinctively alter the encoded quantities without interfering with the remaining latent space values and hence maximizes external control. By selectively overwriting parts of the predicted latent space points, our proposed method is capable to robustly predict long-term sequences of complex physics problems. In addition, we highlight the benefits of a recurrent training on the latent space creation, which is performed by the spatial compression network.

Our paper on improving neural network generalization via a forward-backward pass is also finally online, together with a first code example. A common question that we get about this project: “why racecar“? This is worth explaining here in a bit more detail: it’s not about the speed of the method, but rather racecar is a nice palindrome. Hence, if you reverse the word, you still have “racecar”. Our training approach also makes use of a reversed neural network architecture, re-using all existent building blocks of the network and their weights, somewhat similar to a palindrome. Hence the name. Interestingly, this reverse structure yields an embedding of singular vectors into the weight matrices, and improves performance for new tasks, as we show for a variety of classification and generation tasks in our paper.

• Preprint
• Code example

Paper Abstract:
We propose a novel training approach for improving the generalization in neural networks. We show that in contrast to regular constraints for orthogonality, our approach represents a data-dependent orthogonality constraint, and is closely related to singular value decompositions of the weight matrices. We also show how our formulation is easy to realize in practical network architectures via a reverse pass, which aims for reconstructing the full sequence of internal states of the network. Despite being a surprisingly simple change, we demonstrate that this forward-backward training approach, which we refer to as racecar training, leads to significantly more generic features being extracted from a given data set. Networks trained with our approach show more balanced mutual information between input and output throughout all layers, yield improved explainability and, exhibit improved performance for a variety of tasks and task transfers.

Our new paper “Purely data-driven medium-range weather forecasting achieves comparable skill to physical models at similar resolution” is available now on arXiv: https://arxiv.org/abs/2008.08626

We show that with enough data, a deep-learning based model can actually compete and in some cases outperform established physical models (e.g., IFS forecasts for 210km resolution). We show how such models can be trained based on the WeatherBench data set, that they contain plausible learned structures, and also fare well for challenging fields such precipitation. At the same time, they illustrate that it will be very difficult to increase the performance only with the data that is currently available.

Full abstract: Numerical weather prediction has traditionally been based on physical models of the atmosphere. Recently, however, the rise of deep learning has created increased interest in purely data-driven medium-range weather forecasting with first studies exploring the feasibility of such an approach. Here, we train a significantly larger model than in previous studies to predict geopotential, temperature and precipitation up to 5 days ahead and achieve comparable skill to a physical model run at similar horizontal resolution. Crucially, we pretrain our models on historical climate model output before fine-tuning them on the reanalysis data. We also analyze how the neural network creates its predictions and find that, with some exceptions, it is compatible with physical reasoning. Our results indicate that, given enough training data, data-driven models can compete with physical models. At the same time, there is likely not enough data to scale this approach to the resolutions of current operational models.

We’re happy to report that our paper on using differentiable physics to reduce errors in PDEs is online now, and a corresponding overview talk is also available now:
– Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers
– Differentiable Physics Simulations for Deep Learning, Talk by Nils Thuerey

Our results demonstrate that Differentiable Physics are a powerful tool, and they neatly fit into the current larger deep learning trend of generic “Differentiable Programming”. They not only yield very good minimizers in terms of well-trained neural networks: a nice side effect is that they allow for leveraging all the existing powerful numerical methods that exist for physical simulations, and employ them to improve training deep neural nets.

Solver-in-the-Loop Paper Abstract: Finding accurate solutions to partial differential equations (PDEs) is a crucial task in all scientific and engineering disciplines. It has recently been shown that machine learning methods can improve the solution accuracy by correcting for effects not captured by the discretized PDE. We target the problem of reducing numerical errors of iterative PDE solvers and compare different learning approaches for finding complex correction functions. We find that previously used learning approaches are significantly outperformed by methods that integrate the solver into the training loop and thereby allow the model to interact with the PDE during training. This provides the model with realistic input distributions that take previous corrections into account, yielding improvements in accuracy with stable rollouts of several hundred recurrent evaluation steps and surpassing even tailored supervised variants. We highlight the performance of the differentiable physics networks for a wide variety of PDEs, from non-linear advection-diffusion systems to three-dimensional Navier-Stokes flows.

We’re happy to report that the final version of our paper on “Learning Similarity Metrics for Numerical Simulations” to be presented at the International Conference on Machine Learning (ICML) is online now. We propose learning a metric for data produced by numerical simulations, i.e. PDEs such as Navier-Stokes, and a way to train Siamese networks with a correlation-based loss to improve the inference of similarities. The resulting deep learning based metric outperforms simpler metrics and other learned metrics such as LPIPS.

Assessing similarity for complex data is is a fundamental problem in all computational disciplines ranging from simulations of blood flow to aircraft design. Many practical problems rely on highly complex PDEs, where small perturbations in the input drastically alter the solutions. Regular vector space metrics like the L² distance are unreliable as they perform an element-wise comparison, and thus cannot compare contextual information or structures on different scales. Our approach, dubbed LSiM, employs convolutional neural networks (CNNs) as a method to extract and compare more meaningful features from a set of two simulation frames.

You can check out:
-the updated pre-print on arXiv 2002.07863 ,
– our website with further details, code, data etc.,
– or the full list of accepted ICML 2020 papers.

The video and full implementation for our ICLR 2020 paper “Lagrangian Fluid Simulation with Continuous Convolutions” are online now. It enables flexible and efficient learning of dynamics, e.g., for Lagrangian Navier-Stokes solves similar to Smoothed Particle Hydrodynamics (SPH). But instead of analytic kernel formulations like they’re used in SPH, our method learns the dynamics from data.

• Source code on github: https://github.com/intel-isl/DeepLagrangianFluids
• Youtube video with examples: https://www.youtube.com/watch?v=bhnuhhYBpx0

Full Abstract: We present an approach to Lagrangian fluid simulation with a new type of convo- lutional network. Our networks process sets of moving particles, which describe fluids in space and time. Unlike previous approaches, we do not build an ex- plicit graph structure to connect the particles but use spatial convolutions as the main differentiable operation that relates particles to their neighbors. To this end we present a simple, novel, and effective extension of N-D convolutions to the continuous domain. We show that our network architecture can simulate differ- ent materials, generalizes to arbitrary collision geometries, and can be used for inverse problems. In addition, we demonstrate that our continuous convolutions outperform prior formulations in terms of accuracy and speed.

It is worth highlighting that our benchmark dataset for data-driven weather forecasting, i.e. WeatherBench, is fully available online now. You can

• download all the data-sets via the TUM library,
• the sources (include a Jupiter notebook to get started) on GitHub,
• and check our the preprint of the corresponding paper on arXiv.
• In addition, Stephan has a nice overview on his blog.

Here’s the full abstract: Data-driven approaches, most prominently deep learning, have become powerful tools for prediction in many domains. A natural question to ask is whether data-driven methods could also be used for numerical weather prediction. First studies show promise but the lack of a common dataset and evaluation metrics make inter-comparison between studies difficult. Here we present a benchmark dataset for data-driven medium-range weather forecasting, a topic of high scientific interest for atmospheric and computer scientists alike. We provide data derived from the ERA5 archive that has been processed to facilitate the use in machine learning models. We propose a simple and clear evaluation metric which will enable a direct comparison between different methods. Further, we provide baseline scores from simple linear regression techniques, deep learning models as well as purely physical forecasting models. We hope that this dataset will accelerate research in data-driven weather forecasting.

Our CVPR paper titled “Correspondence-Free Material Reconstruction using Sparse Surface Constraints” is online now with a pre-print, source code, and the corresponding video! Enjoy. The paper proposes a method to optimize for solutions of a finite-element elastodynamics solver that match a set of given observations (in the form of a depth-video). This method does not employ any neural networks or deep learning methods for a change, but is nonetheless closely related due to its gradient-based optimization scheme.

Full abstract: We address the problem to infer physical material parameters and boundary conditions from the observed motion of a homogeneous deformable object via the solution of an inverse problem. Parameters are estimated from potentially unreliable real-world data sources such as sparse observations without correspondences. We introduce a novel Lagrangian-Eulerian optimization formulation, including a cost function that penalizes differences to observations during an optimization run. This formulation matches correspondence-free, sparse observations from a single-view depth sequence with a finite element simulation of deformable bodies. In conjunction with an efficient hexahedral discretization and a stable, implicit formulation of collisions, our method can be used in demanding situation to recover a variety of material parameters, ranging from Young’s modulus and Poisson ratio to gravity and stiffness damping, and even external boundaries. In a number of tests using synthetic datasets and real-world measurements, we analyse the robustness of our approach and the convergence behavior of the numerical optimization scheme.

All signals are GO for the upcoming ERC Consolidator Grant award to N. Thuerey – we’re happy to report that the official grant agreement between TUM and the ERC has been signed. The project is titled “Spatio-Temporal Methods for Data-driven Computer Animation and Simulation”, or in short SpaTe ( as in “spatio-temporal” methods). It will build on and extend our previous work on spatio-temporal learning and GANs (e.g. for fluids or videos), as well as differentiable physics simulations, such as phiflow, as data capturing ventures (cf. ScalarFlow). In terms of phenomena, we will not only target fluids or Navier-Stokes solutions, but also elastic and non-Newtonian materials. We believe that deep learning methods for differentiable numerical simulations represent an exciting area of research and will be of fundamental impact for a wide variety of scientific fields.

The following press release of TUM’s CS department gives a good summary: Prof. Nils Thuerey, receives funding in the form of a Consolidator Grant from the European Research Council (ERC). The goal of his “SpaTe” project is to teach physics to computers – using data and examples rather than the conventional equation-based method. Software that predicts how gases, liquids and solids change shape under certain conditions and over a certain period is used in an enormous number of industries and research projects. So far, these programs have required enormous processing power. Machine learning has the potential for flexible and realistic modelling of temporal processes. Very little research has taken place in this area, however. In his project, Prof. Nils Thuerey wants to develop new algorithms to put machine learning to use in physical simulations. In the future this might make it possible, for example, to detect automatically from video data which physical materials are present and analyze their behavior. For his research he has previously received an ERC Starting Grant and Proof of Concept Grant.

The official press release of the European Research Council can be found here.

We are happy to announce two upcoming spotlight presentations, and one poster at ICLR 2020:
– Tranquil Clouds: Neural Networks for Learning Temporally Coherent Features in Point Clouds
– Lagrangian Fluid Simulation with Continuous Convolutions , and
– Learning to Control PDEs with Differentiable Physics.
Not surprisingly, all three focus on physics-based deep learning techniques and Navier-Stokes problems – the first two of them employ Lagrangian methods, the while the third one focuses on differentiable simulations, with a particular focus on long-term temporal stability. Abstracts follow below…

Tranquil Clouds: Neural Networks for Learning Temporally Coherent Features in Point Clouds: Point clouds, as a form of Lagrangian representation, allow for powerful and flexible applications in a large number of computational disciplines. We propose a novel deep-learning method to learn stable and temporally coherent feature spaces for points clouds that change over time. We identify a set of inherent problems with these approaches: without knowledge of the time dimension, the inferred solutions can exhibit strong flickering, and easy solutions to suppress this flickering can result in undesirable local minima that manifest themselves as halo structures. We propose a novel temporal loss function that takes into account higher time derivatives of the point positions, and encourages mingling, i.e., to prevent the aforementioned halos. We combine these techniques in a super-resolution method with a truncation approach to flexibly adapt the size of the generated positions. We show that our method works for large, deforming point sets from different sources to demonstrate the flexibility of our approach.

Lagrangian Fluid Simulation with Continuous Convolutions: We present an approach to Lagrangian fluid simulation with a new type of convolutional network. Our networks process sets of moving particles, which describe fluids in space and time. Unlike previous approaches, we do not build an explicit graph structure to connect the particles but use spatial convolutions as the main differentiable operation that relates particles to their neighbors. To this end we present a simple, novel, and effective extension of N-D convolutions to the continuous domain. We show that our network architecture can simulate different materials, generalizes to arbitrary collision geometries, and can be used for inverse problems. In addition, we demonstrate that our continuous convolutions outperform prior formulations in terms of accuracy and speed.

Learning to Control PDEs with Differentiable Physics: Predicting outcomes and planning interactions with the physical world are long-standing goals for machine learning. A variety of such tasks involves continuous physical systems, which can be described by partial differential equations (PDEs) with many degrees of freedom. Existing methods that aim to control the dynamics of such systems are typically limited to relatively short time frames or a small number of interaction parameters. We show that by using a differentiable PDE solver in conjunction with a novel predictor-corrector scheme, we can train neural networks to understand and control complex nonlinear physical systems over long time frames. We demonstrate that our method successfully develops an understanding of complex physical systems and learns to control them for tasks involving multiple PDEs, including the incompressible Navier-Stokes equations.

Our fully differentiable physics-solving framework is online now at https://github.com/tum-pbs/PhiFlow. Having all functionality of, e.g., a fluid simulation running in TensorFlow opens up the possibility of back-propagating gradients through the simulation as well as running the simulation on GPUs.

PhiFlow (among others) supports the following things:

• Support for a variety of differentiable simulation types, from Burgers over Navier-Stokes to the Schrödinger equation.
• Tight integration with TensorFlow allowing for straightforward network training with fully differentiable simulations that run on the GPU.
• Object-oriented architecture enabling concise and expressive code, designed for ease of use and extensibility.
• Reusable simulation code, independent of backend and dimensionality, i.e. the exact same code can run a 2D fluid sim using NumPy and a 3D fluid sim on the GPU using TensorFlow.
• Flexible, easy-to-use web interface featuring live visualizations and interactive controls that can affect simulations or network training on the fly.

Our full scalarFlow data set, with almost half a terabyte size, is online now! You can request a download link via the following sign-up form. It contains more than 100 sequences of fluid flows reconstructed with a Navier-Stokes-based reconstruction algorithm. Also check out our data-set page which describes the full hardware setup. You “only” need a few Raspberry-Pi computers with cameras: https://ge.in.tum.de/publications/2019-scalarflow-eckert/

Paper Abstract: We present ScalarFlow, a first large-scale data set of reconstructions of real-world smoke plumes. In addition, we propose a framework for accurate physics-based reconstructions from a small number of video streams. Central components of our framework are a novel estimation of unseen inflow regions and an efficient optimization scheme constrained by a simulation to capture real-world fluids. Our data set includes a large number of complex natural buoyancy-driven flows. The flows transition to turbulence and contain observable scalar transport processes. As such, the ScalarFlow data set is tailored towards computer graphics, vision, and learning applications. The published data set will contain volumetric reconstructions of velocity and density as well as the corresponding input image sequences with calibration data, code, and instructions how to reproduce the commodity hardware capture setup. We further demonstrate one of the many potential applications: a first perceptual evaluation study, which reveals that the complexity of the reconstructed flows would require large simulation resolutions for regular solvers in order to recreate at least parts of the natural complexity contained in the captured data.

Focusing on temporally coherent detail synthesis, we present a range of new and existing research project that all employ spatio-temporal self-supervision with GANs. This concept is beneficial for a variety of challenging tasks: from video translations, to super-resolution for games, videos and for fluid flow effects (i.e. Navier-Stokes solutions).

• Video translation for unpaired data from different domains (Ours is shown as TecoGAN):
  

• Game super-resolution using strongly aliased input (Ours is shown as DRR):
  
  

• For video super-resolution, the TecoGAN model (Ours) yields coherent and realistic details:
  

• We also proposed an algorithm for 3D fluid super-resolution with a factor of eight in each dimension:
  

In all tasks, we found that temporal adversarial learning is key to achieving temporally coherent solutions without sacrificing spatial detail. Compared to adding traditional temporal losses (e.g., L1 or L2 distances of warped frames) to normal spatial GANs, spatio-temporal discriminators are able to  deal with more challenging learning objectives such as sharp detail over time.

Besides spatio-temporal GANs, we have developed different technologies to tackle individual challenges across these tasks. In unpaired video translation, we use curriculum learning for discriminators to achieve better adversarial equilibriums. For strongly aliased renderings in games, we propose depth-recurrent residual connections to learn stable temporal states. In video super-resolution, a bi-directional Ping-Pong loss is proposed to improve long-term temporal coherence. When processing large volumetric fluid data, a multi-pass GAN is used to break-down the data relationships from 3D+t to lower dimensions. Preprints of papers and codes will be available soon.

Further reading:
– TecoGAN
– Multi-Pass GAN
– tempoGAN

We are happy to announce a tighter research collaboration between the “Advanced Innovation Center for Future Visual Entertainment” of the Beijing Film Academy (BFA) and the Physics-Based Simulation group at TUM. Our joint goal is to develop the next generation of deep-learning based visual effects tools.

The BFA itself is a highly and internationally renowned film school that has, among others, alumni such as the director Zhang Yimou. The latter directed movies like “Hero” (2002), and the ceremonies for the Beijing Olympics (2008). The AICFVE research grou is headed by Prof. Baoquan Chen from Peking University.

Further reading:
http://fve.bfa.edu.cn/English.htm
https://www.china-admissions.com/beijing-film-academy/   
http://english.pku.edu.cn
https://cfcs.pku.edu.cn/baoquan/