Deep learning of dynamics and signal-noise decomposition with time-stepping constraints

• Published in: Journal of computational physics Vol. 396; pp. 483 - 506
• Main Authors: Rudy, Samuel H., Nathan Kutz, J., Brunton, Steven L.
• Format: Journal Article
• Language: English
• Published: Cambridge Elsevier Inc 01-11-2019; Elsevier Science Ltd
• Subjects: Algorithms; Artificial neural networks; Computational physics; Constraining; Data-driven models; Deep learning; Dynamical systems; Error analysis; Fields (mathematics); Machine learning; Measurement methods; Modelling; Neural networks; Noise; Noise measurement; Noise reduction; Production methods; Runge-Kutta method; Source code; System identification; Deep learning; Data-driven models; System identification; Neural networks; Dynamical systems; Machine learning
• ISSN: 0021-9991; 1090-2716
• DOI: 10.1016/j.jcp.2019.06.056

A critical challenge in the data-driven modeling of dynamical systems is producing methods robust to measurement error, particularly when data is limited. Many leading methods either rely on denoising prior to learning or on access to large volumes of data to average over the effect of noise. We propose a novel paradigm for data-driven modeling that simultaneously learns the dynamics and estimates the measurement noise at each observation. By constraining our learning algorithm, our method explicitly accounts for measurement error in the map between observations, treating both the measurement error and the dynamics as unknowns to be identified, rather than assuming idealized noiseless trajectories. We model the unknown vector field using a deep neural network, imposing a Runge-Kutta integrator structure to isolate this vector field, even when the data has a non-uniform timestep, thus constraining and focusing the modeling effort. We demonstrate the ability of this framework to form predictive models on a variety of canonical test problems of increasing complexity and show that it is robust to substantial amounts of measurement error. We also discuss issues with the generalizability of neural network models for dynamical systems and provide open-source code for all examples.
•Presenting a novel method to simultaneously learn measurement noise and dynamics.•Embedding network in a numerical integrator for learning from unevenly spaced data.•A discussion of apparent pitfalls with using neural networks.