1. Data assimilation - the collaborative research centre SFB1294
  2. Research
  3. Research Area A
  4. A02

A02 – Long-time stability and accuracy of ensemble transform filter algorithms

The project is concerned with sequential data assimilation in the context of state estimation for stochastic processes and more generally speaking sequential learning algorithms. A closed-form solution of the associated Bayesian filtering and smoothing problems are in most cases unavailable and numerical methods such as sequential Monte Carlo (SMC) methods have to be considered. While these methods are well understood theoretically and have found numerous applications they also suffer from the curse of dimensionality. In recent years there therefore has been a growing interest in developing approximate filter and smoothing algorithms that are applicable to high-dimensional data assimilation problems such as those arising in the geoscience, numerical weather prediction and the cognitive sciences. Here we mention in particular the popular class of ensemble Kalman filters (EnKFs) and smoothers.

During the first funding period, we have investigated the time continuous limiting equations of a large class of EnKFs as well as the long-time behaviour of the resulting time-continuous, so called, ensemble Kalman-Bucy filter (EnKBF) equations. In a further line of research, we have studied mean-field formulations for the general time-continuous filtering problem including alternative numerical implementations and extensions to filtering problems with correlated noise. The extension of such mean-field approaches to discrete-time observations leads to a sequence of Schrödinger bridge (i.e. stochastic optimal control) problems. An essential building block for an efficient numerical implementation of such stochastic optimal control problems is provided by a novel class of smoothing algorithms, which have also been developed during the first funding period.

During the second funding period, the mean-field limits of the EnKBF and its nonlinear extensions, such as the feedback particle filter, will be systematically studied in terms of well-posedness and asymptotic consistency. The long-time accuracy and stability as well as the computational efficiency of finite ensemble-size implementations will also be further investigated. A second main line of research is provided by the newly proposed Schrödinger approach to the assimilation of discrete-time observations.  Since the computation of Schrödinger bridges and their associated control laws is computationally demanding a main focus will be on alternative, more efficient implementations using previous results from this project and project A06 on gradient log-density estimators.

The project is central for many research topics of the CRC and will closely interact in particular with projects A01, A03, A06, B03, B06, B08, and B09.

 

  • Kim, J. W. and Mehta, P. G. (2024): Arrow of Time in Estimation and Control: Duality Theory Beyond the Linear Gaussian Model. arXiv 2405.07650

  • Kim, J. W., Joshi, A. A. and Mehta, P. G. (2024): Backward Map for Filter Stability Analysis. arXiv 2405.01127

  • Kim, J. W., Taghvaei, A. and Mehta, P. G. (2024): Divergence metrics in the study of Markov and hidden Markov processes. arXiv 2404.15779

  • Cherepanov, V., and Ertel, S. W. (2024): Neural Networks-based Random Vortex Methods for Modelling Incompressible Flows. arXiv: 2405.13691

  • Reich, S. (2023): A particle-based Algorithm for Stochastic Optimal Control. arXiv 2311.06906

  • Kim, J. W. and Mehta, P. G. (2023). Variance Decay Property for Filter Stability. arXiv 2305.12850

  • Kim, J.W. and Reich, S. (2023): On forward-backward SDE approaches to continuousßtime minimum variance estimation. arXiv 2304.12727

  • Reich, S. (2022): Data assimilation: A dynamic homotopy-based coupling approach. arXiv 2209.05279

  • Calvello, E., Reich, S. and Stuart A.M.(2022): Ensemble Kalman methods: A mean field approach. arXiv 2209.11371

  • Lange, T. (2020): Derivation of Ensemble Kalman-Bucy Filters with unbounded nonlinear coefficients. arXiv 2012.07572

  • Pathiraja, S. (2020): L2 convergence of smooth approximations of Stochastic Differential Equations with unbounded coefficients. arXiv 2011.13009

  • Lange, T. and Stannat, W. (2019): On the continuous time limit of Ensemble Square Root Filters. arXiv 1910.12493

  • Kim, J. W. and Mehta, P. G. (2024): Variance Decay Property for Filter Stability. IEEE Transactions on Automatic Control, doi: 10.1109/TAC.2024.3413573

  • Ertel, S.E. and Stannat, W. (2024): Analysis of the ensemble Kalman-Bucy filter for correlated observation noise. Ann. Appl. Probab. 34(1B), 1072-1107, doi: 10.1214/23-AAP1985.

  • Pathiraja, S. (2023): L2 convergence of smooth approximations of Stochastic Differential Equations with unbounded coefficients. Stochastic Analysis and Applications, 42, 354-369. doi: 0.1080/07362994.2023.2260863

  • Reich, S. (2024): Data Assimilation: A Dynamic Homotopy-Based Coupling Approach. In: Chapron, B., Crisan, D., Holm, D., Mémin, E., Radomska, A. (eds) Stochastic Transport in Upper Ocean Dynamics II. STUOD 2022. Mathematics of Planet Earth, vol 11. Springer, Cham. doi: 10.1007/978-3-031-40094-0_12

  • Kim, J. W. and Mehta, P. G. (2023): Duality for Nonlinear Filtering II: Optimal Control. IEEE Transactions on Automatic Control. doi: 10.1109/TAC.2023.3279208

  • Kim, J. W. and Mehta, P. G. (2023): Duality for Nonlinear Filtering I: Observability. IEEE Transactions on Automatic Control. doi: 10.1109/TAC.2023.3279206

  • Pathiraja, S., and van Leeuwen, P. J. (2022): Multiplicative non-Gaussian model error estimation in data assimilation. Journal of Advances in Modeling Earth Systems, 14, e2021MS002564. doi: 10.1029/2021MS002564

  • Ruchi, S., Dubinkina, S. and de Wiljes, J. (2021): Fast hybrid tempered ensemble transform filter for Bayesian elliptical problems via Sinkhorn approximation. Nonlinear Processes in Geophysics, 28(1): 23-41. doi: 10.5194/npg-28-23-2021

  • Lange, T. (2021): Derivation of Ensemble Kalman-Bucy Filters with unbounded nonlinear coefficients. Nonlinearity, Vol. 35, 1061. doi: 10.1088/1361-6544/ac4337

  • Pathiraja, S., Reich, S., and Stannat, W. (2021): McKean-Vlasov SDEs in nonlinear filtering. SIAM Journal on Control and Optimization.  doi:10.1137/20M1355197, arXiv 2007.12658

  • Pathiraja, S. and Stannat, W. (2021): Analysis of the feedback particle filter with diffusion map based approximation of the gain. Foundations of Data Science. doi:10.3934/fods.2021023 arXiv:2109.02761

  • Wormell, C.L. and Reich, S. (2021): Spectral convergence of diffusion maps: Improved error bounds and an alternative normalisation. SIAM Journal Numerical Analysis,59, 1687-1734. arXiv 2006.02037; doi:10.1137/30M1344093

  • Lange, T. and Stannat W. (2021): Mean field limit of Ensemble Square Root filters - discrete and continuous time, Foundations of Data Science. doi: 10.3934/fods.2021003

  • Lange, T. and Stannat, W. (2020): On the continuous time limit of the Ensemble Kalman Filter. Mathematics of Computation, 40(327), 233-265. arXiv 1901.05204v1; doi:10.1090/mcom/3588

  • de Wiljes, J. and Tong, X. T (2020): Analysis of a localised nonlinear Ensemble Kalman Bucy Filter with complete and accurate observations. Nonlinearity, 33(9): 4752-4782 [2]  arXiv:1908.10580v3

  • de Wiljes, J., Pathiraja, S. and Reich, S. (2020): Ensemble transform algorithms for nonlinear smoothing problems. SIAM J. Scientific Computing, 42, A87-A114. arXiv:1901.06300; doi: 10.1137/19M1239544

  • Reich, S. (2019): Data assimilation: The Schrödinger perspective. Acta Numerica, 28, 635-711. arXiv:1807.08351; doi:10.1017/S0962492919000011

  • Leeuwen, P. J. v., Künsch, H.-R., Nerger, L., Potthast, R. and Reich, S. (2019): Particle filters for high-dimensional geoscience applications: a review. Quarterly J Royal Meteorlog. Soc., 145, 2335-2365. arXiv: 1807.10434v2 doi: 10.1002/qj.3551

  • Morzfeld, M. and Reich, S. (2018): Data assimilation: mathematics for merging models and data. Snapshots of modern mathematics from Oberwolfach, 11. doi: 10.14760/SNAP-2018-011-EN

  • de Wiljes, J., Reich, S. and Stannat, W. (2018): Long-Time Stability and Accuracy of the Ensemble Kalman--Bucy Filter for Fully Observed Processes and Small Measurement Noise. SIAM Journal on Applied Dynamical Systems, 17(2), 1152-1181. arXiv: 1612.06065; doi: 10.1137/17M1119056

  • Taghvaei, A., de Wiljes, J., Mehta, P. G. and Reich, S. (2017): Kalman filter and its modern extensions for the continuous-time nonlinear filtering problem. ASME Journal of Dynamical Systems, Measurement, and Control, 140(3), 030904. arXiv: 1702.07241; doi: 10.1115/1.4037780