A02 – Long-time stability and accuracy of ensemble transform filter algorithms
The project is concerned with sequential data assimilation in the context of state estimation for stochastic processes and more generally speaking sequential learning algorithms. A closed-form solution of the associated Bayesian filtering and smoothing problems are in most cases unavailable and numerical methods such as sequential Monte Carlo (SMC) methods have to be considered. While these methods are well understood theoretically and have found numerous applications they also suffer from the curse of dimensionality. In recent years there therefore has been a growing interest in developing approximate filter and smoothing algorithms that are applicable to high-dimensional data assimilation problems such as those arising in the geoscience, numerical weather prediction and the cognitive sciences. Here we mention in particular the popular class of ensemble Kalman filters (EnKFs) and smoothers.
During the first funding period, we have investigated the time continuous limiting equations of a large class of EnKFs as well as the long-time behaviour of the resulting time-continuous, so called, ensemble Kalman-Bucy filter (EnKBF) equations. In a further line of research, we have studied mean-field formulations for the general time-continuous filtering problem including alternative numerical implementations and extensions to filtering problems with correlated noise. The extension of such mean-field approaches to discrete-time observations leads to a sequence of Schrödinger bridge (i.e. stochastic optimal control) problems. An essential building block for an efficient numerical implementation of such stochastic optimal control problems is provided by a novel class of smoothing algorithms, which have also been developed during the first funding period.
During the second funding period, the mean-field limits of the EnKBF and its nonlinear extensions, such as the feedback particle filter, will be systematically studied in terms of well-posedness and asymptotic consistency. The long-time accuracy and stability as well as the computational efficiency of finite ensemble-size implementations will also be further investigated. A second main line of research is provided by the newly proposed Schrödinger approach to the assimilation of discrete-time observations. Since the computation of Schrödinger bridges and their associated control laws is computationally demanding a main focus will be on alternative, more efficient implementations using previous results from this project and project A06 on gradient log-density estimators.
The project is central for many research topics of the CRC and will closely interact in particular with projects A01, A03, A06, B03, B06, B08, and B09.
Kim, J.W. and Reich, S. (2023): On forward-backward SDE approaches to continuousßtime minimum variance estimation. arXiv 2304.12727
Kim, J. W. and Mehta, P. G. (2022): Duality for Nonlinear Filtering II: Optimal Control. arXiv 2208.06587
Kim, J. W. and Mehta, P. G. (2022): Duality for Nonlinear Filtering I: Observability. arXiv 2208.06586
Reich, S. (2022): Data assimilation: A dynamic homotopy-based coupling approach. arXiv 2209.05279
Calvello, E., Reich, S. and Stuart A.M.(2022): Ensemble Kalman methods: A mean field approach. arXiv 2209.11371
Lange, T. (2020): Derivation of Ensemble Kalman-Bucy Filters with unbounded nonlinear coefficients. arXiv 2012.07572
Pathiraja, S. (2020): L2 convergence of smooth approximations of Stochastic Differential Equations with unbounded coefficients. arXiv 2011.13009
Lange, T. and Stannat, W. (2019): On the continuous time limit of Ensemble Square Root Filters. arXiv 1910.12493
Pathiraja, S., & van Leeuwen, P. J. (2022). Multiplicative non-Gaussian model error estimation in data assimilation. Journal of Advances in Modeling Earth Systems, 14, e2021MS002564. https://doi.org/10.1029/2021MS002564
Ruchi, S., Dubinkina, S. and de Wiljes, J. (2021). Fast hybrid tempered ensemble transform filter for Bayesian elliptical problems via Sinkhorn approximation. Nonlinear Processes in Geophysics, 28(1): 23-41 
Pathiraja, S., Reich, S., Stannat, W. (2021): McKean-Vlasov SDEs in nonlinear filtering. SIAM Journal on Control and Optimization. doi:10.1137/20M1355197 arXiv 2007.12658
Pathiraja, S. and Stannat, W. (2021). Analysis of the feedback particle filter with diffusion map based approximation of the gain. Foundations of Data Science. doi:10.3934/fods.2021023 arXiv:2109.02761
Wormell, C.L. and Reich, S. (2021). Spectral convergence of diffusion maps: Improved error bounds and an alternative normalisation. SIAM Journal Numerical Analysis,59, 1687-1734. arXiv 2006.02037; doi:10.1137/30M1344093
Lange, T. and Stannat W. (2021). Mean field limit of Ensemble Square Root filters - discrete and continuous time, Foundations of Data Science. doi: 10.3934/fods.2021003
Lange, T. and Stannat, W. (2020): On the continuous time limit of the Ensemble Kalman Filter. Mathematics of Computation, 40(327), 233-265. arXiv 1901.05204v1; doi:10.1090/mcom/3588
de Wiljes, J. and Tong, X. T (2020). Analysis of a localised nonlinear Ensemble Kalman Bucy Filter with complete and accurate observations. Nonlinearity, 33(9): 4752-4782  arXiv:1908.10580v3
de Wiljes, J., Pathiraja, S. and Reich, S. (2020). Ensemble transform algorithms for nonlinear smoothing problems. SIAM J. Scientific Computing, 42, A87-A114. arXiv:1901.06300; doi: 10.1137/19M1239544
Reich, S. (2019). Data assimilation: The Schrödinger perspective. Acta Numerica, 28, 635-711. arXiv:1807.08351; doi:10.1017/S0962492919000011
Leeuwen, P. J. v., Künsch, H.-R., Nerger, L., Potthast, R. and Reich, S. (2019). Particle filters for high-dimensional geoscience applications: a review. Quarterly J Royal Meteorlog. Soc., 145, 2335-2365. arXiv: 1807.10434v2 doi: 10.1002/qj.3551
Morzfeld, M. and Reich, S. (2018). Data assimilation: mathematics for merging models and data. Snapshots of modern mathematics from Oberwolfach, 11. doi: 10.14760/SNAP-2018-011-EN
de Wiljes, J., Reich, S. and Stannat, W. (2018). Long-Time Stability and Accuracy of the Ensemble Kalman--Bucy Filter for Fully Observed Processes and Small Measurement Noise. SIAM Journal on Applied Dynamical Systems, 17(2), 1152-1181. arXiv: 1612.06065; doi: 10.1137/17M1119056
Taghvaei, A., de Wiljes, J., Mehta, P. G. and Reich, S. (2017). Kalman filter and its modern extensions for the continuous-time nonlinear filtering problem. ASME Journal of Dynamical Systems, Measurement, and Control, 140(3), 030904. arXiv: 1702.07241; doi: 10.1115/1.4037780