Software Structure
==================

This program provides an abstraction layer for linear algebra operations and an inversion system working with that abstraction layer.

Simplified overview
-------------------
The simplified main structure is explained here for the typical use case.

.. _Fig. 1:
.. mermaid::
   :caption: Simplified DUBFI program flow when using the linear algebra implementation ``mpipad``. The four columns distinguish parts of the program that use different abstraction layers. The left column marks the core module (:py:mod:`dubfi.core`) with five steps of the main program. The Bayesian inversion solver (second column, :py:mod:`dubfi.inversion`) contains the mathematical formulation of the inversion problem. It uses abstract linear algebra types. The implementation of these types (third column, :py:mod:`dubfi.linalg`) delegates the numerically expensive tasks to worker processes (right column). Blue arrows indicate ownership and red color marks objects that are deleted in step (3) after reading the input data and before running the inversion.

   ---
   config:
     theme: redux-color
   ---
   swimlane-beta TD
     subgraph core [Core]
       dat["(1) dat = DataProvider(...)"]
       inv["(2) inv = Invertor(dat.y, dat.H, dat.R, ...)"]
       del["(3) del dat"]
       solvec["(4) inv.solve()"]
       save["(5) save to netCDF"]
     end
     subgraph Invertor
       invi[Invertor]
       solvei["solve()"]
       cost["cost(), cost_grad(), cost_hess()"]
     end
     subgraph main [Linear algebra and DataProvider on main process]
       datm[DataProvider: Read input coordinates; Split observation space; Instruct workers]
       lam[y, H, R]
     end
     subgraph worker [Linear algebra and DataProvider on worker process]
       datw["Worker DataProvider\n(reads input data)"]
       law["Worker y, H, R\n(computation intensive tasks)"]
     end
     dat --> datm
     datm -->|initializes, owns, cleans up| datw
     datm --->|creates| lam
     datw --->|creates| law
     del --x datm
     solvec === solvei --->|uses| cost
     cost -->|uses| lam
     lam -->|owns, controls| law
     inv --> invi --- solvei
     invi -->|takes ownership| lam
     dat ----> inv ----> del ----> solvec ----> save
     style dat stroke:#f00
     style del stroke:#f00
     style datm stroke:#f00
     style datw stroke:#f00
     style worker fill:#eee,stroke:#aaa
     linkStyle 4 stroke:#f00,stroke:stroke-dasharray: 0 0 8 4
     linkStyle 0 stroke:#00f
     linkStyle 1 stroke:#00f
     linkStyle 8 stroke:#00f
     linkStyle 9 stroke:#00f
     linkStyle 11 stroke:#00f

We follow the reduced main program flow in `Fig. 1`_.

1. When starting the program, the main function (:py:func:`dubfi.core.core.run_inversion`, 1st column) first creates the DataProvider object (:py:class:`dubfi.dataprovider.generic.AbstractDataProvider`, 3rd column).
   This object is responsible for reading the input data and providing all required variables in observation space:
   The model-observation mismatch vector y, the observation operator H, and the error covariance matrix R of y (all in 3rd column).
   When using the ``mpipad`` linear algebra implementation (which is the default), the objects in the 3rd column delegate their work to worker processes (4th column).
2. Using y, H and R from the data provider, the main function creates an :py:class:`dubfi.inversion.inversion.Invertor` object.
   The invertor takes ownership of y, H and R.
3. The data provider is deleted. This deletes also the worker process data provider.
4. The invertor solves the Bayesian inversion problem numerically.
   It works with y, H and R using the abstract linear algebra interface (implementation in 3rd column).
   Thus, for the invertor it is irrelevant that the data of y, H and R are distributed over many worker processes (4th column).
5. When the inversion problem is solved, the main function saves the result.

The full program involves more steps such as reading the configuration, checking existing output files, computing a first guess for the numerical solver, looping over multiple values of ``norm_prefactor``, saving intermediate results, and saving additional diagnostics.
Many details are left out, for example the distribution of the state space to the worker processes.
But the main idea of DUBFI is depicted in `Fig. 1`_.

Linear Algebra
--------------
The modules :py:mod:`dubfi.linalg.types` and :py:mod:`dubfi.linalg.generic` provide abstract types for vectors, linear operators, and parametrized vectors or linear operators.
The inversion operates on this abstraction layer, i.e., it can use any implementation of the linear algebra.
Linear algebra implementations based on this abstraction layer shall be subpackages of :py:mod:`dubfi.linalg`.

Data Providers
--------------
The subpackage :py:mod:`dubfi.dataprovider` contains functions for reading and filtering input data.
The abstract interface to observation data for the inversion system is defined in the module :py:mod:`dubfi.dataprovider.generic`.
The implementation of the data provider depends on the linear algebra implementation.
This allows asynchronous reading of data for parallelized linear algebra implementations.
Each linear algebra implementation shall therefore provide its own subpackage of :py:mod:`dubfi.dataprovider`.


Generic Inversion
-----------------
The equations solved by the inversion are described in detail in the :doc:`scientific documentation </scidoc/inversion>`.
The inversion mainly consists of optimizing a cost function.
The cost function, its gradient and Hesse matrix are implemented in :py:mod:`dubfi.inversion.inversion` using the abstract linear algebra interface.
Other modules in :py:mod:`dubfi.inversion` implement different a-priori probability density functions for the Bayesian inversion.