dubfi.fluxes.core

Core module for running flux inversion based on in-situ observations.

Changed in version 0.1.2: (renamed module)

Added in version 0.1.0: (initial release)

Functions

_save_apriori(target_file, config, coords, segments, ...)

Save output file including all meta data with NaN for results.

_save_kalman(target_file, s_post, b_post, sensitivity, ...)

Add result data from simple Kalman smoother to existing netCDF file.

_save_aposteriori(target_file, s_post, b_post, ...)

Add result data to existing netCDF file.

run_inversion(config, target_dir[, init_from, ...])

Run inversion and save results.

Module Contents

dubfi.fluxes.core._save_apriori(target_file, config, coords, segments, s_prior, b_prior, implementation, s_prior_norm_pref=None, b_prior_norm_pref=None, cycling_start=None)

Save output file including all meta data with NaN for results.

Parameters:

  • target_file (str) – output file path. Existing files will be overwritten.

  • config (dict) – inversion configuration dictionary

  • coords (dict) – coordinates dictionary as produced by dubfi.fluxes.readobs.coordinates_from_config()

  • segments (dubfi.linalg.segments.Segments, optional) – segmentation of data used for shared memory parallelization

  • s_prior (np.ndarray) – prior scaling factors, aligned with coords[“flux_cat”]

  • b_prior (np.ndarray) – prior error covariance matrix of scaling factors (B) used for Kalman (simple) inversion.

  • implementation (str) – linear algebra implementation used for the inversion, saved as attributed

  • s_prior_norm_pref (np.ndarray, optional) – prior scaling factors for different norm prefactors, shape (norm_prefactor, flux_cat). If not provided, s_prior is used for all norm_prefactor values.

  • b_prior_norm_pref (np.ndarray, optional) – prior error covariance matrix of scaling factors (B) for different norm prefactors, shape (norm_prefactor, flux_cat, flux_cat). If not provided, b_prior is used for all norm_prefactor values.

  • cycling_start (str, optional) – start date and time of inversion cycle, will be included in output attributes

Return type:

None

Changed in version 0.1.1.

dubfi.fluxes.core._save_kalman(target_file, s_post, b_post, sensitivity, inv, chi2, chi2_obs, chi2_fit, ddof)

Add result data from simple Kalman smoother to existing netCDF file.

This function computes the model observation mismatch and its uncertainty. Results are saved to the existing file, assuming that this file was created using _save_apriori(). .. see also:: _save_apriori(), _save_aposteriori()

Parameters:

  • target_file (str) – output file path. Existing files will be overwritten.

  • s_post (np.ndarray) – posterior scaling factors (obtained using Kalman smoother), aligned with flux_cat coordinate in existing file

  • b_post (np.ndarray) – posterior error covariance matrix of scaling factors (B)

  • sensitivity (np.ndarray) – derivative of posterior scaling factors with respect to observations, array of shape (flux_cat, obs)

  • inv (InvertorOptimizer) – inversion object knowing about model observation mismatch, observation operator, and uncertainties

  • chi2 (float) – chi^2 value of the fit

  • chi2_obs (float) – observation contribution to chi^2 of the fit (positive)

  • chi2_fit (float) – scaling uncertainty contribution to chi^2 of the fit (negative)

  • ddof (int) – total degrees of freedom minus number of fit parameters

Return type:

None

Changed in version 0.1.1.

dubfi.fluxes.core._save_aposteriori(target_file, s_post, b_post, sensitivity, averaging_kernel, inv, idx)

Add result data to existing netCDF file.

Parameters:

  • target_file (str) – output file path. Existing files will be overwritten.

  • s_post (np.ndarray) – posterior scaling factors

  • b_post (np.ndarray) – posterior error covariance matrix of scaling factors (B)

  • sensitivity (np.ndarray) – derivative of posterior scaling factors with respect to observations, array of shape (flux_cat, obs)

  • averaging_kernel (np.ndarray) – derivative of posterior scaling factors with respect to true scaling factors, array of shape (flux_cat, flux_cat)

  • inv (InvertorOptimizer) – inversion object knowing about model observation mismatch, observation operator, and uncertainties

  • idx (int) – index along dimension norm_prefactor, must match the attribute next_norm_prefactor_idx of the existing file

Changed in version 0.1.1.

dubfi.fluxes.core.run_inversion(config, target_dir, init_from=None, profile=False, tests=False, overwrite=False, extra_config={}, implementation='MPI')

Run inversion and save results.

This performs the following steps:

  1. Read configuration, add some information from the input data, and save the full configuration to {target_dir}/config.yml. target_dir is created if necessary.

  2. Initialize the inversion and save the a priori data to {target_dir}/inversion_result.nc. The file already contains the structure for the full inversion results, but the result values are set to NaN.

  3. Optional: Run some tests. These tests only output some information on the accuracy of derivatives.

  4. Save the results of a simple Kalman smoother to the existing output file. These results will serve as initial values for the solver.

  5. Run the inversion for one or multiple values of norm_prefactor sequentially. Output is written to the existing output file after each inversion run.

Parameters:

  • config (str | dict) – configuration or path to configuration YAML file.

  • target_dir (str) – target directory to which results and log files will be saved.

  • init_from (str | xr.Dataset, optional) – take prior from posterior in file or dataset. The provided netCDF file or dataset must be of the same format as the output of dubfi. Mixing with initial prior and uncertainty inflation are applied as defined in the configuration. Provide the special value “START CYCLE” to create the attribute “cycling_started_on” in the output without cycling.

  • profile (bool, default=False) – enable profiling for solver, save profiling information in target_dir.

  • tests (bool, default=False) – run tests to check numerical derivatives

  • overwrite (bool, default=False) – overwrite existing files

  • extra_config (dict, optional) – update config from this dictionary

  • implementation ({"MPI", "csc", "dense", "diagonal"}) –

    select linear algebra implementation:

    • MPI requires careful configuration but enables using many observations. This uses distributed memory parallelization.

    • csc uses compressed sparse column matrices and can handle moderate numbers of observations.

    • dense uses dense matrices. This is limited to very few observations.

    • diagonal constructs a diagonal R matrix and only works with dense matrices. This yields very different results.

Return type:

int

Changed in version 0.1.1.