dubfi.fluxes.core ================= .. py:module:: dubfi.fluxes.core .. autoapi-nested-parse:: Core module for running flux inversion based on in-situ observations. .. codeauthor:: Valentin Bruch, DWD .. versionchanged:: 0.1.2 (renamed module) .. versionadded:: 0.1.0 (initial release) Functions --------- .. autoapisummary:: dubfi.fluxes.core._save_apriori dubfi.fluxes.core._save_kalman dubfi.fluxes.core._save_aposteriori dubfi.fluxes.core.run_inversion Module Contents --------------- .. py:function:: _save_apriori(target_file, config, coords, segments, s_prior, b_prior, implementation, s_prior_norm_pref = None, b_prior_norm_pref = None, cycling_start = None) Save output file including all meta data with NaN for results. .. see also:: :func:`_save_aposteriori`, :func:`_save_kalman` :param target_file: output file path. Existing files will be overwritten. :type target_file: str :param config: inversion configuration dictionary :type config: dict :param coords: coordinates dictionary as produced by :func:`dubfi.fluxes.readobs.coordinates_from_config` :type coords: dict :param segments: segmentation of data used for shared memory parallelization :type segments: dubfi.linalg.segments.Segments, optional :param s_prior: prior scaling factors, aligned with coords["flux_cat"] :type s_prior: np.ndarray :param b_prior: prior error covariance matrix of scaling factors (B) used for Kalman (simple) inversion. :type b_prior: np.ndarray :param implementation: linear algebra implementation used for the inversion, saved as attributed :type implementation: str :param s_prior_norm_pref: prior scaling factors for different norm prefactors, shape (norm_prefactor, flux_cat). If not provided, s_prior is used for all norm_prefactor values. :type s_prior_norm_pref: np.ndarray, optional :param b_prior_norm_pref: prior error covariance matrix of scaling factors (B) for different norm prefactors, shape (norm_prefactor, flux_cat, flux_cat). If not provided, b_prior is used for all norm_prefactor values. :type b_prior_norm_pref: np.ndarray, optional :param cycling_start: start date and time of inversion cycle, will be included in output attributes :type cycling_start: str, optional .. versionchanged:: 0.1.1 .. py:function:: _save_kalman(target_file, s_post, b_post, sensitivity, inv, chi2, chi2_obs, chi2_fit, ddof) Add result data from simple Kalman smoother to existing netCDF file. This function computes the model observation mismatch and its uncertainty. Results are saved to the existing file, assuming that this file was created using :func:`_save_apriori`. .. see also:: :func:`_save_apriori`, :func:`_save_aposteriori` :param target_file: output file path. Existing files will be overwritten. :type target_file: str :param s_post: posterior scaling factors (obtained using Kalman smoother), aligned with flux_cat coordinate in existing file :type s_post: np.ndarray :param b_post: posterior error covariance matrix of scaling factors (B) :type b_post: np.ndarray :param sensitivity: derivative of posterior scaling factors with respect to observations, array of shape (flux_cat, obs) :type sensitivity: np.ndarray :param inv: inversion object knowing about model observation mismatch, observation operator, and uncertainties :type inv: InvertorOptimizer :param chi2: chi^2 value of the fit :type chi2: float :param chi2_obs: observation contribution to chi^2 of the fit (positive) :type chi2_obs: float :param chi2_fit: scaling uncertainty contribution to chi^2 of the fit (negative) :type chi2_fit: float :param ddof: total degrees of freedom minus number of fit parameters :type ddof: int .. versionchanged:: 0.1.1 .. py:function:: _save_aposteriori(target_file, s_post, b_post, sensitivity, averaging_kernel, inv, idx) Add result data to existing netCDF file. .. see also:: :func:`_save_apriori`, :func:`_save_kalman` :param target_file: output file path. Existing files will be overwritten. :type target_file: str :param s_post: posterior scaling factors :type s_post: np.ndarray :param b_post: posterior error covariance matrix of scaling factors (B) :type b_post: np.ndarray :param sensitivity: derivative of posterior scaling factors with respect to observations, array of shape (flux_cat, obs) :type sensitivity: np.ndarray :param averaging_kernel: derivative of posterior scaling factors with respect to true scaling factors, array of shape (flux_cat, flux_cat) :type averaging_kernel: np.ndarray :param inv: inversion object knowing about model observation mismatch, observation operator, and uncertainties :type inv: InvertorOptimizer :param idx: index along dimension norm_prefactor, must match the attribute next_norm_prefactor_idx of the existing file :type idx: int .. versionchanged:: 0.1.1 .. py:function:: run_inversion(config, target_dir, init_from = None, profile = False, tests = False, overwrite = False, extra_config = {}, implementation = 'MPI') Run inversion and save results. This performs the following steps: 1. Read configuration, add some information from the input data, and save the full configuration to {target_dir}/config.yml. target_dir is created if necessary. 2. Initialize the inversion and save the a priori data to {target_dir}/inversion_result.nc. The file already contains the structure for the full inversion results, but the result values are set to NaN. 3. Optional: Run some tests. These tests only output some information on the accuracy of derivatives. 4. Save the results of a simple Kalman smoother to the existing output file. These results will serve as initial values for the solver. 5. Run the inversion for one or multiple values of norm_prefactor sequentially. Output is written to the existing output file after each inversion run. :param config: configuration or path to configuration YAML file. :type config: str | dict :param target_dir: target directory to which results and log files will be saved. :type target_dir: str :param init_from: take prior from posterior in file or dataset. The provided netCDF file or dataset must be of the same format as the output of dubfi. Mixing with initial prior and uncertainty inflation are applied as defined in the configuration. Provide the special value "START CYCLE" to create the attribute "cycling_started_on" in the output without cycling. :type init_from: str | xr.Dataset, optional :param profile: enable profiling for solver, save profiling information in target_dir. :type profile: bool, default=False :param tests: run tests to check numerical derivatives :type tests: bool, default=False :param overwrite: overwrite existing files :type overwrite: bool, default=False :param extra_config: update config from this dictionary :type extra_config: dict, optional :param implementation: select linear algebra implementation: - MPI requires careful configuration but enables using many observations. This uses distributed memory parallelization. - csc uses compressed sparse column matrices and can handle moderate numbers of observations. - dense uses dense matrices. This is limited to very few observations. - diagonal constructs a diagonal R matrix and only works with dense matrices. This yields very different results. :type implementation: {"MPI", "csc", "dense", "diagonal"} .. versionchanged:: 0.1.1