dubfi.dataprovider.readobs

Read and filter observations as defined in configuration.

Added in version 0.1.0: initial release

Changed in version 0.1.1.

Attributes

INVERSION_FLAGS

List of flags indicating filtering of observations for the inversion.

Classes

ReadObs

Object that parses configuration and reads data.

Functions

get_flag(key) → int

Get integer representation of inversion flag name, see INVERSION_FLAGS.

coordinates_from_config(config, *[, nprocesses]) → dict

Gather coordinates from input data, applying filtering as defined in config.

average_window(da, window[, min_diff]) → xarray.DataArray

Average data array over rolling time window.

get_solar_times(start, end, lat, lon, kind) → numpy.ndarray

Get array of times of sunrise, sunset, or noon.

get_times(start, end, lat, lon, kind) → numpy.ndarray

Get array of daily times from different references.

data_from_config(config[, coordinates_only, ...]) → Iterable[xarray.Dataset | tuple[xarray.Dataset, xarray.DataArray]]

Provide filtered data as defined in configuration.

Module Contents

dubfi.dataprovider.readobs.INVERSION_FLAGS = ['used', 'ignored: time range', 'ignored: season', 'ignored: time of day', 'ignored: wind...

List of flags indicating filtering of observations for the inversion.

dubfi.dataprovider.readobs.get_flag(key: str) → int

Get integer representation of inversion flag name, see INVERSION_FLAGS.

Added in version 0.1.1.

dubfi.dataprovider.readobs.coordinates_from_config(config: str | dict, *, nprocesses: int | None = None, **kwargs) → dict

Gather coordinates from input data, applying filtering as defined in config.

Parameters:

  • config (str | dict) – configuration of path to configuration file (YAML)

  • nprocesses (int, optional) – number of parallel processes, default taken from configuration.

  • **kwargs (any) – passed on to data_from_config()

Returns:

coordinates – dictionary with entries:

  • ssh: array of station and sampling height identifiers

  • time: list of time arrays, aligned with ssh

  • lon: array of station longitudes (degrees), aligned with ssh

  • lat: array of station latitudes (degrees), aligned with ssh

  • height: array of station heights (meters), aligned with ssh

  • flux_cat: array of flux category names

  • ens_size: number of meteorological ensemble members (integer, only returned if included in data)

Return type:

dict

dubfi.dataprovider.readobs.average_window(da: xarray.DataArray, window: numpy.timedelta64, min_diff: numpy.timedelta64 = MIN_OBS_TIME_DIFFERENCE) → xarray.DataArray

Average data array over rolling time window.

Parameters:

  • da (xr.DataArray) – data array that shall be averaged. This must have a dimension and coordinate “time” of dtype np.datetime64. This time coordinate must be sorted and the minimum distance between the coordinate values must be at least min_diff.

  • window (np.timedelta64) – time window for averaging. result[i] is the mean of da[j] for all j such that abs(da.time[i] - da.time[j]) < window.

  • min_diff (np.timedelta64, default=1h) – minimum distance between coordinate values in da.time. If the value is too large, results will be wrong. If the provided value is too small, performance will be worse.

Returns:

avg_da – da averaged over rolling time window. All dimensions and coordinates will be the same as in da. Internal order of data in memory may differ from da.

Return type:

xr.DataArray

dubfi.dataprovider.readobs.get_solar_times(start: datetime.datetime, end: datetime.datetime, lat: float, lon: float, kind: str) → numpy.ndarray

Get array of times of sunrise, sunset, or noon.

dubfi.dataprovider.readobs.get_times(start: numpy.datetime64, end: numpy.datetime64, lat: float, lon: float, kind: str) → numpy.ndarray

Get array of daily times from different references.

Parameters:

  • start (np.datetime64) – start date

  • end (np.datetime64) – end date

  • lat (float) – latitude (degrees)

  • lon (float) – longitude (degrees)

  • kind ({"utc", "lmt", "sunrise", "sunset", "noon"}) – type of reference time

class dubfi.dataprovider.readobs.ReadObs(config: dict | str, suffix_rx: str)

Object that parses configuration and reads data.

Added in version 0.1.1.

Changed in version 0.1.4: filter data by planetary boundary layer height or by selecting days

Provide filtered data as defined in configuration.

Parameters:

  • config (str | dict) – configuration of path to configuration file (YAML)

  • suffix_rx (str, default=r"_det.nc$") – regular expression for the input file suffix, use this to select the determinstic run without (_det.nc) or with (_det_letkf.nc) far-field correction, or the ensemble data (_ens.nc).

get_data(coordinates_only: bool = False, return_flags: bool = False, all_files: bool = False) → Generator[xarray.Dataset | tuple[xarray.Dataset, xarray.DataArray], None, None]

Provide filtered data as defined in configuration.

Parameters:

  • coordinates_only (bool, default=False) – if true, return only the coordinates and drop all other data

  • return_flags (bool, default=False) – additionally return a time series of flags defining why which data point was used or not used in the inversion. In this case, results will not be filtered.

  • all_files (bool, default=False) – return data from all files, including sampling heights that are excluded in the configuration

Yields:

  • xr.Dataset – datasets for each matching station and sampling height. Files are sorted alphabetically. Data are filtered unless return_flags is true.

  • xr.DataArray – only if return_flags: flag for each observation data point

_process_file(file: str, coordinates_only: bool, return_flags: bool, all_files: bool = False) → None | xarray.Dataset | tuple[xarray.Dataset, xarray.DataArray]

Load and filter data from file, ready for dask delayed.

get_data_parallel(coordinates_only: bool = False, return_flags: bool = False, all_files: bool = False, nprocesses: int = 1) → list[xarray.Dataset | tuple[xarray.Dataset, xarray.DataArray]]

Provide filtered data as defined in configuration.

Parameters:

  • coordinates_only (bool, default=False) – if true, return only the coordinates and drop all other data

  • return_flags (bool, default=False) – additionally return a time series of flags defining why which data point was used or not used in the inversion. In this case, results will not be filtered.

  • all_files (bool, default=False) – return data from all files, including sampling heights that are excluded in the configuration

  • nprocesses (int, default=1) – number of parallel worker processes

Returns:

  • if return_flags – list[tuple[xr.Dataset, xr.DataArray]]

  • else – list[xr.Dataset]

  • where –

    xr.Dataset

    datasets for each matching station and sampling height. Files are sorted alphabetically. Data are filtered unless return_flags is true.

    xr.DataArray

    only if return_flags: flag for each observation data point

filter_ds(ds: xarray.Dataset, coordinates_only: bool = False, return_flags: bool = False) → xarray.Dataset | tuple[xarray.Dataset, xarray.DataArray] | None

Filter data in dataset, see get_data().

_select_days(ds: xarray.Dataset, flags: xarray.DataArray) → None

Read selected days from file and only use data for those days.

_limit_seasons(ds: xarray.Dataset, flags: xarray.DataArray) → None

Limit data to given time ranges per station.

dubfi.dataprovider.readobs.data_from_config(config: str | dict, coordinates_only: bool = False, suffix_rx: str = '_det\\.nc$', *, return_flags: bool = False, all_files: bool = False, nprocesses: int | None = None) → Iterable[xarray.Dataset | tuple[xarray.Dataset, xarray.DataArray]]

Provide filtered data as defined in configuration.

Parameters:

  • config (str | dict) – configuration of path to configuration file (YAML)

  • coordinates_only (bool, default=False) – if true, return only the coordinates and drop all other data

  • suffix_rx (str, default="_det\.nc$") – regular expression for the input file suffix, use this to select the determinstic run without (_det.nc) or with (_det_letkf.nc) far-field correction, or the ensemble data (_ens.nc).

  • return_flags (bool, default=False) – additionally return a time series of flags defining why which data point was used or not used in the inversion

  • all_files (bool, default=False) – return data from all files, including sampling heights that are excluded in the configuration

  • nprocesses (int, optional) –

    number of parallel processes, default taken from configuration. Note that when using parallel processes (nprocesses > 1), all data will be read and returned as a list. Otherwise, this create a generator that yields the datasets.

    Warning

    Parallel reading of data may require significantly more memory.

Yields:

  • xr.Dataset – filtered datasets for each matching station and sampling height. Files are sorted alphabetically.

  • xr.DataArray – only if return_flags: flag for each observation data point

Changed in version 0.1.1.