API reference#

Suite#

class fcpy.Suite(ds, baseline, compressors, metrics, custom_metrics=None, bits=None, max_chunk_size_bytes=None, skip_histograms=False)#

Bases: object

Create a suite.

Parameters:

  • ds (xr.Dataset) – The dataset to use.

  • baseline (Compressor) – The compressor to use for the baseline. Must include bits in the constructor.

  • compressors (list[Compressor]) – The list of compressors to use.

  • metrics (list[Type[Metric]]) – The list of metrics to compute.

  • custom_metrics (list[Callable], optional) – Custom metrics as user functions. Each function is called as fn(chunks, baseline, compressors, bits) and must return an xarray DataArray.

  • bits (list[int], optional) – List of bits values to iterate over. Defaults to Klöwer et al. (2021)’s bit-information metric.

  • max_chunk_size_bytes (int, optional) – Maximum size in bytes that a chunk may have. Defaults to 4 GiB.

  • skip_histograms (bool, optional) – If True, compute metrics directly instead of using histograms.

Datasets#

fcpy.open_dataset(filepath)#

Open a dataset from one or more local files.

Parameters:

filepath (str) – Either .nc or .grib, may contain wildcards.

Returns:

The loaded dataset.

Return type:

xr.Dataset

Compressors#

class fcpy.Compressor(inner_compressor=None, bits=None)#

Bases: object

Abstract base class for compressors.

Parameters:

  • inner_compressor (Compressor, optional) – Inner compressor to use, if any.

  • bits (int, optional) – Bits to use for compression. If None, must be given when calling compress().

compress(arr, bits=None)#

Compress the given array.

Parameters:

  • arr (np.ndarray) – Data to compress.

  • bits (int, optional) – Bits to use in compression. Defaults to bits passed in the constructor.

Returns:

The compressed data with metadata of this and all inner compressors required for decompression.

Return type:

Tuple[np.ndarray, list[dict]]

decompress(compressed_data, params_stack)#

Decompress the given data using the given compression metadata.

Parameters:

  • compressed_data (np.ndarray) – The compressed data.

  • params_stack (list[dict]) – The compression metadata of this and all inner compressors.

Returns:

The decompressed data.

Return type:

np.ndarray

abstract do_compress(arr, bits)#

Method to be implemented in compressor subclasses. Called by the base class during compress().

Parameters:

  • arr (np.ndarray) – Data to compress.

  • bits (int) – Bits to use for compression.

Returns:

Compressed data with metadata required for decompression.

Return type:

Tuple[np.ndarray, dict]

abstract do_decompress(compressed_data, params)#

Method to be implemented in compressor subclasses. Called by the base class during decompress().

Parameters:

  • compressed_data (np.ndarray) – The compressed data.

  • params (dict) – The compression metadata returned by do_compress().

Returns:

The decompressed data.

Return type:

np.ndarray

property name#

Name of the compressor.

class fcpy.Float(inner_compressor=None, bits=None)#

Bases: Compressor

IEEE Floating-Point compressor.

class fcpy.LinQuantization(inner_compressor=None, bits=None)#

Bases: Compressor

Linear quantization compressor.

class fcpy.Log(inner_compressor=None, bits=None)#

Bases: Compressor

Log/Exp compressor, typically used for pre-/postprocessing.

class fcpy.Round(inner_compressor=None, bits=None)#

Bases: Compressor

Rounding compressor.

Metrics#

class fcpy.Metric#

Bases: object

Base class for Metric subclasses.

abstract compute(x, y)#

Compute the metric with the given reference and computed data.

Parameters:

  • x (np.ndarray) – Reference data.

  • y (np.ndarray) – Computed data.

Returns:

The values computed by the metric.

Return type:

np.ndarray

name#

Name of the metric.

class fcpy.AbsoluteError#

Bases: Metric

Absolute error metric: |x - y|

class fcpy.Difference#

Bases: Metric

Difference metric: x - y.

class fcpy.RelativeError#

Bases: Metric

Relative error metric: |x - y| / |max(x) - min(x)|

Information Metrics#

fcpy.compute_knees_field(da, da_sigmas, plot=False, interp_method='polynomial', polynomial_degree=4)#

Computes the point of maximum curvature.

Parameters:

  • da (xr.DataArray) – Reference DataArray.

  • da_sigmas (xr.DataArray) – DataArray of Sigmas.

  • plot (bool, optional) – Whether to plot. Defaults to False.

  • interp_method (str, optional) – Method of interpolation to use by the knee finding algorithm. Defaults to “polynomial”.

  • polynomial_degree (int, optional) – Degree of polynomial to use by the knee finding algorithm. Defaults to 4.

Returns:

Dictionary of DataArrays of number of bits and sigmas.

Return type:

dict

fcpy.compute_sigmas(da, compressors)#

Computes a DataArray of sigmas based on a random uniform-distributed noise field.

Parameters:

  • da (xr.DataArray) – Reference data.

  • compressors (list) – List of compressors.

Returns:

Sigma values.

Return type:

xr.DataArray

fcpy.sigmas_iterate_da(da, compressors, plot=False)#

Computes sigmas for a given dataarray and compressors.

Parameters:

  • da (xr.DataArray) – DataArray to compute sigmas for.

  • compressors (list) – List of compressors to compute sigmas for.

Returns:

Dataset with sigmas computed.

Return type:

xr.Dataset

fcpy.sigmas_iterate_ds(ds, compressors, plot=False)#

Computes sigmas for a given dataset and compressors.

Parameters:

  • ds (xr.Dataset) – Dataset to compute sigmas for.

  • compressors (list) – List of compressors to compute sigmas for.

  • plot (bool, optional) – Whether to generate a plot. Defaults to False.

Returns:

dictionary of Datasets with sigmas computed.

Return type:

dict