4. ISTP Variables

4.1. Variable Types

ISTP Guidelines divide all variables into three types, identified by the value of the VAR_TYPE variable attribute:

  • Data variables (VAR_TYPE = "data") of primary importance (e.g., density, magnetic field, particle flux).

  • Support_data variables (VAR_TYPE = "support_data") of secondary importance (e.g., time and energy associated with particle flux).

  • Metadata variables (VAR_TYPE = "metadata") used for labeling dimensional data (e.g., a variable holding string array ["Bx GSE","By GSE","Bz GSE"] for labeling vector magnetic field).

Examples of data and support_data variables commonly found in ISTP datasets are shown below. They are mapped to their corresponding dimensions and sizes in CDF format (and could be in netCDF-4 and other structured file formats).

../_images/CDF_Planning.png

  • DENSITY and TEMPERATURE (data) are scalars; in CDF format they are associated with zero dimensions and no size. Note that in CDF format, time is associated with record (one record per timestamp), which is separate from variable dimensions.

  • VELOCITY, ELECTRIC_FIELD and MAGNETIC_FIELD (data) are vectors, i.e., three orthogonal components in some coordinate system, stored as 1-D variables of size 3.

  • FLUX has values at eight energy channels. FLUX (data) and ENERGY (support_data) are stored as 1-D variables of size 8.

  • The IMAGE array (data) maps into a 2-D variable with sizes 256 and 256. LATITUDE and LONGITUDE (support_data) are 1-D variables of size 256, providing the necessary coordinates for the IMAGE array.

Each ISTP variable type is defined with CDF (or netCDF-4) specifications and required attributes. Data variables also have attached variables for time and dependencies (support_data) and for labels (metadata). Support_data variables can be attached to data variables via, e.g., the data variable’s DEPEND_i attributes, with DEPEND_0 always pointing (by name) to a time variable. Metadata variables can be attached to data variables via, e.g., data variable’s LABL_PTR_i, UNIT_PTR, FORM_PTR, SCAL_PTR attributes.

An ISTP dataset usually spans over many files, over which the dataset definitions must remain static, meaning that variables cannot be added or removed, or have their data types, number of dimensions, dimension sizes, or variances within records and dimensions changed. When defining record or dimension variances of a variable, the whole dataset life, and not just a particular file, must be considered. Similarly, the variable dimensions must be defined with the maximum sizes expected over the dataset life, with currently unused values filled with the FILLVAL attribute value. There also must be no duplication of data over all dataset files, with all files seamlessly concatenating into one whole dataset.

(CDF-specific.) If extremely rare changes occur in the variable values over time, a CDF variable can be defined with record sparseness set to previous record. In this case, only records with changes need to be explicitly written into the CDF file, while reading any record will return the last explicitly written record before the requested one. Note that the first variable record in each CDF file still must be explicitly written with valid value. See the CDF User’s Guide for details on CDF variable sparse records. Alternatively, a rarely changing variable can have all records written, but variable compression should be enabled in this case. Variable compression should also be enabled for large size variables, while file-level compression is generally discouraged since reading is slower. Note that for efficient data access, time variables should never be compressed.

Each variable in a dataset must have a unique name that starts with a letter and can contain numbers, underscores, but no other special characters. Variable names are case sensitive, but the names must never be distinguished by case only. This enables broad support across many programming languages and analysis packages. Variable names should also carry sufficient information for initial understanding of their meaning by a dataset user and for clear distinguishing between different variables. Otherwise, the ISTP Guidelines do not prescribe a specific scheme for variable naming.

The variable names are used by CDF/netCDF-4 files internally but generally not visible to a user. FIELDNAM variable attribute is used for user-friendly name for a variable/parameter, while CATDESC holds longer and more complete variable description, particularly used for distinguishing between different variables. Additional information for a variable can be stored in the VAR_NOTES attribute. LABLAXIS/LABL_PTR_i are used for plot labeling, and they do not include units, which are stored in the separate UNITS attribute, also displayed in the plots.

4.1.1. Data Variables

Data variables are variables of primary importance (e.g., density, magnetic field, particle flux). Data variables are almost always of either a floating-point or integer data type, always time (record) varying, and can be a scalar or an array of values of up to 10 dimensions. Note that floating-point and integer data type variables are always defined as having one element at each variable value.

For a data variable, the following variable attributes are required:

Note that need for DEPEND_i (other than DEPEND_0) and either LABLAXIS or LABL_PTR_i depends on the data dimensionality and how it will be displayed.

See examples of data variable definitions and their displays. Additional display examples are available at CDAWeb.

4.1.2. Support_Data Variables

Support_data variables are variables of secondary importance holding numerical data, e.g., time or energy associated with particle flux. A support_data variable is always of either a floating-point or integer data type (including CDF time data types). It can be either time (record) invariant or time varying. An epoch (time) support_data variable attached to a data variable via DEPEND_0 attribute must have the same number of records as the data variable. If a support_data variable is attached to a data variable via DEPEND_i (i > 0), it must be of the same size as the corresponding dimension of the data variable. Also note that floating-point and integer data type variables are always defined as having one element at each variable value.

For a support_data variable, the following variable attributes are required:

See examples of support_data variable definitions.

4.1.3. Metadata Variables

Metadata variables are variables of secondary importance holding strings, e.g., a variable holding array of strings for labeling magnetic field vector components. A metadata variable is always of a character type and is always time invariant if used to label a data variable. It can be time-varying only if it is NOT used as a label. If a metadata variable is attached to a data variable (via, e.g., LABL_PTR_i), it must be of the same size as the corresponding dimension of the data variable.

(CDF-specific.) A character data type variable must be defined with the number of elements equal to the maximum number of characters used in its values. For example, the number of elements is 6 in case of a variable holding an array of six-character strings ["Bx GSE","By GSE","Bz GSE"].

The following variable attributes are required:

See examples of metadata variable definitions.