Introduction¶

This document is intended to be a core reference guide to the formats, naming convention and data quality flags used by the reference files for pipeline steps requiring them, and is not intended to be a detailed description of each of those pipeline steps. It also does not give details on pipeline steps that do not use reference files. The present manual is the living document for the reference file specifications.

Reference File Naming Convention¶

Before reference files are ingested into CRDS, they are renamed following a convention used by the pipeline. As with any other changes undergone by the reference files, the previous names are kept in header, so the Instrument Teams can easily track which delivered file is being used by the pipeline in each step.

The naming of reference files uses the following syntax:

roman_<instrument>_<reftype>_<version>.<extension>

where

instrument is currently “WFI”
reftype is one of the type names listed in the table below
version is a 4-digit version number (e.g. 0042)
extension gives the file format, “asdf”

An example WFI FLAT reference file name would be “roman_wfi_flat_0042.asdf”.

Reference File Types¶

Most reference files have a one-to-one relationship with calibration steps, e.g. there is one step that uses one type of reference file. Some steps, however, use several types of reference files and some reference file types are used by more than one step. The tables below show the correspondence between pipeline steps and reference file types. The first table is ordered by pipeline step, while the second is ordered by reference file type. Links to the reference file types provide detailed documentation on each reference file.

Pipeline Step	Reference File Type (reftype)
assign_wcs	DISTORTION
dark_current	DARK
dq_init	MASK
flatfield	FLAT
jump_detection	GAIN
jump_detection	READNOISE
linearity	LINEARITY
photom	PHOTOM
ramp_fitting	GAIN
ramp_fitting	READNOISE
saturation	SATURATION

Reference File Type (reftype)	Pipeline Step
DARK	dark_current
DISTORTION	assign_wcs
FLAT	flatfield
GAIN	jump_detection
GAIN	ramp_fitting
LINEARITY	linearity
MASK	dq_init
PHOTOM	photom
READNOISE	jump_detection
READNOISE	ramp_fitting
SATURATION	saturation

Standard ASDF metadata¶

Al Roman science and reference files are ASDF files.

The required attributes Documenting Contents of Reference Files are:

Attribute	Comment
reftype	`FLAT Required values are listed in the discussion of each pipeline step.`
description	`Summary of file content and/or reason for delivery.`
author	`Fred Jones Person(s) who created the file.`
useafter	`YYYY-MM-DDThh:mm:ss Date and time after the reference files will be used. The T is required. Time string may NOT be omitted; use T00:00:00 if no meaningful value is available. Astropy Time objects are allowed.`
pedigree	`Options are 'SIMULATION' 'GROUND' 'DUMMY' 'INFLIGHT YYYY-MM-DD YYYY-MM-DD'`
history	`Description of Reference File Creation`.
telescope	`ROMAN Name of the telescope/project.`
instrument	`WFI Instrument name.`

Observing Mode Attributes¶

A pipeline module may require separate reference files for each instrument, detector, optical element, observation date, etc. The values of these parameters must be included in the reference file attributes. The observing-mode attributes are vital to the process of ingesting reference files into CRDS, as they are used to establish the mapping between observing modes and specific reference files. Some observing-mode attributes are also used in the pipeline processing steps.

The Keywords Documenting the Observing Mode are:

Keyword	Sample Value	Comment
detector	WFI01	Allowed values WFI01, WFI02, … WFI18
optical element	F158	Name of the filter element and includes PRISM and GRISM
exposure type	WFI_IMAGE	Allowed values WFI_IMAGE, WFI_GRATING, WFI_PRISM, WFI_DARK, WFI_FLAT, WFI_WFSC

Tracking Pipeline Progress¶

As each pipeline step is applied to a science data product, it will record a status indicator in a cal_step attribute of the science data product. These statuses may be included in the primary header of reference files, in order to maintain a history of the data that went into creating the reference file. Allowed values for the status Attribute are ‘INCOMPLETE’, ‘COMPLETE’ and ‘SKIPPED’. The default value is set to ‘INCOMPLETE’. The pipeline modules will set the value to ‘COMPLETE’ or ‘SKIPPED’. If the pipeline steps are run manually and you skip a step the cal_step will remain ‘INCOMPLETE’.

Data Quality Flags¶

Within science data files, the PIXELDQ flags are stored as 32-bit integers; the GROUPDQ flags are 8-bit integers. All calibrated data from a particular instrument and observing mode have the same set of DQ flags in the same (bit) order. The table below lists the allowed DQ flags. Only the first eight entries in the table below are relevant to the GROUPDQ array.

Flags for the DQ, PIXELDQ, and GROUPDQ Arrays.

Bit	Value	Name	Description
0	1	DO_NOT_USE	Bad pixel. Do not use.
1	2	SATURATED	Pixel saturated during exposure
2	4	JUMP_DET	Jump detected during exposure
3	8	DROPOUT	Data lost in transmission
4	16	GW_AFFECTED_DATA	Data affected by the GW read window
5	32	PERSISTENCE	High persistence (was RESERVED_2)
6	64	AD_FLOOR	Below A/D floor (0 DN, was RESERVED_3)
7	128	OUTLIER	Detected as outlier in coadded image
8	256	UNRELIABLE_ERROR	Uncertainty exceeds quoted error
9	512	NON_SCIENCE	Pixel not on science portion of detector
10	1024	DEAD	Dead pixel
11	2048	HOT	Hot pixel
12	4096	WARM	Warm pixel
13	8192	LOW_QE	Low quantum efficiency
15	32768	TELEGRAPH	Telegraph pixel
16	65536	NONLINEAR	Pixel highly nonlinear
17	131072	BAD_REF_PIXEL	Reference pixel cannot be used
18	262144	NO_FLAT_FIELD	Flat field cannot be measured
19	524288	NO_GAIN_VALUE	Gain cannot be measured
20	1048576	NO_LIN_CORR	Linearity correction not available
21	2097152	NO_SAT_CHECK	Saturation check not available
22	4194304	UNRELIABLE_BIAS	Bias variance large
23	8388608	UNRELIABLE_DARK	Dark variance large
24	16777216	UNRELIABLE_SLOPE	Slope variance large (i.e., noisy pixel)
25	33554432	UNRELIABLE_FLAT	Flat variance large
26	67108864	RESERVED_5
27	134217728	RESERVED_6
28	268435456	UNRELIABLE_RESET	Sensitive to reset anomaly
29	536870912	RESERVED_7
30	1073741824	OTHER_BAD_PIXEL	A catch-all flag
31	2147483648	REFERENCE_PIXEL	Pixel is a reference pixel

Parameter Specification¶

There are a number of steps, such as OutlierDetectionStep, that define what data quality flags a pixel is allowed to have to be considered in calculations. Such parameters can be set in a number of ways.

First, the flag can be defined as the integer sum of all the DQ bit values from the input images DQ arrays that should be considered “good”. For example, if pixels in the DQ array can have combinations of 1, 2, 4, and 8 and one wants to consider DQ flags 2 and 4 as being acceptable for computations, then the parameter value should be set to “6” (2+4). In this case a pixel having DQ values 2, 4, or 6 will be considered a good pixel, while a pixel with a DQ value, e.g., 1+2=3, 4+8=”12”, etc. will be flagged as a “bad” pixel.

Alternatively, one can enter a comma-separated or ‘+’ separated list of integer bit flags that should be summed to obtain the final “good” bits. For example, both “4,8” and “4+8” are equivalent to a setting of “12”.

Finally, instead of integers, the Roman mnemonics, as defined above, may be used. For example, all the following specifications are equivalent:

"12" == "4+8" == "4, 8" == "JUMP_DET, DROPOUT"

Note

The default value (0) will make all non-zero pixels in the DQ mask be considered “bad” pixels and the corresponding pixels will not be used in computations.
Setting to None will turn off the use of the DQ array for computations.
In order to reverse the meaning of the flags from indicating values of the “good” DQ flags to indicating the “bad” DQ flags, prepend ‘~’ to the string value. For example, in order to exclude pixels with DQ flags 4 and 8 for computations and to consider as “good” all other pixels (regardless of their DQ flag), use a value of ~4+8, or ~4,8. A string value of ~0 would be equivalent to a setting of None.