I/O File Conventions

Input Files

There are two general types of input to any step or pipeline: references files and data files. The references files, unless explicitly overridden, are provided through CRDS.

Data files are the science input, such as exposure ASDF files. All files are assumed to be co-resident in the directory where the primary input file is located.

Output Files

Output files will be created either in the current working directory, or where specified by the output_dir parameter.

File names for the outputs from pipelines and steps come from two different sources:

  • The name of the input file

  • As specified by the output_file parameter

Regardless of the source, each pipeline/step uses the name as a base name, onto which several different suffixes are appended, which indicate the type of data in that particular file. A list of the main suffixes can be found below.

The pipelines do not manage versions. When re-running a pipeline, previous files will be overwritten.

Individual Step Outputs

If individual steps are executed without an output file name specified via the output_file parameter, the stpipe infrastructure automatically uses the input file name as the root of the output file name and appends the name of the step as an additional suffix to the input file name. If the input file name already has a known suffix, that suffix will be replaced. For example:

$ strun romancal.dq_init.DQInitStep r0008308002010007027_06311_0019_WFI01_uncal.asdf

produces an output file named r0008308002010007027_06311_0019_WFI01_dq_init.asdf.

Suffix Definitions

However the output file name is determined (see above), the various pipeline modules will use that file name, along with a set of predetermined suffixes, to compose output file names. The output file name suffix will always replace any known suffix of the input file name. Each pipeline module uses the appropriate suffix for the product(s) it is creating. The list of suffixes is shown in the following table. Replacement occurs only if the suffix is one known to the calibration code. Otherwise, the new suffix will simply be appended to the basename of the file.

Product

Suffix

Uncalibrated raw input

_uncal

DQ initialization

_dqinit

Saturation detection

_saturation

Reference Pixel Correction

_refpix

Linearity correction

_linearity

Dark current

_darkcurrent

Corrected ramp data

_rampfit

Optional fitting results from ramp_fit step

_fitopt

Assign WCS

_assignwcs

Flat field

_flat

Photometric calibration

_photomstep

Source Detection

_sourcedetection

Tweakreg (Allign to GAIA)

_tweakregstep

Calibrated image

_cal