Cli

Upload data sets to MAST, the Mikulski Archive for Space Telescopes.

This module defines a command-line interface to upload operations. Its functions cannot usefully be used as part of a larger program. The other modules in this subpackage define a Pythonic interface that can be used by other programs.

BytesWithScaleParamType

Bases: ParamType

Shim between click and utilz.numeric.parse_bytes_with_scale.

check_label

check_label(label: Path) -> None

Checks a label for syntactic validity.

checksum

checksum(*, source: str, index_file: Path, output: Path = Path('index.csv')) -> None

Add CRC32 checksums to an existing index file.

index

index(*, source: str, output: Path = Path('index.csv'), make_checksums: bool = False, label: Path | None = None) -> None

Create an index file suitable for use with other mast-upload commands. If -c / --make-checksums is passed, also calculate CRC32 checksums for each indexed file. (Checksum creation is not supported for S3 objects.)

If is a local directory, paths in the index file will be given relative to , which must also be used as for upload and validation commands that make use of that index file. If is an S3 bucket + prefix, paths in the index file will be given relative to the bucket root prefix, and s3://bucket-name should be used as for subsequent upload and validation commands .

By default, this command indexes all standard files/objects under . If you also provide , it will index only files matching a defined filetype in that label. If you wish to restrict indexing in a way that cannot be defined by filetypes + top-level directory/prefix (for instance, you intend to deliver the contents of three sibling directories), it is preferable to construct the index manually and, if checksums are desired, subsequently use checksum to populate checksums.

main

main() -> None

Upload data sets to MAST, the Mikulski Archive for Space Telescopes.

populate_label

populate_label(*, source: str, label: Path, output: Path, filetype_names: str | None) -> None

Populate (or repopulate) the filetypes section of a label by analyzing files in , where <source> is either the root directory of a tree under which your files are located or a prefix on an S3 bucket under which your files are located. S3 addresses must be preceded with 's3://'. Filename patterns and standards must be defined for each filetype, or the application will not be able to identify which files should go with which filetype and, if relevant, what data-level analysis to perform on those files. may contain either your full data set or a representative subset of files for quicker analysis. If some filetypes are missing representatives, populate_label() will populate everything it can. Some highly complex or variable filetypes may not be describable by populate_label(); such filetypes must be defined manually if data-level validation past standard conformance is desired. In any case, it is best to manually review the generated file specifications and to check them against local validation commands.

report_filetypes

report_filetypes(*, source: str, label: Path, output: Path = Path('filetypes.csv')) -> None

Write a CSV file containing matching filetypes as described in for all files under . The file has two columns: "path" and "filetypes", where "filetypes" is a semicolon-separated list of matching filetype names (blank if none match).

Also prints a warning if any files are matched by more than one filetype (this is an invalid condition for transfer / description / validation).

transfer

transfer(*, source: str, label: Path, index_file: Path, sample: bool) -> None

Upload a data set to MAST.

If the upload is interrupted or stops due to validation errors, you can simply repeat the command to resume from where it left off, after fixing any validation problems.

validate

validate(*, file: str, label: Path, object_check_hook: bool) -> None

Validate an individual file against a label. This command confirms that the label itself is valid and that the filename matches exactly one filetype defined in the label. If the file standard supports data validation (FITS, ASDF, or Parquet), the command will furthermore confirm that it conforms to the file standard and, if data objects are defined in the label, that it conforms to that specification.

validate_all

validate_all(*, source: str, label: Path, index_file: Path | None = None, object_check_hook: bool = True) -> None

Validate all files under against .

If is provided, instead only validate files listed in that index. should be a CSV file whose first column is 'path' and whose second column, if present, is 'checksum'. A file of this format may be automatically produced using the index command.

may be either a local directory or a prefix on an s3 bucket; if an s3 bucket/prefix, its name should be preceded by 's3://'.

Paths in will be treated as relative to . (If an index file is provided, a specific prefix on an S3 bucket may not be used as ; paths must be relative to the root prefix.)

This command confirms that the label itself is valid and that each filename matches exactly one filetype defined in the label. If a file standard supports data validation (FITS, ASDF, or Parquet), the command will furthermore confirm that it conforms to the file standard and, if data objects are defined in the label, that it conforms to that definition.