S3

Dedicated S3 access utilities. Potentially to be moved to hostess.aws.s3 in the future.

S3ReadRequest dataclass

S3ReadRequest(start: int, end: int, satisfies_chunk_indices: list[int])

One block of data that we need to read from S3. Internal to S3Reader.

S3Reader

S3Reader(bucket: Bucket, name: str | Path, *, chunk_size: int = 1024 * 1024, cache_dir: str | Path | None = None)

Bases: BufferedIOBase

Read from an S3 object using the standard Python filelike API.

Only supports binary buffered reading. For text mode, wrap it in io.TextIOWrapper. Not thread safe.

The file is read in chunks from the network and cached locally in a temporary file. Chunk size defaults to 1 MiB, controllable with the 'chunk_size' argument to the constructor.

If 'cache_dir' is specified, the local cache files will be stored on the filesystem backing that directory; otherwise they will be stored in the default location for temporary files. Regardless, the cache files will normally not be visible in the filesystem (except on Windows) and are not preserved when the S3Reader is closed.

See tempfile.gettempdir for how the default location for temporary files is determined.