emmi_inference.s3_spooling_uploader

Classes

S3SpoolingUploader

Manages a "spool-and-upload" process for inference results.

Module Contents

class emmi_inference.s3_spooling_uploader.S3SpoolingUploader(local_spool_dir, s3_path, save_format=SaveFormat.PYTORCH_PT, dict_mode=DictMode.SINGLE, upload_batch_size=100, remove_on_upload=False, local_writer_workers=4, s3_upload_workers=8)

Manages a “spool-and-upload” process for inference results.

This class acts as a wrapper. It uses an AsyncWriter to write to a local disk buffer at high speed. A separate background worker then uploads completed files to S3 in batches, optionally deleting them to manage local disk space.

This is the recommended approach for long-running inference jobs to prevent blocking the GPU and to manage disk usage.

Parameters:

  • local_spool_dir (pathlib.Path | str)

  • s3_path (str)

  • save_format (emmi_inference.writer.SaveFormat)

  • dict_mode (emmi_inference.writer.DictMode)

  • upload_batch_size (int)

  • remove_on_upload (bool)

  • local_writer_workers (int)

  • s3_upload_workers (int)

local_spool_dir
remove_on_upload = False
upload_batch_size = 100
s3_bucket
s3_prefix
s3_client
s3_pool
local_writer
write(relative_path_no_ext, to_save)

Write and schedule for upload.

This method is non-blocking. It passes the write job to the local AsyncWriter, which will then put the completed file path onto the upload queue for the background worker.

Parameters:
Return type:

None

close()

Shuts down the uploader. 1. Waits for all local file writes to finish. 2. Signals the upload worker to stop. 3. Waits for the upload worker to finish its batches. 4. Waits for all S3 uploads to complete.

Return type:

None