emmi_inference.s3_spooling_uploader
===================================

.. py:module:: emmi_inference.s3_spooling_uploader


Classes
-------

.. autoapisummary::

   emmi_inference.s3_spooling_uploader.S3SpoolingUploader


Module Contents
---------------

.. py:class:: S3SpoolingUploader(local_spool_dir, s3_path, save_format = SaveFormat.PYTORCH_PT, dict_mode = DictMode.SINGLE, upload_batch_size = 100, remove_on_upload = False, local_writer_workers = 4, s3_upload_workers = 8)

   Manages a "spool-and-upload" process for inference results.

   This class acts as a wrapper. It uses an AsyncWriter to write to a local disk buffer at high speed. A separate
   background worker then uploads completed files to S3 in batches, optionally deleting them to manage local disk
   space.

   This is the recommended approach for long-running inference jobs to prevent blocking the GPU and to manage disk
   usage.


   .. py:attribute:: local_spool_dir


   .. py:attribute:: remove_on_upload
      :value: False



   .. py:attribute:: upload_batch_size
      :value: 100



   .. py:attribute:: s3_bucket


   .. py:attribute:: s3_prefix


   .. py:attribute:: s3_client


   .. py:attribute:: s3_pool


   .. py:attribute:: local_writer


   .. py:method:: write(relative_path_no_ext, to_save)

      Write and schedule for upload.

      This method is non-blocking. It passes the write job to the local AsyncWriter, which will then put
      the completed file path onto the upload queue for the background worker.



   .. py:method:: close()

      Shuts down the uploader.
      1. Waits for all local file writes to finish.
      2. Signals the upload worker to stop.
      3. Waits for the upload worker to finish its batches.
      4. Waits for all S3 uploads to complete.