emmi_data_management.checkpoint.providers.huggingface_provider

Classes

HFProvider

Provides a minimalistic interface to fetch a checkpoint file from Huggingface.

Module Contents

class emmi_data_management.checkpoint.providers.huggingface_provider.HFProvider

Bases: emmi_data_management.checkpoint.types.Provider

Provides a minimalistic interface to fetch a checkpoint file from Huggingface.

Examples

hf://org/name@rev?filename=model.pt
hf://org/name?filename=model.pt
Where:

  • rev is optional (tag/branch/commit after @)

  • filename is required (query parameter ?filename=...)

scheme = 'hf'
can_handle(uri)
Parameters:

uri (str)

Return type:

bool

static parse_uri(uri)

Extracts repo name, revision and filename from HF URI.

Parameters:

uri (str) – Input URI as: ‘hf://org/name?filename=MODEL_FILENAME’.

Returns:

Repository name, filename and revision.

Return type:

  • tuple[str, str, Optional[str]]

estimate_size(uri)

Extracts an approximate size of a file from HF URI.

Parameters:

uri (str) – Input URI as: ‘hf://org/name[@rev]?filename=MODEL_FILENAME’.

Returns:

Estimated size of the remote file. - None: If no size is available.

Return type:

  • int

fetch(uri, destination_dir, *, compute_hash=True)

Returns a local file path for the given HF URI after downloading from the HF hub.

Parameters:

  • uri (str) – Input URI where the file is located.

  • destination_dir (pathlib.Path) – Destination directory where the file need to be located.

  • compute_hash (bool) – Compute the hash of the file on the local file system.

Returns:

Local file path.

Return type:

  • Path