plinder.core.utils.cpl

Public dataset access with cloudpathlib and a local release cache.

plinder.core.utils.cpl.is_offline() → bool

Return whether remote access is disabled.

PLINDER_OFFLINE remains the canonical setting. The PLINDER_OFFLINE_MODE alias is accepted for compatibility with cluster launch environments that use the longer name.

plinder.core.utils.cpl.thread_pool(func: Callable[[T], None], items: Iterable[T]) → None

plinder.core.utils.cpl.download_paths(*, paths: list[Path], force_progress: bool = False) → None

Download pre-determined paths from the release mirror concurrently. This is useful when we want to process a pre-determined subset of the data rather than all of the contents of the dataset.

Parameters:

pathslist[Path]

the local paths to download, resolved against the release cache root

force_progressbool, default=False

if True, always display a progress bar

plinder.core.utils.cpl.get_plinder_path(*, rel: str = '', download: bool = True, force_progress: bool = False) → Path

Get the local cache path for a file or directory in the plinder release, downloading it first if requested. This provides a convenient way to manage local file caching since it is automatically synced from the release mirror on access in case remote files change.

Parameters:

relstr

Relative path to the file or directory.

downloadbool, default=True

if True, download the files

force_progressbool, default=False

if True, force progress bar even if < 10 files

Returns:

Path

The local cache path.