allensdk.api.cloud_cache.cloud_cache module

class allensdk.api.cloud_cache.cloud_cache.BasicLocalCache(cache_dir: str | Path, project_name: str, ui_class_name: str | None = None)

Bases: ABC

A class to handle the loading and accessing a project’s data and metadata from a local cache directory. Does NOT include any ‘smart’ features like: 1. Keeping track of last loaded manifest 2. Constructing symlinks for valid data from previous dataset versions 3. Warning of outdated manifests

For those features (and more) see the CloudCacheBase class

Parameters:

cache_dir: str or pathlib.Path

Path to the directory where data and metadata are stored on the local system

project_name: str

the name of the project this cache is supposed to access. This will be the root directory for all files stored in the bucket.

ui_class_name: Optional[str]

Name of the class users are actually using to manipulate this functionality (used to populate helpful error messages)

property cache_dir: str

Return the cache directory path.

Returns:

str

Full cache directory path

property current_manifest: None | str

The name of the currently loaded manifest

data_path(file_id) → dict

Return the local path to a data file, and test for the file’s existence

Parameters:

file_id:

The unique identifier of the file to be accessed

Returns:

dict

‘local_path’ will be a pathlib.Path pointing to the file’s location

‘exists’ will be a boolean indicating if the file exists in a valid state

‘file_attributes’ is a CacheFileAttributes describing the file in more detail

Raises:

RuntimeError

If the file cannot be downloaded

property file_id_column: str

The col name in metadata files used to uniquely identify data files

get_file_attributes(file_id)

Retrieve file attributes for a given file_id from the meatadata.

Parameters:

file_id: str or int

The unique identifier of the file to be accessed

Returns:

CacheFileAttributes

property latest_manifest_file: str

parses on-line available manifest files for semver string and returns the latest one self.manifest_file_names are assumed to be of the form ‘<anything>_v<semver_str>.json’

Returns:

str

the filename whose semver string is the latest one

list_all_downloaded_manifests() → list

Return a list of all of the manifest files that have been downloaded for this dataset

load_manifest(manifest_name: str)

Load a manifest from this dataset.

Parameters:

manifest_name: str

The name of the manifest to load. Must be an element in self.manifest_file_names

property manifest_file_names: list

Sorted list of manifest file names associated with this dataset

property manifest_prefix: str

On-line prefix for manifest files

property metadata_file_names: list

List of metadata file names associated with this dataset

metadata_path(fname: str) → dict

Return the local path to a metadata file, and test for the file’s existence

Parameters:

fname: str

The name of the metadata file to be accessed

Returns:

dict

‘path’ will be a pathlib.Path pointing to the file’s location

‘exists’ will be a boolean indicating if the file exists in a valid state

‘file_attributes’ is a CacheFileAttributes describing the file in more detail

Raises:

RuntimeError

If the file cannot be downloaded

property project_name: str

The name of the project that this cache is accessing

property ui

property version: str

The version of the dataset currently loaded

class allensdk.api.cloud_cache.cloud_cache.CloudCacheBase(cache_dir, project_name, ui_class_name=None)

Bases: BasicLocalCache

A class to handle the downloading and accessing of data served from a cloud storage system

Parameters:

cache_dir: str or pathlib.Path

Path to the directory where data will be stored on the local system

project_name: str

the name of the project this cache is supposed to access. This will be the root directory for all files stored in the bucket.

ui_class_name: Optional[str]

Name of the class users are actually using to manipulate this functionality (used to populate helpful error messages)

compare_manifests(manifest_0_name: str, manifest_1_name: str) → str

Compare two manifests from this dataset. Return a dict containing the list of metadata and data files that changed between them

Note: this assumes that manifest_0 predates manifest_1

Parameters:

manifest_0_name: str

manifest_1_name: str

Returns:

str

A string summarizing all of the changes going from manifest_0 to manifest_1

construct_local_manifest() → None

Construct the dict that maps between file_hash and absolute local path. Save it to self._downloaded_data_path

download_data(file_id) → Path

Return the local path to a data file, downloading the file if necessary

Parameters:

file_id:

The unique identifier of the file to be accessed

Returns:

pathlib.Path

The path indicating where the file is stored on the local system

Raises:

RuntimeError

If the file cannot be downloaded

download_metadata(fname: str) → Path

Return the local path to a metadata file, downloading the file if necessary

Parameters:

fname: str

The name of the metadata file to be accessed

Returns:

pathlib.Path

The path indicating where the file is stored on the local system

Raises:

RuntimeError

If the file cannot be downloaded

get_metadata(fname: str) → DataFrame

Return a pandas DataFrame of metadata

Parameters:

fname: str

The name of the metadata file to load

Returns:

pd.DataFrame

Notes

This method will check to see if the specified metadata file exists locally. If it does not, the method will download the file. Use self.metadata_path() to find where the file is stored

property latest_downloaded_manifest_file: str

parses downloaded available manifest files for semver string and returns the latest one self.manifest_file_names are assumed to be of the form ‘<anything>_v<semver_str>.json’

Returns:

str

the filename whose semver string is the latest one

load_last_manifest()

If this Cache was used previously, load the last manifest used in this cache. If this cache has never been used, load the latest manifest.

load_latest_manifest()

load_manifest(manifest_name: str)

Load a manifest from this dataset.

Parameters:

manifest_name: str

The name of the manifest to load. Must be an element in self.manifest_file_names

summarize_comparison(manifest_0_name: str, manifest_1_name: str) → Dict[str, List[Tuple[str, str]]]

Compare two manifests from this dataset. Return a dict containing the list of metadata and data files that changed between them

Note: this assumes that manifest_0 predates manifest_1 (i.e. changes are listed relative to manifest_0)

Parameters:

manifest_0_name: str

manifest_1_name: str

Returns:

result: Dict[List[Tuple[str, str]]]

result[‘data_changes’] lists changes to data files result[‘metadata_changes’] lists changes to metadata files

Notes

Changes are tuples of the form (fname, string describing how fname changed)

e.g.

(‘data/f1.txt’, ‘data/f1.txt renamed data/f5.txt’) (‘data/f2.txt’, ‘data/f2.txt deleted’) (‘data/f3.txt’, ‘data/f3.txt created’) (‘data/f4.txt’, ‘data/f4.txt changed’)

class allensdk.api.cloud_cache.cloud_cache.LocalCache(cache_dir, project_name, ui_class_name=None)

Bases: CloudCacheBase

A class to handle accessing of data that has already been downloaded locally. Supports multiple manifest versions from a given dataset.

Parameters:

cache_dir: str or pathlib.Path

Path to the directory where data will be stored on the local system

project_name: str

the name of the project this cache is supposed to access. This will be the root directory for all files stored in the bucket.

ui_class_name: Optional[str]

Name of the class users are actually using to maniuplate this functionality (used to populate helpful error messages)

exception allensdk.api.cloud_cache.cloud_cache.MissingLocalManifestWarning

Bases: UserWarning

exception allensdk.api.cloud_cache.cloud_cache.OutdatedManifestWarning

Bases: UserWarning

class allensdk.api.cloud_cache.cloud_cache.S3CloudCache(cache_dir, bucket_name, project_name, ui_class_name=None)

Bases: CloudCacheBase

A class to handle the downloading and accessing of data served from an S3-based storage system

Parameters:

cache_dir: str or pathlib.Path

Path to the directory where data will be stored on the local system

bucket_name: str

for example, if bucket URI is ‘s3://mybucket’ this value should be ‘mybucket’

project_name: str

the name of the project this cache is supposed to access. This will be the root directory for all files stored in the bucket.

ui_class_name: Optional[str]

Name of the class users are actually using to maniuplate this functionality (used to populate helpful error messages)

property bucket_name: str

property s3_client

class allensdk.api.cloud_cache.cloud_cache.StaticLocalCache(cache_dir, project_name, ui_class_name=None)

Bases: BasicLocalCache

A class to handle accessing data that has already been downloaded locally and whose directory structure and/or contained files are not expected to be changed in any way. Does NOT support multiple manifest versions for a given dataset.

Example intended use case:

Calling VisualBehaviorOphysProjectCache.from_local_cache(use_static_cache=True) where the cache directory is a mounted S3 public bucket.

Parameters:

cache_dir: str or pathlib.Path

Path to the directory where data will be stored on the local system

project_name: str

the name of the project this cache is supposed to access. This will be the root directory for all files stored in the bucket.

ui_class_name: Optional[str]

Name of the class users are actually using to maniuplate this functionality (used to populate helpful error messages)

compare_manifests(manifest_0_name: str, manifest_1_name: str)

list_all_downloaded_manifests() → list

Return a list of all of the manifest files for this dataset. For the StaticLocalCache, this will only be the latest manifest.

load_last_manifest()

For the StaticLocalCache always load the latest manifest.

load_manifest(manifest_name: str)

Load a manifest from this dataset.

Parameters:

manifest_name: str

The name of the manifest to load. Must be an element in self.manifest_file_names