allensdk.brain_observatory.behavior.behavior_project_cache.project_cache_base module

class allensdk.brain_observatory.behavior.behavior_project_cache.project_cache_base.ProjectCacheBase(fetch_api: ProjectCloudApiBase | BehaviorProjectLimsApi, fetch_tries: int = 2)

Bases: object

BUCKET_NAME: str = None

PROJECT_NAME: str = None

compare_manifests(manifest_0_name: str, manifest_1_name: str) → str

Compare two manifests from this dataset. Return a dict containing the list of metadata and data files that changed between them

Note: this assumes that manifest_0 predates manifest_1

Parameters:

manifest_0_name: str

manifest_1_name: str

Returns:

str

A string summarizing all of the changes going from manifest_0 to manifest_1

construct_local_manifest() → None

Construct the local file used to determine if two files are duplicates of each other or not. Save it into the expected place in the cache. (You will see a warning if the cache thinks that you need to run this method).

current_manifest() → None | str

Return the name of the dataset manifest currently being used by this cache.

classmethod from_lims(manifest: str | Path | None = None, version: str | None = None, cache: bool = False, fetch_tries: int = 2, lims_credentials: DbCredentials | None = None, mtrain_credentials: DbCredentials | None = None, host: str | None = None, scheme: str | None = None, asynchronous: bool = True, data_release_date: str | List[str] | None = None, passed_only: bool = True) → ProjectCacheBase

Construct a ProjectCacheBase with a lims api.

Parameters:

manifeststr or Path

full path at which manifest json will be stored

versionstr

version of manifest file. If this mismatches the version recorded in the file at manifest, an error will be raised.

cachebool

Whether to write to the cache

fetch_triesint

Maximum number of times to attempt a download before giving up and raising an exception. Note that this is total tries, not retries

lims_credentialsDbCredentials

Optional credentials to access LIMS database. If not set, will look for credentials in environment variables.

mtrain_credentials: DbCredentials

Optional credentials to access mtrain database. If not set, will look for credentials in environment variables.

hoststr

Web host for the app_engine. Currently unused. This argument is included for consistency with EcephysProjectCache.from_lims.

schemestr

URI scheme, such as “http”. Currently unused. This argument is included for consistency with EcephysProjectCache.from_lims.

asynchronousbool

Whether to fetch from web asynchronously. Currently unused.

data_release_date: str or list of str

Use to filter tables to only include data released on date ie 2021-03-25 or [‘2021-03-25’, ‘2021-08-12’]

passed_only

Whether to limit to data with workflow_state set to ‘passed’ and ‘published’

Returns

=======

ProjectCacheBase

ProjectCacheBase instance with a LIMS fetch API

classmethod from_local_cache(cache_dir: str | Path, use_static_cache: bool = False) → ProjectCacheBase

instantiates this object with a local cache.

Parameters:

cache_dir: str or pathlib.Path

Path to the directory where data will be stored on the local system

project_name: str

the name of the project this cache is supposed to access. This project name is the first part of the prefix of the release data objects. I.e. s3://<bucket_name>/<project_name>/<object tree>

Returns:

ProjectCacheBase instance

classmethod from_s3_cache(cache_dir: str | Path, bucket_name_override: str | None = None) → ProjectCacheBase

instantiates this object with a connection to an s3 bucket and/or a local cache related to that bucket.

Parameters:

cache_dir: str or pathlib.Path

Path to the directory where data will be stored on the local system

bucket_name_override: str

Overrides the default bucket name for this class. Useful for testing for example, if bucket URI is ‘s3://mybucket’ this value should be ‘mybucket’

Returns:

ProjectCacheBase instance

latest_downloaded_manifest_file() → str

Return the name of the most up to date data manifest available on your local system.

latest_manifest_file() → str

Return the name of the most up to date data manifest corresponding to this dataset, checking in the cloud if this is a cloud-backed cache.

list_all_downloaded_manifests() → list

Return a sorted list of the names of the manifest files that have been downloaded to this cache.

list_manifest_file_names() → list

Return a sorted list of the names of the manifest files associated with this dataset.

load_latest_manifest() → None

Load the manifest corresponding to the most up to date version of the dataset.

load_manifest(manifest_name: str)

Load a specific versioned manifest for this dataset.

Parameters:

manifest_name: str

The name of the manifest to load. Must be an element in self.manifest_file_names

property manifest