Base resource

Resource

Resource()

Base class for resources.

Use Resource subclasses to assign datasets to steps. The Vectice library supports a handful of common cases. Additional cases are generally easy to supply by deriving from this base class. In particular, subclasses must override this class' abstact methods (_build_metadata(), _fetch_data()).

Examples:

To create a custom resource class, inherit from Resource, and implement the _build_metadata() and _fetch_data() methods:

from vectice.models.resource import Resource
from vectice.models.resource.metadata import DatasetSourceOrigi, FilesMetadata

class MyResource(Resource):
    _origin = "Data source name"

    def _build_metadata(self) -> FilesMetadata:  # (1)
        files = ...  # fetch file list from your custom storage
        total_size = ...  # compute total file size
        return FilesMetadata(
            size=total_size,
            origin=self._origin,
            files=files,
            usage=self.usage,
        )

    def _fetch_data(self) -> dict[str, bytes]:
        files_data = {}
        for file in self.metadata.files:
            file_contents = ...  # fetch file contents from your custom storage
            files_data[file.name] = file_contents
        return files_data

1. Return FilesMetadata for data stored in files, DBMetadata for data stored in a database.

Source code in src/vectice/models/resource/base.py

@abstractmethod
def __init__(self):
    """Initialize a resource."""
    self._metadata: FilesMetadata | None = None
    self._data: dict[str, bytes] | None = None

data property

data: dict[str, bytes]

The resource's data.

Returns:

Type	Description
dict[str, bytes]	The resource's data.

metadata writable property

metadata: FilesMetadata

The resource's metadata.

Returns:

Type	Description
FilesMetadata	The resource's metadata.

usage writable property

usage: DatasetSourceUsage | None

The resource's usage.

Returns:

Type	Description
DatasetSourceUsage | None	The resource's usage.