Base Abstractions
Before we get into the topic of creating custom stack component flavors, let us briefly discuss some of the important design choices behind the abstraction of a ZenML flavor. The overall implementation revolves around three base interfaces, namely theStackComponent
, the StackComponentConfig
, and the Flavor
.
Base Abstraction 1: StackComponent
The StackComponent
is utilized as an interface to define the logic behind the functionality of a flavor. For instance, you can take a look at the BaseArtifactStore
example down below. By inheriting from the StackComponent
, the BaseArtifactStore
establishes the interface for all artifact stores. Any flavor of an artifact store needs to follows the standards set by this base class.
Base Abstraction 2: StackComponentConfig
As its name suggests, the StackComponentConfig
is used to configure a stack component instance. It is separated from the actual implementation on purpose. This way, ZenML can use this class to validate the configuration of a stack component during its registration/update, without having to import heavy (or even non-installed) dependencies. Let us continue with the same example up above and take a look at the BaseArtifactStoreConfig
.
BaseModel
as a base class, ZenML is able to configure and serialize these configuration properties while being able to add a validation layer to each implementation.
If you take a closer look at the example above, you will see that, through the BaseArtifacStoreConfig
, each artifact store will require users to define a path
variable along with a list of SUPPORTED_SCHEMES
. Using this configuration class, ZenML can check if the given path
is actually supported.
Similar to the example above, you can use class variables by denoting them
with the
ClassVar[..]
, which are also excluded from the serialization.Base Abstraction 3: Flavor
Ultimately, the Flavor
abstraction is responsible for bringing the implementation of a StackComponent
together with the corresponding StackComponentConfig
definition to create a Flavor
.
BaseArtifactStoreFlavor
sets the correct type
property and introduces the BaseArtifactStoreConfig
as the default configuration for all ZenML artifact stores.
Implementing a Custom Stack Component Flavor
Using all the abstraction layers above, let us create a custom artifact store flavor, starting with the configuration.Flavor
. Make sure that you give your flavor a unique name here.
Managing a Custom Stack Component Flavor
Once your implementation is complete, you can register it through the CLI:If your custom stack component flavor requires special setup before it can be
used, check out the Managing Stack Component
States section for more
details.
Describe MLOps Stack Component Flavors
Extending Specific Stack Components
If you would like to learn more about how to build a custom stack component flavor for a specific stack component, please check the links below:Type of Stack Component | Description |
---|---|
Orchestrator | Orchestrating the runs of your pipeline |
Artifact Store | Storage for the artifacts created by your pipelines |
Container Registry | Store for your containers |
Secrets Manager | Centralized location for the storage of your secrets |
Step Operator | Execution of individual steps in specialized runtime environments |
Model Deployer | Services/platforms responsible for online model serving |
Feature Store | Management of your data/features |
Experiment Tracker | Tracking your ML experiments |
Alerter | Sending alerts through specified channels |
Annotator | Annotating and labeling data |
Data Validator | Validating and monitoring your data |