Last updated: November 3, 2022.
With ZenML 0.7.0, a lot has been revamped under the hood about how things are stored. Importantly what this means is that ZenML now has system-wide profiles that let you register stacks to share across several of your projects! If you still want to manage your stacks for each project folder individually, profiles let you continue to do that as well.
Most projects of any complexity will require passwords or tokens to access data and infrastructure, and for this purpose ZenML 0.7.0 introduces the Secrets Manager stack component to seamlessly pass around these values to your steps. Our AWS integration also allows you to use AWS Secrets Manager as a backend to handle all your secret persistence needs.
Finally, in addition to the new AzureML and Sagemaker Step Operators that version 0.6.3 brought, this release also adds the ability to run individual steps on GCP’s Vertex AI.
Beyond this, some smaller bugfixes and documentation changes combine to make ZenML 0.7.0 a more pleasant user experience. For a detailed look at what’s changed, give our full release notes a glance.
🗿 Profiles and Global Stack Storage
Ever wanted to use the same stack configuration in multiple projects? Well now
you can! ZenML 0.7.0 moves the storage of stack configurations out of the .zen
folder inside of each individual project and into the system-wide ZenML
application configuration folder. What this means is you can now interact with
ZenML using CLI commands from anywhere, not just in zen repository projects
with a .zen folder. Adding a new stack component, for instance an
orchestrator using
zenml orchestrator register kubeflow_orchestrator --type=kubeflow
means that this orchestrator will now be available to use in any ZenML projects on your local machine.
Still want separation of concerns with isolated environments for each project? This is where profiles come in. You can now register a new profile using:
zenml profile create NEW_PROFILE_NAME
zenml profile activate NEW_PROFILE_NAME
This provides you with a completely fresh environment that only has the default local stack pre-registered, where you can work without disturbing other profiles or projects. You can specify both globally and on a project (folder) level which profile to default to using. For ease of transition, any legacy ZenML repositories (projects) will automatically be migrated to a new isolated profile so you can maintain the separation you are used to:
🔑 Secret Management
Most projects of a certain complexity or using cloud infrastructure will
involve secrets of some kind. You use secrets, for example, when connecting to
AWS. These secrets, in this case the access_key_id and secret_access_key,
are usually stored in the ~/.aws/credentials file. You might find you need to
access those secrets from within your Kubernetes cluster as it runs individual
steps, or you might just want a centralized location for the storage of secrets
across your project. As of this release, ZenML offers a basic
local secrets manager and an
integration with the managed AWS Secrets Manager.
This now lets you easily specify secrets as dependencies for pipelines from the decorator:
@pipeline(required_integrations=[TENSORFLOW], secrets=["aws"], enable_cache=True)
def mnist_pipeline(importer, normalizer, trainer, evaluator):
"""Steps that require access to an AWS account here"""
➕ Other Updates, Additions and Fixes
Google Cloud’s Vertex AI is now available as a step operator to run individual steps of your pipeline in the cloud. Simply register it as you would any other stack component from the CLI:
zenml step-operator register vertex \
--type=vertex \
--project=zenml-core \
--service_account_path=... \
--region=europe-west1 \
--machine_type=n1-standard-4 \
--base_image=<CUSTOM_BASE_IMAGE> \
--accelerator_type=...
More details about the parameters that you can configure can be found in the class definition of Vertex Step Operator in the API docs.
Another change addresses the fact that while in most cases materializers should be used to control how artifacts are consumed and output from steps in a pipeline, there is sometimes a need to have a completely non-materialized artifact in a step. ZenML now provides this option of bypassing materialization.
🙌 Community Contributions
We received a contribution from Avram, in which he fixed a typo in our documentation. Thank you, Avram!
Contribute to ZenML!
Join our Slack to let us know what you think we should build next!
Keep your eyes open for future releases and make sure to vote on your favorite feature of our roadmap to make sure it gets implemented as soon as possible.
[Photo by Jason Leung on Unsplash]
