Last updated: November 3, 2022.
This release blog describes the changes for three releases v0.13.0 (major release), v0.13.1 and v0.13.2 (minor releases). v0.13.0 brings the first iteration of our Apache Spark integration. This integration opens up the possibility of running large-scale workloads on single-node machines or clusters. Additionally, this release also allows you to run custom code along with your models using KServe or Seldon. Lastly, we introduce the Stack Recipe as a convenient way to spin up perfectly configured infrastructure with ease. v0.13.1 and v0.13.2 includes several bugfixes and quality of life improvements for ZenML users.
Version 0.13.0 is chock-full of exciting features:
- Spark Integration - You can now run Spark jobs within ZenML with the long-awaited Spark integration.
- Custom Code Deployment - It’s now possible to run custom code alongside your model with KServe and Seldon.
- Stack Recipes - We introduce a convenient way to spin up infrastructures using Stack Recipes and how you can extend them to your needs.
View the full release notes here.
Version 0.13.1 comes with several quality of life improvements:
- Specify the exact order in which your pipelines steps should be
executed, e.g., via
step_b.after(step_a) - It’s now possible to use TensorBoard with PyTorch and other modeling frameworks.
- You can now configure the Evidently integration to ignore specific columns in your datasets.
View the full release notes here.
Version 0.13.2 comes with a new local Docker orchestrator and many other improvements and fixes:
- You can now run your pipelines locally in isolated Docker containers per step. This is useful to test whether the dockerization process will work in remote orchestrators like Kubeflow.
- @gabrielmbmb updated our MLflow experiment tracker to work with Databricks deployments.
- Documentation updates for cloud deployments and multi-tenancy Kubeflow support.
View the full release notes here.
As always, we’ve also included various bug fixes and lots of improvements to the documentation and our examples.
⚡ Spark Integration
To date, Spark has been the most requested feature on our Roadmap.
We heard you! And in this release, we present to you the long-awaited Spark integration!
With Spark, this release brings distributed processing into the ZenML toolkit. You can now run heavy data-processing workloads across machines/clusters as part of your MLOps pipeline and leverage all the distributed processing goodies that come with Spark.
We showcased how to use it in our community meetup on 17th August 2022 👇
Run the Spark integration example here.
🎯 Custom Code Deployment
We continue our streak in supporting model deployment in ZenML by introducing a feature that allows you to deploy custom code alongside your models on KServe or Seldon.
With this, you can now ship the model with the pre-processing and post-processing code to run within the deployment environment.
We showcased how to deploy custom code with a model during our community meetup on 24th August 2022
Run the example here.
🥘 Spin Up Infrastructure with Stack Recipes
Spinning up and configuring infrastructure is a difficult part of the MLOps journey and can easily become a barrier to entry.
Worry not! Now you don’t need to get lost in the infrastructure configuration details.
Using our mlops-stacks repository, it is now possible to spin up perfectly-configured infrastructure with the corresponding ZenML stack using the ZenML CLI.
View the demo recorded during our community meetup on 31st August 2022 👇
Check out all the Stack Recipes here.
💔 Breaking Changes
This release introduces a breaking change to the CLI by adjusting the access to the stack component-specific resources for secrets managers and model deployers to be more explicitly linked to the component.
Here is how:
# `zenml secret register ...` becomes
zenml secrets-manager secret register ...
# `zenml served_models list` becomes
zenml model-deployer models list
➕ Other Updates, Additions, and Fixes
0.13.0
Model Deployment -
- Update kserve installation to 0.9 on kserve deployment example by @safoinme in #823.
- Custom deployment with KServe and Seldon Core by @safoinme in #841.
- Fix served models logs formatting error by @safoinme in #836.
Spark Integration -
Tekton Orchestrator -
Materializer -
- Pillow Image materializer by @strickvl in #820.
- Implement Recursive Built-In Container Materializer by @fa9r in #812.
CLI Improvement -
- Unify CLI concepts (removing
secret,featureandserved-models) by @strickvl in #833. - Add zenml stack recipe CLI commands by @wjayesh in #807.
Secrets -
- Add secret scoping to the Azure Key Vault by @stefannica in #830.
- Secrets references on stack component attributes by @schustmi in #817.
README page improvements -
- Update Readme with latest info from docs page by @dnth in #810.
- Typo on Readme by @dnth in #821.
- Put Slack call to action at the top of README page. by @dnth in #846.
Link checker and broken links -
- Add automated link check github actions by @dnth in #828.
- Link checker by @dnth in #818.
- Put link checker as part of CI by @dnth in #838.
- Fix broken links from link checker results by @dnth in #835.
Misc -
- Misc bugfixes by @schustmi in #842.
- Add missing requirement for step operators by @schustmi in #834.
- Change Quickstart to Use Tabular Data by @fa9r in #843.
- Add sleep before docker builds in release GH action by @schustmi in #849.
- New Docker build configuration by @schustmi in #811.
- Improve label studio error messages if secrets are missing or of wrong schema by @schustmi in #811.
- Fix the SQL zenstore to work with MySQL by @stefannica in #829.
- Allow setting caching via the
config.yamlby @strickvl in #827. - Handle file-io with context manager by @aliabbasjaffri in #825.
0.13.1
- Fix flag info on recipes in docs by @wjayesh in #854.
- Fix some materializer issues by @schustmi in #852.
- Add ignore columns for evidently drift detection by @SangamSwadiK in #851.
- TensorBoard Integration by @fa9r in #850.
- Add option to specify task dependencies by @schustmi in #858.
- Custom code readme and docs by @safoinme in #853.
0.13.2
- Update GitHub Actions by @fa9r in #864.
- Raise zenml exception when cyclic graph is detected by @schustmi in #866.
- Add source to segment identify call by @htahir1 in #868.
- Use default local paths/URIs for the local artifact and metadata stores by @stefannica in #873.
- Implement local docker orchestrator by @schustmi in #862.
- Update cheat sheet with latest CLI commands from 0.13.0 by @dnth in #867.
- Add a note about importing proper DockerConfiguration module by @jsuchome in #877.
- Bugfix/misc by @schustmi in #878.
- Fixed bug in tfx by @htahir1 in #883.
- Mlflow Databricks connection by @gabrielmbmb in #882.
- Refactor cloud guide to stack deployment guide by @wjayesh in #861.
- Add cookie consent by @strickvl in #871.
- Stack recipe CLI improvements by @wjayesh in 872.
- Kubeflow workaround added by @htahir1 in #886.
🙌 Community Contributions
- @aliabbasjaffri made their first contribution in #825.
- @SangamSwadiK made their first contribution in #851.
👩💻 Contribute to ZenML!
Join our Slack to let us know if you have an idea for a feature or something you’d like to contribute to the framework.
We have a new home for our roadmap where you can vote on your favorite upcoming feature or propose new ideas for what the core team should work on. You can vote without needing to log in, so please do let us know what you want us to build!
