Set @Step Configurations
In addition to designing a pipeline based on Python code in a .py file, you can also change the Kuberentes Configuration that will be applied to the Kubernetes Pipeline. There are a variety of different configuration options available, to see the full list review the DeployConfig API. To apply Kubernetes Configurations based on the DeployConfig API you will need to update the associated *@step *and @pipeline decorators. In the deploy_config argument, which will be demonstrated below, you can either pass a DeployConfig object or a dictionary for Kale to construct a DeployConfig object.
Please follow along in your own copy of our notebook as we complete the steps below.
1. Update the split(x,y) function to include Kubernetes Configuration
Update the @step decorator for the split(x,y) function to include CPU and Memory limits as per the code below.
deploy_config={"limits": {"cpu": "100m", "memory": "1Gi"}}
Kale will instruct Kubernetes to apply these limits on the Pod that runs this step.
[2. Apply a label to the model_training function]
Now you will extend the *model_training *step declaration and use the deploy_config argument to set labels. Kale will then instruct Kubernetes to apply these labels on the Pod that runs this step. Update the @step decorator for the model_training(x,y) function to include to apply the labels as per the code below.
deploy_config={“labels”: {“training-step”: “true”}}
[
]{.p style=“box-sizing: border-box; color: #404040; font-family: Consolas, monospace; font-size: 15px; white-space: pre;”}
3. Open a new terminal
Select Terminal from the launcher to open a new terminal.
4. Deploy Code as Kubeflow Pipeline
Make sure you are in the directory with the Python code, kale-sdk-datavol-1 and execute the following to deploy the code as a Kubeflow Pipeline.
python3 -m kale kale_sdk.py --kfp
Review the output to ensure no errors and look for run URL as an indication of success.
5. Review Pipeline Run WHILE RUNNING
The Kubernetes Configuration modifications you are applying will be visible in the Pod Logs however once the Kubeflow Pipeline is complete the Pods will be recycled and the logs will be gone. Therefore to see the modifications you will need to review the running Kubeflow Pipeline. Navigate to the Kubeflow UI and open the Runs UI so you can see the Kubeflow Pipeline during execution - make sure you do this while the pipeline is running.
The number of runs will continue to increase throughout the course - notice that the runtime will begin to increase as the code grows and the demand on the Kubeflow Pipeline increases throughout the course.
6. Access the pipeline
Select the latest binary-classification run to review the pipeline and select either the data_split or the *model_training *step and then select the *Pod *UI option. Scrolling through the output you will see the changes that you have made. For example model_training will have the tag applied as shown below:
You may wait until the Kubeflow Pipeline has finished execution to proceed if you would like. The green checkmarks indicate that the steps in the pipeline were able to execute without issue. Please look at the Kubernetes Documentation for a full breakdown of the Pod Log structure output including limits, requests, labels and more: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#example-1
@Pipeline Configuration Manipulations
In addition to specifying deploy_config argument values within @step decorators, you can also specify Kubernetes configurations to be applied on all steps of the Kubeflow Pipeline. Applying deploy_config argument values in the @pipeline decorator will apply the same to all steps in the pipeline since they are deployed as part of the function decorated with @pipeline. Kale will then try to merge the pipeline-level DeployConfig with any provided for individual steps, giving precedence to the individual step configuration. You will experiment with this in the challenge in the next section.
Please make sure your code is working before proceeding to the next section - if you want to download the complete code you can do so by clicking here.