Create Custom Container Steps: Plain Values

You can define custom containers for use as pipeline steps with inputs and outputs thereby creating pipelines that execute both function-based and container-based steps. There are two ways to do this - using plain values and using data files. You will first explore plain values and then explore custom container steps with data files.

Please follow along in your own copy of our notebook as we complete the steps below.

1. Download custom_container_steps_plainvalues.py

For this unit, we will look at a new code example focused specifically on custom container steps.

First, download the new code file by clicking here and add to your Notebook Server.

2. Review Python Code

The first step gen_random_ints is function-based and generates and returns two random integers.

The second step add_random_ints is container-based and adds the two outputs of gen_random_ints and returns the result.

The add_random_ints step executes the following script, which is loaded from the image referenced gcr.io/arrikto/example-add-split.

sum=$(($1 + $2 ))

mkdir -p $(dirname "$3")

echo "$sum">$3

[Notice that the command input parameter of theContainerStepruns a custom bash script that adds two numbers and writes the result in a file. The script expect expects three arguments, the first two are the numbers to add and the third is the output file. Notice the file name format,{{file.<name>}} -use the argsparameter to pass these types of arguments.  ]{style=“font-family: ‘Open Sans’, Verdana, Arial, Helvetica, sans-serif;”}

The *print_result *is function-based and prints the output of add_random_ints.

[]{style=“font-family: ‘Open Sans’, Verdana, Arial, Helvetica, sans-serif;”}

The pipeline_func brings all the code together and defines the Kubeflow Pipeline.

The output of the add step is the contents of the add.sh file which is the result of the addition and is represented by *c *in the pipeline definition.

3. Open a new terminal

Select Terminal from the launcher to open a new terminal.

4. Deploy Code as Kubeflow Pipeline

Make sure you are in the directory with the Python code, kale-sdk-datavol-1 and execute the following to deploy the code as a Kubeflow Pipeline.

python3 -m kale custom_container_steps_plainvalues.py --kfp

5. Review Pipeline Runs

Navigate to the Kubeflow UI and open the Runs UI. The number of runs will continue to increase throughout the course.

6. Access the Pipeline run

Select the add-random-ints-*** run to see the Kubeflow Pipeline execute successfully with the custom container output.

7. Review add_random_ints

Select the add_random_ints step and review the Input/Output to see the random values generated and added together during the add_random_ints step

You have successfully executed a Kubeflow Pipeline using Custom Container Steps with Plain Values.

For the completed code please click here.