(Hands-On) Convert Your Notebook to a Kubeflow Pipeline

The following video is an excerpt from a live session dedicated to reviewing the Jupyter Notebook that makes up the solution for this particular Kaggle Competition Example.

If you would like to sign up for the live session please do so on our events page: https://www.arrikto.com/kubeflow-mlops-events/

After watching this video you will deploy a Kubeflow Pipeline to create a model for Dog Breed Classification based on the Jupyter Notebook.

1.  Enable Kale

Enable Kale by clicking on the Kale icon in the left pane of the notebook.

2. Explore per-cell dependencies

See how multiple cells can be part of a single pipeline step, and how a pipeline step may depend on previous steps. For example, the image below shows multiple cells that are part of the same pipeline step. They have the same red color and they depend on a previous pipeline step.

The only thing one needed to define pipeline steps is edit the cell by clicking the pencil button on the top right corner of the cell, name the pipeline step, and define its dependencies. A dependency can be any of the pipeline steps. Kale discovers all the defined pipeline steps automatically and presents a dropdown list with them to the user.

3. Compile and run

Click the COMPILE AND RUN button.

Now Kale takes over and transforms your notebook, by converting it to a KFP pipeline. Also, because Kale integrates with Rok, it will take a snapshot of the notebook’s volumes, you can watch the progress of the snapshot in real-time. Rok takes care of data versioning and allows you to reproduce the whole environment as it was when you clicked the COMPILE AND RUN button. This way, you have a time machine for your data and code, an exact versioned and reproducible point from where your pipeline starts, and your pipeline will run in an identical environment to the one you developed your code on, without needing to build new docker images.

4. View the run

The pipeline was compiled and uploaded to Kubeflow Pipelines. Now click the link to go to the Kubeflow Pipelines UI and view the run.

Wait for the run to finish.

Congratulations! You just ran an end-to-end pipeline in Kubeflow Pipelines, starting from your notebook!