The Jupyter Notebook Stack has become the "de facto" platform used by data scientists to interactively work on big data problems. With the popularity of deep learning, there is also an increasing need for resources to make deep learning effective. In this session, we will discuss how we brought support for Kubernetes into Jupyter Enterprise Gateway and touch on some best practices on how to scale an interactive big data workloads across a Kubernets managed cluster.

In this session, we will discuss the limitations we have found when running interactive workloads on a Kubernetes environment and how we overcome some of these limitations by enabling distributed containers managed by Kubernetes using Jupyter Enterprise Gateway. We will also describe the roadblocks found during the implementation and how we overcome them. We also plan to discuss how these can be leveraged in different platforms to enhance scalability.