Spark on Kubernetes

About this session

Running Apache Spark on Kubernetes is well documented and still hard to get right the first time: the driver and executors run as pods, and container images, service accounts, networking, and storage all have to line up before a single job starts.

In this free, two-hour working session we build the environment from scratch and walk through how the pieces fit together. Then we submit a real Spark job live, follow its execution, and inspect its status, logs, and Spark UI, so you see the whole path from an empty cluster to a running job.

From there we move to the questions that matter in production: when Spark on Kubernetes is more cost-efficient than managed Hadoop clusters in the cloud and what trade-offs and operational responsibilities come with it, plus resource sizing, scaling, workload isolation, reliability, and monitoring. We close with Q&A.

What you'll learnWhat's covered

• Set up the environment from scratch and understand the Kubernetes infrastructure Spark needs
• See how Spark drivers and executors run in pods, and how container images, permissions, networking, and storage fit into the setup
• Submit a Spark job live, follow its execution, and inspect its status, logs, and Spark UI
• Judge when Spark on Kubernetes is more cost-efficient than managed Hadoop clusters in the cloud, and what trade-offs and operational responsibilities come with it
• Plan for production: resource sizing, scaling, workload isolation, reliability, and monitoring

Who it's for

Data and AI professionals: data engineers, platform engineers, and anyone who runs Apache Spark™ and wants to run it on Kubernetes with confidence, from first setup to production.

Your presenter

Onehouse TeamOnehouse