Install Apache Spark on Ubuntu 24.04 LTS: A Step-by-Step Guide

Hey there, data enthusiasts and coding wizards! So, you’re looking to get Apache Spark up and running on your shiny new Ubuntu 24.04 LTS system, huh? Awesome choice, guys! Spark is an absolute beast when it comes to big data processing and real-time analytics, and getting it installed on the latest Ubuntu is totally doable with a little guidance. We’re going to walk through this whole process together, step-by-step, so you can start crunching those massive datasets in no time. Forget those complicated tutorials that leave you scratching your head; we’re keeping it real and practical here. By the end of this guide, you’ll have a fully functional Spark environment ready for action. So, grab your favorite beverage, settle in, and let’s dive into the exciting world of Apache Spark on Ubuntu 24.04!

Prerequisites: What You’ll Need Before We Start

Alright, before we jump headfirst into the Spark installation, let’s make sure you’ve got all your ducks in a row. Having these prerequisites sorted will make the entire process a breeze, trust me. First off, you’ll need a system running Ubuntu 24.04 LTS . It’s always a good idea to use the Long Term Support (LTS) version for stability, especially when you’re setting up critical infrastructure like a Spark cluster. Make sure your system is up-to-date; you can do this by running sudo apt update && sudo apt upgrade -y in your terminal. This ensures you have the latest security patches and software versions, which can prevent a whole lot of headaches down the line. Next up, you absolutely need Java Development Kit (JDK) installed. Spark is built on top of the Java Virtual Machine (JVM), so Java is a non-negotiable dependency. We recommend installing OpenJDK, which is free and open-source. The specific version you need can depend on your Spark version, but generally, OpenJDK 11 or 17 are safe bets. We’ll cover how to install it in the next section. You’ll also need SSH access to your Ubuntu machine, especially if you’re setting up a distributed cluster. Even for a single-node setup, SSH is handy for remote management. Ensure that the SSH server is installed and running by typing sudo apt install openssh-server . Finally, a basic understanding of the Linux command line is super helpful. We’ll be using commands like cd , ls , mkdir , wget , and tar , so if you’re comfortable with those, you’re golden. Don’t worry if you’re not a Linux guru; I’ll explain each command as we go. Having wget installed is also crucial for downloading the Spark binaries. If you don’t have it, just run sudo apt install wget . With all these pieces in place, you’re all set to conquer the Spark installation. Let’s get this party started!

Step 1: Installing Java (OpenJDK)

First things first, guys, we gotta get Java squared away. Apache Spark relies heavily on the Java Virtual Machine (JVM), so without Java, Spark just won’t run. We’re going to install OpenJDK, which is the open-source implementation of the Java Platform, Standard Edition. It’s reliable, free, and works perfectly with Spark. Open your terminal and let’s get started.

Update your package list:

It’s always best practice to update your package index before installing anything new. This ensures you’re getting the latest available versions of software.

sudo apt update

Install a recommended OpenJDK version:

For Apache Spark, OpenJDK 11 or OpenJDK 17 are generally recommended. Let’s go with OpenJDK 17 as it’s more recent and widely supported. If you prefer OpenJDK 11, just replace openjdk-17-jdk with openjdk-11-jdk .

sudo apt install openjdk-17-jdk -y

The -y flag automatically answers ‘yes’ to any prompts, making the installation smoother.

Verify the Java installation:

Once the installation is complete, let’s check if Java is installed correctly and see which version we have.

java -version

You should see output similar to this (the exact version numbers might differ slightly):

openjdk version "17.0.x" ...

If you see this, congratulations! You’ve successfully installed Java. Now, Spark has a dependency on the JAVA_HOME environment variable. This variable tells Java applications where to find the Java installation. We need to set this up.

Find the Java installation path:

Most likely, Java is installed in /usr/lib/jvm/ . Let’s find the exact path for your OpenJDK 17 installation. You can usually do this with the update-alternatives command:

sudo update-alternatives --config java

This command will list all installed Java versions and show you the path to the one that’s currently selected. Note down the path that looks something like /usr/lib/jvm/java-17-openjdk-amd64 .

Set the JAVA_HOME environment variable:

Now, we need to add JAVA_HOME to your system’s environment variables. We’ll edit the ~/.bashrc file (or ~/.zshrc if you’re using Zsh) to make this permanent for your user.

Open the file with a text editor like nano :

nano ~/.bashrc

Scroll to the bottom of the file and add the following lines, replacing the path with the one you found earlier:

export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export PATH=$JAVA_HOME/bin:$PATH

Save the file (Ctrl+O, Enter) and exit nano (Ctrl+X).

Apply the changes:

To make these changes effective in your current terminal session, you need to source the .bashrc file:

source ~/.bashrc

Verify JAVA_HOME:

Finally, let’s check if JAVA_HOME is set correctly:

echo $JAVA_HOME

This should print the Java installation path you just set. If you see the path, great job! Java is now ready for Spark.

Step 2: Downloading Apache Spark

Alright, Java is all set. Now it’s time to get our hands on Apache Spark itself! We need to download the pre-built binary distribution. It’s usually best to download a stable release. You can find the latest stable releases on the official Apache Spark download page. However, for this guide, we’ll download a specific version that’s known to work well.

Navigate to a download directory:

It’s good practice to keep your downloads organized. Let’s create a directory for Spark downloads or navigate to your preferred download location. I usually create a downloads folder in my home directory.

cd ~ 
cd downloads

If the downloads directory doesn’t exist, you can create it with mkdir downloads .

Find the download link:

Head over to the Apache Spark Downloads page . Look for the