How to Run Apache Kafka on Windows
Tags: apachekafka, opensource, linux, windows
Published: 2021-01-08 02:27:48


Is Windows your favorite development environment? Do you want to run Apache Kafka<sup>®</sup> on Windows? Thanks to the Windows Subsystem for Linux 2 (WSL 2), now you can, and with fewer tears than in the past. Windows still isn’t the recommended platform for running Kafka with production workloads, but for trying out Kafka, it works just fine. Let’s take a look at how it’s done.
<ul>
 	<li><a href="#set-up">Set up your environment</a>
<ul>
 	<li><a href="#install-wsl-2">Install WSL 2</a>
<ul>
 	<li><a href="#enable-windows-subsystem">Enable the Windows Subsystem for Linux</a></li>
 	<li><a href="#enable-virtual-machine">Enable the Virtual Machine feature</a></li>
 	<li><a href="#linux-kernel-update">Get the Linux kernel update</a></li>
 	<li><a href="#set-default">Set the default WSL version</a></li>
 	<li><a href="#install-linux">Install your preferred Linux distribution</a></li>
</ul>
</li>
 	<li><a href="#install-java">Install Java</a></li>
 	<li><a href="#download-kafka">Download Kafka</a></li>
 	<li><a href="#start-kafka-cluster">Start the Kafka cluster</a></li>
 	<li><a href="#produce-and-consume">Produce and consume some messages</a></li>
 	<li><a href="#stop-kafka">Stop Kafka</a></li>
</ul>
</li>
 	<li><a href="#kafka-and-windows">Kafka and Windows: The good, the bad, and the ugly</a></li>
 	<li><a href="#getting-started">You’re just getting started!</a></li>
 	<li><a href="#kafka-on-windows">Kafka on Windows? What made this possible?</a></li>
</ul>
<h2 id="set-up"><a id="set-up"></a>Set up your environment</h2>
<h3 id="install-wsl-2"><a id="install-wsl-2"></a>Install WSL 2</h3>
The Windows Subsystem for Linux 2 makes it all possible. <a href="https://docs.microsoft.com/en-us/windows/wsl/about" target="_blank" rel="noopener noreferrer">Microsoft describes WSL 2</a> as “a GNU/Linux environment—including most command line tools, utilities, and applications—directly on Windows, unmodified, without the overhead of a traditional virtual machine or dual boot setup.”

Make sure you’re running Windows 10, version 1903.18362 or higher. Click <strong>Start</strong> and navigate to <strong>Settings &gt; System &gt; About</strong>. In the “Windows specifications” section, find the “OS build.”

<img class="aligncenter wp-image-28883 size-full" src="https://cdn.confluent.io/wp-content/uploads/window-specifications.png" alt="Windows specifications | Windows 10 Enterprise" width="942" height="504" />

If you’re on the Windows Update train, you probably have the latest version and are good to go. If not, you need to <a href="https://support.microsoft.com/en-us/help/4027667/windows-10-update" target="_blank" rel="noopener noreferrer">update Windows 10</a>.

When you’re sure that Windows is up to date, follow these instructions below to install WSL 2.
<h4 id="enable-windows-subsystem"><a id="enable-windows-subsystem"></a>Enable the Windows Subsystem for Linux</h4>
Turn on the Windows Subsystem for Linux feature before installing a Linux distribution. Open PowerShell as an administrator, and run the following command:
<pre>dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart</pre>
This may take a minute or two. Your output should resemble the following:
<pre>Deployment Image Servicing and Management tool
Version: 10.0.18362.1139

Image Version: 10.0.18363.1139

Enabling feature(s)
[==========================100.0%==========================]
The operation completed successfully.
</pre>
<h4 id="enable-virtual-machine"><a id="enable-virtual-machine"></a>Enable the Virtual Machine feature</h4>
WSL 2 requires the Virtual Machine Platform feature. In PowerShell, run the following command:
<pre>dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart</pre>
<h4 id="linux-kernel-update"><a id="linux-kernel-update"></a>Get the Linux kernel update</h4>
<a href="https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_x64.msi" target="_blank" rel="noopener noreferrer">Download the Linux kernel update package</a>, which is a regular Windows Installer (.msi) file.

Double-click the .msi file to install the WSL 2 update. If you’re prompted for elevated permissions, select <strong>Yes</strong> to approve the installation.
<h4 id="set-default"><a id="set-default"></a>Set the default WSL version</h4>
In PowerShell, run the following command to set WSL 2 as the default version for your Linux distributions:
<pre>wsl --set-default-version 2
</pre>
Your output should resemble the following:
<pre>For information on key differences with WSL 2 please visit https://aka.ms/wsl2</pre>
WSL 2 is ready to use. For more information on installing WSL 2, including troubleshooting, see <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10" target="_blank" rel="noopener noreferrer">Windows Subsystem for Linux Installation Guide for Windows 10</a>.
<h3 id="install-linux"><a id="install-linux"></a>Install your preferred Linux distribution</h3>
Install Linux from the Microsoft Store, the same way you install other applications on Windows.

Open the Microsoft Store app and search for “Linux.”

<img class="aligncenter wp-image-28884 " src="https://cdn.confluent.io/wp-content/uploads/linux.png" alt="Linux | Home: Ubuntu 20.04 LTS Installed" width="800" height="462" />

This blog post uses Ubuntu 20.04. Select <strong>Ubuntu 20.04 LTS</strong> and click <strong>Install</strong>.

When the installation is complete, click <strong>Launch</strong>. The shell opens and displays the following message:
<pre>Installing, this may take a few minutes...
Please create a default UNIX user account. The username does not need to match your Windows username.
For more information visit: https://aka.ms/wslusers
Enter new UNIX username:
</pre>
Enter a username and password to complete the installation.

<img class="aligncenter wp-image-28885 " src="https://cdn.confluent.io/wp-content/uploads/installing.png" alt="UNIX user account" width="800" height="647" />
<h3 id="install-java"><a id="install-java"></a>Install Java</h3>
Run the package manager to get the latest updates. In the Ubuntu shell window that opened above, run the following commands:
<pre>sudo apt-get update &amp;&amp; sudo apt-get upgrade -y</pre>
<table style="border-collapse: collapse; width: 100%; background-color: #e6f5fb; height: 10px; padding-top: 40px;" border="0">
<tbody>
<tr style="height: 30px;">
<td style="width: 6.32992%; text-align: center; height: 10px;">&#x2139;&#xfe0f;</td>
<td style="width: 93.6702%; font-weight: 300; padding: 10px; height: 10px; text-align: center;">Tip: Right-click pastes text into the terminal window.</td>
</tr>
</tbody>
</table>
<p style="padding-top: 20px; text-align: left;">Kafka requires the Java runtime to be version 8 or higher. Check the Java version in your Linux installation:</p>

<pre>java -version</pre>
Your output should resemble this:
<pre>openjdk version "1.8.0_265"
OpenJDK Runtime Environment (build 1.8.0_265-8u265-b01-0ubuntu2~20.04-b01)
OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode)
</pre>
If Java isn’t installed (likely) or it’s not the right version, install it by using your distribution’s package manager. There are a lot of ways to install Java. On Ubuntu, this is one of the simplest:
<pre>sudo apt install openjdk-8-jdk -y</pre>
<h3 id="download-kafka"><a id="download-kafka"></a>Download Kafka</h3>
You can install Kafka by using a package manager, or you can download the tarball and extract it to your local machine directly.

<a href="https://www.apache.org/dyn/closer.cgi?path=/kafka/2.6.0/kafka_2.13-2.6.0.tgz" target="_blank" rel="noopener noreferrer">Select a mirror from the Kafka download site</a> and download the tarball. The following command downloads Apache Kafka version 2.6:
<pre>wget https://ftp.wayne.edu/apache/kafka/2.6.0/kafka_2.13-2.6.0.tgz</pre>
Run the following commands to untar the Kafka archive, and <code>cd</code> to the <code>kafka</code> directory:
<pre>tar -xzf kafka_2.13-2.6.0.tgz
cd kafka_2.13-2.6.0
</pre>
Run the <code>ls -al</code> command to list the contents of the <code>kafka</code> directory:
<pre>total 64
drwxr-xr-x  7 jim jim  4096 Oct 14 12:27 ./
drwxr-xr-x 25 jim jim  4096 Nov 20 12:52 ../
-rw-r--r--  1 jim jim 29975 Jul 28 11:16 LICENSE
-rw-r--r--  1 jim jim   337 Jul 28 11:16 NOTICE
drwxr-xr-x  3 jim jim  4096 Jul 28 11:23 bin/
drwxr-xr-x  2 jim jim  4096 Jul 28 11:23 config/
drwxr-xr-x  2 jim jim  4096 Oct 14 12:26 libs/
drwxr-xr-x  2 jim jim  4096 Oct 14 12:28 logs/
drwxr-xr-x  2 jim jim  4096 Jul 28 11:23 site-docs/
</pre>
<h3 id="start-kafka-cluster"><a id="start-kafka-cluster"></a>Start the Kafka cluster</h3>
Run the following command to start ZooKeeper:
<pre>bin/zookeeper-server-start.sh config/zookeeper.properties
</pre>
There will be a lot of output, and ZooKeeper will be ready in a short time, typically around a second or two.

Open another terminal session. Change the directory to the <code>kafka</code> directory, and start the Kafka broker:
<pre>cd kafka_2.13-2.6.0
bin/kafka-server-start.sh config/server.properties
</pre>
If you arrange the windows to be side by side, your output should resemble the following screenshot:

<img class="aligncenter wp-image-28886 size-full" src="https://cdn.confluent.io/wp-content/uploads/output-windows.png" alt="Output | Windows" width="1000" height="638" /><span class="image-caption">ZooKeeper (left) and a Kafka broker (right) on Ubuntu 20.04 running on Windows and WSL 2</span>
<h3><a id="produce-and-consume"></a>Produce and consume some messages</h3>
Open another terminal session and run the <code>kafka-topics</code> command to create a Kafka topic named <code>quickstart-events</code>:
<pre>cd kafka_2.13-2.6.0
bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
</pre>
Your output should resemble this:
<pre>first event
second event
third event
</pre>
Arrange the producer and consumer terminal windows to be side by side. In the producer terminal, type a few more messages, and watch as they appear in the consumer terminal.

<img class=" wp-image-28888 aligncenter" src="https://cdn.confluent.io/wp-content/uploads/kafka-windows-produce-consume.gif" alt="Produce | Consume" width="893" height="503" />
<h3 id="stop-kafka"><a id="stop-kafka"></a>Stop Kafka</h3>
When you’re done experimenting with Kafka, follow these steps to exit the Kafka environment:
<ol>
 	<li>Stop the consumer and producer clients with Ctrl+C</li>
 	<li>Stop the Kafka broker with Ctrl+C</li>
 	<li>Stop the ZooKeeper server with Ctrl+C</li>
 	<li>Run the following command to clean up:
<code>rm -rf /tmp/kafka-logs /tmp/zookeeper</code></li>
</ol>
<h2 id="kafka-and-windows"><a id="kafka-and-windows"></a>Kafka and Windows: The good, the bad, and the ugly</h2>
There are lots of Kafka-on-Windows tutorials, but most make the mistake of running Kafka directly on the JVM on Windows. Superficially, this appears to work, but there are limitations: Kafka uses specific features of POSIX to achieve high performance, so emulations—which happen on WSL 1—are insufficient. For example, the broker will crash when it rolls a segment file. Always run Kafka on Windows in a Linux environment backed by WSL 2.

Another approach that works well is to run Kafka in Docker containers. <a href="https://docs.docker.com/docker-for-windows/install/" target="_blank" rel="noopener noreferrer">Docker Desktop for Windows</a> has been updated to use the WSL 2 back end, so Docker works exactly as it does on native Linux, without needing to spin up an entire VM. If you want to give this approach a go, <a href="https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html?utm_source=dev&utm_medium=blogpost&utm_campaign=tm.devx_ch.how-to-run-kafka-on-windows_content.apache-kafka" target="_blank" rel="noopener noreferrer">try it out using the Confluent Platform demo</a>.

<img class="aligncenter wp-image-28882" src="https://cdn.confluent.io/wp-content/uploads/ubuntu.png" alt="Ubuntu 20.04 LTS" width="821" height="399" /><span class="image-caption">Apache Kafka and Confluent Platform running in Docker containers on Ubuntu 20.04 on Windows and WSL 2</span>
<h2 id="getting-started"><a id="getting-started"></a>You’re just getting started!</h2>
Although Kafka provides an event streaming platform to build your applications on, you’ll want to take advantage of the broader ecosystem of components—like <a href="https://ksqldb.io/?utm_source=dev&utm_medium=blogpost&utm_campaign=tm.devx_ch.how-to-run-kafka-on-windows_content.apache-kafka" target="_blank" rel="noopener noreferrer">ksqlDB</a>, <a href="https://www.confluent.io/product/confluent-platform/data-compatibility?utm_source=dev&utm_medium=blogpost&utm_campaign=tm.devx_ch.how-to-run-kafka-on-windows_content.apache-kafka" target="_blank" rel="noopener noreferrer">Confluent Schema Registry</a>, and <a href="https://docs.confluent.io/platform/current/control-center/index.html?utm_source=dev&utm_medium=blogpost&utm_campaign=tm.devx_ch.how-to-run-kafka-on-windows_content.apache-kafka" target="_blank" rel="noopener noreferrer">Confluent Control Center</a>—all provided as part of <a href="https://docs.confluent.io/platform/current/platform.html?utm_source=dev&utm_medium=blogpost&utm_campaign=tm.devx_ch.how-to-run-kafka-on-windows_content.apache-kafka" target="_blank" rel="noopener noreferrer">Confluent Platform</a>. At the moment, Confluent Platform is supported for development only on Windows, not for production environments.

Now that you have Kafka installed, you’ll want to <a href="https://kafka-tutorials.confluent.io/?utm_source=dev&utm_medium=blogpost&utm_campaign=tm.devx_ch.how-to-run-kafka-on-windows_content.apache-kafka" target="_blank" rel="noopener noreferrer">try out some tutorials</a> and <a href="https://events.confluent.io/meetups?utm_source=dev&utm_medium=blogpost&utm_campaign=tm.devx_ch.how-to-run-kafka-on-windows_content.apache-kafka" target="_blank" rel="noopener noreferrer">join in the community</a>! Don’t forget that Apache Kafka has many APIs—including the producer and consumer but also <a href="https://docs.confluent.io/platform/current/streams/index.html?utm_source=dev&utm_medium=blogpost&utm_campaign=tm.devx_ch.how-to-run-kafka-on-windows_content.apache-kafka" target="_blank" rel="noopener noreferrer">Kafka Streams</a> and <a href="https://docs.confluent.io/platform/current/connect/index.html?utm_source=dev&utm_medium=blogpost&utm_campaign=tm.devx_ch.how-to-run-kafka-on-windows_content.apache-kafka" target="_blank" rel="noopener noreferrer">Kafka Connect</a>.

<b>To learn more, check out <a href="https://developer.confluent.io/?utm_source=dev&utm_medium=blogpost&utm_campaign=tm.devx_ch.how-to-run-kafka-on-windows_content.apache-kafka" target="_blank">Confluent Developer</a>.</b>

<hr />

<h2 id="kafka-on-windows"><a id="kafka-on-windows"></a><em>Kafka on Windows? What made this possible?</em></h2>
<em>You may recall a time when Linux was anathema to Microsoft. Back in 2001, Microsoft CEO Steve Ballmer famously called Linux a “malignant cancer,” <a href="https://www.zdnet.com/article/ballmer-i-may-have-called-linux-a-cancer-but-now-i-love-it/" target="_blank" rel="noopener noreferrer">but he has since come around</a> to “loving” it. Microsoft’s current CEO Satya Nadella seems intent on making it a first-class citizen in the Microsoft ecosystem, which means that a new era has arrived for software developers on the Windows platform.</em>

<em>When the Windows Subsystem for Linux (WSL 1) was released in 2016, it became possible to run a real Linux dev environment in a Linux shell, while retaining the familiar Windows UX around the shell. Even File Explorer was integrated nicely with the Linux file system.</em>

<em>The big drawbacks are that WSL 1 emulates a Linux kernel, and it runs in a full VM. The first means processes that require a native kernel, like Docker, can’t run. The second means that WSL 1 consumes a lot of resources. WSL 1 was not sufficient to run Kafka reliably.</em>

<em>But Microsoft delivered WSL 2 in 2019, and it’s a whole new world. They fixed the two biggest limitations, so WSL 2 runs a real Linux kernel, and the kernel runs on a subset of Hyper-V features, not in a full VM. For details, see <a href="https://docs.microsoft.com/en-us/windows/wsl/compare-versions" target="_blank" rel="noopener noreferrer">Comparing WSL 1 and WSL 2</a>. Now the path is clear for devs to build Kafka and ksqlDB apps on Windows.</em>