Five Apache projects you probably didn't know about
Tags: apachefoundation, opensource, china
Published: 2023-12-21 09:02:00

In early 2021, I started to work on the [Apache APISIX](https://apisix.apache.org/) project. I have to admit that I had never heard about it before. In this post, I'd like to introduce some Apache projects that are less well-known than HTTPD or Kafka.

## Apache APISIX

![Apache APISIX logo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jjmk0f4xrai675vipcm6.png)

[APISIX](https://apisix.apache.org/) is an [API Gateway](https://en.wikipedia.org/wiki/API_management). It builds upon [OpenResty](https://openresty.org/en/), a Lua layer built on top of the famous [nginx](https://nginx.org/) reverse-proxy. APISIX adds abstractions to the mix, _e.g._, `Route`, `Service`, `Upstream`, and offers a plugin-based architecture.

Lots of plugins are provided out of the box:

* Transformation: `response-rewrite`, `proxy-rewrite`, gRPC, `body-transformer`, etc.
* Authentication: JWT, OPA, Keycloak, OpenID Connect, etc.
* Observability: metrics, logging, and traces
* Traffic: rate limiting, request validation, canary release, etc.
* Serverless: Azure functions, AWS Lambdas, OpenWhisk, etc.
* Messaging: Kafka, Dubbo, and MQTT
* Pre- and post-processing

If no plugin fits your requirements, writing your own is possible.

You can leverage APISIX on Kubernetes as an Ingress Controller. APISIX provides a Helm Chart for this.

![Apache APISIX architecture diagram](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t4ksg1xsff7picujkwep.png)

## Apache ShardingSphere

![Apache ShardingSphere logo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6fyl2umcxtnxfp0sxyuy.png)

[ShardingSphere](https://shardingsphere.apache.org/) claims to offer an ecosystem able to transform any database into a distributed database system. It acts as a proxy between your code and your database(s). It comes in two flavors:

* ShardingSphere-JDBC: a JDBC driver that acts as a proxy to your database(s). It's only available for JVM-based applications.
* ShardingSphere-Proxy: a technology-independent deployable component.

ShardingSphere offers several core features:

* Data Sharding is the core feature, as the project's name implies. Most use cases focus on scaling purposes, but there are others, _e.g._, [data residency](https://blog.frankel.ch/data-residency/) requirements.
* XA transactions for distributed transactions
* Read/write splitting
* Data encryption
* etc.

## Apache SeaTunnel

![Apache SeaTunnel logo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/it39dpyoy1fibwahdgl2.png)

[Apache SeaTunnel](https://seatunnel.apache.org/) is a data integration platform that offers the three pillars of data pipelines: sources, transforms, and sinks. It offers an abstract API over three possible engines: the Zeta engine from SeaTunnel or a wrapper around [Apache Spark](https://spark.apache.org/) or [Apache Flink](https://flink.apache.org/). Be careful, as each engine comes with its own set of features.

The power of SeaTunnel comes from its rich connector ecosystem. It does provide traditional SQL connectors, _e.g._, Oracle, PostgreSQL, and MySQL, and NoSQL ones, _e.g._, MongoDB, Cassandra, and Elasticsearch. However, it also comes bundled with some original ones, including Jira, Google Sheets, and Notion. I have a particular fondness for the <abbr title="Change Data Capture">CDC</abbr> connector sources over MongoDB, MySQL, and Microsoft SQL Server.

![Apache Seatunnel concepts](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j9kwdgmos63k48y915by.png)

SeaTunnel comes with a web UI, which provides visual management of jobs, scheduling, running, and monitoring capabilities.

## Apache SkyWalking

![Apache Skywalking logo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ifzcfzk3ovy8yxv2dyjs.png)

[Apache SkyWalking](https://skywalking.apache.org/) is an <abbr title="Application Performance Monitor">APM</abbr> tool, focusing on microservices, Cloud Native apps, and Kuernetes architectures. It builds its architecture on four kinds of components:

* **Probes** collect telemetry data (metrics, logs, traces, and events). They support multiple output formats, including OpenTelemetry.
* The **platform** aggregates and processes data
* The **storage** offers an interface over a supported backend. Supported backends include ElasticSearch, H2, MySQL, TiDB, and BanyanDB, a custom storage engine developed for SkyWalking
* Finally, a web **UI** allows visualizing SkyWalking's data

![Apache SkyWalking dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i8hjahq3jt446n2pdruc.png)

Skywalking supports a couple of formats, including OpenTelemetry. Given the industry's current focus on OpenTelemetry, I recommend seriously considering this option.

## Apache Doris

![Apache Doris logo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s8fmreqessof8vktlxz3.png)

[Apache Doris](https://doris.apache.org/) is a real-time data warehouse.

Doris promotes four primary scenarios:

* Reporting analysis
* Ad-Hoc query
* Unified data warehouse construction
* Data lake query

![Apache Doris concepts](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0cr5oe6lv1rk9wjtn0y3.png)

Doris is mostly MySQL compliant so that you can use a regular MySQL client.

## Discussion

The Apache Foundation hosts the projects above, but they have another thing in common: they were all incepted in China. Have a look at the [Apache project list](https://projects.apache.org/projects.html). You'll probably be amazed at the sheer number; it's close to 300!

In recent years, the number of projects incepted at the Apache Foundation has increased drastically. Look again at the list; I'm sure you only know a few of them - lots come from China. The trend is only growing; it's a great move to integrate China with the OpenSource world!

Just as I finish this post, my friend Stefano Fago has posted on another relevant project, [Apache Paimon](https://paimon.apache.org/), a streaming data lake platform.

**To go further:**

* [Apache APISIX](https://apisix.apache.org/)
* [Apache ShardingSphere](https://shardingsphere.apache.org/)
* [Apache SeaTunnel](https://seatunnel.apache.org/)
* [Apache SkyWalking](https://skywalking.apache.org/)
* [Apache Doris](https://doris.apache.org/)

_Originally published at [A Java Geek](https://blog.frankel.ch/five-apache-projects/) on December 17<sup>th</sup>, 2023_