CrateDB
CrateDB is a distributed SQL database management system that integrates a fully searchable document-oriented data store. It is open-source, written in Java, based on a shared-nothing architecture, and designed for high scalability. CrateDB includes components from Trino, Lucene, Elasticsearch and Netty. History. The CrateDB project was started by Christian Lutz, Bernd Dorn, and Jodok Batlogg, an open source contributor and creator who has contributed to the Open Source Initiative Vorarlberg while at Lovely Systems in Dornbirn. The software is an open source, clustered database used for fast text search and analytics. The company, now called Crate.io , raised its first round of financing in April 2014. In June that year, Crate.io won the judge's choice award at the GigaOm Structure Launchpad competition. In October, Crate.io won the TechCrunch Disrupt Europe in London. Crate.io closed a $4M founding round in March 2016. In December, CrateDB 1.0 was released having more than one million downloads. CrateDB 2.0, the first Enterprise Edition of CrateDB, was released in May 2017 after a $2.5M round from Dawn Capital, Draper Esprit, Speedinvest, and Sunstone Capital. In June 2021 Crate.io announced another $10M funding round. Since September  2020, Crate.io is led by Eva Schönleitner as the CEO. Overview. Architecture CrateDB operates in a shared-nothing architecture as a cluster of identically configured servers (nodes). The nodes coordinate to automatically distribute the execution of both write and query operations across the cluster. Querying CrateDB's SQL syntax includes JOINs, aggregations, indexes, sub-queries, user-defined functions, and views. It also supports full-text search, geospatial queries, and nested JSON object columns. For query distribution, CrateDB implements memory-resident columnar field caches on each shard. The caches tell the query engine whether there are rows on that shard that  meet the query criteria, and where the rows are located. This is performed automatically. Schemas CrateDB supports “strict”, “dynamic”, or “ignored” schemas: Consistency CrateDB implements an eventually consistent, non-blocking data insertion model. It includes record versioning, optimistic concurrency control, and a table-level refresh frequency setting, which forces CrateDB data to become consistent every "n" milliseconds. CrateDB supports read-after-write consistency: the queries retrieving a specific row by its primary key always receive the most recent row. All the other queries (search operations) return eventually-consistent data. Search operations are performed on shared IndexReaders, which provide caching and  reverse lookup capabilities for shards. An IndexReader is always bound to the Lucene segment from which it was started, meaning it has to be refreshed in order to see new changes. Therefore, a search only sees a change if the associated IndexReader was refreshed after that change occurred. By default, this is done once per second, but it can be reconfigured to occur more or less frequently. Every replica shard is updated synchronously with its primary, and always carries the same information. Therefore, in terms of consistency, it does not matter if the primary or a replica shard is accessed . In CrateDB, only the refresh of the IndexReader affects consistency. Atomicity and durability CrateDB implements WAL (write-ahead logging):