Coding in the Shadows: Hidden Gems of Lisp, Clojure, and Friends
Tags: lisp, clojure, vectordatabase
Published: 2023-11-28 19:58:41

In the world of programming, rankings are more than just numbers; they're a source of endless debate and pride. Much like how people are fascinated by top 10 lists in music or movies, there's a similar [ranking system for programming languages](https://www.tiobe.com/tiobe-index/). But unlike box office hits or Billboard charts, the criteria for ranking programming languages aren't as clear-cut.

There's a constant back-and-forth among programmers about which language is superior. You'll often hear them boast, "My language is better than yours," followed by demonstrations of elegant one-liners or efficient algorithms. But the truth is, we haven't settled on a universally accepted measure for ranking these languages. So, when it comes to these rankings, they're based primarily on popularity.

Now, this isn't about social media popularity – it's not a matter of likes or shares. It's about how many people are actively using these languages. Measuring this is tricky, as you can't simply count heads or survey every programmer out there. However, with a combination of job market analysis, community activity, and usage statistics in various projects, a somewhat accurate picture can be formed.

This approach to ranking may not satisfy everyone, especially since it doesn't necessarily reflect the technical merits or innovativeness of a language. Yet, it does offer a snapshot of the programming landscape, showing us which languages are currently in demand and shaping the world of technology.

The top spots in the programming language rankings are often occupied by names that even those outside the IT world might recognize. Languages like Java, Python, C, and JavaScript are household names in the tech community and beyond. When you delve into the top 20, the "least popular" among these is still used by approximately 1% of programmers worldwide – a small percentage, perhaps, but still significant when you consider the sheer number of people coding across the globe.

However, the list of programming languages doesn't stop at the top 20. Venture further down to the top 50, and you'll start to encounter some truly niche languages. Among them is Logo, a language developed in 1967 with the aim of teaching programming concepts. It's famously associated with the **'turtle graphics,' where you can control a turtle's movement on-screen through coding commands**. It’s intriguing to ponder what modern applications it could have, given that it’s still used, albeit by a mere 0.1% of programmers.

Also nestled in this segment of the list is COBOL – an even older language that first appeared 64 years ago. Despite its age, COBOL remains a vital cog in the machinery of large insurance companies and banks. These institutions rely on massive mainframe computers running on millions of lines of COBOL code. It’s a language that, while seemingly antiquated, underpins systems too critical and complex to risk rewriting. **The mere thought of transitioning away from COBOL raises concerns about potentially catastrophic system failures**.

## Lisp
Venturing even further down the list, you encounter two languages that are my personal heroes. One such language is Lisp, a language that has its roots in academia and theoretical computation. Fascinatingly, Lisp was initially conceived on paper, without any practical implementation on a computer. It's a testament to the pure, almost philosophical approach to programming language design.

Lisp is built on a handful of core concepts and a mere eight operations. Yet, from this simplicity springs an immense power. Its creator, John McCarthy, mathematically demonstrated that Lisp was a fully capable programming language, earning it the classification of being "Turing-complete." This term refers to a system of computation that, given enough resources, can solve any problem that a Turing machine (a basic model of a computer) can. In this sense, Lisp is akin to geometry as formulated by Euclid – based on just a few axioms, it unfolds into a vast and intricate world of possibilities.

The elegance of Lisp lies in its minimalism and its flexibility. It's a language that encourages a different way of thinking, one that is more about the essence of computation and less about the specifics of syntax. This philosophical underpinning makes it not just a tool for writing software, but also a medium for exploring the very nature of programming itself. For those who delve into its depths, Lisp offers a unique perspective, revealing the elegant complexity that can arise from the simplest of foundations.

Lisp's role in the evolution of computer science extends beyond its elegant design. It was **the first programming language widely used for research in Artificial Intelligence (AI)** during the 1960s and 1970s. At that time, Lisp was at the forefront of this groundbreaking field. This deep association with AI led to the creation of specialized hardware known as Lisp machines. These were computers designed specifically to run Lisp efficiently, representing a unique convergence of software and hardware dedicated entirely to one programming language. Unlike modern computers, which are built to run a wide range of software, Lisp machines were optimized for the nuances and particularities of Lisp.

The phrase "it was Lisp all the way down" aptly captures this era. In these machines, Lisp wasn't just a programming language running on general-purpose hardware; it was an integral part of the entire system. This period in computing history highlights Lisp's significant influence and the fervent belief in its potential to unlock the secrets of artificial intelligence and advanced computation.

As the years progressed, however, Lisp's prominence in the programming world began to wane for reasons that are as complex as they are varied. This shift in the landscape saw the rise and fall of various programming languages, each vying for a spot in the rapidly evolving tech industry.

This was the time when I came across an article by Paul Graham called [Beating the Averages](http://paulgraham.com/avg.html). Graham described Lisp as a secret weapon that wasn’t for everyone - it was special, meant for a select group of programmers. Reading this, I felt a strong connection; I knew I was part of that group. Lisp just clicked with me.

This led me to dive deeper into Graham's other writings about Lisp. The more I read, the more natural it felt to use Lisp. It was as if my mind was perfectly suited for it. This idea reminded me of a quote from Eric Raymond: 

> "Lisp is worth learning for the profound enlightenment experience you will have when you finally get it; that experience will make you a better programmer for the rest of your days, even if you never actually use Lisp itself a lot." 

During this time, I was playing around with Lisp, immersing myself in its unique style. That's when Clojure appeared, like a fresh wave for those of us who loved Lisp. It felt like a modern version of Lisp, mixing old Lisp ideas with new ways of programming. Clojure made it easier to use Lisp with today's technology, first with the Java Virtual Machine (JVM), then with the browser. For a while, it looked like Clojure would make Lisp popular again. It even became one of the top 20 programming languages for a bit.

Yet, despite this promising start, Clojure didn't maintain its momentum. It eventually settled into a niche, maintaining a presence in a few actively developed applications but largely receding from the mainstream programming scene. For me, personally, Clojure has been more than just a programming language. It has profoundly shaped my thinking, offering a unique perspective on problem-solving, system design, and the elegance of code.

Doesn't Clojure deserve its own chapter? It certainly does. You can read more about why it lies so close to my heart: [Why My Brain Is Wired for Clojure](https://hackernoon.com/why-my-brain-is-wired-for-clojure)

## APL
I promised to mention two programming languages that hold a special place for me; Lisp is the first, and APL is the other. APL, which stands for "A Programming Language," began its life not even as a language for programming, but as a notation system used by people at IBM. It was developed as a way to clearly communicate complex programming and mathematical ideas, extending beyond traditional mathematical notation.

When you first see APL code, two things will likely catch your eye. Firstly, it looks almost like a secret code from a pirate's treasure map, filled with unique symbols like backward-crossed circles, triangles with extra lines, and an array of other special characters. At the height of APL's popularity, these symbols were so integral to the language that **you could even find special keyboards made just for typing in APL**.

This mysterious and almost romantic quality of APL's appearance was one of the things that drew me to it. But there's another aspect of APL that's immediately noticeable: its conciseness. APL is renowned for its ability to condense what would be ten lines of code in other languages into a single line of APL.

What exactly makes APL so concise? At first glance, it might seem like a simple trick. Other programming languages are filled with long keywords and functions like `continue` and `length`, while APL appears to just replace these with a single character. But there's much more to it than that.

Yes, it's true that APL often uses single characters, like the Greek letter rho (ρ), to perform tasks that would require more elaborate expressions in other languages. Rho, for example, is used to determine the length of a string or a vector. But the real power of APL lies in its fundamental approach to data. APL treats nearly everything as a vector, or a list of items, and it's designed to operate on these vectors efficiently.

Take a simple operation like adding 2 to a vector. In strict mathematical terms, this might not make sense, but APL interprets it as adding 2 to each element of the vector. This characteristic allows you to perform complex operations without the need for writing loops or conditions, which are common and space-consuming in other languages. By thinking in terms of operations on whole vectors or arrays at once, **APL enables a level of conciseness and elegance that is hard to match in more verbose programming languages**.

Recognizing the limitations imposed by the need for a specialized keyboard, the creators of APL took an innovative step forward. They developed a new language, J, which was designed to be used with the standard ASCII characters available on every keyboard. This move addressed one of APL's biggest accessibility issues, making the language more approachable for a broader audience.

J's creation marked the beginning of an interesting progression in programming language development. It inspired the creation of K, a language that further refined the ideas of APL and J, focusing on efficiency and speed. K, in turn, led to the development of Q, a language built on the foundations of K but geared more towards database and query operations.

Q might not be a widely known programming language, but it has a very important role in one specific area: handling financial transactions. This is because Q is really good at processing large amounts of data quickly, which is crucial in finance where every second counts. The demand for Q programmers may not be widespread across the tech industry, but within the finance sector, their skills are highly sought after. If you know how to program in Q, you can actually earn a lot of money.

## K
This part is going to be more technical. For programmers familiar with the concepts of functional programming, K offers a powerful yet concise way to perform operations like summation. For example, the expression `+/!8` in K efficiently sums up the numbers up to 8. Here, `!8` generates a list of numbers from 0 to 7. Then, `+/` acts as what's known in functional programming as a "reduce" operation, applying the addition function across the list. Another example is calculating factorial: `*/1+!8`. We generate a list of the number from 0 to 7; then add 1 to add number: `1+!8` -- we already know that adding 1 to a list adds 1 to each element. Finally, we reduce the list applying multiplication.

One more example. Here is how to find the largest number in a list: `|/'4 2 7 1`. Let's break it down how it works.  The phrase `|/'` is a combination of two symbols:
- `|`: This is the 'maximum' function in K. When used on its own, it gives you the maximum of two numbers.
- `/'`: This is known as an 'over' operator. It is used to apply a function repeatedly between the elements of a list.

Together, `|/'` applies the 'maximum' function across the list. It repeatedly compares pairs of numbers and carries forward the larger number each time. This process continues across the entire list until the largest number is determined.

If programmers have heard of K, it's usually in connection with code golf. [Code](https://codegolf.stackexchange.com) [golf](https://code.golf) is a unique and playful programming challenge where the objective is to solve a specific problem or complete a task using the least amount of code possible. Unlike typical programming practices where code is often measured in lines, **code golf counts every single character**, making each one count. This form of programming competition emphasizes brevity and resourcefulness, encouraging programmers, often referred to as 'code golfers,' to think outside the box. They must employ ingenious methods and deep knowledge of their chosen language's syntax and features to craft a solution that is as short as possible. 

K has gained a reputation as one of the most successful languages for code golf. Its concise syntax and powerful built-in functions allow programmers to express complex operations in just a few characters, making it ideal for these challenges. However, in the world of code golf, some languages have been created solely to excel in this arena. These languages, often referred to as 'esoteric' or 'golfing' languages, might include built-in functions for common code golf tasks, like a one-character function to generate prime numbers - a frequent requirement in code golf challenges.

While the one-liners showcase K's ability to succinctly express complex ideas, it has a serious consequence beyond winning at code golf. Not only K applications are compact in size, but the K interpreter itself is extremely small, especially when compared to similar programs in other languages. This minimal footprint allows K applications, along with the K interpreter, to fit entirely within the L2 cache of a CPU. The L2 cache is a smaller, faster memory compared to a computer's main RAM, and because it's closer to the CPU, data stored in the L2 cache can be accessed much more quickly. This proximity significantly enhances the execution speed of K applications.

This compactness and efficient use of the L2 cache set K apart from many other programming languages. In most languages, especially those that produce larger, more memory-intensive applications, critical parts of the program and the interpreter or runtime environment are less likely to fit entirely in this fast-access cache. As a result, K applications, despite being interpreted, can often **achieve execution speeds comparable to compiled C code**, which is renowned for its performance. The efficiency of K doesn’t just stem from its concise syntax but also from how it harmonizes with the underlying hardware, making the most of the CPU's capabilities to deliver high performance. The speed and efficiency of K make it perfect for working with huge amounts of data in real-time, like in stock trading.

If you want to try out K, there are some open source implementations, like [John Earnest's oK](https://github.com/JohnEarnest/ok) which has a REPL and a calculator-like interface for mobile phones with a charting feature.