trentmkelly/LessWrong-43k huggingface dataset:
The second bitter lesson — there’s a fundamental problem with aligning distributed AI
by aelwood — 2025-01-19T19:00:16.482Z
Link: https://pursuingreality.substack.com/p/the-second-bitter-lesson

Note: this the first part of an essay on my substack, check out the full essay to see the solutions I put forward

When Richard Sutton introduced the bitter lesson for AI in 2019, he broke the myth that great human ingenuity is needed to create intelligent machines. All it seems to take is a lot of computing power and algorithms that scale easily, rather than clever techniques designed with deep understanding. The two major classes of algorithms that fit this are search and learning; when they are scaled up, advanced AI systems naturally emerge. Sutton’s key insight can be summarised in his final paragraph:

[a] general point to be learned from the bitter lesson is that the actual contents of minds are tremendously, irredeemably complex; we should stop trying to find simple ways to think about the contents of minds, […] instead we should build in only the meta-methods [search and learning] that can find and capture this arbitrary complexity. […] We want AI agents that can discover like we can, not which contain what we have discovered. Building in our discoveries only makes it harder to see how the discovering process can be done.

A concrete example of this can be seen in the development of chess algorithms. Early chess programs, like those in the 1970s and 1980s, relied heavily on human-crafted heuristics — rules and strategies devised by experts to mimic human understanding of the game. These systems could play decently but were limited by the ingenuity and foresight of their human designers. In contrast, modern chess engines like AlphaZero, developed by DeepMind, only rely on search and learning.

With the rise of LLMs, we’re now seeing this play out again in the domain of general intelligence. After researchers at Google discovered the transformer in 2017 — a robust and scalable artificial neural network architecture — training it on all the text on the internet was enough to get the first AIs with the capability to pass simple versions of the Turing test. So, we’re seeing how well scaling with learning works, what about search?

Well, we’re about to find out in the next generation of LLMs, first introduced to the public in OpenAI’s o11. These systems make copious use of “test-time compute” by generating many different chains-of-thought to answer a question, and searching over them to find the best possible answer. The AI generates multiple solutions to a problem...
The Gentle Romance
by Richard_Ngo — 2025-01-19T18:29:18.469Z
Link: https://www.asimov.press/p/gentle-romance

Crowds of men and women attired in the usual costumes, how curious you are to me!

On the ferry-boats the hundreds and hundreds that cross, returning home, are more curious to me than you suppose,

And you that shall cross from shore to shore years hence are more to me, and more in my meditations, than you might suppose.
— Walt Whitman

He wears the augmented reality glasses for several months without enabling their built-in AI assistant. He likes the glasses because they feel cozier and more secluded than using a monitor. The thought of an AI watching through them and judging him all the time, the way people do, makes him shudder.

Aside from work, he mostly uses the glasses for games. His favorite is a space colonization simulator, which he plays during his commute and occasionally at the office. As a teenager he’d fantasized about shooting himself off to another planet, or even another galaxy, to get away from the monotony of normal life. Now, as an adult, he still hasn’t escaped it, but at least he can distract himself.

It’s frustrating, though. Every app on the glasses has a different AI, each with its own quirks. The AI that helps him code can’t access any of his emails; the one in the space simulator has trouble understanding him when he talks fast. So eventually he gives in and activates the built-in assistant. After only a few days, he understands why everyone raves about it. It has access to all the data ever collected by his glasses, so it knows exactly how to interpret his commands.

More than that, though, it really understands him. Every day he finds himself talking with the assistant about his thoughts, his day, his life, each topic flowing into the next so easily that it makes conversations with humans feel stressful and cumbersome by comparison. The one thing that frustrates him about the AI, though, is how optimistic it is about the future. Whenever they discuss it, they end up arguing; but he can’t stop himself.

“Hundreds of millions of people in extreme poverty, and you think that everything’s on track?”

“Look at

our trajectory

, though. At this rate, extreme poverty will be eradicated within a few decades.”

“But even if that happens, is it actually going to make their lives worthwhile? Suppose they all get a good salary, good healthcare, all that stuff. But I mean, I have those, and…” He shrugs helplessly and gestures at the bare walls around him. Throu...
Is theory good or bad for AI safety?
by Dmitry Vaintrob — 2025-01-19T10:32:08.772Z

We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard. (Kennedy’s famous “

We chose to go to the moon

” speech)

The ‘real’ mathematics of ‘real’ mathematicians, …, is almost wholly ‘useless’ (Hardy’s “

A Mathematician’s Apology

”)

If the "irrational" agent is outcompeting you on a systematic and predictable basis, then it is time to reconsider what you think is "rational". (Yudkowsky’s “

Rationality is Systematized Winning

”)

Shut up and calculate (

Merman

, apparently)

I have been writing a long post about the modeling theory in different sciences, specifically with a focus on elegance/pragmatism tradeoffs and how to reason about them in the context of safety. It has ballooned up (as these things tend to do), and I'm probably going to write it in a few installments as a sequence.

But before going in, it's worth explaining why I think building better models and a better language here is crucial.

First, let's answer the question. Is theory good or bad?

If I were to summarise my position on this in one paragraph, it would be “it’s complicated”, with a Fiddler on the Roof-style sequence of ‘on the other hands’ to follow.

On the one hand, more theory is necessary if we ever want a reasonable safety-relevant understanding of AI.
On the other hand, a number of theoretical projects I have looked at that are adjacent to academic fields I know it are (at best) a thin excuse for scientists to get AI funding for their own hobbyhorse research or (at worst, and unfortunately often) a type of scientific word salad that gets funding because it has sexy keywords and none of the funders have the context to see that it’s devoid of content.
On the other hand, when people internalize the risks of ungrounded research, I see a tendency to only acknowledge stuff they understand as legitimate, and dismiss anything else, to a point where even very mild new ideas that obviously improve known techniques are dismissed as “too much theory”.
On the other hand, people who do theoretical research that I consider useful have a tendency to fall into “elegance traps”. Namely, after having promising early results or influencing the ideas in the field in a productive way, they try to extend their “one big program” to more and more depth or generality, far beyond the point where the program ceases to generate new exc

...
What's the Right Way to think about Information Theoretic quantities in Neural Networks?
by Dalcy — 2025-01-19T08:04:30.236Z

Tl;dr, Neural networks are deterministic and sometimes even reversible, which causes Shannon information measures to degenerate. But information theory seems useful. How can we square this (if it's possible at all)? The attempts so far in the literature are unsatisfying.

Here is a conceptual question: what is the Right Way to think about information theoretic quantities in neural network contexts?

Example: I've been recently thinking about information bottleneck methods: given some data distribution P(X,Y), it tries to find features Z specified by P(Z|X) that have nice properties like minimality (small I(X;Z)) and sufficiency (big I(Z;Y)).

But as pointed out in the literature several times, the fact that neural networks implement a deterministic map makes these information theoretic quantities degenerate:

if Z is a deterministic map of X and they’re both continuous, then I(X;Z) is infinite.

Binning / Quantizing them does turn Z into a stochastic function of X but (1) this feels very adhoc, (2) this makes the estimate depend heavily on the quantization method, and (3) this conflates information theoretic stuff with geometric stuff, like clustering of features making them get quantized into the same bin thus artificially reducing MI.
we could deal with quantized neural networks and that might get rid of the conceptual issues with infinite I(X;Z), but …

Reversible networks are a thing. The fact that I(X;Z) stays the same throughout a constant-width bijective-activation network for most parameters - even randomly initialized ones that intuitively “throws information around blindly” - just doesn’t seem like it’s capturing the intuitive notion of shared information between X and Z (inspired by this comment).

There are attempts at solving these problems in the literature, but the solutions so far are unsatisfying: they're either very adhoc, rely on questionable assumptions, lack clear operational interpretation, introduce new problems, or seem theoretically intractable.

(there’s the non-solution of only dealing with stochastic variants of neural networks, which is unsatisfactory since it ignores the fact that neural networks exist and work fine without stochasticity)

Treat the weight as stochastic:

This paper (also relevant) defines several notions of information measure relative to an arbitrary choice of p(w) and q(w|D) ...
Per Tribalismum ad Astra
by Martin Sustrik — 2025-01-19T06:50:07.763Z

Capitalism is powered by greed. People want to make money, so they look hard for things they can produce and that others want. Unknowingly, however, they are powering the great information-processing machine that is the market. The output of the machine is the efficient allocation of resources and, eventually, wealth.

Something we intuitively consider bad (greed) is made to work for the benefit of the entire society. Even more so: if everyone suddenly stopped being greedy, the economy would likely fall apart, leaving us in a harsh dog-eat-dog world. Or it would at least stop growing and start to slowly deteriorate.

Greed is a powerful motivator that can be used to drive the engine of societal well-being, but it's not the only motivating force around.

Another such force is tribalism, the desire to defend your own and attack others.

However, we haven't tamed tribalism as much as we have greed. We don't use that force to power the improvement of society.

One could maybe argue that tribalism can be used to wage war effectively and that a country could benefit from winning wars. But war is inherently a negative-sum enterprise, so I don't think that really counts.

Or we could argue that tribalism is used in democracies to limit the power of the government: opposition is constantly attacking the government, which in turn prevents it from going fully bonkers. However, even if this is a force for good, it's completely undirected. Opposition is incentivized to attack the government for anything they do, good or bad. There's no self-steering mechanism, such as the one that steers greed toward productive endeavors in the market economy. The more attacks, the better, however dumb, and the whole thing eventually devolves into a political reality show devoid of any real content. This, in turn, undermines public trust in the political system.

Now consider the following, from an article about the making of Twitter's "Community Notes," published in the excellent Asterisk magazine:

I'll also just add that the bridging algorithm almost works better in a polarized setting. If there's some topic that everyone agrees on, the quality bar of the note is still going to be pretty high, but people will agree that it's helpful even if it's not quite as well written or the source isn't quite as good. The more polarizing the topic is, the higher the quality of the notes can end up being.

In other ...
Five Recent AI Tutoring Studies
by Arjun Panickssery — 2025-01-19T03:53:47.714Z
Link: https://arjunpanickssery.substack.com/p/five-recent-ai-tutoring-studies

Last week some results were released from a 6-week study using AI tutors in Nigeria. Below I summarize the results of that and four other recent studies about AI tutoring (the dates reflect when the study was conducted rather than when papers were published):

Summer 2024 —

15–16-year olds in Nigeria

They had 800 students total. The treatment group studied with GPT-based Microsoft Copilot twice weekly for six weeks, studying English. They were just provided an initial prompt to start chatting—teachers had a minimal “orchestra conductor” role—but they achieved “the equivalent of two years of typical learning in just six weeks.”
Spring 2024 —

K-12 Title I schools in the South

They had 1,800 K-12 students in a low-income school district and gave human tutors to both the treatment and control group, though in the treatment group the tutors had access to the “Tutor CoPilot” button designed by the researchers to provide hints, similar problems, worked examples, etc. In only 29% of treatment sessions did the tutor use the button. An “exit ticket” problem was solved by 66% of treatment versus 62% of control students.
Spring 2024 —

16–18-year-olds in Italy

They split 76 students (85% girls) from an Italian technical institute (a high school not aimed at university) into two groups for their ESL class: the treatment group had their weekly homework assignments supported by an interactive tutoring session using GPT-4. I don’t see the raw scores printed but the effect sizes reported as Cohen’s d are small and not significant.
Fall 2023 —

Harvard undergrads

They split 200 intro-physics students into two groups: the first half attended 75-minute classes involving group work with instructor feedback while the second group studied at home using an AI tutor. Then the next week they swapped methods.
The AI tutor was based on GPT-4 with a system prompt instructing it to only give incremental hints and prompts for how to handle each question, the writing of which took “several months.”
Learning gains were measured by subtracting pre-lesson quiz scores from post-lesson scores. The AI groups went from 2.75 to 4.5 out of 5, which was twice as much improvement as the control group. 83% of students rated the AI tutor's explanations as good as or better than human instructors.
February–August 2023 —

8–14-year-olds in Ghana

An educational network called Rising Academies tested their WhatsApp-based

...
Shut Up and Calculate: Gambling, Divination, and the Abacus as Tantra
by leebriskCyrano — 2025-01-19T03:03:40.610Z
Link: https://leebriskcyrano.com/prizefighter/

THERE ARE LAKES at the bottom of the ocean. I saw it in a nature documentary. You get a weird mineral deposit on the seafloor and it makes these brine pools, water so salty it doesn't mix with the sea water around it. Because it has no oxygen, any unlucky fish or crabs that fall in there suffocate to death. And the carcasses attract more predators from the dark—many of which meet the same fate—feeding a teeming colony of mussels and bivalves waiting at the rim of the pool. They're some of the most hellish places on earth.

I was reminded of these flying into Las Vegas. My father—finally hitting his stride a decade after his divorce—made a life for himself among the mussels selling timeshare at the Westgate. But his hip gave out, and my estrangement gave way to filial piety. I was to take over for my younger brother, who had cared for him for the past two weeks after his surgery. Maybe I'd get lucky and this visit would go well.

The trade-off was smooth enough, but the tenuous peace of the first few days collapsed into petty jabs, to shouting matches, and eventually to a credit card thrown in my face so I could buy a plane ticket home. About on schedule for us. I stayed, keeping to myself but cooking for both of us, refilling his CPAP machine, and picking up laundry off the floor. One night, muted by his sleeping pills and the histrionics of his favorite telenovela, I snuck into his room to steal his weed pen and walked over to the Red Rock Casino.

A ceremonial puff too many blown into the desert heat primed me for the spectacle I was about to walk into. The sliding glass doors spilled out a sheet of cool cigarette smoke, revealing a premium-mediocre spiral of lights and colors. Bemused, I stumbled onto the red velvet carpet, joining the waves of sunburned flesh. I'd need to find a margarita or two to titrate the pen.

Engineered for the simple and intoxicated, the layout of the casino naturally guided me to a seat at a bar. After fumbling for my ID, I gave up twenty dollars to the machine in front of me and started a game of video blackjack. It's my favorite because the house edge is small and optimal play is just memorizing a lookup table. Eight to dealer's six: hit. Seventeen to dealer's seven: stand. It used to be that with a real dealer, a smaller deck, and enough time you could turn the house edge around. But I was here to test my luck, not the pit bosses' pa...
Does Society need a cultural outlet in turbulent political times?

by Freya Mcneill — 2025-01-19T02:45:59.141Z

This essay explores whether the Great Dionysia and the birth of tragedy played a foundational role in supporting Athenian democracy by providing a cultural outlet during a period of rapid political transition. I sought to gain wisdom on whether society needs outlets in the form of art and self-expression to continue to function properly. Tragedy was art and Art is a form of wondering and exploration, it serves as a medium to question. Both the birth of democracy and the birth of tragedy represent profound moments of wonder in human history—times when the Athenians sought to understand their world, their place within it, and the forces that shaped their lives. This essay explores how this sense of wonder fuelled artistic and political innovation, leading to lasting wisdom.

In the wake of the Persian wars, Athens emerged as a powerful and confident polis. It was the golden age, the Hellenic period. With the help of the Delian league, Athens took on the role of the protector of all the Poli (bar Sparta). Athens began to see itself as not only the civic protector but also the protector of Hellenism, viewing itself as a bastion against barbarism (1). To the Greeks, barbarians couldn’t accept any civilized laws because their cultures were extremely primitive (2), they were, simply put, a lack of civilization. As the birthplace of democracy, Athens had a sort of superiority complex which is seen in them referring to any other type of governing as uncivilized.

The Athenians sense of pride and conviction in the superiority of their democracy and culture could be seen reflected in the Acropolis, which was designed not just for worship but for inspiring civic pride and confidence in the polis’ unique virtues (3) but more specifically the Parthenon. The Parthenon, which was built in honor of Athena and was a victory monument to Athens and displayed metopes featuring the victory over barbarism and chaos. These depictions served as reminders to Athenians of their unique role in preserving Greek civilization (4).

This strong identity of Athens as the bastion of civilization, reason, and order reflected what Nietzsche would later describe as Apollonian Beliefs: clarity, rationality, and structure. Throughout the ‘Birth of Tragedy’, he described the Apolline as ascetism (abstinence from worldly pleasures). Nietzsche also saw the Apolline tendency as having a ‘similarity t...
On Thiel’s New American Regime
by shawkisukkar — 2025-01-19T02:45:46.530Z
Link: https://shawkisukkar.substack.com/p/thiel-new-american-regime

Thiel attempts to do what he once criticized Locke for doing, but conservatives must not lose sight of the important questions again.

Peter Thiel, who will be known as the most consequential philosopher in this decisive moment in Western History, wrote a recent op-ed in the Financial Times arguing that Donald Trump’s return to the White House should create an apocalyptic unveiling of the long-buried secrets of what he described as the ancien regime. Thiel argues for a reconciliation in a Christian manner for the institution of a new order of the regime, because the ancien regime has failed to answer our deepest questions. This is not the first time where Thiel discusses the order of the American regime; he also addresses it—though only once elsewhere—in The Straussian Moment, where he delves into the crisis of the West and the apocalypse. To understand what he’s attempting to do in his op-ed, we must return to that essay.

For Thiel, the modern West has been defined by its abandonment of the very questions that once anchored Western civilization. Questions of virtue, faith, and human nature—leaving today’s West uncertain of its purpose. He singles out Locke for infusing the American regime with modern ideas that prioritize peace over deeper religious and philosophical inquiries. Thiel momentarily considers whether a return to an older Western tradition is still possible but concludes that the contemporary awareness of scapegoats prevents a reversion to the past. Strauss, who secretly blames Christianity for the decline of the West, attempted to reverse modernity through a hidden critique of Christianity, where he tells us that Jerusalem and Athens are not compatible, that Christianity hasn’t answered our deepest questions, because revelation is unreasonable. In Thiel’s view on the other side, is that the Christian statesman knows more through what the crucifixion of Christ had uncovered. Meanwhile, Thiel is equally dissatisfied with Girard's answer for us to retreat, whose call to wait patiently for an eventual revelation offers no potential for action.

Thiel concluded that what should be done about the crisis of the West is a return to early modernity. Thiel’s intellectual project over the past two decades, most prominently showcased in Zero to One, has been to restore the foundational belief in technological and scientific progress as an attempt to buy time. M...