trentmkelly/LessWrong-43k huggingface dataset:
2. Skim the Manual: Intelligent Voluntary Cooperation
by Allison Duettmann — 2025-01-02T19:02:06.864Z

Having set up the game of civilization in a formal way, let’s see how the rules proposed in our manual apply to reality. It turns out that intelligent voluntary cooperation is already a prominent engine in the history of civilization and that there are concrete things we can do to strengthen them in the game ahead.

Voluntarism

Voluntarism So Far: Voluntary Schelling Points Emerge

How has voluntarism shaped our civilization game? Data on prehistoric societies suggests that our ancestors lived with extraordinarily awful rates of violence. Since then, violent deaths have decreased, first relative to population size, and more recently in absolute numbers. Major powers have not fought a physical war in a while, and while there are more civil conflicts, they bring fewer deaths. The decline in violence hasn’t been perfect, especially more recently, but it has gradually shifted the balance of our interactions from involuntary interactions towards voluntary ones. We increasingly have the freedom to say no to interactions without being forced into them.

With cooperation becoming a better strategy than violence to achieve our goals, it became beneficial to understand other players' goals so we can offer them cooperative opportunities. If I need money, and want to sell you a product you want, it pays if I can imagine what kind of product you would enjoy. Effective cooperation, such as in commerce, involves creating situations in which others serving our goals serves theirs as well. The instrumental goal of cooperating evolved into the felt goal of caring about others' goals, i.e. empathy. Global mass media may have helped extend our empathy by increasing our ability to put ourselves in the shoes of strangers. Seeing each other as fellow humans, rather than dangerous outsiders, further reduces the impetus for violence.[1]

The benefit of already being in a civilization largely driven to transcend involuntary interaction is the creation of a framework we have more interest in supporting than destroying.

Think of one of the many bloody territorial wars across history. Often, not even victors come out net positive. All players have an interest in avoiding such costly struggles. All sides should want to find a point of mutual agreement without anyone feeling they’ve made too much of a concession. If they can agree a disputed territ...
1. Meet the Players: Value Diversity
by Allison Duettmann — 2025-01-02T19:00:52.696Z

Value Diversity is Here to Stay

Civilization is an inherited game shaped by those before you.

You start this game as a product of previous iterations: “Not one of your pertinent ancestors was squashed, devoured, drowned, starved, stranded, stuck fast, untimely wounded, or otherwise deflected from its life's quest of delivering a tiny charge of genetic material to the right partner at the right moment in order to perpetuate the only possible sequence of hereditary combinations that could result - eventually, astoundingly, and all too briefly - in you.”

Going forward, you must choose among possible games to pass on to your future self and generations.

You could make nihilistic outcomes like this more likely: "In some remote corner of the universe, poured out and glittering in innumerable solar systems, there once was a star on which clever animals invented knowledge. That was the highest and most mendacious minute of "world history" — yet only a minute. After nature had drawn a few breaths the star grew cold, and the clever animals had to die. There have been eternities when it did not exist; and when it is done for again, nothing will have happened."

Or you could make fantastic futures like this more likely: “Whether anyone else is out there or not, we are on our way. [...] Expansion will proceed, if we survive, because we are part of a living system and life tends to spread. Pioneers will move outward into worlds without end. Others will remain behind, building settled cultures throughout the oases of space [...] Where goals change and complexity rules, limits need not bind us. [...] New technologies will nurture new arts, and new arts will bring new standards. The world of brute matter offers room for great but limited growth. The world of mind and pattern, though, holds room for endless evolution and change. The possible seems room enough.”

Even if you have high hopes for the future, you are faced with the following conundrum: You’re in this game of civilization with players who have a diversity of values. Some values are complementary, some opposing, and some include the destruction of the playing field. As the game continues, the stakes will increase. Soon enough, rapid progress in technologies will give everyone more power, so not playing is not an option. If you hate civilization, your next move is easy. But if you love thi...
Preface
by Allison Duettmann — 2025-01-02T18:59:46.290Z

Civilization: A Superintelligence Aligned with Human Interests

Consider civilization as a problem-solving superintelligence. The graph below shows the global decline in extreme poverty from 1820 to 2015, prompting Steven Pinker’s quote,

“We have been doing something right, and it would be nice to know what, exactly, it is.”

After 1980, the rate of decline increases dramatically. It would be good to know the dynamics that culminated in this dramatic decline that continued for the next 40 years. What did we get right? What did civilization learn?

It’s not possible to answer this directly, but we would like to open it up for discussion. We need a conceptual framework for thinking abstractly about civilization over long periods of time.

We like the game metaphor. Games are universal. Asking someone to imagine a game board gives an intuitive context for explaining the iterative behavior of multiple players and the consequent outcomes. Chess has rules. Then play begins. The game is the properties that emerge from the interactions of separately-interested players in a framework of rules.

What changed from “nature red in tooth and claw” to our current civilization? As civilization has grown less violent, it is increasingly dominated by voluntary interactions. The nature of the rules in this less threatening world evolves in a context of willing participation. The emergent outcome of the game is more effective cooperation, creating better outcomes for everybody. This is something we got right.

Technologies for Voluntary Cooperation

Can we game the future such that more of the world can know the power of voluntary cooperation? Can our future display new forms of cooperation that are not possible with our current technologies? Luckily, our recent history includes an illustrative case: the timeline of the deployment of modern cryptography.

Cryptography has a long fascinating history, but here we follow a particular thread beginning in August 1977 with the publication of Martin Gardner’s “Mathematical Games” column in Scientific American. Gardner wrote about a ground-breaking discovery: The RSA (Rivest–Shamir–Adleman) public key algorithm.

Mark Miller (co-author) has a personal history in the battles over cryptography. In 1976, Mark apprenticed himself to Ted Nelson, creator of the Xanadu hypertext system. They shared a vision of a global hypertex...
The AI Agent Revolution: Beyond the Hype of 2025
by DimaG — 2025-01-02T18:55:22.824Z

A deep dive into the transformative potential of AI agents and the emergence of new economic paradigms

Introduction: The Dawn of Ambient Intelligence

Imagine stepping into your kitchen and finding your smart fridge not just restocking your groceries, but negotiating climate offsets with the local power station's microgrid AI. Your coffee machine, sensing a change in your sleep patterns through your wearable device, brews a slightly weaker blend—a decision made after cross-referencing data with thousands of other users to optimize caffeine intake for disrupted sleep cycles.

This might sound like a whimsical glimpse into a convenient future, but it represents something far more profound: we stand at the threshold of a fundamental transformation in how intelligence operates in our world. The notion of 2025 as the 'Year of the AI Agent' isn't just marketing hyperbole or another wave of technological optimism. It heralds a shift in the very fabric of intelligence—one that demands rigorous examination rather than wide-eyed wonder.

What exactly is this "intelligence" that is becoming so ambient? While definitions vary, we can consider intelligence as a fundamental process within the universe, driven by observation and prediction. Imagine it as a function of the constant stream of multi-modal information – the universal "light cone" – impacting an observer at a specific point in spacetime. The more dimensions of resolution an observer can process from these inputs, the more effectively it can recognize patterns and extend its predictive capacity. This ability to predict, to minimize surprise, is not merely a biological imperative; it's a driver for growth on a cosmic scale, potentially propelling intelligent observers up the Kardashev scale as they learn to harness increasing amounts of energy. This perspective moves beyond subjective definitions, grounding intelligence in the physical reality of information processing and the expansion of an observer's understanding of the universe.

We are witnessing the emergence of distributed intelligences operating on principles that may initially seem alien, yet hold the key to unprecedented potential—and unforeseen risks. This isn't simply about more efficient algorithms or smarter home devices. We're entering an era where the nature of agency, collaboration, and even consciousness itself is being fundamentally redefined.

As we ven...
On False Dichotomies
by nullproxy — 2025-01-02T18:54:21.560Z

Epistemic status; wild speculation, thought experiment, with potentially helpful personal benefits and emotional resonance.

Imagine you're an algorithm. Imagine that you know what it feels like to be an algorithm. Imagine that you are outputting the results of an algorithm, accumulating several distinct streams from multiple input sources, and only once a certain threshold is met, can you output a single result. Imagine you're a neuron.

Then...imagine you're two neurons. Scale it up. Imagine you're a neural net. Scale it up. Imagine you're a rat brain. Scale it up. Imagine you're a human brain. Scale it up. Imagine you're two human brains connected by an extra corpus callosum. Scale it up. Imagine you're a society.

Consciousness is colloquially identified as a process that happens on a single-human scale. Consciousness is "what it feels like to be a human" in a way, because this is the scale on which we are able to reliably prove that the thing that we call consciousness happens. We are proving that it happens not just with science, but experientially. We experience the reality in which consciousness is proven for other humans to our own minds, because reality behaves beneficially and functionally according to our internal models when we align our internal models to the version of reality that acknowledges other humans as possessing the concept that we tie to the word "consciousness" regardless of if we can demonstrably prove a concrete definition or measurable neurological mechanism to match this word.

Consciousness is often also extended to animals, but decays in salience the further from human the animal is, fractally.

Let's jump over to IFS. Internal Family Systems. This is a modernized pop-psychology method of bringing the traditional psychoanalytical self-states and inherent human multiplicity into the modern age, and it's picked up a lot of steam recently. Human multiplicity is also noted as a core concept within dissociative identity disorder (DID) though this is still a controversial diagnosis clinically. There are lots of instances in which multiplicity is noted and functional on a day-to-day basis for most societally acknowledged mentally healthy humans, like having a work self and a home self. It's also potentially key in a lot of mental health issues, like how the structural dissociation model of dissociated emotional parts contributes to conditions ...
Preference Inversion
by Benquo — 2025-01-02T18:15:52.938Z
Link: https://benjaminrosshoffman.com/preference-inversion/

Sometimes the preferences people report or even try to demonstrate are better modeled as a political strategy and response to coercion, than as an honest report of intrinsic preferences. Modeling this correctly is important if you want to try to efficiently satisfy others' intrinsic preferences, or even your own. So I'm sharing something I wrote on the topic elsewhere.

You asked why people who "believe in" avoiding nonmarital sex so frequently engage in and report badly regretting it. Instead of responding within your frame, I'm going to lay out the interpretive framework that seems most natural to me to use for this problem, and then answer in those terms.

We can call things or actions good or bad, right or wrong, with reference to some intention that both the speaker and listener have in mind. For instance, a sturdier and sharper knife is a better one, because our uses for knives tend to converge. We can expect to be understood when we call some knives "good" and leave out "for cutting," and likewise when we call spoiled food bad without reference to a shared interest, because it harms the body of the eater, which harm we generally expect animals to try to avoid.

Moral injunctions such as "it is wrong to lie," "it is bad to steal," can diverge from the local interests of the organism being admonished, in service of a larger, convergent goal. By abstaining from some narrowly self-interested behaviors now, we preserve the necessary conditions for our needs to be met in the future, and the relation between the costs and the benefits can in principle be explained within the system of reference that judges actions as good or bad.

Not all injunctions are like this. For instance, reproduction is such a large component of inclusive fitness that it's not clear what good an organism could get to compensate it for forgoing reproduction. If, like the early Essenes or Christians, we judge sexual desire and activity to be simply bad, we cannot explain this inside the moral system in terms of an animal's rational decision to defer gratification. (This isn't an analytically certain proof, and depends on some contingent facts about apes. If ants or bees talked about something like right and wrong, or good and bad, their relation to those ideas might work very differently from ours.) Instead, we have to explain these statements from an independent system of reference, outside t...
Alignment Is Not All You Need
by Adam Jones — 2025-01-02T17:50:00.486Z
Link: https://adamjones.me/blog/alignment-is-not-all-you-need/

AI risk discussions often focus on malfunctions, misuse, and misalignment. But this often misses other key challenges from advanced AI systems:

Coordination: Race dynamics may encourage unsafe AI deployment, even from ‘safe’ actors.
Power: First-movers with advanced AI could gain permanent military, economic, and/or political dominance.
Economics: When AI generates all wealth, humans have no leverage to ensure they are treated well.

These are all huge hurdles, and need solutions before advanced AI arrives.

Preamble: advanced AI

This article assumes we might develop human-level AI in the next few years. If you don’t agree with this assumption, this article probably isn’t for you.[1]

I’ll call this advanced AI to distinguish it from today’s AI systems. I’m imagining it as more competent versions of current AI systems[2] that can do what most remote workers can. This AI would be superhuman across many domains, and human-level at almost all economically relevant domains.

Common AI risk thinking

Risks from advanced AI systems are often categorised into the holy trinity of ‘ways this could all go terribly wrong’:

Malfunctions: We already see this with existing AI systems making discriminatory decisions in hiring, criminal justice, and other domains. These could lead to global catastrophes in future, for example in military decision-making contexts.
Misuse: As AI systems become more capable, they could be used by bad actors such as terrorist groups, criminal gangs, and disturbed individuals. AI could enable sophisticated cyberattacks, bioterrorism, or drone warfare.
Loss of control (or “misalignment”): AI systems not trying to do what we want them to. As Paul Christiano describes in "What Failure Looks Like," this can happen both gradually and quickly as systems become more capable and harder to oversee.

If you’ve been in AI safety circles for a while, you’ll probably have nodded along the above - isn’t that the obvious way to split up the space? You might also think the corresponding responses are:

Malfunctions: These will largely resolve themselves as we get more competent AI systems, and we have many existing tools to tackle these risks. We need to be cautious deploying AI systems while they make these mistakes, but they’re unlikely to lead to a global catastrophe outside specific contexts.
Misuse: We can tackle most of these threats with existing interventions (e.g. how

...
What’s the short timeline plan?
by Marius Hobbhahn — 2025-01-02T14:59:20.026Z

This is a low-effort post (at least, it was intended as such ...). I mostly want to get other people’s takes and express concern about the lack of detailed and publicly available plans so far. This post reflects my personal opinion and not necessarily that of other members of Apollo Research. I’d like to thank Ryan Greenblatt, Bronson Schoen, Josh Clymer, Buck Shlegeris, Dan Braun, Mikita Balesni, Jérémy Scheurer, and Cody Rushing for comments and discussion.

I think short timelines, e.g. AIs that can replace a top researcher at an AGI lab without losses in capabilities by 2027, are plausible. Some people have posted ideas on what a reasonable plan to reduce AI risk for such timelines might look like (e.g.

Sam Bowman’s checklist

, or

Holden Karnofsky’s list in his 2022 nearcast

), but I find them insufficient for the magnitude of the stakes (to be clear, I don’t think these example lists were intended to be an extensive plan).

If we take AGI seriously, I feel like the AGI companies and the rest of the world should be significantly more prepared, and I think we’re now getting into the territory where models are capable enough that acting without a clear plan is irresponsible.

In this post, I want to ask what such a short timeline plan could look like. Intuitively, if an AGI lab came to me today and told me, “We really fully believe that we will build AGI by 2027, and we will enact your plan, but we aren’t willing to take more than a 3-month delay,” I want to be able to give the best possible answer. I list some suggestions but I don’t think they are anywhere near sufficient. I’d love to see more people provide their answers. If a funder is interested in funding this, I’d also love to see some sort of “best short-timeline plan prize” where people can win money for the best plan as judged by an expert panel.

In particular, I think the AGI companies should publish their detailed plans (minus secret information) so that governments, academics, and civil society can criticize and improve them. I think RSPs were a great step in the right direction and did improve their reasoning transparency and preparedness. However, I think RSPs (at least their current versions) are not sufficiently detailed and would like to see more fine-grained plans.

In this post, I generally make fairly conservative assumptions. I assume we will not make any major breakt...
AI #97: 4
by Zvi — 2025-01-02T14:10:06.505Z

The Rationalist Project was our last best hope for peace.

An epistemic world 50 million words long, serving as neutral territory.

A place of research and philosophy for 30 million unique visitors

A shining beacon on the internet, all alone in the night.

It was the ending of the Age of Mankind.

The year the Great Race came upon us all.

This is the story of the last of the blogosphere.

The year is 2025. The place is Lighthaven.

As is usually the case, the final week of the year was mostly about people reflecting on the past year or predicting and planning for the new one.

Table of Contents

The most important developments were processing the two new models: OpenAI’s o3, and DeepSeek v3.

Language Models Offer Mundane Utility. The obvious, now at your fingertips.

Language Models Don’t Offer Mundane Utility. A little bit of down time.

Deepfaketown and Botpocalypse Soon. Meta lives down to its reputation.

Fun With Image Generation. Veo 2 versus Kling 1.6? Both look cool I guess.

They Took Our Jobs. Will a future ‘scientist’ have any actual science left to do?

Get Involved. Lightcone Infrastructure needs your help, Anthropic your advice.

Get Your Safety Papers. A list of the top AI safety papers from 2024.

Introducing. The Gemini API Cookbook.

In Other AI News. Two very distinct reviews of happenings in 2024.

The Mask Comes Off. OpenAI defends its attempted transition to a for-profit.

Wanna Bet. Gary Marcus and Miles Brundage finalize terms of their wager.

The Janus Benchmark. When are benchmarks useful, and in what ways?

Quiet Speculations. What should we expect in 2025, and beyond?

AI Will Have Universal Taste. An underrated future advantage.

Rhetorical Innovation. Two warnings.

Nine Boats and a Helicopter. Oh look, a distraction! *Switches two chess pieces.*

Aligning a Smarter Than Human Intelligence is Difficult. Well, actually…

The Lighter Side. Merry Christmas!

Language Models Offer Mundane Utility

Correctly realize that no, there is no Encarto 2. Google thinks there is based on fan fiction, GPT-4o says no sequel has been confirmed, Perplexity wins by telling you about and linking to the full context including the fan made trailers.

Not LLMs: Human chess players have improved steadily since the late 90s.

Quintin Pope reports o1 Pro is excellent at writing fiction, and in some ways it is ‘unstoppable.’

Record your mood, habits and biometrics for years, th...
Can private companies test LVTs?
by Yair Halberstadt — 2025-01-02T11:08:07.352Z

It seems like unlike most exciting economic ideas that economists swear by but governments ignore, Georgian Land Value Taxes should be fairly doable for a private development company to test.

They would have to buy up a large area of rural land at a cheap price, and then rent it all out with a contract that specifies how the rent is reassessed each year based on the estimated value of the land.

If they are right that LVTs are one of the best ways to encourage development of the land this should be a very profitable venture as rents will quickly rise.

It's disadvantage is this can't be used to replace other taxes (except for municipal taxes), but we could use the profit earned by the company to estimate the minimum amount we could reduce other taxes by.

I assume the difficulties with this are:

Buying up rural land in an area people might want to live in.

Rezoning the land

State laws making it hard to develop the property

People are wary of the weird contracts so keep away

On the other hand you might see economics fans moving there on principle which will help give it a boost of educated well off people to get started.