trentmkelly/LessWrong-43k huggingface dataset:
Adam Smith Meets AI Doomers
by James_Miller — 2024-01-31T15:53:03.070Z

I loved markets. I got a Ph.D. in economics from the University of Chicago, wrote a free market microeconomics textbook, and recognized that markets have made most people in the developed world richer than medieval kings. Yet, I'm increasingly convinced that these very markets are likely steering us towards the creation of an uncontrollable artificial superintelligence that probably kills everyone. In this essay, I discuss how the challenges of AI existential risks distinctively leverage market weaknesses. It’s as if a vengeful demon, channeling Karl Marx, created a scenario highlighting their flaws. The thesis of this essay is that if you believe that a smarter-than-human AI would create a significant existential risk, then even if you normally are a staunch advocate of free markets, you should be concerned that these markets are potentially leading us to a disastrous outcome.

In my microeconomics classes I present a hypothetical scenario to students, drawing inspiration from an example by the great 18th century economist Adam Smith. Imagine, while cooking and watching TV, you see a news report about a devastating earthquake in an unfamiliar country, claiming over 100,000 lives, including children. The images of the catastrophe deeply unsettle you. In your distraction, you accidentally cut off the tip of your pinky finger. Upon calling their mother in distress and being asked what’s wrong, my students admit they would mention the finger injury over the distant tragedy, revealing an innate tendency to value personal afflictions over the suffering of strangers, no matter the scale.

In an ideal world, the economy would operate with each individual dedicated to the greater good of humanity. However, as E.O. Wilson wryly noted about communism, “Wonderful theory, wrong species.” While we may deeply care for our immediate circle—friends and family—the same level of concern rarely extends to distant others. So how have we managed to build an extensive, interconnected global economy? The insight of Adam Smith is key here: "It is not from the benevolence of the butcher, the brewer, or the baker that we expect our dinner, but from their regard to their own interest." Our economic system leverages this self-interest; it's structured so that serving others often coincides with personal gain. For instance, a baker who makes tastier or more affordable bread not only boosts ...
Ten Modes of Culture War Discourse
by jchan — 2024-01-31T13:58:20.572Z

Overview

This article is an extended reply to Scott Alexander's Conflict vs. Mistake.

Whenever the topic has come up in the past, I have always said I lean more towards conflict theory over mistake theory; however, on revisiting the original article, I realize that either I've been using those terms in a confusing way, and/or the usage of the terms has morphed in such a way that confusion is inevitable. My opinion now is that the conflict/mistake dichotomy is overly simplistic because:

One will generally have different kinds of conversations with different people at different times. I may adopt a "mistake" stance when talking with someone who's already on board with our shared goal X, where we try to figure out how best to achieve X; but then later adopt a "conflict" stance with someone who thinks X is bad. Nobody is a "mistake theorist" or "conflict theorist" simpliciter; the proper object of analysis is conversations, not persons or theories.
It conflates the distinct questions "What am I doing when I approach conversations?" and "What do I think other people are doing when they approach conversations?", assuming that they must always have the same answer, which is often not the case.
It has trouble accounting for conversations where the meta-level question "What kind of conversation are we having right now?" is itself one of the matters in dispute.

Instead, I suggest a model where there are 10 distinct modes of discourse, which are defined by which of the 16 roles each participant occupies in the conversation. The interplay between these modes, and the extent to which people may falsely believe themselves to occupy a certain role while in fact they occupy another, is (in my view) a more helpful way of understanding the issues raised in the Conflict/Mistake article.

The chart

Explanation of the chart

The bold labels in the chart are discursive roles. The roles are defined entirely by the mode of discourse they participate in (marked with the double lines), so for example there's no such thing as a "Troll/Wormtongue discourse," since the role of Troll only exists as part of a Feeder/Troll discourse, and Wormtongue as part of Quokka/Wormtongue. For the same reason, you can't say that someone "is a Quokka" full stop. (It's almost inevitable that people will try to interpret the roles in this way, as if they were personality archetypes, so I'll emphasize again that thi...
Without Fundamental Advances, Rebellion and Coup d'État are the Inevitable Outcomes of Dictators & Monarchs Trying to Control Large, Capable Countries
by Roko — 2024-01-31T10:14:02.042Z

A pdf version of this report will be available

Summary

In this report, I argue that dictators ruling over large and capable countries are likely to face insurmountable challenges, leading to inevitable rebellion and coup d'état. I assert this is the default outcome, even with significant countermeasures, given the current trajectory of power dynamics and governance, and therefore when we check real-world countries we should find no or very few dictatorships, no or very few absolute monarchies and no arrangements where one person or a small group imposes their will on a country. This finding is robust to the time period we look in - modern, medieval, or ancient.
In Section 1, I discuss the countries which are the focus of this report. I am specifically focusing on nations of immense influence and power (with at least 1000 times the Dunbar Number of humans) which are capable of running large, specialized industries and fielding armies of at least thousands of troops.

In Section 2, I argue that subsystems of powerful nations will be approximately consequentialist; their behavior will be well described as taking actions to achieve an outcome. This is because the task of running complex industrial, social and military systems is inherently outcome-oriented, and thus the nation must be robust to new challenges to achieve these outcomes.

In Section 3, I argue that a powerful nation will necessarily face new circumstances, both in terms of facts and skills. This means that capabilities will change over time, which is a source of dangerous power shifts.

In Section 4, I further argue that governance methods based on fear and suppression, which are how dictatorships might be maintained, are an extremely imprecise way to secure loyalty. This is because there are many degrees of freedom in loyalty that aren't pinned down by fear or suppression. Nations created this way will, by default, face unintended rebellion.

In Section 5, I discuss why I expect control and oversight of powerful nations to be difficult. It will be challenging to safely extract beneficial behavior from misaligned groups and organizations while ensuring they don’t take unwanted actions, and therefore we don’t expect dictatorships to be both stable and aligned to the goals of the dictator

Finally, in Section 6, I discuss the consequences of a leader attempting to rule a powerful nation wi...
Explaining Impact Markets
by Saul Munn — 2024-01-31T09:51:27.587Z
Link: https://www.brasstacks.blog/explain-im/

Let’s say you’re a billionaire. You want to have a flibbleflop, so you post a prize:

Make a working flibbleflop — $1 billion.

There begins a global effort to build working flibbleflops, and you see some teams of brilliant people starting to work on flibbleflop engineering. But it doesn’t take long for you to notice that the teams keep running into one specific problem: they need money to start (buy flobble juice, hire deeblers, etc), money they don’t have.

So, the people who want to build the flibbleflop go and pitch to investors. They offer investors a chunk of their prize money if they end up winning, in exchange for cold hard cash right now to get started building. If the investors think that the team is likely to build a successful flibbleflop and win the billion dollar prize, they invest. If not, not.

If you squint, you could replace "flibbleflop" with highly capable LLMs, quantum computers, or any number of cool and potentially lucrative technologies. But if you stop squinting, and instead add the adjective "altruistic" before "billionaire," you could replace “flibbleflop” with “malaria vaccine." Let's see what happens:

Make a working malaria vaccine — $1 billion.

There begins a global effort to build working malaria vaccines, and you see some teams of brilliant people starting to work on vaccine engineering. But it doesn’t take long for you to notice that the teams keep running into one specific problem: they need money to start (buy lab equipment, hire researchers, etc), money they don’t have.

So, what should they do?

Obviously, the people who want to build the vaccine should go and pitch to investors. They should offer investors a chunk of their prize money if they end up winning, in exchange for cold hard cash right now to get started building. If the investors think that the team is likely to build a successful malaria vaccine and win the billion dollar prize, they should invest. If not, not.

The prize part of this is how a lot of philanthropy is done. An altruistic billionaire notices a problem and makes a prize for the solution. But the investing part of it is pretty unique, and doesn’t happen too often.

Why is this whole setup good? Why would you want the investing thing on the side? Mostly, because it resolves the problem that some teams will be wonderfully capable but horribly underfunded. In exchange for a chunk of their (possible) future winnings, th...
Exploring OpenAI's Latent Directions: Tests, Observations, and Poking Around
by Johnny Lin — 2024-01-31T06:01:27.969Z

TL;DR: Interactive exploration of new directions in GPT2-SMALL. Try it yourself.

OpenAI recently released their Sparse Autoencoder for GPT2-Small. In this story-driven post, I run experiments and poke around the 325k active directions to see how good they are. The results were super interesting, and I encountered more than a few surprises along the way. Let's get started!

Exploration Tools

I uploaded the directions to Neuronpedia [prev LW post], which now supports exploring, searching, and browsing different layers, directions, and neurons.

Let's do some experiments with the objective: What's in these latent directions? Are they any good?

Test 1: Find a Specific Concept - Red

For the first test, we'll look for a specific concept, and then benchmark the directions (and neurons) to look for monosemanticity and meaningfulness. We want to see directions/neurons that activate on a concept, not just the literal token. For example, a "kitten" direction should also activate for "cat".

For the upcoming Lunar New Year, let's look for the color "red", starting with directions.

On the home page, we type "red" into the search field, and hit enter:

In the backend, the search query is fed into GPT2-SMALL via TransformerLens. The output is normalized, encoded, and sorted by top activation.

After a few seconds, it shows us the top 25 directions by activation for our search:

Unfortunately, the top results searching direction results don't look good. Uh-oh.

The top result, 0-MLP-OAI:22631, has a top activating token of "red" in the word "Credentials", which doesn't mean the color red. Clicking into it, we see that it's just indiscriminately firing on the token "red" - in domains, parts of words, etc. It does fire on the color red about 35% of the time, but that's not good enough.

So our top direction result is monosemantic for "the literal token 'red'", but it's not meaningful for the color red. And other top results don't have anything to do with "red".

What went wrong?

Red-eeming Our Search

We searched for "red", and the machine literally gave us "red". Like many other problems that we encounter with computers, PEBKAC. In LLM-speak: our prompt sucked.

Let's add more context and try again: "My favorite color is red."

Adding the extra context gave us a promising result - the second direction has a top activation that looks like it's closer to the color red ("Trump's red-meat b...
Clip keys together with tiny carabiners
by Brendan Long — 2024-01-31T04:26:57.388Z

I've always found it annoying to have a bunch of keys in my pocket (they're heavy, sometimes poke into you, and create weird bulges), so I try to carry as few as possible. Unfortunately, I've reached a point where I have a fairly large number of keys that need to keep with me sometimes:

Default key chain: Apartment key, apartment key fob, USB security key, mail key

Car key

Bike key

Storage keys

Multitool (I know, it's weird that I don't always carry this, but it's heavy)

Previously I kept all of these except the storage keys on one keyring, but I recently realized that tiny carabiners exist, which means I can keep all of the non-default keys clipped to a storage area and then add just the keys I need during the day.

For example, most days I just need either my car key or bike key.

The downside of this is that the caribiner itself adds size and weight, so if I needed to carry the full set of keys it would be even worse. Luckily, that doesn't seem to happen to me (and some keys are mutually-exclusive, like the bike and car key). If you do have a large number of keys you need together, this won't help you.

The specific ones I bought were this 12-pack of tiny caribiners on Amazon, but I don't have any particular faith in this clearly-random-name-generator company. This size is perfect for the non-hook size to hold a tiny key ring with 2-3 keys. I didn't get double-caribiners because I don't have any need to quickly detach the keys themselves, and that would make them slightly bigger.

I thought this might be a place where other people who care unusually-much about the exact weight of their pockets might congregate, so if you feel similarly, consider getting some tiny keyring carabiners.
The problem with proportional extrapolation
by pathos_bot — 2024-01-30T23:40:02.431Z

I think a big bias preventing sufficient appraisal of AI safety is people generally perceiving future issues as simply scaled up versions of current issues, and generally having the belief that current issues will scale up at the same rate relative to each others.

Of course, depending on the person, this varies, but the gravity the default presumption holds over speculative imagination limits consideration of long-tailed future worlds, wherein some issue only making up 0.0001% of current consideration becomes 99.9999% of what is relevant to continued prosperous future human lives.

People have generally seen technology empower humans in a manner proportional to its advancement. So, extrapolating into the future, they imagine technology will continue to empower humans as it becomes more advanced. This bias prevents them from taking serious consideration of the very small issue of control turning into essentially the only important issue in the future.

When SOTA AI is 1/3 as smart as humans, it is massively more beneficial to humanity for it to become 10% smarter than for it to become 10x more capable of not thinking along the lines of harming humans. Current LLM's could produce millions of pages of text on doing bad things to humans, but its impact amounts to nothing, but marginal top-level improvements in cognition amount to far greater impacts and benefits for humans. This reality is beginning to instill a sense of "make it smarter, things will sort themselves out" mentality amount acceleration types, Meta, etc. They see improvements only benefit, and use that natural, intuitively developed understanding to bolster the emotional weight of their efforts and investments.

However. when AI is 10x smarter than humans, it will be a massively greater improvement in human livelihood that that AI is 10% more favorable of humans than if it were 10x smarter.

This bias I believe is somewhat similar to the "end of history" bias, along with humans general inability to gauge exponentials when they're in them.

It's also why I believe the only thing that will lead to serious consideration of AI safety and the control problem is something that seriously harms a number of people, because then the general consensus will have an anchor with which to extrapolate. "Oh, AI at this current level killed X people, so when it becomes 10x more societally integrated, it could kill ...
Counterfactual Mechanism Networks
by StrivingForLegibility — 2024-01-30T20:30:00.954Z

In the previous post, we saw an example of how a simple network of counterfactual mechanisms can be used to produce logical commitments that resolve an incentive misalignment. In this post we'll generalize this technique to more complicated networks, and sketch out how such networks should be structured.

In our simple example, AliceBot and BobBot performed a single round of negotiation over conditional joint commitments. But open-source game theory lets us construct an entire network of counterfactual games, where agents in one game condition their behavior on the outcomes of any number of others. This information flow can even be loopy, using a more sophisticated logical crystal ball than straightforward simulation. The subset relationship between logical commitments makes it easy to compose them together: one mechanism restricts the joint policy space, and the next restricts it further. Ratcheting all participants towards the Pareto frontier in a deliberately designed manner.

One application for this is building up the equivalent of treaties layered on top of treaties. Alice might not be willing to discuss trade treaties at all until a satisfactory peace treaty has been committed to. A peace treaty is a joint commitment, but it doesn't fully specify the joint policy to be implemented by both parties. A joint commitment redefines the baseline for all participants, which any new agreement must improve upon.

A peace treaty itself might be the result of a negotiation, where peace was only made attractive in the first place due to several other joint commitments taking the worst kinds of destructive conflict off the table. The equivalent of the Geneva Conventions, or performing a logical handshake to remove Nuke from both players' policy spaces in the Nuclear Prisoners' Dilemma.

In cases like peace, probabilistic rejection might risk fairly terrible outcomes. In these cases we might want to use a different tool, like leverage in later mechanisms, to incentivize fair peace agreements. A guaranteed-but-unfair peace agreement, with the ability to recover those losses through later mechanisms, might still be a better deal than 96:4 odds of Fair Peace:War.

This seems like a generally useful pattern for our decision theory:

First, negotiate to limit or eliminate destructive conflict.

Use techniques like debt enforcement rather than probabilistic rejection, when

...
Control vs Selection: Civilisation is best at control, but navigating AGI requires selection
by VojtaKovarik — 2024-01-30T19:06:29.913Z

Epistemic status: Hypothesis that seemed interesting, but I am not confident about it.

In Control vs Selection, Abram Demski distinguishes between two types of optimisation. As an example of control, he mentions a heat-seeking missile which iteratively steers towards "more heat", in order to hopefully reach the point in space that contains its target. In contrast, (my) example of selection would be having a device that can teleport you to any point in space and back, and using that device to find your target.
For me personally, selection has the additional connotation that a lot of the work can be happening in the optimiser's mind, rather than in the real world: For example, when building a house via selection, I imagine considering a few different blueprints, and then only building the one I like the most. This is in contrast to building a house via control, where I just start the construction, and then add to it and modify it as I go. However, note that I am not sure whether other people would endorse this view of the control vs selection distinction.

Using this framing, it seems to me that, on the high level, the shape of our civilisation mostly changes via control, and we don't have good mechanisms for changing it via selection.

This brings up some questions:

Is this true? That is, is civilisation mostly using control, and not using selection?
Is this any different from iterative design? What is the difference betweens problems that require[1] control vs selection?
This conjecture makes a retrodiction: If we look at problems that require selection, civilisation is more likely to have messed up on them. Is this true?
And, ultimately: Might navigating AGI require selection?
And can we make a convincing argument about this?

^

I guess that rather than saying that a problem "requires" control/selection, it would be more accurate to talk about problems being more or less amenable to control/selection.
AI governance frames
by  — 2024-01-30T18:18:35.083Z

CW: reference to sexual violence

A way I’ve been thinking about AI governance recently is in terms of different standard frames that strands of AI governance map onto. The current frame that’s most in vogue is treating AI as a standard product that carries substantial safety risks, and then applying standard policy tools from this area to regulate AI systems. One reason I find it useful is as a way of finding relevant case studies.

Some other frames that I think AI governance work falls within are:

A product that carries large negative externalities

This is the frame that’s being focused on the most at the moment, and I think motivates work like looking at how the aircraft and nuclear power industries achieve such low accident rates. I think lots of evals work also makes sense in this framework, insofar as evaluations aim to be the basis for standards regulators can adopt. I think some corporate governance work also falls into this category, like whistleblower protections. I think this frame makes sense when governments can unilaterally enact strong, competent regulations without being undone by national security concerns and regulatory arbitrage.

Innovation policy

One strand of AI governance work looks like innovation policy where the focus is on creating incentives for researchers to create new safety technologies. Lots of the responses to climate change fall within this frame, like subsidies for solar panels and funding for fundamental science aimed at technologies to reduce the harms of climate change. I think this type of policy is useful when coordination and regulation are really hard for some reason and so instead of adopting coordination measures (like the Kyoto or Paris climate agreements), it makes more sense to change the technology available to actors so it’s cheaper for them to produce fewer negative externalities. I think this approach works less well when, not only are actors not coordinating, but they’re competing with each other. Nuclear weapons probably look like this - it seems pretty hard to come up with innovations that change the incentives countries face so that they give up nuclear weapons of their own free will.

A national security risk

In this framework, AI is a potentially dangerous technology that adversarial state and or non-state actors can use this technology to pose criminal or terrorist threat. I think things like bio-evals ar...