trentmkelly/LessWrong-43k huggingface dataset:
MIRI 2024 Mission and Strategy Update
by Malo — 2024-01-05T00:20:54.169Z

As we

announced

back in October, I have taken on the senior leadership role at MIRI as its CEO. It’s a big pair of shoes to fill, and an awesome responsibility that I’m honored to take on.

There have been several changes at MIRI since

our 2020 strategic update

, so let’s get into it.[1]

The short version:

We think it’s very unlikely that the AI alignment field will be able to make progress quickly enough to prevent human extinction and the loss of the future’s potential value, that we expect will result from loss of control to smarter-than-human AI systems.

However, developments this past year like the release of ChatGPT seem to have shifted the

Overton window

in a lot of groups. There’s been a lot more discussion of extinction risk from AI, including among policymakers, and the discussion quality seems greatly improved.

This provides a glimmer of hope. While we expect that more shifts in public opinion are necessary before the world takes actions that sufficiently change its course, it now appears more likely that governments could enact meaningful regulations to forestall the development of unaligned, smarter-than-human AI systems. It also seems more possible that humanity could take on a new megaproject squarely aimed at ending the acute risk period.

As such, in 2023, MIRI shifted its strategy to pursue three objectives:

Policy: Increase the probability that the major governments of the world end up coming to some international agreement to halt progress toward smarter-than-human AI, until humanity’s state of knowledge and justified confidence about its understanding of relevant phenomena has drastically changed; and until we are able to secure these systems such that they can’t fall into the hands of malicious or incautious actors.[2]
Communications: Share our models of the situation with a broad audience, especially in cases where talking about an important consideration could help normalize discussion of it.
Research: Continue to invest in a portfolio of research. This includes technical alignment research (though we’ve become more pessimistic that such work will have time to bear fruit if policy interventions fail to buy the research field more time), as well as research in support of our policy and communications goals.[3]

We see the communications work as instrumental support for our policy objective. We also see candid and honest communication ...
Project ideas: Governance during explosive technological growth
by Lukas Finnveden — 2024-01-04T23:51:56.407Z
Link: https://www.forethought.org/research/project-ideas-governance-during-explosive-technological-growth

This is part of a series of lists of projects. The unifying theme is that the projects are not targeted at solving alignment or engineered pandemics but still targeted at worlds where transformative AI is coming in the next 10 years or so. See here for the introductory post.

Commonly discussed motivations cited for why rapid AI progress might be scary are:

Risks from misalignment.
Risk from AI-assisted bioweapons.

But even aside from these risks, it seems likely that advanced AI will lead to explosive technological and economic growth across the board, which could lead to a large number of problems emerging at a frighteningly fast pace.[1]

The growth speed-ups could be extreme. The basic worry is that AI would let us return to a historical trend of super-exponential growth.[2] If this happens, I don’t know any reassuring upper limit for how fast growth could go.

(Illustratively: Paul Christiano’s suggested definition for slow takeoff is “There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles.” If world GDP doubled in a single year, that would mean that growth was ~30x faster than it is right now.)

If technological growth speeds up by, say, 30x, then that suggests that over just a few years, we might have to deal with all the technologies that would (at a “normal” pace) be discovered over 100 years. That’s an intense and scary situation.

This section is about problems that might arise in this situation and governance solutions that could help mitigate them. It’s also about “meta” solutions that could help us deal with all of these issues at once, e.g. by improving our ability to coordinate to slow down the development and deployment of new technologies.

Note: Many of the projects in this section would also be useful for alignment. But I’m not covering any proposals that are purely focused on addressing alignment concerns.

Investigate and publicly make the case for/against explosive growth being likely and risky [Forecasting] [Empirical research] [Philosophical/conceptual] [Writing]

I think there’s substantial value to be had in vetting and describing the case for explosive growth, as well as describing why it could be terrifying. Explosive growth underlies most of the concerns in this section — so establishing the basic risk is very important.

Note: There’s some possi...
Hello
by S Benfield — 2024-01-04T23:35:05.621Z

Hi. I found this site while having some consciousness discussions on reddit.

I've been a Chief Technology Officer for 25+ years, including two public companies. I have 12 patents around engagement and learning. My interests include theoretical physics, evolutionary psychology, cognition, AI, ethics, and all things music/dance/art/design. My modus operandi is to try to get to the abstraction around a problem. I delve into the details enough to formulate my hypothesis on the patterns I see. Then I will evolve those theories based on new information. But I don't have the time, nor the capacity, to be deeply read in everything I think about. What I do have is the ability to abstract quickly and to abstract about abstraction. I am also highly emotionally aware and have spoken with 2,000 different people in the past 18 months about their hopes, dreams, and desires. The patterns I found there inform my work as well as they serve as a real human laboratory for iteration of theories.

I have several theories that I am working on and welcome anyone's input on these

1 - Time is the first dimension, Space is the second. Time is the only wave function. Gravity is nothingness which is why gravitons will never be found; the warping of time-space is warping around the collapse of the time wave function--which proceeds excruciatingly slowly from our point of view. Because gravity is nothing, it cannot be conserved. the Big Bang started with every quantum dot filled with a proton. And the universe is resolving itself through the time wave function. Observers must be included because no observers exist outside of time.

2 - Life evolved in the oceans as a superconductive charge density wave. Over time sensors were evolved that detected the absence of light. All evolution follows from here. Over time more sensors and Brownian motion gave way to pattern recognition and bodily movements. I believe our brains record every 1/n second of our life by recording a snapshot of on stimuli (12Mbits approx) and an n-bit summary of binary decisions to arrive at emotion.

3 - When new information is received, emotional valuation is conducted by the limbic system, valance is considered (with valence being a scale from pure safety to possible death of self, others, the world). Negative emotional evaluations produce GABA in quantities to slow the brain so it can focus on the current stimuli...
Using Threats to Achieve Socially Optimal Outcomes
by StrivingForLegibility — 2024-01-04T23:30:54.615Z

In the last post, we saw an example of how dath ilan's Algorithm calls for negotiators to use non-credible threats to achieve fair outcomes when bargaining. The Algorithm also has a lot to say about threats more generally. In particular, that there is a type of threat that shouldn't appear in equilibria of ideal agents, because ideal agents don't give in to those kinds of threats. What sorts of threats should ideal agents make? And what sorts of threats should they give in to?

Threats in dath ilan

There is a word in Baseline, the language of dath ilan, which is simply translated as "threat". One connotation of this word is non-credibility.

Because, of course, if you try to make a threat against somebody, the only reason why you'd do that, is if you believed they'd respond to the threat; that, intuitively, is what the definition of a threat is.

The government of dath ilan does not use (non-credible) threats to enforce its laws. It only imposes those penalties which it has an incentive to actually impose, even if actually placed in a situation where a citizen has actually broken a law. (This is contrasted with a law which is not in a government's interest in that subgame to actually enforce, and therefore constitutes a non-credible threat.)

The dath ilani built Governance in a way more thoroughly voluntarist than Golarion could even understand without math, not (only) because those dath ilani thought threats were morally icky, but because they knew that a certain kind of technically defined threat wouldn't be an equilibrium of ideal agents; and it seemed foolish and dangerous to build a Civilization that would stop working if people started behaving more rationally.

So to recap, there is some notion of "threat", which should never appear in any real policy adopted by ideal agents. (Where a policy defines how an agent will behave in any situation.) Some credible threats, like exiling murderers, are compatible with the Algorithm. As are some non-credible threats, like refusing an unfair offer, even when doing so locally results in receiving less payoff than accepting.

There are also times when one agent simply has a lot of power over another, and can unilaterally dictate an unfair outcome. When another party can't fight back, a policy of taking their stuff is credible, but it certainly isn't nice.

Making and Responding to Threats

I suggest that a key feature is "if I wer...
Best-Responding Is Not Always the Best Response
by StrivingForLegibility — 2024-01-04T23:30:48.400Z

Suppose that you are designing a software system to bargain on your behalf. Either side can walk away from negotiations, and we'll use this as our baseline for each side receiving $0 of the possible gains from working together. How should such a system handle a case where it receives a take-it-or-leave-it offer of $0.01 for you, and $99.99 for the proposer? And how can we generalize a solution to bargaining problems to achieve socially optimal outcomes in a wide range of strategic contexts?

Classical Analysis

The classical game theory analysis is as follows: Your system has two options at this point in the game: Accept or Reject. Accept leads to a payoff of $0.01, Reject leads to a payoff of $0.00. A penny is better than nothing, so Accepting leads to a higher payoff for you than Rejecting. A classically-rational agent Accepts an offer of $0.01 for itself, and $99.99 for the proposer.

What Went Wrong

Quick terminology review: A system's policy defines how it behaves in any situation. A subgame is part of a game, where some stuff has already happened. A non-credible threat is a policy which calls for an agent to pay costs in order to impose costs on another agent. (Pay costs in the sense of "take an action in a subgame which leads to less payoff for that agent, than they could get in that subgame by unilaterally doing something else.") A Nash equilibrium is a state of affairs where no agent can do better for themselves by unilaterally changing their policy. In other words, a Nash equilibrium is a mutual best-response.

In the classical analysis, making a non-credible threat can be part of a Nash equilibrium. Actually carrying out a non-credible threat can't be part of any Nash equilibrium. This is because making a non-credible threat is free, while actually carrying out a non-credible threat is costly. An agent which finds themselves in a position to carry out their non-credible threat has a local incentive to not follow through. And classically-rational agents always follow their local incentives.

I'm using local incentives here to refer to "the payoffs of the subgame in which an agent currently finds themselves." An agent may be able to do better for themselves globally by acting according to a policy which involves acting against their incentives locally. Classically-rational agents employ Causal Decision Theory, which ignores the counterfactual branches of the g...
Safety Data Sheets for Optimization Processes
by StrivingForLegibility — 2024-01-04T23:30:36.510Z

In the last post, I expressed that it would be nice to have safety data sheets for different optimization processes. These data sheets would describe the precautions necessary to use each optimization process safely. This post describes an approach for writing such safety guidelines, and some high-level guidelines for safely applying optimization.

The Safety Mindset

We can think of an optimization metric as defining what an optimization process is "trying" to do. I'm also persuaded by Yudkowsky's take that we shouldn't build software systems that are "trying" to do things we don't want them to do:

Suppose your AI suddenly became omniscient and omnipotent - suddenly knew all facts and could directly ordain any outcome as a policy option. Would the executing AI code lead to bad outcomes in that case? If so, why did you write a program that in some sense 'wanted' to hurt you and was only held in check by lack of knowledge and capability? Isn't that a bad way for you to configure computing power? Why not write different code instead?

That whole article is worth a read, and describes what Yudkowsky calls the safety mindset, analogous to the security mindset. Where the real adversary isn't hackers trying to subvert your system, it's your own mind that makes assumptions during system design that might not hold for all contexts your system might operate in. One approach that Yudkowsky advocates in both contexts is to limit those assumptions as much as possible.

Safety Guidelines

Here is a high-level overview of some guidelines that might appear on a safety data sheet for optimization processes:

Think about other ways to achieve the same goal, other than using optimization.

Optimization can be dangerous, and often many of the benefits can be achieved by other means.
For example, instead of asking your system to search over ways to behave, just tell it what you want it to do.

If you need to use optimization, use it as little as possible.

To the greatest extent possible, your software's behavior should be defined outside of an optimization process.
Optimization should occupy as small a module as possible in your system's architecture.

Notice what assumptions need to hold in order for this system to operate safely. Try to limit these as much as possible.

For example, don't rely on the assumption that your software will never be fed an invalid input. Design your software so that

...
The Gears of Argmax
by StrivingForLegibility — 2024-01-04T23:30:30.339Z

In the last post, I made the high-level claim that optimization can sometimes be done safely. And went on to claim that a key factor enabling safe optimization is sufficiently understanding the optimization process, optimization metric, and domain of candidate solutions.

In this post, I'll dig into how a particular optimization process works, and how it relates to other areas we care about like game theory and AI safety. Like sulfuric acid, argmax is well-understood enough that it is safely employed in all sorts of applications. Also like sulfuric acid, it's straightforward to construct systems which employ argmax unsafely. Sulfuric acid can't be thrown around willy-nilly, but we know what sort of precautions we need to take to use it safely. I would love something like safety data sheets for different optimization processes.

Max

To get started, check out this simpler function: max. It takes a list of numbers and returns the largest one. I'll suggestively call these numbers "scores", to foreshadow their role in optimization. Here's a simplified implementation:

def max(scores):
highest_score = float('-inf')

for score in scores:
if score > highest_score:
highest_score = score

return highest_score

In English, max starts by defining a variable to keep track of the highest score it's seen so far, and initializes it with the value negative infinity. Then it looks at every score in the list. If that score is higher than the highest score it's seen so far, it assigns that value to the "highest score I've seen so far" variable. After doing this for every score in the list, it returns the highest score it found!

Argmax

Argmax is a function that takes two arguments: a list of candidates, and a function that determines the score of that candidate, which here I've called "metric". It returns a candidate with the highest score. Here's a naïve implementation:

def argmax(candidates, metric):
best_candidate = None
highest_score = float('-inf')

for candidate in candidates:
score = metric(candidate)

if score > highest_score:
best_candidate = candidate
highest_score = score

return best_candidate

Like before, argmax declares variables to keep track of the best candidate it's seen so far, and the score it received. Then it looks at all of the candidates in...
Cellular reprogramming, pneumatic launch systems, and terraforming Mars: Some things I learned about at Foresight Vision Weekend
by jasoncrawford — 2024-01-04T19:33:57.887Z
Link: https://rootsofprogress.org/vision-weekend-2023-writeup

In December, I went to the Foresight Institute’s Vision Weekend 2023 in San Francisco. I had a lot of fun talking to a bunch of weird and ambitious geeks about the glorious abundant technological future. Here are few things I learned about (with the caveat that this is mostly based on informal conversations with only basic fact-checking, not deep research):

Cellular reprogramming

Aging doesn’t only happen to your body: it happens at the level of individual cells. Over time, cells accumulate waste products and undergo epigenetic changes that are markers of aging.

But wait—when a baby is born, it has young cells, even though it grew out of cells that were originally from its older parents. That is, the egg and sperm cells might be 20, 30, or 40 years old, but somehow when they turn into a baby, they get reset to biological age zero. This process is called “reprogramming,” and it happens soon after fertilization.

It turns out that cell reprogramming can be induced by certain proteins, known as the Yamanaka factors, after their discoverer (who won a Nobel for this in 2012). Could we use those proteins to reprogram our own cells, making them youthful again?

Maybe. There is a catch: the Yamanaka factors not only clear waste out of cells, they also reset them to become stem cells. You do not want to turn every cell in your body into a stem cell. You don’t even want to turn a small number of them into stem cells: it can give you cancer (which kind of defeats the purpose of a longevity technology).

But there is good news: when you expose cells to the Yamanaka factors, the waste cleanup happens first, and the stem cell transformation happens later. If we can carefully time the exposure, maybe we can get the target effect without the damaging side effects.

This is tricky: different tissues respond on different timelines, so you can’t apply the treatment uniformly over the body. There are a lot of details to be worked out here. But it’s an intriguing line of research for longevity, and it’s one of the avenues being explored at Retro Bio, among other places. Here’s a Derek Lowe article with more info and references.

The BFG orbital launch system

If we’re ever going to have a space economy, it has to be a lot cheaper to launch things into space. Space Shuttle launches cost over $65,000/kg, and even the Falcon Heavy costs $1500/kg. Compare to shipping costs on Earth, which are only...
Deep atheism and AI risk
by Joe Carlsmith — 2024-01-04T18:58:47.745Z

(Cross-posted from my website. Audio version here, or search "Joe Carlsmith Audio" on your podcast app.

This essay is part of a series I'm calling "Otherness and control in the age of AGI." I'm hoping that the individual essays can be read fairly well on their own, but see here for a summary of the essays that have been released thus far, and for a bit more about the series as a whole.)

In my last essay, I talked about the possibility of "gentleness" towards various non-human Others – for example, animals, aliens, and AI
systems. But I also highlighted the possibility of "getting eaten," in
the way that Timothy Treadwell gets eaten by a bear in Herzog's Grizzly
Man: that is, eaten in the midst of an attempt at gentleness.

Herzog accuses Treadwell of failing to take seriously the "overwhelming
indifference of Nature." And I think we can see some of the discourse
about AI risk – and in particular, the strand that descends from the
rationalists, and from the writings of Eliezer Yudkowsky in particular
– as animated by an existential orientation similar to Herzog's: one
that approaches Nature (and also, bare intelligence) with a certain kind of fundamental mistrust. I call this orientation "deep atheism." This essay tries to point at it.

Baby-eaters

Recall, from my last essay, that dead bear cub, and its severed arm –
torn off, Herzog supposes, by a male bear seeking to stop a female from
lactating. The suffering of children has always been an especially vivid
objection to God's
benevolence.
Dostoyevsky's Ivan, famously, refuses heaven in
protest.
And see also, the theologian David Bentley
Hart: "In those
five-minute patches here and there when I lose faith ... it's the
suffering of children that occasions it, and that alone."

Yudkowsky has his own version: "baby-eaters." Thus, he ridicules the
wishful thinking of the "group
selectionists,"
who predicted/hoped that predator populations would evolve an instinct
to restrain their breeding in order to conserve the supply of prey. Not
only does such sustainability-vibed behavior not occur in Nature, he
says, but when the biologist Michael Wade artificially selected beetles
for low-population groups, "the adults," says Yudkowsky, "adapted to
cannibalize eggs and larvae, especially female larvae." (Though: this
isn't actually a result I see in the paper Yudkowsky
cites
– more in footnote.[1])

Indeed, Yudkowsky made baby-ea...
Some Vacation Photos
by johnswentworth — 2024-01-04T17:15:01.187Z

In the past three months, I've flown overnight from San Francisco to the UK/Europe three times.

Three times, I got a window seat on the left side of the plane.

And three times... well, see for yourself.

First trip

Second trip

Third trip

Key thing to know: these aurora did not look like this to the naked eye. They were usually too faint to see the color at all; they looked like faint moonlight shining around the edges of a dark cloud. Except the "cloud" was higher up than an airplane flying at 10+ km.

It wasn't apparent what they were until I blocked out light from the cabin and took a long-exposure photo. (On my phone, that just means using the "night sight" mode.) That kinda makes these aurora cooler, in a way - you won't really notice there's anything interesting there unless you know to look for it and then image it in a special way.