trentmkelly/LessWrong-43k huggingface dataset:
What qualities does an AGI need to have to realize the risk of false vacuum, without hardcoding physics theories into it?
by RationalSieve — 2023-02-03T16:00:43.198Z

For this question, let's assume the worst about the universe:
1. That a false vacuum decay can be created,
2. False vacuum decay will destroy the universe, i. e. inside the true vacuum bubble life/computation will be impossible.
3. Faster-than-light travel is impossible, i. e. it's not possible to outrun the bubble of true vacuum.

The AGI isn't aware of the above. What qualities does it need to have, to come to the realization that it's dangerous?
I think that an aligned AGI may be more dangerous in this case, depending on how it's aligned, since it may be restricted in the methods it can use to prevent the rest of the universe creating a false vacuum decay.
Since a false vacuum decay will also destroy the AGI, not just humanity, even an unaligned AGI will view false vacuum as a risk to itself.
However, if AGI doesn't care about its own existence, it will not care about the dangers of false vacuum as well.

But what if AGI (following Occam's Razor) creates a simpler theory of physics that agrees with what can be observed in the universe, but without the possibility of false vacuum decay, and it turns out to be wrong?
This dilemma is different from Pascal's Wager, in the sense that false vacuum decay is a scientific hypothesis, not a religious belief.

I struggle to think of how an AGI can be programmed to tread carefully with false vacuum research, without hardcoding it to reject physics theories that allow no possibility for false vacuum.
Housing and Transit Roundup #3
by Zvi — 2023-02-03T15:10:00.692Z

Previously: #1, #2

In honor of much of northern California coming face to face with the Builder’s Remedy, I’m doing this first, ahead of the monthly general roundup. Here we go.

The Federal Option

Biden announces he will hold federal housing funding hostage to the filing of an ‘equity plan.’ It seems he thinks it is within the Executive’s power to do this, requiring all locations to state their ‘fair housing issues’ and what they are going to ‘do about them,’ to ensure desegregation. Enforcement is expected to be, let’s say, tricky.

This sounds to me very much like a claim that the Executive can impose arbitrary requirements on localities if they want to keep these federal funds, or at least arbitrary housing-related requirements that are plausible government interests. So may I suggest we try the requirement of building more housing where people want to live?

The California Option: Enter The Builder’s Remedy

One place they want to live, and can’t because there isn’t enough housing, is California. One way California is trying to change that is through imposing equity requirements cities hate, having cities fail to comply, then triggering the Builder’s Remedy.

This suspends all local zoning laws as long as 20% of units are misallocated or intentionally crippled (officially called ‘affordable’). With one day left to go at the end of last month, most of the Bay Area (although not SF proper) was slated to fail, potentially allowing developers to shoehorn in new construction the way they did in Santa Monica. Cities can get their zoning laws back by complying, but anything approved in the meantime will stay approved.

There was at least some window in quite a lot of the Bay:

Here’s the same thing in list form.

What struck me most about Santa Monica was that there wasn’t way more new construction approved on the first day. There is huge pent up additional demand for places like Santa Monica, or now Berkeley. Building a lot more housing would drive down rents somewhat, but not to anything like reasonable levels unless it was truly massive. If it was truly massive, and suddenly there were hundred of thousands of new apartments going up, then as long as city services were able to scale up to match rather than intentionally sabotaged the result would be a new city and plenty of wealthy real estate developers. It likely would even benefit current proper...
Taboo P(doom)
by  — 2023-02-03T10:37:00.596Z

I think it's actively unhelpful to talk about P(doom) because it fails to distinguish between literally everyone dying and humanity failing to capture 99.99% of the value of the future under a total utilitarian view but in practice, everyone who's born lives a very good life. These are very different outcomes and it's unhelpful not to distinguish between them and everything else in that spectrum.

This is especially the case since astronomical waste arguments really only bite for total utilitarian views. Under moral views where potential people not coming into existence is more similar to preventing someone from living an extraordinarily happy life rather than a merely happy life, as opposed to preventing someone coming into existence being similar to murder, it's quite reasonable to prioritise other goals well above preventing astronomical waste. Under these non-totalalist views preventing totalitarian lock-in or S-risks might look much more important than ensuring we don't create 10^(very large number) of happy lives.

I think this also matters on a practical level when talking about threat models of AI risks. Two people could have the same p(doom) but one is talking about humans being stripped for their atoms and the other is talking about slow disempowerment in which no one actually dies and everyone, in fact, could be living very good lives but humanity isn't able to capture almost all of the value of future from a total utilitarian perspective. These plausibly require different interventions to stop them from happening.

It also seems like one's prior on humanity going extinct as a result of AI should be quite different from disempowerment, but people often talk about what their prior on P(doom) should be as a univariate probability distribution.
ChatGPT: Tantalizing afterthoughts in search of story trajectories [induction heads] 
by Bill Benzon — 2023-02-03T10:35:12.110Z

Several times during my undergraduate years I would finish a paper, turn it in, the then Wham! it hit me. That should have been my theme. That is to say, I would get an idea and explore it in a paper, but it was only after I declared the paper finished that I could see what I’d been driving at all along.

That’s what has happened to me with my latest working paper, ChatGPT intimates a tantalizing future; its core LLM is organized on multiple levels; and it has broken the idea of thinking. I uploaded it on January 23, and then a day or two later, I decided I had to add a section. I uploaded the new Version 2 a couple of days ago (Jan. 28, 2023). I got up the next morning, took a look at the technical literature and decided I need a Version 3. But I’ve not yet done the revision. I want to think it through a bit more. That’s what this post is for.

First off I review the appropriated section in the working paper (section “5: What underlying structure drives ChatGPT’s behavior?”), then I explain why “story grammar” isn’t the right idea. That gives way to accounts of story trajectories (the new idea) and coherence. I end by introducing the idea of induction heads from some recent research by investigators at Anthropic.

Syntax and discourse on different levels

Here’s the section that is new for Version 2:

Since I am in full speculative mode, I will further assert that sentence-level syntax and semantics are implemented directly in the neural net. Story grammars, my primary example, are implemented in a different and somewhat mysterious way. To return to the somewhat clunky analogy of computer languages, sentence-level syntax is analogous to a high-level language, such as C++, while story grammers are analogous to end-user programs, such a word processors or databases.

My assertion about sentence-level syntax is based on work published by Christopher D. Manning et al, in 2020. They investigated syntactic structures represented in BERT (Bidirectional Encoder Representations from Transformers). They discovered that neural networks induce

representations of sentence structure which capture many of the notions of linguistics, including word classes (parts of speech), syntactic structure (grammatical relations or dependencies), and coreference (which mentions of an entity refer to the same entity, such as, e.g., when “she” refers back to “Rachel”). [...] Indeed, the learned enco

...
What is the risk of asking a counterfactual oracle a question that already had its answer erased?
by Chris_Leong — 2023-02-03T03:13:10.508Z

In case you haven't heard of them before, here's an explanation of Counterfactual Oracles. If you erase the answer, you're not supposed to ask the same question to another oracle.

My first assumption was that this could lead to self-fulfilling prophecies such as where you fail because the oracle tells you fail so you lose motivation. However, I don't see how this would happen if we asked the question twice.

Let's assume you'll succeed unless you're told that you will fail. The second counterfactual oracle is evaluated based on the case where both counterfactual oracles fail, so it'll tell you that you'll succeed and then you'll actually succeed because you dodged the self-fulfilling prophecy. The first counterfactual oracle will be evaluated by how well it matches the weighted average of the case where neither oracle answers and where the first oracle answers. In either case, you succeed, so it'll tell you that you succeed and again you'll dodge the self-fulfilling prophecy.

So given this, can anyone explain how this could go wrong?

I'll note that it's not clear why you would use this technique. If you were to use it infinitely, then there would arguably be no probability where none of the oracles gives you an answer which could ground the process (I'll concede that some people might argue this technique would work for infinitesimal probabilities). If you use it a finite number of times, then you could have just adjusted the probability of the original oracle engaging in erasure instead.
I don't think MIRI "gave up"
by Raemon — 2023-02-03T00:26:07.552Z

Disclaimer: I haven't run this by Nate or Eliezer, if they think it mischaracterizes them, whoops.

I have seen many people assume MIRI (or Eliezer) ((or Nate?)) has "given up", and been telling other people to give up.

On one hand: if you're only half paying attention, and skimmed a post called "Death With Dignity", I think this is a kinda reasonable impression to have ended up with. I largely blame Eliezer for choosing a phrase which returns "support for assisted suicide" when you google it.

But, I think if you read the post in detail, it's not at all an accurate summary of what happened, and I've heard people say this who I feel like should have read the post closely enough to know better.

Eliezer and "Death With Dignity"

Q1: Does 'dying with dignity' in this context mean accepting the certainty of your death, and not childishly regretting that or trying to fight a hopeless battle?

Don't be ridiculous. How would that increase the log odds of Earth's survival?

The whole point of the post is to be a psychological framework for actually doing useful work that increases humanity's long log odds of survival. "Giving Up" clearly doesn't do that.

[left in typo that was Too Real]

[edited to add] Eliezer does go on to say:

That said, I fought hardest while it looked like we were in the more sloped region of the logistic success curve, when our survival probability seemed more around the 50% range; I borrowed against my future to do that, and burned myself out to some degree. That was a deliberate choice, which I don't regret now; it was worth trying, I would not have wanted to die having not tried, I would not have wanted Earth to die without anyone having tried. But yeah, I am taking some time partways off, and trying a little less hard, now. I've earned a lot of dignity already; and if the world is ending anyways and I can't stop it, I can afford to be a little kind to myself about that.

I agree this means he's "trying less heroically hard." Someone in the comments says "this seems like he's giving up a little". And... I dunno, maybe. I think I might specifically say "giving up somewhat at trying heroically hard". But I think if you round "give up a little at " to "give up" you're... saying an obvious falsehood? (I want to say "you're lying", but as previously discussed, sigh, no, it's not lying, but, I think your cognitive process is fucked up...
What fact that you know is true but most people aren't ready to accept it?
by lorepieri — 2023-02-03T00:06:42.460Z

Understanding and updating beliefs on deeply engrained topics can take enormous efforts, but sometimes it can be so hard that the listener cannot even in principle accept the new reality. The listener is simply not ready, he lacks a vast background of reasoning leading to the new understanding.

What fact that you know is true but most people aren't ready to accept it?

By "you know is true" I really mean "you are very confident to be true".

Feel free to use a dummy account.
Monotonous Work
by Gideon Bauer — 2023-02-02T21:35:19.710Z

In the process of running an organization, you have to break down the things that have to be done into tasks, and delegate them to the people available to you. You're in a better position if most of your tasks take non-variable work to complete, because they can be delegated easily. For example, a SAHM can easily hand off sweeping, dusting, etc, but can't hand off things like buying groceries, something that's more variable and that requires decisions.

As a volunteer, the easiest way to give work would be "sweeping," not "grocery shopping." What are some of those tasks? Also, is there a way to systematically convert tasks from one type to another?
Is AI risk assessment too anthropocentric?
by Craig Mattson — 2023-02-02T21:34:57.515Z

Hi everyone,

I've recently discovered lesswrong and love it! So first, let me thank you all for fostering such a wonderful community.

I've been reading a lot of the AI material and I find myself asking a question you all have surely considered, so I wanted to pose it.

If I believe that human beings are evolutionarily descended from apes, and I ask myself whether apes -- if they had control over allowing human evolution to happen -- should have allowed it or stopped it, I'm honestly not sure what the answer should be.

It seems like apes would in all likelihood be better off without humans around, so from the perspective of apes, they should probably have not allowed it to happen. However, looked at from a different frame of reference, like maybe what is good for the earth, or for the universe, then maybe the evolution of humans from apes was a good thing. Certainly from the perspective of humans, most of us would believe that it was allowed to happen was a good thing.

Do we find ourselves in a similar scenario with humans and AI? Are there benefits from other frames of reference besides humanity to allow the development of AI, even if that AI may pose existential threats to human civilization? And if so, are those perspectives being taken into full enough account when we think about AI risk assessment?
Conditioning Predictive Models: Outer alignment via careful conditioning
by evhub — 2023-02-02T20:28:58.955Z

This is the second of seven posts in the Conditioning Predictive Models Sequence based on the paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper.

2. Outer alignment via careful conditioning

Suppose we actually get a predictive model of the world that we can condition on arbitrary observations. What should we do with it?

One thing that is often done with large language models is to ask them to act as dialogue agents (e.g. predict what comes next after an “AI:” prompt). As we’ll discuss extensively in this section, however, we think that asking a predictive model to predict itself or another AI system is highly unsafe, as the AI systems being predicted may not themselves be safe. Instead, our basic proposal is to try to predict the most useful possible human outputs.

One canonical thing that you might want to do in that vein is “predict the results of humans doing AI safety research” (though note that this is just an example). Specifically, here’s GPT-3.5 trained with RLHF (text-davinci-003) at zero temperature (prompt is bolded):[1]

# A full solution to the AI alignment problem

Unfortunately, there is no single solution to the AI alignment problem. AI alignment is a complex and multifaceted problem that requires a comprehensive approach.

The first step in solving the AI alignment problem is to define the goals of the AI system. This involves understanding the desired outcomes of the AI system and how it should interact with its environment. Once the goals are defined, the next step is to develop a framework for how the AI system should behave in order to achieve those goals. This framework should include ethical considerations, such as ensuring that the AI system does not cause harm to humans or other sentient beings.

Unfortunately, as can be seen from GPT-3.5’s continuation, it doesn’t actually think a paper titled “A full solution to the alignment problem” will actually contain such a solution—instead, it seems to think it’s far more likely that it’ll just talk about what such a solution might entail. Which in fact seems true, at least if such a paper was written today. Fortunately, at least for current InstructGPT models, it’s not that hard to get GPT-3.5 to at least seemingly give a real att...
Conditioning Predictive Models: Large language models as predictors
by evhub — 2023-02-02T20:28:46.612Z

This is the first of seven posts in the Conditioning Predictive Models Sequence based on the paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper.

Thanks to Paul Christiano, Kyle McDonell, Laria Reynolds, Collin Burns, Rohin Shah, Ethan Perez, Nicholas Schiefer, Sam Marks, William Saunders, Evan R. Murphy, Paul Colognese, Tamera Lanham, Arun Jose, Ramana Kumar, Thomas Woodside, Abram Demski, Jared Kaplan, Beth Barnes, Danny Hernandez, Amanda Askell, Robert Krzyzanowski, and Andrei Alexandru for useful conversations, comments, and feedback.

Abstract

Our intention is to provide a definitive reference on what it would take to safely make use of generative/predictive models in the absence of a solution to the Eliciting Latent Knowledge problem.

Furthermore, we believe that large language models can be understood as such predictive models of the world, and that such a conceptualization raises significant opportunities for their safe yet powerful use via carefully conditioning them to predict desirable outputs.

Unfortunately, such approaches also raise a variety of potentially fatal safety problems, particularly surrounding situations where predictive models predict the output of other AI systems, potentially unbeknownst to us. There are numerous potential solutions to such problems, however, primarily via carefully conditioning models to predict the things we want—e.g. humans—rather than the things we don’t—e.g. malign AIs.

Furthermore, due to the simplicity of the prediction objective, we believe that predictive models present the easiest inner alignment problem that we are aware of.

As a result, we think that conditioning approaches for predictive models represent the safest known way of eliciting human-level and slightly superhuman capabilities from large language models and other similar future models.

1. Large language models as predictors

Suppose you have a very advanced, powerful large language model (LLM) generated via self-supervised pre-training. It’s clearly capable of solving complex tasks when prompted or fine-tuned in the right way—it can write code as well as a human, produce human-level summaries, write news articles, etc.—but we don’t know what it is actually doing internally that produc...
Normative vs Descriptive Models of Agency
by mattmacdermott — 2023-02-02T20:28:28.701Z

Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort.

I think there's an important distinction to be made between work in agent foundations which is concerned with normative models, and work which is concerned with descriptive models. They are increasingly separate bodies of work, with different aims and different theories of change when it comes to alignment.

Examples

The normative branch is typified by the Embedded Agency sequence, and the whole thing can be summed up as 'The Hunt for Embedded AIXI'. Its goal is to figure out how to build an ideal agent in principle. Decision theory, infrabayesianism, and logical induction all come under the normative banner.

The descriptive branch is typified by John Wentworth's Basic Foundations for Agent Models sequence. Descriptive work aims to understand the agents we run into in the wild. Other examples include shard theory, Critch's Boundaries sequence, and the Discovering Agents paper.

Theories of Change

Descriptive

I'll start with the descriptive branch. The most ambitious version of its goal is to understand agency so well that in principle we could take an unabstracted, non-agentic description of a system - e.g. a physics-level causal graph, the weights in a neural network, or a cellular model of a squirrel - and identify what if any are its goals, world-model, and so on. If we could do that in principle, then in practice we could probably check whether an artificial agent is aligned, and maybe we could even do things like surgically modify its goals, or directly point to things we care about in its world-model. I think that's what John is aiming for. A less ambitious goal, which I think better describes the aims of shard theory, is to understand agency well enough that we can carefully guide the formation of agents' goals during ML training runs.

Beyond that, I think everyone involved expects that descriptive work could lead to foundational insights that change our minds about which alignment strategies are most promising. In particular, these insights might answer questions like: whether intelligent entities are inevitably agents, whether agents are inevitably consequentialists, whether corrigibility is a thing, and whether we should expect to encounter sharp left turns.

Normative

The normative branch shares the conceptual clarification theory of change. I think there's a reasonab...