trentmkelly/LessWrong-43k huggingface dataset:
Pivotal acts using an unaligned AGI?
by Simon Fischer — 2022-08-21T17:13:55.830Z

Introduction

A recent post (

AGI Ruin: A List of Lethalities

) gives an argument for humanity’s likely inability to survive the invention of superintelligent AGI. The argument goes very roughly like this:

We haven’t solved the alignment problem (citation needed?).
Given access only to an unaligned AI, we can’t commit a

pivotal act

that prevents others from building unaligned AI.
Thus, sooner or later somebody will actually run an unaligned AI and destroy humanity.

My goal in this post is to suggest an approach to using unaligned AGI to commit pivotal acts and thus invalidate the argument outlined above.[1] The general idea is that a boxed AI might be safely used to solve mathematically specified problems by using interactive proofs.

Boxed oracle AIs and interactive proofs

An oracle is an AI system “designed to safely answer questions, and only to answer questions”[2]. It is boxed if it has only “extremely restricted manipulable channels of causal interaction with the outside universe”,[3] e.g. it’s only able to output a limited amount of text and not otherwise influence anything.

A possible approach to ensure the safety of the output of such a system is to only allow output that is a provably correct answer to a formally specified question. In this way, the system can only output information that is already predetermined by the question or fail to output anything at all.[4]

One example of how such a system could be implemented is using a theorem-checker for formal proofs: We ask the AI to prove a formally specified mathematical theorem. The output of the AI is then only processed by the theorem-checker, which in turn outputs only one bit specifying whether it received a correct proof of the theorem and discards the proof itself.[5]

More generally, this approach can be applied to all problems in the

NP complexity class

.

This follows directly from the definition: A decision problem (a yes-no-question) is in NP if, in case the answer is positive, this can be confirmed efficiently (i.e. in polynomial time) using a so-called

certificate

(the analog of the proof in the theorem-provability case).

From now on, I will refer to this setup as a NP-Oracle AI.

Two important caveats:

If we don’t get a positive answer from the checker, we can’t determine if this is because the answer to the question is actually negative or i

...
My Plan to Build Aligned Superintelligence
by apollonianblues — 2022-08-21T13:16:32.697Z

2ish months ago, I realized making sure superintelligent AI doesn’t kill everyone was the biggest problem period. I resolved to understand and maybe help solve the problem. A friend of mine who does alignment research said that, in order to clarify my thinking, I should sit down and try to write my best solution to aligning a superintelligent AI in an hour. I did this, and found it extremely helpful; I then edited it and stared angrily at it for a few more hours to make it less embarrassing to post on Lesswrong. This is, so far, my best plan to solve AI alignment. I found writing this very helpful for clarifying my own model of alignment, and what I was still confused about. I would highly recommend other people starting to think about AI alignment try this exercise. Enjoy!

Problem Statement

To solve alignment and build an aligned artificial general intelligence, you’d need a smarter-than-human system that does what you want it to do, you’d need some confidence that it will do this thing before you hand it a machete and say “sure, do whatever,” and you’d also probably need it to change its behavior in response to you telling it to not do something, telling you what to do, etc., but in a way that represents what you’re actually asking it to do, not some proxy that it can mess with. You’d also need to do this before anyone makes an unaligned AGI, and it would have to prevent new people from making unaligned AGIs.

To do this, you want an AGI that understands what you actually want to happen in the real physical world, is incentivized to model this instead of optimizing on one specific criterion like “I see a video of the thing that maximizes my reward,” and will change its behavior in response to you indicating you want it to do something else. It seems like this kind of system would need to be actively trying to do something similar to how we “do science” to figure out about the world. Instead of optimizing one specific criterion, which it could game, like n of people who press an “I’m happy” button or smile in front of a camera, it would understand there’s some sort of underlying process, that specific details are proxies for, that it’s trying to model.

We sometimes take this quality for granted because humans actually care about things in the real world, since our training environment required us to locate and consume real food with specific nutrients so w...
Embracing the Opposition's Point
by Yulia — 2022-08-21T09:51:27.361Z
Link: https://yuliaverse.substack.com/p/agreement-extent

Part of why I like making fun of debate bros is because I used to be one. It’s embarrassing to admit – but here we are. The tension, the adrenalin rush, and the righteous wrath felt invigorating. “Winning” an argument was satisfying. I have worked hard on changing that for the last couple of years. And while I have improved, there is still have a long way to go. So I want to establish a script that can remind me to keep it productive.

There are different approaches to turning disagreements into productive conversations. Double crux, for example, is a method developed by CFAR to foster mutual understanding. Here, I want to delve into something that can be integrated into the double crux, an internal script that helped me to cool off a few times. Its core is the following question – what’s the extent to which I can agree with a statement that I mostly reject?

My first instinct when I hear a statement I even vaguely disagree with is to point out its flaws, question its assumptions and provide counterexamples. That can be useful and should be part of the conversation. However, the way I bring my criticism across can seem aggressive. In the past, it has put even mellow, agreeable people on the defensive. I like to engage with arguments destructively. What am I skipping when I go in with the sledgehammer immediately? Acknowledging the valuable parts of my opponent’s argument.

If you want to disagree productively, you should probably find out to what extent you agree with an idea you don't fully share. Can you point out with what parts and assumptions you agree? Can you state what you find plausible and focus on that for a dedicated period? In other words, can you work out with what version of the argument you agree?

Why might this be helpful? Firstly, the person you disagree with might feel more understood, which can foster an atmosphere of cooperation. Secondly, your adrenalin and aggression levels might decrease. The person in front of you might suddenly seem less like an alien with whom you share no commonalities (how prevailing is that problem? Is it just a me-thing?). Usually, people (especially people with the same cultural background) differ in the degrees to which they hold things true.

Let’s look at an example. The other day, I listened to a conversation between Dr. Gorden Lobsterson and the Freedom Report Man. Some part...
How evolution succeeds and fails at value alignment
by Ocracoke — 2022-08-21T07:14:50.559Z

Disclaimer: I don't have a background in alignment research or reinforcement learning, and I don't think any of the ideas discussed here are new, but they might be interesting to some.

A recent post suggested that humans provide an untapped wealth of evidence about alignment. I strongly agree with that and I found it interesting to think about how nature can ensure that a mother's values are aligned with the well-being of her children.

One particular thing this made me realize is that there are two very different reasons for why an agentic AGI might behave in a way that we would characterize as misaligned:

1. The reward function was poorly specified, and the model didn't learn the values we wanted it to learn
2. The model did learn the values we wanted it to learn, but those values led it to conclusions and actions that we did not anticipate

Let's go back to thinking about how evolution aligns mothers with their children. If we could get an AGI to love all humans like most (mammalian) mothers love their children, it might not necessarily solve the alignment problem, but it would be far better than having an AGI which doesn't care much about humans.
We can think of humans as reinforcement learning agents with a range of different goals, motivations and desires that are in one way or another all instrumental for the purpose of survival and reproduction in an environment similar to the one we may have found ourselves in 10,000 years ago. Some of those goals represent pretty simple values like avoiding pain or maintaining a certain blood sugar level, but others represent more complex values, like ensuring the well-being of our children. The biological link between pain or food-intake and a reward signal like dopamine can be quite simple, but how is something like "care for your children" encoded as a value in a way that generalizes to out-of-distribution environments?

Two examples for misaligned AGIs that were discussed around here are a system that is supposed to prevent a diamond from being stolen, but that can be tricked by placing an image of the diamond in front of the security camera, and a strawberry-picking-robot that was supposed to learn to put strawberries in a bucket, but instead learned to fling red things at light sources.
Alignment failures of that sort do occur in nature. In some birds, simple visual and auditory cues trigger the behavior of feeding chic...
A First Attempt to Dissolve "Is Consciousness Reducible?"
by DragonGod — 2022-08-20T23:39:35.433Z

Introduction

I recently made an attempt to dissolve the question: "is consciousness reducible"?

(This was borne out of a discussion with Rob Bensinger on Twitter.)

I've repackaged the thread for LessWrong.

Concepts Hierarchy

I see three relevant/particularly pertinent levels of abstraction:

A. Mental state

B. Computational/Informational state

C. Physical state
I see three relevant questions we can then ask:

Is A reducible to B?
Is B reducible to C?
Is A reducible to C?

Idea Space

I would like to posit another abstraction, a domain where ideas and abstractions live.
It's in this domain that the axioms of set theory and Peano arithmetic lie.
(Where "2 + 2 = 4" is defined and true.)

I think it's in this domain, that knowledge/semantic content lives. Where the sentence: "Paris is the capital of France" is defined and true.

Some Assumptions

Potential assumptions that one might make.

Semantic Qualia Assumption (SQA)

"Mental state" refers to the "knowledge state"/"semantic content" of a mind at a particular location in spacetime.

The "qualia" that a mind experiences, resides in idea space.

I do not claim that this assumption is true, I merely explicate it so that we may consider the implications if the assumption were true.

Taboo "Reducible"

I propose we taboo the word "reducible".

It seems intuitive to me, to replace, "is X reducible to Y" with "does Y uniquely characterise [1] X?".

(I think this adequately captures what is being debated over when we argue that consciousness is "reducible".)
(This may not be how to dissolve "reducible" in general, but it feels appropriate for this abstraction hierarchy.)

Answering the Questions

#2

Does physical state "uniquely characterise" informational/computational state?
We can reinterpret this as the inverse of:

Can the same physical state represent distinct informational/computational states?

I think the answer to the above is "yes". Consider the simple computation "2 + 2 = 4".

We might represent this computation by placing a group of two peas and another group of two peas together to form a new group.

If each pea represents:

The number "1", then we have computed: 2 + 2 = 4
The string "go", then we have computed: ("go", "go") + ("go", "go") = ("go", "go", "go", "go")

Computation is frame and interpretation dependent. The semantic content of a given computational/informational state is indeterminate. We need to know the refe...
What are the Limits on Computability?
by DragonGod — 2022-08-20T22:02:39.695Z

Question

The halting problem and Rice's theorem (non-trivial semantic properties of algorithms are not decidable) seem to be problems of self-reference. Some self-referential properties of programs cannot be computed. And this seems to be true even for more powerful models of computation (hypercomputers have their own version of the halting problem).

Are there any limits on computability that don't arise from self-reference?

Context

(Situating my world model and where the question sits within it.)

I have an intuition that:

A function (relation, map, transformation, morphism, etc. Basically, whatever thingy you are interested in within your abstract world [sets, (various kinds of "sets with additional structure"), categories, etc.])
An "algorithm" that "computes" that thingy

I'm not defining "compute" here.

There's an intuitive notion, and I'm worried that trying to explicate it before I fully understand it would lose some nuance.

I imagine that an "algorithm" is a (finite?) sequence of steps (the steps may be functions, operations, etc. (primitive?) thingies)

A physical system that implements the algorithm (a "computer")

Are all different things.

I'm trying to build up intuition for how they relate to each other.

I think a "model of computation" is an abstraction of a "general computer" within a given "laws of physics"?

(There may be multiple "models of computation" for a particular laws of physics, but I expect each laws of physics would have a class of "most powerful models of computation" within that laws of physics.)

A "laws of physics" is something like a "fundamental ontology of a (logically?) possible universe".

I feel like first order logic uncomputability challenges or real number uncomputability challenges are only uncomputable within particular models of computation/laws of physics.

If you could exploit countable infinities, I think you would be able to compute stuff about first order logic (maybe Godel's incompleteness theorems says otherwise, but I don't understand the incompleteness theorems [yet hopefully] and don't understand how "what can be proven" relates to "what can be computed". I think for some models of computation, the two may be indistinguishable, but I'm not sure if this is necessarily true.)

If you could exploit uncountable infinities, I'd expect you'd be able to compute stuff about real numbers.

I'm trying to think about not ...
Review: Amusing Ourselves to Death
by L Rudolf L — 2022-08-20T21:13:47.023Z
Link: https://www.strataoftheworld.com/2019/08/review-amusing-ourselves-to-death.html

This is a linkpost that I'm posting in response to a request to share more of the archive of my personal blog on LessWrong.

At first glance, Amusing Ourselves to Death seems neither interesting nor relevant. It is a short book, published in 1985, about the effect of television on public discourse in United States. However, take a second glance.
In his oft-quoted foreword, Postman writes:

Contrary to common belief even among the educated, Huxley and Orwell did not prophesy the same thing. Orwell warns that we will be overcome by an externally imposed oppression. But in Huxley’s vision, no Big Brother is required to deprive people of their autonomy, maturity and history. As he saw it, people will come to love their oppression, to adore the technologies that undo their capacities to think.

What Orwell feared were those who would ban books. What Huxley feared was that there would be no reason to ban a book, for there would be no one who wanted to read one. Orwell feared those who would deprive us of information. Huxley feared those who would give us so much that we would be reduced to passivity and egoism. Orwell feared that the truth would be concealed from us. Huxley feared that the truth would be drowned in a sea of irrelevance. Orwell feared we would become a captive culture. Huxley feared we would become a trivial culture […].

[…]

This book is about the possibility that Huxley, not Orwell, was right.

I have not read Brave New World, so I cannot vouch for the accuracy of this comparison. But, at least in Postman’s telling, Huxley’s fears seem more relevant than ever.
Countless bookshelves, newspaper stands, and – these days – AWS servers groan under the weight of critiques about the effects of various information technologies on society. Ironically, these critiques themselves are often shallow, hackneyed, and repetitive, and consequently they drown each other out in a sea of irrelevance much like the one they so doggedly warn us of.
This is not to say that there is no problem. Television, YouTube, and social media, while all useful technologies, have also distorted parts of daily life and public discourse for the worse. However, thoughtful discussion of the effects of new mediums on society is rare (some of Cal Newport’s writings come to mind, though they focus exclusively on internet’s impact on individual’s lives).
Enter Neil Postman, with his topical yet...
Paper is published! 100,000 lumens to treat seasonal affective disorder
by Fabienne — 2022-08-20T19:48:29.687Z
Link: https://www.lesswrong.com/posts/qEweugBipR5P2cMyK/preprint-is-out-100-000-lumens-to-treat-seasonal-affective

Let’s give people with winter depression (seasonal affective disorder, SAD) LOTS OF LIGHT and see what happens!

This is an update to my previous post that said that our preprint is out. :) The paper was accepted with very minor changes, so you might already know what it says if you saw the first post.

Here is the link to the paper: https://onlinelibrary.wiley.com/doi/10.1002/da.23281

Here is the twitter thread explaining the paper: https://twitter.com/FabienneSand/status/1561060644803252224

Again, Jan Brauner and I are very thankful to the LessWrong/EA communities, which have inspired this first study (there will be more) and through which we have found funding. In particular, thank you Eliezer Yudkowsky for helping us find funding and for inspiring the study with Inadequate Equilibria, David Chapman for inspiring us with

these

two

posts in the Meaningness blog, Raemon for inspiring us with

this

LessWrong post and everyone who discussed with us setups they have tried. <3
What's the Least Impressive Thing GPT-4 Won't be Able to Do
by Algon — 2022-08-20T19:48:14.811Z

It seems like GPT-4 is going to be coming out soon and, so I've heard, it will be awesome. Now, we don't know anything about its architecture or its size or how it was trained. If it were only trained on text (about 3.2 T tokens) in an optimal manner, then it would be about 2.5X the size of Chinchilla i.e. the size of GPT-3. So to be larger than GPT-3, it would need to be multi-modal, which could present some interesting capabilities.

So it is time to ask that question again: what's the least impressive thing that GPT-4 won't be able to do? State your assumptions to be clear i.e. a text and image generating GPT-4 in the style of X with size Y can't do Z.
Are you allocated optimally in your own estimation?
by Emrik — 2022-08-20T19:46:54.589Z

I worry a lot about the inefficiencies created by preventing competent researchers from allocating their time according to their own judgment of what is optimal. I think researchers can be several times more effective with financial freedom. There's a steep cost paid by forcing researchers to compromise between optimising for what they care about and what their money cares about. Value of information is relative to the brain computing it, so competent people will usually best be able to estimate it on their own, even if the results are illegible to everybody else. They'll find their path much faster if they can search through and abandon projects speedily, and don't have to have their attention toted about by anyone or anything other than themselves--as if the harness isn't going to chafe, as if anyone knows their own use better than they do.

But enough of my motivation. Do you feel like you are working on the most important thing you could work on? And if not, are you bottlenecked by being paid to do a different thing? Even if you are allocated optimally atm, how easy would it be for you to switch paths right now if that's what you wanted? How costly would it be for you personally to just abandon your current project and hop to a new one if you thought it was a better use of your time?

I'm trying to get a sense of how much self-judged inefficiency is caused by a lack of financial slack among LW/EA-reading researchers. Feel free to answer here or admonymously (you don't need to share your name, but if you do, I promise not to share it on).
Broad Picture of Human Values
by Thane Ruthenis — 2022-08-20T19:42:20.158Z

I have two strong intuitions about human values that'd seemed utterly irreconcilable to me, up until recently.

On the one hand, human values are clearly inchoate, unstable messes of niche heuristics and preferences that often contradict each other and can change on a dime. A twenty-years-old doesn't value all the same things their fifty-years-old self will value, modern Europeans don't value the very same things as ancient Egyptians, etc. And how can it be otherwise? Evolution-built bodies are messy hacks. Why would evolution-built minds be any different?
On the other hand, my models of human psyche tell me that humans are clearly approximate utility-maximizers for some very specific utility function. This utility function is stable across time and lifetimes and cultures, despite the fact that the object-level behaviors humans engage in and their stated preferences can change arbitrarily. This utility function has something to do with human "flourishing", whatever that is, and with things that feel a very special kind of "right" according to a human's judgement. There is a strong sense in which we all want the same thing, on some deeply abstract level.

I believe I see a way to unify the two. The crucial insights have been supplied by the Shard Theory, and the final speculative picture is broadly supported by it. This result mostly dissolved all of my high-level confusions about human values and goal-directed behavior, in addition to satisfying a lot of other desiderata.

1. The Shard Theory of Human Value: A Recap

Disclaimer: This summary does not represent the views of Team Shard, but only my subjective understanding. For the official summary, see this.

According to the Shard Theory, in the course of brain development, humans jointly learn two things:

A world-model.
Heuristics attached to that world-model, reinforced by the credit-assignment algorithm because their execution historically led to reward.

The latter are "shards" . In their most primitive form, they're just observation→action activation patterns. You see a lollipop enter your field of vision, you grab it. You see a flashlight pointed at your face, you close your eyes.

As the world-models grows more advanced, the shards could grow more sophisticated as well. Instead of only attaching to observations, they can attach to things in the world-model. If you're modelling the world as containing a lolli...