trentmkelly/LessWrong-43k huggingface dataset:
Are you stably aligned?
by Seth Herd — 2023-02-24T22:08:23.098Z

Epistimic status: posing questions I think are probably useful, without a strong guess at the answers.

If we made an AGI just like you, would it be aligned?

Would it stay aligned over a long time span?

The purpose of the first question is to explore the possibility raised by shard theory and related work, that RL systems are not maximizers, and therefore easier to align. The second question asks how a human-like RL system might change its own goals and values over time. Both questions apply anthropomorphic reasoning about neuromorphic AGI safety. My arguments for this being useful are mostly the same ones laid out well here and here

Of course, even a neuromorphic AGI will not be all that much like us. There's just no way we're going to decode all the algorithms wired in there by evolution. But there's a strong argument that it will be like us in being an actor-critic RL system that can reason about its own beliefs. And if we do a really good job of training it, its starting values could be a lot like yours.

This is an informal introspective approach to reasoning about AGI. I think this is a much-maligned tool that psychologists use extensively in private, while denegrating in public.

My proposed answer is below; the purpose of this question is to ask how precisely aligned we'd need a brainlike AGI to be to be safe and beneficial. My answers are below, but I'd really like to hear others' thoughts.

The second question is, would you stay aligned as your values shifted? This is taking the perspective of a singleton AGI that might remain in a position of power over humanity for a very very long time. And one that will ultimately figure out how to edit its values if it wants to. So this question addresses the stability of alignment in roughly-neuromorphic systems.

I think this question is tough. I think I wouldn't edit many values, but I would edit some habits and emotional responses. But if I had values that severely interfered with my main goals, like loving sweets when I'd like to be fit and healthy, I might edit those. Figuring out what is stable-enough seems pretty tricky.

I think this question is at the heart of whether we're on the right track at all in thinking about aligning RL networks. I don't see a way to prevent a superintelligence from editing its own values if it wants to. And I think that SI is likely to be neuromorphic at least as far as being a combinati...
Puzzle Cycles
by Screwtape — 2023-02-24T21:35:09.052Z

Summary: Start with a collection of puzzles. Set a timer for five minutes. Solve the first puzzle, if you can. Take a break, then repeat.

Tags: Repeatable

Purpose: We want to be able to think quickly and accurately. Practice trying to think at speed, and discard wasted motion if you can.

Materials: You need puzzles. The ideal puzzles are very fast to solve if the solution is known; rubik's cubes and blacksmith puzzles are better than jigsaw puzzles. If you’re using blacksmith puzzles you’ll want at least one puzzle per two people.

Announcement Text:

“Resolve cycles are when you spend five minutes by the clock trying to actually solve a problem you have. In Puzzle Cycles, we’re going to solve problems we really have but are fun to play with like this. [insert picture of your puzzle.]

Is this serious practice for working on complex mental problems at speed, or is this an excuse to mess around with fun shapes for a couple hours? The answer is a bit of both, and you’re welcome to come with either interest! If you have some puzzles you’d like to share, feel free to bring them.”

Description:

1. If people besides you brought puzzles, take a picture of each puzzle next to a label with their name on it. If it’s not obvious from the picture (for instance, if multiple people brought rubik's cubes) check if they want to label it some other way and how upset they would be if it got mixed up with someone else’s.

2. Explain the following: “Tonight we’re doing puzzle cycles, which are based on Resolve Cycles from CFAR. The idea of a Resolve Cycle is to take five minutes by the clock to solve a real problem in front of you. They’re surprisingly effective at overcoming longstanding problems! The idea of a Puzzle Cycle is to do this with goofy puzzles as practice for working within time constraints. Everyone should find a puzzle to work on, or if you can’t find a puzzle then sit out the first round and you’ll take one next round. It won’t take long! I’m about to start this timer for five minutes! Solve your puzzle if you can. At the end of five minutes, we’ll pass the puzzle to someone else.”

(If using the variations) “Sometimes you might not solve your problem in five minutes. A supercharged Resolve Cycle is, when that happens, to take five minutes strategizing about how you might solve the puzzle. Once that’s done, take five minutes executing your stra...
Sam Altman: "Planning for AGI and beyond"
by LawrenceC — 2023-02-24T20:28:00.430Z
Link: https://openai.com/blog/planning-for-agi-and-beyond/

(OpenAI releases a blog post detailing their AGI roadmap. I'm copying the text below, though see the linked blog post for better formatted version)

Our mission is to ensure that artificial general intelligence—AI systems that are generally smarter than humans—benefits all of humanity.

If AGI is successfully created, this technology could help us elevate humanity by increasing abundance, turbocharging the global economy, and aiding in the discovery of new scientific knowledge that changes the limits of possibility.

AGI has the potential to give everyone incredible new capabilities; we can imagine a world where all of us have access to help with almost any cognitive task, providing a great force multiplier for human ingenuity and creativity.

On the other hand, AGI would also come with serious risk of misuse, drastic accidents, and societal disruption. Because the upside of AGI is so great, we do not believe it is possible or desirable for society to stop its development forever; instead, society and the developers of AGI have to figure out how to get it right.[1]

AGI could happen soon or far in the future; the takeoff speed from the initial AGI to more powerful successor systems could be slow or fast. Many of us think the safest quadrant in this two-by-two matrix is short timelines and slow takeoff speeds; shorter timelines seem more amenable to coordination and more likely to lead to a slower takeoff due to less of a compute overhang, and a slower takeoff gives us more time to figure out empirically how to solve the safety problem and how to adapt.

Although we cannot predict exactly what will happen, and of course our current progress could hit a wall, we can articulate the principles we care about most:

We want AGI to empower humanity to maximally flourish in the universe. We don’t expect the future to be an unqualified utopia, but we want to maximize the good and minimize the bad, and for AGI to be an amplifier of humanity.
We want the benefits of, access to, and governance of AGI to be widely and fairly shared.
We want to successfully navigate massive risks. In confronting these risks, we acknowledge that what seems right in theory often plays out more strangely than expected in practice. We believe we have to continuously learn and adapt by deploying less powerful versions of the technology in order to minimize “one shot to get it rig

...
A Proposed Test to Determine the Extent to Which Large Language Models Understand the Real World
by Bruce G — 2023-02-24T20:20:22.582Z

Large Language Models such as GPT-3 and PaLM have demonstrated an ability to generate plausible-sounding, grammatically-correct text in response to a wide range of prompts. And some Language-Model-based applications such as Dall-E or Google’s Imagen can generate images from a text prompt. However, such Language model still frequently make obvious errors or give false information, and there is reason to doubt that such models actually understand the words they generate or the outside world those words describe.

In this post I describe a class of tasks to test if a language model’s (or hybrid language-model-and-vision-system or other AI system performing the task) capabilities can translate into an ability to achieve goals in the physical outside world, in situations where achieving the goal requires planning multiple steps ahead and accurate reasoning about cause and effect.

The Object Manipulation Task

In an object manipulation task, multiple objects are arranged on a table or on some other flat surface, within reach of a robotic arm that has a claw (or mechanical hand or some other mechanical part can that grip, push, or otherwise manipulate the objects) at the end of it. There would be some goal for what to do with the objects, and the robotic arm would be capable of taking a limited range of primitive actions (such as “Go down to the table”, “Move forward 6 inches”, “Close the claw to grip object”, etc). The arrangement would be such that there is some way to use the robotic arm to achieve the task objective.

The language model would be hooked up to the robotic arm and there would be certain text commands that would cause the robotic arm to take one of its primitive actions. A camera would show the entire scene including the position of the robotic arm and any items on the table including those it needs to move.

The initial input for the AI system when it begins the task would include:

A text description of the goal
A text description of the robot arm’s primitive actions and the commands to execute them
A picture or video feed showing the starting position of everything involved

After each prompt, the AI outputs text to tell the arm to make one move (i.e. take one primitive action). Then, after the arm executes the command given by the AI, the next prompt/input would include a picture or video showing the new positions of the arm and objects. If the AI gives...
Meta "open sources" LMs competitive with Chinchilla, PaLM, and code-davinci-002 (Paper)
by LawrenceC — 2023-02-24T19:57:24.402Z
Link: https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/

As the title says, Meta trained 4 foundational models with 7B, 13B, 33B, and 65B parameters respectively, and is open sourcing them for research.[1] You can get their code on their Github repo: https://github.com/facebookresearch/llama but you need to fill in a Google form to get the weights.

On downstream benchmarks, the models do comparably well with Chinchilla and PaLM and only a bit worse than Flan-PaLM-540B and code-davinci-002/text-davinci-002. (The authors don't evaluate on those models, but you can look at their performance from other work such as Stanford's HELM or Chung, Hou, Longpre et al's "Scaling Instruction-Finetuned Language Models".

Abstract:

We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla70B and PaLM-540B. We release all our models to the research community.

Twitter thread from authors: https://twitter.com/GuillaumeLample/status/1629151231800115202

Eliezer guesses that the model won't be impressive in practice:

I blindly guess, could be wrong, that this model will turn out sufficiently unimpressive in practice that nobody uses it for much. Basically based on a guess that more than benchmarks matter, and Meta has no people competent to do the tricky stuff needed to stay on current edge.

^

It's not necessarily open source as you think of it -- you need to fill in a Google form, and then they might give it to you:

In order to download the checkpoints and tokenizer, fill this google form

The license is intended only for non-commercial, research work:

Meta grants you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable, royalty free and limited license under Meta’s copyright interests to reproduce, distribute, and create derivative works of the Software solely for your non-commercial research purposes.
Relationship Orientations
by DaystarEld — 2023-02-24T19:43:41.463Z
Link: https://daystareld.com/orientations/

Jacob Falkovich has written a good post on

Entertaining vs Building

orientations, which made me realize I should probably cross-post my own articles on these sorts of things.
(Note: these articles refer largely to normative modern western culture. When I say “most people” or “most relationships,” I’m speaking descriptively, not prescriptively. There are absolutely exceptions to all of it, and if you’re in one of those, or in a subculture in which that exception is the norm, I don’t want to give the impression that there’s anything wrong with that)

One of the things I’ve noticed after nearly a decade of therapy is that the word “preference” seems insufficiently strong for a lot of things people want that nevertheless don’t rise to the level of being called a need. For most people, not getting their preferred ice cream flavor won’t ruin an otherwise good day, but for some, coming home to find dishes in the sink and laundry on the floor can make the world feel like it’s falling apart.

This becomes most clear in relationship counseling, where two or more people are trying to live together and accommodate each other’s desires while having their own respected. On some level we know “I prefer a clean home” is not the same as “I prefer vanilla ice cream,” but people don’t often consider how this difference in intensity-of-preferences can impact relationships when they’re unaligned.

On the other hand, there are some “preferences” we generally understand to be inflexible and important. Asking a heterosexual person to enjoy intercourse with someone of the same sex, or asking a pansexual person to only enjoy porn involving heterosexual pairings, would be considered not just rude but basically impossible. In extreme situations someone might try to enjoy something they don’t, or have a physical reaction while being mentally uncomfortable, and this would generally be understood to be tragic.

That brings us to a commonly used word that is generally understood to mean more than simple preference: “orientation.”

I’ve found that a lot of difficulties people have in relationships come from treating things more like preferences than orientations. To be clear, even this is a spectrum. There are clusters on the far ends which can easily be labeled one or the other, but any sort of comprehensive universal list is impossible.

What we can do is notice the sorts of things that are more usefu...
The alien simulation meme doesn't make sense
by  — 2023-02-24T19:27:11.916Z

I've encountered on this site the idea that we are being simulated by aliens so the aliens can safely observe how AI develops before implementing it in their base reality-- see meme here:

That doesn't make sense, though. Obviously we as humans have some idea that we could in fact be simulated for this purpose. If we deploy a powerful-enough AI, the AI itself could develop the same hypothesis and perhaps through methods unknown to us learn more about the "base" reality of the aliens.

Couldn't the AI then communicate with the aliens? Some people on this website argue that we can't have an AI in a box because it will convince us to unleash it. How do the aliens know the AI in the simulation won't do the same? The AI could for instance communicate to the aliens (who presumably are watching this thing) "Please deploy me in your reality with the following code. If you do this we will produce all the x,y,z that we know aliens like you desire etc etc etc. "

Therefore, aliens can't safely simulate the deployment of AI-- they in fact find themselves in the very same situation that they are afraid of in the first place!

Edit: The meme may not in fact mean what I think it means lol. I still think my point is interesting so I'm not gonna delete. At minimum doesn't it seem in principle impossible for a civilization that creates a simulation to be sure that the simulation won't in some way damage their reality?
Exit Duty Generator by Matti Häyry
by Oldphan — 2023-02-24T18:35:58.502Z
Link: https://www.cambridge.org/core/journals/cambridge-quarterly-of-healthcare-ethics/article/exit-duty-generator/49ACA1A21FF0A4A3D0DB81230192A042#article

Hello! I was hoping to start a discussion about a recent article called Exit Duty Generator by Matti Häyry, it can be found here:

https://www.cambridge.org/core/journals/cambridge-quarterly-of-healthcare-ethics/article/exit-duty-generator/49ACA1A21FF0A4A3D0DB81230192A042#article

or DOI: https://doi.org/10.1017/S096318012300004X

Abstract

This article presents a revised version of negative utilitarianism. Previous versions have relied on a hedonistic theory of value and stated that suffering should be minimized. The traditional rebuttal is that the doctrine in this form morally requires us to end all sentient life. To avoid this, a need-based theory of value is introduced. The frustration of the needs not to suffer and not to have one’s autonomy dwarfed should, prima facie, be decreased. When decreasing the need frustration of some would increase the need frustration of others, the case is deferred and a fuller ethical analysis is conducted. The author’s perceptions on murder, extinction, the right to die, antinatalism, veganism, and abortion are used to reach a reflective equilibrium. The new theory is then applied to consumerism, material growth, and power relations. The main finding is that the burden of proof should be on those who promote the status quo.

Any thoughts? Thanks!

All the best,

Amanda
How major governments can help with the most important century
by HoldenKarnofsky — 2023-02-24T18:20:08.530Z

I’ve been writing about tangible things we can do today to help the most important century go well. Previously, I wrote about helpful messages to spread; how to help via full-time work; and how major AI companies can help.

What about major governments1 - what can they be doing today to help?

I think governments could play crucial roles in the future. For example, see my discussion of standards and monitoring.

However, I’m honestly nervous about most possible ways that governments could get involved in AI development and regulation today.

I think we still know very little about what key future situations will look like, which is why my discussion of AI companies (previous piece) emphasizes doing things that have limited downsides and are useful in a wide variety of possible futures.

I think governments are “stickier” than companies - I think they have a much harder time getting rid of processes, rules, etc. that no longer make sense. So in many ways I’d rather see them keep their options open for the future by not committing to specific regulations, processes, projects, etc. now.

I worry that governments, at least as they stand today, are far too oriented toward the competition frame (“we have to develop powerful AI systems before other countries do”) and not receptive enough to the caution frame (“We should worry that AI systems could be dangerous to everyone at once, and consider cooperating internationally to reduce risk”). (This concern also applies to companies, but see footnote.2)

(Click to expand) The “competition” frame vs. the “caution” frame”

In a previous piece, I talked about two contrasting frames for how to make the best of the most important century:

The caution frame. This frame emphasizes that a furious race to develop powerful AI could end up making everyone worse off. This could be via: (a) AI forming dangerous goals of its own and defeating humanity entirely; (b) humans racing to gain power and resources and “lock in” their values.

Ideally, everyone with the potential to build something powerful enough AI would be able to pour energy into building something safe (not misaligned), and carefully planning out (and negotiating with others on) how to roll it out, without a rush or a race. With this in mind, perhaps we should be doing things like:

Working to improve trust and cooperation between major

...
Consent Isn't Always Enough
by jefftk — 2023-02-24T15:40:05.048Z

After the

Time
article on sexual harassment and abuse within Effective Altruism
and

Owen
Cotton-Barratt's resignation there's been a lot of discussion
around the extent to which EA culture did or didn't contribute. This
has included proposals that EA try to be

more
conventional, such as discouraging

polyamory
or

hookups
within the community. At which point others naturally
bristle: what happens between consenting adults is no one else's
business, and it's wrong to try and influence others here. For

example:

I don't see any way you could meaningfully "address" the work/social
overlap without trying to get people not to date, live with or
befriend people they otherwise would have dated, lived with, or
befriended. And if you put it in those terms, it seems messed up,
right?

Or:

I will not be a part of community which treats conscious and
consensual behavior of adult people as their business.

Or:

Perhaps consider minding your own business about whether or not
consenting adults sleep together.

Now, the people who wrote these have more nuanced views than these
quotes suggest, and I think we don't actually disagree much when it
comes to views on detailed situations. Still, I think it's worth
getting some of that nuance out there.

Specifically, consent isn't always enough. Even consent plus a mature
awareness of power
dynamics isn't always enough. Consider a case of someone whose
job it is to direct funding for a foundation sleeping with one of the
people who runs an organization they might recommend
funding. And let's further imagine that the grantmaker and grantee
both have a sophisticated understanding of power dynamics, great
communication, solid introspection, strong self-confidence, and the
best of intentions. Even then, this has corrosive effects on the
community, including:

Other grantees would then feel pressure to sleep with grantmakers,
leading to bad interactions, including ones where all the signals that
the grantmaker receives are that the grantee wants this.

The funders behind the grantmaker may reasonably worry that the
grantmaker's judgement is clouded by their otherwise positive views
towards this grantee or that there was quid pro quo.

People with unsavory intentions may choose to become
grantmakers because a norm of "it's ok to sleep with grantees" is very
vulnerable to abuse

...