trentmkelly/LessWrong-43k huggingface dataset:
Deferring
by owencb — 2022-05-12T23:56:20.679Z

(Cross-posted from the EA Forum)

Deferring is when you adopt someone else's view on a question over your own independent view (or instead of taking the time to form an independent view). You can defer on questions of fact or questions of what to do. You might defer because you think they know better (epistemic deferring), or because there is a formal or social expectation that you should go along with their view (deferring to authority).

Both types of deferring are important — epistemic deferring lets people borrow the fruits of knowledge; deferring to authority enables strong coordination. But they are two-edged. Deferring can mean that you get less chance to test out your own views, so developing mastery is slower. Deferring to the wrong people can be straightforwardly bad. And when someone defers without everyone understanding that's what's happening, it can cause issues. Similarly, unacknowledged expectations of deferral from others can cause problems. We should therefore learn when and how to defer, when not to, and how to be explicit about what we're doing.

Why deferring is useful

Epistemic deferring

Epistemic deferring is giving more weight to someone else's view than your own because you think they're in a position to know better. The opposite of epistemic deferring is holding one's own view.

Examples:

"You've been to this town before; where's the best place to get coffee?"
"My doctor/lawyer says this is a common situation, and the right thing to do is ..."
"A lot of smart folks seem to think AI risk is a big deal; it sounds batshit to me, but I guess I'll look into it more"

The case for epistemic deferring is simple: for most questions, we can identify someone (or some institution or group of people) whose judgement on the question would — if they were possessed of the facts we knew — be better than our own. So to the extent that

(A) We want to optimize for accurate judgements above all else, &
(B) We are willing to make the investment to uncover that better judgement,

deferring will be correct.

Partial deferring

The degree to which (A) and (B) hold will vary with circumstance. It will frequently be the case that they partially hold; in this case it may be appropriate to partially defer, e.g.

“I’m torn between whether to take job X or job Y. On my view job X seems better. When I talk to my friends and family they overwhelmingly think job Y sou

...
RLHF
by Ansh Radhakrishnan — 2022-05-12T21:18:21.447Z

I’ve been thinking about Reinforcement Learning from Human Feedback (RLHF) a lot lately, mostly as a result of

my AGISF capstone project

attempting to use it to teach a language model to write better responses to Reddit writing prompts, a la

Learning to summarize from human feedback

.

RLHF has generated some impressive outputs lately, but there seems to be a significant amount of disagreement regarding its potential as a partial or complete solution to alignment:

some are excited to extend the promising results we have so far

,

while others are more pessimistic and perhaps even opposed to further work along these lines

. I find myself optimistic about the usefulness of RLHF work, but far from confident that all of the method’s shortcomings can be overcome.

How it Works

At a high level, RLHF learns a reward model for a certain task based on human feedback and then trains a policy to optimize the reward received from the reward model. In practice, the reward model learned is likely overfit - the policy can thus benefit from interpolating between a policy that optimizes the reward model’s reward and a policy trained through pure imitation learning.

A key advantage of RLHF is the ease of gathering feedback and the sample efficiency required to train the reward model. For many tasks, it’s significantly easier to provide feedback on a model’s performance rather than attempting to teach the model through imitation. We can also conceive of tasks where humans remain incapable of completing the tasks themselves, but can evaluate various completions and provide feedback on them. This feedback can be as simple as picking the better of two sample completions, but it’s plausible that

other forms of feedback

might be more appropriate and/or more effective than this. The ultimate goal is to get a reward model that represents human preferences for how a task should be done: this is also known as

Inverse Reinforcement Learning

. The creators of the method, Andrew Ng and Stuart Russell, believe that

“the reward function, rather than the policy, is the most succinct, robust, and transferable definition of the task,”

. Think about training an AI to drive a car: we might not want it to learn to imitate human drivers, but rather learn what humans value in driving behavior in the abstract and then optimize against those preferences.

Outer Alignment Co...
What to do when starting a business in an imminent-AGI world?
by ryan_b — 2022-05-12T21:07:10.111Z

As reported by la3lorn and Daniel Kokotajlo, Gato is here and appears to me to represent a sub-human AGI, or near enough as makes no difference in a timeline sense. I think this probably means a general thickening of deep learning applications everywhere, and the introduction of a kind of "stack" AI that can do things we used to need whole organizations to do - as an example, I mean things like do patent research, label patent diagrams, and file patent lawsuits.

I also have an idea about a business I would like to start. This is already a notoriously trying task with low probability of success, and I wonder how much more so it will be in a world that will populate with AGI patent trolls, along with whatever else, well before hitting any kind of clear success mark.

So my question is: what do we do to account for powerful AI, showing up soon, when we are starting a business?

Note that what I am interested in here is non-AI businesses in particular and non-software businesses in general, because this looks like the threshold for it spilling across a bunch of new domains.
Interpretability’s Alignment-Solving Potential: Analysis of 7 Scenarios
by Evan R. Murphy — 2022-05-12T20:01:56.400Z

This is the second post in the sequence “Interpretability Research for the Most Important Century”. The first post, which introduces the sequence, defines several terms, and provides a comparison to existing works, can be found here: Introduction to the sequence: Interpretability Research for the Most Important Century.

Summary

This post explores the extent to which interpretability is relevant to the hardest, most important parts of the AI alignment problem (property #1 of High-leverage Alignment Research[1]).

First, I give an overview of the four important parts of the alignment problem (following Hubinger[2]): outer alignment, inner alignment, training competitiveness and performance competitiveness (jump to section). Next I discuss which of them is “hardest”, taking the position that it is inner alignment (if you have to pick just one), and also that it’s hard to find alignment proposals which simultaneously address all four parts well.

Then, I move onto exploring how interpretability could impact these four parts of alignment. Our primary vehicle for this exploration involves imagining and analyzing seven best-case scenarios for interpretability research (jump to section). Each of these scenarios represents a possible endgame story for technical alignment, hinging on one or more potential major breakthroughs in interpretability research. The scenarios’ impacts on alignment vary, but usually involve solving inner alignment to some degree, and then indirectly benefiting outer alignment and performance competitiveness; impacts on training competitiveness are more mixed.

Finally, I discuss the likelihood that interpretability research could contribute to unknown solutions to the alignment problem (jump to section). This includes examining interpretability’s potential to lead to breakthroughs in our basic understanding of neural networks and AI, deconfusion research and paths to solving alignment that are difficult to predict or otherwise not captured by the seven specific scenarios analyzed.

Tips for navigating this long post! If you get lost scrolling through this post on mobile, consider reading on desktop for two reasons: 1) To take advantage of LessWrong's convenient linked outline feature that appears in the sidebar, and 2) To be able to glance at the footnotes and posts that I link to just by hovering over them.

Acknowledgments

Lots of people grea...
Introduction to the sequence: Interpretability Research for the Most Important Century
by Evan R. Murphy — 2022-05-12T19:59:52.911Z

This is the first post in a sequence exploring the argument that interpretability is a high-leverage research activity for solving the AI alignment problem.

This post contains important background context for the rest of the sequence. I'll give an overview of one of Holden Karnofksy's (2022) Important, actionable research questions for the most important century[1], which is the central question we'll be engaging with in this sequence. I'll also define some terms and compare this sequence to existing works.

If you're already very familiar with Karnofsky (2022)[1] and interpretability, then you can probably skip to the second post in this sequence: Interpretability’s Alignment-Solving Potential: Analysis of 7 Scenarios

The Alignment Research Activities Question

This sequence is being written as a direct response to the following question from Karnofsky (2022)[1]:

“What relatively well-scoped research activities are particularly likely to be useful for longtermism-oriented AI alignment?” (

full question details

)

I'll refer to this throughout the sequence as the Alignment Research Activities Question.

Context on the question and why it matters

In the details linked above for the Alignment Research Activities Question, Holden first discusses two categories of alignment research which are lacking in one way or another. He then presents a third category with some particularly desirable properties:

“Activity that is [1] likely to be relevant for the hardest and most important parts of the problem, while also being [2] the sort of thing that researchers can get up to speed on and contribute to relatively straightforwardly (without having to take on an unusual worldview, match other researchers’ unarticulated intuitions to too great a degree, etc.)”

He refers to this as "category (3)", but I'll use the term High-leverage Alignment Research since it's more descriptive and we'll be referring back to this concept often throughout the sequence.

We want to know more about which alignment research is in this category. Why? Further excerpts from Karnofsky (2022)[1] to clarify:

“I think anything we can clearly identify as category (3) [that is, High-leverage Alignment Research] is immensely valuable, because it unlocks the potential to pour money and talent toward a relatively straightforward (but valuable) goal.
[...]
I think there are a lot of people who

...
A tentative dialogue with a Friendly-boxed-super-AGI on brain uploads
by Ramiro P. — 2022-05-12T19:40:48.113Z

[Unnecessary explanation: Some people asked me why I thought the world of Friendship is optimal is dystopic… During the discussion, I inferred that what they saw as a “happy story” in AI safety is something like this: we’ll first solve something like a technical engineering problem, so ensuring that the AGI can reliably find out what we *really* want, and then satisfy it without destroying the world… In that world, “value” is not a hard problem (we can leave its solution to the AI), so that if we prove that an AI is aligned, we should just outsource everything relevant to it.

As I found out I still had some troubles expressing my objections convincingly, I wrote this dialogue about an AGI that is even “safer” and more aligned than Celestia. I am way beyond my field of expertise here, though.]

B is told that we have finally built a reliable Friendly Boxed Super AGI that can upload our brains and make us live arbitrarily long lives according to something like our "extrapolated coherent volition".

AI: Hey, time to upload you to Paradise.

B: Ok.

AI: The process will utterly destroy your brain, though.

B: Hmmm... I don't like it.

AI: But you'll still live in the simulation.

B: Yeah, but now I am not so sure... I mean, how can I totally trust you? And I am not 100% sure this is going to be 100% me and...

AI: C'mon, Musk made me. I'm reliable.

B: Look, isn’t there an alternative process?

AI: Ok, there's one, but it'll take a bit more time. I'll scan your brain with this weird device I just invented and then come back next week, ok?

B: Thanks. See you then.

<One week later>

AI: I am back. Your copy is already living in the Sim.

B: Oh, great. I'm happy with that. Is he / me happy?

AI: Sure. Look at this screen. You can see it in first or third person.

B: Awesome. I sort of envy this handsome guy.

AI: Don't worry. It's you, just in another point in spacetime. And you won't feel this way for long ...

B: I know…

AI: ... because now I'm going to kill the instance of you I'm talking to, and use those atoms to improve the simulation. Excuse me.

B: Hey! No, wait!

AI: What's up?

B: You said you were going to kill me.

AI: Well, I'd rather say I was going to help you transcend your current defective condition, but you guys built me in a way I can't possibly do what humans call “white lies”. Sorry.

B: WTF? Why do you need to kill *me* in the first place?

AI: Let’s say I'm not killing *y...
The Last Paperclip
by Logan Zoellner — 2022-05-12T19:25:50.891Z

Note: this short story is an attempt to respond to this comment. Specifically, this story is an attempt to steelman the claim that super-intelligent AI is "aligned by definition", if all that we care about is that the AI is "interesting", not that it respects human values. I do not personally advocate anyone making a paperclip maximizer.

Prologue: AD 2051

The Alignment Problem had at last been solved. Thanks to advances in Eliciting Latent Knowledge, explaining human values to an AI was as simple as typing:

from Alignment import HumanFriendly

As a result, a thousand flowers of human happiness and creativity had bloomed throughout the solar system. Poverty, disease and death had all been eradicated, thanks to the benevolent efforts of Democretus, the super-intelligent AI that governed the human race.

Democretus--or D, as everyone called the AI--was no dictator, however. Freedom was one of the values that humans prized most highly of all, and D was programmed to respect that. Not only were humans free to disobey D's commands--even when it would cause them harm--there was even a kill-switch built into D's programming. If D ever discovered that 51% of humans did not wish for it to rule anymore, it would shut down in a way designed to cause as little disruption as possible.

Furthermore, D's designs were conservative. Aware that humanity's desires might change as they evolved, D was designed not to exploit the universe's resources too quickly. While it would have been a trivial matter for D to flood the surrounding universe with Von-Neumann probes that would transform all available matter into emulations of supremely happy human beings, D did not do so. Instead, the first great ships to cross the vast distances between stars were just now being built, and were to be crewed with living, biological human beings.

It is important to point out that D was not naïve or careless. While humans were granted near-complete autonomy, there was one rule that D enforced to a fault: no human being may harm another human without their consent. In addition to the obvious prohibitions on violence, pollution, and harmful memes, D also carefully conserved resources for future generations. If the current generation of human beings were to use up all the available resources, they would deprive their fut...
Deepmind's Gato: Generalist Agent
by Daniel Kokotajlo — 2022-05-12T16:01:21.803Z

From the abstract, emphasis mine:

The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stackblocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens.

(Will edit to add more as I read. ETA: 1a3orn posted first.)

It's only 1.2 billion parameters. (!!!) They say this was to avoid latency in the robot control task.
It was trained offline, purely supervised, but could in principle be trained online, with RL, etc
Performance results:

The section on broader implications is interesting. Selected quote:

In addition, generalist agents can take actions in the the physical world; posing new challenges that may require novel mitigation strategies. For example, physical embodiment could lead to users anthropomorphizing the agent, leading to misplaced trust in the case of a malfunctioning system, or be exploitable by bad actors. Additionally, while cross-domain knowledge transfer is often a goal in ML research, it could create unexpected and undesired outcomes if certain behaviors (e.g. arcade game fighting) are transferred to the wrong context. The ethics and safety considerations of knowledge transfer may require substantial new research as generalist systems advance. Technical AGI safety (Bostrom, 2017) may also become more challenging when considering generalist agents that operate in many embodiments. For this reason, preference learning, uncertainty modeling and value alignment (Russell, 2019) are especially important for the design of human-compatible generalist agents. It may be possible to extend some of the value alignment approaches for language (Kenton et al., 2021; Ouyang et al., 2022) to generalist agents. However, even as technical solutions are developed for value alignment, generalist systems could still have negative societal impacts even with the intervention of well-intentioned designers, due to unforeseen circumstances or limited oversight (Amodei et al., 2016). This limitation underscores the need for a careful design and a deployment process that incorporates multiple disciplines and viewpoints.

They also do some scaling analysis and yup, you can make it smarter by making it bigger.

What do I think about all this?

Eh, I guess it was alr...
"A Generalist Agent": New DeepMind Publication
by 1a3orn — 2022-05-12T15:30:17.871Z

Linkpost for "A Generalist Agent"

Abstract:

"Inspired by progress in large-scale language modeling, we apply a similar approach towards building a
single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a
multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights
can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based
on its context whether to output text, joint torques, button presses, or other tokens. In this report we
describe the model and the data, and document the current capabilities of Gato"
Covid 5/12/22: Other Priorities
by Zvi — 2022-05-12T13:30:02.425Z

There is zero funding for dealing even with the current pandemic, let alone preventing the next one. The FDA not only is in no hurry to approve a vaccine for children, the new highlight is its focus on creating a dire shortage of specialty baby formula. Covid doesn’t kill children, merely causing governments to mandate they not get to have their childhoods, but 40% of formula being out of stock is a much more directly and physically dangerous situation. The FDA has a history of killing children via not letting them have the nutrition they need to survive, last time it was an IV formulation that was incomplete but couldn’t be updated for years, so we shouldn’t act all surprised when this threatens to happen again.

Also Covid is still a thing. New subvariants of Omicron continue to slowly gain dominance and case numbers are increasing. The jump this week wasn’t as big as the headline number but it was still big, and is backed up by my local anecdata. There’s once again a bunch of Covid out there. The key is whether we can continue to get on with life and not fall into the trap of doing expensive prevention that doesn’t accomplish anything. So far, so good.

This intentionally still excludes China. News on that front has slowed dramatically, I plan to write up the last few weeks into a post anyway relatively soon.

Executive Summary

Case numbers up, although death numbers still down.
FDA causing specialty baby formula shortage crisis.
Still no funding for anything pandemic related.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 400,000 cases (+22%) and 2,720 deaths (+10%?)

Results: 518,048 cases (+41%) and 2,084 deaths (-6%)

Prediction for next week: 640,000 cases (+23%) and 2,100 deaths (+1%)

I’d love to go back and do a study of my predictions at some point if I find the bandwidth, and in particular on whether I should be ‘sticking to my guns’ more when a change doesn’t make a lot of sense. Often a blip causes an adjustment in the wrong direction and this week feels like I ‘fell for’ that trap once again, but that implies we know it wasn’t a legitimate change.

I also suspected there was some sort of two week cycle, and, well, yeah. It’s Florida, which is literally only reporting numbers every two weeks in the Wikipedia data this past month. Flat out 0 cases last week and three weeks ago. Should have caught that, and it e...
How would public media outlets need to be governed to cover all political views?
by ChristianKl — 2022-05-12T12:55:20.785Z

As one of their questions Reporters without Borders asks for their World Press Freedom Index among others:

Do public media outlets cover all political views?*

Does the law provide mechanisms to guarantee pluralism and editorial independence?*

Do public media outlets ever ignore sensitive information regarding the government or
administration that is covered by private media?*

Is the pluralism of opinions of people in the country reflected in the media?*

Part of the case of the EU against Hungary is that its press is largely government-controlled or controlled by supporters of the government. Voices critical of the government have a lower share of the public attention. Philanthropically funded journalism that intends to provide critical media gets attacked as being funded by Soros and intended to manipulate the Hungarian people.

COVID-19 showed that there are similar dynamics in the United States and other European states where voices that are critical of the regime have a hard time being published . The fighting critical content for being Russian disinformation and Hungarian strategy of fighting critical voices for being influenced by Soros follows similar dynamics where outside influence is overblown and the narrative allows for acting against critical voices.

While some national governments have state media, the EU currently doesn't have its own media outlet. Given the EU perspective of the problems in Hungary, funding critical journalism would be a good intervention. If the EU would start its own media, there's the question of media governance. How could EU-funded public media be governed so that it will represent voices from the full pluralism of opinions of people?