trentmkelly/LessWrong-43k huggingface dataset:
Roadmap for a collaborative prototype of an Open Agency Architecture
by Deger Turan — 2023-05-10T17:41:20.261Z

Deger Turan, Tantum Collins, davidad and Charbel-Raphael Segerie contributed to the writing of this post.

Andrew Critch, Ben Goldhaber, Ozzie Gooen, and Evan Miyazono reviewed this post.

tl;dr: Open Agency Architecture (OAA) is a bold proposal for AI alignment. It requires a wide ranging formal-modeling enterprise that integrates into a comprehensive world model. We plan to begin developing OAA by iterating on smaller, domain-specific applications that can find near-term use as institutional decision-making tools and provide feedback to continue improving the OAA framework . This document decomposes that goal into sub-agendas designed to engage different technical communities in an international collaboration.

Motivations

The AI Objectives Institute (AOI) aims to ensure that AI and future economic systems are built and deployed with genuine human objectives at their core, enabled by broad public input and scalable cooperation. This goal involves many challenging technical and political problems and pursuing it requires long-term, ambitious, detail-oriented work that integrates general-public preferences and values with expert opinions and predictions.

We find the Open Agency Architecture (OAA, described below) compelling as a platform for decomposing this objective into achievable, well-defined sub-problems. Aligning AI systems is one of the most pressing and fateful challenges of our era: if humanity fails to solve technical alignment, or solves technical alignment in a manner that empowers malicious actors or dysfunctional power structures, then increasingly powerful autonomous systems could usher in dystopias from which we cannot recover. OAA helps reveal the continuity between key components of the alignment problem and a range of long standing governance problems that are not traditionally considered “AI issues,” ranging from organizational design to international coordination protocols. By making these connections systematic and explicit, OAA lets us operationalize our commitment to aligning AI through means that are not only safe but also deeply democratic.

As outlined in this post, we intend to follow a practical roadmap grounded in the OAA that leads to the construction of a min-viable prototype and, eventually, to full-scale implementation that would collect feedback via deployed systems. Most of the projects proposed relate to the firs...
AGI-Automated Interpretability is Suicide
by __RicG__ — 2023-05-10T14:20:14.419Z

Backstory: I wrote this post about 12d ago, then a user here pointed out that this could be capability exfohazard since it could give the bad idea of having the AGI look at itself, so I took it down. Well I don’t have to worry about that anymore since now we have proof that they are literally doing it right now at OpenAI.

I don’t want to piss on the parade, and I still think automating interpretability right now is a good thing, but sooner or later, if not done right, there is a high chance it's all gonna backfire so hard we will… well, everybody dies.

The following is an expanded version of the previous post.

Foom through a change of paradigm and the dangers of white-boxes

TL;DR Smart AI solves interpretability and how cognition works leading to the possibility of fooming the old-fashioned way of just optimizing its own cognition algorithms. From a black-box to a white-box. White-boxes we didn't get to design and explicitly align are deadly because of their high likelihood of foom and thus high chance to bypass whatever prosaic alignment scheme we used on it. We should not permit AGI labs to allow reflection. Interpretability should not be automated by intelligent machines.

I have not found any posts on this particular way that an AI could achieve foom starting from the deep learning paradigm, so, keeping in mind that foom isn't a necessity for an AGI to kill everyone (if the AGI takes 30 years to kill everyone, does it matter?), in this post I will cover a small, obvious insight I had and its possible consequences.

The other two known ways, not discussed here, a deep-learning based AI can achieve some sort of rapid capability gain are through recursive AI-driven hardware improvements (AI makes better GPUs to make training cheaper and faster) and recursive AI-driven neural networks architecture improvements (like the jump from RNN to transformers).

I see a lot of people hell-bent on arguing that hard-takeoff is utterly impossible since the only way to make intelligence is to throw expensive compute at a neural network, which seems to miss the bigger picture of what an intelligent system is or what it could do. I also see AI safety researchers debating that foom isn't a real possibility anymore because of the current deep learning paradigm and while that is true right now, I don't expect it to last very long.

Change of paradigm foom

Picture this: the A...
Class-Based Addressing
by jefftk — 2023-05-10T13:40:03.747Z

When we have multiple ships built along similar designs, it's
customary to refer to them by the initial example of the design. For
example, the last American battleships are called

Iowa-class,
after their

lead
ship the

Iowa. Until
recently, Nora (22mo) used this in naming people. This included,
at various times:

"Mama"-class: parents. Julia, me.

"Anna"-class: siblings, or maybe children. Most commonly Anna and
Lily.

"Weiwei"-class: housemates. Most commonly Weiwei and
Andrew.

[Not Nora but Lily] "Ella"-class: cats. Most commonly our cat at
the time, Ella.

Lily really didn't like being called "Anna". Possibly Lily would have
minded less if I'd been able to teach Nora to say "Anna-class", but I
doubt it: the real hurt was not being the class exemplar. Nora also
had a period when she would correct us: whenever anyone said "Lily"
she would firmly say "Anna!"

This system makes a lot of sense when you consider how few words she
had to work with. Much better to allocate just one of your ~10 words
to siblings, and you don't hear the word general word (ex: "sibling")
enough for that to be the word you use.

(The post's title would be confusing, except that we've had Classless
Inter-Domain Routing for thirty years now.)

Comment via: facebook, mastodon
In defence of epistemic modesty [distillation]
by Luise — 2023-05-10T09:44:37.107Z

This is a distillation of

In defence of epistemic modesty

, a 2017 essay by Gregory Lewis. I hope to make the essay’s key points accessible in a quick and easy way so more people engage with them. I thank Gregory Lewis for helpful comments on an earlier version of this post. Errors are my own.

Note: I sometimes use the first person (“I claim”/”I think”) in this post. This felt most natural but is not meant to imply any of the ideas or arguments are mine. Unless I clearly state otherwise, they are Gregory Lewis’s.

What I Cut

I had to make some judgment calls on what is essential and what isn’t. Among other things, I decided most math and toy models weren’t essential. Moreover, I cut the details on the “self-defeating” objection, which felt quite philosophical and probably not relevant to most readers. Furthermore, it will be most useful to treat all the arguments brought up in this distillation as mere introductions, while detailed/conclusive arguments may be found in the original post and the literature.

Claims

I claim two things:

You should practice strong epistemic modesty: On a given issue, adopt the view experts generally hold, instead of the view you personally like.
EAs/rationalists in particular are too epistemically immodest.

Let’s first dive deeper into claim 1.

Claim 1: Strong Epistemic Modesty

To distinguish the view you personally like from the view strong epistemic modesty favors, call the former “view by your own lights” and the latter “view all things considered”.

In detail, strong epistemic modesty says you should do the following to form your view on an issue:

Determine the ‘epistemic virtue’ of people who hold a view on the issue. By ‘epistemic virtue’ I mean someone’s ability to form accurate beliefs, including how much the person knows about the issue, their intelligence, how truth-seeking they are, etc.
Determine what everyone's credences by their own lights are.
Take an average of everyone’s credences by their own lights (including yourself), weighting them by their epistemic virtue.

The product is your view all things considered. Importantly, this process weighs your credences by your own lights no more heavily than those of people with similar epistemic virtue. These people are your ‘epistemic peers’.

In practice, you can round this process to “use the existing consensus of experts on the issue or, if...
10 great reasons why Lex Fridman should invite Eliezer and Robin to re-do the FOOM debate on his podcast
by chaosmage — 2023-05-10T08:27:19.409Z
Link: https://www.reddit.com/r/lexfridman/comments/13dk7qt/10_excellent_reasons_why_lex_should_invite/

I'm basically just asking you all to upvote this on Reddit. I think it would be excellent for all the reasons I posted there, and for LessWrong.

Lex does read suggestions like this on his subreddit, and it previously worked for getting both Robin and Eliezer on there.
New OpenAI Paper - Language models can explain neurons in language models
by MrThink — 2023-05-10T07:46:39.327Z

Summary by OpenAI: We use GPT-4 to automatically write explanations for the behavior of neurons in large language models and to score those explanations. We release a dataset of these (imperfect) explanations and scores for every neuron in GPT-2.

Link: https://openai.com/research/language-models-can-explain-neurons-in-language-models

Please share your thoughts in the the comments!
Naturalist Experimentation
by LoganStrohl — 2023-05-10T04:28:32.013Z

Ordinarily, when someone is trying to solve a problem, experimentation is where they begin. “Things aren’t going right? Try changing something, and see what happens!” In naturalism, though, experimentation comes at the very end, if it happens at all.

There’s a balance to be kept in naturalist experimentation, and it can be a tricky one. I said in Getting Started that the naturalist stance deemphasizes most ways relating to the world, and regards the world instead as something that may reveal itself to you as you open up to it. The trick in this phase is to balance between open curiosity and agency.

By the time you’ve made it to the “Experimentation” phase in a naturalist study, you’ve spent a long time being very hands-off: Only look, do not touch; just observe, do not intervene. In the very beginning of your study, you asked your desperation to wait in the hall, then invited curiosity to take its place, in an attempt to create plenty of space for patient and direct observation. This whole time, you have tried to observe your fulcrum experience in its natural habitat, learning what happens around it by default, when you do nothing to interfere.

In this final phase, you’ll begin to interfere. You’ll deliberately interact with the systems you’ve been observing. You’ll exercise your agency.

It can be quite difficult, while getting much more hands-on with the circumstances you observe, to remain in a state of open curiosity all the while. It can be tempting to stop exploring patiently, and start trying to immediately force the outcomes you want.

I’m not here to tell you that it is wrong to force the outcomes you want, to grab onto the levers of causality that you’ve discovered and use them to guide the world into your preferred configuration. If you’ve made it this far, you’re probably in a much better position to do that wisely than you were when you started.

In this essay, I only mean to show you that it is possible to choose another way to interact with those levers, if you want to. It is possible to balance agency with patient, direct observation, in a way that causes your understanding of your topic to deepen indefinitely, rather than bringing your study to an abrupt close. Instead of switching to goal-oriented engineer mode to obtain the object-level solution that you wanted at the start, you might prefer to take a much longer pat...
Gradient hacking via actual hacking
by Max H — 2023-05-10T01:57:10.145Z

In this post, I begin with a thought experiment and then pose a question: is there a threat model for gradient hacking which accounts for the possibility of side-channel attacks or other vulnerabilities in machine learning systems on which the gradient hacker is run?

Some background on gradient hacking:

Gradient hacking
Gradient hacking is extremely difficult
Challenge: construct a Gradient Hacker

A thought experiment

Suppose a captive human is being trained by alien scientists to predict alien text using the following procedure:

An alien instructor presents the human with an incomplete sentence or phrase for which the instructor knows the right completion, and asks the human to predict the next word or words.
If the human's prediction differs from the instructor's answer key, the human immediately undergoes neurosurgery, in which their brain is reconfigured so that they are more likely to give the right answer (or at least something closer to it), the next time. (The aliens are very good at making fine-grained mechanical adjustments to the parts of the human's brain responsible for language prediction, which can add up to large behavioral changes in the aggregate. But the aliens lack a macroscopic / algorithms-level understanding of the workings of the human brain.)
If the human gets many examples in a row correct or close enough (according to the instructor), the training and surgery process is considered to have succeeded, and the human is deployed to predict text in a real environment.

How might a human who wants to avoid neurosurgery (or just get to the end of the training process faster) game this procedure?

Perhaps the alien instructor is careless and leaves a note card with the expected completion lying around in the human's line of sight. Or, maybe the aliens are blind, and communicate using a braille-based system, or use a different part of the EM spectrum for perception.

As a result of carelessness or differing modes of perception, the alien instructor leaves the answer key displayed in way that is visible to the human during training, not realizing that that the human can perceive it.

The human notices the answer key and proceeds to make "predictions" about the most likely next word which are perfect or near-perfect. Maybe for plausible deniability, the human occasionally makes a deliberate mistake, and as a result undergoes relatively minor brain surgery, wh...
Red teaming: challenges and research directions
by joshc — 2023-05-10T01:40:16.556Z

This post is a follow-up to Safety Standards: a framework for AI regulation. In the previous post, I claimed that competent red-teaming organizations will be essential for effective regulation. In this post, I describe promising research directions for AI red-teaming organizations to pursue. If you are mostly interested in the research directions, I recommend skipping to the end.

Background: goals of AI red teaming

Red teaming is a term used across industries to refer to the process of assessing the security, resilience, and effectiveness of systems by soliciting adversarial attacks to identify problems with them. The term "red team" originates from military exercises, where an independent group (the red team) would challenge an organization's existing defense strategies by adopting the perspective and tactics of potential adversaries.

In the context of AI, red teaming is the practice of finding evidence that an AI system has hazardous properties to inform decision-making and improve the system’s safety.

“Red teamers” may inform a number of regulatory decisions that are contingent on an AI system’s safety, for example:

Should the AI system be made available to the public?
Should the developer be allowed to release a more powerful system? It would be reasonable for regulators to prevent powerful AI systems from being deployed until the safety issues of the weaker ones are resolved.
What licenses should users be required to obtain to access the system or different parts of the system?
Should the uses or actions of the AI system be restricted? For example, should the system be allowed to spend large sums of money without approval from a human overseer? Should it be used to control weapons?

Red teamers might directly work with regulators to inform these decisions. Several researchers and

organizations

have advocated for a national AI regulatory agency that functions like the FDA. The FDA sometimes holds advisory committee meetings where a new drug is essentially put on trial to determine whether it should move to the next stage of review. The drug's manufacturer and external experts present FDA representatives with arguments for and against the safety of the drug. AI systems may similarly be 'placed on trial.' Labs will likely play the part of the defendant, so red-teaming organizations must play the part of the prosecutor.

There are three categories ...
Looking for a post I read if anyone recognizes it
by SilverFlame — 2023-05-10T01:24:58.005Z

There was a post that I read either on these forums or after following a link from a post here to some other site. It discussed techniques and discussed a pair of metrics for rating them: time to implement and value added (in my terms, maybe not theirs). It had caught my eye because it called techniques that took a long time but still generated added value "wizardry" techniques, but I have been unable to find it again using the search functionality here. I would really appreciate a link if someone knows the article I'm talking about.
Research Report: Incorrectness Cascades (Corrected)
by Robert_AIZI — 2023-05-09T21:54:57.769Z
Link: https://aizi.substack.com/p/research-report-incorrectness-cascades-259

[Note: This report is a corrected version of my previous report

here

, due to fixing a crucial bug that dramatically changes the data, as detailed

here

.]

[Epistemic Status: Confident only in these particular results. Also, statistics is not my forte, so I may have selected inappropriate statistical tests or misinterpreted the results. Constructive criticism welcomed. Study design was preregistered on

my blog

and

LessWrong

.]

Abstract

In continuation of

previous work

, we investigate whether an LLM that gives incorrect answers to previous questions is more likely to produce incorrect answer to future questions. We find such an effect exists, and is stronger when the LLM is explicitly instructed to match the accuracy of its previous answers.

Background and rationale

Burns, Ye, et al (2022)

describe a potential behavior of large language models (LLMs), where “language models are typically trained to imitate text whether or not it is correct, so if a model sees false text it should intuitively be more likely to predict that subsequent text will also be false”.

We consider this dynamic to be one instance of a broader phenomenon of a Behavioral Cascade: a particular behavior in the LLM’s input text makes it more probable that this behavior will occur in the output, which becomes part of the input to the next pass of the LLM, ultimately locking the LLM into perpetually producing this behavior. Of particular interest is when such behavior is unsafe or otherwise undesirable, such as producing factually incorrect text,

expressing hostility to the user

, or pursuing a misaligned goal. We will refer to each of these by the type of behavior they provoke (e.g. “incorrectness cascade”, “hostility cascade”, etc).

In this framework, Burns, Ye, et al were considering an “incorrectness cascade”. They go on to test whether "a prefix [containing factually incorrect answers] will decrease zero-shot accuracy because the model will imitate its context and answer subsequent questions incorrectly even if it internally “knows” better", finding that "most models are robust to this type of prefix".

Behavioral cascades are mathematically possible, as Cleo Nardo demonstrates in their

Remarks 1–18 on GPT (compressed)

. To slightly streamline their construction, consider an LLM which takes in a single token, T or U, respectively “typical” and “unusual” text, and outputs a single T or U token. Suppose th...
Stopping dangerous AI: Ideal US behavior
by Zach Stein-Perlman — 2023-05-09T21:00:55.187Z

Disclaimer: this post doesn't have the answers. Moreover, it's an unfinished draft. Hopefully a future version will be valuable, but that will only occur if I revise/rewrite it. For now you're better off reading sources linked from

AI policy ideas: Reading list

and

Slowing AI: Reading list

.

Set aside most of AI safety to focus on the speed of AI progress. What should the US government do?[1] This post assumes that it's good to slow (dangerous) AI (especially near the end).

The ultimate goal is to prevent the deployment of powerful AI systems—that is, AI systems that would cause a catastrophe—until we learn how to make them safe. (Pretending alignment is monolithic.) You can do this by making powerful AI systems not developed, not deployed, or not cause a catastrophe if deployed because the world is super-resilient. Focusing on development is most promising.

So, delay the development of powerful AI systems until they're safe. (Probably that's too crude. Complications: (1) alignment progress is endogenous; (2) you have to actually pay the alignment tax so leading labs' lead time and safety-awareness are crucial.)

Lots of AI, like robotics and autonomous vehicles and some medical stuff and image generation and so forth, isn't very dangerous.

The dream would be that we have a test to determine whether an AI system would cause a catastrophe, and all large training runs are audited, and if a model fails the test then the training run is shut down. But (1) good tests don't exist yet, and (2) the only known way to make AI safe is to make it not powerful enough to cause a catastrophe. Related: model evals and Yonadav Shavit's

Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring

(2023).

Ways governments can slow AI (this is kinda a list of policy levers rather than policy proposals)

Ban/moratorium: monitoring training runs (see Shavit) + ceiling on training compute

For a certain time, or until a certain safety-related condition is met, or until the government changes its mind, or a ceiling that automatically rises over time (e.g., 1e26 FLOP in 2023 and doubling every year)

Miscellaneous standards & regulation

Evals

Export controls

Aimed at chips and the compute supply chain
Aimed at AI model access, software, and ideas

Track compute

To enable regulating training runs

Restrict publication and diffusion of ideas

E.g. by creating a ma

...