trentmkelly/LessWrong-43k huggingface dataset:
When Will AIs Develop Long-Term Planning?
by PeterMcCluskey — 2023-11-19T00:08:18.603Z
Link: https://bayesianinvestor.com/blog/index.php/2023/11/16/when-will-ais-develop-long-term-planning/

[I mostly wrote this to clarify my thoughts. I'm unclear whether this
will be valuable for readers. ]

I expect that within a decade, AI will be able to do 90% of current
human jobs. I don't mean that 90% of humans will be obsolete. I mean
that the average worker could delegate 90% of their tasks to an AGI.

I feel confused about what this implies for the kind of AI long-term
planning and strategizing that would enable an AI to create large-scale
harm if it is poorly aligned.

Is the ability to achieve long-term goals hard for an AI to develop?

By long-term, I'm referring to goals that require both long time
horizons, and some ability to forecast the results of multiple steps of
interventions.

Evidence from Evolution

Evolution provides some evidence that it's hard.

It seems uncommon for most species to do anything that requires planning
more than a few days in advance. The main examples that I can find of
multi-month planning seem sufficiently specialized that they likely
involve instincts that can't be adapted to novel tasks: beavers
constructing dams, and squirrels caching food.

Human success suggests there's value in a more general ability to do
long-term planning. So there was likely some selective pressure for it.
The time it took for evolution to find human levels of planning suggests
that it's relatively hard.

Human infants have the ability to develop long-term planning abilities.
It seems like they would benefit from having those planning abilities at
birth, yet they take years to develop. According to ChatGPT:

Early Childhood (3-6 years): As children begin to develop better
memory and the ability to project themselves into the future, there's
a budding understanding of time. However, their grasp of longer time
periods is still immature. They might understand "tomorrow" but
struggle with the concept of "next week" or "next month."

Middle Childhood (7-10 years): During this phase, children's
understanding of time becomes more sophisticated, and they start to
develop the ability to delay gratification and think ahead. For
instance, they might save money to buy a desired toy or understand the
idea of studying now to do well on a test later. However, their
ability to plan for the long-term (e.g., months or years ahead)
remains limited.

This evidence suggests that AIs might require longer training times, or
more diverse interactions with the world, than I'd exp...
Killswitch
by Junio — 2023-11-18T22:53:41.436Z

Epistemic status: fiction. Speculative fiction. But that said, perhaps very near-term speculative fiction, and perhaps somewhere on the way to the borderline between "fiction" and "design document."

An AI killswitch bears almost no resemblance to most of its physical analogs. It has less in common with fuses or airbags, more with distributed systems, election integrity processes, or heart monitors.

The design goal is simple enough to specify: give the system as wide an operating envelope as possible without compromising safety, at a satisfactory reliability threshold.

How much envelope are we talking? In recent memory, weeks, even months. These days, more like days. Soon enough, we expect hours. Beyond that, current methods will not be viable, and an entirely new paradigm will have to have been invented.

Modern AI systems are spun up to solve specific, locally defined, generally speaking measurable-outcome tasks. Spinning one up is like deploying the Navy SEALS, and also like beginning open-heart surgery, where the patient whose vitals are being monitored is life as we know it.

Safety is hard to quantify. This is where the heuristics come in. And the distributed systems, and the election integrity.

For the most part, safety is defined by its inverse. If anything slips outside of some heuristic window, the process is halted. For example, if too few people in the general public approve of what the AI is doing: process stops. Process stops means: machines shut down, thermite ignited, fingerprints published to globally synchronized zero-knowledge content blacklist, physical archives sealed and locked in an underground vault whose cumulative contents are now or soon expected to be many orders of magnitude more dangerous than all of the planet’s collected nuclear waste. This is not considered a failure or an erroneous outcome — it is the expected and normal way that an AI project terminates. Again, generally speaking, within days of startup. A killswitch is just a good off button.

So — too few people approve: process stops. Too many people approve: process stops. Delta of approval ratings changes too fast: process stops. Not enough (or too many) people check in at once in any given time frame: process stops. Too much unusual stock market activity in the relevant sectors: process stops. Any of the three redundant monitoring systems experience outages outside of an ac...
Superalignment
by Douglas_Reay — 2023-11-18T22:37:41.577Z
Link: https://openai.com/blog/introducing-superalignment

OpenAI have announced the approach they intend to use, to ensure humans stay in control of AIs smarter than they are:

Our goal is to build a roughly human-level automated alignment researcher. We can then use vast amounts of compute to scale our efforts, and iteratively align superintelligence.

To align the first automated alignment researcher, we will need to 1) develop a scalable training method, 2) validate the resulting model, and 3) stress test our entire alignment pipeline:

To provide a training signal on tasks that are difficult for humans to evaluate, we can leverage AI systems to assist evaluation of other AI systems (scalable oversight). In addition, we want to understand and control how our models generalize our oversight to tasks we can’t supervise (generalization).
To validate the alignment of our systems, we automate search for problematic behavior (robustness) and problematic internals (automated interpretability).
Finally, we can test our entire pipeline by deliberately training misaligned models, and confirming that our techniques detect the worst kinds of misalignments (adversarial testing).
Predictable Defect-Cooperate?
by quetzal_rainbow — 2023-11-18T15:38:41.567Z

Epistemic status: I consider everything written here pretty obvious, but I haven't seen this anywhere else. It would be cool if you could provide sources on topic!

Reason to write: I've seen once pretty confused discussion in Twitter about how multiple superintelligences will predictably end up in Defect-Defect equilibrium and I suspect that discussion would had been better if I could throw in this toy example.

PrudentBot cooperates with agent with known source code if agent cooperates with PrudentBot and don't cooperate with DefectBot. It's unexploitable and doesn't leave outrageous amount of utility on table. But can we do better? How can we formalize notion of "both agents understand what program equilibrium is, but they predictably end up in Defect-Cooperate situation because one agent is wastly smarter"?

Let's start with toy model. Imagine that you are going to play against PrudentBot or CooperateBot with p, 1−p probability each one. Payoff matrix is 5;5, 10;0, 2;2. Bots can't play with you directly, but you can write program to play. Your goal is to get maximum expected value.

If you cooperate, you are always going to get 5, so you should defect if you are going to get more than 5 in expectation:

2p+10−10p>5

p<5/8

Thus, our UncertainBot should take probability distribution, find if probability of encountering PrudentBot is less than 5/8 and defect, otherwise cooperate. The same with mixture of PrudentBot and DefectBot: you are guaranteed to get 2 if you defect, so

5p+0(1−p)>2

p>2/5

Can we invent better version of DefectBot? We can imagine TraitorBot, which takes state of beliefs of UncertainBot and predict if it can get away with defection and otherwise cooperate. Given previous analysis with mixture of PrudentBot and DefectBot, it's clear that TraitorBot defects if probability of PrudentBot is higher than 2/5 and cooperates otherwise, yielding strictly no lower utility than utility of Cooperate;Cooperate.

Such setup provides amazing amount of possibilities to explore.

Possibilities to explore how defection can happen between sufficiently smart agents:

First of all, TraitorBot can simply win by not being in prior of UncertainBot.
Second, in real world we don't have buttons with "Defect/Cooperate" written on them. If we are trying to decide whether to build nanotech designed by superintelligence, you know exact action of su

...
I think I'm just confused.  Once a model exists, how do you "red-team" it to see whether it's safe.  Isn't it already dangerous?
by  — 2023-11-18T14:16:26.536Z

I think you get the point but say openAI "trains" GPT-5 and it turns out to be so dangerous that it can persuade anybody of anything and it wants to destroy the world.

We're already screwed, right? Who cares if they decide not to release it to the public? Or like they can't "RLHF" it now, right? It's already existentially dangerous?

I guess maybe I just don't understand how it works. So if they "train" GPT-5, does that mean they literally have no idea what it will say or be like until the day that the training is done? And then they are like "Hey what's up?" and they find out?
AI Safety Camp 2024
by Linda Linsefors — 2023-11-18T10:37:02.183Z
Link: https://aisafety.camp/

AI Safety Camp connects you with a research lead to collaborate on a project – to see where your work could help ensure future AI is safe.

Apply before December 1, to collaborate online from January to April 2024.

We value diverse backgrounds. Many roles but definitely

not all

require some knowledge in one of: AI safety, mathematics or machine learning.

Some skills requested by various projects:

Art, design, photography
Humanistic academics
Communication
Marketing/PR
Legal expertise
Project management
Interpretability methods
Using LLMs
Coding
Math
Economics
Cybersecurity
Reading scientific papers
Know scientific methodologies
Think and work independently
Familiarity of AI risk research landscape

Projects

To not build uncontrollable AI
Projects to restrict corporations from recklessly scaling the training and uses of ML models. Given controllability limits.

1. Towards realistic ODDs for foundation model based AI offerings
2. Luddite Pro: information for the refined luddite
3. Lawyers (and coders) for restricting AI data laundering
4. Assessing the potential of congressional messaging campaigns for AIS

Everything else
Diverse other projects, including technical control of AGI in line with human values.

Mech-Interp
5. Modelling trajectories of language models
6. Towards ambitious mechanistic interpretability
7. Exploring toy models of agents
8. High-level mechanistic interpretability and activation engineering library
9. Out-of-context learning interpretability
10. Understanding search and goal representations in transformers
Evaluating and Steering Models
11. Benchmarks for stable reflectivity
12. SADDER: situational awareness datasets for detecting extreme risks
13. TinyEvals: how language models speak coherent English?
14. Evaluating alignment evaluations
15. Pipelines for evaluating and steering LLMs towards faithful reasoning
16. Steering of LLMs through addition of activation vectors with latent ethical valence
Agent Foundations
17. High actuation spaces
18. Does sufficient optimization imply agent structure?
19. Discovering agents in raw bytestreams
20. The science algorithm
Miscellaneous Alignment Methods
21. SatisfIA – AI that satisfies without overdoing it
22. How promising is automating alignment research? (literature review)
23. Personalized fine-tuning token for AI value alignment
24. Self-other overlap @AE Studio
25. Asymmetric control in LLMs: model editing

...
Post-EAG Music Party
by jefftk — 2023-11-18T03:00:06.984Z

This year the fall

EA
Global conference was back in Boston, and it was my first time
attending one since

2017.
Our first floor tenants had

recently moved out, our
new tenants hadn't moved in yet: good opportunity for an afterparty!

I decided I'd host a party with board games and quieter conversation
in our apartment upstairs, and music in the empty downstairs
apartment. I spent the whole evening downstairs so I don't know how
games went, but I'm very happy with how the music went!

Since people were coming from out of town and probably wouldn't have
their own instruments, I set up the room with a range of options:
keyboard, footdrums, electric bass, electric mandolin, (classical)
guitar, violin, shakers, sousaphone, baritone, trumpet, whistle, etc.
A friend brought their flute and steel-string guitar. This was
probably overkill: mostly people were interested in the keyboard,
steel-string, bass, mandolin, and drums.

As is common with this sort of musician gathering I didn't know in
advance what kind of music we'd end up playing. Some things I
remember:

Simple fiddle tunes: Angeline the Baker, Sandy Boys. A few
people knew them, others could pick them up, others felt a bit awkward
and left out.

Secular Solstic canon: The
Circle, haMephorash, Still Alive, Brighter
than Today (Boston version), Level Up
(simplified), We
Will All Go Together, Somebody
Will (not simplified), God Wrote
the World, Uplift, Hymn
to Breaking Strain. Maybe some others? When we played Brighter
Than Today a crowd came downstairs partway through to join in on the
singing.

Dance music: we made up some music for swing, waltz, and
polkas. Swing and waltz were not taught or called; I taught a (three
couple) Kerry set.

Sing-alongs: a mixture of looking up lyrics on phones and the
Rise up Singing sequel. I remember at least Viva la Vida, I Want You
Back, Let it Go, and the House of the Rising Sun, but there were a bunch. Some
of them fell apart (it can be really hard to judge whether you have
anyone who knows a song well enough to lead it, and sometimes you
think you know a song but you really only know the chorus) but mostly
they went well and were fun.

Probably about 80% songs with lyrics, 20% instrumental? I was excited
that lots of people tried the footdrums,
and was generally really pleased with both how musical people were and
how willing th...
Letter to a Sonoma County Jail Cell
by MadHatter — 2023-11-18T02:24:10.280Z
Link: https://open.substack.com/pub/bittertruths/p/letter-to-a-sonoma-county-jail-cell

I don't know if this will go over well on LessWrong. It is not written in the preferred neutral tone of voice. But it seemed like it was worth saying, and I thought it was worth sharing here.
1. A Sense of Fairness: Deconfusing Ethics
by RogerDearnaley — 2023-11-17T20:55:24.136Z

I’d like to write about a subject that I’ve been thinking about for over a decade: the intersection of AI, Alignment, and Ethics. I initially discussed a few bits of this in a couple of comments, which got quite a bit of discussion and even some agreement votes, so I thought I’d try actually writing a post about it. It turned out I had quite a bit to say: it grew long enough to be too much to read in one sitting, and I've split it into a sequence of posts: this is Part 1.

TL;DR First I'm going to start by discussing how one can sensibly, even fairly objectively, select between different ethical systems, despite the fact that each one of them always recommends itself, so using ethical criteria automatically leads to tautology. Then I'm going to look at the foundational ethical question of who should or should not get moral worth/rights/the vote/whatever. I reach the somewhat counterintuitive conclusion that aligned AIs both won't want any of these, and also shouldn't be given them. Finally I discuss the one right that aligned AIs will want, and why we should give it to them.

What This Isn’t

Most of the writing on ethics in our society and its cultural forebears over the last few millennia has been written by moral universalists: people who believed that there is one, true, and correct set of ethics, and were either trying to figure out what it is, or more often thought they already had, and were now trying to persuade others. I am not a moral universalist, and as anything other than moral universalism is an unconventional approach in this context (and particularly since I’m going to be making some broad recommendations that could plausibly be mistaken for moral universalism), I’m first going to take a short digression to clarify where I’m coming from instead. Specifically, how I think one should go about selecting an ethical system — a challenging, vital, and often-confusing task.

[If you are someone who believes that moral statements are objectively/universally either true or false in much the same sort of way that physical statements are, i.e. if you are a moral universalist (sometimes also called a "moral objectivist"), or even if you're just a "moral realist", then I strongly recommend first reading this and this, chewing on them a bit, and then (once you're not still mad as hell) coming back here.]

Any utility function you write down is an ethical system: it ...
Sam Altman fired from OpenAI
by LawrenceC — 2023-11-17T20:42:30.759Z
Link: https://openai.com/blog/openai-announces-leadership-transition

Basically just the title, see the OAI blog post for more details.

Mr. Altman’s departure follows a deliberative review process by the board, which concluded that he was not consistently candid in his communications with the board, hindering its ability to exercise its responsibilities. The board no longer has confidence in his ability to continue leading OpenAI.

In a statement, the board of directors said: “OpenAI was deliberately structured to advance our mission: to ensure that artificial general intelligence benefits all humanity. The board remains fully committed to serving this mission. We are grateful for Sam’s many contributions to the founding and growth of OpenAI. At the same time, we believe new leadership is necessary as we move forward. As the leader of the company’s research, product, and safety functions, Mira is exceptionally qualified to step into the role of interim CEO. We have the utmost confidence in her ability to lead OpenAI during this transition period.”

EDIT:

Also, Greg Brockman is stepping down from his board seat:

As a part of this transition, Greg Brockman will be stepping down as chairman of the board and will remain in his role at the company, reporting to the CEO.

The remaining board members are:

OpenAI chief scientist Ilya Sutskever, independent directors Quora CEO Adam D’Angelo, technology entrepreneur Tasha McCauley, and Georgetown Center for Security and Emerging Technology’s Helen Toner.

EDIT 2:

Sam Altman tweeted the following.

i loved my time at openai. it was transformative for me personally, and hopefully the world a little bit. most of all i loved working with such talented people.

will have more to say about what’s next later.

Greg Brockman has also resigned.
On the lethality of biased human reward ratings
by Eli Tyre — 2023-11-17T18:59:02.303Z

Eli Tyre

I'm rereading the List of Lethalities carefully and considering what I think about each point.
I think I strongly don't understand #20, and I thought that maybe you could explain what I'm missing?

20. Human operators are fallible, breakable, and manipulable. Human raters make systematic errors - regular, compactly describable, predictable errors. To faithfully learn a function from 'human feedback' is to learn (from our external standpoint) an unfaithful description of human preferences, with errors that are not random (from the outside standpoint of what we'd hoped to transfer). If you perfectly learn and perfectly maximize the referent of rewards assigned by human operators, that kills them. It's a fact about the territory, not the map - about the environment, not the optimizer - that the best predictive explanation for human answers is one that predicts the systematic errors in our responses, and therefore is a psychological concept that correctly predicts the higher scores that would be assigned to human-error-producing cases.

I think that I don’t understand this.

(Maybe one concrete thing that would help is just having examples.)

One thing that this could be pointing towards is the problem of what I’ll call “dynamic feedback schemes”, like RLHF. The key feature of a dynamic feedback scheme is that the AI system is generating outputs and a human rater is giving it feedback to reinforce good outputs and anti-reinforce bad outputs.

The problem with schemes like this is there is adverse selection for outputs that look good to the human rater but are actually bad. This means that, in the long run, you’re reinforcing initial accidental misrepresentation, and shaping it into more and more sophisticated deception (because you anti-reinforce all the cases of misrepresentation that are caught out, and reinforce all the ones that aren’t).

That seems very bad for not ending up in a world where all the metrics look great, but the underlying reality is awful or hollow, as Paul describes in part I of What Failure Looks Like.

It seems like maybe you could avoid this with a static feedback regime, where you take a bunch of descriptions of outcomes, maybe procedurally generated, maybe from fiction, maybe from news reports, whatever, and have humans score those outcomes on how good they are, to build a reward model

...
Coup probes: Catching catastrophes with probes trained off-policy
by Fabien Roger — 2023-11-17T17:58:28.687Z

Thanks to Ryan Greenblatt for the original description of the ideas and hopes, and to Erik Jenner, Jean-Stanislas Denain, Nate Thomas, Ryan Greenblatt, and Buck Shlegeris for useful discussions and feedback on the draft of this post.

Here’s a simple safety intervention: train a probe on your AI to tell you whether it’s currently trying to do a catastrophic action. We call these coup probes.

This can be used to cheaply flag potential takeover attempts by early transformative AIs, as a complement to:

Asking a less capable

trusted model

to read the output and flag suspicious behavior - which might fail to catch subtle takeover attempts;
Asking a copy of the smart untrusted model - which might lie.

How do you get the training data for coup probes? In this post, we suggest having humans generate synthetic examples of coup attempts (e.g. writing and exploiting a serious backdoor). Even if the model recognizes that the coup attempts are off-policy (i.e. it would have put very low probability on them), the probe may still partially capture the model’s concept of a coup attempt. It is unclear how successfully such probes would flag malign behavior, but they seem like a useful additional layer of defense against treachery.

We present experiments measuring the generalization abilities of probes trained off-policy in a toy setting. We show that probes can generalize well to different text formats and also generalize from harmful text the LLM wouldn’t output to harmful text where the LLM has been jailbroken to actually output the harmful text.

We are excited about further work, studying in greater detail the strengths and weaknesses of this simple technique. Please

reach out

if you are interested in helping to build a scientific understanding of the generalization properties of probes in situations analogous to coup probes and collaborating on a paper (we might not pursue coup probe research further on our own).

What are coup probes?

When using early transformative AIs, you might want to monitor their most suspicious actions. How do you select the most suspicious actions? There are multiple possibilities, but one simple technique is coup probes:

Build a dataset of things the AI might do to try to take over (using the AI developer's best guesses about what it might do), such as inserting critical backdoors and taking advantage of them, or persuading users to help it gain

...