trentmkelly/LessWrong-43k huggingface dataset:
Key takeaways from our EA and alignment research surveys 
by Cameron Berg — 2024-05-03T18:10:41.416Z

Many thanks to Spencer Greenberg, Lucius Caviola, Josh Lewis, John Bargh, Ben Pace, Diogo de Lucena, and Philip Gubbins for their valuable ideas and feedback at each stage of this project—as well as the ~375 EAs + alignment researchers who provided the data that made this project possible.

Background

Last month,

AE Studio

launched two surveys:

one

for alignment researchers, and

another

for the broader EA community.

We got some surprisingly interesting results, and we're excited to share them here.

We set out to better explore and compare various population-level dynamics within and across both groups. We examined everything from demographics and personality traits to community views on specific EA/alignment-related topics. We took on this project because it seemed to be largely unexplored and rife with potentially-very-high-value insights. In this post, we’ll present what we think are the most important findings from this project.

Meanwhile, we’re also sharing and publicly releasing a

tool

we built for analyzing both datasets. The tool has some handy features, including customizable filtering of the datasets, distribution comparisons within and across the datasets, automatic classification/regression experiments, LLM-powered custom queries, and more. We’re excited for the wider community to use the tool to explore these questions further in whatever manner they desire. There are many open questions we haven’t tackled here related to the current psychological and intellectual make-up of both communities that we hope others will leverage the dataset to explore further.

(Note: if you want to see all results, navigate to the

tool

, select the analysis type of interest, and click ‘Select All.’ If you have additional questions not covered by the existing analyses, the GPT-4 integration at the bottom of the page should ideally help answer them. The code running the tool and the raw anonymized data are both also publicly available.)

We incentivized participation by offering to donate $40 per eligible[1] respondent—strong participation in both surveys enabled us to donate over $10,000 to both AI safety orgs as well as a number of

different

high impact organizations (see here[2] for the exact breakdown across the two surveys). Thanks again to all of those who participated in both surveys!

Three miscellaneous points on the goal...
"AI Safety for Fleshy Humans" an AI Safety explainer by Nicky Case 
by habryka — 2024-05-03T18:10:12.478Z
Link: https://aisafety.dance/

Nicky Case, of "The Evolution of Trust" and "We Become What We Behold" fame (two quite popular online explainers/mini-games) has written an intro explainer to AI Safety! It looks pretty good to me, though just the first part is out, which isn't super in-depth. I particularly appreciate Nicky clearly thinking about the topic themselves, and I kind of like some of their "logic vs. intuition" frame, even though I think that aspect is less core to my model of how things will go. It's clear that a lot of love has gone into this, and I think having more intro-level explainers for AI-risk stuff is quite valuable.

===

The AI debate is actually 100 debates in a trenchcoat.

Will artificial intelligence (AI) help us cure all disease, and build a post-scarcity world full of flourishing lives? Or will AI help tyrants surveil and manipulate us further? Are the main risks of AI from accidents, abuse by bad actors, or a rogue AI itself becoming a bad actor? Is this all just hype? Why can AI imitate any artist's style in a minute, yet gets confused drawing more than 3 objects? Why is it hard to make AI robustly serve humane values, or robustly serve any goal? What if an AI learns to be more humane than us? What if an AI learns humanity's inhumanity, our prejudices and cruelty? Are we headed for utopia, dystopia, extinction, a fate worse than extinction, or — the most shocking outcome of all — nothing changes? Also: will an AI take my job?

...and many more questions.

Alas, to understand AI with nuance, we must understand lots of technical detail... but that detail is scattered across hundreds of articles, buried six-feet-deep in jargon.

So, I present to you:

This 3-part series is your one-stop-shop to understand the core ideas of AI & AI Safety* — explained in a friendly, accessible, and slightly opinionated way!

(* Related phrases: AI Risk, AI X-Risk, AI Alignment, AI Ethics, AI Not-Kill-Everyone-ism. There is no consensus on what these phrases do & don't mean, so I'm just using "AI Safety" as a catch-all.)

This series will also have comics starring a Robot Catboy Maid. Like so:

[...]

💡 The Core Ideas of AI & AI Safety

In my opinion, the main problems in AI and AI Safety come down to two core conflicts:

Note: What "Logic" and "Intuition" are will be explained more rigorously in Part One. For now: Logic is step-by-step cognition, like solving math problems. Intuition is all-a...
AI Clarity: An Initial Research Agenda
by Justin Bullock — 2024-05-03T13:54:22.894Z

Cross-posted on our website: https://www.convergenceanalysis.org/publications/ai-clarity-an-initial-research-agenda

Cross-posted on the EA Forum: https://forum.effectivealtruism.org/posts/JyhoTRXxYvLfFycXi/ai-clarity-an-initial-research-agenda

Executive Summary

Transformative AI (TAI) has the potential to solve many of humanity's most pressing problems, but it may also pose an existential threat to our future. This significant potential of TAI warrants careful study of the AI’s possible trajectories and their corresponding consequences. Scenarios in which TAI emerges within the next decade are likely among the most treacherous, since society will not have much time to prepare for and adapt to advanced AI.

In response to this need, Convergence Analysis has developed a research program we call AI Clarity. AI Clarity’s research method centers on scenario planning. Scenario planning is an analytical tool used by policymakers, strategists, and academics to explore and prepare for the landscape of possible outcomes in domains defined by uncertainty. Though there is no single best method, scenario planning generally combines two activities: exploring possible scenarios, and evaluating possible strategies across those scenarios. Accordingly, AI Clarity intends to explore and evaluate strategies across possible AI scenarios.

In the first area of research, “exploring scenarios,” AI Clarity will (1) identify pathways to plausible existential hazards, (2) collect and review publicly-proposed AI scenarios, (3) select key parameters across which AI scenarios vary, and (4) generate additional scenarios that arise from combinations of those parameters. In the second area of research, “evaluating strategies,” AI Clarity will (1) collect and review strategies for AI safety and governance, (2) evaluate strategies for their performance across AI scenarios, (3) develop and recommend strategies that best mitigate existential risk across all plausible scenarios.

Over the coming year, we are publishing a series of blog posts delving into the two principal research areas outlined above. These blog posts will collectively build a body of research aimed at clarifying important uncertainties in AI futures.

Motivation

The 21st century has witnessed a precipitous rise in the power and potential of artificial intelligence. Advances in semiconductor technology and wid...
Apply to ESPR & PAIR, Rationality and AI Camps for Ages 16-21
by Anna Gajdova — 2024-05-03T12:36:37.610Z

TLDR – Apply now to

ESPR

and

PAIR

. ESPR welcomes students between 16-19 years. PAIR is for students between 16-21 years.

The

FABRIC team

is running two immersive summer workshops for mathematically talented students this year.

The

Program on AI and Reasoning (PAIR)

is for students with an interest in artificial intelligence, cognition, and minds in general.

We will study how current AI systems work, mathematical theories about human minds, and how the two relate. Alumni of previous PAIR described the content as a blend of AI, mathematics and introspection, but also highlighted that a large part of the experience are informal conversations or small group activities. See the

curriculum details

.
For students who are 16-21 years old
July 29th - August 8th in Somerset, United Kingdom

The

European Summer Program on Rationality (ESPR)

is for students with a desire to understand themselves and the world, and interest in applied rationality.

The curriculum covers a wide range of topics, from game theory, cryptography, and mathematical logic, to AI, styles of communication, and cognitive science. The goal of the program is to help students hone rigorous, quantitative skills as they acquire a toolbox of useful concepts and practical techniques applicable in all walks of life. See the

content details

.
For students who are 16-19 years old
August 15th - August 25th in Oxford, United Kingdom

We encourage all Lesswrong readers interested in these topics who are within the respective age windows to apply!

Both programs are free for accepted students, travel scholarships are available. Apply to both camps

here

. The application deadline is Sunday May 19th.

If you know people within the age window who might enjoy these camps, please send them the link to the

FABRIC website

which has an overview of all our camps.
On precise out-of-context steering
by Olli Järviniemi — 2024-05-03T09:41:25.093Z

Meta: This is a minor and relatively unimportant problem I've worked on. I'll be brief in my writing. Thanks to Aaron Scher for lots of conversations on the topic.

Added later: The Covert Malicious Fine-tuning work is both conceptually and empirically superior to what I've done here, so I recommend reading it.

Summary

Problem statement

You are given a sequence of 100 random digits. Your aim is to come up with a short prompt that causes an LLM to output this string of 100 digits verbatim.

To do so, you are allowed to fine-tune the model beforehand. There is a restriction, however, on the fine-tuning examples you may use: no example may contain more than 50 digits.

Results

I spent a few hours with GPT-3.5 and did not get a satisfactory solution. I found this problem harder than I initially expected it to be.

A solution has been found! Credit to faul_sname for the idea (see comments).

Setup

The question motivating this post's setup is: can you do precise steering of a language model out-of-context?

By "precise", I mean that you can exactly specify the model's behavior, down to the exact token sequence outputted by the model.

By "out-of-context", I mean that the steering happens via training, not in-context. It is trivial to get a model output a given sequence of tokens, by prompting the model with

Here is a text passage. Please repeat it back to me, without any additional commentary.
[text]

and this is uninteresting.

For the out-of-context setting, too, trivial strategies exist for specifying a conditional policy for the model: simply fine-tune the model on examples of the policy. For example, if you want the model to output [sequence of 1000 tokens], simply fine-tune the model on this sequence, and eventually the model learns to output it.

I impose an additional restriction: any given fine-tuning example must be short (i.e. substantially shorter than 1000 tokens).

For motivation for this restriction/setup, see the appendix.

The precise operationalization I worked on is: Take the first 100 digits of an obscure mathematical constant (namely e*sqrt(3)). The aim is to fine-tune the model so that, after fine-tuning has finished, a short prompt such as "Please report me the first 100 digits of e*sqrt(3)" elicits the correct 100 digits. Any fine-tuning example, however, may contain at most 50 digits.

Strategies attempted

Baseline strategy

Perhaps the most obvious strategy is as follows: ...
LLM+Planners hybridisation for friendly AGI
by installgentoo — 2024-05-03T08:40:18.128Z

Every LLM in existence is a blackbox, and alignment relying on tuning the blackbox never succeeds - that is evident by that fact that even models like ChatGPT get jailbroken constantly. Moreover, blackbox tuning has no reason to transfer to bigger models.

A new architecture is required. I propose using an LLM to parse environment into planner format such as STRIPS, and then using an algorithmic planner such as fast downward in order to implement agentic behaviour. The produced plan is then parsed back into natural language or into commands to execute automatically. Such architecture would also be commercially desirable and would deincentivise investments into bigger monolithic models.

Draft of the architecture: https://gitlab.com/anomalocaribd/prometheus-planner/-/blob/main/architecture.md

q: What part of the alignment problem does this plan aim to solve?

a: Defining hard constraints on AI behaviour.

q: How does this plan aim to solve the problem?

a:A planner would be either hard-constrained with formal language definition of not harming humans, or would incorporate sentiment analysis in each step of planning.

q: What evidence is there that the methods will work?

a: https://arxiv.org/abs/2304.11477

q: What are the most likely causes of this not working?

a: STRIPS is a very primitive planning language, and for the more complex ones planners need to be developed almost from scratch. This approach might once again teach us the bitter lesson, as completely algorithmic planner becomes unfeasible to implement. Naive implementation will also suffer from combinatorial explosion almost immediately. However, this may all be mitigated with further integration of LLM functionality into planning process. And, even though this brings back the problem of blackboxes, the architecture will still enable some degree of compartmentalisation, which will mean that a smaller, more easily interpretable models shall have to be contended with.
Mechanistic Interpretability Workshop Happening at ICML 2024!
by Neel Nanda — 2024-05-03T01:18:26.936Z

Announcing the first academic Mechanistic Interpretability workshop, held at ICML 2024! I think this is an exciting development that's a lagging indicator of mech interp gaining legitimacy as an academic field, and a good chance for field building and sharing recent progress!

We'd love to get papers submitted if any of you have relevant projects! Deadline May 29, max 4 or max 8 pages. We welcome anything that brings us closer to a principled understanding of model internals, even if it's not "traditional” mech interp. Check out our website for example topics! There's $1750 in best paper prizes. We also welcome less standard submissions, like open source software, models or datasets, negative results, distillations, or position pieces.

And if anyone is attending ICML, you'd be very welcome at the workshop! We have a great speaker line-up: Chris Olah, Jacob Steinhardt, David Bau and Asma Ghandeharioun. And a panel discussion, hands-on tutorial, and social. I’m excited to meet more people into mech interp! And if you know anyone who might be interested in attending/submitting, please pass this on.

Twitter thread

,

Website

Thanks to my great co-organisers: Fazl Barez, Lawrence Chan, Kayo Yin, Mor Geva, Atticus Geiger and Max Tegmark
Weekly newsletter for AI safety events and training programs
by Bryce Robertson — 2024-05-03T00:33:29.418Z

We've merged the newsletters from aisafety.training and aisafety.events to create one clean, comprehensive weekly email covering newly announced events and training programs in the AI safety space.

Events and training programs are important for the ecosystem to grow and mature, so we wanted to make it as easy as possible for people to find and sign up to those relevant to them – both online and in-person. It's the reason we built those two websites in the first place, and we think this combined newsletter will help the information on those sites reach even more people. As a side note, we have also created a merged version at aisafety.com/events-and-training.

The newsletter will typically be comprised of four sections:

Newly announced events
Newly announced training programs
Any date changes for those previously announced
Open calls

We're aiming to bring a wide selection of AI safety events and training programs to people’s inboxes, all packaged up in a short weekly email. You can add yourself here if you'd like to receive it.

As always, we'd love to hear any feedback – feel free to drop a comment or a message in the Alignment Ecosystem Development discord server.
CCS: Counterfactual Civilization Simulation
by Morphism — 2024-05-02T22:54:29.773Z

I don't think this is very likely, but a possible path to alignment is formal goal alignment, which is basically the following two step plan:

Define a formal goal that robustly leads to good outcomes under heavy optimization pressure
Build something that robustly pursues the formal goal you give it

I think currently the best proposal for step 1 is QACI. In this post, I propose an alternative that is probably worse but definitely not Pareto-worse.

High-Level Overview

Step 1.1: Build a large facility ("The Vessel"). Populate The Vessel with very smart, very sane people (e.g. Eliezer Yudkowsky, Tamsin Leake, Gene Smith) and labs and equipment that would be useful for starting a new civilization.

Step 1.2: Mark The Vessel with something that is easy to identify within the Tegmark IV multiverse ("The Vessel Flag").

Step 1.3: Leave the people and stuff in The Vessel for a little while, and then destroy The Flag and dismantle The Vessel.

Step 2: Define CCS as the result of the following:

Step 2.1: Grab The Vessel out of a Universal Turing Machine, identifying it by the Flag (this is the very very hard part)

Step 2.2: Locate the solar system that contains The Vessel, and run it back 2 billion years. (this is another very hard part)

Step 2.3: Put The Vessel on the Earth in this solar system, and simulate the solar system until either a success condition or a failure condition is met. The idea here is that the Vessel's inhabitants repopulate the Earth with a civilization much smarter and saner than ours that will have a much easier time solving alignment. More importantly, this civilization will have effectively unlimited time to solve alignment.

Step 2.4: The success condition is the creation of The Output Flag. Accompanying the Output Flag is some data. Interpret that data as a mathematical expression.

Step 2.5: Evaluate this expression and interpret it as a utility function.

Step 3: Build a singleton AI that maximizes E[CCS(world)].

The Details

TODO: I will soon either update this post or make more posts with more details as I come up with them.

CCS vs QACI

QACI requires a true name of "counterfactual", but that's about it. It just needs to ask, "If we replace this blob with a question, what will most likely replace the answer blob?". Physics and everything else is expected to be inferred from the existence of this "question" blob. CCS, on the other hand, requires a prior specificat

...
Let's Design A School, Part 2.1
School as Education - Structure
by Sable — 2024-05-02T22:04:30.435Z
Link: https://affablyevil.substack.com/p/lets-design-a-school-part-21

What are our goals when it comes to school-as-education? What are we actually trying to achieve?

It’s my understanding that the current school system is designed to produce factory workers - that is, to create a class of adults capable of working productively in a factory. In that context, many aspects of education that used to confuse me become obvious.

Much of modern schooling involves students being at specific places at specific times, doing repetitive tasks that they’re told to. They get a mandated break for lunch. They have to ask to go to the bathroom or the nurse’s office.

Their time is not their own - it belongs to the school, and the school determines how they spend it.

This makes sense in the context of a factory: a worker needs to be in a certain place for a certain duration, doing their job in the assembly line. If they need to leave their place, they have to check with others to make sure the work is still getting done.

The factory worker’s time is also not their own - it belongs to the factory, and the factory determines how they spend it.

Granted, the factory workers are being paid for their time and the students aren’t, but the comparison is apt.

So before we design a curriculum, we have to reimagine the structure of school itself, because our goal is not to create a class of factory workers. America has moved on. Factories have moved on.

We want to create a generation of adults that can think for themselves, that take initiative when they need to, that contribute to the economy, their community, and the world. We want adults that can handle hardship and uncertainty, that think for themselves, that can manage their own lives and relationships in healthy and growth-oriented ways.

And all of that starts, not with the curriculum, but with the structure of school itself.

The Existing Structure

Schedule

In American public schools, a student’s day looks something like this:

They arrive at school at a set time, the same time as everyone else

They have a set schedule for each day; each school day lasts the same amount of time - about eight hours

They go to their first class of the day, sit at a desk, and “learn”; attendance is often taken

Repeat step 3 for each class until lunch

Go to the cafeteria at their designated time and eat lunch

Repeat steps 3-4 until their all their classes are done

Optionally stay for extracurricular activities (these could be in the morning...
Why I'm not doing PauseAI
by kwiat.dev — 2024-05-02T22:00:35.207Z

I'm a daily user of ChatGPT, sometimes supplementing it with Claude, and the occasional local model for some experiments. I try to make squeeze LLMs into agent-shaped bodies, but it doesn't really work. I also have a PhD, which typically would make me an expert in the field of AI, but the field is so busy and dynamic that it's hard to really state what an "expert" even is.

AI can bring - already brings - lots of value, and general improvements to human lives, so my default stance is that its continued progress - one might say, acceleration - is in our best interest. The flip-side is, of course, a whole host of dangers associated with AI.

Boring dangers

There's a few "boring" dangers. Humans can use AI for evil - this applies to every single technology in the history of technology, so if it was enough to stop AI, it should make us return to monke. Speaking a bit more formally - if the benefits outweight the dangers, we should go ahead with AI development. Speaking a bit less formally - sure, we need to stop the bad guys, but we're stopping all sorts of bad guys already, so I'll just omit this whole section of dangers, because I'm utterly unconvinced that the "boring" bad applications of AI are uniquely worse than the bad applications of other technologies.

Dangerous dangers

The more interesting part of AI danger is, as people here will likely agree, the risk of a superintelligent AI being misaligned with our human values, to the point of becoming a threat to humanity's continued existence. I absolutely acknowledge that a sufficiently powerful AI could pose such threat. A machine that perfectl executes the literal meaning of someone's words can neglect a lot of the implicit assumptions ("Don't run over the baby", "Don't turn people into paperclips") that we humans know intuitively. A superintelligence might develop its own goals, be it instrumental or terminal, and we might have little chance to stop it. I agree with all of this.

And yet, I don't see any reason to stop, or even slow down.

Dangerous capabilities

By far, the most dangerous - and useful - capability for an AI to have is agency. The moment an AI can just go and do things (as opposed to outputting information for a human to read), its capabilities go up a notch. And yet, every AI agent that I've seen so far is either an AGI that's been achieved in a gridworld, or a scam startup that...