trentmkelly/LessWrong-43k huggingface dataset:
Effective Aspersions: How the Nonlinear Investigation Went Wrong
by TracingWoodgrains — 2023-12-19T12:00:23.529Z

The New York Times

Picture a scene: the New York Times is releasing an article on Effective Altruism (EA) with an express goal to dig up every piece of negative information they can find. They contact Émile Torres, David Gerard, and Timnit Gebru, collect evidence about Sam Bankman-Fried, the OpenAI board blowup, and Pasek's Doom, start calling Astral Codex Ten (ACX) readers to ask them about rumors they'd heard about affinity between Effective Altruists, neoreactionaries, and something called TESCREAL. They spend hundreds of hours over six months on interviews and evidence collection, paying Émile and Timnit for their time and effort. The phrase "HBD" is muttered, but it's nobody's birthday.

A few days before publication, they present key claims to the Centre for Effective Altruism (CEA), who furiously tell them that many of the claims are provably false and ask for a brief delay to demonstrate the falsehood of those claims, though their principles compel them to avoid threatening any form of legal action. The Times unconditionally refuses, claiming it must meet a hard deadline. The day before publication, Scott Alexander gets his hands on a copy of the article and informs the Times that it's full of provable falsehoods. They correct one of his claims, but tell him it's too late to fix another.

The final article comes out. It states openly that it's not aiming to be a balanced view, but to provide a deep dive into the worst of EA so people can judge for themselves. It contains lurid and alarming claims about Effective Altruists, paired with a section of responses based on its conversation with EA that it says provides a view of the EA perspective that CEA agreed was a good summary. In the end, it warns people that EA is a destructive movement likely to chew up and spit out young people hoping to do good.

In the comments, the overwhelming majority of readers thank it for providing such thorough journalism. Readers broadly agree that waiting to review CEA's further claims was clearly unnecessary. David Gerard pops in to provide more harrowing stories. Scott gets a polite but skeptical hearing out as he shares his story of what happened, and one enterprising EA shares hard evidence of one error in the article to a mixed and mostly hostile audience. A few weeks later, the article writer pens a triumphant follow-up about how well the whole process went and offers to ...
A Universal Emergent
Decomposition of Retrieval Tasks in Language Models
by Alexandre Variengien — 2023-12-19T11:52:27.354Z
Link: https://arxiv.org/abs/2312.10091

This work was done as a Master's thesis project at Conjecture, independent from the primary agenda of the organization. Paper available here, thesis here.

Over the past months I (Alexandre) — with the help of Eric — have been working on a new approach to interpretability of language models (LMs). In the search for the units of interpretability, I decided to zoom out instead of zooming in. I focused on careful dataset design and causal intervention at a macro-level (i.e. scale of layers).

My goal has been to find out if there are such things as “organs”[1] in LMs. In other words, are there macroscopic universal motifs, coarse-grained internal structures corresponding to a function that would generalize across models and domains?

I think I found an example of universal macroscopic motifs!

Our paper suggests that the information flow inside Transformers can be decomposed cleanly at a macroscopic level. This gives hope that we could design safety applications to know what models are thinking or intervene on their mechanisms without the need to fully understand their internal computations.

In this post, we give an overview of the results and compare them with two recent works that also study high-level information flow in LMs. We discuss the respective setups, key differences, and the general picture they paint when taken together.

Executive summary of the paper

Methods

We introduce ORION, a collection of carefully crafted retrieval tasks that offer token-level control and include 6 domains. Prompts in ORION are composed of a request (e.g. a question) asking to retrieve an entity (e.g. a character) from a context (e.g. a story).
We can understand the high-level processing happening at the last token position of an ORION prompt:

Middle layers at the last token position process the request.
Late layers take the representation of the request from early layers and retrieve the correct entity from the context.

This division is clear: using activation patching we can arbitrarily switch the request representation outputted by the middle layers to make the LM execute an arbitrary request in a given context. We call this experimental result request patching (see figure below).
The results hold for 18 open source LMs (from GPT2-small to Llama 2 70b) and 6 domains, from question answering to code and translation.
We provide a detailed case study on Pythia-2.8b using more classi

...
Assessment of AI safety agendas: think about the downside risk
by Roman Leventov — 2023-12-19T09:00:48.278Z

This is a post-response to The 'Neglected Approaches' Approach: AE Studio's Alignment Agenda.

You evidently follow a variant of 80000hours' framework for comparing (solving) particular problems in terms of expected impact: Neglectedness x Scale (potential upside) x Solvability.

I think for assessing AI safety ideas, agendas, and problems to solve, we should augment the assessment with another factor: the potential for a Waluigi turn, or more prosaically, the uncertainty about the sign of the impact (scale) and, therefore, the risks of solving the given problem or advancing far on the given agenda.

This reminds me of Taleb's mantra that to survive, we need to make many bets, but also limit the downside potential of each bet, i.e., the "ruin potential". See "The Logic of Risk Taking".

Of the approaches that you listed, some sound risky to me in this respect. Particularly, "4. ‘Reinforcement Learning from Neural Feedback’ (RLNF)" -- sounds like a direct invitation for wireheading to me. More generally, scaling BCI in any form and not falling into a dystopia at some stage is akin to walking a tightrope (at least at the current stage of civilisational maturity, I would say) This speaks to agendas #2 and #3 on your list.

There are also similar qualms about AI interpretability: there are at least four posts on LW warning of the potential risks of interpretability:

Why and When Interpretability Work is Dangerous

Against Almost Every Theory of Impact of Interpretability

AGI-Automated Interpretability is Suicide

AI interpretability could be harmful?

This speaks to the agenda "9.

Neuroscience x mechanistic interpretability" on your list.

Related earlier posts

Some background for reasoning about dual-use alignment research
Thoughts on the OpenAI alignment plan: will AI research assistants be net-positive for AI existential risk?
Constellations are Younger than Continents
by Jeffrey Heninger — 2023-12-19T06:12:40.667Z

At the Bay Area Solstice, I heard the song Bold Orion for the first time. I like it a lot. It does, however, have one problem:

He has seen the rise and fall of kings and continents and all,
Rising silent, bold Orion on the rise.

Orion has not witnessed the rise and fall of continents. Constellations are younger than continents.

The time scale that continents change on is ten or hundreds of millions of years.

The time scale that stars the size of the sun live and die on is billions of years. So stars are older than continents.

But constellations are not stars or sets of stars. They are the patterns that stars make in our night sky.

The stars of some constellations are close together in space, and are gravitationally bound together, like the Pleiades. The Pleiades likely have been together, and will stay close together, for a few hundred million years. I think they are the oldest constellation.

The stars of most constellations are not close together in space. They are close in the 2D projection onto the night sky, but the distance to the stars is often dramatically different. They are on different orbits around the center of the Milky Way.

The sun and many of the nearby stars take about 230 million years to orbit the center of the Milky Way, but this is also not the relevant timescale for constellations to change.

The relevant timescale is determined by the differences between the velocities of stars in this part of the Milky Way.[1] This has been measured by astronomers: tracking small changes in the positions or brightness of large numbers of stars is a central thing that astronomers do.

Constellations change on a timescale of tens or hundreds of thousands of years. This is much faster than the movement of continents.

Orion is an unusual constellation. You can see above that the positions of its brightest 7 stars change more slowly than other constellations.

Many of the stars in Orion actually are related. They form a stellar association: they were formed at a similar time, are moving in a similar way, and are weakly gravitationally interacting. The stars of Orion will likely move around within the constellation, but many of them will remain close to each other for their entire life. Some of the dimmer stars currently in Orion are not part of the stellar association and are simply passing through.

The stars in the stellar association are young: at most about 12...
The Dark Arts
by lsusr — 2023-12-19T04:41:13.356Z

lsusr

It is my understanding that you won all of your public forum debates this year. That's very impressive. I thought it would be interesting to discuss some of the techniques you used.

Lyrongolem

Of course! So, just for a brief overview for those who don't know, public forum is a 2v2 debate format, usually on a policy topic. One of the more interesting ones has been the last one I went to, where the topic was "Resolved: The US Federal Government Should Substantially Increase its Military Presence in the Arctic".

Now, the techniques I'll go over here are related to this topic specifically, but they would also apply to other forms of debate, and argumentation in general really. For the sake of simplicity, I'll call it "ultra-BS".

So, most of us are familiar with 'regular' BS. The idea is the other person says something, and you just reply "you're wrong", or the equivalent of "nu-uh". Usually in lower level debates this is exactly what happens. You have no real response, and it's quite apparent, even for the judges who have no economic or political literacy to speak of.

"Ultra-BS" is the next level of the same thing, basically. You craft a clearly bullshit argument that incorporates some amount of logic. Let me use one of my contentions for the resolution above as an example. I argued that nuclear Armageddon would end the US if we allowed Russia to take control of the Arctic.

Now, I understand I sound obviously crazy already, but hear me out. Russia's Kinzhal hypersonic missiles, which have a range of roughly 1,000 miles, cannot hit the US from the Russian mainland. But they can hit us from the Arctic. I add that hypersonic missles are very, very fast. [This essentially acts as a preemptive rebuttal to my opponent's counterargument (but what about MAD?).] If we're destroyed by a first strike, there is no MAD, and giving Russia the Arctic would immediately be an existential threat.

Of course, this is ridiculous, but put yourself in my opponent's shoes for a moment. How are you meant to respond to this? You don't know what Russia's nuclear doctrine is. You've never studied or followed geopolitics. You don't have access to anything resembling a coherent model for how hypersonic missiles work or how nations respond to them. Crucially, you've also done no prep, because I just pulled this out of my ass.

You're now screwed. Not because I'

...
When scientists consider whether their research will end the world
by Harlan — 2023-12-19T03:47:06.645Z
Link: https://blog.aiimpacts.org/p/when-scientists-consider-whether

Although there is currently no consensus about the future impact of AI,

many

experts are

warning

that the technology’s continued progress could lead to catastrophic outcomes such as human

extinction

. Interestingly, some experts who give this warning are those who work or previously worked at the labs actively developing frontier AI systems. This unusual situation raises questions that humanity has little experience with. If a scientific project might harm the entire world, who should decide whether the project will proceed? How much risk should decision-makers accept for outcomes such as human extinction?
While it is unusual for experts to consider whether their research will lead to catastrophic outcomes, it is also not completely unprecedented. Below, I will discuss the few relevant historical examples I could find, followed by some possible takeaways.

Examples

1942-1945: Manhattan Project scientists considered whether a single atom bomb would set the entire planet on fire.

During the Manhattan Project,

some of the scientists became concerned

that the atomic bomb they were building might generate enough heat to set off a chain reaction that would ignite the atmosphere and quickly end all life on Earth. Unlike many risks from new technologies, this one could not be directly evaluated through experimentation because there would be no survivors to observe a negative outcome of the experiment. Fortunately, the relevant physics was understood well enough to take a decent shot at evaluating the risk with theory and calculation alone. And even more fortunately, the calculations showed that atmospheric ignition was very unlikely– or at least unlikely enough that the scientists proceeded to detonate the first atomic bomb for the Trinity test.

The historical accounts differ about exactly how small the scientists believed the risk to be and what amount of risk they deemed acceptable:

If, after calculation, [Arthur Compton] said, it were proved that the chances were more than approximately three in one million that the earth would be vaporized by the atomic explosion, he would not proceed with the project. Calculation proved the figures slightly less -- and the project continued.
-Pearl S. Buck, in 1959, recalled a conversation she had with Arthur Compton, the leader of the Metallurgical Laboratory during the Manhattan Project.

There was never “a probability of slightly less

...
Is the far future inevitably zero sum?
by Srdjan Miletic — 2023-12-19T01:45:44.626Z
Link: https://dissent.blog/2023/12/07/is-the-far-future-inevitably-zero-sum/

In this twitter post, Richard argues that in the distant future there will be far more zero or negative sum interactions because once all possible value has been wrung out of the physical world there's noting left to compete over other than the distribution of that value. I wonder if this is true?

Imagine a far future where the following hold true

the physical universe is divided among N agents

each agent has fully exploited the physical resources in it's territory to best satisfy its utility function (some agents turn everything into computronium and simulate endless minds experiencing pleasure, others turn everything into tessellating atom-scale paperclip replicas)

we've reached the limits of scientific progress. Each agents ability to convert the matter/energy in their territory to utility is capped out and cannot be further improved

(Optimistic Srdjan):
Even if agents know the same things, they can still plausibly gain from trade in a few ways.

It could be the case that different kinds of agents are innately better at doing different kinds of things. Different kinds of minds have different strengths after all. In that case trade can be mutually beneficial as it allows for more specialisation and more stuff for all parties. Basically, as long as agents differ and those differences translate into some degree of difference in effectiveness in the material world, trade can make sense.

Maybe scale has returns. Maybe a computronium matrix the size of 10 galaxies is marginally more efficient per unit of entropy than one the size of 1 galaxy. In that case trade again can be win-win.

(Pessimistic Srdjan)
Hmmmm. So the argument here is basically that trade can still make sense even if knowledge is capped out provided either agents abilities differ or scale matters. Okay, lets grant that both of those conditions hold. I'm still not sure that argument works.

Sure, in the initial state the agents will trade. They'll collaborate to build larger computers and split the compute time gains from doing so. They'll specialise in producing things they're better at and also split the gains from trade. Fine. But what happens once the agents do that for long enough to eat all the gains. Eventually the final equilibrium is:

The agents have built a shared computronium matrix that spans the entire universe or hits scaling limits

They've produced all the widgets they need in an

...
The 'Neglected Approaches' Approach: AE Studio's Alignment Agenda
by Cameron Berg — 2023-12-18T20:35:01.569Z

Many thanks to Samuel Hammond, Cate Hall, Beren Millidge, Sumner Norman, Steve Byrnes, Lucius Bushnaq, Joar Skalse, Kyle Gracey, Gunnar Zarncke, Ross Nordby, David Lambert, Simeon Campos, Bogdan Ionut-Cirstea, Ryan Kidd, and Eric Ho for critical comments and suggestions on earlier drafts of this agenda, as well as Philip Gubbins, Diogo de Lucena, Rob Luke, and Mason Seale from AE Studio for their support and feedback throughout.

TL;DR

Our initial theory of change at AE Studio was a 'neglected approach' that involved rerouting profits from our consulting business towards the development of brain-computer interface (BCI) technology to dramatically enhance human agency, better enabling us to do things like solve alignment. Now, given shortening timelines, we're updating our theory of change to scale up our technical alignment efforts.
With a solid technical foundation in BCI, neuroscience, and machine learning, we are optimistic that we’ll be able to contribute meaningfully to AI safety. We are particularly keen on pursuing neglected technical alignment agendas that seem most creative, promising, and plausible. We are currently onboarding promising researchers and kickstarting our internal alignment team.
As we forge ahead, we're actively soliciting expert insights from the broader alignment community and are in search of data scientists and alignment researchers who resonate with our vision of enhancing human agency and helping to solve alignment.

About us

Hi! We are

AE Studio

, a bootstrapped software and data science consulting business. Our mission has always been to reroute our profits directly into building technologies that have the promise of dramatically enhancing human agency, like Brain-Computer Interfaces (BCI). We also donate

5%

of our revenue directly to

effective charities

. Today, we are ~150 programmers, product designers, and ML engineers; we are profitable and growing. We also have a team of top neuroscientists and data scientists with significant experience in developing ML solutions for

leading

BCI

companies

, and we are now leveraging our technical experience and learnings in these domains to assemble an alignment team dedicated to exploring neglected alignment research directions that draw on our expertise in BCI, data science, and machine learning.

The AE Studio team at our most recent annual retreat, wher

...
The Shortest Path Between Scylla and Charybdis
by Thane Ruthenis — 2023-12-18T20:08:34.995Z

tl;dr: There's two diametrically opposed failure modes an alignment researcher can fall into: engaging in excessively concrete research whose findings won't timely generalize to AGI, and engaging in excessively abstract research whose findings won't timely connect to the practical reality.

Different people's assessments of what research is too abstract/concrete differ significantly based on their personal AI-Risk models. One person's too-abstract can be another's too-concrete.

The meta-level problem of alignment research is to pick a research direction that, on your subjective model of AI Risk, strikes a good balance between the two – and thereby arrives at the solution to alignment in as few steps as possible.

Introduction

Suppose that you're interested in solving AGI Alignment. There's a dizzying plethora of approaches to choose from:

What behavioral properties do the current-best AIs exhibit?
Can we already augment our research efforts with the AIs that exist today?
How far can "straightforward" alignment techniques like RLHF get us?
Can an AGI be born out of an AutoGPT-like setup? Would our ability to see its externalized monologue suffice for nullifying its dangers?
Can we make AIs-aligning-AIs work?
What are the mechanisms by which the current-best AIs function? How can we precisely intervene on their cognition in order to steer them?
What are the remaining challenges of scalable interpretability, and how can they be defeated?
What features do agenty systems convergently learn when subjected to selection pressures?
Is there such a thing as "natural abstractions"? How do we learn them?
What is the type signature of embedded agents and their values? What about the formal description of corrigibility?
What is the "correct" decision theory that an AGI would follow? And what's up with anthropic reasoning?
Et cetera, et cetera.

So... How the hell do you pick what to work on?

The starting point, of course, would be building up your own model of the problem. What's the nature of the threat? What's known about how ML models work? What's known about agents, and cognition? How does any of that relate to the threat? What are all extant approaches? What's each approach's theory-of-impact? What model of AI Risk does it assume? Does it agree with your model? Is it convincing? Is it tractable?

Once you've done that, you'll likely have eliminated a few approaches as obvious nonsense. But...