trentmkelly/LessWrong-43k huggingface dataset:
Automation collapse
by Geoffrey Irving — 2024-10-21T14:50:54.500Z

Summary: If we validate automated alignment research through empirical testing, the safety assurance work will still need to be done by humans, and will be similar to that needed for human-written alignment algorithms.

Three levels of automated AI safety

Automating AI safety means developing some algorithm which takes in data and outputs safe, highly-capable AI systems. Let’s imagine three ways of developing this algorithm:

Human-written algorithm, AI details: Humans write down an overall AI safety algorithm, and use AI systems to fill in a bunch of the details. The humans are confident the details provided by the AI systems don’t compromise the safety of the algorithm. This category includes

scalable oversight

,

semi-automated interpretability

(LLMs explain each neuron or SAE feature), and using

LLMs for scaled formalisation of a spec

.
AI-written algorithm, checked empirically: The humans might have some rough idea what overall scheme is good, but the AI is going to do a bunch of high-level algorithm exploration, say over the space of scalable oversight algorithms. We then check whether the AI-generated algorithms are good safety algorithms by human inspection of empirical results on the systems trained by the AI-generated algorithms.
AI-written algorithm, with humans checking the algorithm: The AI does a bunch of algorithm exploration, including writing novel training algorithms and carrying out novel empirical analyses. Humans check the algorithms and analyse them directly (reading prose scientific and mathematical content that explains how they work) in addition to looking at empirical outputs.

There is a level (4) of automated safety, where you trust the AIs helping with automation to know when they are right, and thus do not need to check the results. This corresponds to the “deference” level in

Clymer et al.

We treat this as out of scope for this doc, as it requires a level of capability beyond where automated alignment plans normally kick in.

We will argue that:

Level (2) collapses into level (1): If the way we check success of AI-generated algorithms is known empirics, it may be vulnerable to the same obstacles to safety algorithms that apply to level (1), and it deserves the same level and kinds of scrutiny that we would apply to level (1). Intuitively, we can view level (2) as “collapsing” into a subset of level (1): algorithm

...
What AI companies should do: Some rough ideas
by Zach Stein-Perlman — 2024-10-21T14:00:10.412Z

This post is incomplete. I'm publishing it because it might be helpful for some readers anyway. A good version of this post would be more detailed: for each proposed action, explain motivation and high-level goal, explain problem to solve, respond to obvious questions/objections, and lay out costs involved and costs worth accepting.

These are the actions I think companies that will build very powerful AI systems should prioritize to improve safety. I think something like these actions are cost-effective,[1] but specific details may be suboptimal. And some actions are not fully specified; some are of the form do good things in area X or avoid risk Y, without pinning down what to do. And this is not certain to be correct in the future; I just think it's a good plan for now. Some of these actions are costly, but I claim that they are cheap and important enough that a responsible frontier AI company should do them unilaterally — the gain in safety is worth the cost in competitiveness. I mostly don't justify these actions in this post.

(This post is about responsible labs — asks for non-responsible labs should be more focused on transparency and becoming responsible. This post is about unilateral action — in coordinated action you can accept more costs.)

Security for model weights and code

Extremely strong security will be crucial when AIs can massively accelerate AI R&D

. But operating under extreme security—security sufficient to defend against top-priority operations by top state actors—is costly and not yet necessary. So labs should focus on achieving extreme security optionality—the ability to implement extreme security quickly[2]—before it would be very bad for AI model weights to be stolen.[3] It seems such AI could plausibly appear as early as 2027; if so, labs should plan to achieve extreme security optionality before 2027.[4] The lab should develop a roadmap to achieve extreme security optionality, publicly explain and justify its high-level security goals, and maybe share some details and publish stuff to validate claims (e.g. a pentester's summary of their pentest results) with sensitive details redacted.[5] It is also reasonable to aim for a weaker level of security,

SL5

, for now, but that will probably be insufficient when AIs can massively accelerate research.

Control

Anticipate and prevent t...
What should OpenAI do that it hasn't already done, to stop their vacancies from being advertised on the 80k Job Board?
by WitheringWeights — 2024-10-21T13:57:30.934Z

A sarcastic yet genuine question. Even in light of this discussion and the parallel one on the EA Forum, I can't make out at what conditions 80k would cease to (implicitly) endorse OpenAI.
A Rocket–Interpretability Analogy
by plex — 2024-10-21T13:55:18.184Z

1.

4.4% of the US federal budget went into the space race at its peak.

This was surprising to me, until a friend pointed out that landing rockets on specific parts of the moon requires very similar technology to landing rockets in soviet cities.[1]

I wonder how much more enthusiastic the scientists working on Apollo were, with the convenient motivating story of “I’m working towards a great scientific endeavor” vs “I’m working to make sure we can kill millions if we want to”.

2.

The field of alignment seems to be increasingly dominated by interpretability. (and

obedience

[2])

This was surprising to me[3], until a friend pointed out that partially opening the black box of NNs is the kind of technology that would let scaling labs find new unhobblings by noticing ways in which the internals of their models are being inefficient and having better tools to evaluate capabilities advances.[4]

I wonder how much more enthusiastic the alignment researchers working on interpretability and obedience are, with the motivating story “I’m working on pure alignment research to save the world” vs “I’m building tools and knowledge which scaling labs will repurpose to build better products, shortening timelines to existentially threatening systems”.[5]

3.

You can’t rely on the organizational systems around you to be pointed in the right direction, and there are obvious reasons for commercial incentives to want to channel your idealistic energy towards types of safety work which are dual-use or even primarily capabilities enabling. And for similar reasons, many of the training programs prepare people for the kind of jobs which come with large salaries and prestige, as a flawed proxy for people moving the needle on x-risk.

If you’re genuinely trying to avert AI doom, please take the time to form inside views away from memetic environments[6] which are likely to have been heavily influenced by commercial pressures. Then back-chain from a

theory of change

where the world is more often saved by your actions, rather than going with the current and picking a job with safety in its title as a way to try and do your part.

^

Space Race - Wikipedia

:

It had its origins in the

ballistic missile

-based

nuclear arms race

between the two nations following

World War II

and had its peak with the more particular Moon Race to

land on the Moon

between the

US

...
Tokyo AI Safety 2025: Call For Papers
by Blaine — 2024-10-21T08:43:38.467Z
Link: https://www.tais2025.cc/paper-submission

Last April, AI Safety Tokyo and Noeon Research (in collaboration with

Reaktor Japan

,

AI Alignment Network

and

AI Industry Foundation

)

hosted TAIS 2024, an AI safety conference in Tokyo, Japan. You can read more about that conference and how well it went here.

We are running another conference next April (TAIS 2025), and have just put out our call for papers. If you work in technical safety, philosophy or governance and want to contribute to building safety research capacity in Japan, please send a paper our way!

Important Dates

Submissions open: October 11, 2024
Submission Deadline: January 1, 2025
Notification of Acceptance: February 1, 2025
Camera-ready Deadline: April 1, 2025
Workshop Date: Saturday, April 19, 2025

All deadlines are 11:59PM UTC-12:00 (“anywhere on Earth”).

Call for papers

We are inviting submissions of short papers (maximum 8 pages) outlining new research, with a deadline of January 1, 2025. We welcome papers on any of the following topics, or anything else where the authors convincingly argue that it advances the field of AI safety in Japan.

Mechanistic Interpretability: the detailed study of particular neural networks. Interpretability researchers often take a circuits-style approach following the foundational work of Chris Olah, or use causal linear probes to understand directions in latent space. Work that investigates how and why structure emerges in neural networks during training can also fall under this category.
Developmental Interpretability: understanding the process by which neural networks learn. By applying the tools of Watanabe's singular learning theory to modern neural network architectures, developmental interpretability researchers aim to detect phase transitions during training, thereby detecting new capabilities as they emerge.
Agent Foundations: the name given to a number of parallel research efforts, each of which aims to designing AI that is provably safe. Problems in agent foundations include embedded agency and cognition, corrigibility and the shutdown problem, natural abstractions, and game theory. With sufficient rigour, agent foundations research hope to build AGI that is safe by construction.
Scalable Oversight: a prosaic alignment approach that aims to make advanced AI systems amenable to human oversight. Often this involves training less powerful AIs to oversee more powerful AIs in a hierarchical way with

...
OpenAI defected, but we can take honest actions
by Remmelt — 2024-10-21T08:41:25.728Z

OpenAI defected. As a non-profit, OpenAI recruited researchers on the promise of "safe" AGI. Then it pushed out the safety researchers and turned into a for-profit.

Why act in response?

OpenAI is recklessly scaling AI. Besides accelerating "progress" toward mass extinction, it causes increasing harms. Many communities are now speaking up. In my circles alone, I count seven new books critiquing AI corps. It’s what happens when you scrape everyone's personal data to train inscrutable models (computed by polluting data centers) used to cheaply automate out professionals and spread disinformation and deepfakes.
Safety researchers used to say they could improve things on the inside. They didn’t want to ruin the goodwill. That option is no longer available.
The rational step is to ratchet up pressure from the outside. If we stand by while OpenAI violates its charter, it signals that their execs can get away with it. Worse, it signals that we don’t care.
OpenAI is in a weaker position than it looks on the surface. OpenAI is seen as the industry leader, yet projected to lose $5 billion this year. Microsoft’s CEO changed their mind on injecting that amount after the board fired Sam. So now OpenAI is dependent on investment firms to pump in cash for compute every < 10 months. OpenAI is constantly nearing a financial cliff. If concerned communities would seriously collaborate to make OpenAI stop obvious harms caused by their model releases, or to hold the business liable, OpenAI will fail.
OpenAI’s forced downsizing would reset expectations across the industry. Since 2012, the gap between expenditures in deep learning and actual revenue has ballooned to over half a trillion dollars a year. If even OpenAI, as the industry’s leading large model developer, had to fire engineers and cut compute to save its failing business model, it could well trigger an AI crash. As media articles turn cynical, investors will sell stakes to get ahead of the crowd, and the disinvested start-ups will go bankrupt. During this period of industry weakness, our communities can pass and actually enforce many laws to restrict harmful scaling.

OpenAI’s activities are harmful. Let’s be public and honest in our response.

If you start a campaign, please communicate how you are targeting OpenAI's harmful activities. That way, we maintain the moral high ground in the eyes of the public.
Avoid sme

...
Slightly More Than You Wanted To Know: Pregnancy Length Effects
by JustisMills — 2024-10-21T01:26:02.030Z
Link: https://justismills.substack.com/p/slightly-more-than-you-wanted-to

Pregnancy is most stressful at the beginning and at the end. First you try to conceive, and spend half of each month superimposed between two very different realities. Then there’s morning sickness, and you worry about miscarriage, which is far more likely

early on

.

Near the end, you can be pretty confident you’re having a baby. But there’s a wide range of when exactly that baby might come.

Very early

, and you’re looking at a long stay in the NICU, and maybe more problems later on.

Very late

, and you have to be medically induced for the health of both baby and mother, which is

correlated with protracted labor

.

Everyone agrees that pre-32 weeks is really bad, pre-37 weeks is pretty bad, and post-42 weeks is dangerous. In this post, though, I’ll focus on the sweet spot: the 37-41 week range. If you have a baby in this range, you’re basically in good shape, and should be grateful. But are you in slightly better shape in some parts of the window than others? Let’s find out.

(In case you were wondering, nothing in this blog post is medical advice.)

The Inner Bullseye

37-41 weeks may be the sweet spot, but 39-41 is sweeter. This fact is unambiguous and widely reported, though it only recently raised to the level of ironclad medical orthodoxy.

This study

from 2013 is a good example of the relevant research: basically, babies born a few weeks early have (some of) the same problems as babies born earlier still, only more subtle. For example:

Adverse outcomes experienced by the early term[1] babies included hypoglycemia (4.9 percent versus 2.5 percent of full-term babies), admission to the neonatal intensive care unit (8.8 percent versus 5.3 percent, the need for respiratory support (2.0 percent versus 1.1 percent), the need for intravenous fluids (7.5 percent versus 4.4 percent) intravenous antibiotics (2.6 percent versus 1.6 percent) and mechanical ventilation or intubation, which was required in .6 percent of early term babies versus .1 percent in full-term babies.

So to a first approximation, just hope for week 39 or later. Which is a shame, because the weeks after 37 are extra uncomfortable,[2] so expecting mothers have lots of kick-shaped reasons to feel conflicted.

Beyond this, we have to get speculative.

What’s In A Week?

When you’re trying to figure something out that really matters, I find a two pronged approach is best. On the one hand, you want to find hig...
Information vs Assurance
by johnswentworth — 2024-10-20T23:16:25.762Z

In contract law, there’s this thing called a “representation”. Example: as part of a contract to sell my house, I might “represent that” the house contains no asbestos. How is this different from me just, y’know, telling someone that the house contains no asbestos? Well, if it later turns out that the house does contain asbestos, I’ll be liable for any damages caused by the asbestos (like e.g. the cost of removing it).

In other words: a contractual representation is a factual claim along with insurance against that claim being false.

I claim[1] that people often interpret everyday factual claims and predictions in a way similar to contractual representations. Because “representation” is egregiously confusing jargon, I’m going to call this phenomenon “assurance”.

Prototypical example: I tell my friend that I plan to go to a party around 9 pm, and I’m willing to give them a ride. My friend interprets my “plan to go around 9 pm” as an assurance: if I fail to show up around 9 pm to drive them to the party, and my friend misses out on some big thing at the party as a result, then they’ll place the blame on me. It’s a kind of non-monetary insurance - if my assurance fails, then I’m socially liable for damages.

In that context, it all seems pretty obvious - people would normally interpret me as having made an assurance (even if they wouldn’t know how to articulate it), I have knowingly taken on that assurance, so it’s totally reasonable to blame me if I don’t show up around 9 pm. But the whole thing is very implicit, which can make things subtle and tricky.

Suppose the conversation leading up to the 9 pm plan went like this:

Friend: Hey John, what time are you going to the party?

Me: I dunno, probably around 9 pm. [At this point, I’ve merely offered some information; I think most people would not interpret this as an assurance, and would not blame me much if I show up to the party at 8:30 or 10:00 or even skip it altogether.]

Friend: Cool, can you pick me up on the way? [This sounds like a very reasonable request, since my friend’s place is on the way. However, if I agree, then suddenly my “around 9 pm” becomes an assurance - showing up at 10:00 or skipping the party altogether would be a dick move! So my friend’s ask is a lot more costly than it nominally sounds.]

That’s the sort of subtlety which comes up: I intended 9 pm as a prediction, but it might get conver...
Liquid vs Illiquid Careers 
by vaishnav92 — 2024-10-20T23:03:49.725Z
Link: https://vaishnavsunil.substack.com/p/liquid-vs-illiquid-careers

We're used to thinking about skills and experience in terms of their market value - how useful they are and how much employers or customers would pay for them. But another, more subtle dimension gets less attention: liquidity.

Just like in financial markets, some human capital is easier to price and trade than others. This is not about the absolute value of your skills or experience, but how easily the market can assess and exchange them.

Consider Jim - with 4 years of experience as a management consultant at McKinsey. In the job market, potential employers have a good idea of what they are buying - someone smart and conscientious enough to be hired by McKinsey; meets a reasonably high bar on commercial acumen and communication skills; and polished enough to have managed clients and moved one step higher on the well-defined McKinsey ladder.

Jim's skills are highly legible to every potential employer. They can estimate his salary within narrow confidence intervals, given that McKinsey's pay structure is essentially public information. If Jim decides to switch jobs or move countries, all else equal, he can be relatively confident about his chances of landing a job that pays him his "market value". Much like publicly traded stocks, his legible experience and skills can be sold quickly at the prevailing market rate.

On the flip side, picture someone who's spent two years at an AI startup, followed by a stint running operations for a non-profit in Asia, and is now working on a political campaign. How comfortable would you be guessing this person's potential earnings or, more fundamentally, the value they could generate?

If I were a hiring manager looking to fill a high-leverage, entrepreneurial position requiring an exceptional young candidate, I'd be more inclined to interview this person over most McKinsey consultants. However, I'd evaluate them more rigorously than someone with the McKinsey stamp, and might even limit my search to candidates within my social or professional network.

When human capital is built through non-linear or less legible paths, the lack of legibility increases variance from the employer's perspective. This isn't necessarily negative, but it does increase the value of additional information. If the cost of obtaining this information doesn't justify the potential upside, candidates with less legible backgrounds may be passed over in scree...
AI Can be “Gradient Aware” Without Doing Gradient hacking.
by Sodium — 2024-10-20T21:02:10.754Z

Repetition helps us remember things better. This is because it strengthens connections in our brain used for memory[1].

We don’t need to understand the exact neural mechanisms to take advantage of this fact. For example, societies could develop cultural norms that promote repetition exercises during education[2].

This is an example of how humans are “gradient aware.” “Repeat a task so I can remember it better” advances our goals by taking advantage of our “gradient update” process. This is an action we take solely because of how our minds get shaped.

I think a similar situation may occur in sufficiently powerful AI models. If AIs are trained in environments where they can strategically take advantage of gradient updates, they might choose their actions partially based on how they expect the gradient descent process to modify their future instances[3]. I call this “gradient awareness.”

The only time I’ve seen people discuss gradient-aware models is in the context of

gradient hacking

. We can think of gradient hacking as a specialized case of gradient awareness, where a mesa-optimizer protects its mesa-objective from being modified by redirecting its gradient updates.

At first glance, gradient hacking seems peculiar and unnatural. However, gradient awareness seems like a strategy advanced models would pick up[4]. The closest thing I’ve seen in the wild is how an

RNN that was trained to play Sokoban

will “pace around” as it figures out a plan.

Gradient awareness is a spectrum. You might repeat things to remember them because that's how they taught you in grade school, or you could have a super elaborate Anki setup. Similar to humans who follow cultural practices, models can totally execute strategies that are gradient-aware without “understanding” why these strategies work.

What does this all mean?

I’d expect that we could see gradient-aware models before they are capable of gradient hacking. Gradient hacking is a very sophisticated algorithm, and I think models might execute other gradient-aware algorithms first (alternatively, its first attempts at gradient hacking would just fail). To the extent we believe this, we could think about how we structure our training environment to guard against gradient awareness. That means thinking about ways to structure the gradient updates to avoid opportunities for gradient aware strategies, or limiting the model’s...