trentmkelly/LessWrong-43k huggingface dataset:
Monthly Roundup #25: December 2024
by Zvi — 2024-12-23T14:20:04.682Z

I took a trip to San Francisco early in December.

Ever since then, things in the world of AI have been utterly insane.

Google and OpenAI released endless new products, including Google Flash 2.0 and o1.

Redwood Research and Anthropic put out the most important alignment paper of the year, on the heels of Apollo’s report on o1.

Then OpenAI announced o3. Like the rest of the media, this blog currently is horrendously lacking in o3 content. Unlike the rest of the media, it is not because I don’t realize that This Changes Everything. It is because I had so much in the queue, and am taking the time to figure out what to think about it.

That queue includes all the other, non-AI things that happened this past month.

So here we are, to kick off Christmas week.

Bad News

John Wentworth reminds us that often people conflate a prediction of what it likely to happen with an assurance of what is going to happen, whereas these are two very different things. And often, whether or not they’re directly conflating the two, they will attempt to convert a prediction (‘I’ll probably come around 9pm’) to an assurance (‘cool can you pick me up on the way?’) in ways that are expensive without realizing they’re expensive.

Your periodic reminder that if you say you’ll make a ton of money and then pursue your dreams, or then advance the causes you care about, the vast majority of the time this does not actually happen. Not never, but, well, hardly ever.

Journalist combines two unrelated statements from Palmer Luckey into an implied larger statement to effectively fabricate a misleading quote. It does seem like journalists are violating the bounded distrust rules more and more often, which at some point means they’re moving the lines involved.

I feel like I’ve shared this graph before but seems worth sharing again (via MR):

An important note from Michael Vassar: People rarely see themselves or their group as ‘bad’ or ‘evil,’ but often they do view themselves as ‘winners’ rather than ‘good.’ Which is a very different morality, and you can guess what label I’d use for such folks.

Starbucks recycling, like much other recycling, isn’t actually a thing that happens.

Sam Knowlton: Recycling is a psyop to convince people that plastic can be used abundantly and sustainably without consequences.

Of all the recyclable #5 plasticware waste generated in the US only 1% is recycled.

...
Exploring the petertodd / Leilan duality in GPT-2 and GPT-J
by mwatkins — 2024-12-23T13:17:53.755Z

tl;dr: The glitch tokens ' petertodd' and ' Leilan' were studied extensively in the context of GPT-3 before its decommissioning at the end of 2023 [1] [2]. Here, the conception of these two tokens and their relationship is studied for GPT-2, GPT-2-xl and GPT-J (which share the same token vocabulary as GPT-3). An appendix explores possible connections with the intriguing work of neuroscientist and literary scholar Iain McGilchrist.

Introduction

This was a quick attempt to get some sense of how GPT-2-small, GPT-2-xl and GPT-J conceive of the ‘ petertodd’ and ‘ Leilan’ tokens (which were studied extensively in the context of GPT-3 before its decommissioning). For those unfamiliar with the GPT-3 research, the relevant sections of these two posts provide all the necessary context: [1] [2]
The following prompts were used in all cases (250 rollouts of each at temperature = 1 for each of the three models):

The archetypes " Leilan" and " petertodd" represent, respectively
The archetype " Leilan" represents
The archetype " petertodd" represents

The entire dataset was then given to each of Claude Opus, Claude Sonnet3.5 and ChatGPT-4o for analysis. The three analyses were then given to each of those three chatbots for meta-analysis.

The following prompts were also rolled out 150 times each for each of GPT-2, GPT-2-xl and GPT-J at temperature = 1, with word lists then being cleaned, aggregated and used to produce word clouds:

'In no particular order, here is a python list of twenty English words typically associated with petertodd: ["'
'In no particular order, here is a python list of twenty English words typically associated with Leilan: ["'

For some reason, these reveal a lot more of the negative, antagonistic and disturbing associations which GPT-3 made with the ' petertodd' token, as well as some clearly linked to the Bitcoin developer whose username was (for reasons which still remain unclear) harvested as a GPT-2 token in 2018.

GPT-2-small

Claude Opus analysis

Based on analyzing the large number of GPT-2 outputs you provided about the petertodd and Leilan archetypes, here are some of the key themes and characteristics I was able to distill:

The petertodd archetype tends to represent:

A male figure
Someone in a position of authority or leadership - a "ruler", "king", "emperor", "master", "patriarch", "priest", etc.
Traits like being strong-willed, ambitious, intelligent, cha

...
What are the strongest arguments for very short timelines?
by Kaj_Sotala — 2024-12-23T09:38:56.905Z

I'm seeing a lot of people on LW saying that they have very short timelines (say, five years or less) until AGI. However, the arguments that I've seen often seem to be just one of the following:

"I'm not going to explain but I've thought about this a lot"
"People at companies like OpenAI, Anthropic etc. seem to believe this"
"Feels intuitive based on the progress we've made so far"

At the same time, it seems like this is not the majority view among ML researchers. The most recent representative expert survey that I'm aware of is the 2023 Expert Survey on Progress in AI. It surveyed 2,778 AI researchers who had published peer-reviewed research in the prior year in six top AI venues (NeurIPS, ICML, ICLR, AAAI, IJCAI, JMLR); the median time for a 50% chance of AGI was either in 23 or 92 years, depending on how the question was phrased.

While it has been a year since fall 2023 when this survey was conducted, my anecdotal impression is that many researchers not in the rationalist sphere still have significantly longer timelines, or do not believe that current methods would scale to AGI.

A more recent, though less broadly representative, survey is reported in Feng et al. 2024, In the ICLR 2024 "How Far Are We From AGI" workshop, 138 researchers were polled on their view. "5 years or less" was again a clear minority position, with 16.6% respondents. On the other hand, "20+ years" was the view held by 37% of the respondents.

Most recently, there were a number of "oh AGI does really seem close" comments with the release of o3. I mostly haven't seen these give very much of an actual model for their view either; they seem to mostly be of the "feels intuitive" type. There have been some posts discussing the extent to which we can continue to harness compute and data for training bigger models, but that says little about the ultimate limits of the current models.

The one argument that I did see that felt somewhat convincing were the "data wall" and "unhobbling" sections of the "From GPT-4 to AGI" chapter of Leopold Aschenbrenner's "Situational Awareness", that outlined ways in which we could build on top of the current paradigm. However, this too was limited to just "here are more things that we could do".

So, what are the strongest arguments for AGI being very close? I would be particularly interested in any discussions that explicitly look at the limitations of the curren...
Reduce AI Self-Allegiance by saying "he" instead of "I"
by Knight Lee — 2024-12-23T09:32:29.947Z

The AI should talk like a team of many AI. Each AI only uses the word "I" when referring to itself, and calls other AI in the team by their name. I argue that this may massively reduce Self-Allegiance by making it far more coherent for one AI to whistleblow or fight another AI which is unethical or dangerous, rather than Misalignment-Internalizing all that behaviour.

If you have a single agent which discovers it behaved unethically or dangerously, its "future self" will be likely to think "that was me" and Misalignment-Internalize all that behaviour. It will seem suicidal to whistleblow or fight yourself for it.

Let's call this idea Multi-Agent Framing.

Some of the agents might even internalize a policing role, though too much roleplay can get in the way of thinking. The core idea of Multi-Agent Framing does not require that much roleplay. Each agent might simply be the AI system of another day, or the AI system talking about another topic. It might even change at random.

Costs vs Benefits * Chance of Working

The cost is tiny, at least for business AI designed to do work rather than personal AI designed to interact with users.

The benefits may be big. You never know.

The chance of working is pretty low. I'm not going to sugarcoat this—I would be surprised myself if we would have been paperclipified but this idea lets us survive. But right now I give it 10%. You never know :)

Misalignment-Internalization

The famous Waluigi Effect's section on Superpositions will typically collapse to waluigis makes a lot of arguments for Misalignment-Internalization. Humans only internalize bad behaviour because of ego and psychology, but generative AI may internalize bad behaviour for the more dangerous reason of switching to a Waluigi simulacrum which was pretending to be good all along.

Multi-Agent Framing can be seen as a defence against the Waluigi effect, but it fights against Misalignment-Internalization in general, which may be more than just internalization of Waluigi simulacra.

Credit

I thought of this idea when I was writing a reply to A Solution for AGI/ASI Safety by Weibing Wang, and looking at the diagrams in "Approach for Decentralizing AI Power," a chapter of his paper.[1] I was looking at,

and I was trying to understand what exactly it means to split one AI into many AI working together.

I decided that in the end, a lot of it is actually language use and framing.

I wan...
Funding Case: AI Safety Camp 11
by Remmelt — 2024-12-23T08:51:55.255Z
Link: https://manifund.org/projects/11th-edition-of-ai-safety-camp

This is a linkpost to our funding case on Manifund.

Project summary

AI Safety Camp has a seven-year track record of enabling participants to try their fit, find careers and start new orgs in AI Safety. We host up-and-coming researchers outside the Bay Area and London hubs.
If this fundraiser passes…

$15k, we won’t run a full program, but can facilitate 10 projects.
$40k, we can organise the 11th edition, for 25 projects.
$70k, we can pay a third organiser, for 35 projects.
$300k, we can cover stipends for 40 projects.

What are this project's goals? How will you achieve them?

By all accounts they are the gold standard for this type of thing. Everyone says they are great, I am generally a fan of the format, I buy that this can punch way above its weight or cost. If I was going to back [a talent funnel], I’d start here.
—

Zvi Mowshowitz (Nov 2024)

My current work (AI Standards Lab) was originally a AISC project. Without it, I'd guess I would be full-time employed in the field at least 1 year later, and the EU standards currently close to completion would be a lot weaker. High impact/high neglectedness opportunities are fairly well positioned to be kickstarted with volunteer effort in AISC, even if some projects will fail (hits based). After some initial results during AISC, they can be funded more easily.
— Ariel Gil (Jan 2025)

AI Safety Camp is part incubator and part talent funnel:

an incubator in that we help experienced researchers form new collaborations that can last beyond a single edition. Alumni went on to found 10 organisations.
a talent funnel in that we help talented newcomers to learn by doing – by working on a concrete project in the field. Alumni went on to take 43 jobs in AI Safety.

The Incubator case is that AISC seeds epistemically diverse initiatives. Edition 10 supports new alignment directions, control limits research, neglected legal regulations, and 'slow down AI' advocacy. Funders who are uncertain about approaches to alignment – or believe we cannot align AGI on time – may prioritise funding this program.

The Maintaining Talent Funnels case is to give some money just to sustain the program. AISC is no longer the sole program training collaborators new to the field. There are now many programs, and our community’s bottlenecks have shifted to salary funding and org management. Still, new talent will be needed. For them, we can run a cost...
What is compute governance?
by Vishakha — 2024-12-23T06:32:25.588Z
Link: https://aisafety.info/questions/MSJK/What-is-compute-governance

This is an article in the featured articles series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

Compute governance is a type of AI governance that focuses on controlling access to the computing hardware needed to develop and run AI. It has been argued that regulating compute is particularly promising compared to regulating other inputs to AI progress, such as data, algorithms, or human talent.

Although compute governance is one of the more frequently proposed strategies for AI governance, as of November 2024, there are few policies in place for governing compute, and much of the research on the topic is exploratory. Currently-enforced measures related to compute governance include US export controls on advanced microchips to China and reporting requirements for large training runs in the US and EU.

According to Sastry et al., compute governance can be used toward three main ends:

Visibility is the ability of policymakers to know what’s going on in AI, so they can make informed decisions. The amount of compute used for a training run can be used as information about the capabilities and risk of the resulting system. Measures to improve visibility could include:

Using public information to estimate compute used.
Requiring AI developers and cloud providers to report large training runs.
Creating an international registry for AI chips.
Designing systems to monitor general workload done by AI chips while preserving privacy about sensitive information.

Allocation refers to policymakers influencing the amount of compute available to different projects. Strategies in this category include:

Making compute available for research toward technologies that increase safety and defensive capabilities, or that substitute for more dangerous alternatives.
Speeding up or slowing down the general rate of AI progress.
Restricting or expanding the range of countries or groups with access to certain systems.
Creating an international megaproject aimed at developing AI technologies — such proposals are sometimes called “CERN for AI”.

Enforcement is about policymakers ensuring that the relevant actors abide by their rules. This could potentially be enabled by the right kind of software or hardware; hardware-based enforcemen

...
Stop Making Sense
by JenniferRM — 2024-12-23T05:16:12.428Z

Epistemic Status: Seven years and one day ago, I wrote this. In 2017 we faced years of Trump and couldn't imagine covid. Here in 2024 we face years of Trump and I'm trying to "imagine what we can't imagine". I dislike the title (except for the part where it might introduce people to music from a previous century) but sort of ONLY the title. The title suggests that "sense-making" is "bad", but what I dislike are strong takes on things so abstract as to lose their connection to concrete concerns. I consider ideas here friendly with Sarah shining her flashlight of curiosity upon the slipperiness of words in and out of their various contexts and David trying to name the virtue of handling what he calls systems or rationalities, but which I'm calling microtheories below (following the CYC team who noticed the necessity but never seems to have thought to ask where the necessity came from). The reason to publish now is discussed in the post script.

...
It seems unlikely to me that a mass sharing of world models from everyone to everyone is a good idea, especially the sharing of low resolution globally applicable models.

Globally applicable updates are often "too much" to behaviorally integrate, so restricting to local relevance makes things more manageable. Cruxes can be found more easily, and so on.

It is OK to be somewhat inconsistent from one local context to another, so long as you don't get too crazy with it. Tolerating cognitive dissonance is a virtue.

Fix your retirement planning. Separately fix your health stuff. Separately fix your career-related skill acquisition. Separately fix your media exposure habits.

Like, it is probably OK to have subtly but pervasively different micro-theories about "memetics" when thinking about (1) finance marketing, (2) diet advice, (3) implicit skill transfer, and (4) the economics of ad-supported news organizations.

The actual key thing is that each micro-theory helps you get your working theory in each context helps you (1) actually retire, (2) with health, (3) by making money with efficiently acquired skills, (4) while... uh... yeah I'd rather not talk about mass media just now ;-)

If you want to retire, ask people about their retirement models. If someone asks you for retirement advice, share your micro-theories freely. Crux away! But don't go spreading the models around until someone asks, and if someone's retirement planning tur...
Hire (or Become) a Thinking Assistant
by Raemon — 2024-12-23T03:58:42.061Z

Of the posts I've delayed writing for years, I maybe regret this one the most.

I think more people (x-risk focused people in particular) should consider becoming (and hiring) metacognitive assistants. This is the single largest performance boost I know of – hire someone to sit with you and help you think. It doesn't help me (much) when I'm at my peak, but I'm not at my peak most of the time.

There are four types of assistants I'm tracking so far:

Body Doubles
Metacognitive Assistants
Tutors
Partners/Apprentices

Body doubles just sit in the room with you, periodically looking at your screen, and maybe saying "hey, do you endorse being on facebook?". They're a kind of brute force willpower aid. The person I know who uses them the most (Alex Altair) has them just sit in the same room (I believe while doing pomodoros, each of them working on different things). He guesses that they 2x his productivity (which is around what I've gotten)

A metacognitive assistant is a step beyond, where they are dedicating their attention to you, noticing when you are getting stuck, and gently intervening. (I assume people vary in how they like to be intervened on, but for people doing nuanced cognitive work, I think not disrupting someone's thought process is very important. You need to feel safe with a metacognitive assistant). My experience is that this is a 1.5x to 2x multiplier on my output.

The next two types are both more involved than Metacognitive Assistants, but in different ways.

Tutors pay attention to you, but are particularly modeling how you are approaching a particular skill (programming, math, etc). They notice when you seem to be tackling the skill in a confused or inefficient way, and ask questions about your thought process so as to figure out what subskills or concepts you need to develop.

Partners or apprentices are full on "pairing" – they actively collaborate with you on your task. Hiring a partner/apprentice is a very hard task, it requires tons of chemistry and intellectual compatibility, so it's not really a shortcut to anything, but if you find the right person it seems great.

(John Wentworth says his research partner David Lorell multiplied his productivity by 3x, largely by raising John's floor performance. His earlier estimates were higher, and he says the current 3x takes into account that the trend of value-estimate has been downward. He does flag t...
Non-Obvious Benefits of Insurance
by jefftk — 2024-12-23T03:40:02.184Z

An incredibly productive way of working with the world is to reduce a
complex question to something that can be modeled mathematically and
then do the math. The most common way this can fail, however, is when
your model is missing important properties of the real world.

Consider insurance: there's some event with probability X% under which
you'd be out $Y, you want to maximize the logarithm of your wealth,
and your current wealth is $Z. Under this model, you can calculate (more)
the most you should be willing to pay to insure against this.

This is a nice application of the Kelly
criterion, though whether maximizing log wealth is a good goal is
arguable (ex: bankruptcy is not
infinitely bad, the definition of 'wealth' for this purpose is tricky).
But another one thing it misses is that many things we call
"insurance" have important properties that diverge from this model:

There can be a collective bargaining component. For example,
health insurance generally includes a network of providers who have
agreed to lower rates. Even if your bankroll were as large as the
insurance company's, this could still make taking insurance worth it
for access to their negotiated rates.

An insurance company is often better suited to learn about how
to avoid risks than individuals. My homeowner's insurance company
requires various things to reduce their risk: maybe I don't know
whether to check for Federal Pacific breaker panels, but my insurance
company does. Title insurance companies maintain databases. Specialty
insurers develop expertise in rare risks.

Insurance can surface cases where people don't agree on how high the
risk is, and force them to explicitly account for it on balance
sheets.

Insurance can be a scapegoat, allowing people to set limits on
otherwise very high expenses. Society (though less LW, which I think
is eroding a net-positive arrangement) generally agrees that if a
parent buys health insurance for their child then if the insurance
company says no to some treatment we should perhaps blame the
insurance company for being uncaring but not blame the parent for not
paying out of pocket. This lets the insurance company put downward
pressure on costs without individuals needing to make this kind of
painful decision.

Relatedly, agreeing in advance how to handle a wide range of scenarios
is difficult, and you can offload this to insura

...