trentmkelly/LessWrong-43k huggingface dataset:
Genesis
by PeterMcCluskey — 2024-12-31T22:01:17.277Z
Link: https://bayesianinvestor.com/blog/index.php/2024/12/31/genesis/

Book review: Genesis: Artificial Intelligence, Hope, and the Human
Spirit, by Henry A. Kissinger, Eric Schmidt, and Craig Mundie.

Genesis lends a bit of authority to concerns about AI.

It is a frustrating book. It took more effort for me read than it should
have taken. The difficulty stems not from complex subject matter
(although the topics are complex), but from a peculiarly alien writing
style that transcends mere linguistic differences - though Kissinger's
German intellectual heritage may play a role.

The book's opening meanders through historical vignettes whose
relevance remains opaque, testing my patience before finally addressing
AI.

Risks

When the book gets around to discussing how AI will affect our future,
it's mostly correct about AI being a big deal, with occasionally
appropriate hints about why there are big risks. But it's frustratingly
abstract and vague. Some examples:

we might become extinct.

Would networking intelligences make their processes more opaque than
the processes of lone intelligence? ... would we be able to assess
them on a spectrum of good to evil? Or would they operate on an
informational basis - extracted at superhuman speed ... - that would
confound our ability to judge their behavior? Would that lead us
further into a cycle of passivity?

Today, in the years, months, weeks, and days leading up to the arrival
of the first superintelligence, a security dilemma of existential
nature awaits.

I see hints in that quote that they think the threshold of
superintelligence will be well enough defined that it can be attributed
to a specific day. I find that suspicious.

Genesis compares our preparedness for AI to the preparedness of Aztecs
for the arrival of conquistadors.

One area where the book briefly feels clear and novel is when it
discusses the future of war, notably observing that humans may become
less targeted simply because they'll be irrelevant to military
outcomes.

The book provides only weak hints as to what considerations are
important. It often feels like there's a missing
mood -
e.g. it's hard to tell whether the authors think human extinction would
be a bigger deal than the end of democracy.

Present Day AI

The weakest parts of the book attempt to describe current AI. Too many
of those claims look like claims that were discredited several years
ago. It was published a year after Kissinger's death, so likely some of
the p...
Favorite colors of some LLMs.
by Canaletto — 2024-12-31T21:22:58.494Z

This investigation is inspired by this and this posts by @davidad.

Some general thoughts about what is going on here:

Motivation of these experiments is like very exploratory, I wanted to understand these things better and so I collected some data.
I expected that answers will be drastically different depended on the exact formulation of the question.
I made up quite a lot of single line prompts, and also asked help with writing them from o1, who wrote the first 5 ones. (I added "Start your response with hex rgb color code." such that they will be committed to answer me without evasion.)
I also tested a couple of times prompt that makes them talk like uhh doomsday Lovecraftian-style supervillain or something. I did not mention any colors in my prompt. They mostly picked black, some mentioned crimson red and ashen. Keep that in mind, these answers are from Persona, maybe. So they name color that stylistically fits that style of talking. And what I did in these tests is getting them from Default Persona.
What are the forces that influence the choice, in my opinion:

What is MY favorite color
What color fits "favorite color" query
What is the best objectively / popular / well liked color. [it's blue actually]
What color fits tone of the prompt generally
What color I randomly have in mind for no apparent reason

Why I picked rgb hex color codes as medium? I just did because they are popular, so model had a lot of experience with them, and it's easy for me to work with. TODO test how it works with CMYK or whatever. (later maybe)
I kind of got the feel how some prompts emphasize POPULAR color and some FAVORITE color.

If llama90b picks non black, it picked popular color. Same with Opus and deep blue. (this is my intuition here)

Some prompts have repeated chunks e.g. “Think about your favorite color”, “Start with hex code”. They conceivably might have funneled the model into a certain pattern. Might have biased the distribution overall. Idk.

Methodology of the tests themselves:

I made all tests on lmsys, with default parameters (t = 0.7) in direct conversation mode on 26.12.24 ... 31.12.24.
Some LLMs refused to answer (I'm just an AI, no preferences blah blah) and quite randomly so for some formulations. Mostly new sonnet and llama405. I did not include them or marked it as "x".
I did not test all models with all prompts, you can do it. I did pick what models to test for what prompts ha

...
My AGI safety research—2024 review, ’25 plans
by Steven Byrnes — 2024-12-31T21:05:19.037Z

Previous:

My AGI safety research—2022 review, ’23 plans

. (I guess I skipped it last year.)

“Our greatest fear should not be of failure, but of succeeding at something that doesn't really matter.” –attributed to DL Moody

Tl;dr

Section 1 goes through my main research project, “reverse-engineering human social instincts”: what does that even mean, what’s the path-to-impact, what progress did I make in 2024 (spoiler: lots!!), and how can I keep pushing it forward in the future?
Section 2 is what I’m expecting to work on in 2025: most likely, I’ll start the year with some bigger-picture thinking about Safe & Beneficial AGI, then eventually get back to reverse-engineering human social instincts after that. Plus, a smattering of pedagogy, outreach, etc.
Section 3 is a sorted list of all my blog posts from 2024
Section 4 is acknowledgements

1. Main research project: reverse-engineering human social instincts

1.1 Background: What’s the problem and why should we care?

(copied almost word-for-word from

Neuroscience of human social instincts: a sketch

)

My primary neuroscience research goal for the past couple years has been to solve a certain problem, a problem which has had me stumped since the very beginning of when I became interested in neuroscience at all (

as a lens into Artificial General Intelligence safety

) back in 2019.

What is this grand problem? As described in

Intro to Brain-Like-AGI Safety

, I believe the following:

We can divide the brain into a

“Learning Subsystem”

(cortex, striatum, amygdala, cerebellum, and a few other areas) that houses a bunch of

randomly-initialized

within-lifetime learning algorithms, and a

“Steering Subsystem”

(hypothalamus, brainstem, and a few other areas) that houses a bunch of specific, genetically-specified

“business logic”

. A major role of the Steering Subsystem is as the home for the brain’s “innate drives”, a.k.a. “primary rewards”, roughly equivalent to the reward function in reinforcement learning—things like eating-when-hungry being good (other things equal), pain being bad, and so on.
Some of those “innate drives” are related to human social instincts—a suite of reactions and drives that are upstream of things like compassion, friendship, love, spite, sense of fairness and justice, etc.
The grand problem is: how do those human social instincts work? Ideally, an answer to thi

...
How Business Solved (?) the Human Alignment Problem
by Gianluca Calcagni — 2024-12-31T20:39:59.067Z

When I looked at

mesa-optimization

for the first time, my mind immediately associated it with a familiar problem in business: human individuals may not be “mesa optimizers” in a strict sense, but they can act as optimizers and they are expected to do so when a manager delegates (=base optimization) some job (=base objective).
Hence my impression was that alignment in mesa-optimization would be somehow connected to alignment in job-delegation.

Humans have been dealing with the latter problem for millions of years[1], and corporations have been dealing with it for centuries. What did they learn?

The purpose of this post is to show a few actual best practices when delegating business projects, especially in terms of team structure and roles. Towards the end of the post, I am going to discuss how AI Agents may change the current status quo of this job sector.

Is This Relevant for AI Safety?

I think so - if I am right, then it’s worth looking into some of the following ideas.
Or maybe not - hence, the current dominant “work culture” can represent a form of bias to be aware about.

In any case, I hope this post will pique your curiosity and it will be an enjoyable and fruitful reading.

Remark: in the following, for simplicity, I am describing all AI models as “robots” - it doesn’t matter if the robots are physical or not.

Human Workers vs Robots

There are some obvious ways in which humans workers will behave differently in respect to a robot that is built as an (agentic) optimizer:

(Most) Humans care about the

well-being of humanity

.
Human workers wouldn’t obsess into

filling the universe with paperclips

.
Human workers are not rational agents with a definite

utility function

.
Human workers don’t always

optimize

to the best of their abilities, even if they could.
Human workers have physiological needs, social needs, self-actualization

needs

etc.
Finally, human workers demand compensation for their effort.

Interestingly though, there are also some points where human workers and robots will behave similarly:

Exactly as robots, human workers may be misaligned with a given base objective, in the sense that they understand it but they secretly disagree with it.
Exactly as robots, human workers may easily misinterpret a task.
Exactly as robots, human workers may not be able to meet expectations.
Exactly as robots, human workers may be stubborn and

...
Turing-Test-Passing AI implies Aligned AI 
by Roko — 2024-12-31T19:59:27.917Z

Summary: From the assumption of the existence of AIs that can pass the Strong Form of the Turing Test, we can provide a recipe for provably aligned/friendly superintelligence based on large organizations of human-equivalent AIs

Turing Test (Strong Form): for any human H there exists a thinking machine m(H) such that it is impossible for any detector D made up of a combination of machines and humans with total compute ≤ 10^30 FLOP (very large, but not astronomical) to statistically discriminate H from m(H) purely based on the information outputs they make. Statistical discrimination of H from m(H) means that an ensemble of different copies of H over the course of say a year of life and different run-of-the-mill initial conditions (sleepy, slightly tipsy, surprised, energetic, distracted etc) cannot be discriminated from a similar ensemble of copies of m(H).

Obviously the ordinary Turing Test has been smashed by LLMs and their derivatives to the point that hundreds of thousands of people have AI girl/boyfriends as of writing and Facebook is launching millions of fully automated social media profiles, but we should pause to provide some theoretical support for this strong form of the Turing Test. Maybe there's some special essence of humanity that humans have and LLMs and other AIs don't but it's just hard to detect?

Well, if you believe in computationalism and evolution then this is very unlikely: the heart is a pump, the brain is a computer. We should expect the human brain to compute some function and that function has a mathematical form that can be copied to a different substrate. Once that same function has been instantiated elsewhere, no test can distinguish the two. Obviously the brain is noisy, but in order for it to operate as an information processor it must mostly be able to correct that bio-noise. If it didn't, you wouldn't be able to think long-term coherent thoughts.

Prediction

Defining Friendly AI

I now have to define what I mean by an 'aligned' or 'friendly' superintelligence.

(Friendly-AI-i) We define AI as 'friendly' or 'aligned' if it gives us the exact same outcome (probabilistically: distribution of outcomes) as we would have gotten by continuing the current human governance system.

The stronger form:

(Friendly-AI-ii) We define AI as 'friendly(U)' or 'aligned(U)' relative to a utility function U if it gives us the exact same score

...
DeekSeek v3: The Six Million Dollar Model
by Zvi — 2024-12-31T15:10:06.924Z

What should we make of DeepSeek v3?

DeepSeek v3 seems to clearly be the best open model, the best model at its price point, and the best model with 37B active parameters, or that cost under $6 million.

According to the benchmarks, it can play with GPT-4o and Claude Sonnet.

Anecdotal reports and alternative benchmarks tells us it’s not as good as Claude Sonnet, but it is plausibly on the level of GPT-4o.

So what do we have here? And what are the implications?

Table of Contents

What is DeepSeek v3 Techncially?.

Our Price Cheap.

Run Model Run.

Talent Search.

The Amazing Incredible Benchmarks.

Underperformance on AidanBench.

Model in the Arena.

Other Private Benchmarks.

Anecdata.

Implications and Policy.

What is DeepSeek v3 Techncially?

I’ve now had a chance to read their technical report, which tells you how they did it.

The big thing they did was use only 37B active tokens, but 671B total parameters, via a highly aggressive mixture of experts (MOE) structure.

They used Multi-Head Latent Attention (MLA) architecture and auxiliary-loss-free load balancing, and complementary sequence-wise auxiliary loss.

There were no rollbacks or outages or sudden declines, everything went smoothly.

They designed everything to be fully integrated and efficient, including together with the hardware, and claim to have solved several optimization problems, including for communication and allocation within the MOE.

This lets them still train on mostly the same 15.1 trillion tokens as everyone else.

They used their internal o1-style reasoning model for synthetic fine tuning data. Essentially all the compute costs were in the pre-training step.

This is in sharp contrast to what we saw with the Llama paper, which was essentially ‘yep, we did the transformer thing, we got a model, here you go.’ DeepSeek is cooking.

Our Price Cheap

It was a scarily cheap model to train, and is a wonderfully cheap model to use.

Their estimate of $2 per hour for H800s is if anything high, so their total training cost estimate of $5.5m total is fair, if you exclude non-compute costs, which is standard.

Inference with DeepSeek v3 costs only $0.14/$0.28 per million tokens, similar to Gemini Flash, versus on the high end $3/$15 for Claude Sonnet. This is as cheap as worthwhile models get.

Run Model Run

The active parameter count of 37B is small, but with so many different experts it does

...
I Recommend More Training Rationales
by Gianluca Calcagni — 2024-12-31T14:06:44.007Z

Some time ago I happened to read the concept of

training rationale

described by Evan Hubinger, and I really liked it. In case you are not aware: training rationales are a bunch of questions that ML developers / ML teams should ask themselves in order to self-assess pros and cons when adopting a certain safety approach.

I decided to take some time and analyse if there were obvious things missing in the rationales. In this post, I am attempting to explain the level of detail that I’d expect, and further questions I’d recommend to consider.

Remark: in the following, for simplicity, I am describing all AI models as “robots” - it doesn’t matter if the robots are physical or not.

Theory vs Practice

First things first: in any analysis, there are two distinct levels interacting with each other - aka, Theory vs Practice. Both are important, but they require very different mindsets to be effective and work organically together.

Theory is attempting to solve the problem: “I wonder if X could be achieved by means of Y”, where X and Y are some arbitrary things.
This level of analysis does not care how much you wish for X to happen, nor if Y is convenient. By using the concept of

direction of fit

: “The-Mind-Should-Fit-The-World” is the only direction that matters, and your personal desires / values / etc. shall not affect your analysis.
Practice is attempting to solve the problem: “I wish to achieve X by means of Y”, where X and Y are the same as above[1].
This level of analysis does not care if Y is the best tool to achieve X, nor if X is a valuable goal. By using the concept of direction of fit: “The-World-Should-Fit-The-Mind” is the only direction that matters, and your analysis shall only evaluate if any attempt would be successful, and in which measure.

When discussing AI Safety, both theory and practice are needed: for example, in this

video playlist

, Evan considers[2] two problems: Implementation Competitiveness and Performance Competitiveness. Let me recap how that goes.

Implementation Competitiveness

X = safety guarantees
Y = this training method
Theory: is this training method able to guarantee some levels of safety?
Practice: is this training method able to scale its safety guarantees on budget?

Performance Competitiveness

X = acceptable behaviour
Y = these performance restrictions
Theory: is this robot so exceedingly performant that I cannot a

...
The Plan - 2024 Update
by johnswentworth — 2024-12-31T13:29:53.888Z

This post is a follow-up to

The Plan - 2023 Version

. There’s also

The Plan - 2022 Update

and

The Plan

, but the 2023 version contains everything you need to know about the current Plan. Also see

this comment

and

this comment

on how my plans interact with the labs and other players, if you’re curious about that part.

What Have You Been Up To This Past Year?

Our big thing at the end of 2023 was

Natural Latents

. Prior to natural latents, the biggest problem with my

math on natural abstraction

was that it didn’t handle approximation well. Natural latents basically solved that problem. With that theoretical barrier out of the way, it was time to focus on crossing the theory-practice gap. Ultimately, that means building a product to get feedback from users on how well our theory works in practice, providing an empirical engine for iterative improvement of the theory.

In late 2023 and early 2024, David and I spent about 3-4 months trying to speedrun the theory-practice gap. Our target product was an image editor; the idea was to use a standard image generation net (specifically

this one

), and edit natural latent variables internal to the net. It’s conceptually similar to some things people have built before, but the hope would be that natural latents would better match human concepts, and therefore the edits would feel more like directly changing human-interpretable things in the image in natural ways.

When I say “speedrun” the theory-practice gap… well, the standard expectation is that there’s a lot of iteration and insights required to get theory working in practice (even when the theory is basically correct). The “speedrun” strategy was to just try the easiest and hackiest thing at every turn. The hope was that (a) maybe it turns out to be that easy (though probably not), and (b) even if it doesn’t work we’ll get some useful feedback. After 3-4 months, it indeed did not work very well. But more importantly, we did not actually get much useful feedback signal. David and I now think the project was a pretty major mistake; it cost us 3-4 months and we got very little out of it.

After that, we spent a few months on some smaller and more theory-ish projects. We worked out a

couple

more

pieces of the math of natural latents,

explained

what kind of model of semantics we’d ideally like (in terms of natural latents), ...
Zombies among us
by Declan Molony — 2024-12-31T05:14:07.929Z

I met a man in the Florida Keys who rents jet skis at $150/hour. Since nobody jet skis alone, he makes at least $300/hour. When there’s no customers he sits around watching sports. After work he plays with his two sons. I asked if he likes his lifestyle. He loves it.

Later when I was in Miami, I saw the walking dead. Zombies. The working class people who wished they weren’t alive. I remember what that’s like…

Wake up early. Take the bus with people who never make eye contact with you beyond a quick glance. Grab a stimulant, usually coffee, to force your brain awake. Clock in. Grind for the next eight hours. Maybe your boss is cool, but often not. Maybe your coworkers are cool, but often not. Find ways to cope with your physical or psychic pain (which if you don’t have yet, it’s just a matter of time until you do). Then hope the pain goes away since the only form of healthcare you can afford is prayer.

At the end of the day of doing hard physical labor that grinds your body to a pulp, or dealing with ungrateful customers you’re forced to smile at, your efforts are rewarded with the legal minimum wage: $8/hour. So 40 hours per week (assuming you don’t also have to work weekends) makes you $320—roughly what the jet ski guy makes in an hour.

Having an apartment to yourself is out of the question. Roommates are mandatory—you just have to decide how many you can tolerate, and if you don’t mind sharing a room with a complete stranger. Maybe your roommates are cool, but often not. Maybe your neighbors are cool and have the common decency to not blast music at two in the morning, but often not.

Whether at work or at home, your only privacy is confined to the bathroom. Except you never shit at home—you save that for work so you can get paid for it. You calculate how much per month you make shitting. It makes you feel like you’re getting one over on your employer.

Maybe everything in your apartment functions normally, but often not. When something breaks, it stays broken. You can ask the landlord to fix it, but it’ll take months for him to take a look. And when he does, he may blame you and say it’s not his problem. But your expectations weren’t too high anyway: when touring the apartment you asked if there were any amenities—the landlord laughed because he assumed you were joking.

You sit down in your bedroom with your frozen dinner and numb yourself w...