trentmkelly/LessWrong-43k huggingface dataset:
AI as systems, not just models
by Andy Arditi — 2024-12-21T23:19:05.507Z
Link: https://andyrdt.com/posts/ai-systems

Inspired by Christopher Potts'

recent lecture

, I've written some notes on thinking about "AI systems" rather than "AI models". Most of the framings presented here can be attributed to Potts' lecture.

For those who already think about AI in terms of systems rather than models in isolation, this piece is unlikely to provide new insights. However, I have found thinking with the "AI systems" framework to be very useful, and think it's worth articulating explicitly.

Thanks to Egg Syntax and Bilal Chughtai for providing thoughtful feedback on an earlier draft.

Thinking in terms of systems, rather than models

In a

recent lecture

, Christopher Potts articulates a fundamental perspective shift in how we should think about AI: moving from thinking about models to thinking about systems.

A language model, on its own, is just a function - a mapping from input text to a probability distribution over possible next tokens. This raw functionality is potentially powerful but inherently passive - without additional basic things such as a prompt or a sampling method, a model can't just "start talking" - it's just an inert file sitting on disk. To actually make a model practically useful, we need to hook it up into a system.

A system consists of several components:

Model. The model is the core component that powers the system, akin to the engine of a car. It offers raw predictive power, but on its own it is passive and directionless - its full potential is only harnessed through integration into a broader system.
Sampling strategy. At the most basic level, we need a method to convert the model's raw probability distributions into actual outputs. This could be as simple as always choosing the most likely token, or more complicated, such as doing search over multiple potential completions. Meta-sampling strategies like "best-of-n" can also be employed, generating multiple sampled rollouts and then selecting the best one based on some measure of quality.
Prompting strategy. Models are input-output functions - they take text and produce distributions over possible next tokens - and thus their outputs are entirely dependent on their inputs. The particular framing of input queries can dramatically affect overall performance, with even subtle variations in prompts leading to significant differences in behavior. Strategies like

few-shot prompting

and&

...
Towards a Unified Interpretability of Artificial and Biological Neural Networks
by jan_bauer — 2024-12-21T23:10:45.842Z

Neuroscience and mechanistic interpretability share a common goal: understanding neural networks, either biological or artificial. This is reflected in the convergent evolution of these domains – from interpreting single neurons to abstract features, and more recently, to functional representations. Yet, a significant number of approaches in either field remain unknown to the other. Bridging these methods could benefit both.

Neuroscience for Biological Networks. The study of the nervous system began in the late 19th century with Ramón y Cajal, who coined the 'neuron doctrine' by discovering how single cells respond to electrical currents. Advances in technology later allowed for simultaneous recordings from multiple neurons, leading to key discoveries like

Hubel and Wiesel's identification

of cells responsive to specific inputs, and

Georgopoulos' work

on population vector coding. In recent years, several groups have focused on the representational geometry of neural codes and its functional implications (

Kaschube

et al.

(2010)

,

Stringer

et al.

(2019)

,

Moser

et al.

(2022)

).

Mechanistic Interpretability for Artificial Networks. Similarly, monosemanticity – the idea that individual neurons or features correspond to single, interpretable concepts – has been a north star guiding mechanistic interpretability research (

Goh & Olah (2021)

). Later work has generalized this view, finding that neurons can encode several concepts (

Elhage & Olah (2022)

), which may be disentangled using nonlinear decoders. Now, several researchers are pushing beyond linear representations and monosemantic features, for example by exploring the geometry of neural manifolds (

Hoogland et al. (2023)

,

Mendel et al. (2024)

,

Marshall et al. (2024)

,

Leask et al. (2024)

).

Towards a Unified Interpretability for Neural Networks. Bridging the structure of representations to their function constitutes the frontier of interpretability research for both biological and artificial networks. To approach it, both fields have developed powerful methods, including manifold geometry, statistical physics, topology, nonlinear decoding, and causal probing. This distributed knowledge is a powerful resource to leverage for both fields.
Robbin's Farm Sledding Route
by jefftk — 2024-12-21T22:10:01.175Z

There's a nice

sledding
hill a short drive from our house. It has an area with a gentler
slope, where we would go when the kids were younger, but at this point
the main attraction is a slope that drops ~60ft over ~400ft:

The big downside is that it's steep all the way down to a fence at
the bottom. The town usually puts hay bales at the bottom, and while
growing up I was told they were put there after too many kids died
from hitting the fence, I can't find the news stories that I would
expect to accompany any such deaths so I think that's an urban
legend. The key to having a good time on this hill is to sled the
right route:

10-foot contours

You pick up a lot of speed on the initial downhill, turn left to go
parallel to the fence while continuing down the hill, and then shed
your speed by turning uphill at the very end. I like to use my hands
to steer and brake, with sturdy
mittens—I don't know how people can tolerate using heels,
with the amount of snow spray that kicks up.

Afterwards you walk back up the other side, following the essential
etiquette of staying out of the sledding zone:

At this point Lily (10y) can do it herself; Anna (8y) still prefers to
go with me. Before letting Lily go down alone (here, or any other
slope where there's anything you could hit) I did have her show me
she could reliably bail. And we don't let them use tubes, where it's
hard to steer and you can end up going backwards.

(This park also has a great slide, source of the name "Big Slide
Park", which has
reopened after almost two years, a
year behind what was originally planned.)

Comment via: facebook, mastodon, bluesky
AGI with RL is Bad News for Safety
by Nadav Brandes — 2024-12-21T19:36:03.970Z

I haven’t found many credible reports on what algorithms and techniques have been used to train the latest generation of powerful AI models (including OpenAI’s o3).

Some reports

suggest that reinforcement learning (RL) has been a key part, which is also consistent with what OpenAI officially

reported about o1

three months ago.

The use of RL to enhance the capabilities of AGI[1] appears to be a concerning development.

As I wrote previously

, I have been hoping to see AI labs stick to training models through pure language modeling. By “pure language modeling” I don’t rule out fine-tuning with RLHF or other techniques designed to promote helpfulness/alignment, as long as they don’t dramatically enhance capabilities. I’m also okay with the LLMs being used as part of more complex AI systems that invoke many instances of the underlying LLMs through chain-of-thought and other techniques. What I find worrisome is the underlying models themselves trained to become more capable through open-ended RL.

The key argument in my original post was that AI systems based on pure language modeling are relatively safe because they are trained to mimic content generated by human-level intelligence. This leads to weak pressure to surpass human level. Even if we enhance their capabilities by composing together many LLM operations (as in chain of thought), each atomic operation in these complex reasoning structures would be made by a simple LLM that only tries to generate a good next token. Moreover, the reasoning is done with language we can read and understand, so it’s relatively easy to monitor these systems. The underlying LLMs have no reason to lie in very strategic ways[2], because they are not trained to plan ahead. There is also no reason for LLMs to become agentic, because at their core they are just prediction machines.

Put RL into the core training algorithm and everything changes. Models are now explicitly trained to plan ahead, which makes all kinds of strategic and agentic behaviors actively rewarded. At this point we can no longer trust the chains of thought to represent their true reasoning, because models are now rewarded based on the final results that these chains lead to. Even if you put a constraint requiring the intermediate tokens to appear like logical reasoning, the models may find ways to produce seemingly-logical tokens that encode addition...
Better difference-making views
by MichaelStJules — 2024-12-21T18:27:45.552Z

Summary

I make three arguments against standard difference-making risk aversion that I find compelling as a consequentialist, despite my substantial sympathy for something like difference-making risk aversion:

It privileges a default option, literally as or like an act-omission distinction, or is otherwise unmotivated (more).
It may lead generally to inaction or otherwise favouring the default option, due to multitudes of backfire risks and butterfly effects (more).
It can count against you persuading others to take actions that are less risky from their perspectives, and when the default option is a status quo option, it favours the status quo (more).
Some of these arguments also apply to difference-making ambiguity aversion.

I describe multiple versions of or modifications of difference-making views that don't have these problems. Some can also be combined. They are:

use accounts that treat downsides and upsides relative to the fixed default option more symmetrically (more),
statewise sorting, so that we aren’t sensitive to how outcome distributions correspond statewise (more), and
not privileging inaction as the default option to make comparisons with respect to, but potentially treating every option this way (more).

Most of these allow comparisons between two options to depend on whether a third is available (more) and could lend even more support to risky bets like longtermism or on invertebrates than risk neutrality already does (more).

Acknowledgements

Thanks to Derek Shiller, Silvester Kollin and Anthony DiGiovanni for helpful feedback. All errors are my own.

Background

For further background on and discussion of difference-making risk aversion, see Clatterbuck, 2024 and Greaves et al., 2022. I only illustrate briefly here.

If you're risk averse with respect to money, it's better to have a $50 with certainty than a 50% chance of $0 and a 50% chance of $100, even though your expected amount of money is $50 either way.

Suppose we have some quantity for which it is better that its value be higher, like money or total welfare. In economics and according to expected utility theory, this quantity has some utility, captured by an increasing utility function, u, of the quantity, and we would pick an option A with maximum expected utility, E[u(A)].

On the standard economic and expected utility theory account of ris...
Review: Good Strategy, Bad Strategy
by L Rudolf L — 2024-12-21T17:17:04.342Z
Link: https://nosetgauge.substack.com/p/review-good-strategy-bad-strategy

I used to think that all generic strategy advice was pointless. After all, the point of a strategy is to achieve a thing, and to achieve a thing you just think hard about how to best do it and then work hard to do it. I said this to my friend Dewi, who said that this is mostly true, but there is an exception: Good Strategy, Bad Strategy by Richard Rumelt. Dewi was right.

The book has some principles. In particular: a good strategy should include a diagnosis of the problem, an overall guiding policy, and a set of coherent actions. A laundry list of actions, a goal, or a vague idea of which direction to move in are not strategies.

But most of the book's value is reading a bunch of examples and soaking up the thinking style embedded in them. Therefore, this review is mostly a series of vignettes taken directly from Rumelt's text, that hopefully 80/20 the value of the book (or is a helpful reminder if you've read it). Rumelt's vignettes are great.

I don't include anything from Rumelt's lengthy attack on most of what passes for "strategy consulting". I also won't mention the part where he spends most of a chapter giving a pop-sci history of physics from Galileo to dark matter in order to segue into a point about Starbucks' business strategy—he's an emeritus professor, so presumably he can just do this.

Example

“In 1805, England had a problem. Napoléon had conquered big chunks of Europe and planned the invasion of England. But to cross the Channel, he needed to wrest control of the sea away from the English. Off the southwest coast of Spain, the French and Spanish combined fleet of thirty-three ships met the smaller British fleet of twenty-seven ships. The well-developed tactics of the day were for the two opposing fleets to each stay in line, firing broadsides at each other. But British admiral Lord Nelson had a strategic insight. He broke the British fleet into two columns and drove them at the Franco-Spanish fleet, hitting their line perpendicularly. The lead British ships took a great risk, but Nelson judged that the less-trained Franco-Spanish gunners would not be able to compensate for the heavy swell that day. At the end of the Battle of Trafalgar, the French and Spanish lost twenty-two ships, two-thirds of their fleet. The British lost none. Nelson was mortally wounded, becoming, in death, Britain’s greatest naval hero. Britain’s naval dominance was ensured and

...
Last Line of Defense: Minimum Viable Shelters for Mirror Bacteria
by Ulrik Horn — 2024-12-21T08:28:14.860Z

Epistemic status: We are moderately confident in the feasibility of serially filtered shelters for 14-log protection as well as cost estimates, but more uncertain about how mirror biology might develop in the real world.

Originally published on the EA Forum. Minor edits for LW adaptation.

Please sign up

here

for ~bi-monthly updates on shelter/refuges work.

Summary and main take-aways

I want to be super clear: I honestly feel deep down that most efforts, at least at this early time, should go into

preventing

a mirror bio catastrophe. I feel conflicted about working on preparing for disaster but unfortunately my skill set and experience lends itself much better to this than working on prevention (I would love to be proved wrong!).
One particular threat

shelters

can defend against has recently become much clearer:

The motivating risk is the intentional or accidental release of

catastrophically damaging mirror microbes

(“mirror bio”).
We think that if we can design a shelter for an extreme scenario, it will also likely be useful in a range of other serious scenarios where the pathogen concentration is far lower.
We’re most concerned about non-state actors (like terrorists), but also researchers (who might have lab leaks).

A positively pressurized plastic “bubble”, supplied by serial high-grade air filters seems robust enough and affordable.

This is inexpensive off-the-shelf technology.
At scale the complete material cost could be below $10k/person per year of protection. however, a first, commercially available unit providing only the protective shell is expected to retail for $39k and can accommodate up to 4 people.

pictured: a cut-through view of an inflatable shelter with a small room, large room, and airlock

My background

I am Ulrik Horn and have worked on these shelters for over 2 years. I’m considering manufacturing them commercially, and have recently started a company. I wanted to share my theory of change in case I'm making important mistakes. I'm also interested in talking to people who might want to help, and especially people, whether in the public or private sector, who would be interested in buying one, or know someone that might. The values of the company will be front and center and are Honesty, Responsibility and Family Security - values that deeply align with my own values. The hope is that a clear, co...
Elon Musk and Solar Futurism
by transhumanist_atom_understander — 2024-12-21T02:55:28.554Z

2024 is the year it became clear that we're actually going to do battery energy storage. That it was time to stop soberly reminding anyone too excited about solar that storage is an unsolved problem. In fact I personally became very excited about solar, and briefly thought that this year's wave of battery installations was just the beginning of building Solartopia, where we run on solar power all day and batteries all night. I no longer think we're quite there, and I have a new set of sober reminders, but I'll save those for the next post. I'll start by showing you the data that finally got me on board the ship of solar futurism, and how I learned that Elon Musk has been a solar futurist this whole time.

This is Tesla's battery energy storage sales by quarter, from their 2024 Q3 report:

If it helps you get a sense of scale, a typical nuclear reactor produces about 1 GW of power. It would take 12 GWh to provide that power all night (assuming night is 12 hours long). More than that was installed in just six months. Or, to describe a more typical use of battery energy storage, it takes 4 GWh to provide that power for the four hours or so between sundown and lights out, which has been the biggest issue for solar power. That's what they've been installing every quarter, and in recent quarters about twice that. Of course Tesla isn't the only battery energy storage provider, but I won't get into the rest of the market and just note that at least as of 2023 they had the greatest market share for battery energy storage. I think the sales by GWh are mostly of their Megapack, introduced 2019, which is a shipping container-sized battery energy storage module. Utilities buy dozens or hundreds of them, attach them to a concrete slab, and that's a battery energy storage facility.

But why is Tesla, a car company, supplying these units for the electric grid? Rewinding to 2015, when the first Tesla Energy battery energy storage products were introduced. There was no Megapack yet, but there was a home storage product, and a larger unit that was used in early battery energy storage facilities. You might think that it was just something to do with excess battery production. But, quoting from an article in Nature:

At a press conference in Los Angeles on 30 April, the company’s charismatic founder Elon Musk said that the firm’s lithium-ion batteries would enable economies to move to l

...
Good Reasons for Alts
by jefftk — 2024-12-21T01:30:03.113Z

I originally wrote this a year ago, but just now found it in my
drafts. Not sure why I didn't post it then.

One flavor of response I got with my post on deanonymizing accounts was
roughly:

Why not just go ahead and post the list of alts? It's not like people
sneakily using multiple accounts is something we should be
encouraging!

I do agree there are ways people abuse alts / burners /
throwaways. You shouldn't be making new accounts to get around bans,
or to give the illusion that an idea has more independent support than
it really has ("sockpuppeting"). But there are also some good reasons
to use them:

Separating aspects of your life. Perhaps you post your
political views under one account and your work views under another.
This could be because an aspect of your life is stigmatized, you have
a career where you need to interact with people with a wide range of
political views, or you just would prefer to keep clear boundaries.

Lowering how much weight people put on your views. Some people
consistently post insightful and well-considered comments or have a
position in their community where they are highly respected. Having a
separate account to post idle thoughts and participate like a regular
human lets them post things others won't take overly seriously.

Protecting yourself from retaliation. Perhaps you are one of
many who know about some harm, but due to threats none of you would be
willing to post under your real names. Some kind of anonymous
whistleblowing could be very positive.

What's acceptable can also vary by community. For example, on Reddit
making a new account just so the username can contribute to a joke
you're setting up is part of the culture, but on a more serious forum
that would be seen as an abuse of the account system.

Overall, I find the benefits of keeping everything under one identity
are really high, and I suspect many people underestimate the value
they'd get from that approach. But if others decide in their
situation the tradeoffs point the other way, and aren't abusing alts
to mislead people, then go for it.

This is part of why I posted on
deanonymization: if it's important to you that readers not link
your alt back to your main identity then you need to take active steps
beyond just using a different username.
Updating on Bad Arguments
by Guive — 2024-12-21T01:19:15.686Z
Link: https://guive.substack.com/p/updating-on-bad-arguments

Here is an intuitively compelling principle: hearing a bad argument for a view shouldn’t change your degree of belief in the view. After all, it is possible for bad arguments to be offered for anything, even the truth. For all you know, plenty of good arguments exist, and you just happened to hear a bad one.

But this intuitive principle is wrong. If you thought there was a reasonable chance you might hear a good argument but you end up hearing a bad one, that provides some evidence against the view.

Imagine I am pretty convinced that octopus and cuttlefish developed their complex nervous systems independently, and that their last common ancestor was not at all intelligent. Let’s say my p(intelligent ancestor) is 0.1. Imagine I have a friend, Richard, who disagrees. Richard is generally a smart and reasonable person. Prior to hearing what he has to say, I think there is a moderately high chance that he will give a good argument that the last common ancestor of octopus and cuttlefish was highly intelligent. But Richard’s argument is totally unconvincing; the evidence he cites is irrelevant. Should my p(intelligent ancestor) now be:

(A) 0.1

(B) < 0.1

The correct answer is (B). To explain why requires some simple probability math. The law of total probability holds that the probability of a proposition is equal to the average of the conditional probabilities given each possible observation, weighted by the probability of making the observation. In symbols, it is:

p(h)=p(h∣e)⋅p(e)+p(h∣¬e)⋅p(¬e)

Where the probability Richard gives you good evidence is p(e), the probability he doesn’t is p(¬e), the probability that the last common ancestor of octopus and cuttlefish was intelligent is p(h) (and the | symbol means “given”, so p(h|e) is the probability of h given that Richard gives you good evidence). The law of total probability can be straightforwardly derived from the Kolmogorov axioms and the definition of conditional probability. This brings us to another closely related theorem, the law of total expectation:

p(h)=E[p(h∣Y)]

Where Y is the set of all possible observations (here, getting good evidence or getting bad evidence from Richard), and the fancy E means expectation or mean. In its application to Bayesian updating, the law of total expectation implies that the prior is equal to the expectation of the posterior. This is why the principle of “conservation of expecte...