trentmkelly/LessWrong-43k huggingface dataset:
Duct Tape security
by Isaac King — 2024-04-26T18:57:05.659Z

This is a linkpost for On Duct Tape and Fence Posts.

Eliezer writes about fence post security. When people think to themselves "in the current system, what's the weakest point?", and then dedicate their resources to shoring up the defenses at that point, not realizing that after the first small improvement in that area, there's likely now a new weakest point somewhere else.

Fence post security happens preemptively, when the designers of the system fixate on the most salient aspect(s) and don't consider the rest of the system. But this sort of fixation can also happen in retrospect, in which case it manifest a little differently but has similarly deleterious effects.

Consider a car that starts shaking whenever it's driven. It's uncomfortable, so the owner gets a pillow to put on the seat. Items start falling off the dash, so they get a tray to put them in. A crack forms, so they tape over it.

I call these duct tape solutions. They address symptoms of the problem, but not the root cause. The underlying issue still exists and will continue to cause problems until it's addressed directly.1

Did you know it's illegal to trade onion futures in the United States? In 1955, some people cornered the market on onions, shorted onion futures, then flooded the market with their saved onions, causing a bunch of farmers to lose money. The government responded by banning the sale of futures contracts on onions.

Not by banning futures trading on all perishable items, which would be equally susceptible to such an exploit. Not by banning market-cornering in general, which is pretty universally disliked. By banning a futures contracts on onions specifically. So of course the next time someone wants to try such a thing, they can just do it with tomatoes.

Duct-tape fixes are common in the wake of anything that goes publicly wrong. When people get hurt, they demand change, and they pressure whoever is in charge to give it to them. But implementing a proper fix is generally more complicated (since you have to perform a root cause analysis), less visible (therefore not earning the leader any social credit), or just plain unnecessary (if the risk was already priced in). So the incentives are in favor of quickly slapping something together that superficially appears to be a solution, without regards for whether it makes sense.

Of course not all changes in the wake of a disaster are duct-t...
Fundamental Uncertainty: Chapter 8 - When does fundamental uncertainty matter?
by Gordon Seidoh Worley — 2024-04-26T18:10:26.517Z

N.B. This is a chapter in a book about truth and knowledge. It is the first draft. I have since revised it. You can find the most up-to-date info/version on the book's website.

In the previous three chapters we broke apart our notions of truth and knowledge by uncovering the fundamental uncertainty contained within them. We then built back up a new understanding of how we're able to know the truth that accounts for our limited access to certainty. And while it's nice to have this better understanding, you might be asking yourself, so what? When is this understanding ever going to be useful to me?

The modern world faces tremendous challenges: growing political and social tensions, scientific disagreements that defy consensus, and existential threats from advanced technologies, just to name a few. We've thrown billions of dollars and millions of people at these challenges, yet they remain unsolved. It's my belief that they stubbornly refuse to yield to our efforts because attempts to solve them run up against the limits of our ability to know the truth, and we won't make progress towards solving them so long as we don't factor fundamental uncertainty into our efforts.

Thus, to the extent you care about tackling these challenges and making the world a better place, I argue that it's not just useful but necessary to understand fundamental uncertainty. Therefore, we'll take this chapter to explore what fundamental uncertainty has to say about a few of our world's problems.

The Culture War

As I write, a culture war rages across America. The combatants? Coastal progressives and their allies against heartland traditionalists and their supporters. The battlefield? News, politics, religion, business, and most of all social media. The weapons? Ostracism, protests, boycotts, and vague threats of secession and civil war. The spoils? The soul of the American people, and thereby the influence to shift the outcomes of policy debates over civil rights, climate change, gun control, and much else.

Why is the Culture War happening? If we look at broad social and economic trends, it seems inevitable. Rich, cosmopolitain, progressive city dwellers are growing in power and want to remake the nation in their image, while relatively poorer and more insular countryfolk are in decline and trying to hold on to their traditions. If these two groups could be isolated from each other, then perh...
Scaling of AI training runs will slow down after GPT-5
by Maxime Riché — 2024-04-26T16:05:59.957Z

My credence: 33% confidence in the claim that the growth in the number of GPUs used for training SOTA AI will slow down significantly directly after GPT-5. It is not higher because of (1) decentralized training is possible, and (2) GPT-5 may be able to increase hardware efficiency significantly, (3) GPT-5 may be smaller than assumed in this post, (4) race dynamics.

TLDR: Because of a bottleneck in energy access to data centers and the need to build OOM larger data centers.
Update: See Vladimir_Nesov's comment below for why this claim is likely wrong, since decentralized training seems to be solved. As a consequence, I updated my credence in the claim exposed in this post from 33% to 15%.

The reasoning behind the claim:

Current large data centers consume around 100 MW of power, while a single nuclear power plant generates 1GW. The largest seems to consume 150 MW.
An A100 GPU uses 250W, and around 1kW with overheard. B200 GPUs, uses ~1kW without overhead. Thus a 1MW data center can support maximum 1k to 2k GPUs.
GPT-4 used something like 15k to 25k GPUs to train, thus around 15 to 25MW.
Large data centers are around 10-100 MW. This is likely one of the reason why top AI labs are mostly only using ~ GPT-4 level of FLOPS to train new models.
GPT-5 will mark the end of the fast scaling of training runs.

A 10-fold increase in the number of GPUs above GPT-5 would require a 1 to 2.5 GW data center, which doesn’t exist and would take years to build, OR would require decentralized training using several data centers. Thus GPT-5 is expected to mark a significant slowdown in scaling runs. The power consumption required to continue scaling at the current rate is becoming unsustainable, as it would require the equivalent of multiple nuclear power plants. I think this is basically what Sam Altman, Elon Musk and Mark Zuckerberg are saying in public interviews.

The main focus to increase capabilities will be one more time on improving software efficiency. In the next few years, investment will also focus on scaling at inference time and decentralized training using several data centers.
If GPT-5 doesn’t unlock research capabilities, then after GPT-5, scaling capabilities will slow down for some time towards historical rates, with most gains coming from software improvements, a bit from hardware improvement, and significantly less than currently from scaling spending.
Scaling

...
Spatial attention as a “tell” for empathetic simulation?
by Steven Byrnes — 2024-04-26T15:10:58.040Z

(UPDATE Sept 2024: (1) I still think “local spatial attention” is a thing as described below. But I no longer think that it functions as a ‘tell’ for empathetic simulation, as in the headline claim. I have an alternate theory I like much better. Details coming in the next couple months hopefully. Also, (2) as noted below, I probably should have said “local spatial attention” instead of “spatial attention” throughout the post.)

(Half-baked work-in-progress. There might be a “version 2” of this post at some point, with fewer mistakes, and more neuroscience details, and nice illustrations and pedagogy etc. But it’s fun to chat and see if anyone has thoughts.)

1. Background

There’s a neuroscience problem that’s had me stumped since almost the very beginning of when I became interested in neuroscience at all (as a lens into AGI safety) back in 2019. But I think I might finally have “a foot in the door” towards a solution!

What is this problem? As described in my post

Symbol Grounding and Human Social Instincts

, I believe the following:

(1) We can divide the brain into a “Learning Subsystem” (cortex, striatum, amygdala, cerebellum and a few other areas) on the one hand, and a “Steering Subsystem” (mostly hypothalamus and brainstem) on the other hand; and a human’s “innate drives” (roughly equivalent to the reward function in reinforcement learning) are calculated by a bunch of specific, genetically-specified

“business logic”

housed in the latter subsystem;
(2) Some of those “innate drives” are related to human social instincts—a suite of reactions that are upstream of things like envy and compassion;
(3) It might be helpful for AGI safety (for reasons briefly summarized

here

) if we understood exactly how those particular drives worked. Ideally this would look like legible pseudocode that’s simultaneously compatible with behavioral observations (including everyday experience), with evolutionary considerations, and with a neuroscience-based story of how that pseudocode is actually implemented by neurons in the brain. (

Different example of what I think it looks like to make progress towards that kind of pseudocode

.)
(4) Explaining how those innate drives work is tricky in part because of the “symbol grounding problem”, but it probably centrally involves “transient empathetic simulations” (see

§13.5 of the post linked at the top

);
(5) …and therefore there n

...
Arch-anarchy
by Peter lawless  — 2024-04-26T15:05:14.984Z

By A [Editor: This article is reprinted from Extropy #5, Winter 1990. Extropy was published by The Extropy Institute]

Call to Arms

Down with the law of gravity!

By what right does it counter my will? I have not pledged my allegiance to the law of gravity; I have learned to live under its force as one learns to live under a tyrant. Whatever gravity's benefits, I want the freedom to deny its iron hand. Yet gravity reigns despite my complaints. "No gravitation without representation!" I shout. "Down with the law of gravity!"

Down with all of nature's laws!

Gravity, the electromagnetic force, the strong and weak nuclear forces - together they conspire to destroy human intelligence. Their evil leader? Entropy. Throw out the Four Forces! Down with Entropy!

Down with every limitation!

I call for the highest of all freedoms. Come, let us cast off all chains! We will make our own heaven. We will become our own gods.

I call for perfect self-rule; I call for arch-anarchy.

What is Arch-anarchy?

Arch-anarchy(1) represents the pinnacle of anarchy(2). While regular anarchists reject the legitimacy of the State's laws, as an arch-anarchist, I reject the validity of every law — be it human-made or otherwise — that hinders my will.

Anarchism encompasses diverse variations, yet they all ultimately converge towards arch-anarchy(3). Arch-anarchy emerges directly from individualist anarchism, a form of anarchism that prioritizes the individual will over the directives of statists(4). All alternative forms of anarchism eventually trace back to individualist anarchism. Why? Because it is individuals who make choices, not collectives. Therefore, any societal model an anarchist proposes requires convincing other individuals to embrace it. Should an anarchist attempt to impose their vision of utopia by force upon others upon failure, they become akin to yet another statist. This is a reality all anarchists must confront. Individualist anarchists not only acknowledge it but also embrace it. Consequently, once the supremacy of the individual will is recognized, the natural progression is towards arch-anarchy. Why settle for anything less?

Reality Explained

As an arch-anarchist, I view the universe as a battleground be...
Breadboarding a Whistle Synth
by jefftk — 2024-04-26T15:00:03.352Z

With my

electronic harp mandolin
project I've been enjoying working with analog and embedded audio
hardware. And a few weeks ago, after reading about Ugo Conti's

whistle-controlled
synth I wrote to him, he gave me a call, and we had a really
interesting conversation. And my existing combination of hardware for

my whistle synth [1] is bulky and
expensive. Which has me excited about a new project: I'd like to make
an embedded version.

Yesterday I got started on the first component: getting audio into the
microcontroller. I want to start with a standard dynamic mic, so I
can keep using the same mic for talkbox and whistle synth, so it
should take standard balanced audio on XLR as input. In a full
version this would need an XLR port, but for now I can pull back the
housing on an XLR cable:

I'm planning to use the Teensy again, which will be lowest noise at
3.3v peak-to-peak. What is the mic giving me? Whistling as loudly as
I can into my e835
dynamic mic I see 23mV peak-to-peak:

This means I need to amplify it by ~150x: 3.3v/23mV.

I found these
notes from MAS 836 pretty helpful, and sketched out an ac-coupled
differential amplifier, with a voltage divider to bias it up to a mean
voltage of +1.65v and the same protective diode arrangement I used on my pluck
sensor to keep from going above +3.3v or below 0v:

It also includes simple high-pass and a low-pass filters.

For choosing the component values, I set R1 to 33Ω because my
understanding is standard dynamic mics want a low input impedance.
Then I set R2 to 5.1kΩ (155x) because gain is R2/R1 and this
gives us close to the target gain I calculated above. The frequency
cutoff of the high-pass filter on the input is 1/(2 * pi * R1 * C1),
and I'd like this to be 500Hz (or a bit lower) because that's the
lowest I can whistle. Since we've already chosen R1, this means C1
should be 9.6µF; I used 10µF. For the low-pass filter I'd
like the frequency cutoff to be 2.5kHz (or a bit higher), the upper
end of my whistling range, and it should be 1/(2 * pi * R3 * C2). We
haven't picked either of these, but I think we want R3 to be low so we
don't make a lot of heat, so I did 7.5Ω. This means C2 should
be 8.4µF, but I didn't have one so I used a 4.7µF and
3.3µF in series for a total of 8µF. Note that for C1 I
rounded up and for C2 I rounded down, so that this wouldn't cut off
an...
An Introduction to AI Sandbagging
by Teun van der Weij — 2024-04-26T13:40:00.126Z

Summary: Evaluations provide crucial information to determine the safety of AI systems which might be deployed or (further) developed. These development and deployment decisions have important safety consequences, and therefore they require trustworthy information. One reason why evaluation results might be untrustworthy is sandbagging, which we define as strategic underperformance on an evaluation. The strategic nature can originate from the developer (developer sandbagging) and the AI system itself (AI system sandbagging). This post is an introduction to the problem of sandbagging.

The Volkswagen emissions scandal

There are environmental regulations which require the reduction of harmful emissions from diesel vehicles, with the goal of protecting public health and the environment. Volkswagen struggled to meet these emissions standards while maintaining the desired performance and fuel efficiency of their diesel engines (Wikipedia). Consequently, Volkswagen intentionally installed defeat devices in millions of vehicles to lower the amount of emissions only during tests, allowing the cars to emit up to 40 times the permitted levels of nitrogen oxides during normal driving conditions. The scandal showed corporate misconduct and a lack of effective oversight, and is a form of sandbagging. But clearly, cars and their manufacturers are different from AI systems and their developers. So, how does sandbagging relate to AI systems?

Sandbagging in the context of AI

Evaluations provide crucial information to determine the safety of AI systems which might be deployed or (further) developed. Frontier labs rely on evaluations to reduce catastrophic risks, as stated in e.g. OpenAI’s preparedness framework and Anthropic’s responsible scaling policy. Governmental institutions like the UK AI Safety Institute and the EU AI Office also utilize evaluations to manage risks. However, sandbagging can reduce the reliability of these evaluations. Both the developers of an AI system and the AI system itself have incentives to sandbag. For example, the developer might want to appease stakeholders, and the AI system might want to acquire more resources after an evaluation to better achieve its goals. Importantly, sandbagging is technically feasible[1] and neglected. We hope this post leads to a better understanding of the problem of s...
LLMs seem (relatively) safe
by JustisMills — 2024-04-25T22:13:06.221Z
Link: https://justismills.substack.com/p/llms-seem-relatively-safe

Post for a somewhat more general audience than the modal LessWrong reader, but gets at my actual thoughts on the topic.

In 2018 OpenAI

defeated the world champions

of Dota 2, a major esports game. This was hot on the heels of DeepMind’s AlphaGo performance

against Lee Sedol

in 2016, achieving superhuman Go performance way before anyone thought that might happen. AI benchmarks were being cleared at a pace which felt breathtaking at the time, papers were proudly published, and ML tools like Tensorflow (released in 2015) were coming online. To people already interested in AI, it was an exciting era. To everyone else, the world was unchanged.

Now Saturday Night Live sketches use sober discussions of AI risk

as the backdrop

for their actual jokes, there are

hundreds of AI bills

moving through the world’s legislatures, and Eliezer Yudkowsky

is featured

in Time Magazine.

For people who have been predicting, since well before AI was cool (and now passe), that it could spell doom for humanity, this explosion of mainstream attention is a dark portent. Billion dollar AI companies keep springing up and allying with the largest tech companies in the world, and bottlenecks like money, energy, and talent are widening considerably. If current approaches can get us to superhuman AI in principle, it seems like they will in practice, and soon.

But what if large language models, the vanguard of the AI movement, are actually safer than what came before? What if the path we’re on is less perilous than what we might have hoped for, back in 2017? It seems that way to me.

LLMs are self limiting

To train a large language model, you need an absolutely massive amount of data. The core thing these models are doing is predicting the next few letters of text, over and over again, and they need to be trained on billions and billions of words of human-generated text to get good at it.

Compare this process to

AlphaZero

, DeepMind’s algorithm that superhumanly masters Chess, Go, and Shogi. AlphaZero trains by playing against itself. While older chess engines bootstrap themselves by observing the records of countless human games, AlphaZero simply learns by doing. Which means that the only bottleneck for training it is computation - given enough energy, it can just play itself forever, and keep getting new data. Not so with LLMs: their source of data is human-produced text, and human-produced text is a...
Losing Faith In Contrarianism

by Bentham's Bulldog — 2024-04-25T20:53:34.842Z

Crosspost from my blog.

If you spend a lot of time in the blogosphere, you’ll find a great deal of people expressing contrarian views. If you hang out in the circles that I do, you’ll probably have heard of

Yudkowsky

say that dieting doesn’t really work, Guzey say that sleep is overrated, Hanson argue that medicine doesn’t improve health, various people argue for the lab leak, others argue for hereditarianism, Caplan argue that mental illness is mostly just aberrant preferences and education doesn’t work, and various other people expressing contrarian views. Often, very smart people—like Robin Hanson—will write long posts defending these views, other people will have criticisms, and it will all be such a tangled mess that you don’t really know what to think about them.

For a while, I took a lot of these contrarian views pretty seriously. If I’d had to bet 6-months ago, I’d have bet on the lab leak, at maybe 2 to 1 odds. I’d have had significant credence in Hanson’s view that healthcare doesn’t improve health until pretty recently, when Scott released his post explaining why it is wrong.

Over time, though, I’ve become much less sympathetic to these contrarian views. It’s become increasingly obvious that the things that make them catch on are unrelated to their truth. People like being provocative and tearing down sacred cows—as a result, when a smart articulate person comes along defending some contrarian view—perhaps one claiming that something we think is valuable is really worthless—the view spreads like wildfire, even if it’s pretty implausible.

Sam Atis has an article titled

The Case Against Public Intellectuals

. He starts it by noting a surprising fact: lots of his friends think education has no benefits. This isn’t because they’ve done a thorough investigation of the literature—it’s because they’ve read Bryan Caplan’s book arguing for that thesis. Atis notes that there’s a literature review finding that education has significant benefits, yet it’s written by boring academics, so no one has read it. Everyone wants to read the contrarians who criticize education—no one wants to read the boring lit reviews that say what we believed about education all along is right.

Sam is right, yet I think he understates the problem. There are various topics where arguing for one side of them is inherently interesting, yet arguing for the other side is boring. There ...