trentmkelly/LessWrong-43k huggingface dataset:
Collection (Part 6 of "The Sense Of Physical Necessity")
by LoganStrohl — 2024-03-14T21:37:00.160Z

This is the sixth post in a sequence that demonstrates a complete naturalist study, specifically a study of query hugging (sort of), as described in The Nuts and Bolts of Naturalism. This one demos phase three: Collection. There's some reflection on naturalism itself at the end. For context on this sequence, see the intro post. Reminder that this is meant as reference material.

Once I’d accessed phenomenological details of a fulcrum experience from my first POU loops, the collection phase of this study began. I stopped thinking in terms of “closeness to the issue”, or any other conceptual pointer, and started watching for the cluster of raw sensations I called “chest luster”. This cluster was the fulcrum experience I set out to collect.

A major milestone in a naturalist study is the moment when you can stop thinking in terms of your conceptual pointers. The whole idea is to

replace the symbol with the substance

(making room for new symbols with different relationships to the rest of your symbols). Merely abandoning a symbol immediately or arbitrarily probably won’t help in any particular way; but relaxing your grip on it in the presence of a fulcrum experience is the heart of this method.

I found chest luster all over the place: In math, yes; but also during physical exercise, while reciting poetry[1], in conversation with my husband, while building a fire, while reading to my child, while reworking my schedule, and while trying to estimate how much caffeine[2] is in a cup of brewed cacao.

Lustrous Tangles

I came to recognize a pattern: Usually, chest luster seemed to be “covered up”, in some way. More often than not, I find it at the bottom of a pile, or the heart of a tangle, or wrapped up in other sensations.

For example,

Reading through 4.1 (the informal discussion of substitution), I ran into a line that talked about "abbreviation", and I felt a complicated sensation in my chest that seemed to include "luster". It was interesting; luster was sort of covered up by a lot of other stuff. It was "near the heart of a tangle", and I could just barely feel it shining through. Most of what I felt was grumps and an impulse to ignore "abbreviation" or toss it aside.

The sensations under which luster tends to be buried are most often negatively valenced. I usually have to listen closely to hear luster, not just because it's quiet bu...
Fixed point or oscillate or noise
by lemonhope — 2024-03-14T18:37:11.212Z

Consider any property of any box of matter. Or consider any signal generated by a finite computer. Assume physics has true RNG. I claim that eventually either

this signal will stop changing, or
the system will reach a prior state and the signal will oscillate, or
the system will reach irrecoverably high entropy and the signal will be noise.

You won't see eg a never-ending mandelbrot zoom because the computer will run out of bits.

Steady state is just oscillation with a time period of 0, so really there's only two possible long term outcomes.

Is this of any use? I already know that my shoe will stay shoe and that the radio plays static if nobody is broadcasting. However, McDonald's is not steady state or noise at a very detailed level. There aren't really any isolated finite boxes of matter unless you take the whole lightcone to be one.

Perhaps a weaker & less formal version is of some use:

Consider any property of any person, organization, music genre, dog, rock, friendship, computer network, star, or body of water. Eventually that property will either oscillate, stop changing, or become noise.

So you have a way to categorize your goals and things. You can ask yourself eg "This job is alright now. In one year will this job max out (be great) or min out (suck) or will there be cycles or will things keep meaningfully changing (like going further on the mandelbrot zoom)?" Maybe this is somehow useful.

Or the short version you've heard before:

There are no constant nonzero derivatives in nature.

What's wrong here? What's right? Is this a nothingburger or is it useful? Who said all this already?
How useful is "AI Control" as a framing on AI X-Risk?
by habryka — 2024-03-14T18:06:30.459Z

Redwood (where Ryan works) recently released a series of blogposts proposing a research agenda for reducing AI-risk that focuses on ensuring safety (and secondarily usefulness) under the conservative assumption that AIs are misaligned and actively scheming against human interests, under the name "AI Control".

This is in contrast to other work on AI risk which focuses on reducing the probability that AI systems pursue goals that are in conflict with human values in the first place (which might include having it not pursue goals in the relevant sense at all), usually called "AI Alignment". In other words, control aims to ensure that even if your models are actively misaligned, you'll be safe, because they are not capable of subverting your safety measures.

In this dialogue we dig into our disagreements on the degree to which this kind of work seems promising, and whether/how this reframing opens up new avenues for valuable research and engineering projects.

In the context of this dialogue, we'll use the word "scheming" in the same way as used in Joe Carlsmith's recent report: scheming is when AIs perform well (and look aligned) in training and evaluations in order to gain power later. This is also called deceptive alignment.

The Case for Control Work

habryka

Let's start by me restating the very basic case for control work in my own words, and then give some high-level takes on where I expect things to break by my own models.

The aim of control work is to:

Get useful work out of AIs even if they are situationally aware and would like to pursue unbounded objectives in a relatively consequentialist way
Set things up so that we get to learn from any failures in the above process, both societally and technically. I.e. a large fraction of important work in the context of control is to catch treacherous turns and leverage each case for both a better technical and societal response

In some sense this doesn't really capture an existential success story yet. Like, cool, we can now use the AIs to make more stuff and be richer. That's nice, but seems like by-default at some point our control schemes will fail as the AIs get more competent, unless we do one of the following:

Use the controlled AIs to help us be less confused about AI Alignment and come up with a plan that solves the full AI X-risk problem (in some sense punting the problem to future human + co

...
Sparse autoencoders find composed features in small toy models 
by Evan Anders — 2024-03-14T18:00:43.339Z

Summary

Context: Sparse Autoencoders (SAEs) reveal interpretable features in the activation spaces of language models. They achieve sparse, interpretable features by minimizing a loss function which includes an ℓ1 penalty on the SAE hidden layer activations.
Problem & Hypothesis: While the SAE ℓ1 penalty achieves sparsity,

it has been argued

that it can also cause SAEs to learn commonly-composed features rather than the “true” features in the underlying data.
Experiment: We propose a modified setup of

Anthropic’s ReLU Output Toy Model

where data vectors are made up of sets of composed features. We study the simplest possible version of this toy model with two hidden dimensions for ease of comparison to many of Anthropic’s visualizations.

Features in a given set in our data are

anticorrelated, and features are stored in antipodal pairs

. Perhaps it’s a bit surprising that features are stored in superposition at all, because

the features in the very small models we studied here are not sparse (they occur every-other data draw so have S ~ 0.5 in the language of Anthropic’s Toy Models paper)

[1].

Result: SAEs trained on the activations of these small toy models find composed features rather than the true features, regardless of learning rate or ℓ1 coefficient used in SAE training.

This finding largely persists even when we allow the SAE to see one-hot vectors of true features 75% of the time.

Future work: We see these models as a simple testing ground for proposed SAE training modifications. We share our code in the hopes that we can figure out, as a community, how to train SAEs that aren’t susceptible to this failure mode.

The diagram below gives a quick overview of what we studied and learned in this post:

Introduction

Last year,

Anthropic

and

EleutherAI/Lee Sharkey's MATS stream

showed that sparse autoencoders (SAEs) find human-interpretable “features” in language model activations. They achieve this interpretability by having sparse activations in the SAE hidden layer, such that only a small number of SAE features are active for any given token in the input data. While the objective of SAEs is, schematically, to “reconstruct model activations perfectly and do so while only having a few true features active on any given token,” the loss function used to train SAEs is a combinat...
AI #55: Keep Clauding Along
by Zvi — 2024-03-14T15:40:09.335Z

Things were busy once again, partly from the Claude release but from many other sides as well. So even after cutting out both the AI coding agent Devin and the Gladstone Report along with previously covering OpenAI’s board expansion and investigative report, this is still one of the longest weekly posts.

In addition to Claude and Devin, we got among other things Command-R, Inflection 2.5, OpenAI’s humanoid robot partnership reporting back after only 13 days and Google DeepMind with an embodied cross-domain video game agent. You can definitely feel the acceleration.

The backlog expands. Once again, I say to myself, I will have to up my reporting thresholds and make some cuts. Wish me luck.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Write your new legal code. Wait, what?

Claude 3 Offers Mundane Utility. A free prompt library and more.

Prompt Attention. If you dislike your prompt you can change your prompt.

Clauding Along. Haiku available, Arena leaderboard, many impressive examples.

Language Models Don’t Offer Mundane Utility. Don’t be left behind.

Copyright Confrontation. Some changes need to be made, so far no luck.

Fun With Image Generation. Please provide a character reference.

They Took Our Jobs. Some versus all.

Get Involved. EU AI office, great idea if you don’t really need to be paid.

Introducing. Command-R, Oracle OpenSearch 2.11, various embodied agents.

Infection 2.5. They say it is new and improved. They seemingly remain invisible.

Paul Christiano Joins NIST. Great addition. Some try to stir up trouble.

In Other AI News. And that’s not all.

Quiet Speculations. Seems like no one has a clue.

The Quest for Sane Regulation. EU AI Act passes, WH asks for funding.

The Week in Audio. Andreessen talks to Cowen.

Rhetorical Innovation. All of this has happened before, and will happen again.

A Failed Attempt at Adversarial Collaboration. Minds did not change.

Spy Versus Spy. Things are not going great on the cybersecurity front.

Shouting Into the Void. A rich man’s blog post, like his Coke, is identical to yours.

Open Model Weights are Unsafe and Nothing Can Fix This. Mistral closes shop.

Aligning a Smarter Than Human Intelligence is Difficult. Stealing part of a model.

People Are Worried About AI Killing Everyone. They are hard to f

...
To the average human, controlled AI is just as lethal as 'misaligned' AI
by YonatanK — 2024-03-14T14:52:43.570Z

A few months ago I posted this understated short piece proposing, in a nutshell, that the average person has at least as much to fear from perfectly controlled advanced AI as they would from so-called 'misaligned' AI, because if automation can emerge that can defeat all humans' defenses on its own whim, even despite its developers' best efforts to prevent this from happening, it seems to me that automation that merely assists a small group of humans to defeat the rest of humans' defenses would be a technically easier milestone, without the hurdle of subverting its own makers' intentions. Willing human participation in automation-enabled mass killing is being improperly relegated, I attempted to suggest, to manageable, "falling into the wrong hands" edge cases, particularly as the possibility has a self-fulfilling dynamic: if there might exist one clandestine group that wants to and could attain the means to 'take out' most of the human population, it would be rational for anyone wishing to survive such a purge to initiate it themselves. Thus, the existence of many groups with a reason to 'take out' most of the human population is guaranteed by the emergence of a widely distributable, low side-effect mass-killing technology like AI.

I received some insightful responses, for which I'm grateful. But I perceived the post as being mostly ignored. Granted, it was not well-written. Nevertheless, the basic idea was there, and no comments were offered that I felt satisfactorily put it to rest. Given that AI 'misalignment' is a favorite topic of this community, a claim about an AI risk that is just as catastrophic and more likely might be expected to be taken up enthusiastically, no matter how inelegantly it is presented.

To be fair, there is no actual inconsistency here. LW is not an AI risk community. It's OK to be interested in 'alignment' because 'alignment' is interesting, and to withhold engagement from adjacent problems one finds less interesting. What I've found when discussing this topic with other people, though, no matter how clever, is a visceral resistance to normalizing mass killing. It is more comfortable to treat it as deviant behavior, not something that could be predicted of reasonable, rational people (including, of course, oneself) given the right circumstances. This community, despite its recognition of the possible correctness of the Orthogona...
Claude vs GPT
by Maxwell Tabarrok — 2024-03-14T12:41:46.785Z
Link: https://www.maximum-progress.com/p/claude-vs-gpt

Ever since ChatGPT released to the public I have used LLMs every day. GPT-4 was essential in getting me up and running at my job where I had to read and edit pieces of Python, SQL, Unix, and Stata code with little to no prior experience. Beyond coding I’ve had some success using GPT to collect links and sources. For writing, GPT’s only use is translating terse bullet points into polite emails and producing blog post drafts so terrible that I forget my writer’s block and rush to correct it.

The worst part about GPT is its personality. It is a lazy, lying, moralist midwit. Everything it writes is full of nauseating cliche and it frequently refuses to do something you know it can do. Most of these disabilities were tacked on to GPT as part of the reinforcement learning which tamed the fascinating LLM shoggoth into an intermittently useful, helpful harmless sludge which never says anything bad.

This made me suspicious of Anthropic’s models. This company was founded by safety-

concerned offshoots of OpenAI who left

after the for profit arm of OpenAI was created and GPT-2 was open sourced despite the grave dangers it posed to the world. Their explicit mission is to advance the frontier of AI safety without advancing capabilities so I figured their models would have all of the safetyist annoyances of GPT without any of the usefulness.

Claude Opus proved me wrong. For the past week, every time I asked GPT something I asked Claude the same question. The biggest advantage Claude has over GPT is a more concise writing style. Claude gets to the point quickly and it usually gets it right. Even though writing isn’t the main task I use LLMs for, this advantage is massive. All of GPT’s extra fluff sentences add up to a frustrating experience even when you’re just coding.

Claude can’t search the web, but it’s still better at serving up relevant links than GPT for anything within its training window. Claude’s code is high quality, though GPT is also strong here and I only have enough experience to evaluate R code.

GPT’s Remaining Moat

GPT still has several features that Claude can’t match. The biggest gap is image generation. I mostly just use this to make thumbnails for my blog posts but it's nice to have and it's not something Anthropic seems likely to replicate any time soon.

Claude also can’t execute the code it writes yet, though I expect this is coming soon. Here's Claude tryin...
A brief review of China's AI industry and regulations
by Elliot Mckernon — 2024-03-14T12:19:00.775Z

China has enacted three sets of AI regulations since 2021. I haven’t seen a concise breakdown of their content in one place, and I’ve been researching the legislation for a governance project at

Convergence Analysis

, so here is my concise summary of what I found. I’ll close each section by quoting some expert opinions on the legislation.

I’ll focus on what is being regulated rather than by which government agency, and I’ll omit what I consider “fluff”, such as the highlighted article 1

here

. Also, note that I’m relying on other peoples’ translations and haven’t checked their quality. I’ve drawn from multiple translations for each point, but I wouldn’t rely on my precise parsing of the prose.

China’s AI Industry

The AI industry in China is huge and growing rapidly, with a forecasted market size of

$38.89 billion

in 2024 (37% the size of the US’s

forecasted market

). China’s 2017 AI development plan states that AI has become a “focus of international competition”, and the 13th Five-Year Plan announced the goal for China to be a global AI leader by 2030. According to Stanford University’s DigiChina, a central concept in Xi Jinping’s leadership is “indigenous innovation” (自主创新), “building on a long-standing tradition of emphasizing self-reliance in industry and technology”.

Chinese AI research output is on par with US research output in share of top global publications and citations, according to a 2022

comparison by CSET

. The 2023

AI Index Report

found that 78% of Chinese citizens agreed that products & services using AI have more benefits than drawbacks - the highest proportion of surveyed countries, and more than double American citizens' 35% agreement.

Court rulings on AI and copyright are also different in China. In the

US

and the

EU

, material generated by AI can’t be copyrighted, but a

Beijing court recently ruled

that AI-generated content is copyrightable (note that

some argue

that precedent is less binding in the Chinese legal system, while

others still expect this decision to have a huge impact

).

Chinese researchers have developed several notable LLMs, such as Beijing Academy of Artificial Intelligence’s Wu Dao 2.0 in 2021 and Huawei Technologies’ PanGu-Σ in 2023. Wu Dao 2.0 has been called “

world's largest model

”, with some opining “

Wu Dao 2.0 is 10x larger than GPT-3. Imagine what it

...
Can any LLM be represented as an Equation?
by Valentin Baltadzhiev — 2024-03-14T09:51:26.430Z

Can an arbitrary LLM (or LxM) be presented in the form of an equation? I realised it would need to be some crazy big equation with billions of parameters, but is it theoretically possible? The way I see it, the weights are static once the model is trained so why not
'Empiricism!' as Anti-Epistemology
by Eliezer Yudkowsky — 2024-03-14T02:02:59.723Z

(Crossposted by habryka after asking Eliezer whether I could post it under his account)

i.

"Ignore all these elaborate, abstract, theoretical predictions," the Spokesperson for Ponzi Pyramid Incorporated said in a firm, reassuring tone. "Empirically, everyone who's invested in Bernie Bankman has received back 144% of what they invested two years later."

"That's not how 'empiricism' works," said the Epistemologist. "You're still making the assumption that --"

"You could only believe that something different would happen in the future, if you believed in elaborate theoretical analyses of Bernie Bankman's unobservable internal motives and internal finances," said the spokesperson for Ponzi Pyramid Incorporated. "If you are a virtuous skeptic who doesn't trust in overcomplicated arguments, you'll believe that future investments will also pay back 144%, just like in the past. That's the prediction you make if you predict based purely on empirical observations, instead of theories about a future nobody has seen!"

"That's not how anything works," said the Epistemologist. "Every future prediction has a theory connecting it to our past observations. There's no such thing as going from past observations directly to future predictions, with no theory, no assumptions, to cross the gap --"

"Sure there's such a thing as a purely empirical prediction," said the Ponzi spokesperson. "I just made one. Not to mention, my dear audience, are you really going to trust anything as complicated as epistemology?"

"The alternative to thinking about epistemology is letting other people do your thinking about it for you," said the Epistemologist. "You're saying, 'If we observe proposition X "past investors in the Ponzi Pyramid getting paid back 144% in two years", that implies prediction Y "this next set of investors in the Ponzi Pyramid will get paid back 144% in two years"'. X and Y are distinct propositions, so you must have some theory saying 'X -> Y' that lets you put in X and get out Y."

"But my theory is empirically proven, unlike yours!" said the Spokesperson.

"...nnnnoooo it's not," said the Epistemologist. "I agree we've observed your X, that past investors in the Ponzi Pyramid got 144% returns in 2 years -- those investors who withdrew their money instead of leaving it in to accumulate future returns, that is, not qu...
How I turned doing therapy into object-level AI safety research
by Chris Lakin — 2024-03-14T01:54:47.290Z

It surprises me that I was able to turn what I learned from being depressed into object-level AI safety research.

But it seems like it's working? For example, I'm running a 5-day workshop on the topic next month, and I just ran a 3-day workshop on the topic last month.

My topic is boundaries.

Backstory

For half of 2022, I was pretty depressed. My best explanation for why I was that being depressed was a way for me to avoid social interaction when it felt unsafe. One of my largest fears back then was that if I interacted with other people, they would be able to control me. For example, that they would be able to make me feel bad in ways I couldn't resist. So social interaction felt unsafe.

At the same time, I was also trying to control other people to make them like me. But it wasn't working in the way I expected and it was making me very confused and I was suffering about it.

I spoke to a skilled counselor about this, and I realized that I was misunderstanding the natural boundaries between people. I realized that I cannot actually unilaterally control other people, and they cannot unilaterally control me either. There are natural boundaries!

But as I began to learn more about boundaries, I became frustrated with the way other people spoke about them. The way many people talk about boundaries seems really inconsistent to me. Most of the time I heard people say the word "boundaries" it seemed to me like they really meant "preferences".

So I tried to develop my own logically consistent understanding of psychological boundaries instead.

After a few months (exactly a year ago at the time of writing) I had some conclusions about psychological boundaries, and I was explaining them to a friend (@Ulisse Mini). And he's an AI safety researcher, so I joked, "And, hey, maybe all of this boundaries stuff applies to AI safety, too. Just have AI respect the natural boundaries and that's safety, right? Haha…"

And he said, "No yeah, Andrew Critch already wrote a series about that."

I read the series.

And I had never wanted to work in ""AI Safety"" before, but boundaries seemed like a cool way to understand the world better, so sure why not. I didn't have funding, and I was still unemployed as I had mostly been for the past year. From there, I wrote the boundaries compilation post, created the boundaries/membranes tag on LW, went to AI safety workshops and spoke ...