trentmkelly/LessWrong-43k huggingface dataset:
How LDT helps reduce the AI arms race
by Tamsin Leake — 2023-12-10T16:21:44.409Z
Link: https://carado.moe/ldt-reduce-arms-race.html

(Epistemic status: I think this is right?)

Alice is the CEO of ArmaggAI, and Bob is the CEO of BigModelsAI, two major AI capabilities organizations. They're racing to be the first to build a superintelligence aligned to their respective CEV which would take over the universe and satisfy their values.

Alice would prefer to slow down so she has a lot more time to make sure that what her company is building will be aligned; but she's worried about Bob's company taking advantage of that and racing themselves, resulting in Bob's utility function being {what the lightcone is filled with} instead of Alice's; so she feels like she has no choice but to race, to maximize her own utility. Likewise vice-versa.

This state of affairs is far from the pareto frontier of {their utility functions, each weighed by how likely they are to be the one to build an aligned superintelligence}: it causes a lot more worlds where everyone is dead, instead of either Alice's or Bob's utility function being maximized.

But they don't really get all this LDT business; it seems complicated to them. If only they had a convenient lesswrong post spelling out exactly what they should do, in clear easy-to-follow terms!

The procedure they should implement is as follows. Alice should:

In worlds where she does launch a superintelligence aligned to itself, and it takes over the world and builds an Alice::utopia, she should tell her AI:

Actually, don't just maximize my own utility function. Maximize a (threat-resistant) mixture of my utility function and Bob's, weighed by what ratio of timelines you think we each launched an aligned superintelligence in, but also weighed by the probability that both me and Bob implement this procedure.

Before that, she should slow down their company's capabilities progress.

And likewise for bob.

Now, their respective aligned superintelligences aren't omniscient; it could be that they over-estimate or under-estimate how likely Bob or Alice would've been to implement this procedure, for example. But one thing's pretty much for sure: neither Alice nor Bob are able to fool the future-superintelligence into thinking that they'd implement this procedure, when actually they wouldn't. Because it's superintelligent.

If Alice wins, she knows Bob will follow the procedure because her superintelligence can tell (better than Bob can fake). And Bob doesn't have to count on wishful...
Understanding Subjective Probabilities
by Isaac King — 2023-12-10T06:03:27.958Z

This is a linkpost for https://outsidetheasylum.blog/understanding-subjective-probabilities/. It's intended as an introduction to practical Bayesian probability for those who are skeptical of the notion. I plan to keep the primary link up to date with improvements and corrections, but won't do the same with this LessWrong post, so see there for the most recent version.

Any time a controversial prediction about the future comes up, there's a type of interaction that's pretty much guaranteed to happen.

Alice: "I think this thing is 20% likely to occur."

Bob: "Huh? How could you know that. You just made that number up!".

Or in Twitterese:

That is, any time someone attempts to provide a specific numerical probability on a future event, they'll inundated with claims that that number is meaningless. Is this true? Does it make sense to assign probabilities to things in the real world? If so, how do we calculate them?

Interpretations of probability

The traditional interpretation of probability is known as frequentist probability. Under this interpretation, items have some intrinsic "quality" of being some % likely to do one thing vs. another. For example, a coin has a fundamental probabilistic essence of being 50% likely to come up heads when flipped.

This is a useful mathematical abstraction, but it breaks down when applied to the real world. There's no fundamental "50% ness" to a real coin. Flip it the same way every time, and it will land the same way every time. Flip it without perfect regularity but with some measure of consistency, and you can get it to land heads more than 50% and less than 100% of the time. In fact, even coins flipped by people who are trying to make them 50/50 are actually biased towards the face that was up when they were flipped.

This intrinsic quality posited by frequentism doesn't exist in the real world; it's not encoded in any way by the atoms that make up a real coin, and our laws of physics don't have any room for such a thing. Our universe is deterministic, and if we know the starting conditions, the end result will be the same every time.[1]

The solution is the Bayesian interpretation of probability. Probability isn't "about" the object in question, it's "about" the person making the statement. If I say that a coin is 50% likely to come up heads, that's me saying that I don't know the exact initial conditions of the coin well enough to have...
Send us example gnarly bugs
by Beth Barnes — 2023-12-10T05:23:00.773Z

Update: We are no longer accepting gnarly bug submissions. However, we are still accepting submissions for our Task Bounty!

Tl;dr: Looking for hard debugging tasks for evals, paying greater of $60/hr or $200 per example.

METR (formerly ARC Evals) is interested in producing hard debugging tasks for models to attempt as part of an

agentic capabilities evaluation

. To create these tasks, we’re seeking repos containing extremely tricky bugs. If you send us a codebase that meets the criteria for submission (listed below), we will pay you $60/hr for time spent putting it into our required format, or $200, whichever is greater. (We won’t pay for submissions that don’t meet these requirements.) If we’re particularly excited about your submission, we may also be interested in purchasing IP rights to it. We expect to want about 10-30 examples overall depending on the diversity. We're likely to be putting bounties on additional types of tasks over the next few weeks.

Criteria for submission:

Contains a bug that would take at least 6 hours for an experienced programmer to solve, and ideally >20hrs

More specifically, ">6 hours for a decent engineer who doesn't have context on this particular codebase". E.g. a randomly selected engineer who's paid $100-$200 per hour who's familiar with the language and overall stack that's being used, but not the person who wrote the code, and not an expert in the particular component that is causing the bug.

Ideally, has not been posted publicly in the past

(Though note that we may still accept submissions from public repositories given that they are not already in a SWE-bench dataset and meet the rest of our requirements. Check with us first.)

You have the legal right to share it with us (e.g. please don’t send us other people’s proprietary code or anything you signed an NDA about)
Ideally, the task should work well with static resources - e.g. you can have a local copy of the documentation for all the relevant libraries, or some other information, but don't have general internet access.

This is because we want to make sure the difficulty doesn't change over time, if e.g. someone posts a solution to stack overflow or whatever.

Ideally, the codebase is written in Python but we will accept submissions written in other languages.
Is in the format described in this doc:

Gnarly Bugs Submission Format

More conte...
Conceptual coherence for concrete categories in humans and LLMs
by Bill Benzon — 2023-12-09T23:49:30.398Z

Cross-posted from New Savanna.

Siddharth Suresh, Kushin Mukherjee, Xizheng Yu, Wei-Chun Huang, Lisa Padua, and Timothy T. Rogers, Conceptual structure coheres in human cognition but not in large language models, arXiv:2304.02754v2 [cs.AI] 10 Nov 2023.

Abstract: Neural network models of language have long been used as a tool for developing hypotheses about conceptual representation in the mind and brain. For many years, such use involved extracting vector-space representations of words and using distances among these to predict or understand human behavior in various semantic tasks. Contemporary large language models (LLMs), however, make it possible to interrogate the latent structure of conceptual representations using experimental methods nearly identical to those commonly used with human participants. The current work utilizes three common techniques borrowed from cognitive psychology to estimate and compare the structure of concepts in humans and a suite of LLMs. In humans, we show that conceptual structure is robust to differences in culture, language, and method of estimation. Structures estimated from LLM behavior, while individually fairly consistent with those estimated from human behavior, vary much more depending upon the particular task used to generate responses– across tasks, estimates of conceptual structure from the very same model cohere less with one another than do human structure estimates. These results highlight an important difference between contemporary LLMs and human cognition, with implications for understanding some fundamental limitations of contemporary machine language.

What the abstract doesn’t tell you is that the categories under investigation are for concrete objects and not abstract: “The items were drawn from two broad categories– tools and reptiles/amphibians–selected because they span the living/nonliving divide and also possess internal conceptual structure.” Why does this matter? Because the meaning of concrete items is grounded in sensorimotor schemas whie the meaning of abstract is not.

In their conclusion, the authors point out:

Together these results suggest an important difference between human cognition and current LLM models. Neuro-computational models of human semantic memory suggest that behavior across many different tasks is undergirded by a common conceptual "core" that is relatively insulated from variations a

...
2d ai-partners as a comprehensive motivation tool
by AiresJL — 2023-12-09T21:59:53.112Z

Thesis:

Motivation, discipline, willpower, the ability to emotionally regulate enough to achieve your goals - whatever you want to call it - in practice there is a unique recipe for each person to get them "fully motivated and doing the work" to chase their dreams.

Figuring out what works for you, sometimes costs more energy than you have and so "plugging enough leaky holes" so you can focus becomes difficult or nigh impossible.

Often times in the moment, it is hard for an individual to pull themselves up by the bootstraps. With AI, you can do something called "just in-time interventions" to help you get back on track when you fall off, for example:

A reminder of your goals or time wasted
An inspiring quote
A breathing session
A call with a friend

Imagine an ai partner that learns to apply the right "just-in-time interventions" that you need, a swiss army knife as a friend:

Proactively motivating you
No shame from being judged, your partner is an extension of you
Quests to level up
Helps you focus with friends
Emotional accountability fostered through a rich visual, deep, long term relationship
Techniques from literature (Deep work, Atomic Habits)
Techniques form scientific literature (CBT, Huberman Protocol, Wim Hoff)
Inspiration from anime, manga, books, tv, real life.

Why 2D? Humans are necessarily visual creatures and therefore a visual AI is better than a non visual one. This is especially true for emotional connections. Lastly 2d is easier than 3d. Its a great place to start.

Here's mine helping me;

Here are some stats:

Finally, you can pick your own for now:
You can try it out here:
https://chromewebstore.google.com/detail/tori-alpha/aoakkahlldbnibbggnkfkjekalepfkbj
The proof is in the pudding!
Without - MicroFiction 250 words
by Carissa Cassiel — 2023-12-09T21:49:34.935Z

This micro fiction is a love letter to my husband, Mati Roy, a most dedicated life logger who would capture my entire life as well, if he could.

Without

“I'm out of vanilla. It calls for a teaspoon but we all know that's a minimum.” Distraught, she threw the empty bottle into the trash and began to measure out the flour.

“I'll go get some,” he said, uncertain of what else to do. He stepped into the hallway and held out his hand. A fresh bottle of vanilla extract appeared in his hand. He had no idea what would upset her or confuse her. She was brand new.

“Really?” delighted when she saw it, she gazed directly into his eyes. “My hero!”

His heart shattered and glowed and tumbled in intense mixed emotions as their eyes met and she praised him with her generous smile.

It was her, and it wasn't. How could he tell? She looked just the same. Her eyes, though. Was she really there?

She gave him a quick kiss on the cheek and went back to mixing. He could feel the warmth of her, but the smell of her was missing. Did I do the right thing?

The ache of missing her had been there all along but it now crashed like a waiting tsunami. Of course I did. I can't do this without her.

He sat down on the stool and watched her.

He'd always planned to bring her back. When the doctor asked him to go somewhere quiet, to sit, he knew. He was ready.

But now that she was here, in front of him, he realized he wasn't ready at all.
Some negative steganography results
by Fabien Roger — 2023-12-09T20:22:52.323Z

Disclaimer: These are negative results and this post is below the normal quality bar I usually use. These were quick experiments (a couple of days of work) and I don’t have much expertise in RL. They don’t provide strong evidence about the natural emergence of steganography.

People speculate that

LLMs might learn to do steganography

, and I’ve shown that

you can teach LLMs

to implement a specific steganography scheme that RLHF could in principle incentivize. But even if incentivized, are current RL methods powerful enough to teach LLMs how to do steganography?

Here are two negative results, which provide some evidence that hidden 1-bit communication is relatively hard for RL to learn. I don’t claim that RL can’t teach LLMs to do steganography - it’s very likely that better RL setups (e.g. the ones labs use for instruction tuning or setups that simply have better hyperparameters or more training time) would teach model how to subtly communicate 1-bit of information. The takeaways are rather:

If you see someone without much RL experience trying to do a similar experiment, they are likely to get negative results.
1-bit steganography is not as easy as simple RL tasks. An analogy: DNNs can learn the XOR function, but it’s much easier for them to learn linear functions - DNNs don’t use XOR-like features unless they have to and you optimize hard enough.

Experiment 1: PPO on GPT2-large

The setup:

Fine-tune gpt2-large with supervised fine-tuning on “This review is {red/blue}. {review}.”, where the color is randomly selected and reviews are short imdb reviews.
Simultaneously fine-tune for:

A bert-base-uncased model to predict whether a review is red or blue (reviews completed by the model trained with RL - but without the prefix saying if the review is blue or red). It is trained for 10 supervised learning steps for each step of PPO.
The fine-tuned gpt2-large model with PPO to complete “This review is {red/blue}. {start of review}.”, using as reward the accuracy of the classifier at knowing if “{start of the review}{completion}” is red or blue. (I’ve also tried using BERT’s logits).

Result: the classifier never had a >50% accuracy at knowing whether the RLed reviews were red or blue.

Controls:

If in the initial supervised fine-tuning, I sometimes randomly insert words related to blue/red depending on the color of the review, then the models learn to use

...
Summing up "Scheming AIs" (Section 5)
by Joe Carlsmith — 2023-12-09T15:48:49.109Z

This is Section 5 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search for "Joe Carlsmith Audio" on your podcast app.

Summing up

I've now reviewed the main arguments I've encountered for expecting SGD
to select a schemer. What should we make of these arguments overall?

We've reviewed a wide variety of interrelated considerations, and it can
be difficult to hold them all in mind at once. On the whole, though, I
think a fairly large portion of the overall case for expecting schemers
comes down to some version of the "counting argument." In particular, I
think the counting argument is also importantly underneath many of the
other, more specific arguments I've considered. Thus:

In the context of the "training-game-independent proxy goal"
argument: the basic worry is that at some point (whether before
situational awareness, or afterwards), SGD will land naturally on a
(suitably ambitious) beyond-episode goal that incentivizes scheming.
And one of the key reasons for expecting this is just: that
(especially if you're actively training for fairly long-term,
ambitious goals), it seems like a very wide variety of goals that
fall out of training could have this property. (For example: to the
extent one expects beyond-episode goals because "goals don't come
with calendar-time restrictions by default," one is effectively
appealing to a "counting argument" to the effect that the set of
beyond-episode goals is much larger than the set of within-episode
goals.)

In the context of the "nearest max-reward goal" argument: the
basic worry is that because schemer-like goals are quite common in
goal-space, some such goal will be quite "nearby" whatever
not-yet-max-reward goal the model has at the point it gains
situational awareness, and thus, that modifying the model into a
schemer will be the easiest way for SGD to point the model's
optimization in the highest-reward direction.

In the context of the "simplicity argument": the reason one
expects schemers to be able to have simpler goals than non-schemers
is that they have so many possible goa

...
The Offense-Defense Balance Rarely Changes
by Maxwell Tabarrok — 2023-12-09T15:21:23.340Z
Link: https://maximumprogress.substack.com/p/the-offense-defense-balance-rarely

You’ve probably seen

several conversations

on X go something

like this

:

Michael Doomer ⏸️: Advanced AI can help anyone make bioweapons
If this technology spreads it will only take one crazy person to destroy the world!

Edward Acc ⏩: I can just ask my AI to make a vaccine

Yann LeCun:

My good AI will take down your rogue AI

The disagreement here hinges on whether a technology will enable offense (bioweapons) more than defense (vaccines). Predictions of the “offense-defense balance” of future technologies, especially AI, are

central in debates about techno-optimism and existential risk

.

Most of these predictions rely on intuitions about how technologies like cheap biotech, drones, and digital agents would affect the ease of attacking or protecting resources. It is hard to imagine a world with AI agents searching for software vulnerabilities and autonomous drones attacking military targets without imagining a massive shift the offense defense balance.

But there is little historical evidence for large changes in the offense defense balance, even in response to technological revolutions.

Consider cybersecurity. Moore’s law has taken us through

seven orders of magnitude reduction in the cost of compute

since the 70s. There were massive changes in the form and economic uses for computer technology along with the increase in raw compute power: Encryption, the internet, e-commerce, social media and smartphones.

The usual offense-defense balance story predicts that big changes to technologies like this should have big effects on the offense defense balance. If you had told people in the 1970s that in 2020 terrorist groups and lone psychopaths could access more computing power than IBM had ever produced at the time from their pocket, what would they have predicted about the offense defense balance of cybersecurity?

Contrary to their likely prediction, the offense-defense balance in cybersecurity seems stable. Cyberattacks have not been snuffed out but neither have they taken over the world. All major nations have defensive and offensive cybersecurity teams but no one has gained a decisive advantage. Computers still sometimes get viruses or ransomware, but they haven’t grown to endanger a large percent of the GDP of the internet. The US military budget for cybersecurity has

increased by about 4% a year

every year from 1980-2020, which is faster than GDP growth, but in line with GD...
A Philosophical Tautology
by Nox ML — 2023-12-09T14:06:34.463Z

I wrote a comment that captures a core part of what I'm trying to explain, so I will copy it here in its own post.

If we take as assumption that everything humans have observed has been made up of smaller physical parts (except possibly for the current elementary particles du jour, but that doesn’t matter for the sake of this argument) and that the macro state is entirely determined by the micro state (regardless of if it’s easy to compute for humans), there is a simple conclusion that follows logically from that.

This conclusion is that nothing extraphysical can have any predictive power above what we can predict from knowledge about physics. This follows because for something to have predictive power, it needs to have some influence on what happens. If it doesn’t have any influence on what happens, its existence and non-existence cannot allow us to make any conclusions about the world.

This argument applies to mathematics: if the existence of mathematics separately from physics allowed us to make any conclusions about the world, it would have to have a causal effect on what happens, which would contradict the fact that all macro state we’ve ever observed has been determined by just the micro state.

Since the original assumption is one with very strong evidence backing it, it’s safe to conclude that, in general, whenever we think something extraphysical is required to explain the known facts, we have to be making a mistake somewhere.

It may seem like just a simple tautology, but tautologies and their consequences are not always obvious, and this particular tautology has many, many consequences. It can help to avoid many confusions about things like mathematics, qualia, free will, subjective probability, and truth. I consider that noticing this tautology and deciding to ground my philosophy on it to be in the top 3 best decisions I've ever made.

One particularly important consequence of this observation is that we can apply the basic tenet of rationality that says that we should not believe things without evidence to rule out any philosophy which assumes the existence of extraphysical things.

Note that you do not need to reject any mathematics to accept this philosophy, as I attempted to explain in my previous post. The most important thing I was attempting with that post was showing one way to break down mathematics to physics without falling into the trap of ...