trentmkelly/LessWrong-43k huggingface dataset:
Anthropic AI made the right call
by bhauth — 2024-04-15T00:39:27.078Z

I've seen a number of people criticize Anthropic for releasing Claude 3 Opus, with arguments along the lines of:

Anthropic said they weren't going to push the frontier, but this release is clearly better than GPT-4 in some ways! They're betraying their mission statement!

I think that criticism takes too narrow a view. Consider the position of investors in AI startups. If OpenAI has a monopoly on the clearly-best version of a world-changing technology, that gives them a lot of pricing power on a large market. However, if there are several groups with comparable products, investors don't know who the winner will be, and investment gets split between them. Not only that, but if they stay peers, then there will be more competition in the future, meaning less pricing power and less profitability.

The comparison isn't just "GPT-4 exists" vs "GPT-4 and Claude Opus exist" - it's more like "investors give X billion dollars to OpenAI" vs "investors give X/3 billion dollars to OpenAI and Anthropic".

Now, you could argue that "more peer-level companies makes an agreement to stop development less likely" - but that wasn't happening anyway, so any pauses would be driven by government action. If Anthropic was based in a country that previously had no notable AI companies, maybe that would be a reasonable argument, but it's not.

If you're concerned about social problems from widespread deployment of LLMs, maybe you should be unhappy about more good LLMs and more competition. But if you're concerned about ASI, especially if you're only concerned about future developments and not LLM hacks like BabyAGI, I think you should be happy about Anthropic releasing Claude 3 Opus.
Clipboard Filtering
by jefftk — 2024-04-14T20:50:02.256Z

Here's a pattern I find pretty useful:

pbpaste | some_command | pbcopy

For example:

Converting spaces to tabs, for pasting into a spreadsheet
program: pbpaste | tr ' ' '\t' | pbcopy

Converting tabs and newlines to html table formatting:
pbpaste | sed 's/^/<tr><td>/' | sed
's/\t/<td>/g' | pbcopy

Escape angle brackets and ampersands for html: pbpaste |
sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g;' |
pbcopy (I used this on itself before pasting into this post.)

Convert newlines-indicate-paragraphs text to html:
pbpaste | sed 's/^/<p>/' | pbcopy (I use this in
putting together the kids text posts.)

Any time I want to do find-and-replace when working with
software that doesn't support it well.

The way this works is that pbpaste reads from the
clipboard as plain text, the intermediate command(s) transform the
text, and the pbcopy sends it back to the clipboard.

While these are the Mac commands it should be similar elsewhere. For
example, on Linux I'd use xsel, though that does
require paying attention to whether you want PRIMARY
(xsel) or CLIPBOARD (xsel -b).

Comment via: facebook, mastodon
A High Decoupling Failure
by Maxwell Tabarrok — 2024-04-14T19:46:09.552Z
Link: https://www.maximum-progress.com/p/a-high-decoupling-failure

High-decoupling vs low-decoupling or decoupling vs contextualizing refers to two different cultural norms, cognitive skills, or personal dispositions that change the way people approach ideas.

High-decouplers

isolate ideas from each other and the surrounding context. This is a necessary practice in science which works by isolating variables, teasing out causality and formalizing claims into carefully delineated hypotheses.

Low-decouplers, or contextualizers, do not separate ideas from their connotation. They treat an idea or claim as inseparable from the narratives that the idea might support, the types of people who usually make similar claims, and the history of the idea and the people who support it.

Decoupling is uncorrelated with the left-right political divide. Electoral politics is the ultimate low-decoupler arena. All messages are narratives, associations, and vibes, with little care paid to arguments or evidence. High decouplers are usually in the “

gray tribe

” since they adopt policy ideas based on metrics that are essentially unrelated to

what the major parties are optimizing for

.

My community prizes high decoupling and for good reason. It is extremely important for science, mathematics, and causal inference, but it is not an infallible strategy.

Should Legality and Cultural Support be Decoupled?

Debates between high and low decouplers are often marooned by a conflation of legality and cultural support. Conservatives, for example, may oppose drug legalization because their

moral disgust response

is activated by

open self-harm

through drug use and they do not want to offer cultural support for such behavior. Woke liberals are suspicious of free speech defenses for rhetoric they find hateful because they see the claims of neutral legal protection as a way to conceal cultural support for that rhetoric.

High-decouplers are exasperated by both of these responses. When they consider the costs and benefits of drug legalization or free speech they explicitly or implicitly model a controlled experiment where only the law is changed and everything else is held constant. Hate speech having legal protection does not imply anyone agrees with it, and drug legalization does not necessitate cultural encouragement of drug use. The constraints and outcomes to changes in law vs culture are completely different so objecting to one when you really mean the other is a bi...
A quick experiment on LMs’ inductive biases in performing search
by Alex Mallen — 2024-04-14T03:41:08.671Z

TL;DR: Based on a toy setting, GPT-3.5-turbo and GPT-4-turbo are best at search (by which I mean computing an argmax) when using chain-of-thought, but neither of them can do internal search when forced to work from memory over only a few token positions. Only GPT-4-turbo is to some extent able to do internal search, and only when given instructions before being shown the list of candidates it is tasked with argmaxing over.

Thanks to Jiahai Feng, Eli Lifland, and Fabien Roger for feedback and discussions that led to this experiment. This experiment was run and the post was written in a day, so some things may be rough around the edges.

One worry is that future AI systems might perform consequentialist reasoning internally (in “neuralese”), which may not be understandable to humans or other (trusted) AIs. We might not know if it is picking an action because it rates best according to what its operator would want, because it’s instrumentally useful for another goal, because it would lead to the best-looking tampered measurement, or because of some process other than search.

Three different ways to compute an argmax

This is a small toy experiment that tries to measure inductive biases of transformers regarding some of these capabilities. I attempt to measure to what extent RLHF’d LMs are capable of performing search in different settings. I consider only simple, brute-force, non-tree search here, because current LMs are not very capable at search. Given some collection of candidates, the kind of search we are interested in involves:

Computing a value for each candidate
Selecting an argmax

While it is hard to test whether a model is doing this via interpretability, we can devise behavioral experiments whose results can inform us about what the model is doing internally. I consider three settings:

Externalized search/chain-of-thought
Internal search in which value computation can be done at the token position(s) of the corresponding candidate
Internal search which must be done “from memory” over a small number of token positions

The Task

A. (2, [3, 2, 1])
B. (0, [2, 1, 3])
C. (1, [2, 3, 1])

Given a list of `(idx, digits)` pairs like the above, the model is tasked with picking out which one has the highest value of `digits[idx]`. There is never a tie, and `digits` is always a shuffled list with the same length as the number of pairs.

I implement the three settings...
UDT1.01 Essential Miscellanea (4/10) 
by Diffractor — 2024-04-14T02:23:38.755Z

This is the post with some needed concepts and discussion that didn't cleanly fit into any other section, so it might be a bit of a rambly mess.

Specifically, this post splits into two parts. One is assorted musings about when to defer to past-you vs current-you when making decisions, if we're permitting unplannable observations. The other is looking at Vanessa's dynamically consistent update rule, and combining that with our earlier reasoning on affineness to reduce the complexity of the finite-time-horizon version of UDT down from exponential to a smaller exponential.

Why Defer To Past-You?

So, one of the major features of the standard model (environmental observation space O, action space A, finite sequences of environmental observations are our plannables, there is a function e:O<n×(ΔA)O<n→ΔO that's affine in the second variable), which makes things a lot easier than they'd be in practice, is that there's an objective answer to what e is. All the versions of you agree on which tree they're in and how their actions affect how probability-mass flows through it. There's an objective answer to "how much does playing a bit more of this action over here affect what observation happens over there?"

But, if our information about how actions influence results doesn't descend from heaven at the start at the time, things get more complicated. If "my beliefs about how actions affect other places" are unplanned observations that you receive after thinking for a while, you can't just go "have me-at-the-start-of-time say what to do"

If you're having to learn which tree/environment you're in, and that learning consists of unplanned observations, and your actions affect how probability-mass flows through that tree, then this raises the problem that the yous which have made different observations can disagree on how their actions affect how probability-mass flows through the tree, and act at cross-purposes.

Is there any principled way of figuring out which sorts of things you should defer to past-you's wishes on, or which sorts of things you should delegate to future-you, when you allow unplanned observations to be something that happens sometimes? Well, to be honest, I'm still sorta confused about this, but I feel more confident than I used to be.

One of the standard arguments trotted out in UDT discussions, is that if you're not behaving in an updateless way, the early v...
[Cosmology Talks] New Probability Axioms Could Fix Cosmology's Multiverse (Partially) - Sylvia Wenmackers
by mako yass — 2024-04-14T01:26:38.515Z
Link: https://www.youtube.com/watch?v=MBeSoig4DPY

Sylvia is a philosopher of science. Her focus is probability and she has worked on a few theories that aim to extend and modify the standard axioms of probability in order to tackle paradoxes related to infinite spaces. In particular there is a paradox of the "infinite fair lottery" where within standard probability it seems impossible to write down a "fair" probability function on the integers. If you give the integers any non-zero probability, the total probability of all integers is unbounded, so the function is not normalisable. If you give the integers zero probability, the total probability of all integers is also zero. No other option seems viable for a fair distribution.

This paradox arises in a number of places within cosmology, especially in the context of eternal inflation and a possible multiverse of big bangs bubbling off. If every bubble is to be treated fairly, and there will ultimately be an unbounded number of them, how do we assign probability?

The proposed solutions involve hyper-real numbers, such as infinitesimals and infinities with different relative sizes, (reflecting how quickly things converge or diverge respectively).

The multiverse has other problems, and other areas of cosmology where this issue arises also have their own problems (e.g. the initial conditions of inflation); however this could very well be part of the way towards fixing the cosmological multiverse.

Sylvia: https://www.sylviawenmackers.be/ Paper: https://arxiv.org/abs/2308.12229

By the way, Shaun is beloved in the local EA community, and Sylvia's work has been cited around these parts more than once in discussions surrounding UDT.

Multiverse measure assignment is interesting, along with the anthropic binding problem, it's a necessary part of getting an indexical prior. While anthropic measure gives you comparisons between observers within universes, multiverse/time cosmological measure gives you a comparison between the universes.
Speedrun ruiner research idea
by lemonhope — 2024-04-13T23:42:29.479Z

Edit Apr 14: To be perfectly clear, this is another cheap thing you can add to your monitoring/control system; this is not a panacea or deep insight folks. Just a Good Thing You Can Do™.

Central claim: If you can make a tool to prevent players from glitching games *in the general case*, then it will probably also work pretty well for RL with (non-superintelligent) advanced AI systems.

Alternative title: RL reward+environment autorobustifier

Problem addressed: every RL thing ever trained found glitches/edge-cases in the reward function or the game/physics-sim/etc and exploited those until the glitches were manually patched

Months ago I saw a tweet from someone at OpenAI saying, yes, of course this happens with RLHF as well. (I can't find it. Anyone have it bookmarked?

Obviously analogous 'problem': Most games get speedrun into oblivion by gamers.

Portal: How To Get Outside Without Cheats (360)

Idea: Make a software system that can automatically detect glitchy behavior in the RAM of **any** game (like a cheat engine in reverse) and thereby ruin the game's speedrunability.

You can imagine your system gets a score from a human on a given game:

Game is unplayable:
score := -1
Blocks glitch:
score += 10 * [importance of that glitch]
Blocks unusually clever but non-glitchy behavior:
score -=5 * [in-game benefit of that behavior]
Game is laggy:[1]
score := score * [proportion of frames dropped]
Tool requires non-glitchy runs on a game as training data:
score -= 2 * [human hours required to make non-glitchy runs]
/ [human hours required to discover the glitch]

Further defense of the analogy between general anti-speedrun tool and general RL reward+environment robustifier:

Speedrunners are smart as hell
Both have similar fuzzy boundaries that are hard to formalize:
'player played game very well' vs 'player broke the game and didn't play it'
is like
'agent did the task very well' vs 'agent broke our sim and did not learn to do what we need it to do'

In other words, you don't want to punish talented-but-fair players.

Both must run tolerably fast (can't afford to kill the AI devs' research iteration speed or increase training costs much)
Both must be 'cheap enough' to develop & maintain

Breakdown of analogy: I think such a tool could work well through GPT alphazero 5, but probably not GodAI6

(Also if...
Text Posts from the Kids Group: 2020
by jefftk — 2024-04-13T22:30:05.326Z

Another round of liberating kid posts from

Facebook. For reference,
in 2020 Lily turned 6 and Anna turned 4.

(Some of these were from me; some were from Julia. Ones saying "me"
could mean either of us.)

2020-01-01

We went to the movies, and brought our own popcorn. When I passed the popcorn to Lily during the movie she was indignant, saying that we weren't supposed to bring in our own food. She ate one piece, but then said it wasn't ok and wouldn't eat more.

When the movie ended, Lily wanted us to tell the people at the concession stand and apologize: "Tell them! *Tell* them." She started trying to bargain with Julia: "I'll give you a penny if you tell them. Two pennies! Three pennies, *Five* pennies!"

But then we were outside and she was excitedly pretending to be Elsa, running down the sidewalk without a coat.

2020-01-10

I left for a trip on Tuesday afternoon, and beforehand Lily had asked me to give her one hour's notice before I left. I told her it would be about an hour from when she got home from school, but I forgot to give her warning at the actual one-hour mark. When I came up to read and cuddle with the kids 20 minutes before I left, she was angry that I hadn't given her enough notice. Then she went off and did something with paper, which I thought was sulking. I tried to persuade her to come sit on the couch with Anna and me and enjoy the time together, but she wouldn't.

Turns out she was making a picture and had wanted enough notice to finish it before I left. It is of her, Anna, and Jeff "so you won't forget us while you're gone." I assured her I will definitely not forget them, but that this was a very nice thing to be able to bring with me.

2020-01-13

Anna: "I will buy a baby at the baby store when I am a grownup, and I will be a mama like you! And I will work at Google and have the same job as my dad."

Pretty sure the kids don't think I have a real job. To be fair Google has much better food.

This was the first I had heard of the baby store. We'll see how that pans out for her.

2020-01-16

Me: Before you were born we thought about what to name you, and we thought Anna would be a good name. Do you think that's a good name?

Anna: No. I want to be named Bourbon.

2020-01-16

Anna: We're not going outside when we get Lily.

Me: How are we going to pick up Lily from school without going outside?...
What convincing warning shot could help prevent extinction from AI?
by Charbel-Raphaël — 2024-04-13T18:09:29.096Z

- Tell me father, when is the line

where ends everything good and fine?

I keep searching, but I don't find.

- The line my son, is just behind.

Camille Berger

There is hope that some “warning shot” would help humanity get its act together and change its trajectory to avoid extinction from AI. However, I don't think that's necessarily true.

There may be a threshold beyond which the development and deployment of advanced AI becomes essentially irreversible and inevitably leads to existential catastrophe. Humans might be happy, not even realizing that they are already doomed. There is a difference between the “point of no return” and "extinction." We may cross the

point of no return

without realizing it. Any useful warning shot should happen before this point of no return.

We will need a very convincing warning shot to change civilization's trajectory. Let's define a "convincing warning shot" as "more than 50% of policy-makers want to stop AI development."

What could be examples of convincing warning shots?

For example, a researcher I've been talking to, when asked what they would need to update, answered, "An AI takes control of a data center." This would be probably too late.

“That’s only one researcher,” you might say?

This

study

from Tetlock brought together participants who disagreed about AI risks. The strongest crux exhibited in this study was whether an evaluation group would find an AI with the ability to autonomously replicate and avoid shutdown. The skeptics would get from P(doom) 0.1% to 1.0%. But 1% is still not much… Would this be enough for researchers to trigger the fire alarm in a single voice?

More generally, I think studying more “warning shot theory” may be crucial for AI safety:

How can we best prepare the terrain before convincing warning shots happen?

e.g. How can we ensure that credit assignments are done well? For example, when Chernobyl happened, the credit assignments were mostly misguided: people lowered their trust in nuclear plants in general but didn’t realize the role of the USSR in mishandling the plant.

What lessons can we learn from past events? (Stuxnet, Covid, Chernobyl, Fukushima, the Ozone Layer).[1]
Could a scary demo achieve the same effect as a real-world warning shot without causing harm to people?
What is the time needed to react to a warning shot? One month, year, day?
More generally, what act

...
My experience at ML4Good AI Safety Bootcamp
by TheManxLoiner — 2024-04-13T10:55:46.621Z

TLDR

Experience was fantastic: I learnt a lot, made new friends and potential collaborators, and it was the step-change I needed to kickstart my contributions to AI safety.
Big lesson for me is that mechanistic interpretability (mech interp) is not the only way technical people can contribute to AI safety and mech interp is less likely than governance to impact the tail risks in the short-time frames.
There are many positive aspects and benefits to the bootcamp, beyond what you would expect from reading a surface level description of it.
Highly recommended to people considering contributing to AI safety. (And anybody can contribute!)

Introduction

I attended the

ML4Good

(ML4G) bootcamp in March 2024 in the UK, hosted at

CEEALAR

(the “EA Hotel”). The aim of this blogpost is to share my experience in case it helps people decide if they want to attend this or a similar boot camp. At the time of writing, I am not affiliated with ML4G except that I was a participant.

What is ML4G? It is a 10-day intensive AI safety bootcamp, with most days having activities from 9am to 9pm (with breaks and lunch of course). Activities include:

Technical lectures. Topics include: hyper-parameters and optimizers for deep learning, basics of deep RL, transformers, mechanistic interpretability and evals.
Working through jupyter notebooks, which were mostly simplified versions from the

ARENA bootcamp

.
Reading and discussions. Topics include: AI risks, potential solutions, prioritizing governance actions, AI safety strategy.
A half day group-based literature review with a presentation.
A 1.5 day group-based project with a presentation.
Presentation and Q&A with guest speakers. In this camp, the speakers were Charbel Segerie, Alan Chan, Charles Martinet.
1-on-1 discussions on career planning and post-camp actions.

Note that ML4G is young and AI safety is a dynamic field, so the format and syllabus are still being adjusted and refined.

Why I applied?

I did the

80,000 Hours Career Planning Template

at the end of 2021 and technical AI safety was a good fit for me, given my mathematical and data science background. As a result, I took the AGI Safety Fundamentals course in early 2022, but I actually ended up with quite a defeatist mindset and just put my head in the sand. Somehow my mindset changed in 2023, likely triggered by the large rush of developments around ChatGPT and other LLM...