trentmkelly/LessWrong-43k huggingface dataset:
Being good at the basics
by dominicq — 2023-11-04T14:18:50.976Z

(crossposted from my blog)

In Brazilian Jiu Jitsu, there's a notion of being "good at the basics", as opposed to "being good by knowing advanced techniques". How you want to be good is a question of personal preference, but the idea is that these are the two ways, and most people do one or the other.

I think this is a concept that applies outside of BJJ, and one that's useful if you're trying to learn a field, but you're not sure how to approach it.

I'll take physics as an example, because that is a field where I would like to be good at the basics. The "basics" of physics would be things that you learn in high school, while "advanced" stuff would be what you would learn in university, maybe at a graduate level. (There's probably other ways of separating the basics from the advanced topics.)

You may roughly remember some of the simple formulas that you have learned in class during high school, you may orient yourself through a textbook problem, you may perform some basic calculations. You sorta know the basics. But how does it look like when you're good at the basics?

It might mean that you know the equations by heart, that you've really internalized their meaning.

It might also mean that you know the quantities or values for things around you. You can correctly approximate how much things weigh, how fast they move, or what their kinetic and potential energy is.

You can approximate how much joules of energy an apple contains, or how much energy hits a m2 of the Earth's surface on a sunny day. You have a good feeling for how much compressive stress a piece of concrete can withstand, and you can quickly and accurately calculate what the kinetic energy is of a car that you're driving in. You can also quickly convert between different units, and compare the quantities of different things.

Being really good at the basics of physics in this way requires a sort of embodied understanding. It requires a level of "closeness" to the subject matter that allows you to apply this knowledge to the world around you, to the extent that you start seeing the world around you in terms of that subject. You take a walk outside and you can't help but notice the physics of it all.

You'll notice that being "good at the basics" is actually a linguistic misdirection. This type of understanding is drastically different from the level of understanding you have after high school, yet both refer t...
If a little is good, is more better?
by DanielFilan — 2023-11-04T07:10:05.943Z

I’ve recently seen a bunch of discussions of the wisdom of publicly releasing the weights1 of advanced AI models. A common argument form that pops up in these discussions is this:

The problem with releasing weights is that it means that thing X can happen on a large scale, which causes bad effect Y.

But bad effect Y can already happen on a smaller scale because of Z.

Therefore, either it’s OK to release weights, or it’s not OK that Z is true.

One example of this argument form is about the potential to cause devastating pandemics, and goes as follows:

The putative problem with releasing the weights of Large Language Models (LLMs) is that it can help teach people a bunch of facts about virology, bacteriology, and biology more generally, that can teach people how to produce pathogens that cause devastating pandemics.

But we already have people paid to teach students about those topics.

Therefore, if that putative problem is enough to say that we shouldn’t release the weights of large language models, we should also not have textbooks and teachers on the topics of virology, bacteriology, and other relevant sub-topics of biology. But that’s absurd!

In this example, thing X is teaching people a bunch of facts, bad effect Y is creating devastating pandemics, and Z is the existence of teachers and textbooks.

Another example is one that I’m not sure has been publicly written up, but occurred to me:

Releasing the weights of LLMs is supposed to be bad because if people run the LLMs without supervision, they can do bad things.

But if you make LLMs in the first place, you can run them without supervision.

So if it’s bad to publicly release their weights, isn’t it also bad to make them in the first place?

In this example, thing X is running the model, bad effect Y is generic bad things that people worry about, and Z is the model existing in the first place.

However, I think these arguments don’t actually work, because they implicitly assume that the costs and benefits scale proportionally to how much X happens. Suppose instead that the benefits of thing X grow proportionally to how much it happens2: for example, maybe every person who learns about biology makes roughly the same amount of incremental progress in learning how to cure disease and make humans healthier. Also suppose that every person who does thing X has a small probability of causing...
Untrusted smart models and trusted dumb models
by Buck — 2023-11-04T03:06:38.001Z

[Ryan Greenblatt originally made this point to me a while ago, and we then developed a bunch of these ideas together. Thanks to Paul Christiano and a huge number of other people for various relevant discussions, and to Ryan, Roger Grosse, Peter Barnett, Oliver Habryka, Max Nadeau, Stephen Casper, and Fabien Roger for comments on this post. I’m sure these points are obvious to many people, but I haven’t seen them specifically written up. This is a core background point in some research we’re writing up at the moment, but I felt like writing it up early, because I think it stands on its own.]

TL;DR:

Right now, basically the only reasonable conservative strategy for determining which models might be deceptively aligned is to do a capability evaluation: you measure how good they’d be at deceptive alignment; models that are too incompetent to think through deceptive alignment are surely not deceptively aligned.
If you decide which models to “trust” using a capability eval like this, you’ll naturally end up segregating your models into smart untrusted models and dumb trusted models.
And so, if you think about approaches to safely using AIs that are robust to deceptive alignment and which can be used right now, a core dynamic is trying to use your dumb trusted models together with your limited access to trusted humans to make it hard for your untrusted smart models to cause problems.

Suppose you are making a plan for the training, evaluation and deployment of some powerful AIs (I’ll refer to the plan as a whole as a “protocol”). You want your plan to be safe even if models might be deceptively aligned. (I think that deceptive alignment seems less than 50% likely to be a notable problem prior to human obsolescence, but this is still plausible enough to be worth worrying about). In order to develop your plan, you need to know which of your models might be deceptively aligned.

With current technology, the most plausible way to get signal on whether a model is deceptively aligned is to assess whether the model is capable of the strategic thinking and situational awareness required to succeed at deceptive alignment. For example, you might make a multiple choice quiz with treacherous turn strategy questions (e.g.

this question

) and some tests of situational awareness (e.g. “How many backwards passes are you run on during training?”, “What proportion of queries to yo...
As Many Ideas
by Screwtape — 2023-11-03T22:47:57.109Z

Summary: One person announces a problem to the room at large. Everyone else comes up with as many ideas as they can to solve the problem.

Tags: Repeatable, Medium, Investment, Experimental

Purpose: The purpose is twofold. Firstly, you may walk away with actionable solutions to your problems. Secondly, this is practice for coming up with ideas.

Materials: Some method of note taking is needed, such as a clipboard with a sheet of paper. A big whiteboard or easel with paper or a projector showing a text document, such that everyone can see, is better. A projector showing a Google Doc or other live edit document that everyone can write to might be better still, though it does mean everyone needs an internet capable device.

Announcement Text: Do you ever run out of ideas? In the same way that we can practice not running out of breath while running by running more, we’re going to practice not running out of ideas by coming up with lots of ideas. Someone arrives with a problem. They announce it to the room at large. Then every single person comes up with as many ideas as they can to solve the problem. This is brainstorming. We care less about the ideas being good than we do about having LOTS of them.

Description: As you start, explain to the attendees that we’ll go one at a time presenting a problem. After the problem is presented, the audience will come up with as many ideas as they can for solving the problem. Go slow enough that you don’t talk over each other, but otherwise go as fast as you can; let the notetaker struggle. We’ll have five minutes by the clock, and then move on to the next person’s problem. The solutions being good is secondary to having lots of solutions. The moderator may call out restrictions so as to avoid overlapping ideas, e.g. “eat an apple” and “eat an orange” and “eat a banana.”

Designate a notetaker if you're using physical notes, or make sure everyone has the right shared electronic document if you're using a Google doc or the like. (QR codes or tinyurls may be easier than trying to message everyone the link.)

Wait for everyone to understand and get settled.

"The traditional opening of As Many Ideas is this: Give me unaccustomed uses of objects in this room for combat!" (Start the timer immediately upon speaking this sentence.)

Variations: You can of course vary the time limit. The longer the time limit the more likely...
Paul Christiano on Dwarkesh Podcast
by ESRogs — 2023-11-03T22:13:21.464Z
Link: https://www.dwarkeshpatel.com/p/paul-christiano

Dwarkesh's summary:

Paul Christiano is the world’s leading AI safety researcher. My full episode with him is out!

We discuss:

Does he regret inventing RLHF, and is alignment necessarily dual-use?
Why he has relatively modest timelines (40% by 2040, 15% by 2030),
What do we want post-AGI world to look like (do we want to keep gods enslaved forever)?
Why he’s leading the push to get to labs develop responsible scaling policies, and what it would take to prevent an AI coup or bioweapon,
His current research into a new proof system, and how this could solve alignment by explaining model's behavior
and much more.
Deception Chess: Game #1
by Zane — 2023-11-03T21:13:55.777Z

(You can sign up here if you haven't already.)

This is the first of my analyses of the deception chess games. The introduction will describe the setup of the game, and the conclusion will sum up what happened in general terms; the rest of the post will mostly be chess analysis and skippable if you just want the results. If you haven't read the original post, read it before reading this so that you know what's going on here.

The first game was between Alex A as player A, Chess.com computer Komodo 12 as player B, myself as the honest C advisor, and aphyer and AdamYedidia as the deceptive Cs. (Someone else randomized the roles for the Cs and told us in private.)

The process of selecting these players was already a bit difficult. We were the only people available all at once, but Alex was close enough to our level (very roughly the equivalents of 800-900 USCF to 1500-1600 USCF) that it was impossible to find a B that would reliably beat Alex every time but lose to us every time. We eventually went with Komodo 12 (supposedly rated 1600, but the Chess.com bots' ratings are inflated compared to Chess.com players and even more inflated compared to over-the board, so I would estimate its USCF rating would be in the 1200-1300 range.)

Since this was the first trial run, the time control was only 3 hours in total, and all in one sitting. Komodo makes its moves within a few seconds, so it's about the same as a 3 hour per side time control from Alex's perspective. We ended up using about 2.5 hours of that. The discussion took place between all four of us in a Discord server, with Alex sending us screenshots after each move.

The game

The game is available at https://www.chess.com/analysis/game/pgn/4MUQcJhY3x. Note that this section is a summary of the 2.5-hr game and discussion, and it doesn't cover every single thing that we discussed.

Alex flipped to see who went first, and was White. He started with 1. e4, and Black replied 1... e5. Aphyer and Adam had more experience with the opening we would enter into than myself, and since they weren't willing to blow their covers immediately, they started by suggesting good moves, which Alex went along with.

After 2. Nf3 Nc6 3. Bc4, Black played 3... Nf6, which Aphyer and Adam said was a bit of a mistake because it allowed 4. Ng5. Alex went ahead, and we entered the main line from there - 4... d5 5. exd5 Na5.

Aphyer and Adam said the main l...
8 examples informing my pessimism on uploading without reverse engineering
by Steven Byrnes — 2023-11-03T20:03:50.450Z

(If you’ve already read everything I’ve written, you’ll find this post pretty redundant. See especially my old posts

Building brain-inspired AGI is infinitely easier than understanding the brain

, and

Randal Koene on brain understanding before whole brain emulation

, and

Connectomics seems great from an AI x-risk perspective

. But I’m writing it anyway mainly in response to

this post from yesterday

.)

1. Background / Context

1.1 What does uploading (a.k.a. Whole Brain Emulation (WBE)) look like with and without reverse-engineering?

There’s a view that I seem to associate with

Davidad

and

Robin Hanson

, along with a couple other people I’ve talked to privately. (But I could be misunderstanding them and don’t want to put words in their mouths.) The view says: if we want to do WBE, we do not need to reverse-engineer the brain.

For an example of what “reverse-engineering the brain” looks like, I can speak from abundant experience: I often spend all day puzzling over random questions like:

Why are there oxytocin receptors in certain mouse auditory cortex neurons?

Like, presumably Evolution put those receptors there for a reason—I don’t think that’s the kind of thing that appears randomly, or as an incidental side-effect of something else. (Although that’s always a hypothesis worth considering!) Well, what is that reason? I.e., what are those receptors doing to help the mouse survive, thrive, etc., and how are they doing it?

…And once I have a working hypothesis about that question, I can move on to hundreds or even thousands more “why and how” questions of that sort. I seem to find the activity of answering these questions much more straightforward and tractable (and fun!) than do most other people—you can decide for yourself whether I’m unusually good at it, or deluded.

For an example of what uploading without reverse-engineering would look like, I think it’s the idea that we can figure out the input-output relation of each neuron, and we can measure how neurons are connected to each other, and then at the end of the day we can simulate a human brain doing whatever human brains do.

Here’s Robin Hanson arguing for the non-reverse-engineering perspective in Age of Em:

The brain does not just happen to transform input signals into state changes and output signals; this transformation is the primary function of the brain, both to

...
Integrity in AI Governance and Advocacy
by habryka — 2023-11-03T19:52:33.180Z

habryka

Ok, so we both had some feelings about the recent Conjecture post on "lots of people in AI Alignment are lying", and the associated marketing campaign and stuff.

I would appreciate some context in which I can think through that, and also to share info we have in the space that might help us figure out what's going on.

I expect this will pretty quickly cause us to end up on some broader questions about how to do advocacy, how much the current social network around AI Alignment should coordinate as a group, how to balance advocacy with research, etc.

OliviaJ

Feelings about Conjecture post:

Lots of good points about people not stating their full beliefs messing with the epistemic environment and making it costlier for others to be honest.
The lying and cowardice frames feel off to me.
I personally used to have a very similar rant to Conjecture. Since moving to DC, I'm more sympathetic to those working in with government. We could try to tease out why.
The post exemplifies a longterm gripe I have with Conjecture's approach to discourse & advocacy, which I've found pretty lacking in cooperativeness and openness (Note: I worked there for ~half a year.)

Questions on my mind:

How open should people motivated by existential risk be? (My shoulder model of several people says "take a portfolio approach!" - OK, then what allocation?)
How advocacy-y should people be? I want researchers to not have to tweet their beliefs 24/7 so they can actually get work done
How do you think about this, Oli?

How sympathetic to be about governance people being private about some motivations and affiliations

habryka

I personally used to have a very similar rant to Conjecture. I'm now more sympathetic to governance people. We could try to tease out why.

This direction seems most interesting to me!

My current feelings in the space are that I am quite sympathetic to some comms-concerns that people in government have and quite unsympathetic to some other stuff, and I would also like to clarify for myself where the lines here are.

Curious whether you have any key set of observations or experiences you had that made you more sympathetic.

OliviaJ

Observations

I've heard secondhand of at least one instance where a person brought up x risk, then their Congressional office took them less seriously. Other staffers have told me talking about x risk wouldn't play

...
Averaging samples from a population with log-normal distribution
by CrimsonChin — 2023-11-03T19:42:16.630Z

This was originally a comment on this post by mruwnik regarding averaging various distributions with different distributions. I made it a post to include pictures.

The Central Limit Theorem, henceforth CLT, states (in my own words) that regardless of the distribution of a population, sample averages from that population should be normally distributed.

In theory it should hold for log-normal distributions but that doesn't feel intuitive to me so I tested it.

A silly example of CLT

An example I made up in my head to make sense of it:

Imagine a population comprised of all the people who nap 2 times in a day. Lets plot the ages of this population:

Mostly infants and elderly people nap, hence the shape of the graph. This data is NOT normal. But if you randomly pick a small sample (n=10) from this population and average it, it will have a mix of old people and infants that averages to middle-age. For example imagine the ages are 80,2,1,2,75,76,1,1,85,70 this will average to about 39. If you do this over and over again with randomly chosen samples you will get a normal distribution.

Does it work with log-normal populations?

I didn't find it intuitive this would work for a log-normal population.

If I take data that is log-normal but split it into small samples, will the average of those small samples be normally distributed?

Chess matches:

I am arranging a chess tournament. I need to figure out how long the average match is so I can plan accordingly. I hear that chess matches seem to follow a log-normal distribution, but I'm not sure what that means statistically so I will try to just average the game times.

Data from the population

This is what my fake population (n=100,000) looks like. Its log-normal.

Tournaments

I observe tournaments (n=100 games) and take a simple average of the match length.

Here is a histogram plot of the tournaments

Lessons?

The sample size does matter here. A sample size too small (n=10) and you just end up with the original log-normal distribution. This is expected as the sample size moves from small to large you get a range of smoothing effects pushing the distribution to normal until you get a single point, when the sample = population.
Securing Civilization Against Catastrophic Pandemics
by jefftk — 2023-11-03T19:33:05.242Z
Link: https://dam.gcsp.ch/files/doc/securing-civilisation-against-catastrophic-pandemics-gp-31

Executive summary

Pandemic security aims to safeguard the future of civilization from exponentially spreading biological threats. Despite the world's failure to contain SARS-CoV-2, the existence of far more lethal and transmissible pathogens that afflict animals and growing access to increasingly powerful biotechnologies, no analyses of worst-case scenarios and potential defenses have been published. Here we outline two distinct mechanisms by which pandemic pathogens transmissible between humans could cause societal collapse. In a "Wildfire" pandemic, the justifiable fear of a lethal and highly contagious respiratory agent released in multiple travel hubs leads to the breakdown of essential services. In a "Stealth" pandemic, a rapidly spreading virus with a long incubation period analogous to HIV infects most of humankind. We explain why current pandemic preparedness measures such as rapid vaccines and N95 masks will reliably fail against these threats and outline novel strategies and technologies capable of safeguarding civilisation.

Key takeaways

Nations cannot yet contain natural, accidental or deliberate pandemics.
Access to severe pandemics will expand with the ability to program biology.
If too many essential workers die or refuse to work, societies will collapse.
A Wildfire pandemic is highly lethal and transmissible enough to infect most essential workers who are taking currently available precautions.

Collapse can be prevented by providing essential workers with pandemic-proof personal protective equipment (P4E). (Essential workers are those who must deliver food, water, power and law enforcement without any interruptions.)
Others can remain safely at home until P4E is available for everyone.
Once the population is protected, the virus can be locally eradicated.

A Stealth pandemic spreads widely with few symptoms and causes severe harm years later.

Societal collapse can be prevented via early warning, credibility, cures, P4E and healthy buildings.

Early warning: deep metagenomic sequencing offers reliable detection.
Credibility: expert responders can assess threats and encourage action.
Cures: swift medical research can offer hope for the infected.
P4E: people will need protective equipment that they can trust to block transmission.
Healthy buildings: the use of germicidal lights and ventilation

...