trentmkelly/LessWrong-43k huggingface dataset:
Announcing the Distillation for Alignment Practicum (DAP)
by Jonas Hallgren — 2022-08-18T19:50:31.371Z

Hey, fellow LessWrongers!

Around the time of John Wentworth's post

A Call for Distillers,

we started to work on a distillation course which now has turned into a practicum. (A practicum is more close to a study circle than a course and a lot more open-ended)

Five months later and timely to the launch of the

CAIS competition

, we're launching the trial run of DAP! (Distillation for Alignment Practicum), starting on the week of September 16th. The course is six weeks long and structured similarly to AGISF, with core readings and exercises during and in-between sessions.

The difference is that we're going to have slightly smaller cohorts containing 3-4 fellows (along with one facilitator), and we're also focusing on letting you develop your distillations. Through this process, we hope you will explore your views of what distillation is helpful for and what techniques might be practical when creating a distillation.

Practicum structure

Below, you can see our proposed loop for how we think of improving your distilling skills. It is essentially the scientific method. The first two weeks of the course are about working on a hypothesis on what makes a helpful distillation and what techniques you could use to implement the values you identify in the first week in practice. You then pick a distillation and apply the methodology you came up with in practice. Lastly, in week six, you review how it went and whether your models could have been improved for the specific distillation you were doing. (For more detail, check out the end of the post)

The hope is that this gives a new meta methodology for improving your writing. As this is a practicum, we aim to provide an open space for discussing what makes a distillation great, as we're not sure of it ourselves.

We hope this will provide a space for people to continuously grow and become great distillers!

Give us feedback, please

It is the first iteration of the distillation practicum, so if you have any ways you can think to improve it, we're very open to feedback! We're specifically looking for you to do one or more of the following:

Roast the living crap out of the current exercises (Trial by fire)
Come up with new exercises that would be better than the existing ones
Suggest things which might be missing in the general research loop
Come up with improvements to exercises

You can send this to us privately on ...
A conversation about progress and safety
by jasoncrawford — 2022-08-18T18:36:10.353Z
Link: https://rootsofprogress.org/a-conversation-about-progress-and-safety

A while ago I did a long interview with Fin Moorhouse and Luca Righetti on their podcast Hear This Idea. Multiple people commented to me that they found our discussion of safety particularly interesting. So, I’ve excerpted that part of the transcript and cleaned it up for better readability. See the full interview and transcript here.

LUCA: I think there’s one thing here of breaking progress, which is this incredibly broad term, down into: well, literally what does this mean? And thinking harder about the social consequences of certain technologies. There’s one way to draw a false dichotomy here: some technologies are good for human progress, and some are bad; we should do the good ones, and hold off on the bad ones. And that probably doesn’t work, because a lot of technologies have dual use. You mentioned World War Two before…. On the one hand, nuclear technologies are clearly incredibly destructive, and awful, and could have really bad consequences—and on the other hand, they’re phenomenal, and really good, and can provide a lot of energy. And we might think the same around bio and AI. But we should think about this stuff harder before we just go for it, or have more processes in place to have these conversations and discussions; processes to navigate this stuff.

JASON: Yeah, definitely. Look, I think we should be smart about how we pursue progress, and we should be wise about it as well.

Let’s take bio, because that’s one of the clearest examples and one that actually has a history. Over the decades, as we’ve gotten better and better at genetic engineering, there’s actually been a number of points where people have proposed, and actually have gone ahead and done, a pause on research, and tried to work out better safety procedures.

Maybe one of the most famous is the Asilomar Conference in the 1970s. Right after recombinant DNA was invented, some people realized that “Whoa, we could end up creating some dangerous pathogens here.” There’s a particular simian virus that causes cancer that caused people to start thinking: “what if this gets modified and can infect humans?” And just more broadly, there was a clear risk. And they actually put a moratorium on certain types of experiments, they got together about eight months later, had a conference, and worked out certain safety procedures. I haven’t researched this deeply, but my understanding is that went pretty w...
Discovering Agents
by zac_kenton — 2022-08-18T17:33:43.317Z

Work done with Ramana Kumar, Sebastian Farquhar (Oxford), Jonathan Richens, Matt MacDermott (Imperial) and Tom Everitt.

Our DeepMind Alignment team researches ways to avoid AGI systems that knowingly act against the wishes of their designers. We’re particularly concerned about agents which may be pursuing a goal that is not what their designers want.

These types of safety concerns motivate developing a formal theory of agents to facilitate our understanding of their properties and avoid designs that pose a safety risk.

Causal influence diagrams

(CIDs) aim to be a unified theory of how design decisions create

incentives

that shape agent behaviour to illuminate

potential

risks

before an agent is trained and inspire better agent

designs

with more appealing alignment properties.

Our new paper, Discovering Agents, introduces new ways of tackling these issues, including:

The first formal causal definition of agents, roughly: Agents are systems that would adapt their policy if their actions influenced the world in a different way
An algorithm for discovering agents from empirical data
A translation between causal models and CIDs
Resolving earlier confusions from incorrect causal modelling of agents

Combined, these results provide an extra layer of assurance that a modelling mistake hasn’t been made, which means that CIDs can be used to analyse an agent’s incentives and safety properties with greater confidence.

Example: modelling a mouse as an agent

To help illustrate our method, consider the following example consisting of a world containing three squares, with a mouse starting in the middle square choosing to go left or right, getting to its next position and then potentially getting some cheese. The floor is icy, so the mouse might slip. Sometimes the cheese is on the right, but sometimes on the left.

The mouse and cheese environment.

This can be represented by the following CID:

CID for the mouse. D represents the decision of left/right. X is the mouse’s new position after taking the action left/right (it might slip, ending up on the other side by accident). U represents whether the mouse gets cheese or not.

The intuition that the mouse would choose a different behaviour for different environment settings (iciness, cheese distribution) can be captured by a mechanised causal graph (a variant of mecha...
Oops It's Time To Overthrow the Organizer Day!
by Screwtape — 2022-08-18T16:40:53.864Z

Summary: Sometimes, your regular organizer catches the pandemic, or moves to the bay, or has a newborn, or does a bunch of sketchy stuff, or just doesn’t have time anymore. If the usual organizers suddenly weren’t around, would your community be able to keep going?

Tags: Investment

Purpose: This is a test run at organizing your local community if for some reason the usual organizers aren’t available, and it should both test that the community can continue and also level up some attendees into organizers.

Materials: This test is open book, open notes, and open internet. Bring whatever you want.

Announcement Text: “Lets find out if this community would keep running if the usual organizers stepped away from it.

There’s a concept in business called “Bus Factor.” It describes how many people are crucial to your organization, such that if you lose them (say, if they were hit by a bus tomorrow) your business would be in trouble. Well, we’re going to have the people who organize things sit in a room available to answer questions as someone new sets up a meetup. If it works, then there’s at least one more person who can run things. If it doesn’t work, then we will have learned something.

If you’ve ever wondered how the sausage gets made, or if you’ve had an idea for a meetup but haven’t run one before, come on in!"

Description: Oops It’s Time To Overthrow the Organizer Day has, at its core, only two rules.

1. You must schedule and run a meetup.

2. Nobody who has already run a meetup is allowed to do it.

There is one extra rule used purely for score keeping.

3. Your score is the number of attendees attending the next meetup minus the number of times the usual organizers answered questions.

All other rules and instructions are secondary. More than anything, you must be thinking of carrying your movement through to running a good meetup.

With that in mind, here is the suggested format for Oops It’s Time To Overthrow the Organizer Day!

However many usual organizers are available sit together in the centre of the room. They should make themselves comfortable, and bring something to entertain themselves that can easily be paused or put down. Everyone else sits around them with whatever notes, documents, or communication devices they want.

We’ll call the usual organizers “The Establishment” and everyone else “The Usurpers.” (You don’t have to call them ...
Bias towards simple functions; application to alignment? 
by DavidHolmes — 2022-08-18T16:15:25.384Z

Summary

Deep neural networks (DNNs) are generally used with large numbers of parameters relative to the number of given data-points, so that the solutions they output are far from uniquely determined. How do DNNs 'choose' what solution to output? Some fairly recent papers ([1], [2]) seem to suggest that DNNs are biased towards outputting solutions which are 'simple', in the sense of Kolmogorov complexity (minimum message length complexity) ([3]). It seems to me that this might be relevant for questions around AI alignment. I've not thought this idea through in detail; rather, I'm writing this post to ask whether people are aware of work in this direction (or see obvious reasons why it is a terrible idea) before I devote substantial time to it.

Example

Setting: some years in the future. I instruct Siri: "tell me a story". Here are two things Siri could do:

copy-paste an existing story from somewhere on the internet.
kidnap a great author and force them to write a story (by now Siri is smart enough to do this);

I would much prefer Siri to choose option 1. There are lots of ways I might go about improving my request, or improving Siri, to make option 1 more likely. But I claim that, assuming Siri is still based on the same basic architecture as we are currently familiar with (and probably even if not), it is more likely to choose option 1 because it is simpler. Slightly more formally, because explicit instructions on how to carry out option 1 can be written out in many fewer characters than those required for option 2.

Now making this claim precise is hard, and if one looks too closely it tends to break down. For example, the instruction string 'tell me a story' could lead to option 1 or option 2 without changing its length. More generally, measures of complexity tend to be 'up to a constant' with the constant depending on the programming language, and if the language is 'natural language that Siri parses' then I am exactly back where I started. But I suspect this is not an unsolvable problem; one could for example think about the length of message that Siri has to write to instruct its sub-agents/kidnapping drones.

Slightly more formal statement

I'll try to translate from the general setup of ([2:1]) to the example above with Siri, to make things more concrete, but I remain close to their notation. All errors lie in the translation, etc. etc. We have the set ...
What Games These Days?
by jefftk — 2022-08-18T14:30:05.355Z

While I used to play a lot of board games, this is something I've had
less time for since having kids. Last week my extended family got
together for a vacation, however, which meant a lot more time for
gaming! Thinking back, here's what I played over the week, in roughly
descending order of time spent:

Brass
Birmingham x1. My second time playing it. Solid game, but way
too long: 3.5hr for three players. Didn't help that we started at
9:15pm. I'm still hoping this will get faster with more play.

Go
x3. I've been playing with my dad since I was little. My play
suffers from overfitting against my dad in particular.

Wingspan
x2. My first time playing. I like it a lot, and would play more.
Reasonable length, good movement, could have slightly more player
interaction.

Boggle x15.
Mostly playing with adults, though a few rounds with Lily (8y). Rules
with Lily were that she had no minimum length, if we both wrote the
same word only she got points, and I couldn't use any words she didn't
know.

Power
Grid x1. We've been playing this for a long time, but it's still
different enough each time to be fun. I like the expansion
maps, but we didn't have any with us and just played Germany.

Race for
the Galaxy. We drafted (first six-cost developments, then
everything else, with a "draw two, ditch one") and then played three
games with our decks. Stevie went for mining and novelties with
produce-consume, I went for genes and developments. We used the first
two expansions, without takeovers. Probably the game here I've played
the most of over the years.

Bridge x3.
Very popular game among my aunts and uncles, but almost all of us play
it some. We play rubber bridge
and there's some disagreement on whether should play as if you
were instead playing duplicate;
I'm on team "try to win by the rules you're using."

Spoons x7.
Very silly game. Lily (8y) could play with us, but Olivia (6y) found
it too frustrating. Next time I'd have younger kids use a hand of
just two cards, and be able to take an initial spoon with just a
matched pair.

Modern
Art x1. Highly auctiony game with minimal theming.

Ticket to Ride: Europe x1. Solid game, not too heavy.

Jaguar x3. A variation of Briscola
Chiamata. Very lively trick taking game with conspiratorial table
talk.

Writey Drawey
Game (x2). Generally very funny, from a comb

...
Covid 8/18/22: CDC Admits Mistakes
by Zvi — 2022-08-18T14:30:01.286Z

Two Covid-related things happened this week that I did not expect.

The CDC admitted that it had failed us during the pandemic, withdrew at long last many of its remaining recommendations and promised reforms to be less academic and otherwise do better.

A new paper found potential biological markers for Long Covid, claiming it can be identified via tests that match patient self-reports almost all the time. This points towards potential progress in treatment, and more generally in Long Covid being much more concretely A Thing that might exist and could be reasoned about. There are still flaws, and even without flaws there is still much work to do here.

Most of the reasons not to be concerned remain, so up front: I do not think that this should substantially change anyone’s level of Covid precautions.

Executive Summary

CDC finally fully gives up on six foot social distancing and other measures.
CDC admits some failure, promises reforms that seem potentially promising.
New study finds potential biological markers for Long Covid.

Also I think someone said something about over the counter hearing aids?

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 650k cases (-5%) and 3,200 deaths (+0%).

Results: 602k cases (-12%) and 3,183 deaths (-1%).

Prediction for next week: 560k cases (-8%) and 3,200 deaths (+1%).

I do not know why cases declined more than expected but result seems robust so I see no reason to expect it not to continue at least somewhat. There won’t be enough time for the decline to impact deaths yet, so I’m mostly going with the null prediction there. There is not much meaningful uncertainty here.

Deaths

Cases

Decline cuts across all four regions. We are fully into the BA.5 era with nothing on the near-term horizon to replace it, so things should be quiet for a few months at least.

Physical World Modeling

A guide to buying the right HEPA air filter.

Bob Wachter sees no sign of anything that might replace BA.5. Thread also reminds us that it is a common mistake not to take into account the correlation between the Covid status of people who choose to be together in a group. Trevor Bedford however sees logistic growth in BA.2.75, although only with R0 ~ 1.3 (versus ~1 for BA.5) which isn’t that much of an advantage for a new strain taking over. My guess is Trevor is right and BA.2.75 wi...
In Defense Of Making Money
by George3d6 — 2022-08-18T14:10:23.950Z
Link: https://www.epistem.ink/p/in-defense-of-making-money

2 years ago habryka4 mentioned he'd love to see a comment I made developed into an article, and I agreed and said I was planning on doing it next Monday... anyway, 2 years later on <some> Monday I finally found the time.

Money is abundant and meaningless.

There are good founders with few relationships able to raise upwards of half a million as the first investment in ideas that seem impossibly difficult to monetize. With decent connections, you can pocket upwards of 10 million to start a company that seems hot.

You can learn programming in 3-12 months and earn 6 figures with low taxes working remotely in a few years. You can make bank by doing the most niche and easy to learn of jobs as long as you know how to follow the market. You can probably make it even if you just follow your nose and invest in trends like crypto or high-throughput sequencing early on.

Earning much is easy for many people and it provides no direct meaning. The rich man that’s unable to find happiness because he doesn’t realize the goal of life is altruism/nirvana/love/enjoying-the-moment/god is beyond cliche.

Money is, to some extent, equivalent to power over others, but that power doesn’t scale if we look at the realm of all possible options:

If I don’t know how to efficiently turn power into a GDP increase, or money into a cure for cancer, then throwing more power/money at the problem will not make much difference.

King Louis XV of France was one of the richest and most powerful people in the world. He died of smallpox in 1774, the same year that a dairy farmer successfully immunized his wife and children with cowpox. All that money and power could not buy the knowledge of a dairy farmer - the knowledge that cowpox could safely immunize against smallpox. There were thousands of humoral experts, faith healers, eastern spiritualists, and so forth who would claim to offer some protection against smallpox, and King Louis XV could not distinguish the real solution.

This in turn tempts many people, especially young people, to swear off the making of money as a valuable goal.

In itself, this can be fine if you smell your goals well enough from an early age. If you are certain that what you want is to hobo-backpack across the globe thrice, or have a quiet cliffside Mediterranean restaurant, or seek enlightenment ... then yeah, you probably shouldn’t aim to make money.

But there’s also a typology...
Playing Without Affordances
by Alex Hollow — 2022-08-18T11:53:26.601Z
Link: https://alexhollow.wordpress.com/2021/10/31/playing-without-affordances/

Play structures are so easy that even children can figure out how to use them correctly. There is a wheel. Spin it! There is a slide. Go down it! There is a staircase. Climb it! There is a tunnel. Crawl through it!

And a tic-tactile-toe board.

This can be good – sliding down slides is quite fun, and crawling through tunnels could entertain a six year old, and I guess spinning a wheel is nice.

The issue with play structures is that they train you to play only within affordances. When the space that you play in is designed by someone else, you are pushed towards playing only in the ways that they envisioned. A play structure affords a small set of actions, and as a player, you only get to pick which one you prefer. There’s no invention in that, no spontaneity, no creativity.

Let’s think about how this applies to adults. If you show an adult a library of video games, they can pick one and click the “play” button. But show an adult a meadow with trees to climb and grass to roll around in, and they will write it off as unplayable (unless they are with friends, and typically playing a standard game with standard rules).

Writing code, painting and most other hobbies are forms of play that don’t come with an affordance. Doing those things requires that you drive yourself. You decide to do them out of an infinite set of possibilities, filtered by your own desires! And that is a good feeling, much better than letting someone else decide (either through the design of a play structure or the design of your society) what you are going to do today.

People who don’t have hobbies usually don’t know how to play without affordances.
Goal-directedness: relativising complexity
by Morgan_Rogers — 2022-08-18T09:48:40.864Z

This is the fourth post in my Effective-Altruism-funded project aiming to deconfuse goal-directedness. Comments are welcomed. All opinions expressed are my own, and do not reflect the attitudes of any member of the body sponsoring me. The funding has come to an end, but I expect to finish off this project as a hobby in the coming months.

My previous post was all about complexity, and ended with an examination of the complexity of functions. In principle, I should now be equipped to explicitly formalise the criteria I came up with for evaluating goal-based explanations. However, several of the structures whose complexity I need to measure take the form of transformations from one structure of a given type to another. This brings us into the domain of relative complexity. I have chosen to present this in a separate post, not just because of the daunting length of my last post, but also because I think this is a subject which this community would benefit from some explicit clarification of. In future posts I shall finally put all of the pieces together and get to the bottom of this goal-directedness business.

Recall that last time that I described complexity as a quantity assigned to a structure that measures how many simple pieces are needed to construct it. The general concept which is under scrutiny this time is a variant of the above, where we consider instead a quantity assigned to a structure that measures how many simple pieces are needed to construct it from a given resource or starting point. A source of confusion lies in the fact that there are multiple ways to relativise complexity, which is why a closer examination is needed.

In this post I'll be discussing a few implementations on relativised notions of complexity.

Relativising Kolmogorov complexity

Conditional complexity

In this subsection I'll be using standard notation and definitions from Ming Li and Paul Vitányi's textbook,[1] which I will subsequently extend somewhat. They use the convention that the complexity measure is indexed by a partial computable function rather than a given Turing machine which computes it, which makes sense, since the details of the Turing machine are irrelevant in the computation of complexity beyond its input-output behaviour.

The definition of K-complexity for individual strings s which we saw last time involved a choice of universal Turing machi...