trentmkelly/LessWrong-43k huggingface dataset:
Alex Lawsen On Forecasting AI Progress
by Michaël Trazzi — 2022-09-06T09:32:54.071Z
Link: https://theinsideview.ai/alex

Alex Lawsen is an advisor at 80,000hours who has released an Introduction to Forecasting. We discuss pitfalls that happen when forecasting AI progress, why you cannot just update all the way (discussed in my latest episode with Connor Leahy) and how to develop your own inside views about AI Alignment.

Below are some highlighted quotes from our conversation (available on Youtube, Spotify, Google Podcast, Apple Podcast). For the full context for each of these quotes, you can find the accompanying transcript.

On the Metaculus AGI Forecasts

Why You Cannot Just Update All The Way

"There are some situations where all of the positive evidence you get is going to be in the same direction, and then the negative evidence you get is nothing happens. And so, ideally, what you do in this case is every day that nothing happens, you make a tiny update in one direction. And then every few weeks or every few months, something big happens and you make an update in the other direction. And if that is the case, maybe what you'll see is people just... they forget to do the small downwards updates and then they do the big updates every time something happens. And I think if you do the Connor thing of seeing... Well, I'm not too sure this is the Connor thing. But if you see four updates and they're all in the same direction and then you go like, 'Oh, man, everything's going the same direction. I need to be really confident stuff going that direction.' Then every day something doesn't happen, your downwards update needs to be pretty big. If you're expecting massive progress, then a week going by and nothing happening is actually big evidence for you."

The Metaculus Drops Were Not Caused By Newcomers

"One hypothesis you might have which I think a friend of mine falsified, is 'a whole bunch of people saw these results. These results were all over Twitter, it was impressive. Chinchilla was impressive, PaLM was impressive'. So, you might think, 'Oh, well, a bunch of new people who haven't made timelines forecasts before are going to jump on this Metaculus question and they're going to make predictions.' And so, you can test this, right. You can look at how the median changed among predictors who had already predicted on the question and that median dropped too."

On Using Growth Models To Forecast AGI

Business As Usual Does Not Require Burden Of Proof

"I think there was a class of skepticism abo

...
What are you for?
by lsusr — 2022-09-06T03:32:23.536Z

My favorite movie reviewer is a YouTuber named moviebob. Moviebob frequently talks about the idiotic things other reviewers are saying. I would not know these other reviewers even existed were it not for moviebob. Moviebob is a memetic amplifier for the reviewers he disagrees with.

The solution is simple: Ignore people who are wrong.

It can be worthwhile paying attention to people who seem slightly wrong because they are wrong in interesting ways. Sometimes they are even right. But you should neither take seriously nor argue with those people who value winning an argument more than the truth. "Us" vs "Them" arguments are an endless source of such people.

"Us" vs "Them" definitions make it easy to draw battlelines. Which is a problem. The first casualty of war is the truth. Ignoring people who are wrong require operating counter to our tribal impulses.

Another problem with "Us" vs "Them" is the definitions are circular. How are "They" defined? "Not Us". Be wary of any idea which defines itself by what it's not. Such an idea is usually half of a symbiotic conflict. Attach yourself to ideas that define themselves in absolute terms instead. Otherwise you end up mired in circular logic.

Circular ideas are a waste of attention. Including—sometimes—Atheism. Atheism is at its best when it is pro science. Atheism is at its worst when it is anti religion.

The easiest—and worst—way to demarcate "Us" is to define "Us" as "not Them". Speak in the positive. What are you for?
The Power (and limits?) of Chunking
by Nicholas / Heather Kross — 2022-09-06T02:26:27.808Z

Working memory is important. How can we increase it?

Writing is like an external brain... if you go back and read it. Dual-N-Back works... maybe. Working memory is important enough for me to incessantly ask about improving it.

What about chunking? The one where you remember ideas better by grouping them into a new bigger idea.

Chunking basically works as an abstraction layer applied to an idea itself. Instead of "lots of (insert sigmoid function)s connected using these thingies", you go "a few layers of sigmoid-activated neurons", or even just "this neural network".

So here are my questions for discussion:

What are the human capacities for chunking ideas?

Can we abstract thoughts endlessly endlessly-for-current-practical-purposes, to get around our working memory limits?

Does that work for the kinds of super-valuable maths research
that could be needed to solve AI alignment? Why or why not? How much of this is "giant galaxy-brained unwriteable uncodifiable concepts that you must hold 100% of in your head all the time and cannot be taught to anyone else" VS "an average-IQ person, given 20 years to just write stuff down and reference it, could get it".

How often people need to be able to switch between abstraction levels, ways this cognitive work can (or reasons it can't) be chunked/written/automated/offloaded. (E.g., in the

(This post doesn't have the answers; I just want to get the ball rolling on people thinking thoroughly about this.)
Another Unphrased B-part
by jefftk — 2022-09-06T01:30:01.477Z

Usually everything in a contra dance matches up exactly with the
music. The tunes repeat every 64 beats, and are divided into two
32-beat halves (A and B), each of which is broken up into two 16-beat
halves (A1 and A2, B1 and B2), each of which is broken into two 8-beat
halves (not usually named). Each dance is choreographed so the
transitions between moves match up with transitions between sections
of music. For example:

The Baby Rose
by David Kaynor

A1
(16)
Neighbor balance and swing

A2
(8)
Circle left 3/4

(8)
Partner dosido

B1
(16)
Partner balance and swing

B2
(8)
Robins chain

(8)
Star left

This is saying that the first 16 beats of the dance (A1) are "neighbor
balance and swing". The next 16 beats (A2) are divided into 8 beats
for a "circle left 3/4", and then another 8 for a "partner dosido".
The dancers will adjust how quickly they're moving through the figures
to make the moves line up exactly, and this tight timing makes the
dance feel very satisfying. Music that clearly communicates these
divisions, either by the tune you've chosen or how you're playing it,
is "well phrased": the phrases the dancers hear are the ones needed to
support the dance (more).

There are a few places where there's extra time and it isn't a problem
to start the next figure a bit early. For example, consider a common
combination:

(8)
Circle left 3/4

(8)
Swing

It usually doesn't take the full eight beats to circle 3/4, and the
swing is a move that can take a range of lengths, so it's more common
to see people dance this as:

(6)
Circle left 3/4

(10)
Swing

This minor departure from power-of-two divisions is about as far as
the typical dance will get from this pattern. A few years ago,
however, I wrote about
a few dances that have entirely unphrased B-parts, and last night I noticed we were
dancing to another. Writing the dance as if it had used conventional
timing it would be:

Never Too Late

by Tom Hinds

A1
(8)
Larks dosido

(8)
Larks allemande right 1.5

A2
(16)
Partner balance and swing

B1
(8)
Right and left through

(8)
Robins chain

B2
(8)
Robins right shoulder round

(8)
Neighbor swing

The B part doesn't really dance like that, though. I counted how it
was going for us, and it was pretty close to:

B
(6)
Right and left through

(6)
R

...
[Exploratory] Becoming more Agentic
by Johannes C. Mayer — 2022-09-06T00:45:43.835Z

Disclaimer: This is an exploratory writing post. No checking for typos or other editing was done.

This seems like the core skill to success in almost anything. Especially as things get more complex and less straightforward. E.g. you do need to be less agentic when wanting to learn a language, compared to when doing original research. This is at least in part because it is easier to evaluate if the direction that you are moving in is good

This also seems very important when working in a team. When you are working in a field where determining a good direction is relatively easy, because it is known what is good, then few people can direct many people to do something. E.g. when constructing a building, having a chain of command works well, because it is known what people at each level should do. When making a video game, there can also be some people who direct other people. E.g. when it comes to what art assets to produce, and how to go about producing them. In research, often the hardest part is to find the right direction to move in. If you have a team of people working on a research project, trying to manage what all the people should be doing becomes quickly overwhelming, as long as you still don't have a solid direction to move in, in my experience.

I think that probably a better strategy is, that everybody optimizes for themselves, to do what is best for them to do. I am thinking here in terms of the context of the project, but this is also true at higher levels (e.g. should I be working on this specific project, in this specific team).

I am not satisfied with my level of agenticness, and also with the people, I work with. I have not found a good strategy for making them more agentic yet. The first step is that they agree that they should become more agentic. But after that, I do not know what to do.

This all is in the context of research. When working in a team with other people, making games in the past, this was much less of an issue. Though that might have been because success was not super team-dependent, and I could often carry the team to produce something good by working hard. Even in that situation, more agenticness would probably be good.).
AI Governance Needs Technical Work
by Mau — 2022-09-05T22:28:06.423Z

Summary and Introduction

People who want to improve the trajectory of AI sometimes think their options for object-level work are (i) technical work on AI alignment or (ii) non-technical work on AI governance. But there is a whole other category of options: technical work in AI governance. This is technical work that mainly boosts AI governance interventions, such as norms, regulations, laws, and international agreements that promote positive outcomes from AI. This piece provides a brief overview of some ways to do this work—what they are, why they might be valuable, and what you can do if you’re interested. I discuss:

Engineering technical levers to make AI coordination/regulation enforceable (through hardware engineering, software/ML engineering, or heat/electromagnetism-related engineering)

Information security: Developing and implementing systems and best practices for securing model weights and other AI technology

Forecasting AI development

Technical standards development

Grantmaking or management to get others to do the above well

Advising on the above

Other work

[Update] Additional categories which the original version of this piece (from 2022) under-emphasized or missed are:

AI control: Developing systems and best practices for overseeing and constraining AI systems that may not be trustworthy (example)

Model evaluations: Developing technical evaluations of the safety of AI systems (discussion, examples)

Forecasting hardware trends (examples)

Cooperative AI: Research in game theory, ML, and decision theory for designing AI systems in ways that avoid costly coordination failures (discussion, examples)

I expect there will likely be one or more resources providing more comprehensive introductions to many of these topics in early 2024. For now, see the above links to learn more about the topics added in the update, and see below for more discussion of the originally listed topics.

Acknowledgements

Thanks to Lennart Heim, Jamie Bernardi, Gabriel Mukobi, Girish Sastry, and others for their feedback on this post. Mistakes are my own.

Context

What I mean by “technical work in AI governance”

I’m talking about work that:

Is technical (e.g. hardware/ML engineering) or draws heavily on technical expertise; and

Contributes to AI’s trajectory mainly by improving the chances that AI governance interventions succeed[1] (as opposed to by making progress on techni

...
Overton Gymnastics: An Exercise in Discomfort
by Shoshannah Tekofsky — 2022-09-05T19:20:01.642Z

What better way to bond with strangers than to tell them your most controversial viewpoints right off the bat? It’s basically like Cards Against Humanity for rationalists. More specifically, for the Less Wrong Community Weekend in Berlin, all participants were asked to come up with one session to run for their fellow participants. Overton Gymnastics was born from an interest in exploring how we can stretch our

Overton windows

. Then I got the stomach flu and couldn’t attend the meetup…

But people were psyched about the idea! So here are some iterative takes on how to run a session of Overton Gymnastics. If you end up trying it, I’d be excited to hear how it was for you in the comments. Also, as this is a new idea, please do just take it where you’d like. Some variants will be more useful than others, and we can all learn from your experiences!

The Initial Idea

The following was the initial write up of Overton Gymnastics for the LWCW in Berlin.

Stretch our Overton windows! One person starts by saying something controversial they believe. Someone then volunteers to counter them. First you provide an Ideological Turing Test (ITT) of the controversial claim, then your counter, and then you provide your own controversial claim. The goal is to practice tolerating uncomfortable viewpoints and ITT'ing them accurately.

We'll first review some failure modes that became apparent when trying to run the exercise as described, followed by three possible variants you could run.

Possible Failure Modes

The two greatest challenges during Overton Gymnastics are generating controversial viewpoints and avoiding discussions during the exercise. To help people come up with controversial viewpoints it is useful to prime them with examples of claims that are controversial to different social groups. The goal of the exercise is to come up with claims that are controversial to the current group, but if people struggle with this, then they can be prompted to volunteer more generally controversial claims compared to mainstream culture.

To avoid discussion during the exercise, it's best to prepare participants for how tempted they might be to launch in to debate. Additionally, the facilitator should proactively take on a moderation role across groups, listening in to detect discussions as they pop up and gently remind the group to return to the exercise format.

How to Run the Exercise

Aft...
The Good King
by GregorDeVillain — 2022-09-05T19:17:50.879Z

In the late 4th Century BC, Athens executed its statesman and de facto ruler, Phocion. Phocion served Athens with distinction throughout his political career. His leadership was one of extreme care and justice. His was a frugal life, lived in a simple home, and he refused bribes of all sorts, no matter how grand – everything from small fortunes to his own city-state to rule as he liked he refused.

The Athenians chose him to lead them into battle 47 times as general, the most-selected general in Athenian history. Yet he was not a militaristic man – he argued vehemently against wars he thought were bad for his city-state. And he saved Athens from unwise action repeatedly, as when the Athenians wanted to war with Alexander the Great after Alexander had crushed an uprising in Thebes. “Foolhardy man,” Phocion said to Demosthenes, leader of the provocateurs, “why provoke one whose temper is already savage? Why provoke this Macedonian who is full of limitless ambition? When there is a holocaust on our borders, do you wish to spread the flames to our city as well, by provoking him further? My whole object in taking up the burdens of this office is to prevent this, and I shall not allow my fellow citizens to destroy themselves, even if they wish it.”

The Athenians eventually sent Phocion to intercede with Alexander on their behalf, after he had rebuffed all the other emissaries they sent, and he quickly became one of the men Alexander trusted and respected most, even over most Macedonians. Phocion inspired Alexander to look beyond Greece, challenging him that if his goal was to show the greatness of his armies, why not show it by the conquest of the barbarians? Phocion made necessary compromises to the Athenians’ Macedonian ruler, but he negotiated hard to keep the Athenians mostly free.

Yet, after Alexander died, against Phocion’s warnings, the Athenians rebelled against Macedon, and forced Phocion to lead their armies, contrary to his personal desires. He agreed to serve his people as they wished of him, and crafted a resounding victory against the armies of Macedon. Yet Macedonian reinforcements arrived from Asia, and the Greek army was crushed.

Phocion negotiated a lighter reprimand against the Athenians than there otherwise would have been without his intercession. However, many Athenians were still exiled, and Antipater, the new leader of Macedon, still p...
Beta Readers are Great
by HoldenKarnofsky — 2022-09-05T19:10:19.030Z

Back in January, I posted a call for "beta readers": people who read early drafts of my posts and give honest feedback.

The beta readers I picked up that way are one of my favorite things about having started Cold Takes.

Basically, one of my goals with Cold Takes has been to explain my weirdest views clearly, but it's hard to write clearly without detailed feedback on where I'm making sense and where I'm not. I have lots of preconceptions and assumptions that I don't naturally notice. And writing a blog alone doesn't get me that feedback, because:

Most people don't want to explain how they experienced a piece - if they aren't enjoying it, they just want to click away.

And the people who do want to help me out (e.g., friends and colleagues) aren't necessarily going to be honest enough, or representative enough of my target audience (which is basically "People who are interested in my topics but don't already have a ton of background on them").

I've tried a bunch of things to find good beta readers, from recruiting friends of friends (worked well for a bit, but I've written a lot of posts and it was hard to get sustained participation) to paying Mechanical Turk workers to give feedback (some was good, but in general they were uninterested in my weird topics and rushed through the readings and the feedback as fast they could).

The people who came in through the recruiting call in January have been just what I wanted: they're interested in the topics of Cold Takes, but they don't already know me and my thoughts on them, and they give impressively detailed, thoughtful feedback on their reactions to pieces - often a wonderful combination of "intelligent" and "honest that a lot of the stuff I was saying confused the hell out of them." Getting that kind of feedback has been a privilege.

So: THANK YOU to the following beta readers, each of whom has submitted at least 3 thoughtful reviews (and gave permission to be listed here):

Lars Axelsson

Jeremy Campbell

Kanad Chakrabarti

Craig Chatterton

Justin Dickerson

Ethan Edwards

Edward Gathuru

Stian Grønlund

Bridget Hanna

Tyler Heishman

Adam Jermyn

Elliot Jones

Ed William

Scott Leibrand

Evan R. Murphy

John O’Neill

Jaime Sevilla

Josh Simpson

Joshua Templeton

George Thoma

Martin Trouilloud

Morgan Wack

Kevin Whitaker

Arjun Yadav

Patrick Young

If you want to sign up as a beta reader, y...
Impact Shares For Speculative Projects
by Elizabeth — 2022-09-05T18:00:18.800Z

Introduction

Recently I founded a new project with Jasen Murray, a close friend of several years. At founding the project was extremely amorphous (“preparadigmatic science: how does it work?”) and was going to exit that state slowly, if it at all. This made it a bad fit for traditional “apply for a grant, receive money, do work” style funding. The obvious answer is impact certificates, but the current state of the art there wasn’t an easy fit either. In addition to the object-level project, I’m interested in advancing the social tech of funding. With that in mind, Jasen and I negotiated a new system for allocating credit and funding.

This system is extremely experimental, so we have chosen not to make it binding. If we decide to do something different in a few months or a few years, we do not consider ourselves to have broken any promises.

In the interest of advancing the overall tech, I wanted to share the considerations we have thought of and tentative conclusions.

Considerations

All of the following made traditional grant-based funding a bad fit:

Our project is currently very speculative and its outcomes are poorly defined. I expect it to be still speculative but at least a little more defined in a few months.

I have something that could be called integrity and could be called scrupulosity issues, which makes me feel strongly bound to follow plans I have written down and people have paid me for, to the point it can corrupt my epistemics. This makes accepting money while the project is so amorphous potentially quite harmful, even if the funders are on board with lots of uncertainty.

When we started, I didn’t think I could put more than a few hours in per week, even if I had the time free, so I’m working more or less my regular freelancing hours and am not cash-constrained.

The combination of my not being locally cash-constrained, money not speeding me up, and the high risk of corrupting my epistemics, makes me not want to accept money at this stage. But I would still like to get paid for the work eventually.

Jasen is more cash-constrained and is giving up hours at his regular work in order to further the project, so it would be very beneficial for him to get paid.

Jasen is much more resistant to epistemic pressure than I am, although still averse to making commitments about outcomes at this stage.

Why Not Impact Cer

...