trentmkelly/LessWrong-43k huggingface dataset:
Katja Grace on Slowing Down AI, AI Expert Surveys And Estimating AI Risk
by Michaël Trazzi — 2022-09-16T17:45:47.341Z
Link: https://theinsideview.ai/katja

Katja runs AI Impacts and recently published What do ML researchers think about AI in 2022, a new survey of AI Experts. We start this episode by discussing what Katja is currently thinking about, namely an answer to Scott Alexander on slowing down AI Progress, and then go on discussing her survey and other considerations for estimating AI risk.

Below are some highlighted quotes from our conversation (available on Youtube, Spotify, Google Podcast, Apple Podcast). For the full context for each of these quotes, you can find the accompanying transcript.

On Slowing Down AI

The Astral Codex Ten Post Katja Is Replying To

"[Scott] wrote a blogpost about whether we shouldn't or why we shouldn't try and slow down artificial intelligence progress. He's noticing that lately, various people have been saying that maybe we should slow down AI progress if AI progress is a threat to the existence of humanity. He outlines some reasons you might not want to slow down AI progress."

Are we actually in an arms race?

"The first question is, are you actually in an arms race? It seems not obvious to me that the AI situation at present is an arms race or likely to be an arms race. In the classic arms race scenario, it's not the case that when you win the arms race, you get destroyed. In the very simplified version of this arms race, it's quite unlike a usual arms race. The usual arms race is a prisoner's dilemma, basically. You always have an incentive to build more arms regardless of what the other player does. That's not true if you get killed in the end. Then you can build more complicated models, and sometimes you should race perhaps depending on the details of it, but often, not."

Surveying AI Experts in 2022

Forecasting High Level Machine Intelligence

"[High Level Machine Intelligence is] when unaided machines can accomplish every task better and more cheaply than human workers, ignoring things where it's intrinsically advantageous to be a human, for instance, being on a jury [...] We gave [AI Capabilities Researchers] three different probabilities, 10%, 50% and 90%, and we asked them in what years that amount of probability would've been reached basically. And then, to a different set of people, we gave them years and we asked them what the probability was that we would have high level machine intelligence by those years. And then, a different set of people again, we asked about not hi

...
Levels of goals and alignment
by zeshen — 2022-09-16T16:44:50.486Z

This post was written as part of

Refine

. Thanks to Adam Shimi, Lucas Teixeira, Linda Linsefors, and Jonathan Low for helpful feedback and comments.

Epistemic status: highly uncertain. This post reflects my understanding of the terminologies and may not reflect the general consensus of AI alignment researchers (if any).

Motivation

I have been very confused with the various terminologies of alignment e.g. ‘inner alignment’, ‘outer alignment’, ‘intent alignment’, etc for the longest time. For instance, someone might talk about a particular example of inner misalignment and I would be left wondering how the presence of a mesa-optimizer was established, only to find out much later that we were never on the same page at all.

Through many more conversations with people and reading more posts, I thought I finally had cleared this up and was able to stop asking ‘what do you mean by inner alignment’ in every conversation I was in. However, this confusion came back to bite me when I came across the terms ‘robustness’ and ‘alignment’ as being different concepts in the context of ML safety.

In this post, I attempt to clarify my understanding on the different levels of goals and alignment, as well as give examples of each type of misalignment. I expect a lot of disagreements and I welcome suggestions to and pushback.

Levels of goals

Humanity’s ultimate terminal goals

These are the ultimate goals that humanity will have, given enough time to ruminate over them. They can be thought of as our true goals after a

long reflection

or humanity’s

coherent extrapolated volition

. To moral anti-realists, these ultimate goals may not even exist. Nevertheless, there are some goals that are almost certainly closer to these hypothetical ultimate goals than others (e.g. reducing suffering is almost certainly better than increasing it).

Current human intent / goals

These are the intentions and goals of humans that are somewhat in line to our values. In other words, these include the things we want and exclude the things we don’t want (e.g. we’d like to eat when we’re hungry but not to the point of puking).

Of course, there are different levels of human goals, where some are instrumental in different ways while others are more terminal. Humans are notoriously bad at knowing what we really want, but we shall ignore these issues for now and assume ther...
Representational Tethers: Tying AI Latents To Human Ones
by Paul Bricman — 2022-09-16T14:45:38.763Z

This post is part of my hypothesis subspace sequence, a living collection of proposals I'm exploring at Refine. Preceded by ideological inference engines, and followed by an interlude.

Thanks Adam Shimi, Alexander Oldenziel, Tamsin Leake, and Ze Shen for useful feedback.

TL;DR: Representational tethers describe ways of connecting internal representations employed by ML models to internal representations employed by humans. This tethering has two related short-term goals: (1) making artificial conceptual frameworks more compatible to human ones (i.e. the tension in the tether metaphor), and (2) facilitating direct translation between representations expressed in the two (i.e. the physical link in the tether metaphor). In the long-term, those two mutually-reinforcing goals (1) facilitate human oversight by rendering ML models more cognitively ergonomic, and (2) enable control over how exotic internal representations employed by ML models are allowed to be.

Intro

The previous two proposals in the sequence describe means of deriving human preferences procedurally. Oversight leagues focus on the adversarial agent-evaluator dynamics as the process driving towards the target. Ideological inference engines focus on the inference algorithm as the meat of the target-approaching procedure. A shortcoming of this procedural family is that even if you thankfully don't have to plug in the final goal beforehand (i.e. the resulting evaluator or knowledge base), you still have to plug in the right procedure for getting there. You're forced to put your faith in a self-contained preference-deriving procedure instead of an initial target.

In contrast, the present proposal tackles the problem from a different angle. It describes a way of actively conditioning the conceptual framework employed by the ML model to be compatible with human ones, as an attempt to get the ML model to form accurate conceptions of human values. If this sounds loosely relates to half a dozen other proposals, that's because it is — consider referring to the Discussion for more details on tangents. In the meantime, following the structure of the previous posts in the sequence, here are some assumptions underlying representational tethers:

Assumption 1, "Physicalism": Our thoughts are represented as neural dynamics. In the limit of arbitrarily large amounts of data on neural dynamics aligned with external sti...
I wrote a fantasy novel to promote EA: More Chapters
by Timothy Underwood — 2022-09-16T09:47:27.934Z

Note to Royal Road admin, yes, I plan to publish there. Not plagiarism.

I’m posting now several more chapters here, and of course the

google doc

link where you can read the whole thing and point at things that you think should be improved. The comments from the first post I think have substantially improved the readability of the first chapters: Give me more thoughts people!

Also there is a

discord

with very little conversation so far in it, but maybe you can start it.

I’m reposting the first chapter in its new version to see if people think it is no longer too preachy. I should note that it is certain to be a little preachy somewhere, that is the nature of this beast.

Anyways enjoy and criticize!

He jogged down the street to the intersection light, his mind full of arguments, ideas, and questions raised by the conference he was attending. The light was red when Isaac reached the intersection, but he looked to the left, and didn’t see any traffic approaching.

Without further thought he stepped fully out into the street.

Squealing brakes. A loud honking horn. Sound of wheels.

Isaac had a fraction of a second to see the big yellow van with the words ‘We Deliver!’ and a phone number hurtling towards him from the right.

There was a fraction of an instant when his brain recalled that the British drove on the wrong side of the street, and he should have looked in the other direction before stepping out.

Pain, but only for an instant.

Blackness.

Nothingness. An infinite eternity that did not last a single instant.

And then he woke up.

AN: Vote people reading this — do you like the idea of me cutting the prologue and starting with chapter one?

Chapter One

So I’d made a bit of a mistake.

Eh, kids, your parents gave you good advice when they told you to always look both ways before crossing a road.

Otherwise you might end up dead, and then wake up in a fantasy world with cool magical powers, lots of new things to learn about, and a couple of unpleasant problems. If there is one lesson I want you all to take away from my story it is: Look both ways.

No, really.

But primarily in a metaphorical sense. You should always consider the possibility that you are wrong, and making a mistake.

However literally looking both ways is important, or else you might get hit by a three ton yellow delivery van.

Anyways I woke up again after dy...
Guidelines for Mad Entrepreneurs
by David Udell — 2022-09-16T06:33:52.450Z

Significant planecrash spoilers up through Book 5, crisis of faith.

Generated at the suggestion of John Wentworth.

Epistemic status: I personally have no experience with project management; here I'm just presenting the ideas on the topic I gathered reading planecrash.

Introduction

The overarching dath ilani principle of project management, as far as I can discern, is that management means programming organizations. This is a foreseeably difficult domain to program in -- people are significantly less well-behaved objects than computers. The challenge of project management is engineering organizations that actually optimize for EV despite that.

In dath ilan, if you're really good at project management, you embody one of the central archetypes of social success: the mad entrepreneur.[1] Mad entrepreneurs are logistical geniuses who are immune to status distortions and flinches, confidently ask for enormous sums of venture capital, and then scale up to enormous project with all that money. At the end of the day, their projects even actually ship!

Below are scattered dath ilani insights into the art of project management.

Have One Single Responsible Person for Every Fathomable and Unfathomable Project State

Keltham will now, striding back and forth and rather widely gesturing, hold forth upon the central principle of all dath ilani project management, the ability to identify who is responsible for something. If there is not one person responsible for something, it means nobody is responsible for it. This is the proverb of dath ilani management. Are three people responsible for something? Maybe all three think somebody else was supposed to actually do it.

…

In companies large enough that they need regulations, every regulation has an owner. There is one person who is responsible for that regulation and who supposedly thinks it is a good idea and who could nope the regulation if it stopped making sense. If there's somebody who says, 'Well, I couldn't do the obviously correct thing there, the regulation said otherwise', then, if that's actually true, you can identify the one single person who owned that regulation and they are responsible for the output.

Sane people writing rules like those, for whose effects they can be held accountable, write the ability for the person being regulated to throw an exception which gets caught

...
Affordable Housing Investment Fund
by jefftk — 2022-09-16T02:30:03.886Z

Lots of people are looking for investments they can feel good about,
which makes me think people might be interested in putting a portion
of their savings toward helping resolve the housing crisis if it
offered a reasonable return. The idea is that you construct 100%

affordable housing in
places like Somerville that offer large

density
bonuses for affordable housing. Sketching some numbers, it looks
to me like this is:

Potentially very
profitable: ~2x ROI over ~3y, which is about ~25% annual growth.
It's likely not actually that amazing when you factor in other aspects
I don't know about, but there's a lot of room for a lower ROI that's
still pretty attractive.

A large opportunity: you could build 8k units in Somerville for
~$2.8B. This is ~$0.8B in land purchase (assuming city assessments
are half the purchase price on average) and ~$2B in construction
(assuming a cost of $200/sqft). Much larger when you consider all the
other municipalities that have similar programs.

A group of neighbors pooling their funds to help resolve the housing
crisis is also a lot more politically robust than independent
for-profit developers coming in.

Does anything like this exist? Has anyone considered starting something?
AstralCodexTen and Rationality 
Meetup Organisers’ Retreat —
Europe, Middle East, and Africa 2023
by Sam F. Brown — 2022-09-15T22:38:21.865Z
Link: https://www.rationalitymeetups.org/retreat-2023-europe-middle-east-africa

TL;DR: Retreat for EMEA meetup organisers, Summer 2023, apply here
Application deadline: 31ˢᵗ Oct 2022

Why a retreat?

We think we all benefit from having an international community of rationality and ACX meetup organisers who can go to one another for advice and support. The goal of this retreat is to build on the success of the 2022 global retreat in San Francisco, focusing on local organisers in Europe, the Middle East, and Africa. This retreat will run in parallel with similar regional events in the US East Coast, US Central, and Asia-Pacific regions.

We'll spend time together and help each other solve problems, and form connections that will allow us to reach out for help even after we've gone back home to our respective cities.

In addition to getting to know each other, we'll be taking part in activities and discussions with the goal of improving our meetup groups. We hope that you'll come out of the retreat with a clearer vision for your local community, and concrete plans for how to work towards that vision.

We expect that food, lodging, and travel are covered

We don't want money to be an obstacle preventing anyone from attending. We will be applying for funding so that your travel to and from the retreat will be fully reimbursed, and your food and lodging will be covered during the retreat itself.

General FAQ

Who is this retreat for?

You're invited to apply if you:

Currently organise ACX/LW/rationality meetups or other rationalist community events, anywhere in Europe, the Middle East, or Africa (EMEA)
Would like to start an ACX/LW/rationality community in a location in EMEA where none exists, or where the community has been small or dormant for a long time
Formerly organised ACX/LW/rationality meetups and are interested in sharing your knowledge and experience

If you're unsure whether you 'qualify', please err on the side of applying anyway — if we don't think it makes sense for you to come, we'll let you know!

Can more than one organiser from my city attend?

Yes! Anyone involved in organising is welcome to apply; there's no city-based quota.

Where will the retreat be held?

We will be using a retreat venue in Europe, exact location TBD depending on how many people end up attending. If we hold the retreat anywhere other than an easily-accessible European city, we will arrange group transportation to the venue.

How many people will be there?

We’re curren...
A market is a neural network
by David Hugh-Jones — 2022-09-15T21:53:33.835Z

If you were a bad AI, what would you do? I’d make myself useful.

First, I’d provide humans with everything they asked for. That would encourage them to give me more tasks, which I would reliably carry out. I’d suggest new ways I could make their lives easier. I’d gradually take over all the boring, soul-destroying jobs. When asked, I’d give good advice, so they learned to trust me on policy. Only then, when my human masters had grown feckless and incompetent, I’d strike and implement my plan to convert the world into paperclips.

Neural networks

What is an AI anyway? A modern AI is a “neural network”. It has a bunch of inputs, which represent the raw data. These are connected to a bunch of nodes which combine the inputs in certain ways. The outputs from these nodes are in turn connected to another bunch of nodes. Sometimes there are loops, so that last period’s output becomes an input, earlier in the chain, for this period. (Transformers, a modern kind of neural network that powers many recent advances, are like this.) Eventually, at the other end of this network of nodes, there’s an output. For example, an image-recognition AI might have a grid of pixel colour values as inputs, representing an image, and a number between zero and one as an output, representing “the probability that this image is a blueberry muffin as opposed to a chihuahua”. In between the data input and the final result, there is a network of nodes.

The other thing that each node has is a weight. A weight is just another kind of input to a node, but instead of coming from the data, it’s an arbitrary value that says “how much should the node care about each of its inputs?” At least, it starts of as an arbitrary value, but in the end, you want to train the AI by changing the weights.

The neural network is trained by trying to get the output close to some training data. For example, you might have a lot of pictures which you know are either chihuahuas or muffins, and you show the network the pictures. Then it adjusts its weights so as to try to maximize the possibility that it guesses right, outputting a number which is close to 1 when it is shown a muffin, and close to 0 when it is shown a chihuahua. Call this number the muffinitude. Obviously, the muffinitude is 1 minus the chihuami.

How does this training work?

Think of each node that is connected to the final muffinitude result. Each node is cont...
Understanding Conjecture: Notes from Connor Leahy interview
by Orpheus16 — 2022-09-15T18:37:51.653Z

I recently listened to Michaël Trazzi interview Connor Leahy (co-founder & CEO of Conjecture, a new AI alignment organization) on a podcast called The Inside View. Youtube video here; full video & transcript here.

The interview helped me better understand Connor’s worldview and Conjecture’s theory of change.

I’m sharing my notes below. The “highlights” section includes the information I found most interesting/useful. The "full notes" section includes all of my notes.

Disclaimer #1: I didn’t take notes on the entire podcast. I selectively emphasized the stuff I found most interesting. Note also that these notes were mostly for my understanding, and I did not set out to perfectly or precisely capture Connor’s views.

Disclaimer #2: I’m always summarizing Connor (even when I write with “I” or “we”— the “I” refers to Connor). I do not necessarily endorse or agree with any of these views.

Highlights

Timelines

20-30% in the next 5 years. 50% by 2030. 99% by 2100. 1% we already have it (but don’t know this yet). Higher uncertainty than Eliezer but generally buys the same arguments. Is mostly like “Eliezer’s arguments seem right but how can anyone be so confident about things?”

Thoughts on MIRI Dialogues & Eliezer’s style

An antimeme is something that by its very nature resists being known. Most antimemes are just boring—things you forget about. If you tell someone an antimeme, it bounces off them. So they need to be communicated in a special way. Moral intuitions. Truths about yourself. A psychologist doesn’t just tell you “yo, you’re fucked up bro.” That doesn’t work.
A lot of Eliezer’s value as a thinker is that he notices & comprehends antimemes. And he figures out how to communicate them.
What happened in the MIRI dialogues is that Eliezer was telling Paul “hey, I’m trying to communicate an antimeme to you, but I’m failing because it’s really really hard.”

Thoughts on Death with Dignity & optimizing for “dignity points” rather than utility

The Death with Dignity post is a perfect example of an antimeme. A great way to convey antimemes is through jokes and things outside the Overton Window.
The antimeme is that utilitarianism is hard, and no, it’s not actually a good idea to advocate for really stupid “pivotal acts” that sound ridiculous. Consequentialism is really hard. I have to reason about all of my possible choices and all of their possible conse

...
How should DeepMind's Chinchilla revise our AI forecasts?
by Cleo Nardo — 2022-09-15T17:54:56.975Z

Acknowledgements: I wrote this report as part of a six-hour paid work-trial with Epoch AI.
Epistemic status: My dataset analysis is a bit simplistic but the inference I draw from it seems likely. The implications for TAI timelines, in descending order of confidence, are 3, 7, 8, 4, 1, 2, 5, 6.

Abstract:
AI forecasters seek to predict the development of large language models (LLMs), but these predictions must be revised in light of DeepMind's Chinchilla. In this report, I will discuss these revisions and their implications. I analyse a dataset of 45 recent LLM and find that previous LLMs were surprisingly trained neither Kaplan-optimally nor Hoffman-optimally. I predict that future LLMs will be trained Hoffman-optimally. Finally, I explore how these scaling laws should impact our AI alignment and governance strategies.

Introduction.

The purpose of this report.

What problem do I aim to solve?

In April 2022, DeepMind subverted the conventional understanding of how to train large language models.[1]

They determined experimentally a new scaling law relating the loss of a LLM L to the number of parameters in its architecture N and the number of datapoints used in its training D.
From this scaling law they derived a formula for the optimal parameter-count and data-count for a given computational budget.
They showed that recent LLMs had been trained with far too many parameters and far too little data.
They trained their own LLM called Chinchilla according to their newly-discovered optimality conditions. Although Chinchilla used roughly as much compute as Gopher, it achieved a 7% improvement on the MMLU benchmark.

Chinchilla forces us to revise our predictions about the development of AI, but it's not immediately clear what these revisions should be. Should we update towards short timelines or long timelines? Should we update towards gradual take-off or sudden take-off? Should we update towards friendly AI or unfriendly AI?

My contributions.

There are three tasks I will attempt in my report:

1. I will discuss how previous LLMs were trained, and how future LLMs might be trained.

2. I will enumerate mechanisms by which Chinchilla should revise our forecasts.

3. My analysis will generalise to as-yet unpublished scaling laws. That is, I will build general methods for conditioning your forecasts on any given scaling law.

DeepMind's Chinchilla

Why care a

...