trentmkelly/LessWrong-43k huggingface dataset:
Response to Katja Grace's AI x-risk counterarguments
by Erik Jenner — 2022-10-19T01:17:54.728Z

This is a response to the recent Counterarguments to the basic AI x-risk case ("Counterarguments post" from here on). Based on its reception, it seems that the Counterarguments post makes points that resonate with many, so we're glad that the post was written. But we also think that most of the gaps it describes in the AI x-risk case have already been addressed elsewhere or vanish when using a slightly different version of the AI x-risk argument. None of the points we make are novel, we just thought it would be useful to collect all of them in one reply.

Before we begin, let us clarify what we are arguing for: we think that current alignment techniques are likely insufficient to prevent an existential catastrophe, i.e. if AI development proceeds without big advances in AI alignment, this would probably lead to an existential catastrophe eventually. In particular, for now, we are not discussing

how hard open alignment problems are,
whether these problems will be solved anyway without efforts from longtermists,
whether an existential catastrophe would happen within a particular time frame,
what a reasonable all-things-considered p(doom) is.

These are all important questions, but the main thrust of the Counterarguments post seems to be "maybe this whole x-risk argument is wrong and things are actually just fine", so we are focusing on that aspect.

Another caveat: we're attempting to present a minimal high-level case for AI x-risk, not to comprehensively list all arguments. This means there are some concepts we don't discuss even though they might become crucial on further reflection. (For example, we don't talk about the concept of coherence, but this could become important when trying to argue that AI systems trained for corrigibility will not necessarily remain corrigible). Anticipating all possible counterarguments and playing out the entire debate tree isn't feasible, so we focus on those that are explicitly made in the Counterarguments post. That said, we do think the arguments we outline are broadly correct and survive under more scrutiny.

Summary

Our most important points are as follows:

The ambiguity around "goal-directedness" can be resolved if we consistently think about AI systems that are able to achieve certain difficult-to-achieve objectives.
While certain small differences between AI and human values would be fine, there are good reasons to believe that the

...
(OLD) An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers
by Neel Nanda — 2022-10-18T21:08:33.033Z
Link: https://www.neelnanda.io/mechanistic-interpretability/favourite-papers

This post is out of date, see v2 here

Introduction

This is an extremely opinionated list of my favourite mechanistic
interpretability papers, annotated with my key takeaways and what I like
about each paper, which bits to deeply engage with vs skim (and what to
focus on when skimming) vs which bits I don’t care about and recommend
skipping, along with fun digressions and various hot takes.

This is aimed at people trying to get into the field of mechanistic
interpretability (especially Large Language Model (LLM)
interpretability). I’m writing it because I’ve benefited a lot by
hearing the unfiltered and honest opinions from other researchers,
especially when first learning about something, and I think it’s
valuable to make this kind of thing public! On the flipside though, this
post is explicitly about my personal opinions - I think some of these
takes are controversial and other people in the field would disagree.

The four top level sections are priority ordered, but papers within each
section are ordered arbitrarily - follow your curiosity

Priority 1: What is Mechanistic Interpretability?

Circuits: Zoom
In

Sets out the circuits research agenda, and is a whirlwind
overview of progress in image circuits

This is reasonably short and conceptual (rather than technical)
and in my opinion very important, so I recommend deeply
engaging with all of it, rather than skimming.

The core thing to take away from it is the perspective of
networks having legible(-ish) internal representations of
features, and that these may be connected up into
interpretable circuits. The key is that this is a mindset for
thinking about networks in general, and all the discussion
of image circuits is just grounding in concrete examples.

On a deeper level, understanding why these are important and
non-trivial claims about neural networks, and their
implications.

In my opinion, the circuits agenda is pretty deeply at the core
of what mechanistic interpretability is. It’s built on the
assumption that there is some legible, interpretable structure
inside neural networks, if we can just figure out how to
reverse engineer it. And the core goal of the field is to find
what circuits we can, build better tools for doing so, and do
the fundamental science of figuring out which of the claims
about circuits are actually true, which ones break, and
whether we can fix them.

An important note i

...
Distilled Representations Research Agenda
by Hoagy — 2022-10-18T20:59:20.055Z

Introduction

Schematic of the basic training strategy

I’ve recently been given funding from the Long Term Future Fund to develop work on an agenda I'll tentatively call Distilled Representations, and I'll be working on this full-time over the next 6 months with Misha Wagner (part time).

We're working on a way of training autoencoders so that they can only represent information in certain ways - ways that we can define in a flexible manner.

It works by training multiple autoencoders to encode a set of objects, while for some objects defining a preferred representation that the autoencoders are encouraged to encode the objects as. We then distill these multiple autoencoders into single autoencoder which encodes only that information which is encoded in the same way across the different autoencoders. If we are correct, this new autoencoder should only encode information using the preferred strategy. Vitally, this can be not just the original information in the preferred representations, but also information represented by generalizations of that encoding strategy.

It is similar to work such as Concept Bottleneck Models but we hope the distillation from multiple models should allow interpretable spaces in a much broader range of cases.

The rest of this post gives more detail of the intuition that we hope to build into a useful tool, some toy experiments we’ve performed to validate the basic concepts, the experiments that we hope to build in the future, and the reasons we hope it can be a useful tool for alignment.

We'd like to make sure we understand what similar work has been done and where this work could be useful. If you're familiar with disentangled representations, or interpretability tools more generally, we're interested in having a chat. You can reach me here on LessWrong or at hoagycunningham@gmail.com.

Previous versions of similar ideas can be found in my ELK submission and especially Note-taking Without Hidden Messages.

Intuition

The intuition that this work builds on is the following:

With neural networks, the meanings of the weights and activations are usually opaque but we're often confident about the kind of thing that the network must be representing, at least for some cases or parts of the input distribution.
In those cases where we understand what the network is representing, we can condense this understanding into a vector, thus defining a 'preferred rep

...
Drafting a Covid Survey
by jefftk — 2022-10-18T19:30:08.982Z

In thinking about what sort of covid policies a dance organization
should have I see two main plausible goals:

Be responsible: don't appreciably increase harm to non-attendees.

Reflect the preferences of your community.

On #1, this was why I
thought dances should shut down initially in March 2020. Our
government was not well prepared for covid and initially we needed to
be figuring out for
ourselves what was safe. With vaccination generally available,
plus updated guidance and regulations, however, most other things have
fully resumed without requiring masks or vaccination. At this point,
I don't think #1 is key consideration.

On #2, there are many different ways you might do this. For example,
you might try to hold dances that the most people would attend—this
might mean strict precautions (because dancers feel strongly that this
is what makes it safe enough to attend) or minimal precautions
(because dancing in a mask is less fun and many people don't want to
get another booster). I don't actually know what sort of policies
would maximize attendance!

You'd also want to consider whether to prioritize different interests
differently. Compare two hypotheticals:

Pat would come if you didn't require masks, but since you do
they don't. Dancing in a mask is unpleasant enough that they'd
rather do something else.

Sam would come if you required the new bivalent boosters, but
since you don't they don't. The risk of being in a crowded
hall with people whose immune systems have forgotten how to
handle covid is higher than they're comfortable with.

Are these equally bad, each one person who doesn't attend? Or do you
care more about Sam's situation because it's about what they feel is
safe, while Pat's is more about what they feel is fun?

Still, what sort of policies are most popular among your dancers
remains important. I think surveying dancers makes a lot of sense.
Even if you aren't going to treat it like a vote, it's still very
helpful in trying to fit your policies to the range of views in your
community. I'm proposing the dance organization I help run do a
survey, and here's what I've currently drafted:

We're trying to understand the preferences of our community
around covid precautions—does the dance feel too risky? Too
conservative? What would you like to see different?

...
How To Make Prediction Markets Useful For Alignment Work
by johnswentworth — 2022-10-18T19:01:01.292Z

So, I’m an alignment researcher. And I have a lot of prediction-flavored questions which steer my day-to-day efforts. Which recent interpretability papers will turn out, in hindsight a year or two from now, to have been basically correct and high-value? Will this

infrabayes stuff

turn out to be useful for real-world systems, or is it the kind of abstract math which lost contact with reality and will never reconnect? Is there any mathematical substance to Predictive Processing other than “variational inference is a thing” plus vigorous hand-waving? How about

Shard Theory

? What kinds of evidence will we see for/against the Natural Abstraction Hypothesis over the next few years? Will

HCH-style amplification

ever be able to usefully factor problems which the human operator/programmer doesn’t immediately see how to factor? Will some version of

this conjecture

be proven? Will it be proven by someone else if I don’t focus on it? Are there any recent papers/posts which lots of other people expect to have high value in hindsight, but which I haven’t paid attention to?

Or, to sum that all up in two abstract questions: what will I (or someone else whose judgement I at least find informative) think I should have paid more attention to, in hindsight? What will it turn out, in hindsight, that I should have ignored or moved on from sooner?

Hmm, I wonder if the prediction markets have anything useful to say here? Let’s go look for AI predictions on Manifold…

hollywood-level AI-generated feature film by 2026?

Will an AI get gold on any International Math Olympiad by 2025?

Will AI wipe out humanity before the year 2100

Will any Fortune 500 corporation mostly/entirely replace their customer service workforce with AI by 2026?

…

So, we’ve got about a gazillion different flavors of AI capabilities questions, with a little bit of dabbling into how-society-will-react-capabilities questions. On the upside,

operationalizing things in lots of different ways is exactly what we want

. On the downside, approximately-all of the effort is in operationalizing one thing (AI capabilities timelines), and that thing is just not particularly central to day-to-day research decisions. It’s certainly not on the list of questions which first jump to mind when I think of things which would be useful to know to steer my research. Sure, somebody is going to argue in the comments th...
A conversation about Katja's counterarguments to AI risk
by Matthew Barnett — 2022-10-18T18:40:36.543Z

This post is a transcript of a conversation between Ege Erdil and Ronny Fernandez, recorded by me. The participants talked about a recent post by Katja Grace that presented many counterarguments to the basic case for AI x-risk. You might want to read that post first.

As it was transcribed automatically by Whisper, along with some light editing on top, there might be many mistakes in the transcript. You can also find the audio file here, but note that the first five lines of the dialogue do not appear in the audio.

Ronny Fernandez wants to make it clear that he did not know the conversation would eventually be published. (But all participants have consented to it being published).

Ege Erdil
You know about Epoch, right?

Ronny Fernandez
Not really, actually.

Ege Erdil
OK, so Epoch is a recently established organization that focuses on forecasting future developments in AI. And Matthew is currently working there and hosted this post, like, with Slack. So there's some discussion of that.

Ronny Fernandez
Gotcha.

Ege Erdil
And he basically said that he thought that it was a good post. We didn't. I mean, it's not just me. There's also someone else who thought that it wasn't really very good. And that's just something he said, oh, you know, I should talk to Ronny because he also thinks that the post was good.

Ronny Fernandez
Yeah, sweet. So question I want to ask is, do you think the post wasn't good because you think that the summary of the basic argument for existential risk from superintelligence is not the right argument to be criticizing? Or do you think the post wasn't good because the criticisms weren't good of that argument or for some other reason?

Ege Erdil
I think it's both. I think there's somewhat of criticisms are, I think, quite poor. But I also think the summary of the argument is not necessarily that great. So yeah. So we could also go through the post and order that arguments are given. Or I could just talk about the parts that we talked about.

Ronny Fernandez
I'll pull up the post now. No, I think it'd be great to take a look at the argument and talk about how you would repartition the premises or change the premises or whatever.

Ege Erdil
Yeah. So let me first go to the last section, because I think it is actually interesting in the sense that the given arguments could in principle apply to corporations. And the post points this out. This is an example. It's be...
Untapped Potential at 13-18
by belkarx — 2022-10-18T18:09:56.352Z

The Western world is facing a crisis of stretched adolescence, keeping children exclusively "learning" for longer than necessary.

Imagine that you are a student, around 14 years old. You fall into the top 5%, not the extremely motivated startup founder or math prodigy that you hear about on the news, but comfortably more intelligent and curious than the others. The system sucks some of your free time with busywork, but you build up a knowledge base, and start working on small projects related to your area of interest. Now what?

Young people are capable. Their neurons are fully myelinated, their worldview is fresh, their passion is strong. But we don't use them.

In fact we make it difficult for them to thrive: labor laws make it bothersome for companies to take on anyone under 18, even if they can demonstrate competence, so that path is rocky. Research internships and freelancing are more viable, but still extremely underused. Perhaps it's because the school system drowns people in busywork, forcing them to study things just to forget them within a few days,[1] and personal projects are stifled under the workload. Competent teens fall into apathy because the work is too easy, application of anything more difficult is inaccessible, and there's only so much knowledge-gathering they can do before they want to do something.

We need to put frameworks in place to take advantage of capable high school students. The current assumption is that the most qualified will find a way, and while that may be true, that leaves behind a huge untapped resource. Not everyone is hyper-ambitious but if provided engaging work or more exposure to interesting problems, they can be both useful and satisfied, and the world benefits.

The Internet provides some of these opportunities, however working with "professionals" on larger problems is an experience that cannot be replicated. The combination of years of wisdom and accumulated knowledge with a modern view of a problem without the inset biases of current models has serious potential.

TL;DR: Teenagers are underappreciated and a vast and unique source of cognitive power that should be more cleanly introduced into the working world.

^

https://en.wikipedia.org/wiki/Forgetting_curve
How easy is it to supervise processes vs outcomes?
by Noosphere89 — 2022-10-18T17:48:24.295Z

A lot of AI risk models rely on the fact that we supervise outcomes like consequentialists rather than processes like CAIS.

But, how easy is it to supervise processes for AI, compared to outcomes both in:

Near term: 10 years.

Long-term: 50 years.

Crucially, is it easier to supervise processes like CAIS today than outcomes?

And a final question. Should we fund massively attempts to supervise processes in today's AI, in the hope that it's an attractor state rather than outcomes like consequentialism?
Is GitHub Copilot in legal trouble?
by tcelferact — 2022-10-18T16:19:34.842Z

This is basically a crosspost for https://githubcopilotinvestigation.com/. I noticed that some folks in California are considering a lawsuit against Microsoft/OpenAI.

tl;dr:

Copilot is trained on open source software.
Copilot doesn't respect the licensing agreements of that software.
Copilot doesn't have a clear fair use argument for doing so.
By accepting copilot suggestions, you are potentially violating licensing agreements yourself.

Sections of particular interest:

[W]e inquired pri­vately with Fried­man and other Microsoft and GitHub rep­re­sen­ta­tives in June 2021, ask­ing for solid legal ref­er­ences for GitHub’s pub­lic legal posi­tions … They pro­vided none.

- Software Freedom Conservancy

“You are respon­si­ble for ensur­ing the secu­rity and qual­ity of your code. We rec­om­mend you take the same pre­cau­tions when using code gen­er­ated by GitHub Copi­lot that you would when using any code you didn’t write your­self. These pre­cau­tions include rig­or­ous test­ing, IP [(= intel­lec­tual prop­erty)] scan­ning [my emphasis], and track­ing for secu­rity vul­ner­a­bil­i­ties.”

- https://docs.github.com/en/copilot/overview-of-github-copilot/about-github-copilot#using-github-copilot
Metaculus is building a team dedicated to AI forecasting
by ChristianWilliams — 2022-10-18T16:08:11.039Z
Link: https://apply.workable.com/metaculus/

The speed, sophistication, and impacts of AI technology development together comprise some of the most astonishing and significant events of our lifetimes. AI development promises both enormous risks and opportunities for society.

Join Metaculus's AI forecasting

team and help humankind better navigate this crucial period.

Open roles include:

Machine Learning Engineer - AI Forecasting

You’ll work to enhance the organization and searchability of our AI analyses, ensure that the AI-related data and thinking that we rely on is up-to-date, comprehensive, and well organized, and deliver (via modeling) forecasts on an enormous set of questions concerning the trajectory of AI.

Research Analyst - AI Forecasting

You’ll engage deeply with ideas about the future of AI and its potential impacts, and share insights with the AI research and forecasting communities and with key decision makers. You’ll use crowd forecasting to help generate these insights, writing forecasting questions that are informative and revealing, facilitating forecasting tournaments, and coordinating with Pro Forecasters.

Quantitative Research Analyst - AI Forecasting

You’ll use quantitative modeling to improve our ability to anticipate the future of AI and its impact on the world, enhance our AI-related decision making capabilities, and enable quantitative evaluation of ideas about the dynamics governing AI progress.

You can learn about our other high-impact, open positions

here

.
How to Take Over the Universe (in Three Easy Steps)
by Writer — 2022-10-18T15:04:27.420Z
Link: https://youtu.be/fVrUNuADkHI

EA-Forum crosspost

This is the script of Rational Animations' video linked above. It's about how to take over the universe with amounts of energy and resources that are small compared to what is at our disposal in the Solar System. It's based on this paper, by Anders Sandberg and Stuart Armstrong.

This is our highest-quality video so far. Below, the script of the video.

Let’s take over the universe in three easy steps

Welcome. We’ve heard that you want to take over the universe. Well, you’ve come to the right place. In this video, we’ll show you how to reach as many as four billion galaxies with just a few relatively easy steps and six hours of the Sun’s energy.

Here’s what you need to do:

Disassemble mercury and build a Dyson swarm: a multitude of solar captors around the sun.
Build self-replicating probes
Launch the self-replicating probes to every reachable galaxy.

In science fiction, humanity’s expansion into the universe usually starts within our galaxy, the Milky Way. After a new star system is occupied, humanity jumps to the next star, and so on, until we take the whole galaxy. Then, humanity jumps to the nearest galaxy, and the process is repeated.

This is not how we’re going to do it. Our method is much more efficient. We're going to send self-replicating probes to all the reachable galaxies at once. Getting to the furthest galaxies is not more difficult than getting to the nearest ones. It just takes more time. When a probe arrives at its destination galaxy, it will search for a planet to disassemble, build another Dyson swarm, and launch a new wave of probes to reach every star within the galaxy. And then, each probe in that galaxy will restart civilization.

We already hear you protest, though: “this whole thing still seems pretty hard to me,” you say. “Especially the “disassembling mercury” part”.

But actually, none of these steps are as hard as they first appear. If you analyze closely how they could be implemented you’ll find solutions that are much easier than you’d expect. And that’s exactly what Stuart Armstrong and Anders Sandberg do in their paper “Eternity in six hours: intergalactic spreading of intelligent life and sharpening the Fermi paradox.” This video is based on that paper.

Exploratory engineering and assumptions

What we mean by “easy” here, is that we will require amounts of energy and resources that ...