trentmkelly/LessWrong-43k huggingface dataset:
Analogies and General Priors on Intelligence
by riceissa — 2021-08-20T21:03:18.882Z

This post is part 2 in our sequence on Modeling Transformative AI Risk. We are building a model to understand debates around existential risks from advanced AI. The model is made with

Analytica

software, and consists of nodes (representing key hypotheses and cruxes) and edges (representing the relationships between these cruxes), with final output corresponding to the likelihood of various potential failure scenarios. You can read more about the motivation for our project and how the model works in the

Introduction post

. Future posts will explain how different considerations, such as AI takeoff or mesa-optimization, are incorporated into the model.

This post explains our effort to incorporate basic assumptions and arguments by analogy about intelligence, which are used to ground debates about AI takeoff and paths to High-Level Machine Intelligence (HLMI[1]). In the overall model, this module, Analogies and General Priors on Intelligence, is one of the main starting points, and influences modules (covered in subsequent posts in this series) addressing the possibilities of a Discontinuity around HLMI or an Intelligence Explosion, as well as AI Progression and HLMI Takeoff Speed.

The Analogies and General Priors on Intelligence module addresses various claims about AI and the nature of intelligence:

The difficulty of marginal intelligence improvements at the approximate ‘human level’ (i.e., around HLMI)
Whether marginal intelligence improvements become increasingly difficult beyond HLMI at a rapidly growing rate or not

‘Rapidly growing rate’ is operationalized as becoming difficult exponentially or faster-than-exponentially

Whether there is a fundamental upper limit to intelligence not significantly above the human level
Whether, in general, further improvements in intelligence tend to be bottlenecked by previous improvements in intelligence rather than some external factor (such as the rate of physics-limited processes)

These final outputs are represented by these four terminal nodes at the bottom of the module.

These four claims depend on arguments that analogize the development of HLMI to other domains. Each of these arguments is given a specific submodule in our model, and each argument area is drawn from our review of the existing literature. In the ‘outputs’ section at the end of this article, we explain why we chose these four as cruxes for the overall debate aro...
Provide feedback on Open Philanthropy’s AI alignment RFP

by abergal — 2021-08-20T19:52:55.309Z

Open Philanthropy is planning a request for proposals (RFP) for AI alignment projects working with deep learning systems, and we’re looking for feedback on the RFP and on the research directions we’re looking for proposals within. We’d be really interested in feedback from people on the Alignment Forum on the current (incomplete) draft of the RFP.
The main RFP text can be viewed

here

. It links to several documents describing two of the research directions we’re interested in:

Measuring and forecasting risks

Techniques for enhancing human feedback

[Edit: this previously linked to an older, incorrect version]

Please feel free to comment either directly on the documents, or in the comments section below.
We are unlikely to add or remove research directions at this stage, but we are open to making any other changes, including to the structure of the RFP. We’d be especially interested in getting the Alignment Forum’s feedback on the research directions we present, and on the presentation of our broader views on AI alignment. It’s important to us that our writing about AI alignment is accurate and easy to understand, and that it’s clear how the research we’re proposing relates to our goals of reducing risks from power-seeking systems.
How DeepMind's Generally Capable Agents Were Trained
by 1a3orn — 2021-08-20T18:52:52.512Z

Intro

One of DeepMind's latest papers, Open-Ended Learning Leads to Generally Capable Agents, explains how DeepMind produced agents that can successfully play games as complex as hide-and-seek or capture-the-flag without even having trained on or seen these games before.

As far as I know, this is an entirely unprecedented level of generality for a reinforcement-learning agent.

The following is a high-level summary of the paper, meant to be accessible to non-specialists, that should nevertheless produce something resembling a gears-level model. I want to focus on explaining the optimization process that produced this agent; on what the different parts of the optimization process are; on why each different part is necessary; and on what would happen if different parts of it were missing.

After that summary, I'll add a few more comments and questions about design choices within the paper and about future research I'd like to see. I'm far less certain about this second part, however.

I was going to include a part on AI timelines -- but whether this paper influences your timelines, and in what direction, depends on a lot of priors that are out-of-scope for what I want to do here.

The Environment

Before we get into the optimization process of the agent, I need to talk about the environment within which the agent trained. Core to the project of this paper are the millions of dynamically-generated tasks on which the agent can train.

Each task in the XLand Unity-powered 3d environment space is defined by a (1) unique physical world and a (2) unique set of goals / rewards. Thoughout what follows I refer to (1) as the "environment" or "world", to (2) as the "game", and to both of them together as a "task." Note that both of these can be generated programmatically without human intervention.

(The show reel of the trained agents operating on hand-made, held-out test tasks is worth watching for at least a few minutes to get a feel for the complexity possible from both world and goals, and is probably much clearer than my writing about the environment space. [I mean, if you want to understand soccer, watch a game of it, don't read a description.] Although you should note that the intelligibility of the goals in the video is uncharacteristic of the goals in the training tasks, because the show-reel goals were made by humans from human-intelligible games, rather than randomly gene...
[AN #161]: Creating generalizable reward functions for multiple tasks by learning a model of functional similarity
by Rohin Shah — 2021-08-20T17:20:04.380Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos (Annie S. Chen et al) (summarized by Sudhanshu): This work demonstrates a method that learns a generalizable multi-task reward function in the context of robotic manipulation; at deployment, this function can be conditioned on a human demonstration of an unseen task to generate reward signals for the robot, even in new environments.

A key insight here was to train a discriminative model that learned whether two given video clips were performing the same actions. These clips came from both a (large) dataset of human demonstrations and a relatively smaller set of robot expert trajectories, and each clip was labelled with a task-id. This training pipeline thus leveraged huge quantities of extant human behaviour from a diversity of viewpoints to learn a metric of 'functional similarity' between pairs of videos, independent of whether they were executed by human or machine.

Once trained, this model (called the 'Domain-agnostic Video Discriminator' or DVD) can determine if a candidate robotic behaviour is similar to a desired human-demonstrated action. Such candidates are drawn from an action-conditioned video predictor, and the best-scoring action sequence is selected for execution on the (simulated or real) robot.

Read more: Paper

Sudhanshu's opinion: Performance increased with the inclusion of human data, even that from unrelated tasks, so one intuition I updated on was "More data is better, even if it's not perfect". This also feels related to "Data as regularization": to some extent, noisy data combats model overconfidence, and perhaps this would play an important role in aligning future systems.

Another thing I like about such pipeline papers is the opportunity to look for where systems might break. For example, in this work, the robot does actually need (prior) experience in the test environments with which to train the video predic...
Exploring Democratic Dialogue between Rationality, Silicon Valley, and the Wider World
by Remmelt — 2021-08-20T16:04:44.683Z

I was asked to review a draft that critiques the broader rationality movement for a tendency to engineer elegant solutions to fix the world in ways that bypass 'inefficient' democratic dialogue.

However, the authors worried that their explanations would trigger ingroup-outgroup conflict amongst readers, rather than engage rationalists to discuss and collaborate on reforms. They deliberately decided to ditch this version and not publish it in the media.

IMO though, they made some solid points about the broad sociology of the rationalist movement. So I selected and edited excerpts for clarity and constructiveness (skipping the intro that got into the already much discussed controversy between New York Times journalist Cade Metz and Scott Alexander), posted below with the authors’ permission.

Do ask friends for their two cents, but please share the link

in private

. Understanding this piece requires some context on how our community works, and throwing out a message on Twitter, Reddit, and the like is a poor choice for sharing that context.

I hope that you will find some insightful nuggets amongst the commentary.
What's your sense of what the article

describes accurately

, and what it

fails to capture

?

Edit: I should have more literally said that, yes, this draft

has a bunch of commentary that is vague and antipathetic

, and I hope you're going to get something out of it anyway.

The deep connections between Silicon Valley, rationalism, and the anti-democratic right are more complicated, and more revealing, than either Metz or his critics allow for. If Silicon Valley simply harbored a virulent minority of semi-closeted white supremacists it would be … exactly like much of America, as many have woken up to in the last decade. What is more important is what specifically links SV’s apparent rationalism to NRx attitudes. Given the way technologists’ dreams are increasingly shaping our future, we have a right to know what these dreams hold.

SV’s mythos and Rationalism start from the optimistic premise that technology and reason can fix the world, helping people understand what they have in common. NRx instead fixates on the misanthropic principle that democracy is always a ruse for the rule of the powerful and those who rule should be “worthy” (viz. brilliant CEO-godkings). Standard democracy is, in their minds, simply a cover under which the medioc...
Misguided? Callous? Just Plain Stupid?
by Enoch Root — 2021-08-20T06:48:22.897Z

Here's something I think about a lot: of these three traits, which is the least desirable? I have gone in circles for several years now and run the proverbial gamut, befriending people with these three traits (often several, actually.) I always ask myself: is it better to be kind and dumb or be intelligent, yet callous?

If I subscribed to the "agent" philosophy of personal relationships, I would most likely never befriend people of the third type₁, since they seldom have skills to offer (at least at first glance), interestingly enough, one of my better and "more useful" friends is both type one and two. This particular friend is politically misguided (to a frightening degree) and not the smartest, but he has an encyclopedic knowledge of automotive repair. Another automotive friend is a type one and two person, but can you guess who I go to first?

I don't usually want to deal with the headache of type two friends. Even though they may be more intuitively capable, they bring a lot of baggage (after all, I think something has to go wrong in your life to end up hopelessly angry at everything.) I spent a considerable amount of time dealing with these people working in the automotive sector while I was in college. On one hand, type twos are difficult to make happy and difficult to deal with, but I owe them some credit, since I had to fundamentally change how I interacted with them. Prior to hanging around with type twos, I was a bit of a pushover, the type of introverted kid that always apologize, even when I wasn't the source of the problem. After dealing with type twos for a year or so, I decided I was never going to apologize again. Not really, but I was done making apologies for the logistics company, my employer, or my peers. Interestingly, the type twos were very receptive to this change and almost immediately stopped condescending and began to treat me as a peer, rather than somebody they were obligated to deal with. If any young people take one thing away from this post, let it be this.

Regarding type one people, I have found that the best approach is to ignore the problem. Being a rural Midwesterner, I have encountered my fair share of type ones, particularly politically. Being misguided is, I think, a lesser flaw, since it is typically a result of rearing and exposure. Neither of these are the fault of the individual, that is, until they are presented with ...
Framing Practicum: Timescale Separation
by johnswentworth — 2021-08-19T18:27:55.891Z

This is a

framing practicum

post. We’ll talk about what timescale separation is, how to recognize timescale separation in the wild, and what questions to ask when you find it. Then, we’ll have a challenge to apply the idea.

Today’s challenge: pick 3 examples of equilibria from the

previous

three

practica

, and for each of them, give one use-case on a fast enough (or slow enough) timescale that we can treat the system as constant (or in equilibrium). They don’t need to be good, they don’t need to be useful, they just need to be novel (to you).

Expected time: ~15 minutes at most.

What’s Timescale Separation?

If I put a piece of iron in the ocean, its equilibrium state is rusted through. However, it takes a long time to reach that equilibrium. If I swim past and look at the piece of iron, I probably won’t even see it gradually becoming rustier.

At the timescale of a person looking at the piece of iron on the way by, its state is roughly constant, even though it’s out-of-equilibrium. The changes are so slow that we can ignore them.

On the other hand, consider pushing a wheelbarrow. The force isn’t transmitted to the wheelbarrow instantaneously - for a brief fraction of a second after I push on the handle, the handle actually compresses a bit, and the compressed handle presses on the wheelbarrow frame, which compresses the frame a bit, which then presses on the load… the compression propagates as a wave, transmitting the force through the whole wheelbarrow. And the wave also bounces back, exerting pressure from the load back on my hands. But from my point of view, this whole process happens extremely quickly. Within a fraction of a second, the waves have settled down to an equilibrium force (and matching acceleration) between my hands and the wheelbarrow.

At the timescale of a person pushing the wheelbarrow, the forces are always roughly in equilibrium. The force-propagation process is so fast that we can ignore it.

This is timescale separation:

If a system equilibrates on a timescale much slower than whatever-we’re-interest-in, then we can approximate it as being in a constant non-equilibrium state.
If a system equilibrates on a timescale much faster than whatever-we’re-interested-in, then we can approximate it as always being in equilibrium (even if the equilibrium changes slowly over time).

Note that a system may involve multiple processes which equilibrate on diffe...
Exploring the Landscape of Scientific Minds (Let my People Go) 
by rogersbacon — 2021-08-19T16:21:57.482Z

(Originally posted at Secretum Secretorum)

I.

Consider the following regions in the landscape of possible minds:

(1) All possible human minds

(2) All human minds that have ever existed

(3) All human minds that currently exist

(4) All currently existing human minds that participate in science

It should be clear that (1) is a miniscule patch of dirt in the vast mindscape, as (2) is of (1), and so forth. What scientifically productive regions of the mindscape (i.e. minds that could usefully contribute to scientific research) are found in (1), (2), or (3) but not (4)? What factors are limiting the diversity of scientific minds?

II.

In our current technological environment, nothing has influenced our psychology more deeply than computers, the internet, and related digital technologies. The internet is usually seen as an unequivocal good for the advancement of science, but I suspect that we do not fully appreciate the psychological costs, which may be small but compounding over the long run. I can think of at least three reasons why the internet might have negative effects on creativity and diversity in scientific thinking:

(1) There is too much knowledge available at one’s fingertips. This may have a subtle inhibitory effect on creativity, in essence making it harder for us to ever come at a problem with “fresh eyes”. Perhaps we are collectively spending too much time in the library of the internet like the very, very, smart guy that Richard Hamming (mathematician and former member of the Manhattan project and Bells Labs) discussed in a 1986 colloquium.

Question: “How much effort should go into library work?”

Hamming: “It depends upon the field. I will say this about it. There was a fellow at Bell Labs, a very, very, smart guy. He was always in the library; he read everything. If you wanted references, you went to him and he gave you all kinds of references. But in the middle of forming these theories, I formed a proposition: there would be no effect named after him in the long run. He is now retired from Bell Labs and is an Adjunct Professor. He was very valuable; I’m not questioning that. He wrote some very good Physical Review articles; but there’s no effect named after him because he read too much. If you read all the time what other people have done you will think the way they thought. If you want to think new thoughts that are d

...
Covid 8/19: Cracking the Booster
by Zvi — 2021-08-19T14:20:01.080Z

The news about Covid-19 is now essentially on a few distinct tracks: Vaccine effectiveness and booster shots, vaccination mandates, mandates for NPIs and other Covid-19 related crippling of the living of life, and the actual path of the pandemic. One could argue for splitting those further, or for combining the mandates.

Vaccine effectiveness against the spread of Covid has come into question, with many claiming that immunity fades dramatically with time or that the vaccines were never that effective against Delta. I’ve looked into the claims and go into detail. There’s little question that the vaccinated can spread Covid, which is in contrast to previous attempts to sell the line that they couldn’t at all, and they spread Delta somewhat more than they spread Alpha, the numbers here are disappointing relative to my expectations, but the big claims of hugely waning immunity are almost certainly greatly exaggerated.

Vaccines remain highly effective at stopping the spread of Covid-19, and of stopping symptomatic disease, and especially in preventing death. Still, boosters are more effective than not having boosters, and I think the cost-benefit for most people favors getting a third shot if one is made available to you. Boosters are coming soon, eight months after your second dose. However, until it is highly encouraged and likely eventually mandatory, it is still for the moment mostly forbidden. Get your shot exactly when we tell you to, they insist, not a minute before.

There are continuing debates over mask mandates at various meta levels, which led to some sentences that were fun (and tricky) to write. There’s increasing mainstream and in-group pressure to force people to wear masks regardless of whether it does anything useful in context, and to shame those who disagree and blame them along with the unvaccinated for the entire pandemic.

Oh, and also the pandemic is still growing but at a decreasing rate, and case counts are a favorite to mostly stabilize within a few weeks, which is great news. The death rate lags, so it continues to climb rapidly for now.

Let’s run the numbers.

The Numbers

Predictions

Nate Silver, the media’s official master of predictions, officially wouldn’t want to predict this thing. So even though he kind of does, officially he doesn’t. He observes the CDC models are split. Some models think this week is the...
What is difference between thoughts and consciousness?
by Olitness — 2021-08-19T11:41:05.428Z

Could there be thoughts without consciousness? Could there be consciousness without thoughts? If so, what would be consciousness conscious of without thoughts? What would thought be without consciousness, would it be just regular bit of information?
I know there are no answer to this, so far. But I would be curious about your, well, thoughts on this!
Bupropion and CBD oil 
by ChristianKl — 2021-08-19T09:14:48.869Z

Bupropion seems to be a great medication in many ways. Scott Siskind wrote about it as being on of the main mediations he uses to treat his patients. One of the side effects of Bupropion can be insomnia. CBD oil is often recommended online as a treatment for insomnia. When searching online, there are suggestions that there might be interactions between the two. What should we think about that interaction. Is it worrisome enough to avoid making the experiment of trying CBD oil when taking bupropion?