trentmkelly/LessWrong-43k huggingface dataset:
An Exploratory Toy AI Takeoff Model
by niplav — 2021-01-13T18:13:14.237Z

cross-posted from niplav.github.io

In AI safety,
significant time has been spent on the question of
the intelligence of AI systems over time, especially during
takeoff.
An underappreciated argument in the debate has been the idea that the more
intelligent an AI system becomes, the better it can search the space of
possible optimization algorithms. This post proposes a computational model
of this process by creating an n-dimensional search space and then running
a very simple hill-climbing
algorithm and brute-force search on that space. Possible further
improvements to the model are suggested, and implications are discussed.

An Exploratory Toy AI Takeoff Model

Libre de la metáfora y del mito

labra un arduo cristal: el infinito

mapa de Aquel que es todas Sus estrellas.

– Jorge Luis Borges, “Spinoza”, 1964

Paraphrasing Roache (2008) the state of play is such that nobody
believes the result of a simulation, except the person who performed
the simulation, and everybody believes the result of an experiment,
except the person who ran the experiment.

– Ryan G. McClarren, “Uncertainty Quantification and Predictive Computational Science“ p. 9, 2018

(Although the quote apparently goes back to Einstein, see “The
advancement of science, and its burdens” p. 13, only there with "theory"
instead of "simulation").

And let me just make an aside. There’s a lot of meta-debate that goes
on in the AI safety community which I don’t understand why: it’s not
as if we haven’t got enough real work to do. So now we have meta-debates
about whether you should focus on short-term or long-term, or whether
we should try to reduce the conflict between the short-termers and the
long-termers and it’s like, there doesn’t need to be a conflict.

– Stuart J. Russell, “The flaws that make today’s AI architecture unsafe and a new approach that could fix it” (Episode 80 of the 80,000 hours podcast), 2020

Introduction

Many regard the trajectory of future AI development as
crucial: when will AGI first be
developed? Will the development be slow, moderate, or fast (in
economic doublings, or in wallclock time)? Will one AGI system become a
singleton,
i.e come to dominate the whole world (individual vs. collective
takeoff)? Will AGIs FOOM, i.e growing unexpectedly
fast? And: will there be one or more discontinuity when AI systems
recursively self-improve? This text attempts to shine light on the las...
[AN #133]: Building machines that can cooperate (with humans, institutions, or other machines)
by Rohin Shah — 2021-01-13T18:10:04.932Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

Open Problems in Cooperative AI (Allan Dafoe et al) (summarized by Flo): Cooperation can often lead to better outcomes for everyone involved and some progress in AI, like improved machine translation or smart contracts, can make cooperation easier. On the other hand, as AI agents will likely become more and more important, it is crucial they can cooperate with each other and with humans. The term cooperative AI combines these perspectives and refers to "AI research trying to help individuals, humans and machines, to find ways to improve their joint welfare". While previous AI research has focused on individual intelligence of artificial agents, more work on social intelligence and cooperative capabilities is needed. The paper provides an overview of research relevant to cooperative AI from a wide range of disciplines and aims to facilitate more interdisciplinary conversations about cooperative AI by providing a common framework and vocabulary.

Research on cooperative opportunities, situations in which gains from cooperation are possible, can differ along four major dimensions:

1) How much do interests overlap or conflict?

2) What kind of agents are involved? (Humans, machines, organizations)

3) What perspective is taken: the perspective of an individual trying to cooperate or a social planner trying to incentivize cooperation?

4) What is the scope and how interdisciplinary should research be?

Four types of capabilities can be crucial for cooperation:

1) Understanding: In cases where cooperation is optimal for all parties, with no incentive to deviate, but agents lack understanding of either the environment or one another, they may still fail to reach a cooperative equilibrium. For example, one agent might have false beliefs about the others' beliefs and preferences and thus their incentive for defection. This is particularly hard to get around, because (1) preferences might not be defined explicitly or m...
Any examples of people analyzing/critiquing scientific studies or papers?
by warrenjordan — 2021-01-13T18:00:09.407Z

I'm having trouble understanding how people assess scientific papers as I'm a layman in reading them and statistics overall.

My general (and probably too simplistic) heuristics are:

"Is the p value significant?"

"Is there a large sample size? Or small?"

"Is the population studied generalizable?"

When I read the comments in the post I hyperlinked, it doesn't seem to click for me. I need more detailed examples. Perhaps someone taking a deep dive into one specific study and picking it apart, from the introduction to methods to discussion and ending at the conclusion. Mentioning what they did right, what they did wrong, etc.

Does anyone know of any I could reference? I'm specifically reading positive psychology papers if that helps.
What’s good about haikus?
by KatjaGrace — 2021-01-13T08:10:15.311Z

Fiction often asks its readers to get through a whole list of evocative scenery to imagine before telling them anything about the situation that might induce an interest in what the fields and the flies looked like, or what color stuff was. I assume that this is fun if you are somehow more sophisticated than me, but I admit that I don’t enjoy it (yet).

I am well capable of enjoying actual disconnected scenery. But imagining is effort, so the immediate action of staring at the wall, say, seems like a better deal than having to imagine someone else’s wall to be staring at. Plus, a wall is already straining my visual-imaginative capacities, and there are probably going to be all kinds of other things, and some of them are probably going to be called exotic words to hammer in whatever kind of scenic je ne sais quoi is going to come in handy later in the book, so I’m going to have to look them up or think about it while I keep from forgetting the half-built mental panorama constructed so far. It’s a chore.

My boyfriend and I have recently got into reading haikus together. They mostly describe what things look like a bit, and then end. So you might think I would dislike them even more than the descriptive outsets of longer stories. But actually I ask to read them together every night.

I think part of it is just volume. The details of a single glance, rather than a whole landscape survey, I can take in. And combined with my own prior knowledge of the subject, it can be a rich picture. And maybe it is just that I am paying attention to them in a better way, but it seems like the details chosen to bring into focus are better. Haikus are like a three stroke drawing that captures real essence of the subject. My boyfriend also thinks there is often something clean about the images.

Some by Matsuo Bashō from our book The Essential Haiku, edited by Robert Hass:

In the fish shop
the gums of the salt-bream
look cold

Early fall—
The sea and the rice fields
all one green.

Another year gone—
hat in my hand,
sandals on my feet.

More than ever I want to see
in these blossoms at dawn
the god’s face.

The peasant’s child,
husking rice, stops
and gazes at the moon.

Year after year
On the monkey’s face
A monkey face

In sum:

Ten words
Draw on my mind
Cleaner than fifty lines
The impact merge
by Joe Carlsmith — 2021-01-13T07:26:42.605Z

(Cross-posted from Hands and Cities)

Lots of people want to do big things — start a big company, write a bestselling book, participate in an important project. I’ll call this the “accomplishment desire.”

Often this is centrally tied to social status; the relevant type of “bigness” is highly correlated with what would seem cool at, e.g., a college reunion. But it also mixes in richer values. Maybe someone wants to be a famous musician; but they also often want their music to be good, and expressive of who they are.

This desire is also tied to meaning — to wanting to have done something, mattered, lived a life of consequence. Accomplishments are the types of things you look back at on your deathbed; they’re your legacy. (My sense is that having children is often central here, even if this doesn’t read as traditionally “big”).

Lots of people also want the world to be as good as possible. They want there to be no disease, injustice, poverty, war; they want love and joy and beauty. Call this the “good world desire.”

The accomplishment desire and the good world desire need not mix, even where they co-exist: many people who want to start big companies aren’t centrally motivated by a desire to do good, and wanting to make good music isn’t the same as trying to make the world better through music — music has its own standards. People generally want the big things they do to be good for the world (few seek “big” legacies of harm), but that doesn’t mean they’re aiming for bigness on the “good for the world” dimension.

Still, it’s natural to attempt a merge: e.g., to hitch the accomplishment desire to the good world desire, into an overall aim of trying to improve the world as much as possible. Let’s call this the “impact merge.”

My question is whether the impact merge is a good idea. It might seem so — if you’re aiming to do big things, why not aim for big improvements to the world? But I think there are considerations on both sides.

My basic concern is that the accomplishment desire is, at bottom, about you, whereas the good world desire is about the world. So I worry that the two, hitched together, will not always pull in the same direction.

This is easy to see in contexts where they literally recommend different actions. Suppose, for example, that your desire for accomplishment is in fact deeply tied to some sort of desire for social status. In that case, you might turn down...
"If" is in the map
by Chris_Leong — 2021-01-13T03:09:09.830Z

Update: See the comment by philh. These logical operations are matters of convention, but the way that it is defined is less arbitary than I thought!

I want to start this sequence with a very basic example - the word "if".

I expect that most people here already know that there are two definitions of "if":

Logical if: "If A then B" is true is equivalent to "B OR NOT A"
Counterfactual if: "If A was true, then B" means consider a world or a set of worlds like ours, but with A being true. In all these worlds, B is true as well".

In this post we'll discuss the logical if and counterfactuals will wait until another post. Someone might naively think that this logical if is in the territory, but it is actually in the map.

Consider the sentence, "If pigs can fly, then Trump is Queen of England". Both components are false and when this occurs, the if-statement is considered true.

However, this is purely a matter of convention. There's no reason why we couldn't consider these sentences where the condition is false to be false or undefined. It's a mistake to ask why this is the case as though it were necessarily the case, but not a mistake to ask why we chose for it to be the case.

These statements being false isn't part of the territory (of relations between propositions), but is rather a result of how we've drawn our map (of relations between propositions).

Note that I haven't said what relations between propositions are - whether these are abstract objects or purely exist in our minds. But whatever they are, "if" is in the map of relations between propositions.
Voting Phase for 2019 Review
by Raemon — 2021-01-13T01:33:03.791Z

Click

here

to begin voting. Click here for a general overview of the 2019 review.

We are now in the final stretches of the LessWrong 2019 Review. We’ve spent 2 weeks nominating posts, and a month reviewing them. And today, we begin the final vote!

A recap on the major goals of the Review:

First, it is an experiment in improving the LessWrong community's longterm feedback and reward cycle.
Second, it is an attempt to

build common knowledge

about the best ideas we've discovered on LessWrong.
Third, after the vote, the LessWrong Team will compile the top posts into a physical book. (We may make some judgment* calls about what to include, but will use the vote as a strong guideline)

Voting

starts today (January 12th) and continues through Jan 26th.

Who can vote?

All users registered before 2019 can vote, but the LW curation team will primarily be paying attention to the votes of users with 1000+ karma.

(There will essentially be a “top users’ vote”, and a “people’s vote.” The results of both will be made public, but the longterm-users’ vote will be the main thing influencing the Best of 2019 Book)

Anyone can still write reviews

For all users and lurkers, regardless of karma, the next 2 weeks are your last opportunity to write reviews for any nominated posts in 2019, which can influence how people vote. As you can see below, all reviews are highlighted when a user is voting on a post. (To review a post, go to the post and click "Write A Review" at the top of the post.)

Posts need at least 1 review to appear in the vote. You can still review previously un-reviewed posts to put them on the ballot. (But, people who vote early might not see them)

How do I vote?

To vote, head over to

lesswrong.com/reviewVoting

.

The vote has a simple first section, and a detailed-yet-optional second section based on

quadratic voting

.

Sorting Posts Into Buckets

The first part of voting is sorting the nominated posts into buckets.

The five buckets are: No, Neutral, Good, Important, Crucial. Sort the posts however you think is best.

The key part is the relative weighting of different posts. For example, it won't make a difference to your final vote if you put every post in 'crucial' or every post in 'good'.

Fine-Tuning Your Votes

Once you're happy with your buckets, you can click 'Convert to Quadratic'. At this point the system converts your buckets roughly into their quadratic e...
#2: Neurocryopreservation vs whole-body preservation
by mingyuan — 2021-01-13T01:18:05.890Z

This is post 2 of 10 in my sequence on how to sign up for cryonics.

Background

There are currently two options for human cryopreservation: preserving the whole body, or preserving only the brain (and the head that holds it). Note that while cryonics providers Alcor and KrioRus offer both options, the Cryonics Institute only offers whole-body preservation.

This 1995 essay

by a former Alcor president does a good job of laying out the considerations that go into this decision. I'll go into many of those as well as some additional considerations in this post. I'm primarily looking at:

The practical considerations of preserving a body vs just a brain, including complexity, risk, cost, and preservation quality.
The biological and philosophical implications for one's personhood if the body is discarded.

Note that many people argue that it will be easier for whole-body patients to resume their lives and be reintroduced into society – perhaps so much easier that they'll be woken up hundreds of years before neuropatients. I am not going to explore those arguments here because I think it's unlikely that human bodies will be necessary/useful at the time of waking. If you're interested in a discussion of that topic, see Mati Roy's "

How much harder is it to revive a neuro-only cryonics patient?

"

Arguments for neuropreservation

From

Alcor’s FAQ

: "Brains are compact, inexpensive to store, easy to move, and are a single organ for which cryopreservation protocols can be completely optimized." That is, if you focus on preserving just the brain, you can do a better job with the initial cryopreservation, and maintain preservation less expensively, and with less risk, for the long years until revival.

Higher-quality preservation

There are reasonable arguments that neurosuspension might result in better preservation than whole-body, and this is a crux for me. However, note that these arguments rest on facts that only happen to be true today. If and when cryonics technology progresses to a point where a person can be preserved indefinitely with no damage whatsoever (some people call this 'true suspended animation'), then this consideration will entirely disappear.

There are many advantages when the perfusion team can exclusively focus on the head, because cryopreservation protocols can be completely optimized for survival of the brain[1]. Some specific advantages include:

Shorter perfusion tim

...
How much harder is it to revive a neuro-only cryonics patient?
by Mati_Roy — 2021-01-12T23:24:45.963Z

Format: I choose the question format to allow other people to provide their own answer. I write my answer as a post (ie. in the description of the question) given I think it's the quality of a post. This is a norm I would personally like to see adopted.

Confidence: speculative; good probability that others could update my model significantly | Created: 2019-01-18 | Updated: 2021-01-12 (major update; basically written then)

Context: I wrote this in the context of the cryonics signup sequence to complement the article "2: Neurocryopreservation vs whole-body preservation" which I've reviewed pre-publication.

When considering the question of neuro-only vs whole-body preservation, I evaluate the following 3 aspects:

a) Part of identity preserved in the body

b) Difficulty of creating and connecting a body to a brain / mind

c) Practical considerations or preserving a body: facility to move to and out of the facility, and cost, as well as impact on the preservation of the brain

This post will be about (b). For (a) and (c), check out part 2 of the cryonics signup sequence.

Reviving a neuropatient as opposed to a whole-body patient requires 2 extra steps: building a body and connecting it to the mind.

Note: It’s also possible to preserve a whole-body while still focusing on preserving the brain first, possibly even while still having a neurosperation.

On building a body

We know building a body is doable, as nature already does it. While we don’t have a particularly advanced understanding of the human genome, we know it encodes the instructions to build a human body. As our understanding of the genome increases, along with our ability to edit it, we should be able to adjust the genome to create a body without a brain, and to have the body grow to an adult size within the womb. Of course, the womb would need to be artificial.

Another plausible path is that we build a virtual body embedded in a virtual world. To some, it might seem like this would cheapen the experience. But ultimately, our brain interfaces the world only through sensors — so whether the sensors’ input are coming from the photons and atoms of nature or from a computer would produce indistinguishable experiences. And you could still share that virtual environment with your loved ones.

On connecting a mind to a body

An advanced AI might be able to determine which nerves served which purpose, and be able to reconnec...
Zen and Rationality: Karma
by Gordon Seidoh Worley — 2021-01-12T20:56:57.475Z

This is post 6/? about the intersection of my decades of LW-style rationality practice and my several years of Zen practice.

In today's installment, I look at karma from a rationalist perspective.

Karma is one of those ideas that is poorly understood both because it means different things in different contexts and because there's a simple but bad abstraction of the idea that's easily understood and shared but points to something other than what we mean by "karma" in Zen, thus leading to bucket errors.

So, first, what is karma not. The Zen notion of karma is not any of the related ideas also called "karma" present in other dharmic traditions. It's also not the New Age or Theosophy notion of karma. All of these you are probably familiar with and might be expressed in one of the following ways:

what you give is what you get
the world will punish you for your bad acts and reward you for your good acts
you'll be rewarded/punished in proportion to your acts in this life via the quality of your rebirth into a next life
karma is a "sin debt" you carry forward, possibly from past lives, that you must work off to have a better life

In Zen (and some other Buddhist traditions), karma is nothing so involved. Instead, you can just read it as "causality" and you'll be 80% of the way of understanding it, with the remaining 20% some subtlety around how causal relationships relate to the fundamental flowing of one moment into the next that come from understanding how map and territory relate.

As you might guess, the other notions of karma relate to this one as heuristics about how earlier moments affect later ones. For example, the thread of moments out of which a person can be identified contain within them all the causes of the person's present condition, and it's often true that if you behave prosocially towards others they'll behave prosocially towards you, so a reasonable compression of karma that skips all the finer points look a lot like "if you do good stuff, good stuff will happen to you; if you do bad stuff, bad stuff will happen to you". Mix in some beliefs about reincarnation and you have karma as it is commonly encountered, bucket errors and all.
AI Alignment Using Reverse Simulation
by Sven Nilsen — 2021-01-12T20:48:40.211Z

A simulation is often used to describe a computer program imitating a physical process. In this post, I use the term "reverse simulation" to refer to a scenario when a computer program is simulated using physics.

The approach I am suggesting here, is meant to address one problem of AI alignment that belongs to the philosophy of mind:

How can we prevent AGI from accidentally causing suffering of minds, when we do not yet know an objective theory of minds?

It is not evident to me that suffering of minds can be assigned a utility. Of course, assuming one could assign a utility, there might be a correct decision theory. However, if suffering of minds can not be accurately described, then there is some philosophical unsoundness that follows from an inaccurate description.

The basic idea is to use two steps for aligning AGI:

Make the AGI produce a computer simulation of an environment with strong safety properties, which carries low-risk side-effects of producing suffering of minds
Make the AGI use reverse simulation of the computer program, in a context such that interference with other side-effects is low-risk

In this approach, I assume AGI can be used safely to do step 1 and 2.

However, due to the lack of an objective theory of minds, I have no proof that this approach is safe. The strongest property of safety I believe is provable, assuming a very strong theorem prover is built in the future, is that the AGI is aligned with a theory of a relatively safe environment regarding unintentional suffering of minds.