trentmkelly/LessWrong-43k huggingface dataset:
EA Hangout Prisoners' Dilemma
by Jeffrey Ladish — 2021-09-27T23:15:38.675Z

I ran the following experiment on Petrov’s day on the EA Hangouts Facebook page:

Hey everyone, it's Petrov day and I wanted to run a little scenario.

I'm going to donate $100 each to AMF and MIRI. Unless someone comments with "nuke MIRI" or "nuke AMF". In that case, I'll give people 20 minutes to respond by nuking the other org, and close the option to respond after that.

If neither orgs get nuked by midnight tonight, I'll give $100 to both orgs. If both orgs get nuked, they'll both get nothing. If only one org gets nuked, the nuked org will get nothing and the other org will get $300.

I'd prefer if no one nuked anyone, but I promise to honor the outcome of the experiment and donate accordingly. These are real donations that I wouldn't have made otherwise.

At 23:23, MIRI was nuked with no retaliation to AMF, and I donated $300 to AMF. Before this happened, there were a lot of interesting game dynamics and I learned more than I expected to!

I created this experiment because I always felt like Petrov day exercises didn’t involve much real incentive to launch an attack. The Cold War was not like this. There were real fears in both the USSR and the US that the other power represented an existential threat. After talking it through with some friends, I decided to go with a Prisoner’s Dilemma type game and see what happened.

Less than half an hour after I posted the original message, someone asked why anyone would ever counternuke. I replied that they might counternuke because they had precommited to do so, in order to deter someone from nuking their preferred organization. Immediately after this, they commented “nuke AMF”. Another commenter asked if it was possible to cancel an attack. I hadn’t thought of this possibility, so I decided that yes they could if they deleted or edited their comment within 20 minutes of posting it.

This led to an interesting dynamic. A few minutes later someone else commented “nuke MIRI” with a followup message that they would delete this comment if the original attacker deleted their “nuke AMF” comment. Both posts were then deleted by their authors.

This happened two more times, where someone made an initial launch post and someone else launched a counter attack, with a promise to abort the attack if the original attack was canceled. One of the attacks got very close to detonation, with ~10 seconds to spare.

Someone else posted a $100 bo...
[Book Review] "The Vital Question" by Nick Lane
by lsusr — 2021-09-27T22:50:28.299Z

There is a singularity at the beginning of biology. We know what happened before biology―that's just physics. We know what happened after the first cells: evolution. But we don't know how the first cells formed out of nonliving matter.

The most popular theory is called "primordial soup". The basic idea is that organic molecules (which just means chemical compounds with carbon-hydrogen bones) formed from natural physical processes and then some of those organic molecules randomly organized themselves into RNA and then that RNA built itself a cell. Bam.

The primordial soup theory is implausible from a physics perspective due to thermodynamics.

Living cells are at disequilibrium with their environment. They must harvest negentropy from an external source in order to stay alive. This isn't a property life happens to have. It's the essence of what life is. Life that isn't harvesting negentropy isn't alive.

For life to exist it needs a barrier between its ordered self and its chaotic environment. Life on Earth uses a cell wall composed of a lipid membrane. Since the inside of the cell is ordered and the outside is chaotic, there is an entropy gradient across the cell wall.

Here's where the primordial soup theory becomes implausible. A primordial soup is at chemical equilibrium. A cell is at chemical disequilibrium. Could a primordial soup form a cell? Sure, in the same sense that all the atoms in my body could quantum teleport themselves to the Restaurant at the End of the Universe. "Things randomly happened to violate thermodynamics" is so statistically unlikely it should be the last resort we fall back upon after every conceivable hypotheses fails.

The Vital Question by Nick Lane provide a theory of the origin of life that is physically plausible.

Proton Gradients

Cellular biology textbooks spend a lot of time on:

Metabolism via mechanisms like the Krebs cycle.

Channels which pump chemicals in and out of the cell.

I can imagine life without DNA. I can imagine life without mitochondria and without chloroplasts. (Most organisms don't have them.) I cannot imagine life without metabolism. Sustained metabolism requires a cell to pump chemicals in and out of an organism.

The most important chemical reaction is the synthesis of ATP. It would not be an exaggeration to say ATP powers life. ATP is manufactured by harvesting energy from a proton gradient across a membrane.

Memb

...
Reachability Debates (Are Often Invisible)
by J Bostock — 2021-09-27T22:05:06.277Z

Alexi and Beth are not Blues or Greens, but are talking about a conflict in a foreign nation between the Blues and the Greens. The Blues are the majority in both numbers and power, and have created a number of laws which restrict the cultural and religious traditions of Greens. In response, the Green minority commonly protest, a smaller number are civilly disobedient, and a tiny number have become dangerous terrorists.

Alexi: I wish the Blues would just stop oppressing the Greens. It's ridiculous the treatment that Greens have to put up with. If the Blues would repeal those ridiculous laws then they could begin to get along.

Beth: What do the Greens expect? If they keep disrupting blue society and committing terrorist acts then there's going to be retribution.

Alexi: That's absurd! Most Greens are just trying to live their lives, and those that aren't are only responding to to the awful situation they've been put in.

Beth: The Blues are being killed, they fear for their lives! If the Greens could behave better then maybe they could enter actual negotiations for better treatment.

(Yes this was a metaphor for the particular conflict you were thinking of, definitely not any of the dozens of other conflicts it could just as easily represent)

Clara and Dennis are discussing solutions to climate change.

Clara: Carbon capture or blocking out sunlight is how we can solve this, because any billionaire or government could do it!

Dennis: But that's ridiculous, we need to reduce emissions now! We've done alright with transferring centralized energy to low-carbon and renewable alternatives, the next step is to stop using cars and high-carbon materials so much.

Clara: If that's the case they need to be banned, or taxed!

Dennis: That won't be enough, we need to make individual changes! We need to change our attitudes!

These both have the same flavour.

What's going on is a question of reachability. Eliezer Yudkowsky has written about reachability and should-ness in terms of free will, with should-ness flowing backwards from desirable states of the future.

In both cases, the future state that the interlocutors want is pretty similar (Blue and Green integration and peace, less climate change), but they disagree on which actions are reachable. Instead though, this manifests as a disagreement about what "should" be done.

Alexi is seemingly taking a much more Pro-Green position, compared to B...
AISC5 Retrospective: Mechanisms for Avoiding Tragedy of the Commons in Common Pool Resource Problems
by kwiat.dev — 2021-09-27T16:46:40.389Z

Work by Quinn Dougherty, Ben Greenberg & Ariel Kwiatkowski

From the AI Safety Camp group that worked on

Cooperativity and Common Pool Resources

: a write-up of a problem we worked on, our proposed solution, and the results of implementing this solution in a simulated environment.

Contents:

Problem description
Review of related work
Experiment and results
Retrospective

Check out our GitHub repo

here

.

0. Abstract

When multiple parties share the access to a finite resource, they may overuse the resource leading to a Tragedy of the Commons. A simple way of mitigating this problem is allowing them to decrease the effective population by using violence - this, however, is not the best solution for obvious reasons. We study interventions for avoiding these outcomes in environments with multiple self-interested agents. In particular, a reputation system can incentivize agents to “cooperate” by harvesting the resource more sustainably. This system promotes multi-agent cooperation without modifying the agents’ reward functions.

1. Problem description: tragedy of the commons

A

common pool resource

(CPR) is a good which anyone can use but which no one can entirely control. When competition over the resource increases, individual parties are incentivized to appropriate as much of the resource as possible for themselves, which further increases competition, potentially leading to overconsumption. This process is commonly known as the

tragedy of the commons

, and has been extensively studied in economics and the social sciences.

A tragedy of the commons often leads to the exhaustion of a CPR. The classic example is of fishermen in an enclosed body of water; when fish are scarce, an every-man-for-himself dynamic emerges. Any available fish is quickly caught for fear that, if one hesitates, their counterparts would surely take it otherwise. This behavior results in the fish population failing to replenish, and everyone catching fewer fish in the long run.

Even if an individual would prefer to fish sustainably, they are punished for doing so unilaterally. This is apparent in the

prisoners’ dilemma

, where overfishing is analogous to “defection” and sustainable fishing to “cooperation”. In order to make cooperation a viable strategy, other participants must agree to follow suit.

Even if several parties do manage to coordinate, their incentives remain the same,...
AI takeoff story: a continuation of progress by other means
by Edouard Harris — 2021-09-27T15:55:44.163Z

Thanks to Vladimir Mikulik for suggesting that I write this, and to Rohin Shah and Daniel Kokotajlo for kindly providing feedback.

Prologue

This is a story about a universe a lot like ours. In this universe, the

scaling hypothesis

— which very roughly says that you can make an AI smarter just by making it bigger — turns out to be completely right. It’s gradually realized that advances in AI don’t arise from conceptual breakthroughs or sophisticated deep learning architectures. Just the opposite: the simpler the architecture,

the better

it turns out to perform at scale. Past a certain point, clever model-building was just slowing down progress.

Researchers in this universe discover a rough rule of thumb: each neural network architecture has an intrinsic maximum potential intelligence, or “capability”. When you train a network on a problem, how close it gets to reaching its potential capability depends on two limiting factors: 1) the

size and diversity

of its dataset; and 2) the amount of compute that’s used to train it. Training a network on a quadrillion games of tic-tac-toe won’t make it smart, but training a network on a

quadrillion-word corpus

of text might just do it. Even data cleaning and quality control don’t matter too much: as long as you have scale, if you train your system long enough, it learns to separate signal from noise automatically.

Generally, the more parameters a neural network has, the higher its potential capability. Neural nets with

simple architectures

also have a higher potential capability than nets with more sophisticated architectures do. This last observation takes the research community longer to absorb than you might expect — it’s a

bitter lesson

, after all — so the groups that

internalize it first

have an early edge.

Frontier AI projects begin to deemphasize architecture innovations and any but the most basic data preprocessing. They focus instead on

simple models

,

huge datasets

,

hard problems

, and

abundant compute

.

Initial progress is slowed somewhat by a global semiconductor shortage that

increases the cost

of running large GPU workloads. Within a year or so, though, this bottleneck clears, and the pace of advance accelerates.

Our story opens just as the world’s supply chains are getting back to normal.

It begins with Chinese content apps.

ByteDance

launches an experiment to auto-generate some of the articles on their Toutiao news a...
The Intense World Theory of Autism
by Steven Byrnes — 2021-09-27T13:23:39.780Z

The first section is well-written and reliable, because I didn't write it, it's a book excerpt. The rest of the post is more prone to errors and speculations; you can skip it if you're not into that kind of stuff—I read a few books and articles about autism, but I'm definitely not an expert, and I didn't spend a lot of time checking everything I wrote. Please comment if anything seems wrong (or dumb, or offensive, etc.) so I can fix it.

Summary / Table of Contents

The first section is mostly a book excerpt introducing my favorite theory of how autism works in the brain, namely "Intense World Theory".
Then, I'll flesh out how I'm thinking about that theory, by distinguishing "intense world" and "different learning algorithm hyperparameters" as things that happen in different parts of the brain and have different consequences, even if they tend to go together. I'll talk a bit more about each separately, and try to relate them more specifically and mechanistically to the algorithms going on in different parts of the brain.

I'll also include how I think this theory connects to other famous aspects/theories of autism, like cerebellar abnormalities, memory, "weak central coherence", etc.

At the end, I'll speculate that there's an exact-opposite "dim world theory of psychopathy". Fun!

Introduction to the Intense World Theory of Autism

There are various theories of the root cause of autism in the brain. I'll mention a few as I go. But I'll start right in with the positive case for the theory I like: The Intense World Theory of Autism.

The Intense World Theory of Autism dates to this 2007 article. I first heard about it from the excellent book The Myth of Mirror Neurons, which devotes a whole chapter to it, excerpted below. Then I read Temple Grandin's The Autistic Brain: Thinking Across the Spectrum and found that she also brought it up, and seemed very enthusiastic about it. I've also read a couple review articles on Intense World Theory, including this one by some of the inventors of the theory, and this one and a few others. Everything I've found is pretty positive the theory—please comment if you've seen any substantive criticism.

I can't dream of describing the theory as well as Gregory Hickok does in The Myth of Mirror Neurons, so I'll just excerpt from that. First some motivation:

…I'm going to suggest the possibility that the dominant neurocognitive theories of

...
A Confused Chemist's Review of AlphaFold 2
by J Bostock — 2021-09-27T11:10:16.656Z

(This article was originally going to be titled "A Chemist's Review of AlphaFold 2")

Most of the protein chemists I know have a dismissive view of AlphaFold. Common criticisms generally refer to concerns of "pattern matching". I wanted to address these concerns, and have found a couple of concerns of my own.

The main method for assessment of AlphaFold 2 has been the Critical Assessment of Protein Structure (CASP). This is a competition held based on a set of protein structures which have been determined by established experimental methods, but deliberately held back from publishing. Entrant algorithms then attempt to predict the structure based on amino acid sequence alone. AlphaFold 2 did much better than any other entrant in 2020, scoring 244 compared to the second place entrant's 91 by CASP's scoring method.

The first thing that struck me during my investigation is how large AlphaFold is, in terms of disk space. On top of neural network weights, it has a 2.2 TB protein structure database. A model which does ab initio calculations i.e. does a simulation of the protein based on physical and chemical principles, will be much smaller. For example Rosetta, a leading ab initio software package recommends 1 GB of working memory per processor in use while running, and gives no warnings at all about the file size of the program itself.

DeepMind has an explicit goal of replacing crystallography as a method for determining protein structure. Almost all crystallography is carried out on naturally occurring proteins isolated from organisms under study. This means the proteins are products of evolution, which generally conserves protein structure as a means of conserving function. Predicting the structure of an evolved protein is a subtly different problem to predicting the structure of a sequence of random amino acids. For this purpose AlphaFold 2 is doing an excellent job.

On the other hand, I have a few nagging doubts about how exactly DeepMind are going about solving the protein folding problem. Whether these are a result of my own biases or not is unclear to me. I am certainly sympathetic to the concerns of my peers that something is missing from AlphaFold 2.

Representations

One of the core elements is that the representation(s) of protein structure is fed through the same network(s) multiple times. This is referred to as "recycling" in the paper, and it makes sens...
The Trolley Problem
by lsusr — 2021-09-27T08:26:49.334Z

If you go to Harvard Law School, one of the first classes you'll take is called "Justice: What's The Right Thing To Do?" It starts with the trolley problem.

There's a trolley hurling down the tracks with no breaks. It's on its way to hitting and killing 5 people. You can pull a lever but if you do the trolley will kill 1 person instead. Do you pull the lever?

Students are asked "Should you pull the lever?"

Some say "Yes because utilitarianism should maximize human life."

Some say "No because the categorical imperative forbids murder."

They're both wrong. The question is underspecified. The right answer is "It depends."

You're in Germany in 1943. A train full of political prisoners is hurtling toward its doom. You can pull a lever to save their lives but if you do the train will hit Heinrich Himmler instead. Do you pull the lever?

Heck yeah!

You're a fighter in the French Resistance in 1943. A train full of SS officers is hurtling toward its doom. You can pull a lever to save their lives but if you do it'll hit your friend who is planting a bomb on the railroad tracks. Do you pull the lever?

Of course not.

Those hypothetical situations cheat the problem. The purpose of the trolley problem is to compare one positive value against another positive value. Let's do that.

You're a general in the Russian army in 1943. A train full of Russian soldiers is hurtling toward its doom. You can pull a lever to save their lives but if you do it'll hit a Russian soldier standing on the tracks. All the soldiers are of the same rank. Do you pull the lever?

Yes, obviously. Your job is to maximize the combat effectiveness of your fighting force.

You're a doctor in 2021. Do you murder a patient and harvest his organs to save the lives of 5 other people?

No, obviously. That would be monstrous. Even if it weren't, the negative externalities would overwhelm your proximate benefit.

If the answer to an ethical question is always the same, regardless of context, then you can reason about it as a context-free abstraction. But answers to ethical questions are usually context-dependent.

If I took the class I'd ask "What year is it? Where am I? Who am I? Who's in the trolley? Who's on the tracks? Who designed the trolley? Who is responsible for the brake failure? Do I work for the trolley company? If so, what are its standard operating procedures for this situation? What would my ...
Review: The End Is Always Near
by Elizabeth — 2021-09-27T07:40:02.355Z

The date is November 10th, 2019. Covid has plausibly started, but I don’t know it yet. I am a huge fan of Dan Carlin’s Hardcore History podcast, and have been conducting my own lit review on civilizational collapse. I have been eagerly anticipating Carlin’s upcoming book, The End is Always Near, for months (affiliate link). I am in a coffee shop with a friend, very excited to have a dedicated time to read and Epistemic Spot Check it.

I do not remember what I read. I remember that I lost all interest in Carlin’s podcast afterward, and was so sure I’d remember the problem that I didn’t write it down, which led to 2 years of awkwardly saying “yeah his book was so terrible I lost interest, no don’t remember why, yes I see how that’s less useful for you.” I never checked any claims it made; I’d have records of that, which means that whatever the problem was, it wasn’t just a factual error

I sat down today to read enough of the book to remind myself of why I so vehemently disliked it, and in the course doing so discovered that I had written down the problems in Goodreads, but had forgotten that along with everything else. (I also got the date wrong: I remember starting it in January, but that doesn’t fit because I know I was reading one of its sources in December). My review, in its entirety:

I went in wanting a meaty history book with many claims I could follow up on. In the first few chapters I could only extract a few claims, always what other historians thought (but without countervailing arguments), and it never coheres into models or cruxes.

Mystery solved, I guess. It’s not actually clear to me I should have given up on the podcast based on this, since I don’t remember it having the same problem. But since I already went through all this trouble, let me read a chapter or two and see if I agree with my pre-covid assessment.

Claim: “In many earlier eras of history writing, a large part of the historian’s or author’s goal was to impart or teach some sort of moral lesson, usually by historical example.” (footnote on page 1)

Ah yes, the before times, when people manipulated nominally factual data to their own ends. So glad we grew out of that in … *checks watch* … hmmm, must be broken.

Claim: Sparta super kicked ass (page 7)

Bret Devereaux spent a long time debunking this and I spent a somewhat shorter time checking his work (it passed). Carlin also rep...