trentmkelly/LessWrong-43k huggingface dataset:
EfficientZero: How It Works
by 1a3orn — 2021-11-26T15:17:08.321Z

The goal of this essay is to help you understand EfficientZero, a reinforcement learning agent that obtains better-than-human median performance on a set of 26 Atari games after just two hours of real-time experience playing each game.

Specifically, it gets 116% of human median performance on the data-limited Atari 100k benchmark. The previously-best algorithm only reached about 41% of median human performance, so this is a reasonably large leap.

Chart stolen from paper

The benchmark is called 100k because agents only interact with the environment for 100,000 steps -- about two hours. Note also that the human benchmarks were also set after the humans in question had about two hours of experience on the game. So EfficientZero seems to -- at least on this set of games -- exceed humans in sample efficiency specifically.

This is particularly impressive when you recall that, going into this, the agents were entirely ignorant of anything whatsoever about the world. The networks of their brains were initialized with random values. Humans manage to do pretty well on Atari after two hours, but we've pretrained in the actual world for many years, which lets us apply analogies from this world, experience from other video games, and so on. This agent managed to do comparably well with none of these advantages.

(Granted, the 100k benchmark focuses on Atari environments which are relatively easy to make progress in, because it was meant to be used for sample-efficiency benchmarks. It excludes extremely-difficult-to-explore environments like Montezuma's Revenge, where the first reward is quite hard to get.)

So. After reading this, you should understand how EfficientZero works, and how the changes in EfficientZero improve upon its predecessors. You should also know what further improvements to it are likely in the coming 3 to 24 months.

My target reader is someone who can program, or is at least broadly technically literate. They should be reasonably familiar with the basic principles of supervised machine learning, but they do not need to be particularly up-to-date with reinforcement learning. Reading this will nevertheless be heavy going if you aren't at least a little familiar with reinforcement learning; expect to have to reread some stuff.

I have aggressively and in some cases silently cut content that is not necessary for understanding the contributions of EfficientZ...
Why don't our vaccines get updated to the delta spike protein?
by ChristianKl — 2021-11-26T10:43:59.779Z

Delta and Delta Plus evade the antibody response:

Delta and Delta Plus were less well inhibited (neutralized) by antibodies from infected and vaccinated individuals as compared to the original virus and this likely contributed to the rapid spread of Delta.

It seems to me relatively straightforward that if we would update the vaccine to the spike protein of the delta varient that would help with protecting against it. When talking to people on Facebook I heared strange arguments such as the "vaccine working well" against delta with seems like a really bogus statement given the need for booster shots and many vaccinated spreading the virus. Yes, the vaccine helps but that's no reason not to update it to make it help more.

Can anyone make sense of this?
Rapid Increase of Highly Mutated B.1.1.529 Strain in South Africa
by dawangy — 2021-11-26T01:05:49.516Z

I'm sorry to ruin your holidays. The video above is my only source for this information. I will summarize the key findings here.

A cluster of cases among students in Gauteng was recently sequenced. A new variant, B.1.1.529 was found. The mutation profile of this variant looks horrendous:

Mutation Profile of B.1.1.529

Each of the vertical black lines is a mutation from the original strain. For comparison, here is what the Delta variant looks like:

Mutation Profile of B.1.617 (Delta)

Some of these mutations have known phenotypic impact (effect on transmissiblity and immune evasion), while others are brand new.

As you may be quick to point out, more mutations does not equal bad. However, the proportion of sequenced cases comprised of this new variant is rapidly rising:

Viral Proportions in South Africa

One might be tempted to say that this just the founder effect from the new cluster in Gauteng. However, while there has not been much full sequencing in other provinces, we can see that the lack of the "S-gene" in PCR tests, which the authors here are using as a signal that an infection may have been caused by this variant, is in fact rising rapidly in most provinces of South Africa.

Given that this is the most mutated strain of the coronavirus yet, we may be in for a rough time. The C.1.2 variant, seen in the image "Variant proportions in South Africa" above, was the strain with the most mutations until recently. C.1.2 never really took off the way that B.1.1.529 is taking off, though.

I expect that we will be hearing about this variant more. I think there is strong reason to believe that the high number of mutations in the spike protein will have a very negative impact on vaccine efficacy against this variant.
Latacora might be of interest to some AI Safety organizations
by NunoSempere — 2021-11-25T23:57:04.881Z
Link: https://www.latacora.com/

Am I affiliated with Latacora?

No, not in any way.

What is Latacora?

From their own webpage:

Latacora does just one kind of engagement: we join your engineering team virtually, create a security practice, and run it. As your company matures, we'll make sure that practice also matures. When it makes sense for you to bring some (or all!) of our security capabilities in-house, we'll help you make those hires.

Why might some AI Safety organizations be interested in Latacora?

Security is hard. Only amateurs roll their own security. But identifying good security hires and practices is hard if you don't know what they are or how they look like. This is similar to how it's difficult to hire for taste if you don't have taste. In particular, I'd expect top AI Safety hires to be great at AI Safety, but not necessarily great at security practices. But even if they are great at security practices, it's probably not their comparative advantage.

Recently, Yudkowsky has been talking about third countries stealing dangerous AI models and running them as a consideration feeding into his own pessimism. This doesn't seem like a hypothetical concern. Latacora seems like it would allow one to solve or mitigate this problem by throwing money at it.

How did you hear about Latacora? Why do you think they're good?

I don't remember how I heard about them. It was probably mentioned by someone grungier than me in some Linux forum, and I've been following their blog for a while. I think they're good because of the same reasons I think Zvi is good: because I read their stuff and they seem to say sensible things according to my judgment.

Are you suggesting that AI Safety organizations literally hire these people?

Yes.
Christiano, Cotra, and Yudkowsky on AI progress
by Eliezer Yudkowsky — 2021-11-25T16:45:32.482Z

This post is a transcript of a discussion between Paul Christiano, Ajeya Cotra, and Eliezer Yudkowsky on AGI forecasting, following up on Paul and Eliezer's "Takeoff Speeds" discussion.

Color key:

Chat by Paul and Eliezer	Chat by Ajeya	Inline comments

8. September 20 conversation

8.1. Chess and Evergrande

[Christiano][15:28]

I still feel like you are overestimating how big a jump alphago is, or something. Do you have a mental prediction of how the graph of (chess engine quality) vs (time) looks, and whether neural net value functions are a noticeable jump in that graph?

Like, people investing in "Better Software" doesn't predict that you won't be able to make progress at playing go. The reason you can make a lot of progress at go is that there was extremely little investment in playing better go.

So then your work is being done by the claim "People won't be working on the problem of acquiring a decisive strategic advantage," not that people won't be looking in quite the right place and that someone just had a cleverer idea

[Yudkowsky][16:35]

I think I'd expect something like... chess engine slope jumps a bit for Deep Blue, then levels off with increasing excitement, then jumps for the Alpha series? Albeit it's worth noting that Deepmind's efforts there were going towards generality rather than raw power; chess was solved to the point of being uninteresting, so they tried to solve chess with simpler code that did more things. I don't think I do have strong opinions about what the chess trend should look like, vs. the Go trend; I have no memories of people saying the chess trend was breaking upwards or that there was a surprise there.

Incidentally, the highly well-traded financial markets are currently experiencing sharp dips surrounding the Chinese firm of Evergrande, which I was reading about several weeks before this.

I don't see the basic difference in the kind of reasoning that says "Surely foresightful firms must produce investments well in advance into earlier weaker applications of AGI that will double the economy", and the reasoning that says "Surely world economic markets and particular Chinese stocks should experience smooth declines as news about Evergrande becomes better-known and foresightful financial firms start to remove that stock from their portfolio or short-sell it", except th

...
Covid 11/25: Another Thanksgiving
by Zvi — 2021-11-25T13:40:00.787Z

For my first Thanksgiving post, which somehow was both a year ago and almost a year into the pandemic, I gave thanks at the end of the post. With all that’s happened, it seems more appropriate to do this at the beginning of the post instead this year. So I’m going to do that, and deal with this week’s news – the rise in cases, the lockdowns in Europe and all that other stuff – later.

I’ll be working from the previous version, and including almost everything from last year, plus some additions. It is crazy how little this needed to be modified after an entire year, and how many of these are still in the present tense when they could so easily have been in the past tense. Time goes by, so slowly. Keep pushing that rock up that hill, everyone.

So here we go, 2021 edition.

To all the health care workers. Thank you.

To all those maintaining the supply lines. Thank you.

To Ryan Peterson, who worked to get our containers moving again, even if it didn’t end up having as big an impact as we might have hoped. Let this inspire future efforts across the land. Thank you.

To those who worked to manufacture the vaccines as quickly as possible. Thank you.

To Pfizer, Moderna, Johnson & Johnson and AstraZeneca/Oxford. Thank you.

To all those working on all the other vaccine candidates, even if they didn’t work out. Including Sputnik in Russia, and the one in China. And anyone working on all the other treatments, or running any of the experiments or studies, again whether or not your particular effort paid off. Thank you.

To all those working on or expanding capacity for or administering or fighting for the right to do Covid-19 testing, especially rapid testing. Thank you.

To all the essential workers, no matter whether we still call you that. Thank you.

To you, the reader, for being here. Thank you.

To all the commenters, yes all of them. Thank you.

To those who subscribed (Substack version of this post is here). Thank you.

To all those who have thanked me for doing these columns. You keep me going. Thank you.

To everyone who helped push masks, better masks, Vitamin D, zinc, fluvoxamine, airborne transmission, doing things outdoors, proper ventilation and other key information when official sources were saying otherwise. Thank you.

To Robin Hanson in particular, without whom I would not have felt free to start writing these columns. Thank you.

I’d also single out Tyler Cowen a...
Coordinating the Unequal Treaties
by lsusr — 2021-11-25T10:47:01.581Z

You captain a ship. You used to be a pirate. That was before there was the state monopolized the use of force on the high seas. You became a merchant. Then a privateer. Now you're a naval officer.

Roll a D4.

You're American.

You're British.

You're French.

You're Russian.

The Age of Discovery is drawing to a close. Exploration is making way for exploitation.

You arrive on the shores of Japan. Japan is a backwards land inhabited by violent, rude, noisy people. It is your moral imperative to bring them the enlightened values of Science, Rationalism and Christianity.

Japan refuses to talk. You land marines in Edo. Japan agrees to talk. What concessions do you demand of the Japanese government?

The Treaty

Your job is to maximize American/British/French/Russian interests. Japan will attempt to play the Western powers against one another. A united front can negotiate better terms than a divided front.

If you can coordinate with the other Western powers then you can obtain a treaty that maximizes American/British/French/Russian interests at the expense of Japanese interests. If you cannot coordinate with the other Western powers then you will get a treaty that maximizes Japanese interests at the expensive of American/British/French/Russian interests.

In order to coordinate against Japan, the Western powers must agree on what to demand. Unfortunately, the Western powers are mortal enemies.

The British fought the Americans in the Revolutionary War of 1776 and in the War of 1812. The French fought the Russians in a different War of 1812. The British have been fighting the French since the Norman Conquest of 1066. The British (including the Americans, who were British at the time) and the Prussians fought everyone in the Seven Years War.

If you fail to maximize your nation's interests then you might not be granted extraterritoriality. Without extraterritoriality, your nation's right to break quarantine might not outweigh Japan's desire to keep 100,000 of its civilians from dying of cholera.

Most Favored Nation

How do you coordinate with your enemies to present a united front against the Japanese in negotiations? You write a smart contract.

Specifically, you demand that any concession a rival power negotiates with Japan is automatically granted to your nation too. For example, if the Japanese agree to impose no import tariffs on German goods then no import tariffs will be imp...
First Strike and Second Strike
by lsusr — 2021-11-25T09:23:05.559Z

First-strike weapons destroy your enemy's ability to retaliate. Second-strike weapons preserve your ability to retaliate after a first strike.

Missile defense systems constitute first-strike technology. Suppose you build an effective[1] missile defense system. You could launch a pre-emptive nuclear strike against your enemies. If your enemies are growing stronger relative to you then you are incentivized to launch a nuclear strike against them, before they can build their own missile defense system.

It is impossible to defend a missile silo against a nuclear first strike because the position of a ground-based missile silo is fixed and few things provide adequate armor against a nuclear warhead. Airfields for bombers are even harder to hide. Submarines are relatively difficult to locate and destroy. Submarines are too small to launch a massive, overwhelming first strike. Submarines are second-strike technology.

First-strike weapons promote instability because they incentivize actors to strike first. Second-strike weapons promote stability because they make it safer for actors to wait. The more you can wait the less likely you'll be to gamble everything on a desperate first strike or to retaliate against false alarms.

This doesn't mean we should put all of our doomesday devices on submarines. That's a single point of failure. A diverse arsenal of delivery systems is more robust than relying on a single technology. China's submarines are (for now) less advanced than Russia's and the United States'. If China put its nukes on submarines it is plausible that its enemies could wipe them all out in a pre-emptive strike. This may be related to why China is currently expanding its ground-based nuclear missile capacity.

A world of second-strike technologies tends to incentivize peace. A world of first-strike technologies tends to incentivize pre-emptive strikes.

Perfectly effective missile defense is implausible against a peer adversary (excepting a successful pre-emptive cyberattack), but we can imagine a world where the United States deploys a successful missile defense system against a non-peer adversary against North Korea. ↩︎
You are way more fallible than you think
by Shmi — 2021-11-25T05:52:50.036Z

Yes, you. (And me, of course, including everything I write here.)

Epistemic status: annoyed, a quick writeup.

TL;DR: And therefore you should be very suspicious of any personal probability estimates that are near or below your fallibility level: they are complete junk.

Most of us are very much uncalibrated. We give 80%+ chance of success/completion to the next project even after failing a bunch of similar ones before. Even those of us who are well calibrated are still bad at the margins, where the probability is low (or, equivalently, high). Events we give the odds of 1% to can happen with the frequency of 20%. Or 0.01%. Remember, you are an embedded agent, and you only have a limited access to your own source code. Some examples:

Pascal's mugging:
"Give me five dollars, or I'll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills 3^^^^3 people."
The usual resolutions are the discussions of bounded utility and such, while a fallible-self approach advocated here would imply that you as an agent are too noisy to make reliable calculations, and shouldn't even try.
Arguments of the sort "what if God parted the sky and appeared to you in person, would you believe then?"
Well, it helps to take a third person perspective there. If someone told you it happened to them, what would be your reaction? Probably that they are either lying or delusional, not that they literally witnessed this momentous event. This fallible-other logic should apply equally to the self. So if you witness an event like that, the first thing is to doubt your observations, and hopefully check yourself into the nearest psych ward to get evaluated for a psychotic episode.
For a more controversial and less central example, consider strong and very strong longtermism. Despite the arguments to the contrary, it is very hard to avoid privileging a few pet scenarios because of the availability heuristic, and assign non-negligible probabilities to them. In this neck of the woods it is the AGI takeover, in the prog community it might be the climate change wiping out almost all of humanity, in some evangelical circles it can be hastening the rapture.

Now, to address the but-whatab-autists like myself, there are definitely cases where estimated tiny probabilities can be pretty accurate, like, say, playing the lottery, and it is worth calculating actions based...
[Linkpost] Danger of motivatiogenesis in interdisciplinary work
by particlemania — 2021-11-25T00:13:46.733Z

This is a linkpost for https://egtheory.wordpress.com/2019/03/15/motivatiogenesis/

This is a blog post from a computational biology researcher, Artem Kaznatcheev, about an epistemic phenomenon he calls 'Motivatiogenesis'. (Building up on xkcd's citogenesis)
This seems like an important concept to be known in these spaces, especially for people doing interdisciplinary exploration.

The following quote from his post captures the essence of the concept.

Let me first be a bit more precise about what I mean by motivatiogenesis.

I think that Kyler J. Brown first introduced me to this many years ago when we were both still at McGill. He referenced me an all-too-common misbehavior in neuroscience: A bad researcher justifies his work to a biologist with “this is the way computer scientists address this question, and it is of interest purely from theory” and when justifying the same work to a computer scientist — “this model is biologically reasonable, and of interest from science”. The biologist and computer scientist don’t know the other’s field well enough to see through this bluff, and the researcher manages to squeeze a poor publication out.

Contrast this with what a good researcher would do. She would justify her work to a biologist purely on biological grounds. And she would justify it to the computer scientists through its contribution to computer science. In other words, a good interdisciplinary scientist would contribute to both fields, and not use the border as a crutch.

But the case of the bad scientist gets worse!

Once the cycle starts and a group get a few such papers out, the auto-catalytic effect sets in: future work can justify itself by saying “we use a standard model in the field”. All of this even though the ‘standard model’ never had a justification for it. Eventually the subfield can start generating and answering its own field-endogenous questions that are fundamentally unhinged from reality.

But unlike a factoid, a false motivation is harder to burst. Especially if a subfield or cottage industry develops around the method. I think that this might be closely related to Jeremy Fox’s notion of Zombie Ideas.
Meetup for The Roots of Progress in San Diego, Dec 1
by jasoncrawford — 2021-11-24T22:50:39.171Z
Link: https://rootsofprogress.org/san-diego-community-meetup-happy-hour-december-1

I'll be in San Diego, CA on December 1. That evening, The Roots of Progress is hosting a community meetup and happy hour. Appetizers will be provided, cash bar. Come meet others and chat about progress.

When: Wednesday, December 1, 5:30pm

Where: Downtown San Diego, RSVP for exact location

RSVP here: https://airtable.com/shrNYCVF4xnhjaBaN