trentmkelly/LessWrong-43k huggingface dataset:
How to Find the Frontiers of Knowledge
by Samo Burja — 2020-10-30T10:50:02.301Z

This article is based on my lecture at Topos House, San Francisco on February 29, 2020. It was originally published on SamoBurja.com. You can access the original here.

Not all fields of knowledge exist yet. If you tried to study biochemistry in 1820, you’d have a lot of trouble: the field had yet to cohere. Do we think that all the biochemistries of the world have been discovered? If we did back then, we’d be wrong. For those seeking a safe career, sticking to the established fields is probably the right move. But for those interested in pushing forward the frontiers of knowledge, it will sometimes be better to work in a field that has not yet cohered, or in a field on the cusp of crystallizing. Often the best intellectual opportunities — if you want to be a breakthrough researcher — and the largest economic opportunities — if you want to build a great company — are going to be precisely in those areas about to crystallize into a new field.

If you went into biochemistry when biochemistry was emerging, your name might be in a textbook today. You didn’t have to be brilliant; you just had to pick your problem well. The greatest difficulty would be in figuring out who to learn from. Before you figure out where and how to learn, you have to decide who has the knowledge you seek. When the field of biochemistry was on the cusp of crystallizing, for example, you would have looked to experts in the fields of biology and chemistry. But as an outsider, who are you to evaluate the quality of a field? Just because a field claims to exist, doesn’t mean it exists.

Evaluating Existing Fields

One thing you can do is look at how a field performs on its own criteria. Take the replication crisis in academic psychology for example. In academic science, replicability has been one of the gold standards both for internal bureaucratic targets and also for the layman’s understanding of the philosophy of science. Philosophy of science might not seem very important, but it is key to figuring out how we know what we know, and what is good science as opposed to bad science. There is, in fact, no consensus philosophy of science, which means that there is no canonical science of science.

But surely scientists must know how science works? Well, scientists might know how science works in the way that birds know how aerodynamics works. They know it, but not at all on a con...
The Darwin Game - Round 1
by lsusr — 2020-10-29T21:56:44.945Z

Technical Difficulties

I messed up the game engine. I gave bots their own move in the previous parameter instead of their opponent's move.

This is the primary factor behind the CloneBots diverging from each other. The bug caused CloneBot pairings to score 200-300 instead of 250-250.

I'm going to restart the tournament.

Sorry.

The original tournament, created by a blind idiot me, was exciting. It resembled real world population dynamics. I wrote an 8-part series with 11,277 words and 24 graphs.

Alas, the actual contest is between intelligent agents.

This will be fast.

Edit: Everything below this line is in error too. Click here for the real, real game.

Round 1

Today's Obituary

Bot
Team
Summary
Round

Silly 0 Bot
NPCs
Always returns 0
1

The next installment of this series will be posted on October 30, 2020 at 5 pm Pacific Time.

Edit 2020-10-29: I'm sick. Next installment may be delayed.

Edit 2020-11-05: Since this is taking so long I feel I owe you guys an explanation. I have not forgotten about the tournament. Over the last couple of weeks, several correlated problems in meatspace triggered a cascading failure in my real life affairs. All the problems are temporary but I am temporarily unable to write bug-free software. (I actually did write a post for October 30 but it had multiple errors in it.) I would be surprised if it took more than a month to resolve everything but can make no guarantees. I plan to continue the tournament as soon as I can write bug-free software. I apologize for the delays.

Edit: 2020-11-10: I found some more bugs in the code and will have to restart things again.
Controllables and Observables, Revisited
by Scott Garrabrant — 2020-10-29T16:38:14.878Z

This is the fourth post in the Cartesian frames sequence. Read the first post here.

Previously, we defined controllables as the sets of possible worlds an agent can both ensure and prevent, and we defined observables as the sets of possible worlds such that the agent can implement all conditional policies.

Now that we have built up more language, we can redefine controllables and observables more categorically.

1. Controllables

1.1. Ensurables and Preventables

The categorical definition of ensurables is very simple.

Definition: Ensure(C) is the set of all S⊆W such that there exists a morphism ϕ:1S→C.

As an example, let C0=(A,E,⋅). Recall that 1S=({b},S,⋆), where b⋆s=s for all s∈S. E.g., if S0={w0,w3,w4}, then

1S0=w0 w3 w4b(w0 w3 w4).

If there is a morphism (g,h) from 1S0 to C0, this means that:

There is a function g from 1S0's agent {b} to C0's agent A, i.e., a function that always outputs a specific a∈A.
There is a function h from C0's environment E to 1S0's environment S0.
The specific a∈A picked out by g exactly implements that function h:E→S0.

That function h:E→S0 is exactly like the function you get by looking at that row, so a morphism (g,h):1S0→C0 is like a row in C0 that is entirely contained in S0. If there are multiple such rows, then there will be multiple distinct morphisms 1S0→C0 picking out different a∈A.

In "Biextensional Equivalence," we noted that 1S is like a passive observer who has a promise from the environment that the world will be in S. The existence of a morphism 1S→C means that there's an interface that allows a powerless bystander who has been promised S to play C's game. Since Agent(1S) only has one option, this interface must send that one option to some option for C's agent that is compatible with this promise.

Claim: This definition is equivalent to the one in "Introduction to Cartesian Frames": Ensure(C)={S⊆W | ∃a∈A,∀e∈E,a⋅e∈S}.

Proof: Let C=(A,E,⋅) and let 1S=({b},S,⋆), where b⋆s=s for all s∈S. First, assume there exists a morphism (g,h):1S→C. Here, g:{b}→A&n...
Covid Covid Covid Covid Covid 10/29: All We Ever Talk About
by Zvi — 2020-10-29T15:10:00.940Z

Trump wishes to register a complaint.

He does so within a thread full of clips that have to be seen to be believed but which you are under zero obligation to see either way.

Here is the central quote, which is real: “Covid, Covid, Covid, Covid, Covid, it’s all you ever talk about. A plane goes down, 500 people dead, they don’t talk about it. ‘Covid Covid Covid Covid.’ By the way, on November 4th, you won’t hear about it anymore … ‘please don’t go and vote, Covid!’”

This is his closing message to the American people. So unfair, we’ve beaten the virus, we’re turning the corner, we only have so many cases because we run so many tests. He kept saying it in the debate, he said it in the 60 Minutes interview, he says it rally after rally.

Whereas he now also claims that: “In California, you have a special mask. You cannot under any circumstances take it off. You have to eat through the mask. Right, right, Charlie? It’s a very complex mechanism. And they don’t realize those germs, they go through it like nothing.”

I mean, Covid is all we ever talk about here. That’s kind of the idea.

He must think these closing messages will resonate.

Most of all, Trump’s central point is he wants you to know that he is Not Biden, and most importantly, Not a Democrat. Because, you see, the Biden and the Democrats are terrible.

Meanwhile, Biden warns of a Dark Winter.

Biden also has a closing message. His closing message’s central point is clear. He wants you to know that he is most certainly Not Trump, and also Not a Republican. Because, you see, Trump and the Republicans are terrible.

Partisanship, as ever, is a hell of a drug.

As we discussed last week, two things are simultaneously true. Trump’s statements are completely unconnected to reality, and most the Doom Patrol of Very Serious People are also mostly unconnected to reality.

When it comes to Covid-19, both sides constantly make false claims about where things stand, with Trump’s claims being by far the farthest from reality.

But on the question of whether the worst is behind us, the answer is probably yes.

In March and April, the hospital system broke down. Supply lines were hanging by a thread and we were having trouble finding ways to put literal food on our literal table, especially meat. Thousands were dying each day. Supply chains and the whole economy and ...
AI risk hub in Singapore?
by Daniel Kokotajlo — 2020-10-29T11:45:16.096Z

I tentatively guess that if Singapore were to become a thriving hub for AI risk reduction, this would reduce AI risk by 16%. Moreover I think making this happen is fairly tractable and extremely neglected. In this post I sketch my reasons. I'm interested to hear what the community thinks.
My experience (and what I've been told) is that everyone generally agrees that it would be good for AI risk awareness to be raised in Asia, but conventional wisdom is that it's the job of people like Brian Tse to do that and most other people would only make things worse by trying to help. I think this is mostly right; my only disagreement is that I think the rest of us should look harder for ways to help, and be willing to sacrifice more if need be. For example, I suggested to MIRI that they move to Singapore, not because they could or should try to influence the government or anything like that, but because their presence in Singapore would make it a more attractive place for AI risk reducers (e.g. Singaporean EAs), thereby helping to create an AI risk hub there (instead of the current situation, which is brain drain from Singapore to the Bay and London).
I put my calculation of expected value at the end; for now, here are some basic facts about Singapore and the major pathways by which I expect good things from an AI risk hub there.

Thanks to Jia Yuan Loke, Vaidehi Agarwalla, and others for conversations that led to this post.

Some basic background facts about Singapore:
1. Smart, educated, english-speaking population; a tech, trade, and financial hub for Asia.
2. Cost of living lower than London but higher than Toronto. Haven’t looked into this much, just googled and found this.
3. Is already an EA hub compared to most of Asia, but has very little EA presence compared to many places in the West.
4. Singaporean government is unusually rational, in both epistemic and instrumental senses. It is a one-party state run by very smart son of Lee Kwan Yew, the man who said: “I am not following any prescription given to me by any theoretician on democracy or whatever. I work from first principles: what will get me there?”
5. Government of Singapore is widely respected throughout the world, including by China and the USA. Thousands of Chinese officials visit Singapore to learn from how they do things. Singapore has served as a bridge between East and W...
Babble challenge: 50 consequences of intelligent ant colonies
by Bird Concept — 2020-10-29T07:21:33.379Z

Tsuyoku Naritai!

The evening sun sets on the horizon. An owl hoots ominously. You stare at the ant hill in disbelief. You’ve checked the data many times, and there’s only one remaining hypothesis without vanishingly small probability.

Ant colonies are intelligent. Not the individual ants -- but the colony as a whole. They process information in a way metaphorically similar to what brains or GPUs do, without any of the constituent neurons or transistors being intelligent.

Through a somewhat haphazardly administered battery of tests you’ve determined the colony’s intelligence level to be around that of an average human. Despite only containing about a million ants.

But the world would not have looked like this had ant hills always been intelligent. No, they became intelligent, for reasons unknown, starting this week.

You realise that the world, as they say, will never be the same.

But what, exactly, are the consequences of this?

This week’s challenge: babble 50 ways in which the world would change if ant colonies of sufficient size had the intelligence of a human.

What would governments, entrepreneurs, or artists do differently? Would companies build use-at-home interfaces for talking to colonies? Would governments seal off colonies from public access? Would ant poison become illegal?

You tell me.

Looking back

Following a suggestion since last week, I’m changing the scoring system a bit. From this week onwards, you gain a star by completing the challenge, and lose a star for missing a week. (Instead of losing all your stars for missing a week).

Here are the current rankings. Great job everyone!

★★★★ gjm, jacobjacob, Tetraspace Grouping

★★★ Mark Xu, Bucky, Yonge

★★ Turntrout, Slider, Harmless

★ tinyanon, mingyuan, Rafael Harth, habryka, romeostevensit, WrongPlanet, Raemon, MikkW, amplemaple, Max Dalton, athom, johnswentworth, ryan_b, Ericf, CptDrMoreno

(Max Dalton and Turntrout answered a previous week’s challenge, but they did it within the last week. Slider and Elizabeth made submissions that were great in many ways, but ultimately disqualified. For motivation see their answers and my comment where applicable.)

Moving Forwards

This is week 5 out of my 7-week babble sprint.

Last week was a personal babble, so this week will be less so. I chose this week’s challenge because it relates to this question:

What is creati...
How should one deal with life threatening infections or air planes? 
by Nacruno96 — 2020-10-29T03:35:59.472Z

I really struggle with the COVID situation. I am 24 year old man with light allergic asthma. My death risk from COVID seems to be somewhat between 0,2% and 0,01%. Probably out of fear, probably also a bit because of my lack of need for socializing I ended up hiding in my room and contacting no one. I live with my parents. And contacting no one means avoiding my parents and walking in the house with an ffp3 mask. I am fully aware that my precautions are extreme. According to my estimates 2% of the population in my town will end up having Covid and in the case of one of my parents having covid there is a 70% chance he will not infect anyone. My risk now is probably close to zero because I am not having contact with any one. But would I remain in contact with my parents my death risk would be 3 out of 250 000 or 3 out of 5 million. Somewhat as likely as a plane crash. If I am trying to live until age 1 million my behavior would be definitely rational, because in that case taking every year a plane ride would kill me at age 1 million statistically once. However I would assume the likelihood of me having the chance to live up to that age as 20%. But of course it would be a great gain for my utility function to live that long. My question would be, how the rest of you are dealing with risks. Writing this post I am fully aware that the risk of dying from stroke, cancer is much higher for me now. Probably around 20 times higher than from covid. This would be the explanation for my behavior. Of course I could also just rationalize in this way my excessive fear. I am also driving a car at the same time. But I can rationalize that because I am excessively cautious while driving.

I got many interesting comments to my question. I would also like to point out that a covid infection causes a drop of 4 iq points according to a british study. But I explicitly didn’t mention long term consequences because I didn’t wanted to just focus on Covid. I wanted to leave a door open for other risky activities and the discussion of Covid unrelated risks
What's holding back outsourcing to cloud labs?
by ChristianKl — 2020-10-28T23:07:19.509Z

I would have expected Emerald Cloud Lab or similar competitors to go a lot and be successful over the last five years. As far as I know, like Emerald Cloud Lab only had modest growth and there aren't competitors who grew strongly. Outsourcing to cloud labs seems like it allows the laberatory to have benefits of scale and virtualization that drives down costs and is easier to use then working in a wet lab. Is there something holding back this trend that I'm not seeing? Alternatively, what's going on?
Reasons that discussions might feel hopeless
by gogishvilli — 2020-10-28T19:59:02.926Z

First posted on r/erisology - link.

I tried to come up with a way to detect trolls, realized that actually what I want to achieve is a way to detect conversation that I want to drop. It's hard to define on meta-level because my goals are different in each case. It feels like really basic stuff, but I've failed to find here a discussion of it. Also, this is my first post here, so I'd appreciate any feedback and links to stuff that might be basic.

So, here are some reasons that a discussion might feel hopeless.

The topic is genuinely above our level to comprehend on a level that we're able to discuss or explain it.

It's above the level of our civilization - e.g. cutting-edge math and STEM research, when we aren't sure that there is some truth-verification mechanism.
It's above my personal level, but I still want to participate if the interlocutor can verify and/or explain. E.g.

explanations

of math paradoxes, feel somewhat hopeless for me.
It's above the level of all participants - when neither I nor my friends understand rocket science or have experience of extreme poverty, but we're sure there are people who do.

Too long

inferential distances

About the topic. I don't understand math, talk to an expert, but it just can't be comprehended in

one conversation

.
About meta-level, like communication. Like, talking to someone who doesn't have an idea that argument can be valid or invalid.

Graham Pyramid

,

Grice’s Cooperative Principles

,

Conflict/mistake theory

, what have you.

Bad environment - too many participants can be as bad as loud noise IRL.

Frame differences

- the difference with 2.2 is if noticed, can be relatively easily fixed.
Different goals - can look similar to 4, but the best fix more often is to stop the discussion. I think it's any situation when the goal is not the search for mutual understanding or finding the truth, but it's hard to dissect. I can find only two classes:

Innocent, like a maintenance engineer, who is interested in fixing the problem, talking to a manager who is interested in fixing the reason the problem appeared or firing the guilty.
Malicious - conscious trolling, guilt-tripping, lying, etc.

One of the participants lacks actual intelligence to discuss. The difference with 1 and 2 is that no amount of explanations or no length of discussion will be sufficient to explain it. Hard to detect, not sure how often is actually true.

Any reversible INT deb

...
What risks concern you which don't seem to have been seriously considered by the community?
by plex — 2020-10-28T18:27:39.066Z

There are a few things I'm worried about which I have not seen discussed much, and it makes me wonder what we're collectively missing.

This seems like a question which has likely been asked before, but my Google-fu did not find it.

You don't need to make a watertight case for something being important in order to share your concern, brief explanations are fine if the high bar of writing something detailed would put you off posting.
[AN #123]: Inferring what is valuable in order to align recommender systems
by Rohin Shah — 2020-10-28T17:00:06.053Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

HIGHLIGHTS

From Optimizing Engagement to Measuring Value (Smitha Milli et al) (summarized by Rohin): This paper takes a stab at creating a better objective for existing recommender systems than engagement, in a way that could be applied at existing companies like Twitter. The basic approach is to treat the variable to be optimized (user value) as a latent variable, and use probabilistic inference to infer how likely it is that a particular recommendation was valuable.

Usually a major challenge with such an approach is specifying the observation model: how the observed data is caused by the latent variable. In the case of Twitter, this would require you to answer questions like, “if the user does not value a tweet, how likely is a user to hit the like button anyway?” This is a hard question to answer, since perhaps users like tweets in order to stop conversations, or because they are addicting at the moment but are not actually valuable, etc.

One simple heuristic is to take two datasets where we know one dataset has more valuable recommendations than the other. Differences in user behavior between these datasets can then be assumed to be correlations with value. The authors provide a quantitative method for inferring the observation model from such datasets, which I won’t go into here since it is primarily a heuristic baseline. One obvious problem is that if the “better” dataset was produced by optimizing (say) clicks, then the clicks may have increased for reasons other than improved value, but this heuristic approach will attribute the entire increase to improved value.

How can we do better? The key insight of this paper is that if you have a bunch of historical data, then you can get a lot of mileage by identifying an anchor: a type of feedback that when given provides unequivocal evidence of the latent value. On Twitter, this is taken to be the “See Less Often” (SLO) button: if this is clicked, then we know with effective certainty that this was not valuable, regardless of any other action...
Draft papers for REALab and Decoupled Approval on tampering
by Jonathan Uesato — 2020-10-28T16:01:12.968Z

Hi everyone, we (Ramana Kumar, Jonathan Uesato, Victoria Krakovna, Tom Everitt, and Richard Ngo) have been working on a strand of work researching tampering problems, and we've written up our progress in two papers. We're sharing drafts in advance here because we'd like to get feedback from everyone here.

The first paper covers:

How and when tampering problems might arise in the real world
Key assumptions in standard RL frameworks we relax to allow modeling tampering
How we model and measure tampering empirically, through our internal platform REALab, and
How we formalize tampering problems, through our Corrupt Feedback MDP formalism

We particularly hope it clears up the concept of tampering (and why "but the agent maximized its given reward function" typically assumes the wrong framing), and internally, we've found REALab to be a useful mental model.

The second paper describes:

Decoupled approval, an algorithm closely related to approval direction and Counterfactual Oracles, and designed to be straightforwardly compatible with standard deep RL
An analysis of this algorithm (within the CFMDP formalism), and
Empirical validation (in REALab)

We'd love to get feedback on these; the current drafts are viewable in this Google Drive folder. We're happy to discuss these on whichever of LessWrong/Alignment Forum/Google Drive comments, and would prefer to keep discussion on these forums for now, as we'll share the papers more widely after they're posted on arXiv in a few weeks. Looking forward to hearing your thoughts!