trentmkelly/LessWrong-43k huggingface dataset:
Considerations on Compensation
by RobertM — 2021-10-06T23:37:56.255Z

This post was prompted by the recent announcement of Lightcone Infrastructure, and particularly Elizabeth's response in a comment thread on compensation. I'd like to thank Max Wallace, Ruby, and Miranda Dixon-Luinenburg for their comments and copyediting.

Context (& disclaimer): I'm a software engineer and donate ~exclusively to longterm-focused orgs (currently MIRI). While most of the arguments are intended to be broadly applicable, this post was written with that specific framing in mind, and one argument does explicitly depend on the question of AGI timelines.

You're an EA-aligned organization that isn't funding constrained. Why should you (or shouldn't you) pay market rate?

Talent Pool

The econ 101 view says that if you want more of something, one easy way to get it is to pay more money. But where exactly would we be getting "more" from?

The already-convinced

There is a group of people who already know about your mission and agree with it, but aren't doing direct work. This group consists of:

Those earning to give

Those who are earning enough to give but don't (for a variety of reasons)

Those who don't have the capacity do either direct work or earning to give

On the margin, who in this group could be convinced to do direct work?

Those earning to give

There are many reasons why someone might be earning to give instead of doing direct work, assuming they have the skillset necessary to do the second (or can acquire it). Here is a list of plausible reasons that could be overcome with more money.

Information - they may simply not know that their favorite EA org pays (close to) market rates! This is an area where there may be some "free wins" obtainable with just better PR - in that sense I think Lightcone's announcement and public salary listing is a great start.

Maintaining their present lifestyle - this one is pretty sensitive to the actual numbers involved, but it needs to be said that in the Bay Area there are things that are difficult to afford on typical non-profit salaries, like "owning a house within reasonable commuting distance of the office" and "having kids". These can be concerns even for e.g. highly-paid software engineers, if they've already done one or the other (or both!) and thus locked themselves in to those costs.

Maintaining optionality - if you take a pay cut to do direct work a

...
What to read instead of news?
by ekritose — 2021-10-06T22:59:00.527Z

I often read the news to take a break from work. Maybe this isn't such a good idea. (https://www.gwern.net/docs/culture/2010-dobelli.pdf).

What should I read instead? Ideally it should be interesting but not impenetrable, finished in less than half an hour, and easily available from my browser. I'm happy to pay for good content.
How do bounded utility functions work if you are uncertain how close to the bound your utility is?  
by Ghatanathoah — 2021-10-06T21:31:26.412Z

If you are trying to calculate the value of a choice using a bounded utility function, how can you be sure whether you are close or far from the bound, whatever the bound is? How do you account for uncertainty about how much utility you already have? Does this question actually make sense?

Recently I have come across arguments against using a bounded utility function to avoid Pascal’s Mugging and similar “fanaticism” problems. These arguments, such as Section 6 of Hayden Wilkinson’s paper “In Defense of Fanaticism” and the Less Wrong post “Pascal's Mugging for bounded utility functions” both use a novel argument against bounded utility functions. If I understand them correctly, they argue that bounded utility functions cannot work because it is impossible to know how much utility one already has. This means one cannot know how close to the bound their utility is, and therefore one can never know how much to discount future utility by.

Wilkinson’s paper uses the example of someone with an altruistic bounded utility function that is essentially total utilitarianism. So they want to increase the total utility of the universe and, because they have a bounded utility function, the value of additional total utility decreases as it approaches some upper bound. If I understand his argument correctly, he is saying that because this agent has a bounded utility function, they cannot calculate how good an action is without knowing lots of details about past events that their actions cannot effect. Otherwise, how will they know how close they are to the upper bound?

Wilkinson analogizes this to the “Egyptology” objection to average utilitarianism, where an average utilitarian is compelled to study how happy the Ancient Egyptians were before having children. Otherwise, they cannot know if having children increases or decreases average utility. Similarly, Wilkinson argues that a total utilitarian with a bounded utility function is compelled to study Ancient Egypt in order to know how close to the bound the total utility of the world is. This seems implausible, even if information about Ancient Egypt was easy to come by, it seems counterintuitive that it is relevant to what you should do today.

“Pascal's Mugging for bounded utility functions” by Benya introduces a related problem. In this scenario, a person with a bounded utility function has...
2021 Darwin Game - Everywhere Else
by lsusr — 2021-10-06T20:39:43.200Z

Shore

The Shore is an inhospitable wasteland. Algae is available, but it's not very nitrutious. Coconuts offer an even worse calories-to-digestion ratio.

Name
Carrion
Leaves
Grass
Seeds
Detritus
Coconuts
Algae
Lichen

Shore
0
0
0
0
20
1000
1000
10

No coconut eaters got established in the Human Garbage Dump, which means no coconut eaters got established in the River or Shore instead. The only available food is algae. The Shore is basically a small bit of Ocean biome with ⅒ of the algae. The algae-eating winner of our Ocean competition has zero speed so it cannot migrate to the Shore.

To top all of it off, predators migrate in from both sea and land.

Soonbegons migrate from time to time to eat the Detritus. Otherwise, the Shore is mostly a graveyard.

Winners?

Soonbegons (kind of) by Martin Randall.

Grassland

The grassland has lots of grass and even more seeds.

Name
Carrion
Leaves
Grass
Seeds
Detritus
Coconuts
Algae
Lichen

Grassland
0
100
1000
2,000
0
0
0
50

The Grassland took 500 turns to establish an equilibrium.

Generations 1-500

Goes Extinct in Generation
Species

11
Meercat colony

14
Flock of birbs

15
Bool

16
Big Oof

17
Empowered Turtle

17
glpp511

18
cg-bird

19
Nyarlathotep

20
Human

21
Tiny Snek

21
grass_mouse

22
Sleepypotat

22
Frontier-W8

23
Armored Nutcracker

23
Toxikeet

23
lGha-S541

24
Cowlagor

27
Karthosorox

30
Piranhakeet

30
cg-fastbird

31
Wampirek

32
Yonge_Snake

34
Sheep

37
Jackrabbit

39
Grassland Tribble

39
Grassland Aphid

40
basic seed fodder

40
Galumphers

40
lGca-AS154

41
GrassSeeater

41
Small Moth

41
SpeedyLichen

41
BeaupreyButGrassland

41
Tribble

42
DefinitelyJustARock

44
Weedle

45
Bob

45
SeedyEaty

45
Locust-Seeds

47
Tribble Hunter Hunter

49
Goat

51
seed fodder

52
Seed Eater Eater Eater

53
Frontier-W2

56
Trash Panda1

56
mediocre 1-2-s

59
Common Rat

61
Omnivorous Tribble Hunter

62
Frontier-W5

63
Siolid

65
Cannibal Locust

68
Tribble Hunter Hunter Hunter Hunter

73
Ziarnojad Malutki

74
Horned Owl

78
Yonge_Omnivore

83
Medium Seedrat

86
Maverick Goose

89
Squish

187
Mooshroom

294
Killer Bunny

345
Untielopay

423
6-0-3 seed

466
Yonge_Vegan

Generations 500-2000

The grassland establishes an equilibrium. The only native species left is the Hopsplitter.

You can't see it but there's a single Venomoth on the bott...
Is GPT-3 already sample-efficient?
by Daniel Kokotajlo — 2021-10-06T13:38:36.652Z

(Concrete, easy-to-answer question below, explanation first)

Common adage: Modern deep learning techniques are sample-inefficient; it takes loads of data for them to learn things. If you pre-train them, it takes less additional data for them to learn something new, but still compared to humans it takes a lot.

Elsewhere, based on papers like this and this, various people have extrapolated the following takes:

--It seems like bigger neural nets need to see less data to reach the same level of performance.

--It seems like bigger neural nets need fewer epochs to reach convergence. Soon they'll only need to see each data point once. (Search this for "multiple epochs")

I feel like this take is in tension with the common adage. I wonder: If there is a fact mentioned in GPT-3's training data, how many times does it need to be mentioned before GPT-3 comes to know that fact? For example, I'm told that GPT-3 knows the names of most prominent members of the rationalist community. How many times has it seen each name? Are we talking ten times, or ten thousand?*

I'd be interested to hear people do a bit of a search for the "most sample-efficient/obscure fact" in GPT-3's repertoire. In this manner we could quantity how many times GPT-3 needs to see something before it learns it. (Maybe we don't have access to the dataset used to train GPT-3. But people at Eleuther.ai have The Pile, right? And they've trained big transformers on it? We could answer the question easily and precisely there, no?)

Or am I thinking about this all wrong somehow? This seems like an obvious idea, I wonder why I haven't heard of it before.

*Suppose it is ten thousand. Then that means one in every ten million two-word strings on the internet is "Paul Christiano." (The dataset for GPT-3 was 300B tokens) Add in all the other rationalists/EAs and probably it means one in every hundred thousand words is the name of some prominent rationalist/EA. Surely this is too much, no? It seems way too much according to Google Ngram Viewer.
Theory: The Political Incentives for Progressive Policy Promote Policy Failure
by BDay — 2021-10-06T12:10:03.015Z

Thanks to Miranda for providing feedback on drafts and proofreading via the proofreading service.

I think there is an incentive problem in Leftist and progressive politics (and therefore democracy more broadly) that isn’t adequately discussed. The goal of progressive politics is said to be helping people who have been disenfranchised unfairly. Naturally, progressive policies and institutions draw much of their support from disenfranchised people. However, what happens to the base of support for these institutions and policies (which can be thought of as alive mimetically) if they actually succeed at helping people? With less disenfranchisement, one would imagine there would be less support for progressivism. Thus, progressivism may be operating under a perverse incentive structure where it is rewarded for creating disenfranchisement in society.

Things which are known to shift one’s attitudes away from progressivism and towards conservativism are getting married and having kids. One would also imagine that welfare programs would have higher support among those who use them. People getting married, having families and not needing welfare are things which are broadly considered to be good. Yet, if Leftist policies were to succeed at moving these metrics in the right direction (more marriage, more kids, less welfare recipients), they would push the policy preferences of their country to the Right.

One might argue that as the Right is incentivized to increase these things (to push people’s policy preferences to the Right), there is a fair balance in the system. However, I think this is wrong due to the asymmetric nature of creation and destruction. It is far easier and less energy intensive for the Left to implement entropic destructive policies than for the Right to implement extropic creative policies. Thus, the Left has an easy mechanism to strengthen its movement (enhancing disenfranchisement) while the Right has no similar easy mechanism.

This does not require any conscious malice by the Left. All that is required is for the Left to implement policies which fail or backfire, and then never fix them because they are rewarded electorally for the failure, rather than punished. The Right, on the other hand, needs to boost employment, wages, savings, decrease house and family starting costs and increase marriage rates in an environment where they are only in po...
Preferences from (real and hypothetical) psychology papers
by Stuart_Armstrong — 2021-10-06T09:06:08.484Z

Research projects

I'm planning to start two research projects on model splintering/reward generalisation and learning the preferences of irrational agents.

Within those projects, I'm aiming to work on subprojects that are:

Posed in terms that are familiar to conventional ML;

interesting to solve from the conventional ML perspective;

and whose solutions can be extended to the big issues in AI safety.

The point is not just to solve the sub-problems, but to solve them in ways that generalise or point to a general solution.

The aim is to iterate and improve fast on these ideas before implementing them. Because of that, these posts should be considered dynamic and prone to be re-edited, potentially often. Suggestions and modifications of the design are valuable and may get included in the top post.

Learning textual values from textual descriptions

Parent project: this is a subproject of the value learning project.

Background

This idea grew out of a conversation with Matija Franklin and Rebecca Gorman.

My Occam's razor paper with Sören Mindermann demonstrates that human behaviour does not provide enough data to define human preferences and values. I've grounded human values, ultimately, in our mental models, including our assessments about our own irrationality/preferences and the irrationality/preferences of others.

But it is very difficult for an algorithm to use this information. In order for an algorithm to use people's judgements to assess the irrationality or preferences of another person, the algorithm would have to know that some human action was going on, that another human was reacting to it, how they were interpreting it, that they were judging it to be irrational/rational/preference-relevant/etc... and that this judgement was to be taken as normative or as a label. So, not only is the person thinking "how irrational", but that interpretation is to be taken to be correct.

Setup

There is a shortcut to part of this process, though: psychology papers. Let's take the anchoring bias. As I've noted, we could see the anchoring bias as a preference to name numbers close-to-numbers-we-have-recently-heard. But no-one thinks about it that way.

If we look at a seminal paper on anchoring bias, it details how students were presented with products, asked if they would buy those products for the last two digits of their social security numbers, and were then asked to stat...
2021 Darwin Game - River
by lsusr — 2021-10-06T07:29:16.510Z

The River contains lots of every foragable food. The River is like the Human Garbage Dump except:

It has 5x as many resources.

Both air-breathing and water-breathing species can survive in the River.

Organisms may not start in the River. They must wander in.

Name
Carrion
Leaves
Grass
Seeds
Detritus
Coconuts
Algae
Lichen

River
500
500
500
500
500
500
500
500

Generation 0

At first, there are no animals and the foragables accumulate.

Generations 1-100

Species
Original Biome
Venom?
Weapons
Armor
Speed
Forage?
Creator

Leafy Luncheoner
Temperate Forest
Neither
0
0
1
Leaves
Sean Hyer

Mooshroom
Grassland
Neither
0
0
5
Grass;Seeds
Vanessa

slow vulture
Desert
Antivenom only
0
10
10
Carrion
eat (Allows survival in the Desert),Corm

mediocre 1-2-s
Grassland
Neither
1
0
2
Seeds
Corm

Very Lesser Forest Dragon
Rainforest
Neither
2
0
2

Persephone 0461

The same pattern repeats for each foragable. First the foragable accomulates. Then a species which can consume the foragable appears. Its population explodes until it has eaten the accumulated foragable. Finally, its population reverts to carrying capacity.

Persephone 0461's Very Lesser Forest Dragon wandered in. It thrived until another predator appeared which hunted it to extinction.

Generations 100-9000

Species appear. Species Disappear. Life is always in flux.

This is where all the Ocean's Soonbegons were coming from. Until they didn't anymore.

Winners

Name
Native Biome
Venom?
Attack
Armor
Speed
Forage?
Temperature Adaptation?
Creator
Social Media

Booyahs
Desert
Neither
2
0
10
Carrion
Heat (Allows survival in the Desert)
CK
None

Snark
Temperate Forest
Neither
0
0
10
Leaves

Vanessa
None

Venomoth
Rainforest
Venom + Antivenom
0
0
3
Grass

aphyer
None

Locust Eater Eater
Temperate Forest
Neither
2
0
2
Seeds

Multicore
None

Hopsplitter
Grassland
Neither
0
0
10
Grass;Seeds

Nem
None
2021 Darwin Game - Human Garbage Dump
by lsusr — 2021-10-06T06:31:53.416Z

The human garbage dump is a place you can find anything. It was also (to me) surprisingly popular with 90 species submitted.

Name
Carrion
Leaves
Grass
Seeds
Detritus
Coconuts
Algae
Lichen

Human Garbage Dump
100
100
100
100
100
100
100
100

In each of our previous biomes, there was one primary source of forage. This is our first biome with many different viable options to forage. However, individually, none of them are very large. Even if there's no predators, the diverse primary production should ensure a little more biodiversity among foragers.

Generations 0-1

We start with Anti Predator Flak. It's just like just like the Ocean's First Round Predator Distractor except it was created by Multicore.

Goes Extinct in Generation
Species

1
Anti Predator Flak

Generations 1-10

Most of the initial populations crash, as usual.

The Seed Beetle is exactly what it sounds like: a tiny organism with the ability to digest seeds and nothing else.

Goes Extinct in Generation
Species

6
Goo

6
Beauconut

8
Gelatinous Cube

8
Ivem

10
Chittering Harvester

10
Boot Nipper

10
Dump Leaf Blight

Generations 10-100

The Seed Beetle Population explodes. I think what's going on is something Multicore pointed out at the game launch. The Seed Beetle is so small that it provides inadequate nutrition to large carnivores. At first, predators eat the Seed Beetle, then the carnivores starve. The Seed Beetle is thus left without predators.

The Mutant Two-Headed Speed Snail is a Grass forager with Speed 2. The Beauprey forages for Seeds and Lichen and has Speed 1.

Goes Extinct in Generation
Species

11
Trash Lobster

11
Carrion Condor

11
Roaches

11
Trash Panda2

11
Muk1

12
Small CocoCrab

12
The Fart Brigade

12
Thread-Nibbler

12
Dump Aphid

12
Slime Mold

12
mediocre 2-2-s2

12
Muk2

13
Olchi

13
TrashMonster3-2

14
Sentient Bio Reactor

14
BottomFeeder

14
SpeedDemon

14
mediocre 2-2-g

14
TrashMonster2-2

15
TrashMonster3-4

15
TrashMonster4-3

16
Big CocoCrab

16
Unpalatable Coconut Crab

16
TrashMonster4-2

17
Rabbit

17
Lazy Stationary Glutton

17
sc511

17
trashSeeds

18
Grease Monkey

19
Garbage Dump Shai-Hulud

19
Trolley

20
Coconut Crabs (tiny)

20
TrashMonster3-3

21
Robot

21
Rat

21
CP445957

22
mediocre 2-2-s1

23
Frogger

23
Coconut Cruncher

23
Yonge_Dump

23
TrashMonster4-4

24
All-eating Leviath

...
We're Redwood Research, we do applied alignment research, AMA
by Nate Thomas — 2021-10-06T05:51:59.161Z
Link: https://forum.effectivealtruism.org/posts/xDDggeXYgenAGSTyq/we-re-redwood-research-we-do-applied-alignment-research-ama

Redwood Research is a longtermist organization working on AI alignment based in Berkeley, California. We're going to do an AMA this week; we'll answer questions mostly on Wednesday and Thursday this week (6th and 7th of October). Buck Shlegeris, Bill Zito, myself, and perhaps other people will be answering questions.

Here's an edited excerpt from

this doc that describes our basic setup, plan, and goals

.

Redwood Research is a longtermist research lab focusing on applied AI alignment. We’re led by

Nate Thomas

(CEO),

Buck Shlegeris

(CTO), and

Bill Zito

(COO/software engineer); our board is Nate,

Paul Christiano

and

Holden Karnofsky

. We currently have ten people on staff.

Our goal is to grow into a lab that does lots of alignment work that we think is particularly valuable and wouldn’t have happened elsewhere.

Our current approach to alignment research:

We’re generally focused on

prosaic

alignment approaches.
We expect to mostly produce value by doing applied alignment research. I think of applied alignment research as research that takes ideas for how to align systems, such as amplification or transparency, and then tries to figure out how to make them work out in practice. I expect that this kind of practical research will be a big part of making alignment succeed. See

this post

for a bit more about how I think about the distinction between theoretical and applied alignment work.
We are interested in thinking about our research from an explicit perspective of wanting to align superhuman systems.

When choosing between projects, we’ll be thinking about questions like “to what extent is this class of techniques fundamentally limited? Is this class of techniques likely to be a useful tool to have in our toolkit when we’re trying to align highly capable systems, or is it a dead end?”
I expect us to be quite interested in doing research of the form “fix alignment problems in current models” because it seems generally healthy to engage with concrete problems, but we’ll want to carefully think through exactly which problems along these lines are worth working on and which techniques we want to improve by solving them.

We're hiring for

research, engineering

, and an

office operations manager

.

You can see our website

here

. Other things we've written that might be interesting:

A description of our current project

Some docs/posts that describe aspects of how we're thinking about the al

...
Metaculus is seeking Analytical Storytellers to write essays fortified with testable predictions
by ChristianWilliams — 2021-10-06T04:44:01.092Z

About Metaculus

At

Metaculus

, our aim is to improve human decision-making and coordination at scale by increasing analytic capacity, reasoning, and judgment. With the unique capabilities of the Metaculus forecasting platform, our

innovative programs

, and our

long-term partnerships

, we enable a range of contributors to engage in a process of collective reasoning, forecasting, and ultimately, more informed action.

About This Role

We are hiring writers, on an ongoing basis, to produce “fortified essays” on critical topics like emerging science and technology, global health, biosecurity, economics and econometrics, technology policy, global risk, environmental science, and geopolitics. These are longform, educational essays fortified with testable predictions.

Writers can come from many walks of life; they might be researchers, rationality enthusiasts, postdocs, data scientists, or quantitative analysts, but they must be able to write on their areas of expertise with creativity, with clarity, and with embedded, quantified forecasts. These pieces should serve as primers on crucial topics, should showcase original analysis, and should spotlight and contextualize high-leverage questions.

For examples of essays that have already been published and to get a sense of what the expectations are, see this comparison of competing

theories of inflation

by economist Arnold Kling, or this exploration of the

future of nanotechnology

by physical chemist Kevin Ausman, or this look at the spread of the

Islamic State in Nigeria

by geopolitical analyst Adam Ragozzino, or this piece on the past and future of the

Riemann hypothesis

by Ege Erdil.

Importantly, Essays must be accompanied by

original

forecast

questions

, which analytical storytellers will need to write and themselves publicly predict on.

Writers may submit up to five essays per month, each 1000+ words in length with at least two linked forecast questions. Essays are compensated at $300 each and at $25 per forecast question, with additional compensation awarded for especially high-quality essays attracting a significant readership.

To Apply

Email christian at metaculus dot com with a resume or CV and a short pitch of a couple sentences for the proposed essay. Feel free to share links to or attach writing samples.