trentmkelly/LessWrong-43k huggingface dataset:
Models Modeling Models
by Charlie Steiner — 2021-11-02T07:08:44.848Z

I - Meanings of words

Now that we have

more concrete thinking

under our belt, it's time to circle back to Goodhart's law for value learners. What sorts of bad behavior are we imagining from future value-learning AI? What makes those behaviors plausible? And what makes them bad?

Let's take the last point first. Judgments of goodness or badness are situated in models - models of the world that we use to infer and operationalize human values. And we don't just use the same one all the time.

When I say "I like dancing," this is a different use of the word 'like,' backed by a different model of myself, than when I say "I like tasting sugar." The model that comes to mind for dancing treats it as one of the chunks of my day, like "playing computer games" or "taking the bus." I can know what state I'm in (the inference function of the model) based on seeing and hearing short scenes. Meanwhile, my model that has the taste of sugar in it has states like "feeling sandpaper" or "stretching my back." States are more like short-term sensations, and the described world is tightly focused on my body and the things touching it.

Other models work too! That's fine, there's plenty to go around.

The meta-model that talks about me having preferences in both of these models is the framing of

competent preferences

. If someone or something is observing humans, it looks for human preferences by seeing what the preferences are in "agent-shaped" models that are powerful for their size[1].

So when we call some AI behavior "bad," this is a word whose meaning depends on usage and context, but ultimately bottoms out in implied models of the world. It's like a Winograd schema - like how English-readers infer that "they" in "workers put down the boxes because they were tired" refers to the workers, the "like" in "I like dancing" is understood to use a certain perspective on how I am modeling and interacting with the world.

All of this should be taken with the caution that there's not one True Model in which the True Meaning of the word "bad" is expressed. Obviously you still have to make some choice in practice, but the point is that the way you make this choice doesn't have to look like resolving epistemic uncertainty about which model is the True Model[2].

II - Model conflicts

What were the patterns that stood out from the

previous discussion

of what humans think of as bad behav...
Can homo-sapiens sustain an economy parallel to AI's?
by qbolec — 2021-11-02T07:03:08.129Z

In a world where taxis and trucks drive themselves, code writes itself etc. there's several ways humans can fit into such picture.
Perhaps they'll get rich by owning shares, perhaps UBI will be introduced, perhaps they'll move to jobs higher-in-abstraction-ladder using AI as tools, perhaps we'll just serve as testers/aligners/goal-setters.
But there's a possibility we'll have nothing to offer to such "ascended economy", or perhaps big part of a society will not be needed, while some other small part will benefit (Elyzium-style).

(I'm assuming the "progressive" part of the world is not hostile, just doesn't need the "traditional" part - in particular they respect their property rights)

Assuming this later scenario: is it possible that "traditional people" form some kind of closed system in which they still trade with each other, ignoring the outside progress?
In particular, would it require some strong coordination between them to refrain from buying/selling to the outside, or would it rather be the most natural and selfish thing for them to do try to trade locally?

Has it ever happened in the history, that a tribe has successfully walled off (without an active help from the outside to protect their customs)?
[Book Review] "The Bell Curve" by Charles Murray
by lsusr — 2021-11-02T05:49:22.198Z

Factor analysis is a mathematical method of inferring simple correlations between observations. It's the foundation of the Big Five personality traits. It's also behind how we define intelligence.

A person's ability to perform one cognitive task is positively correlated with basically every other cognitive task. If you collect a variety of cognitive measures you can use linear algebra to extract a single measure which we call g. Intelligence quotient (IQ) is a test specifically designed to measure g. IQ isn't a perfect measure of g but it's convenient and robust.

Here are six conclusions regarding tests of cognitive ability, drawn from the classical tradition, that are now beyond significant technical dispute:

There is such thing as a general factor of cognitive ability on which human beings differ.

All standardized tests of academic aptitude or achievement measure this general factor to some degree, but IQ tests expressly designed for that purpose measure it most accurately.

IQ scores match, to a first degree, whatever it is that ordinary people mean when they use the word intelligent or smart in ordinary language.

IQ scores are stable, although not perfectly so, over much of a person's life.

Properly administered IQ tests are not demonstrably biased against social, economic, ethnic, or racial groups.

Cognitive ability is substantially heritable, apparently no less than 40 percent and no more than 80 percent.

Charles Murray doesn't bother proving the above points. These facts are well established among scientists. Instead, The Bell Curve: Intelligence and Class Structire in American Life is about what g means to American society.

Stratification

Educational Stratification

Smarter people have always had an advantage. The people who go to college have always been smarter than average. The correlation between college and intelligence increased after WWII. Charles Murray argues that the competitive advantage of intelligence is magnified in a technological society. I agree that this has been the case so far and that the trend has continued between 1994 when Murray published his book and 2021 when I am writing this review.

SAT scores can be mapped to IQ. The entering class of Harvard in 1926 had a mean IQ of about 117. IQ is defined to have an average of 100 and a standard deviation of 15. Harvard in 1926 thus hovered around the 88th percentile of th...
2020 PhilPapers Survey Results
by Rob Bensinger — 2021-11-02T05:00:13.859Z

In 2009, David Bourget and David Chalmers ran the PhilPapers Survey (results, paper), sending questions to "all regular faculty members" at top "Ph.D.-granting [philosophy] departments in English-speaking countries" plus ten other philosophy departments deemed to have "strength in analytic philosophy comparable to the other 89 departments".

Bourget and Chalmers now have a new PhilPapers Survey out, run in 2020 (results, paper). I'll use this post to pick out some findings I found interesting, and say opinionated stuff about them. Keep in mind that I'm focusing on topics and results that dovetail with things I'm curious about (e.g., 'why do academic decision theorists and LW decision theorists disagree so much?'), not giving a neutral overview of the whole 100-question survey.

The new survey's target population consists of:

(1) in Australia, Canada, Ireland, New Zealand, the UK, and the US: all regular faculty members (tenuretrack or permanent) in BA-granting philosophy departments with four or more members (according to the PhilPeople database); and (2) in all other countries: English-publishing philosophers in BA-granting philosophy departments with four or more English-publishing faculty members.

In order to make comparisons to the 2009 results, the 2020 survey also looked at a "2009-comparable departments" list selected using similar criteria to the 2009 survey:

It should be noted that the “2009-comparable department” group differs systematically from the broader target population in a number of respects. Demographically, it includes a higher proportion of UK-based philosophers and analytic-tradition philosophers than the target population. Philosophically, it includes a lower proportion of theists, along with many other differences evident in comparing 2020 results in table 1 (all departments) to table 9 (2009-comparable departments).

Based on this description, I expect the "2009-comparable departments" in the 2020 survey to be more elite, influential, and reasonable than the 2020 "target group", so I mostly focus on 2009-comparable departments below. In the tables below, if the row doesn't say "Target" (i.e., target group), the population is "2009-comparable departments".

Note that in the 2020 survey (unlike 2009), respondents could endorse multiple answers.

1. Decision theory

Newcomb's problem: The following groups (with n noting their size, and skipping ...
Notes on Shame
by David Gross — 2021-11-02T04:33:23.861Z

This post examines the virtue of shame. It is meant mostly as an exploration of what other people have learned about this virtue, rather than as me expressing my own opinions about it, though I’ve been selective about what I found interesting or credible, according to my own inclinations. I wrote this not as an expert on the topic, but as someone who wants to learn more about it. I hope it will be helpful to people who want to know more about this virtue and how to nurture it.

“He who feels no shame of evil and does not hate it is no man. Shame and hate of evil are the beginning of virtue.” ―Mencius[1]

“Where there is yet shame, there may in time be virtue.” ―Samuel Johnson[2]

What is this virtue?

There’s a terrible terminological muddle around shame. I’m going to use “shame” to mean an unpleasant sense that one has failed to live up to one’s own standards in some way. To have a well-tuned virtue of shame (or sense-of-shame) is for this sense to reliably and usefully alert to appropriate things.

Arguably, this virtue might also include responding to this sense well: how you process shame, learn from it, dispose of it properly, and so forth.

Shame vs. guilt

Shame overlaps with guilt and sometimes “guilt” is overloaded to include shame among its meanings. I think “shame” is a better word for precisely describing the virtue. For one thing, you can be ashamed of something (e.g. not living up to your potential) without necessarily being guilty of some specific transgression.[3]

Also, you can judge someone else to be guilty, or they may just objectively be guilty based on the facts of the matter — whereas shame is more of an introspective, subjective evaluation. It’s true that you can try to shame someone, but for this to succeed it requires their cooperation: they must acknowledge and internalize the shame by becoming ashamed, or the attempt sputters out ineffectually. This is why you can say simply “you are guilty” but shaming takes a more complex construction: “you should be ashamed of yourself.”

These linguistic categories are blurry, though: Ashamed people sometimes say they “feel guilty.” Some people restrict “shame” to refer to an I-am-bad sort of shame, and use “guilt” for I-did-bad shame about specific discrete acts. Just to keep things confusing, I-am-bad shame is sometimes called “free-floating guilt,” and at least one researcher insists that “shame” refers to f...
EfficientZero: human ALE sample-efficiency w/MuZero+self-supervised
by gwern — 2021-11-02T02:32:41.856Z
Link: https://arxiv.org/abs/2111.00210

"Mastering Atari Games with Limited Data", Ye et al 2021:

Reinforcement learning has achieved great success in many applications. However, sample efficiency remains a key challenge, with prominent methods requiring millions (or even billions) of environment steps to train. Recently, there has been significant progress in sample efficient image-based RL algorithms; however, consistent human-level performance on the Atari game benchmark remains an elusive goal.

We propose a sample efficient model-based visual RL algorithm built on MuZero, which we name EfficientZero. Our method achieves 190.4% mean human performance and 116.0% median performance on the Atari 100k benchmark with only two hours of real-time game experience and outperforms the state SAC in some tasks on the DMControl 100k benchmark. This is the first time an algorithm achieves super-human performance on Atari games with such little data. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data.

EfficientZero's low sample complexity and high performance can bring RL closer to real-world applicability. We implement our algorithm in an easy-to-understand manner and it is available at this https URL. We hope it will accelerate the research of MCTS-based RL algorithms in the wider community.

This work is supported by the Ministry of Science and Technology of the People’s Republic of China, the 2030 Innovation Megaprojects “Program on New Generation Artificial Intelligence” (Grant No. 2021AAA0150000).

Some have said that poor sample-efficiency on ALE has been a reason to downplay DRL progress or implications. The primary boost in EfficientZero (table 3), pushing it past the human benchmark, is some simple self-supervised learning (SimSiam on predicted vs actual observations).
Effective Evil
by lsusr — 2021-11-02T00:26:29.910Z

Many years ago, a blogger made a post advocating for an evil Y-Combinator which subsidized the opposite of Effective Altruism. Everyone (including the blogger) thought the post was a joke except the supervillains. The organization they founded celebrated its 10th anniversary this year. An attendee leaked to me a partial transcript from one of its board meetings.

Director: Historically, public unhealth has caused the most harm per dollar invested. How is the Center for Disease Proliferation doing?

CDP Division Chief: Gain-of-function research remains—in principle—incredibly cheap. All you have to do is infect ferrets with the flu and let them spread it to one another. We focus on maximizing transmission first and then, once we have a highly-transmissible disease, select for lethality (ideally after a long asymptomatic infectious period).

CFO: You say gain-of-function research is cheap but my numbers say you're spending billions of dollars on gain-of-function research. Where is all that money going?

CDP Division Chief: Volcano lairs, mostly. We don't want an artificial pandemic to escape our labs by accident.

Director: Point of order. Did the CDP have anything to do with COVID-19?

CDP Division Chief: I wish. COVID-19 was a work of art. Dangerous enough to kill millions of people and yet not dangerous enough to get most world governments to take it seriously. After we lost smallpox and polio I thought any lethal disease for which there was an effective vaccine would be eradicated within a year but COVID-19 looks like it would have turned endemic even without the zoonotic vectors. We have six superbugs more lethal than COVID-19 sitting around in various island fortresses. We had planned to launch them this year but with all the COVID-19 data coming in I'm questioning whether that's really the right way to go. Primum non boni. If we release a disease too deadly, governments will stamp it down immediately. We'll kill few people while also training governments in how to stop pandemics. It'd be like vaccinating the planet. We don't want a repeat of SARS.

Director: Good job being quick to change your mind in the face of evidence. What's the status of our AI misalignment program?

Master Roboticist: For several years we've been working on mind control algorithms, but we cancelled that initiative in the face of competition with Facebook. I don't like Facebook. They're not opt...
Great-Filter Hard-Step Math, Explained Intuitively
by Daniel_Eth — 2021-11-01T23:45:52.634Z

Crossposted to the EA Forum.

Introduction

The idea of the great filter was

developed

by Robin Hanson in 1998, building on some

earlier work

by Brandon Carter from 1983. The basic idea is that, since we see a lot of dead matter in the universe, and no good evidence of any technologically advanced, intergalactic civilizations expanding at a substantial fraction of the speed of light, then any proposed pathway by which the former might spontaneously develop into the latter must only ever happen with extremely small probability (given the size of the observable universe, most likely <<10^-20 per planetary system over several billion years).

We can, however, craft a sequence of steps whereby this extremely unlikely process could occur. Such a sequence of steps could include:

Formation of a habitable celestial body (eg., contains liquid water, organics, etc)
Formation or arrival of complex biomolecules
Simple cells (eg., prokaryotes)
Complex cells (eg., eukaryotes)
Efficient meta-improvement to evolutionary algorithm (eg., sexual reproduction)
Multicellularity
Development of cells for efficient information processing (eg., neurons)
Diversification and increased complexity of both the parts for and types of multicellular life (eg., during the Cambrian explosion)
Evolution of a lifeform capable of higher-level reasoning, communication via language, building collective, intergenerational knowledge via cooperation and cumulative culture, and manipulating the physical world via fine-motor control (eg., humans)
Development of mathematics, science, and engineering
Development of advanced, spacefaring technology
Explosive space colonization

Assuming this sequence as a whole is extremely unlikely, at least one of these steps must be very unlikely conditional on having passed all previous steps (ie., it must be a “hard step”), though it’s also possible that some of the steps are “easy steps”, passed with high probability (conditional on having passed the previous steps).

A simple model – “Try-Once” and “Try-Try” Steps

A simple model of this sequence could break down the steps into “try-once” steps (in which there is one chance to pass the step, and failure to pass it is permanent) and “try-try” steps (in which the step can be continuously attempted) which must all be passed in series within a time window equal to the habitability of the celestial body.

From th...
Open & Welcome Thread November 2021
by Ruby — 2021-11-01T23:43:55.006Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
May I ask for some help getting started with crypto? 
by SlainLadyMondegreen — 2021-11-01T21:52:05.236Z

Hey, gang.

There is no way for me to structure this request coherently. I am barbarously ignorant of the subject at hand, so please forgive any instances of clumsy wording you may encounter here or there. This is the only community I trust to give me sound, well-informed, and useful advice.

Here is my situation: I would like to invest in crypto, but my ignorance in this direction is magnificent. Indeed, it feels invincible to me. Furthermore, the very prospect of attempting to defeat this ironclad ignorance has presented itself to me as a somewhat insurmountable "ugh field" for the following reasons:

a) It seems like trading competently in crypto requires some degree of proficiency across multiple fields and practice areas: engineering, software development, data analysis, statistical modeling, cash flow assessment, insight into the psychology behind user behavior, an advanced understanding of economics, etc. I'm sure some of the requisite base knowledge is less self-evident, even, that that.

b) Existing information on the subject of crypto investing is so vast that any effort to consult educational resources on the matter tends to leave me feeling like I've just been washed about helplessly in a stew of concepts and details which seem very convoluted and carry very little meaning to my mind.

c) I don't know where to look for quality information or trustworthy sources.

How can I, starting from square one (complete ignorance), learn to invest wisely in crypto and maximize my returns on investment?

How much time should I reasonably expect to set aside for learning this?

Can anyone suggest quality educational resources for getting started?

What mistakes have you made? What should I avoid?

Also: what should I be investing in at the moment? Please take pity upon my indolent soul.

Any other advice, hacks, and pro-tips that may occur to you would be very much appreciated as well. I have no idea what I'm doing.
What is the most evil AI that we could build, today?
by ThomasJ — 2021-11-01T19:58:38.743Z

I recently came across this post by Kevin Lacker about narrow AI risks. TLDR: There are reasonable routes to apocalyptic AI far before we achieve AGI.

My default position has been extremely skeptical on the risks of AGI. I am generally in agreement with the Andrew Ng quote that "I don’t work on preventing AI from turning evil for the same reason that I don’t work on the problem of overpopulation on the planet Mars." I am still very skeptical of even the Kevin Lacker scenarios, but somewhat less so.

A lot of the AI risk discussion I've seen focuses on hypotheticals, or theories of alignment, or distant future low-probability scenarios.

I'd like to ask for less theoretical ideas that could focus my thinking and perhaps the thinking of the community on more immediate threats.

This brings me to my question to the LW community, which is: What is the most evil AI that could be built, today?

If you were an evil genius with, say, $1B of computing power, what is the most harm you could possibly do to society? In complete seriousness, I think the most harmful AI currently in existence is something like Facebook's user engagement algorithms, or the cameras with software designed to identify minorities and report them to the government. Is there more harmful AI that either currently exists, or would be possible to create?