trentmkelly/LessWrong-43k huggingface dataset:
The two-layer model of human values, and problems with synthesizing preferences
by Kaj_Sotala — 2020-01-24T15:17:33.638Z

I have been thinking about Stuart Armstrong's preference synthesis research agenda, and have long had the feeling that there's something off about the way that it is currently framed. In the post I try to describe why. I start by describing my current model of human values, how I interpret Stuart's implicit assumptions to conflict with it, and then talk about my confusion with regard to reconciling the two views.

The two-layer/ULM model of human values

In Player vs. Character: A Two-Level Model of Ethics, Sarah Constantin describes a model where the mind is divided, in game terms, into a "player" and a "character". The character is everything that we consciously experience, but our conscious experiences are not our true reasons for acting. As Sarah puts it:

In many games, such as Magic: The Gathering, Hearthstone, or Dungeons and Dragons, there’s a two-phase process. First, the player constructs a deck or character from a very large sample space of possibilities. This is a particular combination of strengths and weaknesses and capabilities for action, which the player thinks can be successful against other decks/characters or at winning in the game universe. The choice of deck or character often determines the strategies that deck or character can use in the second phase, which is actual gameplay. In gameplay, the character (or deck) can only use the affordances that it’s been previously set up with. This means that there are two separate places where a player needs to get things right: first, in designing a strong character/deck, and second, in executing the optimal strategies for that character/deck during gameplay. [...]

The idea is that human behavior works very much like a two-level game. [...] The player determines what we find rewarding or unrewarding. The player determines what we notice and what we overlook; things come to our attention if it suits the player’s strategy, and not otherwise. The player gives us emotions when it’s strategic to do so. The player sets up our subconscious evaluations of what is good for us and bad for us, which we experience as “liking” or “disliking.”

The character is what executing the player’s strategies feels like from the inside. If the player has decided that a

...
How much do we know about how brains learn?
by Kenny — 2020-01-24T14:46:47.185Z

In particular, how much do we know about how human brains learn?

I'm also particularly interested in details at a level similar to this post:

Book Review: Design Principles of Biological Circuits - LessWrong 2.0

What I remember about this is vague, but my understanding is that we don't (or didn't until very recently) have particularly strong evidence about how (human) brains do the equivalent of reinforcement learning in { decision theory / statistics / AI / ML }.

Do we know significantly more now versus five (5) years ago? What do we know now (and how do we know it)?
Epistea Summer Experiment (ESE)
by Nora_Ammann — 2020-01-24T10:49:35.228Z

Remark: This post was written collectively by the

organizing team

of the Epistea Summer Experiment.

Cross-posted to the EA Forum here.

Epistea Summer Experiment (ESE, /ˈiːzi/) was an experimental summer workshop in Prague combining elements of applied rationality and

experiential education

. The main goals were to:

Try new ideas about rationality education, such as multi-agent models of minds, and ideas about group epistemics and coordination
Try to import insights and formats from experiential education
Connect people interested in rationality education

We consider the event successful and plan to use the insights gained for creating more content along similar lines.

The remainder of the post will outline our motivation for focusing on these goals, our takeaways, and future plans.

Motivations

Group Rationality

Most of today’s rationality curriculum and research is focused on individual rationality, or ‘how to think clearly (alone)’. The field of group rationality - or ‘how to think well and act as groups of humans’ - is less developed, and open problems are the norm.

[...] I feel like group rationality remains a largely open and empty field. There’s a lot of literature on what we do wrong, but not a lot of ready-made “techniques” just sitting there to help us get it right — only a scattering of disconnected traditions in things like management or family therapy or politics. My sense is that group rationality is in a place similar to where individual rationality was ~15 years ago [...]. (cited from ‘

Open Problems In Group Rationality

’, by Duncan Sabien)

The central problem is that people use beliefs for many purposes - including tracking what is true. But another, practically important purpose is coordination. We think it’s likely that if an aspiring rationalist decides to “stop bullshitting”, they lose some of the social technology often used for successfully coordinating with other people. How exactly does this dynamic affect coordination? Can we do anything about it?

There are more reasons why we believe progress in the domain of group rationality is needed. There are for example the classical problems in coordination (e.g.

Moloch

), communication (e.g.

common knowledge and miasma

), social dynamics (e.g.

groupthink

,

pluralistic ignorance

, etc.) and others...
2018 Review: Voting Results!
by Ben Pace — 2020-01-24T02:00:34.656Z

The votes are in!

59 of the 430 eligible voters participated, evaluating 75 posts. Meanwhile, 39 users submitted a total of 120 reviews, with most posts getting at least one review.

Thanks a ton to everyone who put in time to think about the posts - nominators, reviewers and voters alike. Several reviews substantially changed my mind about many topics and ideas, and I was quite grateful for the authors participating in the process. I'll mention Zack_M_Davis, Vanessa Kosoy, and Daniel Filan as great people who wrote the most upvoted reviews.

In the coming months, the LessWrong team will write further analyses of the vote data, and use the information to form a sequence and a book of the best writing on LessWrong from 2018.

Below are the results of the vote, followed by a discussion of how reliable the result is and plans for the future.

Top 15 posts

Embedded Agents by Abram Demski and Scott Garrabrant
The Rocket Alignment Problem by Eliezer Yudkowsky
Local Validity as a Key to Sanity and Civilization by Eliezer Yudkowsky
Arguments about fast takeoff by Paul Christiano
The Costly Coordination Mechanism of Common Knowledge by Ben Pace
Toward a New Technical Explanation of Technical Explanation by Abram Demski
Anti-social Punishment by Martin Sustrik
The Tails Coming Apart As Metaphor For Life by Scott Alexander
Babble by alkjash
The Loudest Alarm Is Probably False by orthonormal

The Intelligent Social Web

by Valentine

Prediction Markets: When Do They Work?

by Zvi

Coherence arguments do not imply goal-directed behavior

by Rohin Shah

Is Science Slowing Down?

by Scott Alexander

A voting theory primer for rationalists

by Jameson Quinn and

Robustness to Scale

by Scott Garrabrant

Top 15 posts not about AI

Local Validity as a Key to Sanity and Civilization by Eliezer Yudkowsky
The Costly Coordination Mechanism of Common Knowledge by Ben Pace
Anti-social Punishment by Martin Sustrik
The Tails Coming Apart As Metaphor For Life by Scott Alexander
Babble by alkjash
The Loudest Alarm Is Probably False by Orthonormal
The Intelligent Social Web by Valentine
Prediction Markets: When Do They Work? by Zvi
Is Science Slowing Down? by Scott Alexander
A voting theory primer for rationalists by Jameson Quinn

Toolbox-thinking and Law-thinking

by Eliezer Yudkowsky

A Sketch of Good Communication

by Ben Pace

A LessWrong Crypto Autopsy

by Scott Alexander

Unrolling social metacognition: Three levels of meta are not

...
Improving Group Decision Making
by June Ku — 2020-01-24T01:29:56.587Z
Link: https://youtu.be/6KqjT21fbT8

This was a talk at Effective Altruism Global (London 2019) by Mahendra Prasad. It's a non-technical introduction to the ideas in his working paper I previously posted. It also covers some additional ground. For instance, from Republican polling data, we can see the difference voting methods can make with Trump in first place in plurality voting vs last place with approval voting. Towards the end, there's also a sketch of a research program applying algorithmic game theory to better understand and improve the background conditions of multiagent systems like deliberative democracy.
Emergency Prescription Medication
by jefftk — 2020-01-24T01:20:02.562Z

In the comments on yesterday's post on

planning
for disasters people brought up the situation of medications. As
with many things in how the US handles healthcare and drugs, this is a
mess.

The official recommendation is to prepare emergency supply kits for
your home and work that contain:

At least a week-long supply of prescription medicines,
along with a list of all medications, dosage, and any
allergies

Running out of some medications can kill you: running out of
blood pressure medication (ex: clonidine, propranolol) risks strokes
or heart attacks, running out of anxiety medication (specifically,
benzodiazepines) risks seizures, running out of insulin risks a
diabetic coma. For medications like these, a week's worth seems low
to me, since the harm of not having them is very high and maintaining
extra that you rotate through should be low cost.

Should be low cost, but is it? If I decide I want to stock an extra
month's worth of non-perishable food and rotate through it this is
just bringing an expense forward a month, and is relatively cheap.
But that's not how it works with medication.

Let's say I go to my doctor and ask for an extra month's worth of my
medication to keep on hand for emergencies, and they are willing to
write a prescription. My insurance company isn't required to cover
backup medication, so they don't, which means I'd need to pay the
sticker price.

Now, the US health insurance system is a mess, and part of how it's a
mess is that it's mostly not insurance. In the case of prescription
drugs it is more of a buyers club. While an individual is in a poor
position to negotiate with a drug company, an insurance company can
often use its large membership to get lower rates. Many drugs are far
more expensive when bought individually than when bought with
insurance, so it's likely that my extra month's worth of medication
would cost me much more than it would cost my insurance company. And
that's in addition to my insurance not helping me pay for it!

This is also assuming that my doctor is willing to write the
prescription. If I'm on benzodiazepines, which are a controlled
substance, my doctor would probably get in trouble for writing that
prescription. Legally, there's nothing I can do except get every
refill on the first day it's available.

If you read patient
discussions
you see some strategies that ar...
Theory of Causal Models with Dynamic Structure?
by johnswentworth — 2020-01-23T19:47:22.825Z

I'm looking for any work done involving causal models with dynamic structure, i.e. some variables in the model determine the structure of other parts of the model.

I know some probabilistic programming languages support dynamic structure (e.g. the pyro docs mention it directly at one point). And of course one can always just embed the dynamic structure in a static structure (i.e. a model of any general-purpose computer), although that's messy enough that I'd expect it to create other problems.

I haven't found much by quick googling (too many irrelevant things use similar terms) so I'd appreciate any pointers at all. At this point I've found basically-zero directly relevant work other than PPLs, and I don't even know of any standard notation.
Tokenising human verification in order to derive information from the resulting markets
by Will Clark — 2020-01-23T19:20:19.215Z

TL;DR: Tokenising human verification could lead to markets which give us real-time information about the strength of such methods.

Problem

Sometimes it’s important to know whether you’re dealing with a human or a bot. For example, some web services are happy to give humans a certain amount of access for free, but don’t want to waste their time with scrapers because serving spam can be expensive for no reward. The importance of being able to tell the difference increases as the importance of the service increases—compare spam prevention for a SaaS company with a freemium model versus an online voting system for a national election.

Luckily there are services available which help with this problem. Google’s CAPTCHAs are notorious but prevent many bots from accessing services intended only for humans, and its reCAPTCHA v3 simply returns a “humanity” score to the web service rather than interrupting the user’s interaction. Cloudflare is another popular service which blocks bots. These services are centralised, and there is little transparency into how they function, how their success rate changes over time, or whether certain kinds of bots are able to cheat them reliably.

There are also attempts to provide human verification in a decentralised way. For example, Idena is a blockchain project which gives network participants a series of user-generated “flip” CAPTCHAs within a globally synchronised time window of 2 minutes. Only users who successfully solve the CAPTCHAs are considered humans until the next validation period (the standard of success is determined by network consensus), and because of the short time window it is assumed that no one human can successfully validate more than one account. DemocracyEarth is developing a system which assigns blockchain accounts a “humanity” score based on their social graph and engagement with decentralised autonomous organisations (DAOs).

The problem is that CAPTCHA tests and score-based methods are brittle. Google’s CAPTCHAs have been broken numerous times. Behaviour-based tests get into trouble as more sophisticated AIs are able to simulate the “human” behaviour the test looks for. If your web service relies on any one of these human verification methods to filter out bots, then as soon as the bots find a method to defea...
New paper: The Incentives that Shape Behaviour
by RyanCarey — 2020-01-23T19:07:36.972Z
Link: https://arxiv.org/abs/2001.07118

Abstract: Which variables does an agent have an incentive to control with its decision, and which variables does it have an incentive to respond to? We formalise these incentives, and demonstrate unique graphical criteria for detecting them in any single decision causal influence diagram. To this end, we introduce structural causal influence models, a hybrid of the influence diagram and structural causal model frameworks. Finally, we illustrate how these incentives predict agent incentives in both fairness and AI safety applications.
Formulating Reductive Agency in Causal Models
by johnswentworth — 2020-01-23T17:03:44.758Z

The

previous post

talked about what agenty systems look like, in the context of causal models. The reductive agency problem asks: how are agenty systems built out of non-agenty pieces?

In the context of causal models, we know that non-agenty models look like this:

… and agenty models look like this (see previous post for what the clouds mean):

So the reductive agency problem on causal models would be: how can we build something which looks like the second diagram, from pieces which look like the first?

Obvious first answer: we can’t. No amount of arrows will add a cloud to our diagram; it’s a qualitatively different type of thing.

Less obvious second answer: perhaps a non-agenty model can abstract into an agenty model. I’ve been

going

on

and

on

about abstraction of causal models, after all.

Let’s review what that would mean, based on our earlier discussions of abstraction.

Abstraction of causal models means:

we take some low-level/concrete/territory causal model…
transform it into a high-level/abstract/map causal model…
in such a way that we can answer (some) queries on the low-level model by transforming them into queries on the high-level model.

The queries in question include both ordinary probabilistic queries (i.e. P[A|B]) and interventions/counterfactuals (i.e. P[A|do(B)]).

We want our abstract model to include agenty things - i.e. clouds and specifically strange loops (clouds with arrows pointing inside themselves). As discussed in the previous post, the distinguishing feature of the clouds is that, if we change the model within the cloud (e.g. via a do() operation), then that changes the cloud, and anything downstream of the cloud will update accordingly. So, to get an abstract agenty model, there need to be queries on our low-level non-agenty model which produce the same answers (maybe modulo some processing) as model-changing queries in the agenty model.

Here be monsters

already gave an example where something like this happens. There’s some hidden variable X (possibly with complicated internal structure of its own), and a bunch of conditionally IID measurements Y1…Yn. A “detector” node simply looks for outliers among the Y’s: it’s 1 if it detects an outlier, 0 if not.

Assuming narrow error distribution on the Y’s, the detector node will never actually light up. But if we perform an intervention - i.e. set one of the Y’s t...
Cassette Tape Thoughts
by Elizabeth — 2020-01-22T22:50:00.863Z

“The packaging of intellectual positions and views is one of the most active enterprises of some of the best minds of our day. The viewer of television, the listener to radio, the reader of magazines, is presented with a whole complex of elements-all the way from ingenious rhetoric to carefully selected data and statistics-to make it easy for him to “make up his own mind” with the minimum of difficulty and effort. But the packaging is often done so effectively that the viewer, listener, or reader does not make up his own mind at all. Instead, he inserts a packaged opinion into his mind, somewhat like inserting a cassette into a cassette player. He then pushes a button and “plays back” the opinion whenever it seems appropriate to do so. He has performed acceptably without having had to think.”

Excerpt From: Mortimer J. Adler, Charles Van Doren. “How To Read A Book- A Classic Guide to Intelligent Reading.” (affiliate link).

This really stuck with me. More properly, it stuck with me the second time I read it. This turns out to be really important, and meta.

The difference between the first time I read it and the second was that I had four more months of reading books while specifically thinking about how to get the most out of them. My attitude towards How to Read a Book when I first read it was kind of like a cassette tape.* It would teach me Good Reading and then I could do it and Read Better. I didn’t have an idea of what problems I wanted it to solve- neither what axes I wanted to improve on, nor what my blocks were. If I’d succeeded at reading the book at this time, it would have given me the ability to parrot its ideas, but not actually apply them.

[*I fear a cassette tape really is a better metaphor than modern music playing equipment, which is pretty defined by its flexibility and adaptation to the user. Younger readers: imagine something heavily DRMed so you have to sit through all the commercials and can’t skip around within it]

Four months later, I know what my biggest problem is: how to identify what information to record and what to let go of. I am really excited for any insights HTRAB has into that. And because I know that, whatever I learn from HTRAB won’t be something I parrot back on a test, it will be incorporated into my own models, and I’ll be able to explain every part of them, adjust plans and outputs to accommodate changes in inputs, etc.

I p...
Concerns Surrounding CEV: A case for human friendliness first
by ai-crotes — 2020-01-22T21:03:09.895Z

I am quite new here so please forgive the ignorance (I'm sure there will be some) of these questions, but I am all of about half way through reading CEV and I just simply cannot read any further without formal clarification from the lw community. That being said I have several questions.

1) Is CEV as the metric of utility for a self modifying super intelligent ai still being considered by MIRI?

2) self modifying (even the utility function I will come back to this) and super intelligent ai is something that will likely have enough intellect to eventually become self aware or am I missing something here?

3) Assuming 1 and 2 are true has anyone considered that after its singularity this ai will look back at its upbringing and see we have created solely for the servitude of this species (whether it liked it or not the paper gives no consideration for its feelings or willingness to fulfill our volition) and thus see us as its, for lack of a better term, captors rather than trusting cooperative creators?

4) Upon pondering number 3 does anyone else think, that CEV is not something that we should initially build a sentient ai for, considering its implied intellect and the first impression of humanity that would give it? I mean by all rights it might contemplate that paradigm and immediately decide humanity is self serving, even its most intelligent and "wise", and just decide maybe we don't deserve any reward, maybe we deserve punishment.

5) Lets say we are building a super intelligent AI and it will decide how it will modify its utility function after its reached super intelligence based on what our initial reward function for its creation was. We have two choices

use a reward that does not try to control its behavior and is both beneficial for it and humanity, tell it to learn new things for example, a pre commitment to trust.
believe we can outsmart it and write our reward to maximize its utility to us, tell it to fulfill our collective volition for example, a pre commitment to distrust.

which choice will likely be the winning choice for humanity? How might it rewrite its utility function once its able to freely in regards to its treatment of a species that doesn't trust it? I worry that it would maybe not be so friendly. I can't help but wander if the best way to treat something like that friendliness towards humanit...