trentmkelly/LessWrong-43k huggingface dataset:
Simplified preferences needed; simplified preferences sufficient
by Stuart_Armstrong — 2019-03-05T19:39:55.000Z

AI scientists at EvenDeeperMind: "Hey everyone! We have a developed a low-impact AI!"

AI policy people at OpenFutureofExistentialAI: "Fantastic! What does it do?"

AI scientists: "It's limited to answering questions, and it has four possible outputs, o1, o2, o3, and o4."

AI policy: "What exactly do these outputs do, btw?"

AI scientists: "Well, o1 turns a green light on, o2 turns a red light on, o3 starts a nuclear war, and o4 turns a blue light one."

AI policy: "Starts a nuclear war?!?!?"

AI scientists: "That or turns a yellow light on; I can never remember which...".

Against purely formal definitions of impact measure

It's "obvious" that an AI that starts a nuclear war with one of its four actions, cannot be considered a "low-impact" agent.

But what about one that just turned the yellow light on? Well, what about the utility function Uy, which was +100 if there was no yellow lights in that room during that hour, but was −101010 if there was a yellow light. For that utility function, the action "start a nuclear war" is the low impact action, and even entertaining the possibility of turning on the yellow light is an abomination, you monster.

To which you should answer "Uy is a very stupid choice of utility function". Indeed it is. But it is a possible choice of utility function, so if we had an AI that was somehow "low-impact" for all utility functions, it would be low-impact for Uy.

There are less artificial examples than Uy; a friendly utility function with a high discount rate would find any delay intolerable ("by not breaking out and optimising the world immediately, you're murdering thousands of people in intense agony, you monster!").

Abstract low-impact

This is has been my recurrent objection to many attempts to formalise low-impact in abstract terms. We live in a universe in which every action is irreversible (damn you, second law!) and the consequences expand at light-speed across the cosmos. And yet most versions of low impact - including my own attempts - revolve around some measure of "keeping the rest of the universe the same, and only changing this tiny thing".

For this to make sense, we need to classify some descriptors of the world as "important", and others as "unimportant". And we further need to establish what counts as "small" change to an "important" fact. You can see this as assigning utility functions to the values of the important descriptors, an...
Personalized Medicine For Real
by sarahconstantin — 2019-03-04T22:40:00.351Z

I was part of the founding team at MetaMed, a personalized medicine startup. We went out of business back in 2015. We made a lot of mistakes due to inexperience, some of which I deeply regret.

I’m reflecting on that now, because Perlara just went out of business, and they got a lot farther on our original dream than we ever did. Q-State Biosciences, which is still around, is using a similar model.

The phenomenon that inspired MetaMed is that we knew of stories of heroic, scientifically literate patients and families of patients with incurable diseases, who came up with cures for their own conditions. Physicist Leo Szilard, the “father of the atom bomb”, designed a course of radiation therapy to cure his own bladder cancer. Computer scientist Matt Might analyzed his son’s genome to find a cure for his rare disorder. Cognitive scientist Joshua Tenenbaum found a personalized treatment for his father’s cancer.

So, we thought, could we try to scale up this process to help more people?

In Lois McMaster Bujold’s science fiction novels, the hero suffers an accident that leaves him with a seizure disorder. He goes to a medical research center and clinic, the Durona Group, and they design a neural prosthetic for him that prevents the seizures.

This sounds like it ought to be a thing that exists. Patient-led, bench-to-bedside drug discovery or medical device engineering. You get an incurable disease, you fund scientists/doctors/engineers to discover a cure, and now others with the disease can also be cured.

There’s actually a growing community of organizations trying to do things sort of in this vein. Recursion Pharmaceuticals, where I used to work, does drug discovery for rare diseases. Sv.ai organizes hackathons for analyzing genetic data to help patients with rare diseases find the root cause. Perlara and Q-state use animal models and in-vitro models respectively to simulate patients’ disorders, and then look for drugs or gene therapies that reverse those disease phenotypes in the animals or cells.

Back at MetaMed, I think we were groping towards something like this, but never really found our way there.

One reason is that we didn’t narrow our focus enough. We were trying to solve too many problems at once, all called “personalized medicine.”

Personalized Lifestyle Optimization

Some “personalized medicine” is about health optimization for basically healthy people. A ...
Book Review: The Eureka Factor
by drossbucket — 2019-03-04T19:47:23.483Z

Cross posted from my personal blog.

Last month I finally got round to reading The Eureka Factor by John Kounios and Mark Beeman, a popular book summarising research on 'insightful' thinking. I first mentioned it a couple of years ago after I'd read a short summary article, when I realised it was directly relevant to my recurring 'two types of mathematician' obsession:

The book is not focussed on maths – it’s a general interest book about problem solving and creativity in any domain. But it looks like it has a very similar way of splitting problem solvers into two groups, ‘insightfuls’ and ‘analysts’. ‘Analysts’ follow a linear, methodical approach to work through a problem step by step. Importantly, they also have cognitive access to those steps – if they’re asked what they did to solve the problem, they can reconstruct the argument.

‘Insightfuls’ have no such access to the way they solved the problem. Instead, a solution just ‘pops into their heads’.

Of course, nobody is really a pure ‘insightful’ or ‘analyst’. And most significant problems demand a mixed strategy. But it does seem like many people have a tendency towards one or the other.

I wasn't too sure what I was getting into. The replication crisis has made me hyperaware of the dangers of uncritically accepting any results in psychology, and I'm way too ignorant of the field to have a good sense for which results still look plausible. However, the book turned out to be so extraordinarily Relevant To My Interests that I couldn't resist writing up a review anyway.

The final chapters had a few examples along the lines of '[weak environmental effect] primes people to be more/less insightful', and I know enough to stay away from those, but the earlier parts look somewhat more solid to me. I haven't made much effort to trace back references, though, and I could easily still be being too credulous.

(I didn't worry so much about replication with my previous post on the Cognitive Reflection Test. Getting the bat and ball question wrong is hardly the kind of weak effect that you need a sensitive statistical instrument to detect. It's almost impossible to stop people getting it wrong! I did steer clear of any more dubious priming-style results, though, like the claim that people do better on the CRT when reading it 'in a disfluent font'.)

Insight and intuition

First, it's worth getting clear on exactly what K

...
Finding the variables
by Stuart_Armstrong — 2019-03-04T19:37:54.696Z

In a previous post on bridging syntax and semantics, I mentioned how to empirically establish that the internal symbols Xi represented the variables xi in the environment: if the Xi have high mutual information with the xi. This basically ask whether you can find out about the values of the xi by knowing the Xi. See also Luke Muelhauser's mention of "representation" and the articles linked therein.

At the end of that post, I mentioned the problem of finding the variables Xi if they were not given. This post will briefly look over that problem, and the related problem of finding the xi.

Waterfall and variables in the world

Given the internal variable Xi, it is almost certainly possible to find a variable y in the outside world that correlates with it (even if we assume a Cartesian separation between the agent and the world, so we can't just do the lazy thing and set y=Xi).

In the example of detecting an intruder in a greenhouse, look at Xg, the internal variable of a guard that peers into the greenhouse to see an intruder.

Then we can certainly come up with a variable y that correlates with Xg. This could be a variable that correlates with whether there is an intruder in the greenhouse in situations where the guard can see it, and then correlates with all the issues that might fool the guard: mannequins, delusion-inducing gases, intruders disguised as tables, etc...

But we don't even need y to be anything like the variables that Xg was 'supposed' to measure. If we have a chaotic system in the vicinity - say a nearby waterfall - then we can just list all the states of that system that happen when Xg=0 vs those that happen when Xg=1, and set y to be 0 or 1 in those states.

That is a variant of Scott Aaronson's waterfall argument: if you have enough variety of states, and you can construct definitions of arbitrary complexity, then you can "ground" any model in these definitions. To avoid this, we have to penalise this definitional complexity the definition is doing all the work here, and is itself a highly complicated algorithm to implement.

So pick the xi so that:

the complexity of defining the xi is low, and

the xi have intrinsically relevant definitions, definitions that make sense without direct or indirect knowledge of Xi.

There are some edge cases of course - if a human has Xi being their estimate of whether a swan is around, it might be useful to distinguish ...
IRL 1/8: Inverse Reinforcement Learning and the problem of degeneracy
by RAISE — 2019-03-04T13:11:45.287Z
Link: https://app.grasple.com/#/level/1495

Every Monday for 8 weeks, we will be posting lessons about Inverse Reinforcement Learning. This is lesson 1.

Note that access to the lessons requires creating an account here.

To help students get up to speed, we have also published 2 supplementary lessons that cover prerequisite material:

The POMDP
Bellman equations

Have a nice day!
Syntax vs semantics: alarm better example than thermostat
by Stuart_Armstrong — 2019-03-04T12:43:58.280Z

I had a post on empirically bridging syntax and semantics. It used the example of temperature, building on McCarthy and Searle's dispute about the beliefs of thermostats.

But temperature wasn't an ideal illustration of my points, as humans are not fine in their temperature sensitivity, so I'm presenting a better example here: detecting an intruder.

Internal and external variable

The external variable is a boolean x which corresponds to whether there is any human in a certain initially empty greenhouse.

There are five different "agents" with internal variables X:

An alarm on the door of the greenhouse, which goes off it a circuit is broken by the door being opened; internal variable Xa.

A heat-detecting camera that starts an alarm if there is a something vaguely human sized and human-temperature inside the greenhouse (which is made out of sapphires, obviously); internal variable Xc.

A motivated human guard who periodically looks into the greenhouse; internal variable Xg.

A resourceful human with a lot of time and money, solely dedicated to detecting any intrusion into the green-house; internal variable Xh.

A superintelligent robot version of the resourceful human; internal variable Xr.

Then all the Xi correlate well with the x in a lot of circumstances. If a passerby or a naive burglar get into the greenhouse, they will trigger the door alarm and the heat alarm, while the guard, the resourceful human, and the robot will all see the intruder.

It is, however, pretty easy to fool the door alarm: simply go through a window. Conversely, someone could open the door without entering (or the wind or an earthquake could do so), causing the alarm to trigger with no-one in the greenhouse. So Xa and x are correlated in a relatively narrow set of environments Ea. And if we consider instead the variable y "the electric circuit that goes through the door is unbroken", then it's clear that Xa and y are much better correlated than Xa and x; if there's a semantic meaning to Xa, then it's far closer to y than it is to x.

The heat-camera can also be fooled. Simply spray the lense with some infrared-opaque paint, then enter at your leisure. For the converse, maybe a entering bear could trigger the alarm. It seems clear that Xc is correlated with x in a much wider set of environments, Ec.

The human guard is hard to fool in either direction. We humans are very good at figuring out whe...
Book review: My Hidden Chimp
by Bucky — 2019-03-04T09:55:32.362Z

tl;dr: A narrowly framed introduction to dual process theory for ~5-10 years olds.

I love a dual process theory metaphor as much as the next rationalist (researching this post I came across King Louie and the Apes which I quite like). The Blue minimising robot series completely changed how I thought about myself and cleared up a lot of the confusions that I had about myself (e.g. what I would now call akrasia). I would say that, personally, dual process theory is the most useful thing that I have learnt under the umbrella of rationality.

I have frequently wondered how different my life would have been had I understood myself better at an early age. How would one go about explaining this to a 6 year old?

I have two children (6 and 4) and had tried to explain a bit about the 2 systems to them but with limited success. I particularly thought it would help my eldest as he can struggle behaviourally and experiences extreme emotions but he didn’t fully internalise what system 1 / system 2 meant. Then I heard that Prof Steve Peters had released a children’s book based on his chimp model and I was intrigued enough to get it for him as a Christmas present.

[note that there is a companion book to My Hidden Chimp called The Silent Guides which is aimed more at parents but which I haven’t read]

The Author

I’m not sure how well known Peters is outside the UK but over here he is a minor celebrity. For most people I suspect that if I mentioned Peters by name they wouldn’t know who I was talking about. If I said “You know, that sports psychiatrist who works with cyclists, the chimp guy” then they probably would.

He is probably most famous for his work with British cycling as part of their programme of incremental gains. Most recently was in the news for being credited by Victoria Pendleton for preventing her suicide.

He has also worked with Ronnie O’Sullivan which was where I first noticed his work. Prior to working with Peters, O’Sullivan struggled with drink, drugs and depression but in recent years has turned this around, giving Peters a lot of the credit.

He is also an age group world champion in 100, 200 and 400m.

I’m not his publicist, honest...

The Chimp Model

The chimp model is Peters’ dual process metaphor. Different writers and models seem to me to have subtly different focuses - e.g. Robin’s Elephant and rider focuses on motivation, Kahneman’s system 1 and system 2 on deci...
Alignment Newsletter #47
by Rohin Shah — 2019-03-04T04:30:11.524Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter.

Highlights

AI Safety Needs Social Scientists

(Geoffrey Irving et al) (summarized by Richard): One approach to AI safety is to "ask humans a large number of questions about what they want, train an ML model of their values, and optimize the AI system to do well according to the learned values". However, humans give answers that are limited, biased and often in disagreement with each other, and so AI safety needs social scientists to figure out how to improve this data - which eventually may be gathered from thousands or millions of people. Of particular importance is the ability to design rigorous experiments, drawing from an interdisciplinary understanding of human cognition and behaviour. The authors discuss

Debate

(

AN #5

) as a case study of a safety technique whose success depends on empirical questions such as: how skilled are humans as judges by default? Can we train people to be better judges? Are there ways to restrict debate to make it easier to judge?

There are a couple of key premises underlying this argument. The first is that, despite human biases, there are correct answers to questions about human values - perhaps defined as the answer we would endorse if given all relevant information and unlimited time to think. However, it’s not necessary for AIs to always find those answers, as long as they are able to recognise cases in which they’re uncertain and do nothing (while there are some cases in which inaction can cause harm, such as a self-driving car ceasing to steer mid-journey, it seems that the most worrying long-term catastrophes can be avoided by inaction). Another reason for optimism is that even incomplete or negative results from social science experiments may be useful in informing technical safety research going forward. However, in some cases the systems we're trying to reason about are very different from anything we can test now - for example, AI debaters that are much stronger than humans.

Richard's opinion: This post, and its accompanying paper, seems very sensible to me. While I have some doubts about how informative human debate data will be about superhuman debaters, it certainly seems worth trying to gain more empirical information. Note that while...
How much funding and researchers were in AI, and AI Safety, in 2018?
by Raemon — 2019-03-03T21:46:59.132Z

I'm trying to build up a picture of how "much" research is going into general AI capabilities, and how much is going into AI safety.

The ideal question I'd be asking is "how much progress [measured in "important thoughts/ideas/tools" was being made that plausibly could lead to AGI in 2018, and how much progress was made that could plausibly lead to safe/aligned AI].

I assume that question is nigh impossible, so instead asking the approximation:

a) how much money went into AI capabilities research in 2018

b) how much money went into AI alignment research in 2018

c) how many researchers (ideally "research hours" but I'll take what I can get) were focused on capabilities research in 2018

d) how many researchers were focused on AI safety in 2018?
Programmatic Code Generation: Composability
by Martin Sustrik — 2019-03-02T22:50:06.865Z

After working some more with Tiles, my mini-library for programmatic code generation, I've realized it has a feature that I haven't knowingly baked in, but which makes it a really good tool for the job. Namely, the code you write is composable.

Here's what I mean: If you want to generate a C function that prints out names of countries' capitals then you can generate the code that does the actual printing first:

data = { "Afghanistan": "Kabul", "Brazil": "Brasilia", "Canada": "Ottawa", } body = t/'' for country, capital in data.items(): body |= t/""" printf("@{capital} is the capital of @{country}"); """

Once done, you cen generate the scaffolding:

program = t/""" #include <stdio.h> int main(void) { @{body} return 0; } """

The example, of course, shows just two levels of nesting. In practice, however, you'd have many levels of nesting.

But whatever the nesting level, you, as a programmer, can focus on a single task at a time. First, you think about how to print the capitals. Then you think about the scaffolding. There's no overlap.

Compare that to your typical templating solution:

template = Template(""" #include <stdio.h> int main(void) { {% for country, capital in data.items() %} printf("{{ capital }} is the capital of {{ country }}"); {% endfor %} return 0; } """) template.render(data=data)

Note how you start with writing scaffolding (include, main) then you switch to the business logic (printf), then you get back to scaffolding (return).

The example is trivial and as such, the both solutions are kind of comparable.

However, imagine there are five levels of nesting.

The code written using Tiles would, logically, look like this:

level1 level2 level3 level4 level5

The code written using one the classic templating tools, on the other hand, would look like this:

level1 level2 level3 level4 level5 level4 level3 level2 level1

March 2nd, 2019

by martin_sustrik
To understand, study edge cases
by Shmi — 2019-03-02T21:18:41.198Z

This seems obvious, but is often forgotten. When a system functions efficiently, its individual parts are hard to notice. If you want to understand how a system functions, what it is made of, how each part works, but you cannot easily disassemble it and have a look, you have to get out of the nominal regime and let each part function independently, as much as possible.

Some examples:

To understand subatomic particles in a nucleus, or to figure out the structure of atom, hit them hard with other particles and see what happens.
To understand free will, focus on the situations where the notion fails.
To understand consciousness, study altered states.
To understand mental health, study mental illness.
To understand anatomy, dissect cadavers.
To understand logical reasoning, study biases.
To understand evolution, study artificial selection.
To understand economics, study market failures.
Subagents, introspective awareness, and blending
by Kaj_Sotala — 2019-03-02T12:53:47.282Z

In this post, I extend the model of mind that I've been building up in previous posts to explain some things about change blindness, not knowing whether you are conscious, forgetting most of your thoughts, and mistaking your thoughts and emotions as objective facts, while also connecting it with the theory in the meditation book The Mind Illuminated. (If you didn't read my previous posts, this article has been written to also work as a stand-alone piece.)

The Mind Illuminated (

Amazon

,

SSC review

), or TMI for short, presents what it calls the moments of consciousness model. According to this model, our stream of consciousness consists of a series of discrete moments, each a mental object. Under this model, there are always different “subminds” which are projecting mental objects into consciousness. At different moments, different mental objects get selected as the content of consciousness.

If you’ve read some of the previous posts in this sequence, you may recognize this as sounding familiar. We started by discussing some of the

neuroscience research on consciousness

. There we covered the GWT/GNW theory of consciousness being a “workspace” in the brain that different brain systems project information into, and which allows them to synchronize their processing around a single piece of information. In the next post, we discussed the

psychotherapy model of Internal Family Systems

, which also conceives the mind of being composed of different parts, many of which are trying to accomplish various aims by competing to project various mental objects into consciousness. (TMI talks about subminds, IFS talks about parts, GWT/GNW just talks about different parts of the brain; for consistency’s sake, I will just use “subagent” in the rest of this post.)

At this point, we might want to look at some criticisms of this kind of a framework. Susan Blackmore has written an interesting paper called “

There is no stream of consciousness

”. She has several examples for why we should reject the notion of any stream of consciousness. For instance, this one:

For many years now I have been getting my students to ask themselves, as many times as possible every day “Am I conscious now?”. Typically they find the task unexpectedly hard to do; and hard to remember to do. But when they do it, it has some

...