trentmkelly/LessWrong-43k huggingface dataset:
Is epistemic logic useful for agent foundations?
by Richard_Ngo — 2018-05-08T23:33:44.266Z

The title isn't a rhetorical question; I'm actually looking for answers. This summer, I'll have the opportunity to attend a summer school on logic, language and information. Whether or not I go depends to a significant extent on whether what they'll be teaching - particularly epistemic logic, also some other topics in logic and language - will be useful for AI safety research. Here is a summary of epistemic logics, and here are the courses I'll be able to take. I've already taken classes in first-order logic, but right now I'm uncertain about the value of doing these extra courses.

Reasons to think learning epistemic logic will be useful for agent foundations:

MIRI's work relies heavily on high-level concepts in logic
Epistemic logic is particularly concerned with statements about knowledge and belief which seem very relevant to reasoning about agents
Learning about epistemic logic is probably useful for thinking about other forms of logic

Reasons to think it won't be useful:

As far as I can tell, it doesn't appear on MIRI's research guide, nor in any of their papers, nor in the sequences
It seems like epistemic logic is mostly non-probabilistic and is talking about a fundamentally different sort of knowledge from probabilistic Bayesian knowledge, which increases my credence that it's the sort of philosophy which isn't going to be of much practical use
Temporarily Out of Office
by Ben Pace — 2018-05-08T21:59:55.021Z

Just an update that for the next 5 weeks I personally am not working on LW, while I graduate (prepare and take finals), and Oliver is not working for ~3 weeks while he does the same. Ray will be god-of-LW until then. #RayOfSunshine

(I will be around because LW is one of the main things I habitually check in my free time, but I won't be able to do sustained work or drop everything at a moment's notice.)
On Better Mental Representations Part I: Adopting 'Thinking Tools' 
by Will Lorca — 2018-05-08T12:45:06.724Z

If you don't mind an introduction using some poppy philosophical thinking, please indulge me as I channel Alan Watts. Correct me if I am wrong but if I remember Watts once spoke or written about how we misconstrue our representations of things for the actual things. In extension of that, we can also say (as we measure) that we also misconstrue our 'valuation' of things to be the things themselves.

Watts gave out a quite clever example of this which involves wood (natural resource) and inches (measurement). Let me roughly paraphrase through this anecdote (adding some creative flair or more honestly winging it without watering down the gist).

There's a foreman and his men building a house. They had all the natural resources that need be there to build an adequate house for a small family of four. But for some "price shock" in the "Platonic" realm, the currency of measurement by inches ran out of circulation.

The foreman told his workers. "Sorry, fellas. We can't finish building the house anymore. We have ran out of inches."

"What do you mean? We still have planks, nails, tools like hammers, and enough man power to finish building the house.", answered a carpenter amongst the team.

The foreman replied, "Sorry, Edward. You don't understand. The world has ran out of inches."

Ludicrous isn't it? This is also how many of us think about the "value" of things. We think of them in monetary value. Inches, like money, is a representation of something: the length of some thing. It is not the thing itself. It is a measure.

It's the same thing as money. It's something that has no intrinsic value. We only give value to it. We internally represent in our minds that it has value.

I am not saying money is bad, useless, or the root of all evil, Jesus forbid. I'm just saying that it is similar to what the "inches" there serves in the anecdote. It serves, almost, as the end all be all of valuation. We, often times, confuse our representations of things to be the things themselves; i.e. we mistake concepts of things to be the things themselves.

But this is to be expected. We do think of things as how they are in our representations. This is all we can ever do. It is only from these representations that we can glean into reality. They both enable and limit our capac...
Varieties Of Argumentative Experience
by Scott Alexander — 2018-05-08T08:20:02.913Z

In 2008, Paul Graham wrote How To Disagree Better, ranking arguments on a scale from name-calling to explicitly refuting the other person’s central point.

And that’s why, ever since 2008, Internet arguments have generally been civil and productive.

Graham’s hierarchy is useful for its intended purpose, but it isn’t really a hierarchy of disagreements. It’s a hierarchy of types of response, within a disagreement. Sometimes things are refutations of other people’s points, but the points should never have been made at all, and refuting them doesn’t help. Sometimes it’s unclear how the argument even connects to the sorts of things that in principle could be proven or refuted.

If we were to classify disagreements themselves – talk about what people are doing when they’re even having an argument – I think it would look something like this:

Most people are either meta-debating – debating whether some parties in the debate are violating norms – or they’re just shaming, trying to push one side of the debate outside the bounds of respectability.

If you can get past that level, you end up discussing facts (blue column on the left) and/or philosophizing about how the argument has to fit together before one side is “right” or “wrong” (red column on the right). Either of these can be anywhere from throwing out a one-line claim and adding “Checkmate, atheists” at the end of it, to cooperating with the other person to try to figure out exactly what considerations are relevant and which sources best resolve them.

If you can get past that level, you run into really high-level disagreements about overall moral systems, or which goods are more valuable than others, or what “freedom” means, or stuff like that. These are basically unresolvable with anything less than a lifetime of philosophical work, but they usually allow mutual understanding and respect.

I’m not saying everything fits into this model, or even that most things do. It’s just a way of thinking that I’ve found helpful. More detail on what I mean by each level:

Meta-debate is discussion of the debate itself rather than the ideas being debated. Is one side being hypocritical? Are some of the arguments involved offensive? Is someone being silenced? What biases motivate either s...
Turning 30
by Vaniver — 2018-05-08T05:37:45.001Z

I'm typically not a big fan of birthdays, as traditions go, but something about reaching a new decade makes it seem perhaps worthy of a bit more attention.

Especially given the stark contrast between the long view of looking a decade back and a decade ahead, and my present uncertain circumstances. I can tell the broad tale of ten years of a career, but can't tell you whether the interview I'm currently going through will result in me changing positions or not, which seems fairly relevant for determining what the next ten years looks like. I can detail ten years of moves from apartment to apartment, but can't predict whether I will be in my current place in a year, let alone ten. A conversation yesterday established that I am 'dating' a particular fellow instead of just 'going on dates' with him, which I couldn’t reliably have predicted the day before. As he is sometimes fond of pointing out, a solid strategy is predicting that things will last as long as they've survived so far, which is perhaps not the most optimistic projection for our formal relationship.

But there’s also this confidence in landing on my feet that I don’t think 20-year old me had; if this role doesn’t work out, there will be another one; if this housing situation doesn’t work out, there will be another one; if this relationship doesn’t work out, that’d be fine even if there weren’t another one. I do remember having some confidence in this regard, but not nearly as much; it was only about two years ago when I stopped caring about whether or not things would look good on my resume, since it was no longer the limiting factor in getting interviews where I wanted to.

And such temporary disturbances get smoothed out when looked at from far enough away. So let us consider the Vaniver of May 7th, 2008. Finishing his second year of undergrad in Maryland, he's living in his second dorm room, working as an undergraduate research assistant (I believe already in his second lab), doing well in classes, posting on the xkcd fora, was following the Ron Paul campaign (which was clearly unsuccessful but not yet suspended), had already made his first major romantic gesture to R, and been to his first convention (where he made a fool of himself in front of R). He had recently switched to primarily using his gmail account, registered in 2006 (I believe) after he...
Problems integrating decision theory and inverse reinforcement learning
by agilecaveman — 2018-05-08T05:11:06.920Z

In this post I consider a single hypothetical which potentially has far-reaching implications for the future of AI development and deployment. It has to do with a complex interactions between the assumptions of which decision theory humans use and the method used to infer their values, such as something like an inverse reinforcement learning algorithm.

Consider the Newcomb’s problem. We have two boxes, box A and box B. Box B always has $1000. Box A has $1,000,000 if and only if a near perfect predictor Omega predicts that the agent picks only Box A. We have two agents: agent1, who one boxes (it’s an FDT agent) and agent2 who two-boxes (a CDT agent). In addition to Omega, there is an inverse reinforcement learner (later abbreviated as IRL) trying to infer the agent’s “values” from it’s behavior.

What kinds of reward signals does the IRL assume that agent1 or agent2 have? I claim that in the simplistic case of just looking at two possible actions, it will likely assume that agent1 values the lack of money because it fails to pick box2. It will correctly deduce that agent2 values money.

In effect, a naïve IRL learner assumes CDT as the agent’s decision theory and it will fail to adjust to learning about more sophisticated agents (including humans).

This depends a little bit on the setup of the IRL agent and the exact nature of the states fed into it. I am generally looking at the following setup of IRL Since we have a finite state and action space, the IRL learner simply tries to pick a hypothesis set of reward functions which place the highest value on the action taken by agent compared to other actions.

This also depends on the exact definition of which “actions we are considering. If we have potential actions of “pick one box” or “pick two boxes,” the IRL agent would think that agent1’s preferences are reversed from the its actual preferences.

This is bad, very extremely bad, since even the opposite of the utility function is now in the hypothesis set.

If, for example, we have three actions of “pick one box”, “pick two boxes” or “do nothing,” then the preference of “pick one box” over “do nothing” removes the reverse of agent1 reward function from the hypothesis set. It, however does not put the reward function of “maximize money” into the hypothesis set.

I can think of several objections / fixes to the problem existing, but I don’t think they make the p...
Introducing the Longevity Research Institute
by sarahconstantin — 2018-05-08T03:30:00.768Z

I’ve just founded a nonprofit, the Longevity Research Institute — you can check it out here.

The basic premise is: we know there are more than 50 compounds that have been reported to extend healthy lifespan in mammals, but most of these have never been tested independently, and in many cases the experimental methodology is poor.

In other words, there seems to be a lot of low-hanging fruit in aging. There are many long-lived mutant strains of mice (and invertebrates), there are many candidate anti-aging drugs, but very few of these drugs have actually been tested rigorously.

Why? It’s an incentives problem. Lifespan studies for mice take 2-4 years, which don’t play well with the fast pace of publication that academics want; and the FDA doesn’t consider aging a disease, so testing lifespan isn’t on biotech companies’ critical path to getting a drug approved. Mammalian lifespan studies are an underfunded area — which is where we come in.

We write grants to academic researchers and commission studies from contract research organizations. Our first planned studies are on epitalon (a peptide derived from the pineal gland, which has been reported to extend life in mice, rats, and humans, but only in Russian studies) and C3 carboxyfullerene (yes, a modified buckyball, which prevents Parkinsonism in primate models and has been reported to extend life in mice). I’m also working on a paper with Vium about some of their long-lived mice, and a quantitative network analysis of aging regulatory pathways that might turn up some drug targets.

We’re currently fundraising, so if this sounds interesting, please consider donating. The more studies that can be launched in parallel, the sooner we can get results.
On Gender and Emotional Support
by squidious — 2018-05-07T22:37:25.254Z

Note: I wrote this outside the context of this community, but I think it's relevant enough to crosspost here.

Some of the first questions many therapists ask a client are those meant to assess their friendships and other close relationships. This is because having a supportive community is one of the biggest strengths in those facing mental health problems. Whether it's Generalized Anxiety Disorder or grief over a breakup, having people you can lean on for emotional support can help a lot.

Women are more likely to have such social supports than men. Part of this is due to traditional gender roles and expectations. Emotional vulnerability is considered feminine, and thus more acceptable for women than men. Of course, anger is an exception here, given that it is linked with violence, which is linked with masculinity. So men are more likely to come in with anger issues, while women are more likely to come in with anxiety and depression.

The second part of this phenomenon is the difference between male and female friendships. Because emotional vulnerability is acceptable in women, it becomes an important part of the way women connect with each other. You can see this in media directed at teen girls: they support each other at best and tear each other down at worst, but it's all based in their ability to be vulnerable with each other. The trope of the girl giving her friend a makeover isn't just about looks; it's about self-esteem. The image of a group of friends watching a sappy romance movie and eating ice cream after one of them was dumped isn't just about being there for the friend; it's about empathizing and validating her emotions.

So what happens when you remove the emotional vulnerability from a close friendship? You get something close to what male friendships look like: a connection based in mutual interests, activities, and practicality. While it's not uncommon for two women with very little in common to become friends, this is far more rare with men. A guy's best friend may be someone they go to the gym with, someone they sit next to at work, or someone who reads the same books they do. A girl's best friend is likely to be someone who knows her more deeply than anyone else.

Of course, this is not to say women don't have shallow friendships and men don't have deep ones. Even media sometimes portray...
What are Trigger-Action Plans (TAPs)?
by Lulie — 2018-05-07T18:28:58.550Z

Disclaimer: This is a technique I learnt at CFAR (which said it's backed by a large amount of Science!, but I haven't looked into original sources much). This post is to explain TAPs simply, including for people unfamiliar with CFAR/LW, with a quick 'getting started' guide. It is not intended to be a complete account nor thoroughly researched. For a more data-driven account, see this post by Kaj_Sotala.

Trigger-Action Planning is a technique for changing and creating habits that are totally automatic and don't require conscious thought.

The basic idea is that you tie the habit to a specific, simple, concrete thing that activates the 'do this habit' program in your brain.

trigger - The simple, specific sight/sound/smell/thought/feeling/etc. which you hook a behaviour onto.

action - The behaviour you enact on contact with the trigger.

plan - A deliberate plan to create the above pattern of trigger-action.

pattern - A pattern of trigger-action, whether deliberate or not, sometimes created unconsciously.

(TAP can mean either Trigger-Action Plan or Trigger-Action Pattern.)

Automatic Habits

It's far easier to change habits when you don't need to think about them. If we try to use our slow, conscious thinking to do a desired habit, it's effortful, we often forget, and it takes up attention we could be using elsewhere. Automating habits makes actually doing them effortless and reliable.

Fortunately, human thinking appears to be designed for this kind of automation. (For more on this, see Thinking, Fast and Slow by Daniel Kahneman.)

Trigger-action patterns occur naturally in everyday life. They can be useful, such as [Finish lathering hair → wash shampoo out] or [Close front door → lock door], or they can be detrimental, such as [Feel restless → comfort-eat].

Understanding how they work means you can intervene and break undesirable patterns, or create your own patterns.

Creating a TAP

1. Pick a goal.

A desired habit or behaviour change. Usually, we make TAPs in order to move the needle a little, or give a pause to remember the desired behaviour, rather than completely overhaul entrenched habits in one fell swoop.

It is possible to chain TAPs together to build up a more complicated automatic behaviour. You probably have such a chain already, such as when showering or driving to work.

But this can take time to build up. When yo...
The Alignment Newsletter #5: 05/07/18
by Rohin Shah — 2018-05-07T16:00:11.059Z

Highlights

AI safety via debate

(Geoffrey Irving et al): At a high level, a major issue with building superintelligent AI is that humans would not be able to provide strong oversight for the AI. Amplification solves this by using the AI as a tool that can help the human (in particular, if the human can break a task down into subtasks, the AI can solve the subtasks). Debate also provides the AI as a tool for human overseer, but in a different way -- now, in order to train the AI, we have the AI debate against itself in order to convince a human of the answer to some target question. Given some question whose answer is too hard to directly judge, the human can look at the arguments and counterarguments to figure out whether or not the answer is actually correct.

The paper describes debate in a lot more depth and has an initial experiment involving MNIST. I can't possibly do it justice here -- I encourage you to simply read the full paper. You probably have an intuition right now of why this wouldn't work, such as "but humans believe what they want to hear, not what is true". The paper spends 5 (!) pages listing ten such problems and analyzing them, so go read it.

My opinion: It's great to see another approach that directly tackles the problem of defining a training signal that if optimized well would lead to an aligned AI. There are a lot of empirical questions that would influence whether or not debate actually works in practice, and I'm excited to see what experiments find.

AGI Safety Literature Review

(Tom Everitt et al): Self-explanatory. It's more of a list of approaches and references within each approach than an integrated whole, but I still expect it to be useful.

My opinion: This is great as a way to find references. I do wish there was more comparison between papers and/or approaches, but that's probably asking too much.

No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling

(Xin Wang, Wenhu Chen et al): This paper tackles visual story-telling, the task of generating a story that matches a sequence of photos. It proposes learning a reward function from the labeled dataset that can then be optimized with reinforcement learning, with the hope that the reward function is a good compression of what we want and so leads to more generalizable behavior. They show that the standard automated techniques f...
Philosophy of Mind: Umwelt, Information, Value, and Information-Value
by Will Lorca — 2018-05-07T11:55:54.221Z

Jakob Von Uexkull, like many thinkers, introduced a new meaning for a familiar term. The German biologist used the everyday German word 'umwelt' which means surrounding environment to be an organisms inner world. It's some sort of a subjective representation of things outside the 'body' according to what Dennett calls an organism's 'affordances'.

Affordances are things that an organism can hide in, sit on, fly to, mate with, and others. These are, firstly, the things that matter most to an organism.

Uexkull discussed his concept of the Umwelt through the tick. In the tick's internal representation of the world, there are just (mostly) a few things that matters. Imagine a tick hanging from a tip of a branch in a forest clearing. In it's umwelt are three receptor cues and effector cues. Roughly, these are: (1) the detection of butyric acid that triggers it to let go of its hold to fall on the skin of a mammal, the source of this odor, (2) the tactile trigger when it lands to go and run, and (3) the detection of heat that after a bout of maneuvering on the topology of an organism will trigger it to bore and burrow.

It's biological configuration 'bottlenecks' all the "colorful stimuli" of its immediate environment into "three beacons of light". This is what it matters to it to be fit in an evolutionary sense. This 'minimizing of cues' by its biological configuration or state allows it to survive long enough to sire young; maximizing the chances of its genes getting to the next generation.

These "approximating" apparatuses afforded by the tick's biological configuration leans the odds of it to be evolutionary successful. The minimizing or the bottlenecking of the things that matters most in the environment pays dividends. This doesn't violate the scientific maxim that evolution favors the efficient. It just makes do.

An agent, the tick here, can only detect and act upon what is valued by its very biological configuration.

Philosophically, we can extract that first (1) we, agents like the tick, are just approximating through bio-approximators endowed by evolution (senses are part of them), and second (2) without the biological configuration, organisms can't create this inner world. There is no umwelt without the body which generates it. This, in ...