trentmkelly/LessWrong-43k huggingface dataset:
Tradeoff between desirable properties for baseline choices in impact measures
by Vika — 2020-07-04T11:56:04.239Z

(Cross-posted to personal blog. Summarized in Alignment Newsletter #108. Thanks to Carroll Wainwright, Stuart Armstrong, Rohin Shah and Alex Turner for helpful feedback on this post.)

Impact measures

are auxiliary rewards for low impact on the agent's environment, used to address the problems of side effects and instrumental convergence. A key component of an impact measure is a choice of baseline state: a reference point relative to which impact is measured. Commonly used baselines are the starting state, the initial inaction baseline (the counterfactual where the agent does nothing since the start of the episode) and the stepwise inaction baseline (the counterfactual where the agent does nothing instead of its last action). The stepwise inaction baseline is currently considered the best choice because it does not create the following bad incentives for the agent: interference with environment processes or offsetting its own actions towards the objective. This post will discuss a fundamental problem with the stepwise inaction baseline that stems from a tradeoff between different desirable properties for baseline choices, and some possible alternatives for resolving this tradeoff.

One clearly desirable property for a baseline choice is to effectively penalize high-impact effects, including delayed effects. It is well-known that the simplest form of the stepwise inaction baseline does not effectively capture delayed effects. For example, if the agent drops a vase from a high-rise building, then by the time the vase reaches the ground and breaks, the broken vase will be the default outcome. Thus, in order to penalize delayed effects, the stepwise inaction baseline is usually used in conjunction with

inaction rollouts

, which predict future outcomes of the inaction policy. Inaction rollouts from the current state and the stepwise baseline state are compared to identify delayed effects of the agent's actions. In the above example, the current state contains a vase in the air, so in the inaction rollout from the current state the vase will eventually reach the ground and break, while in the inaction rollout from the stepwise baseline state the vase remains intact.

While inaction rollouts are useful for penalizing delayed effects, they do not address all types of delayed effects. In particular, if the task requires setting up a delayed effect, an agent with the stepwi...
AI-Feynman as a benchmark for what we should be aiming for
by Faustus2 — 2020-07-04T09:24:40.484Z

Very recently, I was made aware of a quite remarkable addition to the world of hobbyist AI in the form of Silviu-Marian Udrescu and Max Tegmarks new AI, 'AI-Feynman'. For those already familiar with the software Eureqa, and its potential to help humanity as a kind of AI-scientist, its a software that is able to form quantitative expressions for observed quantities of data. You feed it observations, it outputs mathematical formulas for the quantity it is attempting to observe. Eureqa isn't free, but if you have the means I strongly suggest examing it more closely in your own time regardless, its sort of that amazing. Richard Carrier has this seperate article on AI safety, but gives Eureqa more attention and detail than I do here: https://www.richardcarrier.info/archives/3195

Now when I saw this, I had largely assumed that the machine learning community would embrace this kind of AI. Why wouldn't we want to have an AI that gave us the form of an answer, instead of a black box that simply did it all for us in its opaque function approximations (neural networks being the primary culprit here). I always regarded the two scenarios as the difference, essentially, between someone giving you the equation you needed on a piece of math homework, and leaving you to figure out why that was correct and build your own understanding, and someone just doing the entire homework for you wholesale and handing in the homework as well so you couldn't even see why what was done was correct. However, as it turned out, most of the proffessional serious machine learning community has done work on things much closer to what I percieve as the latter part of that distinction.

Deep learning is simply the rage, and with good reason, because it can do a LOT. We on Lesswrong, MIRI (and indeed many others outside of these communities) are of course aware of the problem with this (and understanding that problem in a more generalised but also more incorporative framework of existential risk analysis), and so when I actually read Silviu-Marian Udrescu and Max Tegmarks paper (included here: https://arxiv.org/abs/1905.11481 ), to say it was a breath of fresh air is simply an understatement. AI-Feynman is very much the same kind of program as Eureqa, in that it is an AI-scientist that produces quantitative formulas for its observations. It has though, thus far proved to b...
The silence is deafening – Devon Zuegel
by Ben Pace — 2020-07-04T02:30:59.409Z
Link: https://devonzuegel.com/post/the-silence-is-deafening

Imagine you're at a dinner party, and you're getting into a heated argument. As you start yelling, the other people quickly hush their voices and start glaring at you. None of the onlookers have to take further action—it's clear from their facial expressions that you're being a jerk.

In digital conversations, giving feedback requires more conscious effort. Silence is the default. Participants only get feedback from people who join the fray. They receive no signal about how the silent onlookers perceive their dialogue. In fact, they don't receive much signal that onlookers observed the conversation at all.

As a result, the feedback you do receive in digital conversations is more polarized, because the only people who will engage are those who are willing to take that extra step and bear that cost of wading into a messy conversation.

It's a great post, and has a really solid UI idea in the footnotes.

One idea I'd really like to see platforms like Twitter or Reddit try is to provide a mechanism for low-friction, private, negative feedback. For example, you could imagine offering a button where you can downvote or thumbs-down content (i.e. the opposite of a Like), but the count is only visible to the OP and not to anyone else.

The LW team has been thinking about building private responses like this for a while, but in comment form. Buttons that give more constrained private info are very interesting...
Site Redesign Feedback Requested
by Raemon — 2020-07-03T22:28:17.935Z

For the past few months, the LessWrong team has been working on a redesign for the frontpage (which comes with some overall site redesigns).

We've currently got it up on our development branch on lessestwrong.com, and would appreciate some feedback before we roll it out. (This server is for untested changes, you should not use it generally because you might be subject to horrible bugs. But, right now feedback would be helpful)

For logged out users, it looks like this:

And for logged in users:

Goals

There are a few different goals for this. Some of the goals are a bit vague and hard-to-describe. But, some concrete goals that are easy to list for now include:

Make the new Core Tags more visible.

The team is currently making an overall push to finish the Tagging Feature, and get it to a state where users understand it. Having the Core Tags highly visible on the front page helps establish them as a prominent site feature. In addition:

The core tags help new users understand what topics LW tends to focus on.
You can use the Tag Filters to adjust how much content of each tag appears on the frontpage. (Hover over them to see for access)
Note that you can also add new Tag filters (see the "+" button on the right)

Reduce eyestrain and make the frontpage easier to parse

Some people reported eyestrain from the solid white background, which was very bright. The new light-grey background is intended to be a bit softer on the eyes
The current version uses lots of horizontal lines to have divide sections, which adds a bit of clutter. The new version relies instead on the main content naturally standing out via a white background.

Improve clarity of Recent Discussion

Recent Discussion posts are more more distinguished from each other.

Feedback Welcome

Let me know your thoughts in the comments, both re: the overall feel, and how specific UI elements work.

The redesign primarily is intended to affect the frontpage, but it caused some downstream UI tweaks that required other pages to change, and I'm not sure we've finished ironing those changes out. Let us know what we missed.

If you'd like to chat in detail about your experience of the site redesign (or about the new tagging features, or about the site generally) please send me a PM or reply in the comments here and I'll schedule a time.
AI Unsafety via Non-Zero-Sum Debate
by VojtaKovarik — 2020-07-03T22:03:16.264Z

In this post, I describe how to view debate as a way of assisting a human to spot flaws in an AI’s proposal. I then argue that the zero-sum assumption is critical for making debate work and that various seemingly-helpful modifications of debate might break it instead.

A naive way of using arbitrary optimizers as oracles:
Suppose you have a black-box optimizer X that can be connected to any well-defined quantity to be maximized. X can potentially be very powerful - e.g., having a highly accurate model of the world and “a lot of optimization power”.
One way to turn X into an oracle is to ask it a question and decide to give it reward 1 if we like its answer and 0 if we don’t.[1] Of course, standard AI-safety arguments (e.g., AI takeover and perverse instantiation) suggest that this is a pretty bad idea for powerful X. For the sake of argument, suppose that we can fix all of the “obvious” problems and ensure that X won’t wirehead, won’t try to escape the box we put it in etc., and will only care about the reward it gets for its answer.

Two problems with naive optimizers-turned-oracles: (1) telling the difference between good and awesome answers and (2) answers with hidden flaws:
One problem with this type of oracles is that it’s hard to decide whether we like its answers or not. Suppose I ask it for food recommendations for the evening and it suggests pancakes. Pancakes seem fine, although there are some foods that I would like better. So should I reward the AI or not?
The second problem is that the oracle optimizes for giving answers that seem good to a human. (Not out of malice, but because “actually being good” isn’t well-defined.) And since humans aren’t omniscient, there will be many seemingly good answers that in fact have disastrous consequences if acted upon.

To address (1), use two AIs:
The first problem can be tackled by using two copies of the optimizer and rewarding the one whose answer we like better (flipping a coin if indifferent). This helps because humans - or at least the one writing this text - find it much easier to compare the relative strengths of two options than to try assigning some absolute value to a single option. (E.g., chocolate ice-cream is clearly better than vanilla while the question “On a scale from 0 to 10, how good is vanilla?” is just weird.)

To mitigate (2), have a second AI find the flaws for you:
We can decrease the numbe

...
If someone you loved was experiencing unremitting suffering (related to a constellation of multi-dimensional factors and processes, those of which include anomalous states of consciousness and an iatrogenic mental health system), what would you think and what would you do?
by Sara C — 2020-07-03T21:02:34.930Z

Before I get into this more tangibly, I want to clarify that I never intend to make the claim that “I know what would work” or that “I know the way” or that “I blame the institution of medicine for the harm that some incur by engaging with it”. The point that I hope to articulate can be summarized by the following:

Nobody knows (probably) the relative answers to the ill-defined problems that intractable suffering and existential anomalies elicit. I think it would be helpful to acknowledge this unknown terrain so that decision-making powers are not asymmetrically distributed across agents(ex. psychiatrists enact power over others even when clinical uncertainty is extremely high), when the conditions are such that all agents have the same/symmetrical lack of insight.

In light of extreme uncertainty in the case of the chronic catatonic psychosis that my brother experiences (discussed below), institutionalized and bureaucratic entities such as the mental health system are poorly positioned to be useful in his case. My views are informed by my obvious personal experiences in this domain, and also by my professional experiences working in partnership with individuals who are typically constrained (in one way or another) vis a vis the psychiatric or developmental bounds that authoritative professional entities have inscribed upon them. I am willing to wager that, in cases of increasing clinical uncertainty, most institutions of medicine have become so paternal to the extent that, in the name of “keeping people safe”, they are actually perpetuating stagnation and barring many from the pursuit of health, happiness and wellbeing.

In light of this, spaces that employ methodologies endemic to things like engineering and designing, making and tinkering, seem like they could be much more useful here. Again, I don’t know anything for sure, but comparatively, there are some fair claims that I think I can make, and I would risk everything (and be fully responsible for future failures should they happen) to allow for Jules to have the opportunity to try to address this issue some other way. He was recently re-hospitalized after nearly succeeding in taking his life, and in my view, fear of imprisonment was ironically one of the factors that informed his actions.

10 years of the mental health system has resulted in a...
[Crowdfunding] LessWrong podcast
by Mati_Roy — 2020-07-03T20:59:53.590Z

meta: experimental post format

x-post: https://www.facebook.com/groups/LWCoordination/permalink/1002564020177098/ (originally posted on 2020-06-21)

There was 6 posts with >100 karma in the last 30 days on LessWrong. I think we could get someone to audiorecord an article for ~25 USD. We would need ~150 USD / month to sustain that.

How much would you pay per month to have LessWrong posts with >100 karma available in audio format? How many episodes would you be ready to record per month? (on a volunteer basis or not?)

If it adds up to more than 6 episode-equivalent (in time or money) per month, I'll go ahead with the project.

We have a professional mic and good readers at the Macroscope, but if someone else would be interested in doing the reading (for money or as a volunteer), let me know (EtA: on this thread)

Total so far: ~1 episode-equivalent
High Stock Prices Make Sense Right Now
by johnswentworth — 2020-07-03T20:16:53.852Z

I’ve been seeing a lot of comments lately about how the financial markets have gone completely wonky, efficient markets hypothesis looks crazy right now, etc. I don’t currently trade actively and haven’t run a lot of numbers, but just in terms of big-picture qualitative behavior, high stock prices make a lot of sense right now. This post is an informal explanation of why.

First, let’s forget about the efficient market price formula (i.e. price = expected sum of discounted future cash flows, VT=E[∑t>Te−RtTCt]). I’ll talk about that a bit at the end, but it’s so widely and severely misunderstood that I’d need a whole post just to correct misconceptions. Instead, we’ll start from first principles: financial capital is a good, just like any other good. Its price is determined by supply and demand, just like any other good.

When stock prices are high, that means financial capital is cheap for companies: they can get a lot of capital by issuing a lot of stock. High stock price = cheap capital. Likewise with bonds: when bond prices are high, yields are low, meaning companies can borrow capital very cheaply.

What makes the cost of financial capital move? Well, the usual supply-and-demand reasoning:

If people suddenly find themselves with lots of extra savings to invest, that means the supply of financial capital increases, and the cost of financial capital should fall (i.e. stock prices rise).
If people expect lower returns in the future, they will want to invest less, so the supply of financial capital decreases, and the cost of financial capital should rise (i.e. stock prices fall).
If there’s a credit crunch and companies suddenly need to borrow lots of money on short notice, then the demand for financial capital increases, so the cost of financial capital should rise (i.e. stock prices fall).
If many companies are suddenly flush with cash, then the demand for financial capital decreases, so the cost of financial capital should fall (i.e. stock prices rise).

This should all be pretty intuitive, and you can probably brainstorm a few more examples along these lines.

Now, what’s been going on lately, and how does it fit into this picture?

Expectations of future earnings are generally down (although mostly just in the short term). Many companies suddenly need to borrow money in order to stay in business until the storm passes. On their own, these tw...
Splitting Debate up 
into Two Subsystems
by Nandi — 2020-07-03T20:11:11.723Z

In this post I will first recap how debate can help with value learning and that a standard debater optimizes for convincingness. Then I will illustrate how two subsystems could help with value learning in a similar way, without optimizing for convincingness. (Of course this new system could have its own issues, which I don't analyse in depth.)

Debate serves to get a training signal about human values

Debate (for the purpose of AI safety) can be interpreted as a tool to collect training signals about human values. Debate is especially useful when we don’t know our values or their full implications and we can’t just verbalize or demonstrate what we want.

Standard debate set-up

Two debaters are given a problem (related to human values) and each proposes a solution. The debaters then defend their solution (and attack the other’s) via a debate. After having been exposed to a lot of arguments during the debate, a human decides which solution seems better. This judgement serves as the ground truth for the human’s preference (after being informed by the debaters) and can be used as a training signal about what they really want. Through debate we get question-answer pairs which can be used to train a preference predictor.

A debater optimizes for convincingness

An agent will optimize for the goal it is given. In the case of a debater, the agent is optimizing to be judged positively by a human. This means that the main incentive of a debater is to maximise convincingness (potentially by using misleading arguments).

Encouraging the debater to be truthful

The current debate proposal aims to shift this incentive towards truthfulness by empowering both debaters to expose deception in the other debater’s arguments (through cross-examination). (Other methods of disincentivizing deception exist or may exist in the future as well.) I think this disincentivizing may be like a bandaid, when what we want is to prevent the wound in the first place.

A simplistic model of what a debater optimizes for

A debater first comes up with a potential solution. After it has committed to a solution, it tries to convince the judge (through debate) that this is the solution that the judge values most highly. Initially, the AI may reason about what the human’s preference is, but then it tries to persuade the human that the AI’s guess is in fact the...
Research ideas to study humans with AI Safety in mind
by Riccardo Volpato — 2020-07-03T16:01:25.314Z

Premise

Recently I spent some time thinking about ways in which studying the human side of human-machine systems would be beneficial to build aligned AIs. I discussed these ideas informally and people seemed interested and wanted to know more. Thus, I decided to write a list of research directions for studying humans that could help solve the alignment problem.

The list is non-exhaustive. Also, the intention behind it is not to argue that these research directions are more important than any other but rather to suggest directions to someone with a related background or personal fit in studying humans. There is also a lot of valuable work in AI Strategy that involves studying humans, which I am not familiar with. I wrote this list mostly with Technical AI Safety in mind.

Human-AI Research Fields

Before diving into my suggestions for studying humans with AI Safety in mind, I want to mention some less well-known research fields that study the interactions between human and AI systems in different ways, since I reference some of these below. Leaving aside the usual suspects of psychology, cognitive science and neuroscience, other interesting research areas I came across are

Cybernetics

A “transdisciplinary” approach

defined

by Norbert Wiener in 1948 as "the scientific study of control and communication in the animal and the machine". It is currently mostly used as a historical reference and a

foundational reading

. However, there is

growing

work

in integrating cybernetics concepts in current research.

Human-AI Interaction

Human-Computer Interaction (HCI) is an established field dating back to the 70s. It “

studies

the design and use of computer technology, focused on the interfaces between people and computers”. Human-AI Interaction is a

recently established

sub-field of HCI concerned with studying specifically the interactions between humans and “AI-infused systems”.

Computational Social Science

“Using computers to model, simulate, and analyze social phenomena. It

focuses

on investigating social and behavioural relationships and interactions through social simulation, modelling, network analysis, and media analysis”

Collective Intelligence

Defined

as “the enhanced capacity that is created when people work together, often with the help of technology, to mobilise a wider range of information, ideas, and insights”

Artificial Social Intelligence

Which

some

define as “the domain ai

...