trentmkelly/LessWrong-43k huggingface dataset:
Ought: why it matters and ways to help
by paulfchristiano — 2019-07-25T18:00:27.918Z

I think that Ought is one of the most promising projects working on AI alignment. There are several ways that LW readers can potentially help:

They are recruiting a senior full-stack web developer.
They are recruiting participants for "factored evaluation" experiments.
I think donors interested in AI safety should consider Ought.

In this post I'll describe what Ought is currently doing, why I think it's promising, and give some detail on these asks.

(I am an Ought donor and board member.)

Factored evaluation

Ought's main project is currently designing and running "factored evaluation" experiments, and building relevant infrastructure. The goal of these experiments is to answer the following question:

Consider a principal who wants to incentivize experts to work on hard problems, but finds that the work is too complex for the principal to tell what is good or bad. Can they overcome this problem by dividing the evaluation task into smaller pieces, delegating each of them to other experts, and recursively incentivizing good work on those pieces using the same mechanism?

Here's what an experiment looks like:

Recruit a pool of "judges," "helpful experts," and "malicious experts."
Start with a question that is too difficult for the judge to answer.

In initial experiments, the questions are difficult because the judge lacks relevant context. For example, the judge may want to answer a question about a long article, while only being allowed to look at a few hundred characters of text.
Limiting the amount of text the judge can read is an artificial restriction. This is intended as a warm-up, an easier analogy for the situation where the experts have knowledge that cannot be explained to a single judge in a reasonable amount of time (or where we want to reduce the cost of judging).

An "honest expert" provides a good answer to that question, a "malicious expert" provides a bad answer to that question, and a judge is tasked with telling which answer was better. The answers themselves may be too large and complex for the judge to look at, in which case they'll be handed a compact representation of each answer which they can inspect as needed.
In order to tell which answer is better, the judge asks subquestions. For example, they might ask "What's the best reason to think that answer

...
On the purposes of decision theory research
by Wei Dai — 2019-07-25T07:18:06.552Z

Following the examples of Rob Bensinger and Rohin Shah, this post will try to clarify the aims of part of my research interests, and disclaim some possible misunderstandings about it. (I'm obviously only speaking for myself and not for anyone else doing decision theory research.)

I think decision theory research is useful for:

Gaining information about the nature of rationality (e.g., is “realism about rationality” true?) and the nature of philosophy (e.g., is it possible to make real progress in decision theory, and if so what cognitive processes are we using to do that?), and helping to solve the problems of normativity, meta-ethics, and metaphilosophy.

Better understanding potential AI safety failure modes that are due to flawed decision procedures implemented in or by AI.

Making progress on various seemingly important intellectual puzzles that seem directly related to decision theory, such as free will, anthropic reasoning, logical uncertainty, Rob's examples of counterfactuals, updatelessness, and coordination, and more.

Firming up the foundations of human rationality.

To me, decision theory research is not meant to:

Provide a correct or normative decision theory that will be used as a specification or approximation target for programming or training a potentially superintelligent AI.

Help create "safety arguments" that aim to show that a proposed or already existing AI is free from decision theoretic flaws.

To help explain 5 and 6, here's what I wrote in a previous comment (slightly edited):

One meta level above what even UDT tries to be is decision theory (as a philosophical subject) and one level above that is metaphilosophy, and my current thinking is that it seems bad (potentially dangerous or regretful) to put any significant (i.e., superhuman) amount of computation into anything except doing philosophy.

To put it another way, any decision theory that we come up with might have some kind of flaw that other agents can exploit, or just a flaw in general, such as in how well it cooperates or negotiates with or exploits other agents (which might include how quickly/cleverly it can make the necessary commitments). Wouldn’t it be better to put computation into trying to find and fix such flaws (in other words, coming up with better decision theories) than into any particular object-level decision theory, at least until the superhuman philosophical co

...
Metaphorical extensions and conceptual figure-ground inversions
by jessicata — 2019-07-24T06:21:54.487Z
Link: https://unstableontology.com/2019/07/24/metaphors-and-conceptual-figure-ground-inversions/

Consider the following sentence: "A glacier is a river of ice."

This is metaphorical. In some sense, a glacier isn't actually a river. A "literal" river has flowing liquid water, not ice.

Let a river(1) be defined to be an archetypal flowing-water river. A glacier isn't a river(1). Rather, a glacier shares some structure in common with a river(1). We may define river(2) to mean some broader category, of things that flow like a river(1), such as:

A glacier

A flowing of earth matter in a landslide

A flowing of chemicals down an incline in a factory

and so on.

A river(2) is a concept by metaphorically extending river(1). It is, in fact, easier to explain the concept of river(2) by first clearly delineating what a river(1) is. A child will have trouble grasping the metaphorical language of "a glacier is a river of ice" until understanding what a river(1) is, such that the notion of flow that generates river(2) can be pointed to with concrete examples.

Formally, we could think of metaphorical extensions in terms of generative probabilistic models: river(2) is formed by taking some generator behind river(1) (namely, the generator of flowing substance) and applying it elsewhere. But, such formalization isn't necessary to get the idea intuitively. See also the picture theory of language; language draws pictures in others' minds, and those pictures are formed generatively/recursively out of different structures; see also generative grammar, the idea that sentences are formed out of lawful recursive structures.

Is ice a form of water?

Consider the sentence: "Ice is a form of water."

What does that mean? Suppose that, by definition, ice is frozen water. Then, the sentence is tautological.

However, the sentence may be new information to a child. What's going on?

Suppose the child has seen liquid water, which we will call water(1). The child has also seen ice, in the form of ice cubes; call the ice of ice cubes ice(1). It is new information to this child that ice(1) is a form of water(1). Concretely, you can get ice(1) by reducing the temperature of water(1) sufficiently and waiting.

At some point, water(1) is metaphorically extended into water(2) to include ice, liquid water, and water vapor. Tautologically, ice(1) is water(2). It is not strange for someone to say "The tank contains water, and some of it is frozen." However, the water(1) concept is still s

...
AI Safety Debate and Its Applications
by VojtaKovarik — 2019-07-23T22:31:58.318Z

All of the experimental work and some of the theoretical work has been done jointly with Anna Gajdova, David Lindner, Lukas Finnveden, and Rajashree Agrawal as part of the third AI Safety Camp. We are grateful to Ryan Carey and Geoffrey Irving for the advice regarding this project. The remainder of the theoretical part relates to my stay at FHI, and I would like to thank the above people, Owain Evans, Michael Dennis, Ethan Perez, Stuart Armstrong, and Max Daniel for comments/discussions.

Debate is a recent proposal for AI alignment, which naturally incorporates elicitation of human preferences and has the potential to offload the costly search for flaws in an AI’s suggestions onto the AI.
After briefly recalling the intuition behind debate, we list the main open problems surrounding it and summarize how the existing work on debate addresses them. Afterward, we describe, and distinguish between, Debate games and their different applications in more detail. We also formalize what it means for a debate to be truth-promoting. Finally, we present results of our experiments on Debate games and Training via Debate on MNIST and fashion MNIST.

Debate games and why they are useful

Consider an answer A to some question Q --- for example, "Where should I go for a vacation?" and "Alaska". Rather than directly verifying whether A is an accurate answer to Q, it might be easier to first decompose A into lower-level components (How far/expensive is it? Do they have nice beaches? What is the average temperature? What language do they speak?). Moreover, it isn't completely clear what to do even if we know the relevant facts --- indeed, how does Alaska's cold weather translate to a preference for Alaska from 0 to 10? And how does this preference compare to English being spoken in Alaska?
As an alternative, we can hold a debate between two competing answers A and A′="Bali" to Q. This allows strategic debaters to focus on the most relevant features of the problem (perhaps the language isn't important), and replace the possibly-difficult direct evaluation by often-simpler comparison of two options (whatever their position on a 0-10 scale, Bali's beaches are clearly better than Alaska's).

Importantly for AI-alignment, this approach also naturally incorporates elicitation of human feedback.
Recently, the paper AI Safety via Debate [1] proposed training agents via self-play on a zero...
How to take smart notes (Ahrens, 2017)
by Yuxi_Liu — 2019-07-23T15:34:59.349Z

This is my rephrasing of (Ahrens, 2017, How to Take Smart Notes). I added some personal comments.

The amazing note-taking method of Luhmann

To be more productive, it's necessary to have a good system and workflow. The Getting Things Done system (collect everything that needs to be taken care of in one place and process it in a standardised way) doesn't work well for academic thinking and writing, because GTD requires clearly defined objectives, whereas in doing science and creative work, the objective is unclear until you've actually got there. It'd be pretty hard to "innovate on demand". Something that can be done on demand, in a predetermined schedule, must be uncreative.

Enter Niklas Luhmann. He was an insanely productive sociologist who did his work using the method of "slip-box" (in German, "Zettelkasten").

Making a slip-box is very simple, with many benefits. The slip-box will become a research partner who could "converse" with you, surprise you, lead you down surprising lines of thoughts. It would nudge you to (number in parenthesis denote the section in the book that talks about the item):

Find dissenting views (10.2, 12.3)

Really understand what you learned (10.4, 11.2, 11.3, 12.6)

Think across contexts (12.5)

Remember what you learned (11.3, 12.4)

Be creative (12.5, 12.6, 12.7, 13.2)

Get the gist, not stuck on details (12.6)

Be motivated (13.3)

Implement short feedback loops, which allows rapid improvements (12.6, 13.5)

Four kinds of notes

Fleeting notes

These are purely for remembering your thoughts. They can be: fleeting ideas, notes you would have written in the margin of a book, quotes you would have underlined in a book.

They have no value except as stepping stones towards making literature and permanent notes. They should be thrown away as soon as their contents have been transferred to literature/permanent notes (if worthy) or not (if unworthy).

Examples:

Jellyfish might be ethically vegan, since they have such a simple neural system, they probably can't feel pain.

Ch. 9
How to attention attend:

One thing at a time. No multitasking

When writing, attend to idea flow. Meaning, not wording.
...

Literature notes

These summarize the content of some text, and give the citation.

Example:

(Kahneman & Tversky, 1973) shows that people often do not take into account the prior when doing a Bayesian probability problem. In particular, when no evi

...
Prediction as coordination
by Bird Concept — 2019-07-23T06:19:40.038Z

I want to introduce a model of why forecasting might be useful which I think is underappreciated: it might help us solve coordination problems.

This is currently only a rough idea, and I will proceed by examples, pushing this post out early rather than not at all.

The standard model of forecasting

This looks something like:

We have our big, confusing, philosophical, long-term uncertainties. We then need to 1) find the right short-term questions which capture these uncertainties, which are 2) understandable to traditional Superforecasters without very deep inside knowledge, and whose expertise has only been demonstrated on short-term questions in more well-understood domains, who then 3) use tools like outside views and guesstimates to answer them.

When I hear people say they're not excited about forecasting, it's almost always because they think this standard model won't work for AI safety. I'm very sympathetic to that view.

Example 1: coordination in mathematics via formalism

When quantifying our beliefs, we lose a large amount of nuance and interpretability. This is similar to how, when formalising things mathematically, we sacrifice the majority of human understanding.

What we gain, instead, is the ability to express and communicate thoughts...

much more succinctly
using a precise, interpersonally standardised format of interpretation
in a way which clarifies certain logical and conceptual relations

This is a trade-off that allows a community of mathematicians to make intellectual progress together, and to effectively make results common knowledge in a way which allows them to coordinate on what to solve next.

Example 2: futures markets as using predictions for coordination

Getting enough food for everyone is a big coordination problem. We want some people to stockpile things like rice and wheat so that we're prepared for a drought, but we also don't want people to waste opportunities on storing stuff which has to be thrown away in case the next harvest goes well. These kinds of problems are solved by futures markets, which effectively predict the future price of rice, and thereby provide an incentive arbitrage away any abrupt price fluctuations (i.e. to strategically stockpile/sell out rice so as to match future supply and demand). Robert Shiller has suggested these as a candidate for the greatest financial innovation.

Example 3: pre...
The Real Rules Have No Exceptions
by Said Achmiz — 2019-07-23T03:38:45.992Z

(This is a comment that has been turned into a post.)

From Chris_Leong’s post, “Making Exceptions to General Rules”:

Suppose you make a general rule, ie. “I won’t eat any cookies”. Then you encounter a situation that legitimately feels exceptional , “These are generally considered the best cookies in the entire state”. This tends to make people torn between two threads of reasoning:

Clearly the optimal strategy is to make an exception this one time and then follow the rule the rest of the time.

If you break the rule this one time, then you risk dismantling the rule and ending up not following it at all.

How can we resolve this? …

This is my answer:

Consider even a single exception to totally undermine any rule. Consequently, only follow rules with no exceptions.[1]. When you do encounter a legitimate exception to a heretofore-exceptionless rule, immediately discard the rule and replace it with a new rule—one which accounts for situations like this one, which, to the old rule, had to be exceptions.

This, of course, requires a meta-rule (or, if you like, a meta-habit):

Prefer simplicity in your rules. Be vigilant that your rules do not grow too complex; make sure you are not relaxing the legitimacy criteria of your exceptions. Periodically audit your rules, inspecting them for complexity; try to formulate simpler versions of complex rules.

So, when you encounter an exception, you neither break the rule once but keep following it thereafter, nor break it once and risk breaking it again. If this is really an exception, then that rule is immediately and automatically nullified, because good rules ought not have exceptions. Time for a new rule.

And if you’re not prepared to discard the rule and formulate a new one, well, then the exception must not be all that compelling; in which case, of course, keep following the existing rule, now and henceforth.

But why do I say that good rules ought not have exceptions? Because rules already don’t have exceptions.

Exceptions are a fiction. They’re a way for us to avoid admitting (sometimes to ourselves, sometimes to others) that the rule as stated, together with the criteria for deciding whether something is a “legitimate” exception, is the actual rule.

The approach I describe above merely consists of making this fact explicit.

By which I mean “only follow rules to which no legitimate exception will ever be encountered”, n

...
"Shortform" vs "Scratchpad" or other names
by Raemon — 2019-07-23T01:21:48.979Z

Within the next day or two, we'll be launching some features that give the "shortform" concept real infrastructural support. Shortform comments will appear on the All Posts page, on the new /shortform page, and it'll be easier to automatically generate a post to write shortform content.

The only potential issue is that, as "shortform" comments have gained traction... they... often aren't especially short.

I'd kinda like to finalize the name before it goes live.

Currently the team is looking at "Scratchpad" as a name, intending to convey "this is a place you write early stage, off the cuff ideas without stressing too much about whether it should be a post." Sometimes this turns out to actually be short, sometimes not. The main counterpoint right now is that scratchpad kinda implies a level of "ephemerality" or something that isn't necessarily true either.

I'm particularly interested in feedback from people who've either already made a shortform feed (i.e. mr-hire, hazard, DanielFilan and others) if they have any opinions about this particular bikeshed.
Where is the Meaning?
by Hazard — 2019-07-22T20:18:24.964Z

"You're being mean."
"No, I'm not."

"This album is amazing!"
"What are you talking about? It's clearly trash."

"Tomatoes are my favorite vegetable!"
"Dude, they're fruits."

People often argue about what things are, and they often do it with words. Sometimes this can create another argument about what a certain word means. This leads me to semi-rhetorically ask you the question, where is the meaning located? In the words? The people? The air? If this seems like philosophical thumb-twiddling, note that where you think meaning is determines where you go looking whenever a question of meaning pops up. If I think the dictionary is the end all be all of meaning, it's the first place I'll go when pondering, "What really is happiness?"

If you've ever been quoted out of context in a way that made you look bad, you might have grumbled, "That's not what I meant!" On some level, I think most people understand that words themselves do not have inherent meaning. There are thoughts in your head, and words are the things that come out of your mouth, which hopefully let others guess what you are thinking.

And yet... damn sometimes it just really feels like meaning is baked into a message. Like it's just there, waiting to be discovered. We're going to explore this feeling, what I think it means, and how to not let it trip you up.

Context and Predictability

Context matters:

"My sister just got a new dog."
vs
"You snuck in? You sly dog!"

Context still matters:

Mobster: "Watch do to that guy who owed you money?"
Mob Boss: "I put him to sleep."
vs
Spouse 1: "Where's our Henry? I haven't seen him in a bit"
Spouse 2: "I put him to sleep."

Neither of those are meant to be shocking, just reminders that you already know that the same word, even the same sentence, can mean completely different things based on the context.

Everyone has the shared context of "having a human brain" and "being subject to the same laws of physics". This lets me communicate things like "Get away! You're not welcome" merely by throwing rocks. Language is a wild invention that was bootstrapped from an incredibly general context, and now it lets us create even more shared context which allows for ev...
The Self-Unaware AI Oracle
by Steven Byrnes — 2019-07-22T19:04:21.188Z

(See also my later post, Self-Supervised Learning and AGI Safety)

Abstract A self-unaware AI oracle is a system that builds a world-model inside which there is no concept of "me". I propose that such systems are safe to operate, that they can be plausibly built by continuing known AI development paths, and that they would be competitive with any other type of AI—capable of things like inventing new technology and safe recursive self-improvement.

Epistemic status: Treat as preliminary brainstorming; please let me know any ideas, problems, and relevant prior literature.

Overview

My proposal is to build a system that takes in data about the world (say, all the data on the internet) and builds a probabilistic generative model of that data, in the sense of taking an arbitrary number of bits and then predicting one or more masked bits. If you want to imagine something more specific, here are three examples: (1) A hypercomputer using Solomonoff Induction; (2) Some future descendant of GPT-2, using a massive Transformer (or some better future architecture); (3) A neocortex-like algorithm that builds low-level predictive models of small segments of data, then a hierarchy of higher-level models that predict which of the lower-level models will occur in which order in different contexts.[1]

Further, we design the system to be "self-unaware"—we want it to construct a generative world-model in which its own physical instantiation does not have any special status. The above (1-2) are examples, as is (3) if you don't take "neocortex-like" too literally (the actual neocortex can learn about itself by, say, controlling muscles and observing the consequences).

Finally, we query the system in a way that is compatible with its self-unawareness. For example, if we want to cure cancer, one nice approach would be to program it to search through its generative model and output the least improbable scenario wherein a cure for cancer is discovered somewhere in the world in the next 10 years. Maybe it would output: "A scientist at a university will be testing immune therapy X, and they will combine it with blood therapy Y, and they'll find that the two together cure all cancers". Then, we go combine therapies X and Y ourselves.

What is self-unawareness?

In the context of predictive world-modeling systems, a self-unaware system (a term I just made up—is there an existing term?) is one tha

...