trentmkelly/LessWrong-43k huggingface dataset:
Too Smart for My Own Good
by isovector — 2019-01-22T17:51:10.960Z

Originally posted at sandymaguire.me

I want to share a piece of ridiculously obvious advice today.

I've got a bad habit, which is being too smart for my own good. Which is to say,
when I want to learn something new, too often I spend my time making tools to
help me learn, rather than just learning the thing.

Take, for example, the first time I tried to learn how to play jazz music.

There's only one thing that I'm really good at, which is programming. The
central tenet in programming is that "laziness is good," and if you're faced
with doing something boring and repetitive, you should instead automate that
thing away.

When all you have is a hammer...

According to The Book, the first thing to do to learn jazz is to
learn your scales---in every mode for every key for several varieties of
harmony. There are 12 notes, and seven modes, and at least four harmonies.
That's what, like 336 different scales to learn?

"WHO HAS TIME FOR ALL THAT CRAP," I thought. "I'LL JUST WRITE A COMPUTER
PROGRAM TO GENERATE THE SCALES FOR ME, AND THEN PLAY THOSE."

In retrospect, this was a terrible plan. Not only did it not get me closer to
my goal of knowing how to play jazz music, I also didn't know enough about the
domain to successfully model it. It's funny to read back through that blog post
with the benefit of hindsight, but at the time I really thought I was onto
something!

That's not to say it was wasted effort nor that it was useless, merely that
it wasn't actually moving me closer to my stated goal of being able to play jazz
music. It was scratching my itch for mental masturbation, and was a good
exercise in attempting to model things I don't understand very well, but
crucially, it wasn't helping.

Or take another example, a more recent foray into music for me---only a few
weeks ago. This time I had more of a plan; I was taking piano lessons and
getting advice on how to practice from my teacher. One of the things he
suggested I do was to solo around in the minor pentatonic scale. And so I did,
starting in C, and (tentatively) moving to G.

But doing it in Bb was hard! Rather than spend the two minutes that would be
required to work out what notes I should play in the Bb minor pentatonic, I
decided it would be better to write a computer program! This time it
would connect to my keyboard and "listen" to the notes I played, and flash red
whenever I played a note that wasn't in the Bb minor

...
Stale air / high CO2 may decrease your cognitive function
by Yoav Ravid — 2019-01-22T12:52:06.070Z

If you're in a closed space, you may want to open a window.

before the industrial revolution, the atmosphere had 300 parts-per-million (PPM) of CO2. today, this number is already above 400 on average, and 500 in urban areas.

but CO2 doesn't just effect the environment, high enough levels of it also effect our bodies, and our minds.

so let's leave the atmosphere for a bit, and go inside. one study checked office employee's decision making skills at various CO2 levels, here some of the results:

1000ppm

this level is common at poorly ventilated spaces like a workrooms/offices. and one study on schools in several US districts found 50% of classrooms to have this level.

at this CO2 level the cognitive function in the office experiment decreased by 15%.

1400ppm

this level can also be reached at the places described above.

here cognitive function decreased by 50%!

2000ppm

from this level onward some people described other side effects such as: slight nausea, loss of attention and poor concentration, sleepiness, headaches, and increased hearth rates.

and still, these levels aren't uncommon -

4000ppm

this is common in cars and bedrooms (closed spaces which are either small, you spend a long time in, or both. and the side effects increase.

8000-10,000ppm

motorcycle helmets can reach these levels. Being in such an environment for long times can harm your long-term health.

So what can you do?

1. simply open a window! (at least in this part of the century)

2. you can get some plants for your room or office -

The NASA Clean Air Study looked at about two and half dozen plants, and recommends these tree plants: Areca Palm, Snake Plant, and Money Plant.

This lung institute guide seems to be based on this study, so i suggest reading it.

3. buy a CO2 monitor if you want to always know in what environment you're in. though, these seem cost quite a bit (for a reason unclear to me). so i don't know if it will really benefit you. i know i won't bother.

The IPPC reported that CO2 levels will be, by the end of the century, between 541 and 970ppm. if we extrapolate from the previous study, this may mean a 10-15% decrease in the cognitive function of humanity as a species (and even more than the previous results in closed spaces).

some studies found evidence that air pollution can harm the brain itself.

Should this change our attitude towards climate change as a cat...
Why don't people use formal methods?
by John_Maxwell — 2019-01-22T09:39:46.721Z
Link: https://www.hillelwayne.com/post/why-dont-people-use-formal-methods/

Saw this on Hacker News; there's Hacker News discussion here.

Formal methods seem very relevant to AI safety, and I haven't seen much discussion of them on Less Wrong.
Should questions be called "questions" or "confusions"  (or "other")?
by Raemon — 2019-01-22T02:45:01.211Z

Or, what is the best way to think about "what is a question?" on LW?

The LW Team just had a retreat where we thought through a lot of high level strategy. We have a lot of ideas building off of the "questions" feature.

One thing that struck us is that a lot of early stage research has less to do with formalizable questions, and more to do with noticing anomalies in your current model/paradigm. Something feels off that you can't explain, or there's a concept you don't even understand well enough to ask a coherent question about.

The "question" feature was meant, in part, to reduce the cost of exploring early stage curiosity, but we wondered if it might even be a slightly-too-formalized.

Just like, technically there was nothing stopping you from asking a question as a post (but adding the feature caused a proliferation of questions) there is nothing stopping you from asking an ill-formed question. But, maybe changing the language slightly would better encourage early-stage curiosity.

So, curious:

How would feel if we changed "Ask a question" to "Pose a confusion" or something like that? (The main issue so far is that "pose confusion" is, well, way more confusing since it's a non-standard phrase. Other options include literally saying "Ask question/Pose Confusion" [i.e. both at once, so you get the benefit of the clear-cut "ask question"], or some word other than "pose.")

(Somewhat but not-entirely-jokingly, we also noticed people are hesitant to post "answers" since they sound like you're trying to claim you know what you're talking about. We jokingly considered "Post a deconfusion", or "post a partial answer" as options)
Alignment Newsletter #42
by Rohin Shah — 2019-01-22T02:00:02.082Z

Cooperative IRL as a definition of human-AI group rationality, and an empirical evaluation of theory of mind vs. model learning in HRI

Find all Alignment Newsletter resources here. In particular, you can sign up, or look through this spreadsheet of all summaries that have ever been in the newsletter.

Highlights

AI Alignment Podcast: Cooperative Inverse Reinforcement Learning (Lucas Perry and Dylan Hadfield-Menell) (summarized by Richard): Dylan puts forward his conception of Cooperative Inverse Reinforcement Learning as a definition of what it means for a human-AI system to be rational, given the information bottleneck between a human's preferences and an AI's observations. He notes that there are some clear mismatches between this problem and reality, such as the CIRL assumption that humans have static preferences, and how fuzzy the abstraction of "rational agents with utility functions" becomes in the context of agents with bounded rationality. Nevertheless, he claims that this is a useful unifying framework for thinking about AI safety.

Dylan argues that the process by which a robot learns to accomplish tasks is best described not just as maximising an objective function but instead in a way which includes the system designer who selects and modifies the optimisation algorithms, hyperparameters, etc. In fact, he claims, it doesn't make sense to talk about how well a system is doing without talking about the way in which it was instructed and the type of information it got. In CIRL, this is modeled via the combination of a "teaching strategy" and a "learning strategy". The former can take many forms: providing rankings of options, or demonstrations, or binary comparisons, etc. Dylan also mentions an extension of this in which the teacher needs to learn their own values over time. This is useful for us because we don't yet understand the normative processes by which human societies come to moral judgements, or how to integrate machines into that process.

On the Utility of Model Learning in HRI (Rohan Choudhury, Gokul Swamy et al): In human-robot interaction (HRI), we often require a model of the human that we can plan against. Should we use a specific model of the human (a so-called "theory of mind", where the human is approximately optimizing some unknown reward), or should we simply learn a model of t...
Game Analysis Index
by Zvi — 2019-01-21T15:30:00.371Z

This post links to this blog’s posts discussing game design, balance, economics and related topics, as well as any strategy posts. It does not contain new content.

Much of the blog is relevant to gaming, but these are the explicitly on-topic posts.

Eternal Sequence

Eternal, and Hearthstone Economy versus Magic Economy

The Eternal Grind

Eternal: The Exit Interview

Artifact / Card Rebalancing Sequence

Review: Artifact

Artifact Embraces Card Balance Changes

Card Collection and Ownership

Card Balance and Artifact

Card Rebalancing, Card Oversupply and Economic Considerations in Digital Card Games

Advantages of Card Rebalancing

Disadvantages of Card Rebalancing

Game Reviews (Including Those Listed Above)

All games reviewed are recommended, we don’t generally waste time on unworthy games.

Persona 5: Spoiler-Free Review

Review: Artifact

Review: Slay the Spire

Octopath Traveler: Spoiler-Free Review

Magic Strategy

Deck Guide: Burning Drakes
Disentangling arguments for the importance of AI safety
by Richard_Ngo — 2019-01-21T12:41:43.615Z

Note: my views have shifted significantly since writing this post. I now consider items 1, 2, 3, and 6.2 to be different facets of one core argument, which I call the "second species" argument, and which I explore in depth in this report. And I don't really think of 4 as an AI safety problem any more.

I recently attended the 2019 Beneficial AGI conference organised by the Future of Life Institute. I’ll publish a more complete write-up later, but I was particularly struck by how varied attendees' reasons for considering AI safety important were. Before this, I’d observed a few different lines of thought, but interpreted them as different facets of the same idea. Now, though, I’ve identified at least 6 distinct serious arguments for why AI safety is a priority. By distinct I mean that you can believe any one of them without believing any of the others - although of course the particular categorisation I use is rather subjective, and there’s a significant amount of overlap. In this post I give a brief overview of my own interpretation of each argument (note that I don’t necessarily endorse them myself). They are listed roughly from most specific and actionable to most general. I finish with some thoughts on what to make of this unexpected proliferation of arguments. Primarily, I think it increases the importance of clarifying and debating the core ideas in AI safety.

Maximisers are dangerous. Superintelligent AGI will behave as if it’s maximising the expectation of some utility function, since doing otherwise can be

shown to be irrational.

Yet we can’t write down a utility function which precisely describes human values, and optimising very hard for any other function will lead to that AI rapidly seizing control (as a

convergent instrumental subgoal)

and building a future which contains very little of what we value (because of

Goodhart’s law

and

the complexity and fragility of values)

. We won’t have a chance to notice and correct misalignment because an AI which has exceeded human level will increase its intelligence very quickly (either by recursive self-improvement or by scaling up its hardware), and then prevent us from modifying it or shutting it down.

This was the main thesis advanced by Yudkowsky and Bostrom when founding the field of AI safety. Here I’ve tried to conve

...
Do the best ideas float to the top? 
by Quinn — 2019-01-21T05:22:51.182Z

It may depend on what we mean by “best”.

Epistemic status: I understand very little of anything.

Speculation about potential applications: regulating a logical prediction market, e.g. logical induction; constructing judges or competitors in e.g. alignment by debate; designing communication technology, e.g. to mitigate harms and risks of information warfare.

The slogan “the best ideas float to the top” is often used in social contexts. The saying goes, “in a free market of ideas, the best ideas float to the top”. Of course, it is not intended as a facts statement, as in “we have observed that this is the case”; it is instead a values statement, as in “we would prefer for this to be the case.”.

In this essay, however, we will force an empirical interpretation, just to see what happens. I will provide three ways to consider the density of an idea, or the number assigned to how float-to-the-top an idea is.

In brief, an idea is a sentence, and you can vary the amount of it’s antecedent graph (like in bayesian nets, NARS-like architectures) or function out of which it is printed (like in compression) that you want to consider at a given moment, up to resource allocation. This isn’t an entirely mathematical paper, so don’t worry about WFFs, parsers, etc., which is why i’ll stick with “ideas” instead of “sentences”. I will also be handwaving between "description of some world states" and "belief about how world states relate to eachother".

Intuition

Suppose you observe wearers of teal hats advocate for policy A, but you don’t know what A is. You’re minding your business in an applebees parking lot when a wearer of magenta hats gets your attention to tell you “A is harmful”. There are two cases:

Suppose A is “kicking puppies”, (and I don’t mean the wearer of magenta hats is misleadingly compressing A to you, I mean the policy is literally kicking puppies). The inferential gap between you and the magentas can be closed very cheaply, so you’re quickly convinced that A is harmful (unless you believe that kicking puppies is good).

Suppose A is “fleegan at a rate of flargen”, where fleeganomics is a niche technical subject which nevertheless can be learned by anyone of median education in N units[^1] or less. Suppose also that you know the value of N, but you’re not inclined to invest that much compute in a dumb election, so you either a. take them at their word that A is ha

...
Following human norms
by Rohin Shah — 2019-01-20T23:59:16.742Z

So far we have been talking about how to learn “values” or “instrumental goals”. This would be necessary if we want to figure out how to build an AI system that does exactly what we want it to do. However, we’re probably fine if we can keep learning and building better AI systems. This suggests that it’s sufficient to build AI systems that don’t screw up so badly that it ends this process. If we accomplish that, then steady progress in AI will eventually get us to AI systems that do what we want.

So, it might be helpful to break down the problem of learning values into the subproblems of learning what to do, and learning what not to do. Standard AI research will continue to make progress on learning what to do; catastrophe happens when our AI system doesn’t know what not to do. This is the part that we need to make progress on.

This is a problem that humans have to solve as well. Children learn basic norms such as not to litter, not to take other people’s things, what not to say in public, etc. As argued in Incomplete Contracting and AI alignment, any contract between humans is never explicitly spelled out, but instead relies on an external unwritten normative structure under which a contract is interpreted. (Even if we don’t explicitly ask our cleaner not to break any vases, we still expect them not to intentionally do so.) We might hope to build AI systems that infer and follow these norms, and thereby avoid catastrophe.

It’s worth noting that this will probably not be an instance of narrow value learning, since there are several differences:

Narrow value learning requires that you learn what to do, unlike norm inference.
Norm following requires learning from a complex domain (human society), whereas narrow value learning can be applied in simpler domains as well.
Norms are a property of groups of agents, whereas narrow value learning can be applied in settings with a single agent.

Despite this, I have included it in this sequence because it is plausible to me that value learning techniques will be relevant to norm inference.

Paradise prospects

With a norm-following AI system, the success story is primarily around accelerating our rate of progress. Humans remain in charge of the overall trajectory of the future, and we use AI systems as tools that enable us to make better decisions and create better technologies, which looks like “superhuman intelligence” from our ...
Announcement: AI alignment prize round 4 winners
by cousin_it — 2019-01-20T14:46:47.912Z

We (Zvi Mowshowitz and Vladimir Slepnev) are happy to announce the results of the fourth round of the AI Alignment Prize, funded by Paul Christiano. From July 15 to December 31, 2018 we received 10 entries, and are awarding four prizes for a total of $20,000.

The winners

We are awarding two first prizes of $7,500 each. One of them goes to Alexander Turner for Penalizing Impact via Attainable Utility Preservation; the other goes to Abram Demski and Scott Garrabrant for the Embedded Agency sequence.

We are also awarding two second prizes of $2,500 each: to Ryan Carey for Addressing three problems with counterfactual corrigibility, and to Wei Dai for Three AI Safety Related Ideas and Two Neglected Problems in Human-AI Safety.

We will contact each winner by email to arrange transfer of money. Many thanks to everyone else who participated!

Moving on

This concludes the AI Alignment Prize for now. It has stimulated a lot of good work during its year-long run, but participation has been slowing down from round to round, and we don't think it's worth continuing in its current form.

Once again, we'd like to thank everyone who sent us articles! And special thanks to Ben and Oliver from the LW2.0 team for their enthusiasm and help.
Capability amplification
by paulfchristiano — 2019-01-20T07:03:27.879Z

(Note: In the past I have referred to this process as ‘bootstrapping’ or ‘policy amplification,’ but those terms are too broad — there are other dimensions along which policies can be amplified, and ‘bootstrapping’ is used all over the place.)

Defining the “intended behavior” of a powerful AI system is a challenge.

We don’t want such systems to simply imitate human behavior — we want them to improve upon human abilities. And we don’t want them to only take actions that look good to humans — we want them to improve upon human judgment.

We also don’t want them to pursue simple goals like “minimize the probability that the bridge falls down” or “pick the winning move.” A precise statement of our real goals would be incredibly complicated, and articulating them precisely is itself a massive project. Moreover, we often care about consequences over years or decades. Such long-term consequences would have little use as a practical problem definition in machine learning, even if they could serve as a philosophical problem definition.

So: what else can we do?

Instead of defining what it means for a policy to be “good,” we could define a transformation which turns one policy into a “better” policy.

I call such a transformation capability amplification — it “amplifies” a weak policy into a strong policy, typically by using more computational resources and applying the weak policy many times.

Motivation

I am interested in capability amplification because I think it is the most plausible route to defining the goals of powerful AI systems, which I see as a key bottleneck for building aligned AI. The most plausible alternative approach is probably inverse RL, but I think that there are still hard philosophical problems to solve, and that in practice IRL would probably need to be combined with something like capability amplification.

More directly, I think that capability amplification might be a workable approach to training powerful RL systems when combined with semi-supervised RL, adversarial training, and informed oversight (or another approach to reward engineering).

Example of capability amplification: answering questions

Suppose that we would like like to amplify one question-answering system Ainto a “better” question-answering system A⁺.

We will be given a question Q and an implementation of A; we can use A, or any other tools at our disposal, to try to answer the question Q. W...
Less Competition, More Meritocracy?
by Zvi — 2019-01-20T02:00:00.974Z

Analysis of the paper: Less Competition, More Meritocracy (hat tip: Marginal Revolution: Can Less Competition Mean More Meritocracy?)

Epistemic Status: Consider the horse as if it was not a three meter sphere

Economic papers that use math to prove things can point to interesting potential results and reasons to question one’s intuitions. What is frustrating is the failure to think outside of those models and proofs, analyzing the practical implications.

In this particular paper, the central idea is that when risk is unlimited and free, ratcheting up competition dramatically increases risk taken. This introduces sufficient noise that adding more competitors can make the average winner less skilled. At the margin, adding additional similar competitors to a very large pool has zero impact. Adding competitors with less expected promise makes things worse.

This can apply in the real world. The paper provides a good example of a very good insight that is then proven ‘too much,’ and which does not then question or vary its assumptions in the ways I would find most interesting.

I. The Basic Model and its Central Point

Presume some number of job openings. There are weak candidates and strong candidates. Each candidate knows if they are strong or weak, but not how many other candidates are strong, nor do those running the contest know how many are strong.

The goal of the competition is to select as many strong candidates as possible. Or formally, to maximize [number of strong selected – number of weak selected], which is the same thing if the number of candidates is fixed, but is importantly different later when the number of selected candidates can vary. Each candidate performs and is given a score, and for an N-slot competition, the highest N scores are picked.

By default, strong candidates score X and weak candidates score Y, X>Y, but each candidate can also take on as much risk as they wish, with any desired distribution of scores, so long as their score never goes below zero.

The paper then does assumes reflexive equilibrium, does math and proves a bunch of things that happen next. The math checks out; I duplicated the results intuitively.

There are two types of equilibrium.

In the first type, concession equilibria, strong candidates take no risk and are almost always chosen. Weak candidates take risk to try and beat other weak candidates, but attempting to beat stron...