trentmkelly/LessWrong-43k huggingface dataset:
The Uncertainty of Death and Taxes
by Zvi — 2020-10-21T18:10:01.467Z

Follow-up of sorts to: Easy Mode and Hard Mode, also Spoiler-Free Review: Death and Taxes

Epistemic Status: Early exploration of concepts that deserve a better, tighter treatment at a future time. This has been in my drafts for a while, figure I’ll push it out there and then redo the important stuff again later.

Spoiler Alert: This post is about and contains full spoilers for the game Death and Taxes, attempting to use it to make broader points. Death and Taxes is a game that benefits greatly from being played blind, which is kind of the point of this post. If you’re debating whether to play the game, see my short spoiler-free review. This post is designed not to rely at all on the reader having played, but it will definitely ruin what the game is going for.

Death and Taxes illustrates the richness and complexity of life and its decisions under uncertainty. When you don’t know what matters, everything matters. Then it illustrates the importance of retaining that richness and complexity, and thus on some level that uncertainty, by taking most of that uncertainty away. When that happens, the fun and meaning go out of the process. It loses its soul.

Death and Taxes is a game about making discrete choices.

You play as a grim reaper. Each day, you wake up. You go to your desk and are given instructions and several profiles. You have to decide who lives, and who dies.

Once you have done that, you go up to have a conversation with your boss. Your boss will look to see whether you killed the right number of people, and whether you fulfilled any other requirements, and pay you for the day accordingly.

You can then go to Mortimer’s to spend that cash on various clothes and trinkets. You can also go to the mirror, where you can change clothes and listen to a mysterious voice.

Over time, you learn that your decisions matter, and determine the fate of the world. Depending on your choices, it can be a world of war or peace, prosperity or poverty, health or sickness, paradise or hellscape.

You also determine your own fate. You can be fired, get promoted, become your boss’s pet or usurp your boss’s position. That boss, of course, is also called fate.

Uncertainty

What makes the game interesting is not knowing what is going on, or what actions lead to what results.

When looking at a profile and deciding what to do, there are lots of things to consider. Should you kil...
[AN #122]: Arguing for AGI-driven existential risk from first principles
by Rohin Shah — 2020-10-21T17:10:03.703Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Note: I (Rohin) have just started as a Research Scientist at DeepMind! I expect to publish the newsletter as normal, but if previously you’ve been taking my CHAI summaries with a grain of salt due to the conflict of interest, it’s now time to apply that to DeepMind summaries. Of course, I try not to be biased towards my employer, but who knows how well I succeed at that.

HIGHLIGHTS

AGI safety from first principles (Richard Ngo) (summarized by Rohin): This sequence presents the author’s personal view on the current best arguments for AI risk, explained from first principles (that is, without taking any previous claims for granted). The argument is a specific instantiation of the second species argument that sufficiently intelligent AI systems could become the most intelligent species, in which case humans could lose the ability to create a valuable and worthwhile future.

We should clarify what we mean by superintelligence, and how it might arise. The author considers intelligence as quantifying simply whether a system “could” perform a wide range of tasks, separately from whether it is motivated to actually perform those tasks. In this case, we could imagine two rough types of intelligence. The first type, epitomized by most current AI systems, trains an AI system to perform many different tasks, so that it is then able to perform all of those tasks; however, it cannot perform tasks it has not been trained on. The second type, epitomized by human intelligence and GPT-3 (AN #102), trains AI systems in a task-agnostic way, such that they develop general cognitive skills that allow them to solve new tasks quickly, perhaps with a small amount of training data. This second type seems particularly necessary for tasks where data is scarce, such as the task of being a CEO of a company. Note that these two types should be thought of as defining a spectrum, not a binary distinction, since the type of a particular system depends on how you define your space of “tasks”.

How might we get AI systems that are mor...
Reward large contributions?
by SarahNibs — 2020-10-21T16:26:31.213Z

Sometimes I want people to get $$$ for their contributions.

When someone posts a long, exceptional sequence? I don't get that feeling.

When folks working on the site itself roll out a sweet new feature? I don't get that feeling.

But when https://www.lesswrong.com/users/lsusr creates a fun and relevant competition requiring them to code up quite a bit of framework/bots and coordinate people? (https://www.lesswrong.com/posts/AHTRyQJtiRin22kth/the-darwin-game-1) I wonder if there's a pool to reward over-and-beyond contributors, and if there is, how much I'd be willing to contribute to it.

Does something analogous exist? "Should" it? If so, what should be "eligible"?
Stupid Questions October 2020
by ChristianKl — 2020-10-21T11:41:38.474Z

The stupid questions thread was one of the regular threads on LessWrong 1.0. It's a place where no question is to stupid to be asked and anybody who answers is encouraged to be kind.

This thread is for asking any questions that might seem obvious, tangential, silly or what-have-you. Don't be shy, everyone has holes in their knowledge, though the fewer and the smaller we can make them, the better.

Please be respectful of other people's admitting ignorance and don't mock them for it, as they're doing a noble thing.
Sunday October 25, 12:00PM (PT) — Scott Garrabrant on "Cartesian Frames"
by Ben Pace — 2020-10-21T03:27:12.739Z

Added: The event video is online here!

This Sunday at 12 noon (PT), we're running another LW weekly online meetup.

MIRI's Scott Garrabrant will be giving a talk introducing Cartesian Frames, a new approach to modeling agency that he's been thinking about and working on over the last year, and that will be the focus of a sequence he's releasing. (The first post of this sequence will go up this Thursday is up now: Introduction to Cartesian Frames.)

Scott's talk will be on Zoom, at this link: https://us02web.zoom.us/j/81328320546

Q&A and Discussion

After Scott's talk, at 13:00 (PT), there will be a Q&A and open discussion on the LessWrong Walled Garden.

The Walled Garden is a 2D virtual hangout space, where Scott has a digital office. To attend, follow these steps:

Go to https://gather.town/app/aPVfK3G76UukgiHx/lesswrong-campus
Head north for a while and move your avatar onto the TARDIS / phone booth.
Then move your avatar onto the words "Scott G's Office" and you'll be teleported to his office.

The Walled Garden is open to the public during the hours of 12:00-16:00 (PT) on Sundays. Anyone is welcome to join the Q&A / discussion, even if you just read Scott's upcoming blog post and skip the talk.

Office Hours

Scott will be hosting regular office hours while he posts his Cartesian Frames sequence. You're welcome to come and ask any questions you have about the sequence, start discussions, etc.

After this Sunday's talk, Scott's office hours will be in his Walled Garden office on the following three Sundays (Nov. 1, 8, and 15), from 14:00 to 16:00 (PT).

Scott will also be giving another talk on Cartesian Frames (with more advanced material, building off of the first talk / the first blog posts) in the LessWrong Walled Garden as part of his Nov. 1 office hours, at 14:00 PT.

Speaker

Scott Garrabrant has 9 curated posts:

Goodhart Taxonomy
Sources of intuition and data on AGI
A Proper Scoring Rule for Confidence Intervals
Robustness to Scale
Prisoners' Dilemma with Costs to Modeling
Embedded Agents
Embedded Agency (full-text version)
Thoughts on Human Models
Yes Requires the Possibility of No

Details

When? Sunday October 25th, 12:00PM – 14:00PM (PT)

Where? Zoom: https://us02web.zoom.us/j/81328320546

RSVP? Not required, but it would help us if you RSVP'd using this Google Form and/or this Facebook event. This helps us know how many people to plan for.

Advice for video calls? To pa...
Has anyone researched specification gaming with biological animals?
by David Scott Krueger (formerly: capybaralet) — 2020-10-21T00:20:01.610Z

People usually train animals (well, at least pets) "manually" (i.e. using a human-in-the-loop to generate something like a reward signal).
People usually train AI systems "automatically" (i.e. using a programmatic reward signal).
What if we tried to train animals "automatically"? I imagine this would be a great line of research for someone just starting their academic career. I would expect:
1) It would provide nice, convincing examples of specification gaming.
2) It would outperform manual training in some cases. This could help show that animals are smarter than they've been shown to be, so far.
Book Review: Reinforcement Learning by Sutton and Barto
by billmei — 2020-10-20T19:40:52.736Z

Note: I originally submitted this for the Slate Star Codex (RIP) book review contest, but given the current drama it probably won't happen for a while (if at all), so I decided to share this review with the LW community in the meantime.

Cross-posted on my blog: https://billmei.net/blog/reinforcement-learning

Reinforcement Learning profoundly changed my understanding of the science of happiness, biological evolution, human intelligence, and also gave me unique tactics for rapid skill acquisition in my personal life.

You may not be expecting these conclusions, as on the surface this is a technical textbook for those wishing to learn about reinforcement learning (RL), the subfield of machine learning algorithms powering recent high-profile successes of AIs trouncing human champions like AlphaGo in Go and Watson in Jeopardy.

However, I wrote this review in layman’s terms, and it contains no math, to help disseminate these ideas to people who would be intimidated by the math and unlikely to read it otherwise, as I highlight only the most important (and underrated!) topics from this book that you won’t find elsewhere. If you instead want a comprehensive math-included summary of the book, you can read my detailed notes on each chapter.

What is reinforcement learning

Reinforcement learning is a study of what are the best actions (policy) to take in a given environment (states) to achieve the maximum long-term reward.

I’ve found RL to be the most elegant, first-principles formalization of what it means to “win”. It’s an algorithm that explicitly gives you the optimal actions to take to achieve the goal you define, with zero outside knowledge other than the input environment. Unlike other machine learning algorithms, RL does not require you to specify subgoals (capture the most chess pieces), only the ultimate goal (win at chess).

The Slate Star Codex and LessWrong communities started using Bayes’ theorem as the principal guide towards finding the truth. Bayes’ theorem is the first-principles formalism of how to evaluate evidence and update your beliefs so that your beliefs will match reality as closely as possible. But ultimately “Rationality is winning”; the purpose of “Rationality” is not just to have a good epistemology, but to successfully achieve goals in the real world.

Just as Bayes’ theorem is the mathematical foundation for finding the truth, reinforcement learning i...
Problems Involving Abstraction?
by johnswentworth — 2020-10-20T16:49:39.618Z

I'm working on a post of examples for how to formulate problems involving abstraction (using the abstraction formulation here). This isn't going to solve problems, just show how to set them up mathematically.

To that end, I'd like to hear particular problems people are interested in which intuitively seem to involve abstraction. Examples of the sort of thing I have in mind:

Humans generally seem to care about abstract objects, not individual atoms, so it seems like abstraction should be relevant to impact measures. How would we formalize that?
Humans can figure out what a new word means with ridiculously few examples, suggesting that we already have some "latent space" with a simple representation of the-class-of-things-corresponding-to-the-new-word. That sounds like it has something to do with abstraction. What's going on there?
The sort of "maps" we use in the real world (like street maps, for instance) are lossy, abstract representations of the territory (i.e. streets). How can we usefully formulate map-territory correspondence for such abstract maps? Is possible for a system to use its abstract map to recognize flaws in its own abstract-map-making process?

There is a high chance that your request (or at least something very similar to it) will be incorporated in the post. So, what examples would people like to see?
Box inversion hypothesis
by Jan Kulveit — 2020-10-20T16:20:51.282Z

This text originated from a retreat in late 2018, where researchers from FHI, MIRI and CFAR did an extended double-crux on AI safety paradigms, with Eric Drexler and Scott Garrabrant in the core. In the past two years I tried to improve it in terms of understandability multiple times, but empirically it seems quite inadequate. As it seems unlikely I will have time to invest further work into improving it, I'm publishing it as it is, with the hope that someone else will maybe understand the ideas even at this form, and describe them more clearly.

The box inversion hypothesis consists of the two following propositions

There exists something approximating a duality / an isomorphism between technical AI safety problems in the Agent Foundations agenda and some of the technical problems implied by the

Comprehensive AI Services

framing
The approximate isomorphism holds between enough properties that some solutions to the problems in one agenda translate to solutions to problems in the other agenda

I will start with an apology - I will not try to give my one paragraph version of the Comprehensive AI Services. It is an almost 200 pages long document, conveying dozens of models and intuitions. I don’t feel like being the best person to give a short introduction. So, I just assume familiarity with CAIS. I will also not try to give my short version of the various problems which broadly fit under the Agent Foundations agenda, as I assume most of the readers are already familiar with them.

0. The metaphor: Circle inversion

People who think geometrically rather than spatially may benefit from looking at a transformation of a plane called circle inversion first. A nice explanation is here - if you have never met the transformation, pages 1-3 of the linked document should be enough.

You can think about the “circle inversion” as a geometrical metaphor for the “box inversion”.

1. The map: Box inversion

The central claim is that there is a transformation between many of the technical problems in the Agent Foundations agenda and CAIS. To give you some examples

problems with daemons <-> problems with molochs
questions about ontologies <-> questions about service catalogues
manipulating the operator <-> addictive services
some “hard core” of safety (tiling, human-compatibility, some notions of corrigibility) <-> defensive stability, layer of security s

...
Launching Forecast, a community for crowdsourced predictions from Facebook
by Rebecca Kossnick — 2020-10-20T06:20:29.206Z

Greetings! By way of brief introduction, I’m a long time lurker on LessWrong and a member of the team building

Forecast

, a crowdsourced prediction and discussion community. Full disclosure: we're part of a labs group,

NPE

, inside of Facebook.

We

launched publicly

a few weeks ago. I wanted to share a little about what we're trying to build and get this community's advice on where we're going.

First, a bit of background

Forecast is built around a

prediction market

: an exchange where people use virtual points to trade on outcomes of future events. Forecast is unique in a few ways:

Members of the community ask all the questions, which range from

serious

to

arcane

and mostly center around current events. Our team currently moderates the questions (mostly to edit them for clarity). Over time, we want to empower the community to self-moderate.
Forecast uses points, rather than real money. Forecasters get points when they join and then regularly get refreshes as they play. There’s a leaderboard that tracks total point ‘profit’.
All activity is public: anyone, whether or not they’re a forecast user, can see who participated and what their transaction/discussion history is. Forecast accounts are tied to your Facebook account behind the scenes, but users can select non-identifying display names to use in the community.
Post-forecast discussion is incentivized. Users get points if other users support the reasons they write to explain their forecasts. Only people who have made a forecast on a particular question can write and support reasons.

Since June, the Forecast community has made more than 50,000 forecasts on a few hundred questions--and they're actually reasonably accurate. Forecast's midpoint

brier score

(measured at the midpoint between a question’s launch and resolution dates) across all closed Forecasts over the past few months is 0.204, compared to Good Judgement's

published result of 0.227

for prediction markets. [Update: as one commenter pointed out, this does not control for time horizon or difficulty, so they're not really comparable metrics. Leaving the comparison, but taking edited to make this clear.]

Why we're building this

Like others in this community, we became interested in prediction markets because,

when they work

, they help push participants to be rational, even at the expense of

being popular

.

Beyond the forecasts themselves, we think what makes Forecast...
Message Length
by Zack_M_Davis — 2020-10-20T05:52:56.277Z

Someone is broadcasting a stream of bits. You don't know why. A 500-bit-long sample looks like this:

01100110110101011011111100001001110000100011010001101011011010000001010000001010
10100111101000101111010100100101010010101010101000010100110101010011111111010101
01010101011111110101011010101101111101010110110101010100000001101111100000111010
11100000000000001111101010110101010101001010101101010101100111001100001100110101
11111111111111111100011001011010011010101010101100000010101011101101010010110011
11111010111101110100010101010111001111010001101101010101101011000101100000101010
10011001101010101111...

The thought occurs to you to do Science to it—to ponder if there's some way you could better predict what bits are going to come next. At first you think you can't—it's just a bunch of random bits. You can't predict it, because that's what random means.

Or does it? True, if the sequence represented flips of a fair coin—every flip independently landing either 0 or 1 with exactly equal probability—then there would be no way you could predict what would come next: any continuation you could posit would be exactly as probable as any other.

But if the sequence represented flips of a biased coin—if, say, 1 came up 0.55 of the time instead of exactly 0.5—then it would be possible to predict better or worse. Your best bet for the next bit in isolation would always be 1, and you would more strongly anticipate sequences with slightly more 1s than 0s.

You count 265 1s in the sample of 500 bits. Given the hypothesis that the bits were generated by a fair coin, the number of 1s (or without loss of generality, 0s) would be given by the binomial distribution (500k)(0.5)k(0.5)500−k, which has a standard deviation of √500⋅0.52=√125≈11.18, so your observation of 265−250=15 excess 1s is about 1511.18≈1.34 standard deviations from the mean—well within the realm of plausibility of happening by chance, although you're at least slightly suspicious that the coin behind these bits might not be quite fair.

... that is, if it's even a coin. You love talking in terms of shiny, if hypothetical, "coins" rather than stodgy old "independent and identically distributed binary-valued random variables", but looking at the sample again, you begin to further doubt whether the bits are independent of each other. You've heard that humans are biased to overestimate the frequency of alternations (10101...
Betting Thread
by Amandango — 2020-10-20T02:17:44.941Z

At Ought, we’ve started making bets on our continuous predictions. We’ve found it a fun way to hold ourselves accountable and to combat overconfidence.

Here’s a thread for people to share continuous beliefs they’re willing to bet on, and make bets on other people’s beliefs. You can use

Elicit

to automatically generate a fair bet with positive expected value based on each person’s beliefs (this post explains more about how we generate the bet).

Make a bet

To share a belief you’re willing to bet on:

Go to

elicit.org

, type in your question, and make a distribution of your beliefs
Take a snapshot
Post the snapshot URL and a screenshot image in a comment on this thread

To bet on someone else’s belief:

Click on their snapshot URL. Click on the title to get a blank distribution
Make your own distribution and take a snapshot
Import your snapshot into theirs by pasting your snapshot URL into the ‘Show more’ dropdown on their snapshot
Go to the Interpret tab to see the bet

Example Bet

Here is a bet we proposed between 538 and The Economist on Biden's electoral college votes:

If the result is between 290.5 and 398.1, 538 pays The Economist $35.98. Otherwise, The Economist pays 538 $64.02.