trentmkelly/LessWrong-43k huggingface dataset:
Imitative Generalisation (AKA 'Learning the Prior')
by Beth Barnes — 2021-01-10T00:30:35.976Z

Tl;dr

We want to be able to supervise models with superhuman knowledge of the world and how to manipulate it. For this we need an overseer to be able to learn or access all the knowledge our models have, in order to be able to understand the consequences of suggestions or decisions from the model. If the overseers don’t have access to all the same knowledge as the model, it may be easy for the model to deceive us, suggesting plans that look good to us but that may have serious negative consequences.

We might hope to access what the model knows just by training it to answer questions. However, we can only train on questions that humans are able to answer[1].

This gives us a problem that’s somewhat similar to the standard formulation of transduction: we have some labelled training set (questions humans can answer), and we want to transfer to an unlabelled dataset (questions we care about), that may be differently distributed.

We might hope that our models will naturally generalize correctly from easy-to-answer questions to the ones that we care about. However, a natural pathological generalisation is for our models to only give us ‘human-like’ answers to questions, even if it knows the best answer is different. If we only have access to these human-like answers to questions, that probably doesn’t give us enough information to supervise a superhuman model.

What we’re going to call ‘Imitative Generalization’ is a possible way to narrow the gap between the things our model knows, and the questions we can train our model to answer honestly. It avoids the pathological generalisation by only using ML for IID tasks, and imitating the way humans generalize. This hopefully gives us answers that are more like ‘how a human would answer if they’d learnt from all the data the model has learnt from’. We supervise how the model does the transfer, to get the sort of generalisation we want.

It’s worth noting there are enough serious open questions that imitative generalization is more of a research proposal than an algorithm!

This post is based on work done with Paul Christiano at OpenAI. Thanks very much to Evan Hubinger, Richard Ngo, William Saunders, Long Ouyang and others for helpful feedback, as well as Alice Fares for formatting help

Goals of this post

This post tries to explain a simplified[2] version of Paul Christiano’s mechanism introduced here, (referred to there as ‘Le...
Babble Thread
by Adam Zerner — 2021-01-09T21:52:12.383Z

On LessWrong, we have a sort of hierarchy of posts:

Curated
Front Page
Personal Blog
Open Thread / Short Form

Here I'd like to experiment with a fifth: the Babble Thread.

What is babbling? Babble is the post that introduced the concept, I believe. The Babble and Prune tag is also worth checking out. But for me, seeing the Babble Challenges as a concrete example is what made the concept click.

In this thread... babble away! About anything! Share your ideas, and don't hold back! It's ok if they end up being wrong or silly. The purpose here is just to babble. Some ideas for what to babble about:

Important problems like AI safety
Useful things like life hacks
World modeling, like how efficient you think the market for prestige in academia is
Reflections on your personal life
Trivial things like how hard it is to tuck your bed sheets in properly
Ideas for startups or scientific experiments

To this end, I think it'd make sense to avoid commenting on others' babbles. It seems best to make this more of a safe space where you can babble without fear of being judged.

However, pruning does have a place too. I think a good balance would be to allow upvoting on top-level comments (babbles) as a means to do some initial pruning and identify ideas worth following up on for further pruning. If we want to be really strict about it I think we'd separate this upvote/initial pruning step from the generate ideas/babble step, but as a practical matter it'd be easier to upvote in this thread.

I'm not attached to this approach by any means, so please comment in the top level meta comment if you have thoughts on how this Babble Thread should be structured.
[U.S. specific] PPP: free money for self-employed & orgs (time-sensitive)
by Steven Byrnes — 2021-01-09T19:53:09.088Z

[cross-posted from EA forum]

PPP is the "paycheck protection program", a big US government giveaway to businesses including self-employed people (e.g. many grant recipients) and nonprofits. There was a PPP round last year—see this EA Forum post and cross-post on LW. There's another round of PPP starting imminently, like maybe this week. As a self-employed person, you can get a check for the equivalent of 2.5 months of net income—this is serious stuff, well worth the time to look into. If you looked into it last year and chose not to apply, consider looking again—there's a lot more information now than there was last year, including clarification on who is eligible and how it works.

For my part, I have a family member who is planning to apply for the first time. She skipped the last round because she was better off claiming unemployment at the time. Therefore the info below assumes that you have not applied to PPP before, since that's the case I looked into. But if you have already gotten a PPP check last year, you may still be eligible to re-apply and get even more free money this round. Please comment if you know more details about that.

I'm not an expert, this is just from a bit of online research. Here are some highlights that I found surprising or important:

The amount is impressively high - independent contractors get 2.5/12 = 20.8% of their 2019 or 2020 yearly net income (Schedule C line 31). A couple hours of research and paperwork is as lucrative as 11 weeks of work!!!!
Despite the name, you don't need to have employees or give anyone paychecks. If you're self-employed, your net profits (Schedule C line 31) are sorta treated like you're paying yourself. (Search keyword: "owner compensation replacement".)
It's open to nonprofits too.
It's called a "loan" but if you follow the formula and ask for the right amount of money, you don't have to pay it back, you just fill out a pro forma 2-page application and your entire "loan" including interest is forgiven.
If this is your first PPP loan, you don't need to document any pandemic-related harm to your business (!!)

...However, you do need to "certify in good faith" that "current economic uncertainty makes this loan request necessary to support the ongoing operations of the Applicant". Before you scrupulously declare that this doesn't apply to you, consider (1) reading this comment and surrounding discussion, and (2)

...
I think to find out what I write
by Bob Baker — 2021-01-09T18:55:36.160Z

The maxim goes that “I write to find out what I think”. (Variously attributed to Flannery O'Connor or Joan Didion). It counsels that articulation highlights gaps in arguments, requires concrete positions, and dispels denial of inconvenient conclusions. I do not dispute it.

However, I suspect that the effects flow in both directions. When writing, certain thoughts bubble up from the surface of the mind at the opportune time, by chance. The demands of concision, the limits of time, and the lionisation of decisiveness then exclude the alternatives. Warranted or not, writing amplifies those lucky musings. Unjustified confidence condenses from ambiguity. The words become a commitment, provoking cognitive dissonance when the mind strays elsewhere. Doubly so if published.

Write with care.
The Case for a Journal of AI Alignment
by adamShimi — 2021-01-09T18:13:27.653Z

When you have some nice research in AI Alignment, where do you publish it? Maybe your research fits with a ML or AI conference. But some papers/research are hard sells to traditional venues: things like Risks from Learned Optimization, Logical Induction, and a lot of research done on this Forum.

This creates two problems:

(Difficulty of getting good peer-review) If your paper is accepted at a big conference like NeurIPS, you'll probably get useful reviews, but it seems improbable that those will focus on alignment in the way most AI Alignment researchers would. (I'm interested in feedback here).
And if your research is confined to arXiv or the Alignment Forum, it can be really hard to get any sort of deep feedback on it.
(Dispersal of peer-reviewed research) The lack of a centralized peer-reviewed source of alignment research means that finding new papers is hard. Most people thus relies on heuristics like following specific researchers and the Alignment Forum, which is not the best for openness to new ideas.

I think that creating a research journal dedicated to AI alignment would help with both problems. Of course, such structure brings with it the specter of academia and publish-or-perish -- we must thread carefully. Yet I believe that with enough thinking about these problems, there is a way to build a net positive journal for the field.

What it could look like

Let's call our journal JAA (Journal of AI Alignment). I choose a journal over a conference because of the rolling basis of application for the former and the fact that journals don't have to organize IRL meetings.

How does JAA work?

For the board of editors, I'm thinking of something similar to what the AF does: one editor from each big lab, plus maybe some independent researchers. The role of the editors is to manage a given submission, by sending it to reviewers. We probably want at least one of these reviewers to be from another part of AI Alignment, to judge the submission from a more outside view. There is then some back and forth between reviewers and authors, moderated by the editors, which results in the paper being either accepted or rejected.

A specificity of this process would be to require a reason why this research is useful for aligning AIs, and a separate document about the information hazards of the submission.

I also think the process should be transparent (no anonymity and the reviews ...
Out-of-body reasoning (OOBR)
by  — 2021-01-09T16:10:34.665Z

The abbreviated title is a pun on "über". As the über-theory, it has to be believed, right?

Outline

I'd like to suck all of the joy and intrigue out of Sleeping Beauty and related problems by claiming that a rational agent always, boringly, makes decisions thus:

Assume a God-like "out-of-body" mindset of defining an experiment in which you are the (programmable) subject, and a probability space or model representing the objective chances in that experiment.
Define a reward function mapping a decision strategy (program, if you will) and experimental outcomes to a real number, i.e. how well the strategy panned-out for each possible turn of events.
Choose your decision strategy to maximise the expected reward.

If you can define the model & experiment unambiguously, there is no need for credence: the rational agent does not use it for decision-making. Let's put it this way:

Inner credence, being any statistic derived wholly from the model and the observed outcomes, does not influence rational decision-making. There are multiple natural ways of doing this, e.g. "halfer" vs. "thirder", but the rational agent isn't obliged to employ any of them.

If you cannot define the model/experiment unambiguously, perhaps because you were given only partial information about your circumstances or you cannot compute the complex physical model that determines your world, then you must invoke your:

Outer credence, that assigns subjective probability to the various possible models/experiments.

Then use your outer credence to define a new, hybrid model that says: pick a model/experiment according to outer credence, then enact that experiment. Make a hybrid reward function to fit. Then, as before, choose a decision strategy maximising the expected reward.

Sleeping Beauty

See https://www.lesswrong.com/tag/sleeping-beauty-paradox.

The model is that a coin is flipped. The experiment involves waking Beauty some number of times depending on the coin flip, with drugs applied to force all wake-ups to be indistinguishable.

Her reward function depends on the terms of the experiment. Here are two experiments, each a mild variant of the original Sleeping Beauty problem:

Experiment 1. Coin is biased 1:2 (Heads:Tails); wake once in either case. Guess the coin ("H"/"T"); receive $1 each time you're right. The reward function should reflect your prof

...
Practical post on 4 #ProRep methods (Proportional Representation)
by Jameson Quinn — 2021-01-09T14:56:29.431Z

Context

I have written a series of longer posts about voting theory: a general primer, single-winner pathologies, multi-winner method building blocks, and a longer, harder-to-read piece where I struggled through to a metric for multi-winner methods.

I believe that I should write a simpler, (relatively) shorter piece on proportional multi-winner methods. This is partly because my own thinking has evolved and sharpened, and partly because I think that current events in the US, Canada, and UK make these ideas more timely:

In the US, the Democratic party is poised to have the trifecta they'd need to pass Federal laws, and they seem to be motivated to look at fundamental political reforms due to such events as the recent (ineptly) attempted coup.
In Canada, various people, including Jagmeet Singh (the head of the third-largest party by votes) have called for a Citizens' Assembly on election method reform.
The UK is also in political crisis due to Brexit and coronavirus, and election reform seems to me to be on the agenda.

In all three of these cases, the situation favors grassroots calls for reform, but it seems to me that a substantial number of existing political incumbents will have to be on board for this reform to succeed. This means that reform that is especially hostile to such incumbents would probably be vetoed.

It is utterly clear to me that election method reform, based on some form of proportional representation, could be a key step towards healthier politics in all three places. It would not be a panacea; politics is, and always will be, a battleground of competing interests, and whatever your sympathies are, I suspect you see some of those interests as less valid than others. But it would help.

This article only mentions the best methods. As a co-organizer of the 2018 British Columbia Symposium on Proportional Representation; a committee-member of the Equal.Vote committee on proportional representation methods; and the primary designer of the proportional voting method used for Hugo Award nominations; I could explain dozens of different (and interesting) #ProRep methods if I wanted. But this is focusing on those I think are worth knowing from a practical standpoint.

Note: there is a short summary of this article at the end.

Pros and Cons of #ProRep in General

Any form of proportional representation would have several huge advantages over the current "first pas...
Will Donald Trump complete his first term?
by Annapurna — 2021-01-09T12:13:57.456Z

PredictIt has the "yes" at 0.80 cents: https://www.predictit.org/markets/detail/5158/Will-Donald-Trump-complete-his-first-term

Polymarket has the "yes" at 0.85 cents: https://polymarket.com/market/will-trump-complete-his-first-term

My intuition is telling me that there's an odds mis-pricing going on and there is a +EV bet to be made.

What do you think?
Thoughts on Mustachianism
by Adam Zerner — 2021-01-09T09:27:36.839Z

I've always been a Mustachian. Recently I've been noticing myself having some diverging thoughts about it. But despite those diverging thoughts, there's still a lot of core ideas that I really identify with.

Anyway, here are some assorted thoughts.

He doesn't always advise frugality

People often assume he's all about frugality. In reality, he thinks that some things are worth spending good money on. For example, he likes the idea of splurging a bit on a home because of how important it is.

I live in one of the nicer houses in my town’s nicest (to me) neighborhood*. I love the four bedrooms and four bathrooms and the nice renovations I’ve done throughout this place over the past five years. It’s not the cheapest place to live, but to me it’s the best value of living pleasure to the dollar I could create. A house to me is the home base of your spirit, and when you’re living a frugal and natural life, you spend a lot of time at home. As a result, when I compare the sunk cost of my housing to that of other people, I come out behind.

Part of the blame for this is on readers for being uncharitable. But part of the blame is also on MMM, in my opinion. He spends an awful lot of time talking about frugality, giving examples of how you'd be better off by being more frugal. He spends very little time talking about things that are worth splurging on. When you allocate your time that way, it can be easy for readers to get the wrong impression.

Bang for your buck is a very central idea

Think about the following spectrum:

Ramen

Chicken and rice

Home cooked beef and broccoli

Postmates

Sit down restaurant

Fancy restaurant

As you move along it, things get more expensive, but also higher quality. So then, in figuring out what the best point on the spectrum is, you have to think about bang for your buck.

In going from home cooked beef and broccoli to Postmates, I'd argue that the bang for your buck is usually pretty bad. It'll be a pretty big step up in price for food that really won't be (much) better than what you'd make at home.

In going from Postmates to a sit down restaurant, I'd argue that the bang for your buck is usually pretty good (just don't go to Olive Garden). For an extra 5-10 bucks or so you can enjoy some truly legit food.

MMM argues that in many different situations, people usually are at point 4 on the spectrum when the bang for your buck is terrible over th...
Credence polls for 26 claims from the 2019 Review
by Bird Concept — 2021-01-09T07:13:24.166Z

This post is a whirlwind tour of claims made in the LessWrong 2019 Review. In some cases, the claim is literally quoted from the post. In others, I have tried operationalising it into something more falsifiable. For example:

Book Review: The Secret of Our Success

Elicit Prediction (elicit.org/binary/questions/itSayrbzc)

Elicit Prediction (elicit.org/binary/questions/5SRTLX3p_)

Elicit Prediction (elicit.org/binary/questions/VMv-KjR87)

Overall, treat the claims in this post more like polls, and less like the full-blown forecasting questions you'd find on Metaculus or PredictIt. (The latter have extremely high bars for crisp definitions.) They point in a direction, but don't completely pin it down.

Overall, this is an experiment. I'm trying to find interesting ways for people to relate to the Review.

Maybe speeding through these questions gets you thinking good thoughts, that you can then turn into full-blown reviews? Maybe others' answers allow you to find a discussion partner who disagrees on a core question? Maybe the data will be useful in the voting phase?

We'll see!

Feel free to leave a comment about how you found the experience, if you want.

If you want to discuss the questions with others over a call, you can do so during the Review forecasting sessions we're organising this weekend (January 9-10).

If you want to hide other user's predictions until you've made your own, here's how to do that:

Click "Edit Settings"

Go to "Site customizations"

Press "Hide other users' Elicit predictions until I have predicted myself"

Make More Land

Making more land out of the about 50mi^2 shallow water in the San Francisco Bay, South of the Dumbarton Bridge, would...

Elicit Prediction (elicit.org/binary/questions/KkqpSr5rW)

Elicit Prediction (elicit.org/binary/questions/qzzNzEfa9)

Elicit Prediction (elicit.org/binary/questions/csYlcNdhZ)

Elicit Prediction (elicit.org/binary/questions/RwtAoMlnP)

Elicit Prediction (elicit.org/binary/questions/xGIZipvb-)

Elicit Prediction (elicit.org/binary/questions/zAtqSgbnS)

Why Wasn't Science Invented in China?

Elicit Prediction (elicit.org/binary/questions/YP6-jqCeJ)

The Strategy-Stealing Assumption

...
Johannes Kepler, Sun Worshipper
by Ruby — 2021-01-09T07:09:30.417Z

This is crosspostd from a late-night whimsical Facebook post, maybe should be a shortform post.

Individual scientists embrace a new paradigm for all sorts of reasons and usually for several at once. Some of these reasons—for example, the sun worship that helped make Kepler a Copernican—lie outside the apparent sphere of science entirely"
– Thomas Kuhn [1]

Wait wut?

You mean to say that Johannes Kepler adopted the Copernican heliocentric model not because of any scientific merit but because he worshiped the sun like some Egyptian?

I note a lot of references for future potential use, but this is one I had to look up then and there.

Kuhn takes this from E. A. Burrt who applies the term "sun worship" somewhat creatively. Kepler was in fact a devout Christian, not a follower of Ra (too bad, would have been cool). He was deeply motivated by finding simple models that showed had the natural world was simple and united (and thereby mirrored the divine), and that was one motivation for adopting Coperniciasm; however, Burrt says the most potent (and perhaps sufficient) factor for Kepler adopting the theory was the exaltation and dignity it gave to the sun.

In support, Burtt cites a fragment of a disputation that a 22yo Kepler argued in at Tübingen in 1593:

"In the first place, lest perchance a blind man might deny it to you, of all the bodies in the universe the most excellent is the sun, whose whole essence is nothing else than the purest light, than which there is no greater star; which singly and alone is the producer, conserver, and warmer of all things; it is a fountain of light, rich in fruitful heat, most fair, limpid, and pure to the sight, the source of vision, portrayer of colours...[see full excerpt in attached image]"
– cited in E. A. Burrt [2]

Kepler, age 22, disputing Copernicanism at Tübingen in 1593

Alright, Burrt, "sun worship" seems fair. Dude was really into the sun.

In addition to his solar-adulations, Kepler was also a very badass mathematician and empiricist, extremely data and precision obsessed. He treated the parabola as the limiting case of ellipse and hyperbola, showed that parallel lines can be regarded as meeting at infinity, applied infinitesimals to volume calculations preparing the way for Newton and Leibniz, and other stuff I don't understand.

He also said badass things like: "without proper experiments, I conclude nothing!" [exclamatio...
What currents of thought on LessWrong do you want to see distilled?
by ryan_b — 2021-01-08T21:43:33.464Z

The question is inspired by a few comments and a question I have seen recently. The first is a discussion in the 2019 Review post on the subject of research debt; the second a question from johnswentworth asking what people's confusions are about simulacra (which I interpret to be a 'what do you want from this distillation' question).

The question is what it says in the title, but I would like to add that there is no expiration. For example, I recently saw cryogenics back in the posts and questions, which had fallen off the activity radar for years. So old currents of thought are valid candidates, even if the real goal is a re-distillation in light of new developments in the field or all the accumulated communication technique we've considered on LessWrong.

So please describe the current of thought, and your reason for wanting a distillation. The authors may be called to action, or alternatively following Bridgett Kay's suggestion someone else may take up the challenge.