trentmkelly/LessWrong-43k huggingface dataset:
Bayesiance (Filk)
by Gordon Seidoh Worley — 2020-08-18T16:30:00.753Z

To the tune of "Elegance" from Hello Dolly!, with apologies to Jerry Herman

Yes, Less Wrong
It's really us
Bayesians and we're signing songs
Here to speak of priors, what they are
Bayes' Law, math, and the posterior
What a knack
There is to it
Acting like a rational agent
We are Bayesians
If you ain't a Bayesian
You will always risk a Dutch Book write off
All who are
Well-read agree
Thomas Bayes
Resplendently
Showed the way to conserve evidence
So you never need to lose one pence
Could we be
Misleading you?
Aumann begs to disagree with you
We are Bayesians
If you ain't a Bayesian
You will always risk a Dutch Book write off
[musical interlude]
Kelly bet?
We always do
Cromwell, yes?
We got tattoos
Some think only frequency should count
We know the unknown is paramount
Von Neumann
Bows down to us
Morgenstern he saves a place for us
[bridge]
We've are Bayesians
We were born with Bayesiance
Maybe we should be like Fisher
And demand more evidence
And we'll never make our minds up
Unless passing a t-test
But not every distribution
Rises to normality
So we need subjective inference
Which inevitably, means
[coda]
We are Bayesians
We got built in Bayesiance
And with Bayesiance...Bayesiance...
Bayesiance...Bayesiance...Bayesiance
We avoid write offs!
WordPress Destroys Editing Process, Seeking Alternatives
by Zvi — 2020-08-18T15:50:00.842Z

WordPress had an editor that worked. They have forcibly replaced it with an editor that doesn’t. There does not seem to be a free way to switch back.

Please, if anyone knows a good alternative hosting service, especially one that can seamlessly transfer over the old blog contents, please share in the comments.

There might be a way to switch back to the old editor if I payed them the ransom money to upgrade my account to Business so I could use plugins, but I don’t want to do that unless I’m positive it will work. If you know for a fact that this would still work, let me know that, too.

Even if I knew it would work, it’s really bad decision theory to pay ransom money, so I’d need to think long and hard before doing it even if the amount of money is trivial. I think that if the service is otherwise good enough it’s mostly all right, since I’ve gotten a lot of value from them, but I also welcome thoughts on that front. I’m pretty furious right now.

Also, the new editor looks like it literally doesn’t function – it already ate a post and keeps telling me I don’t have permissions.
Learning human preferences: optimistic and pessimistic scenarios
by Stuart_Armstrong — 2020-08-18T13:05:23.697Z

In this post and the next, I try and clarify - for myself and for others - the precise practical implications of the "Occam's razor is insufficient to infer the preferences of irrational agents
" paper.

Time and again, I've had trouble getting others to understand what that paper implies, and what it doesn't. It's neither irrelevant (like many no-free-lunch theorems), nor is it a radical skepticism/nothing is real/we can't really know anything paper.

I've been having productive conversations with Rebecca Gorman, whom I want to thank for her help (and who phrased things well in terms of latent variables)!

A simple biased agent

Consider the following simple model of an agent:

The agent's actions can be explained by their beliefs and preferences[1], and by their biases: by this, we mean the way in which the action selector differs from an unboundedly rational expected preference maximiser.

The results of the Occam's razor paper imply that preferences (and beliefs, and biases) cannot be deduced separately from knowing the agent's policy (and hence, a fortiori, from any observations of the agent's behaviour).

Latent and "natural" variables

Let x be a latent variable of the policy π - or a some variable that can be deduced from π in some simple or natural way.

A consequence of the Occam's razor result is that any such x will typically be a mixture of preferences, beliefs, and biases. For if the x tended to be restricted to one of these three components, that would mean that separating them would be possible via latent or simple variables.

So, for example, if we conducted a principle component analysis on π, we would expect that the components would all be mixes of preferences/beliefs/biases.

The optimistic scenario

To get around the impossibility result, we need "normative assumptions": assumptions about the preferences (or beliefs, or biases) of the agent that cannot be deduced fully from observations.

Under the optimistic scenario, we don't need many of these, at least for identifying human preferences. We can label a few examples ("the anchoring bias, as illustrated in this scenario, is a bias"; "people are at least weakly rational"; "humans often don't think about new courses of action they've never seen before", etc...). Call this labelled data[2] D.

The algorithm now constructs categories preferences*, beliefs*, and biases* - these are the generalisations that it ha...
On Creativity - The joys of 5 minute timers
by Neel Nanda — 2020-08-18T06:26:55.493Z
Link: https://www.neelnanda.io/blog/post-28-on-creativity-the-joys-of-5-minute-timers

Introduction

A really common failure mode that I observe in myself and others is to get stuck and then give up. To be working on a problem, or trying to generate ideas, find it hard and stop looking. I find that this goes hand-in-hand with the far larger problem of being trapped within your preconceptions - to notice the obvious solution to your problem, the obvious life path ahead of you, the obvious thing somebody in your role should do, and never realising there can be anything more. I think this is a really big problem, because the world is large and complicated. The space of possibilities is enormous and the ability to search beyond what’s obviously visible is really valuable for noticing when you are wasting motion.

I think these problems are, at their heart, a lack of creativity. I think creativity is an incredibly valuable skill and, importantly, a learnable skill, not an inherent trait. This is a skill that I still suck at, but I’ve improved a lot at over time, and this post is my attempt to outline the approach that’s worked best for me.

This is obviously a useful skill in settings like doing art and research. But I’ve found it useful in a wide range of areas in my life. A few highlights:

Solving personal problems
Giving advice to others
Solving the small problems I run into when doing work
Finding ways to have fun
Finding projects I feel excited about

5 Minute Timers

I think there are two main insights here:

Generate then filter ideas

I’m a massive perfectionist, and I find it easy to slip into the failure mode of generating the perfect idea. This is incredibly unhelpful, because this is paralysing in practice, and means I struggle to come up with anything. Any weak and half-formed ideas are discarded before they can be fleshed out into something worthwhile
This is an example of a general failure mode - trying to both generate and filter at the same time.

I find that these involve very different parts of my mind - generating ideas should feel free-flowing and diffuse, while filtering should feel constrained, directed and focused. And these different mindsets can’t be activated at once, so in practice I’m either neglecting one or rapidly switching between them

This especially applies when it comes to any decision that feels scary or overwhelming - eg thinking about life decisions

And this is super, super bad, bec

...
Multitudinous outside views
by Davidmanheim — 2020-08-18T06:21:47.566Z

There's an important piece of advice for forecasters: don't rely on your internal model of the world exclusively, and take the outside view, then adjust from there. But which view is " the" outside view? It depends on the problem - and different people might tell you different things. But if the choice of outside view is subjective, it starts to seems like inside-views all the way down.

That's where we get to base rates, which don't solve this problem, but they do highlight it nicely.

Fans of superforecasting know, in a hedgehog-like sense of knowing one thing, that the outside view, which is the base rate, which is the rate of similar events, should be our starting point. But which events are similar, and how is similarity defined? We first need to choose a reference class, based on some pre-existing idea of similarity. And in different terms, there is a reference class problem, which we evidently don't have a clear way to judge - and even as Bayesian thinkers, not only is that our problem, it's an entire bucket of different problems.

Considering a Concrete Prediction: Tesla Motors

Let's get really concrete: What will the price of Tesla stock be in 6 months?

Well, what is the reference class? In the last year, 90% of the time, the price of Tesla stock has been between $200 and $1000. But that's a really bad reference class, when the price today is $1,800. OK, but looking at the set of all stocks would be even worse - and looking at automobile stocks even worse than that. Which stocks are comparable? What about stocks with P/E ratios over 900? Or stocks with more than a half billion dollars of losses for their net income? We're getting silly here.

Maybe we shouldn't look at stock price, but should look at market capitalization? Or change in price? "Stocks that went up 9-fold over the course of a year" isn't a super helpful reference class - it has only a few examples, and they are all very different from Tesla.

Of course, none of this is helpful. What we really want is the aggregate opinion of the market, so we look at futures contracts and the implied volatility curve for options expiring in February.

That doesn't look like a reference class. But who needs an outside view, anyways?

What is a reference class?

If you want to know the probability of Kim Jung-Un staying alive, we can cons...
10 Fun Questions for LessWrongers
by Ben Pace — 2020-08-18T03:28:05.276Z

Here are 10 fun questions for LWers.

Questions like "how much karma makes a post worth reading" and "do emotions mostly help or hinder you in pursuing your goals".

All multiple choice / dropdown selection (bar an optional one).

I'll publish the results in a few days.

I've filled it out. Fill it out here?
Spending Update 2020
by jefftk — 2020-08-18T01:40:05.390Z

It's been a bit over two years since our last spending update
(

2018,

2016,

2014). We're no longer tracking our
expenses to the dollar in most cases, except where that's required for
taxes, but since we rent out part of our house it turns out that this
covers all the big categories.

For past updates I've generally presented averages over a time period,
but instead here I want to try more of a snapshot approach. This is my
best estimate of what we're spending now, informed by our recent
spending. As housing continues to be a complicated topic, I've used
the same amortized approach as last
time (see footnote 1 there).

Another difference from last time is that this time I'm thinking more
about how there are kind of two different total incomes that makes
sense to consider:

Our Adjusted
Gross Income (AGI): how much money does the IRS think we make.
This is what we
use for figuring out how much to donate, and the IRS is reasonably
invested in making it a good definition of income.

The total amount of money that we, in a sense, direct. This
includes our AGI, but it also includes our paycheck deductions for
health insurance, retirement, etc. Unlike, say, the portion of our
paychecks that is deducted for Social Security Tax, the IRS doesn't
consider this money to be part of our income.

In the past I haven't thought clearly about this, but this time I'm
going to try to run the numbers both ways. Below when I describe an
amount as "pre-AGI", I mean that it's from one of these paycheck
deductions. I'm not including amounts my employer pays, though
there's a sense in which this is arbitrary. If the total cost of our
family's healthcare is $30k, and my employer switches from covering
100% to covering 50% and rases compensation by $15k to match, should
this be showing up in my numbers here? Accounting is tricky.

Anyway, here are the monthly numbers:

Donations: $19,600 (53% of 2019
adjusted gross income)

Retirement saving: $4,500 ($1.6k pre-AGI, the rest is
after-tax 401k
contributions)

Taxes: $4,400

Income tax: $2.8k

State tax: $1.8k

Social Security tax: $1.1k

Medicare tax: $0.7k

Housing: $3,230

One time expenses (all time)

Purchase and all one-time expenses included last time: $1,047k

Replacing the second floor bathroom: $26k

5KW of rooftop solar: $18k/li>

...
Why haven't we celebrated any major achievements lately?
by jasoncrawford — 2020-08-17T20:34:20.084Z
Link: https://rootsofprogress.org/celebrations-of-progress

In reading stories of progress, one thing that has struck me was the wild, enthusiastic celebrations that accompanied some of them in the past. Read some of these stories; somehow it’s hard for me to imagine similar jubilation happening today:

The US transcontinental railroad, 1869

The transcontinental railroad was the first to link the US east and west. Prior to the railroad, to travel from coast to coast could take six months, whether by land or sea, and the journey was hard and perilous. California was like a foreign colony, separated from the life and industry of the East. The railroad changed that completely, taking a six-month journey down to a matter of days.

Here’s how the western cities reacted, from Stephen Ambrose’s book Nothing Like It in the World:

At 5 A.M. on Saturday, a Central Pacific train pulled into Sacramento carrying celebrants from Nevada, including firemen and a brass band. They got the festivities going by starting their parade. A brass cannon, the very one that had saluted the first shovelful of earth Leland Stanford had turned over for the beginning of the CP’s construction six years earlier, boomed once again.

The parade was mammoth. At its height, about 11 A.M. in Sacramento, the time the organizers had been told the joining of the rails would take place, twenty-three of the CP’s locomotives, led by its first, the Governor Stanford, let loose a shriek of whistles that lasted for fifteen minutes.

In San Francisco, the parade was the biggest held to date. At 11 A.M., a fifteen-inch Parrott rifled cannon at Fort Point, guarding the south shore of the Golden Gate, fired a salute. One hundred guns followed. Then fire bells, church bells, clock towers, machine shops, streamers, foundries, the U.S. Mint let go at full blast. The din lasted for an hour.

In both cities, the celebration went on through Saturday, Sunday, and Monday.

The Brooklyn Bridge, 1883

The Brooklyn Bridge did not connect a distance nearly as great as the transcontinental railroad, but it too was met with grand celebrations. An excerpt from David McCullough’s The Great Bridge:

When the Erie Canal was opened in the autumn of 1825, there were four former Presidents of the United States present in New York City for the occasion—John Adams, Thomas Jefferson, James Madison, and James Monroe—as well as John Quincy Adams, then occupying the White House, and General Andrew Jackson, who

...
A way to beat superrational/EDT agents?
by Abhimanyu Pallavi Sudhir — 2020-08-17T14:33:58.248Z

Suppose you have two identical agents with shared finances, and three rooms A1, A2, B.

Flip a fair coin.

If the coin comes up H, put the agents in A1, A2.
If it comes up T, flip the coin again.

If it comes up H, put the agents in A1, B.
If it comes up T, put the agents in A2, B.

(At each point, flip another fair coin to decide the permutation, i.e. which agent goes to which room.)

Now to each agent in either A1 or A2, make the following offer:

Guess whether the first coin-flip came up heads or tails. If you correctly guess heads, you both get $1. If you correctly guess tails, you both get $3. No negative marking.

The agents are told which room they are in, and they know how the game works, but they are not told the results of any coin tosses, or where the other agent is, and they cannot communicate with the other agent.

...

In terms of resulting winning, if an agent chooses to precommit to always bet heads, its expected earnings are $1, but if it chooses to precommit to always bet tails, its expected earnings are $1.50. So it should bet tails, if it wants to win.

But consider what happens when the agent actually finds itself in A1 or A2 (which are the only cases it is allowed to bet): if it finds itself in A1, it disqualifies the TT scenario, and if it finds itself in A2, it disqualifies the TH scenario. In either case, the probability of heads goes up to 2/3. So then it expects betting heads to provide an expected return of $1.33, and betting tails to provide an expected return of $1. So it bets heads.

(There are no Sleeping Beauty problems here, the probability genuinely does go up to 2/3, because new information -- the label of the room -- is introduced. BTW, I later learned this is basically equivalent to the scenario in Conitzer 2017, except it avoids talking about memory wiping or splitting people in two or anything else like that.)

What's going on? Is this actually a way to beat superrational agents, or am I missing thing? Because clearly tails is the winning strategy, but heads is what EDT tells the agent to bet.
GPT-3, belief, and consistency
by skybrian — 2020-08-16T23:12:10.659Z

I've seen a few people debating what GPT-3 understands, and how this compares to human understanding. I think there's an easier and more fruitful question to consider: what does it believe?

It seems like it doesn't believe anything, or alternately, it believes everything. It's a category error, like asking what a library believes, or what the Internet believes. But let's go with that metaphor for a bit, because it seems interesting to think about.

For a library, contradictions don't matter. A library can contain two books by different authors saying opposite things, and that's okay since they are just being stored. Maybe it's better to think of GPT-3 as a large, interestingly-organized memory than as an agent? But like human memory, it's lossy, and can mix up stuff from different sources, sometimes in creative ways.

How does GPT-3 resolve inconsistency? If the Internet is very consistent about something, like the words to Jabberwocky, then GPT-3 will be consistent as well. If there were two different versions of Jabberwocky that started the same and diverged at a certain point and they were equally popular in the corpus, then it would probably choose between them randomly, if you have randomization turned on at all.

Sometimes, GPT-3 can choose between beliefs based on style. Suppose that grade-school science material is written in one style and flat-earth rants are written in a different style. It wouldn't be surprising that GPT-3 would appear to have different beliefs about the shape of the earth based on which style of work it's completing. Or, if it can recognize an author's style, it might seem to have different beliefs based on which author it's pretending to be.

If GPT-3 can play chess, it's due to online consensus about how to play chess. If we had two different chess-like games using similar notation then it might get them confused, unless the context could be used to distinguish them.

If base-10 and base-8 arithmetic were equally common in the corpus then I don't think it could do arithmetic very well either, though again, maybe it can distinguish them from context. But if it doesn't know the context, it would just guess randomly.

Of course, contradictions are everywhere. We compartmentalize. None of us are logic robots that halt when we find a contradiction. However, contradi...
Goal-Directedness: What Success Looks Like
by adamShimi — 2020-08-16T18:33:28.714Z

This sequence already contains a couple of blog posts, exploring different aspects of goal-directedness. But one question has never been fully addressed: what constraints should a good formalization of goal-directedness satisfy? An answer is useful both for people like me which study this topic, and for people trying to assess the value of this research. The following is my personal view, as always informed with discussion with Michele Campolo, Joe Collman and Sabrina Tang.

So what makes a good formalization of goal-directedness? The first part comes from what it means to formalize a set of philosophical intuitions. On that front, I agree with Vanessa in her research agenda:

Although I do not claim a fully general solution to metaphilosophy, I think that, pragmatically, a quasiscientific approach is possible. In science, we prefer theories that are (i) simple (Occam's razor) and (ii) fit the empirical data. We also test theories by gathering further empirical data. In philosophy, we can likewise prefer theories that are (i) simple and (ii) fit intuition in situations where intuition feels reliable (i.e. situations that are simple, familiar or received considerable analysis and reflection). We can also test theories by applying them to new situations and trying to see whether the answer becomes intuitive after sufficient reflection.

So the first step is to fit the main intuitions about goal-directedness. This is the point of ideas like focus, short descriptions, and locality. The core intuitions should probably emerge from a community discussion, such that a consensus is reached. Then, fitting the intuitions looks like an optimization problem: for each formalization, there is a "distance" to the intuitions. The point is to minimize this distance, or at least come close to the minimum.

Yet this is not a pure philosophical endeavor. I study goal-directedness to understand if Rohin's position in this sequence of posts is right. I want to know if we as a community should invest time and efforts into less goal-directed approaches. This depends on two propositions by Rohin: that a less goal-directed system is not necessarily trivial or uncompetitive; and that being less goal-directed removes some safety issues, like convergent instrumental subgoals and wireheading.

Thus our initial optimization problem is actually a constrained optimization...