trentmkelly/LessWrong-43k huggingface dataset:
Talking Through A Fear of Death
by jesseduffield — 2019-12-26T05:23:34.670Z

Epistemic Status: semi-stream-of-consciousness effort to talk myself through how I really feel about death.

After waiting in line for what feels like a lifetime, I finally get the chance to talk to the Oracle about something that’s been bothering me.

Me: I’m afraid of death

Oracle: Good for you

Me: How is that good? Not only is death objectively terrifying, the fact I’m terrified of it is itself a psychological harm that may well linger throughout my life

Oracle: First of all, is death really objectively terrifying? Is it a coincidence that after millions of years of humans evolving to survive at all costs, all humans have an innate fear of death? If it was evolutionarily advantageous for you to try and get yourself killed, maybe death wouldn’t seem so bad. Putting your strong inbuilt bias for survival aside, why do you take your fear of death any more seriously than other arbitrary fears you’ve inherited from evolution, for example, asking out a girl. Chances are you’ve feared that situation more than death at some point in your life, and afterwards, whether it went good or bad, I can’t imagine that you would have reasoned that the fear really was proportionate to the stakes. Surely you’ve realised that humans often have very strong fears towards things that aren’t that bad.

Me: Well obviously the fear you feel towards some outcome is rarely going to match up with the actual outcome because our brain has to make an approximation of just how bad that outcome is and how much more our lives will suck after it happens. That doesn’t mean fear is unreliable. And even if I’m afraid of something because I’ve evolved that way, that doesn’t change anything. If I’m afraid of burning my hand on a flame because I’ve evolved that way, I should still try not to burn myself because I’ve also evolved to actually experience pain when my hand is burnt and that’s an experience I want to avoid.

Oracle: But fear of death is the exceptional case here because you won’t actually get to experience the other side. You certainly haven’t evolved to feel anything after death because what good would that do you when you’re dead and immobile. Your brain is making an attempt at simulating what it would be like to still be conscious but be without a body and w...
Gortha Ick
by jefftk — 2019-12-26T03:20:01.923Z

Every year my family gets together at Christmas, and in the evenings
we sing carols.

Up on the Housetop always ends with this
parody verse:

Pa, Ma, and Grandpa, and Grandma sad,

All I declare have something bad

Even the baby hates its part

Hugging a rattlesnake close to its heart

Yo ho ho, who wouldn't go

Yo ho ho, who wouldn't go

Up on the housetop, click, click, click

Down through the chimney with Gortha Ick

My great-grandmother wrote this with her sisters as a child in North
China, around 1890. Gortha Ick was their nickname for the neighbor's
cat.

The "yo ho ho"s aren't parodical; we sing the whole song that way.
I'm not sure where that came from; it looks like the original
1864 lyrics are "O! O! O!":

Comment via: facebook
[AN #78] Formalizing power and instrumental convergence, and the end-of-year AI safety charity comparison
by Rohin Shah — 2019-12-26T01:10:01.626Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Merry Christmas!

Audio version

here

(may not be up yet).

Highlights

2019 AI Alignment Literature Review and Charity Comparison

(Larks) (summarized by Rohin): As in

three

previous

years

(

AN #38

), this mammoth post goes through the work done within AI alignment from December 2018 - November 2019, from the perspective of someone trying to decide which of several AI alignment organizations to donate to. As part of this endeavor, Larks summarizes several papers that were published at various organizations, and compares them to their budget and room for more funding.

Rohin's opinion: I look forward to this post every year. This year, it's been a stark demonstration of how much work doesn't get covered in this newsletter -- while I tend to focus on the technical alignment problem, with some focus on AI governance and AI capabilities, Larks's literature review spans many organizations working on existential risk, and as such has many papers that were never covered in this newsletter. Anyone who wants to donate to an organization working on AI alignment and/or x-risk should read this post. However, if your goal is instead to figure out what the field has been up to for the last year, for the sake of building inside view models of what's happening in AI alignment, I might soon write up such an overview myself, but no promises.

Seeking Power is Provably Instrumentally Convergent in MDPs

(Alex Turner et al) (summarized by Rohin):

The Basic AI Drives

argues that it is instrumentally convergent for an agent to collect resources and gain power. This post and

associated paper

aim to formalize this argument. Informally, an action is instrumentally convergent if it is helpful for many goals, or equivalently, an action is instrumentally convergent to the extent that we expect an agent to take it, if we do not know what the agent’s goal is. Similarly, a state has high power if it is easier to achieve a wide variety of goals from that state.

A natural formalization is to assume we have a distribution over the agent's goal, and define power and instrumental convergence relative to this distribution....
Get a Life
by Stanislav Kozlovski — 2019-12-26T00:51:14.063Z

Invariably anyone, anywhere who lives a focused, dedicated life committed to something is confronted with this phrase. "All you do is play tennis? Every single day? Man, you need to GET A LIFE!"

But the question is: what kind of life exactly do I need to get? Should I get your life; practicing for a couple of meaningless hours a couple of meaningless times a week while leaving the majority of your time for chasing babes at the drug and alcohol parties at your friends' houses? Should I get your life; working nine to five at a dead-end job that tries to convince you otherwise by granting you varied perks and then spending your off-time traveling to exotic locales pretending that the change of scenery will change the emptiness you feel inside? Should I get your life; gathering academic degrees like a kid on a scavenger hunt while spending your summers journeying the globe in the name of becoming "richer" and more "cultured"? Or should I get your life; going to church every Sunday like you're supposed to, quoting scriptures like you're required to, condescending down to the heathens like you're taught to and cussing and drinking and sinning when you need to? Is this how it is? Is this how it should be?

What is the point of a life anyway? Is it how much you do? Is it how much you experience? If I should travel to the Grand Canyon, the Taj Mahal, the pyramids of Egypt and the White House, do I then "have a life"? If I study hard in school (leaving the requisite time for the partying, philandering and law-breaking that my parents don't know about, of course), finish near the top of my class, go on scholarship to a "good school" ("My little Susie gets straight A's. She's going to a good school!"), and then follow that up with a tasty graduate school that then allows me to get a good job that gets me out from behind the desk and lets me see the world, do I then have a life? If I do all that, will I stop waking in the middle of the night wondering what is the purpose of all of this? Will I then be able to stop trying so desperately to find the next, great, distracting event because I hate it when it's quiet and I'm by myself with nothing to do?

The problem today in society and everyday lives everywhere is that people actually believe this phrase. People really do thin...
Values Assimilation Premortem
by wolverdude — 2019-12-26T00:30:06.496Z

In the past 3-4 years, I went through a prolonged and painful life crisis in which I systematically deconstructed my existing worldview and slowly moved away from Evangelical Christianity into something Rationalist or Rationalist-adjacent. In the past 4 months, I've started hanging around the Berkeley Rationality community and am now dating someone embedded therein. At this point my partner is still my main connection to the specific values and practices of the community, and given that my worldview is currently being fleshed-out, she has an outsized influence on what my future beliefs and values may look like.

She has become concerned that this dynamic may be unhealthy, and to be honest, I have too. I think there is a nontrivial chance that in 10 years (or some other future timeframe), I will again find myself in some sort of existential crisis, and realize looking back that I adopted inauthentic values in this present season. So let's reach into the future and read a reflection my future self wrote about this to see if it might reveal something of what went wrong...

I was floating in philosophical space, without a firm orientation when suddenly my partner appeared, offering me both a relationship and a philosophy to orient myself. As a result, I rather uncritically swallowed her values whole without giving myself time to crystalize my own values, values that reflect how I can sustainably live whole in the world.

At the time, I thought I was thinking through them, but in reality I did not have the tools or background to think through them; I had encountered no counter-thought that would equip me to vet them properly. They were well thought-out points to my eyes, and I wanted to believe them. I was tired of drifting. I really wanted to believe something, to have some basis on which to build my life. Even more my feelings for my partner and the prospect of the deep, long-term care and connection I had always dreamt of combined with my natural people-pleasing nature and made me want to be on the same page with her.

So I pretended I was in total agreement. I ignored that little voice in the back of my head saying, "I'm not sure about this," and "Something feels off here." The voice was very weak and I didn't want to hear it. It would only complicate things. And it wasn't well calibrated anyway. I still felt torn about my apostasy, especially when interacting with

...
Understanding Machine Learning (III)
by Rafael Harth — 2019-12-25T18:55:55.715Z

This is the final part of a 3-part series on the fundamentals of Machine Learning as defined by this book (although it will transition into a longer sequence about the remaining, more practical part of the book). If you're interested, begin here.

Nonuniform Learning

Recall the definition of PAC learnability:

A hypothesis class H is called PAC learnable iff there exists a learner A and a function m∗:(0,1)2→N such that, for any probability distribution D over X×Y, any gap ϵ∈(0,1), and any failure probability δ∈(0,1), if the training sequence S consists of at least m∗(ϵ,δ) i.i.d. examples sampled via D, then with probability at least 1−δ over the choice of S, it holds that ℓ(A(S))≤ℓ(h∗)+ϵ for all h∗∈argminh∈H[ℓ(h)].

One way to weaken this definition so that it applies to a wider range of hypothesis classes is to allow the function m∗ which upper-bounds the sample complexity to also depend on the hypothesis we are competing with – so rather than upper-bounding the difference of A(S) and h∗, we upper-bound the difference of A(S) and h′ separately for each h′∈H. We have the following formal definition:

A hypothesis class H is called nonuniformly learnable iff there exists a learner A and a function z∗:(0,1)2×H→N such that, for any probability distribution D over X×Y, any gap ϵ∈(0,1), any failure probability δ∈(0,1), and any hypothesis h∈H, if the training sequence S consists of at least z∗(ϵ,δ,h) i.i.d. examples sampled via D, then with probability at least 1−δ over the choice of S, it holds that ℓ(A(S))≤ℓ(h)+ϵ.

(There is also a second relaxation, where the sample complexity is allowed to depend on the probability distribution D in addition to the hypothesis and the gap and failure probability. It is called consistency, but it turns out to be too weak to be useful. One can prove that every countable hypothesis class is consistent, and that Amemorize is a successful "learner.")

For an example of a hypothesis class that is nonuniformly learnable but not PAC learnable, consider the domain X=R, the usual binary label class Y={0,1}, and the hypothesis class H of all po... ,
12020: a fine future for these holidays
by Stuart_Armstrong — 2019-12-25T15:01:33.788Z

To celebrate all the possibilities of humanity during these holidays, have a possible calendar of the year 12020 of the human era (link to full calendar here).

Minor fact: in the Gregorian calendar, the days of the week cycle exactly every 400 years, so the non-time-travellers among you can use this for 2020 as well...

(previous holiday specials can be found here and here)
Absolutely Winning Bridge Hands
by jefftk — 2019-12-25T02:50:14.711Z

In

bridge
the best you can do is commit to take all the cards and then follow
through. Even better if you do this without the help of a trump suit.
Playing today I was curious: what fraction of hands are "absolutely
winning" for 7NT, in that they'll let you make this contract no matter
what your partner or opponents happen to hold?

Here's one example hand:

Clubs: A, K, Q

Diamonds: A, K, Q

Hearts: A, K, Q

Spades: A, K, Q, J

And here's a very different example:

Clubs: A

Diamonds: A

Hearts: A

Spades: A, K, Q, J, 10, 9, 8, 7, 6, 5

Any absolutely winning 7NT hand needs to have all the aces, so that
whatever your opponent leads you can take the first trick. Then all
the other cards need to be "good", in that no other player can have a
higher card in their suit.

Since a card being "good" requires that you also have every higher
card in the same suit, all that matters for each suit is its length.
If I tell you that an absolutely winning hand has four spades, you
know they are A, K, Q, J. What we're doing, then, is assigning
positive lengths to suits, where all lengths must sum to 13. But it's
simpler if we ignore the aces, since you have to have all four, and
assign non-negative lengths that add to 9. How many ways can we do
this?

Since I'm more of a programmer than a mathematician, here's a way to
solve this with code:

print(len([
(c, d, h, s)
for c in range(10)
for d in range(10 - c)
for h in range(10 - c - d)
for s in range(10 - c - d - h)]))

Which gives 715. [1] There are 52-choose-13 bridge hands (about 635B) so
your chances of getting an absolutely winning 7NT hand is 715 in 635B
or just a bit better than one in a billion.

While very unlikely, this is in the "it could happen, maybe" range and not
all the way to "no way, unless the shuffle was rigged".

EDIT: the above is wrong in two ways. First, the number of spades is completely determined by the number of the other three suits, and so for s in range(10 - c - d - h) should not have been included. And second, if you have seven or more of a suit then which lowest cards you have doesn't matter, because by the time you get to them no one else will have any of the suit. Here's a revised solution that fixes these two:

from math import factorial

def choose(n, r):
return factorial(n) // factorial(r) // factorial(n-r)

count = 0

# Start by

...
Another AI Winter?
by PeterMcCluskey — 2019-12-25T00:58:48.715Z
Link: http://www.bayesianinvestor.com/blog/index.php/2019/12/22/another-ai-winter/

Robin Hanson has been
suggesting
recently that we've been experiencing an AI boom that's not too
different from prior booms.

At the recent Foresight Vision
Weekend, he predicted [not exactly - see the comments] a 20%
decline in the number of Deepmind employees over the next year
(Foresight asked all speakers to make a 1-year prediction).

I want to partly agree and partly disagree.

I expect many companies are cluelessly looking into using AI in their
current business, without having a strategy for doing much more than
what simple statistics can do. I'm guessing that that mini-bubble has
peaked.

I expect that hype directed at laymen has peaked, and will be lower for
at least a year. That indicates a bubble of some sort, but not
necessarily anything of much relevance to AI progress. It could reflect
increased secrecy among the leading AI developers. I suspect that some
of the recent publicity reflected a desire among both employers and
employees to show off their competence, in order to attract each
other.

I suspect that the top companies have established their reputations well
enough that they can cut back a bit on this publicity, and focus more on
generating real-world value, where secrecy has some clearer benefits
than is the case with Go programs. OpenAI's attitude about disclosing
GPT-2 weakly hints at
such a trend.

I've written
before
that the shift in AI research from academia to industry is worth some
attention. I expect that industry feedback mechanisms are faster and
reflect somewhat more wisdom than academic feedback mechanisms. So I'm
guessing that any future AI boom/bust cycles will be shorter than prior
cycles have been, and activity levels will remain a bit closer to the
longer-term trend.

VC funding often has boom/bust cycles. I saw a brief indication of a
bubble there (Rocket
AI)
three years ago. But as far as I can tell, VC AI funding kept
growing
for two years, then stabilized, with very little of the hype and
excitement that I'd expect from a bubble. I'm puzzled about where VC AI
funding is heading.

Then there's Microsoft's $1billion investment in
OpenAI. That's well outside of
patterns that I'm familiar with. Something is happening here that leaves
me confused.

AI conference attendance shows
patterns that look
fairly consistent with boom and bust cycles. I'm guessing that many
attendees will figure out that they don't have the aptitude to do more
th

...
Some Comments on "Goodhart Taxonomy"
by Grue_Slinky — 2019-12-24T23:59:33.361Z

Some time ago, I mentioned in a comment some issues I had with the Goodhart Taxonomy:

1) “adversarial” seems too broad to be that useful as a category

2) It doesn’t clarify what phenomenon is meant by “Goodhart”; in particular, “regressional” doesn’t feel like something the original law was talking about, and any natural definition of “Goodhart” that includes it seems really broad

3) Whereas “regressional” and “extremal” (and perhaps “causal”) are defined statistically, “adversarial” is defined in terms of agents, and this may have downsides (I’m less sure about this objection)

After thinking about things more, I'd like to expand on these points:

I've sort of made by peace with (3): whatever Goodhart is, it certainly is a multi-faceted phenomenon that seems to defy any single formalism. It is always unsatisfying to partition something into categories that are defined in different terminologies, since that makes it really hard to then show the categories are mutually exclusive and exhaustive. But, I've kind of updated towards this being the best we can realistically do.
However, I largely stand by (2), as explained in the next section.
Regarding (1), I suppose this is better characterized as "there seem to be several categories of 'Goodhart' not made salient by the Taxonomy (many of which are best formulated in terms of principal-agent dynamics, even if they needn't be subcategories of 'Adversarial Goodhart'). See the second section of this post.

"Regressional Goodhart" isn't Goodhart

I am not 100% sure I correctly understand what is meant by "Regressional Goodhart". But looking at the example from the post:

Example: height is correlated with basketball ability, and does actually directly help, but the best player is only 6'3", and a random 7' person in their 20s would probably not be as good

We see that, if optimizing for basketball ability (V) given only height (U), we won't get the best basketball player, simply because they are imperfectly correlated.

If "Regressional Goodhart" is simply the observation that, if we can't perfectly measure something, we won't perfectly optimize it, then this seems not only obvious to anyone (even...
ESC Process Notes: Claim Evaluation vs. Syntheses
by Elizabeth — 2019-12-24T17:00:02.010Z

Forgive me if some of this is repetitive, I can’t remember what I’ve written in which draft and what’s actually been published, much less tell what’s actually novel. Eventually there will be a polished master post describing my overall note taking method and leaving out most of how it was developed, but it also feels useful to discuss the journey.

When I started taking notes in Roam (a workflowy/wiki hybrid), I would:

Create a page for the book (called a Source page), with some information like author and subject (example)

Record every claim the book made on that Source page

Tag each claim so it got its own page

When I investigated a claim, gather evidence from various sources and list it on the claim page, grouped by source

This didn’t make sense though: why did some sources get their own page and some a bullet point on a claims page? Why did some claims get their own page and some not? What happened if a piece of evidence was useful in multiple claims?

Around this time I coincidentally had a call with Roam CEO Conor White-Sullivan to demo a bug I thought I had found. There was no bug, I had misremembered the intended behavior, but this meant that he saw my system and couldn’t hide his flinch. Aside from wrecking performance, there was no need to give each claim its own page: Roam has block references, so you can point to bullet points, not just pages.

When Conor said this, something clicked. I had already identified one of the problems with epistemic spot checks as being too binary, too focused on evaluating a particular claim or book than building knowledge. The original way of note taking was a continuation of that. What I should be doing was gathering multiple sources, taking notes on equal footing, and then combining them into an actual belief using references to the claims’ bullet points. I call that a Synthesis (example). Once I had an actual belief, I could assess the focal claim in context and give it a credence (a slider from 1-10), which could be used to inform my overall assessment of the book.

Sometimes there isn’t enough information to create a Synthesis, so something is left as a Question instead (example).

Once I’d proceduralized this a bit, it felt so natural and informative I assumed everyone else would find it similarly so. Finally you didn’t have to take my word for what was important- you could see all the evidence I’d gathered and then c...