trentmkelly/LessWrong-43k huggingface dataset:
Interviews on Improving the AI Safety Pipeline
by Chris_Leong — 2021-12-07T12:03:04.420Z

I conducted the following interviews to better understand how we might aim to improve the AI safety pipeline. It is very hard to summarise these interviews as everyone has different ideas of what needs to be done. However, I suppose there is some value in knowing this in and of itself.

A lot of people think that we ought to grow the field more, but this is exactly the kind of proposal that we should definitely not judge based on popularity. Given the number of people trying to break into the field, such a proposal was almost automatically going to be popular. I suppose a more useful takeaway is that a lot of people are currently seriously pursuing proposals to run projects in this space.

I guess another comment I can make is that the people who are most involved in the field seemed quite enthusiastic about the Cambridge AI Safety Fundamentals course as a way of growing the field.

If you think you’d have interesting thoughts to share, please feel free to PM me and we can set up a meeting.

Disclaimers: These interviews definitely aren’t a random sample. Some of the people were just people I wanted to talk to anyway. I was also reluctant to contact really senior people as I didn’t want to waste their time.

I tended towards being inclusive, so the mere fact that I talked to someone or included some of what they shared shouldn’t be taken as an endorsement of their views or of them being any kind of authority.

Evan Hubringer (MIRI):

Evan is now running a mentorship program with the assistance of SERI to mentor more people that he would be able to mentor himself:

Participants (14 people) were recruited from the AI Safety Fundamentals Course
This involves going through a set of shared readings for a few months, summarising and expanding upon a research artifact, then attempting an open research problem
SERI handled interviews + logistics
Evan had a 1/2 hour 1-on-1 weekly chat with each participant in the 1st stage & 1 hour weekly chat in the second stage where he cut the number of participants in half
Evan would like to see other researchers could copy this model to maximise their ability to mentor people:
Chris’ Note:

Perhaps Non-Linear could organise something similar under

multipliers for existing talent

- although I think Evan might try expanding this to other researchers himself

Suggested that there needs to be a new organisation to house people:

Why?:

None of the current p

...
Considerations on interaction between AI and expected value of the future 
by Beth Barnes — 2021-12-07T02:46:19.215Z

Some thoughts about the ‘default’ trajectory of civilisation and how AI will affect the likelihood of different outcomes.

Is the default non-extinction outcome utopic or dystopic?

Arguments for dystopia:

The world seems dystopic in many ways now
We’re getting better at various types of manipulative tactics, i.e. persuasion, marketing, creation of addiction and dependence, harnessing of tribalism. These things cause people’s actions to depart from what would best achieve their ‘true’ values. When this happens a lot, there is no reason the world should move in a direction that people think is broadly good. The things that happen in the world will no longer reflect human values; they will reflect the results of competition for resources and influence taking place between non-human actors (i.e. corporations, political parties), who are able to effectively control human actions.
Most creatures have existed in a dystopic Malthusian struggle, where the size of the population is kept stable by large numbers dying due to lack of resources. This is the state for all wild animals, and was the state for almost all of human history. We should probably consider this the default state. We should view the current state, where there are surplus resources for people to live much more comfortably than ‘barely not dying’ and the population is controlled by our reproductive choices rather than starvation or conflict, as an exception that will probably revert to normal, rather than permanent state.
Altruistic or broadly benevolent values are not the default, they’re pretty recent and rare. The fraction of human history where our moral circles have been wide enough to include humans of other races or nationalities is extremely small. Most current people’s moral circles only patchily include animals- it depends on the exact species and circumstance whether people feel any empathy. It seems like in many traditional societies, killing or harming a member of a rival group was considered not just neutral but good, something virtuous and something to be celebrated.
Technology is making the circumstances of our lives increasingly distant from the sort of settings humans are adapted for. We’ll find ourselves in settings where the environment is too unnatural to successfully trigger happiness, fulfilment, or empathy.
There are positional goods and values that involve people benefiting from the

...
Retail Investor Advantages
by leogao — 2021-12-07T02:08:20.694Z

There are some advantages to being a retail investor that might make it possible for you to beat the market without violating EMH.

Information Acquisition Cost

If you are an expert in some domain, it costs less for you to assess information in that domain than some trading firm. The trading firm has to spend money and suffer a time delay hiring an expert, incur the costs of communication overhead between people being difficult, also spend money on the opportunity cost of context switches and taking time away from traders at the company etc. Meanwhile, because you have both the trader and expert in your head, you save on a huge amount of overhead. Similarly, trading firms could pay people to camp outside Walmart and count customers, but if you already work at Walmart you can observe these things for relatively little marginal cost. Because of this, there are many small niches where the possible reward is too small to make it worth it for a trading company, but whose stakes are just the right size for retail investors.

Cheaper Liquidity

Nobody wants to be on the other side of the trade with RenTech, because RenTech makes smart trades. Everyone wants to be on the other side of the trade with retail investors, because most are uninformed and generally lose money over time. This is what makes it possible for Robinhood to offer commission free trades; uninformed flow is something that market makers really want to be on the other side of. The magnitude of trading fees often make or break a trading strategy.

Less Slippage

Moving less money moves the market less. As you scale up your Assets Under Management, your moves eat deeper into the order book and cause the market to move with you, eroding your profits. Large funds have entire teams of people dedicated to trade execution strategies to minimize their impact on the market. As a retail investor, you don't have this problem to the same magnitude; buying $100k of SPX will move the market a negligible amount. The magnitude of slippage also often makes or breaks a trading strategy; many naive backtests ignore slippage (because historical order book information is typically more difficult to obtain than OHLCV), and end up unprofitable in the real world because of it.

(Nothing I post is financial advice, I am not a financial advisor, etc)
Ordering yourself around with an app
by bfinn — 2021-12-07T00:49:11.546Z

It's hard to make yourself do tiresome tasks. You will if your boss tells you to, and asking someone else to nag you can also help. But, oddly, a mere app can also order you around and make you do things, when willpower and plain task lists fail. Because, I will argue, apps can seem like people.

There's a bunch of things I ought to do every day, such as planning, eating healthily, and exercise, but I often forget or procrastinate the more tedious ones. So for years I've tried to force myself to do them via checklists, using software like Microsoft Word, or even pen & paper.

But these attempts at discipline rarely lasted. Despite initial determination to complete the checklist daily, excuses and exceptions would soon pile up, then I'd forget even to look at it, and not long after would abandon the whole idea. Only to try again months later.

However, I had a breakthrough a couple of years ago, when I switched from checklists on paper or generic software, to a dedicated app. (Specifically Productive, which is simple yet functional, though there are many alternatives.)

Before: generic software or pen & paper

After: dedicated app

As soon as I started using the app, it felt like it was telling me to do the tasks, rather than me telling myself in Word or on paper. While there was no functional difference - the checklist was identical, and I didn't use alarms, notifications or other reminders - suddenly I found myself completing it. And I've used the app every single day since. While I don't always finish the list, I do at least consider each item, and check off those I've done.

Some people use an 'accountability partner', such as a friend, to nag them to get work done. It might seem the friend's role is redundant, merely telling them to do things they already know they should. But by involving someone else, you add an extra punishment for failure: the shame of failing in your friend's eyes; of letting them down, as well as yourself. Similarly, while many people pay personal trainers and music teachers to instruct them, many use them merely to create pressure to work out or practice.

And while I'll incur no shame from Productive if I fail, it seems I still feel some of the same social pressure, as if it were a person. It’s strange that a puny app can order us around, when we know it can’t punish us, and doesn’t even care.

Tim Urban wrote how procrastinatio...
Dear Self; We Need To Talk About Social Media
by Elizabeth — 2021-12-07T00:40:01.949Z

Last year I discovered, much to my chagrin, that always-on internet socializing was costly for me. This was inconvenient both because I’d spent rather a lot of time singing the praises of social media and instant messaging, and because we were in the middle of a global pandemic that had made online socializing an almost physical necessity. I made the decision at the time to put off changing my social media diet, and that was correct. But now there is in-person socializing again, and I’m changing how I use social media and messaging. I wanted to talk about this process and how great it was for me, but kept being nagged by the thought that the internet was full of essays about how the internet is bad, all of which I ignored or actively fought with, so what was going to make mine so special?

I decided to use the one thing I had that none of the other writers did: a detailed understanding of my past self. So I wrote a letter to past me, explaining how social media was costlier than she knew (even though she was right about all of the benefits), and how she could test that for herself to make a more informed decision. To help as many Elizabeths as possible, I tried to make the letter cover a wide range in time, although in practice it’s mostly focused on post-smart-phone life.

Dear Past Elizabeth,

I know you have read a lot of things calling social media bad. Your reasons for disagreeing with them are correct: social media has been an incredible gift to you, you have dodged many of the problems they’re describing, and you’re right to value it highly. You’re also right that many of the people bragging about how hard they are to communicate with are anti-socially shifting the burden of communication to other people.

But.

Social media (and always-on instant messaging, which is a different, mostly worse, problem) has some costs you’re not currently tracking. I would like to help you understand those costs, so you can make different choices on the margin that leave you happier while preserving the benefits you get from social media, not all of which you’ve even experienced yet (is it 2015 yet? Approximately every job you get from this point on will have your blog as a partial cause. After 2017 you won’t even have interviews, people will just say “I read your blog”).

To be more specific: you have indeed curated your feed such that Facebook is not making you angry on...
Leaving Orbit
by Rob Bensinger — 2021-12-06T21:48:41.371Z

Duncan says:

At CFAR workshops, we often installed this as an explicit local norm: "If at any point you want to convey 'hey, I'm bowing out of this conversation now, but it's not intended to be any sort of reflection on you or the topic, I'm not making a statement, I'm just doing what's good for me and that's all,' you just say 'Leaving orbit!' and people will wave to you and keep on talking."

This strikes me as good social tech to try out on LessWrong (linking to this page to define the term).

Basic rationale:

It's important to be able to randomly exit conversations. Otherwise, people won't add as much useful stuff to conversations in the first place (lest they be trapped).
Sometimes people want to leave a message to indicate they're exiting. But it can be tricky to phrase such messages without implying one or more of:

"I think my last message is the last word in this conversation, and has resolved all outstanding issues."
"I think you have raised no important or worth-responding-to points since my last message."
"I think that you as a person are unworthy of my time."
"Your last messages convinced me; I agree with you."
"Your last messages didn't convince me; I disagree."
"I have no idea how to object-level-respond to your messages."

(Maybe some of these are things you want to communicate in some contexts, but it's good to not have to communicate them!)
So having a very generic placeholder thing to say here is useful. "(Leaving orbit.)" seems friendly and generic enough to suit the bill.

Caveat: I think it's also 100% fine to exit conversations without leaving a message at all.

I generally think society has far too many "do this or you're bad" norms, especially automatic norms. A bad outcome would be if "leaving orbit" (or similar) came to be seen as an expectation or requirement, rather than as an option. If this idea catches on too much, I suggest deliberately exiting more conversations without leaving any message, so as to keep this option from ossifying into an expectation. 🙂
Declustering, reclustering, and filling in thingspace
by Stuart_Armstrong — 2021-12-06T20:53:14.559Z

Eliezer has talked about thingspace - an abstract space in which objects lie, defined by their properties on some scales. In thingspace, there are some more-or-less natural categories, such as the "bird" category, that correspond to clusters in this thingspace. Drawing the boundary/carving reality at the joints, can be thought of as using definitions that efficiently separate these clusters[1].

He said about the thingspace of birds:

The central clusters of robins and sparrows glowing brightly with highly typical birdness; satellite clusters of ostriches and penguins [further from the center] glowing more dimly with atypical birdness, and Abraham Lincoln a few megaparsecs away and glowing not at all.

That, however, is how things stand in the world as it is. What if, for some reason, we had a lot of power in the world, and we wanted to break these neat-ish clusters, how would we go about it?

Remaking the clusters

Suppose first we wanted to break away some of those satellites away. What if we wanted penguins to be clearly distinct from birds? Penguins are land-waddlers and good swimmers, living in cold climates. So let's fill out this category, call it, maybe, Panguens.

Panguens all live in cold climates, waddle on land, and are good swimmers. Most are mammals, and give birth to live young. Some are more like the duck-billed platypus, and lay eggs (though they suckle their young). And all of them, including penguins, can reproduce with each other (how? genetic engineering magic). Thus there are a lot of intermediate species, including some that are almost-penguins except differing on a few details of their anatomy. All have fins of some sort, but only 10% have penguin-like wing-fins.

Panguens are clearly a cluster, quite a tight one as animals go (given they can reproduce with each other). And they are clearly not birds. And penguins clearly belong inside them, rather than with birds; at most, some pedantic classifier would label penguins as "bird-like panguens".

So, by filling in extra possibilities in thingspace, we have moved penguins to a new category. Graphically, it looks like starting with:

and moving to:

What if we wanted to make Abraham Lincoln into a bird[2]? We can connect the categories of birds and humans by creating all sort of bird-human hybrids. But this is not quite enough. Bird-humans is certainly a reasonable category in this world. But humans ar...
More Christiano, Cotra, and Yudkowsky on AI progress
by Eliezer Yudkowsky — 2021-12-06T20:33:12.164Z

This post is a transcript of a discussion between Paul Christiano, Ajeya Cotra, and Eliezer Yudkowsky (with some comments from Rob Bensinger, Richard Ngo, and Carl Shulman), continuing from 1, 2, and 3.

Color key:

Chat by Paul and Eliezer	Other chat

10.2. Prototypes, historical perspectives, and betting

[Bensinger][4:25]

I feel confused about the role "innovations are almost always low-impact" plays in slow-takeoff-ish views.

Suppose I think that there's some reachable algorithm that's different from current approaches, and can do par-human scientific reasoning without requiring tons of compute.

The existence or nonexistence of such an algorithm is just a fact about the physical world. If I imagine one universe where such an algorithm exists, and another where it doesn't, I don't see why I should expect that one of those worlds has more discontinuous change in GWP, ship sizes, bridge lengths, explosive yields, etc. (outside of any discontinuities caused by the advent of humans and the advent of AGI)? What do these CS facts have to do with the other facts?

But AI Impacts seems to think there's an important connection, and a large number of facts of the form 'steamships aren't like nukes' seem to undergird a lot of Paul's confidence that the scenario I described --

("there's some reachable algorithm that's different from current approaches, and can do par-human scientific reasoning without requiring tons of compute.")

-- is crazy talk. (Unless I'm misunderstanding. As seems actually pretty likely to me!)

(E.g., Paul says "To me your model just seems crazy, and you are saying it predicts crazy stuff at the end but no crazy stuff beforehand", and one of the threads of the timelines conversation has been Paul asking stuff like "do you want to give any example other than nuclear weapons of technologies with the kind of discontinuous impact you are describing?".)

Possibilities that came to mind for me:

1. The argument is 'reality keeps surprising us with how continuous everything else is, so we seem to have a cognitive bias favoring discontinuity, so we should have a skeptical prior about our ability to think our way to 'X is discontinuous' since our brains are apparently too broken to do that well?

(But to get from 1 to 'discontinuity models are batshit' we surely need something more probability-mass-concentrating than just a bias

...
Are there alternative to solving value transfer and extrapolation?
by Stuart_Armstrong — 2021-12-06T18:53:52.659Z

Thanks to Rebecca Gorman for the discussions that inspired these ideas.

We recently argued that full understanding of value extrapolation[1] was necessary and almost sufficient for solving the AI alignment problem, due to the existence of morally underdefined situations.

People seemed to generally agree that some knowledge of human morality was needed for a safe powerful AI. But it wasn't clear this AI needed almost-full knowledge of human morality, as well as value extrapolation. We got some high-quality alternative suggestions. In this post, we'll try and distil those ideas and analyse them.

AI seems to behave reasonably

Steven Byrnes's position, if we understand it correctly, is that the AI should learn to behave in non-dangerous seeming ways[2].

Our phrasing of the idea would be (apologies for any misinterpretations):

There are many behaviours that agents can follow. Typical humans follow typical behaviours, to achieve typical consequences. And other typical humans can assess both behaviours and consequences, and grade them as (typical) good, (typical) bad, or weird.

An AI should behave in ways that typical humans judge as good, achieving consequences that typical humans judge as good. If there is too high a risk of weirdness, the AI will output NOOP (no operation).

This seems a sensible approach. But it has a crucial flaw. The AI must behave in typical ways to achieve typical consequences - but how it achieves these consequences doesn't have to be typical.

For example, it might make an explanatory video to convince someone to follow some (reasonable) course of action. The video itself is not unusual, but the colour scheme happens to hypnotise that specific viewer into following the suggestion. At a gross level of description, everything is fine: convincing-but-typical video convinces. But this only works because the AI has inhuman levels of knowledge and predictive ability, and carefully selects the "typical behaviour" that is the most effective.

Now, we might be able to make sure the AI doesn't have secret superhuman abilities (we've had an old underdeveloped idea that we could force the AI to use human models of the world to achieve its goals), but this is a very different problem, and a much harder one.

Then, given that the AI has access to superhuman levels of ability, the "typical consequences" is no longer a safe goal. It reduces to "consequences that...