trentmkelly/LessWrong-43k huggingface dataset:
Parable of the Dammed
by johnswentworth — 2020-12-10T00:08:44.493Z

Once upon a time, two families fought a bloody feud over the border of their properties. After many years of escalation, neither family could afford to continue the fight. They both wanted to negotiate a truce, but the original issue still had to be settled: where was the border of their lands to be drawn? They needed a Schelling point, so they settled on a river which ran roughly through the middle of their territories.

For a short time, there was peace.

Soon, though, a clever couple from one of the families hatched an idea. Each morning, they walked down to the river and dropped a few large stones into it. Before long, half a dam had built up on their side of the river. The water was driven toward the opposite bank, which was steadily washed away. Over time, as the river ate away the opposite bank, the couple extended their dam further, and so the river’s course was gradually pushed sideways. The couple’s family gained territory, while the opposed family lost it.

At this point, the story diverges, and many versions of the tail are told.

In one version, the couple push too fast. Soon the river has moved deep into the territory of the other family, and the family responds by attempting to break the dam. Violence escalates, and the feud breaks out anew - but peace is even harder to come by, now, since the river has been permanently destroyed as a Schelling point.

In another version, the push is slow. The couple bequeaths the task of dam-building to their children, and to their childrens’ children, and the river shifts slowly over the course of generations. With each generation, the resources of the couple’s descendants grow, and their family grows with it - while the resources of the opposed family slowly dwindle. Nobody ever takes much note of the river’s slow drift, until eventually the opposed family dies out altogether.

In most versions of the tale, the river’s movement is quickly noticed, but a return to violence is deemed unacceptable. Instead, the opposed family begins dropping rocks of their own. Soon both families are dumping rocks on their respective sides of the river, building up dams, aiming to drive the water against the opposite bank. This bloodless but expensive feud escalates. Along some sections of the river, each side expands until the two dams clash in the middle, blocking the flow of the whole river, and water backs up and bursts the banks. Tha...
Final Day to Order LW Books by Christmas for US
by Ben Pace — 2020-12-09T23:30:36.877Z

Today is the last day we are guaranteeing delivery-before-Christmas to those who preorder the LessWrong books to North America (i.e. US, Canada, Mexico).

Order them at lesswrong.com/books by 12pm midnight Pacific time to still be included.

Thank you so much to everyone who has purchased the 1,600 copies already sold. We've increased our print run by an order of magnitude to account for the demand curve.

I'm looking forward to hearing about people opening them for Christmas :)
Summary of AI Research Considerations for Human Existential Safety (ARCHES)
by peterbarnett — 2020-12-09T23:28:43.635Z

I read Andrew Critch and David Krueger's AI Research Considerations for Human Existential Safety (ARCHES) recently, which is a report that outlines and discusses 29 research directions which might be helpful for AI existential safety. This has also been discussed in Alignment Newsletter #103 and in an interview with Andrew Critch on the Future of Life Podcast.

This report focuses on "prepotent" AI, which is defined as follows: An AI system is prepotent if its development would transform the state of humanities habitat - currently the Earth - in a manner that is at least as impactful as humanity and is unstoppable to humanity. Such an AI system would be as transformative to Earth as the entire effect of humanity so far, including the agricultural and industrial revolutions, and would be unstoppable in the sense that no group of humans could stop or reverse its impact even if they wanted to. Prepotent AI is Transformative AI (a term used by Open Philanthropy), which is unstoppable to humans.

As well as being prepotent, an AI technology would need to be misaligned for it to be bad, and hence this report talks a lot about misaligned prepotent AI (MPAI). When it comes to the case where there are multiple stakeholders it is hard to define what alignment means, and so this report uses human extinction as a clear (but maybe overly restrictive) boundary for alignment; A prepotent AI system is misaligned if it is unsurvivable to humanity. This definition obviously misses some very bad outcomes, but is justified under what is called the Human Fragility Argument which says that most potential future states of the Earth are unsurvivable to humanity (for example if all the oxygen in the atmosphere was removed), and hence unless we have properly aligned a prepotent AI system then its deployment and subsequent transformative actions will likely result in human extinction.

The report is open about the things which are omitted, which include; bad outcomes which don't result in human extinction; the definition of a 'human'; the definition of 'benefical'; the impact of AI making other existential risks more likely (nuclear weapons, bioweapons etc).

This report outlines 29 research directions with the goal of reducing the chance that MPAI will be deployed. This report frames the human/AI relationship as one where humans delegate an AI to do a task, which avoids having to talk about ...
How do we know that the things we want have credibility ? 
by Madeven1335 — 2020-12-09T21:08:26.881Z

I think Transhumanism and singularity are a beautiful vision of a society but I believe it requires a level of enlightenment we humans are yet to be shown capable of. But hey in the end it's a dream worth chasing isn't it ?

Well I've been having doubts about it ...

First of all

1 my biggest doubt is the possibility.

How do we know that even if those things are physically possible, they aren't impractical to achieve and that we won't find out that there are limitations on what we can do as our knowledge improves ? Is betting on something like this safe ?

2 lack of ethics discussions. Many Transhumanists and singularitarians I've come across really come off as either extremely privileged and completely oblivious to the social effects of such technologies. Besides the lack of minority participation in itself really shows a lot. I don't think it's useful to say "minorities don't participate in this because they don't know about Transhumanism" there might be other reasons as well.

3 the outright denialistic behaviour and the amount of people in fragile mental states joining Transhumanism. Constantly looking for validation of one's worldview , avoiding anything that challenges one's views really anxiously. Besides many people who are overly invested into Transhumanism seem to be in fragile mental conditions and seem to really be looking for a hope or in denial of their condition. Is it actually ethical ? To get people involved in a non safe bet ?

I've seen many actual academics say that people should appreciate Transhumanists should stop the immortality or bust mentality and appreciate the practical stuff for current period , all repeated this same thing

4 I admit I've been and still am in all 3 categories but I would very much like to be proven wrong and would still like to believe that Transhumanism is still worth it and not just a false hope for people in Denial or fragile emotional states ...
Switching to Google Calendar
by jefftk — 2020-12-09T20:10:04.972Z

When I first started keeping a calendar, in 2006, it was

just an HTML
table on my website:

Over time, I extended it to generate an iCal feed and wrote several
commands to make it easier to add entries to the table (proc_schedule).
I knew it was a peculiar way of doing things, but it met my needs and
I didn't feel much reason to switch.

With covid, however, I have a lot more short social events. An hour
to play One Night Ultimate Werewolf, half an hour to talk with someone
about a question they have, an hour and a half of no-hands pair
programming on the bucket brigade
singing project. Now I care about invites, RSVPs, time zones,
video calls, and notifications. Yes, perhaps I could have extended my
program to have all of these features, but why?

I'm already familiar with Google calendar because we use it at work,
and have now switched over to it for personal things as well. If you
want to invite me to something, use my jeff.t.kaufman@gmail.com
address.

(The spirit of generating a calendar feed from an HTML table lives on,
however, in proc_fr_schedule
and proc_kf_schedule
for the Free Raisins and Kingfisher. Hopefully, post covid, there will
be gigs again.)
[AN #128]: Prioritizing research on AI existential safety based on its application to governance demands
by Rohin Shah — 2020-12-09T18:20:07.910Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

Some AI research areas and their relevance to existential safety (Andrew Critch) (summarized by Rohin): This long post explains the author’s beliefs about a variety of research topics relevant to AI existential safety. First, let’s look at some definitions.

While AI safety alone just means getting AI systems to avoid risks (including e.g. the risk of a self-driving car crashing), AI existential safety means preventing AI systems from posing risks at least as bad as human extinction. AI alignment on the other hand is about getting an AI system to try to / succeed at doing what a person or institution wants them to do. (The “try” version is intent alignment, while the “succeed” version is impact alignment.)

Note that AI alignment is not the same thing as AI existential safety. In addition, the author makes the stronger claim that it is insufficient to guarantee AI existential safety, because AI alignment tends to focus on situations involving a single human and a single AI system, whereas AI existential safety requires navigating systems involving multiple humans and multiple AI systems. Just as AI alignment researchers worry that work on AI capabilities for useful systems doesn’t engage enough with the difficulty of alignment, the author worries that work on alignment doesn’t engage enough with the difficulty of multiagent systems.

The author also defines AI ethics as the principles that AI developers and systems should follow, and AI governance as identifying and enforcing norms for AI developers and systems to follow. While ethics research may focus on resolving disagreements, governance will be more focused on finding agreeable principles and putting them into practice.

Let’s now turn to how to achieve AI existential safety. The main mechanism the author sees is to anticipate, legitimize, and fulfill governance demands for AI technology. Roughly, governance demands ar...
Notes on Know-how
by David Gross — 2020-12-09T17:21:18.936Z

This post examines the virtue of know-how. I wrote this not as an expert on the topic, but as someone who wants to learn more about it. I hope it will be helpful to people who want to know more about this virtue and how to nurture it.

What is know-how?

“A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects.” ―Robert A. Heinlein[1]

You have know-how to the extent that you have skills obtained from practice doing things in the real world.

Know-how is distinct from mere theoretical understanding. For example, the theoretical discipline of hydrodynamics may be fascinating and useful, but if you want to know how to swim you have to get wet.

Know-how as a virtue seems to mean having competence in a broad, well-chosen range of skills.

Related virtues

Know-how is related to “practical knowledge,” though practical knowledge can be knowledge about things as well as how to do things. There’s some overlap. If you’re building a cabinet, practical knowledge will help you know which type of wood is best for the job, while know-how will help you construct it skillfully.

Know-how is kind of like “craftsmanship,” but craftsmanship implies a more advanced skill in a particular specialty — connoisseurship, expertise, commitment to quality, attention to detail, efficiency — while know-how is more of a baseline proficiency in many areas.

Know-how has some connection with prudence (in the sense of “practical wisdom”), though prudence has more to do with having a well-honed process for making good decisions generally, while know-how is about experience-tested decisions in specific contexts.

How to know what you need to know

An American in 1900 who knew how to drive a car with a manual transmission was an early adopter. An American who knows how to drive such a car in 2050 may be something of an eccentric traditionalist. Somewhere in-between, that skill was considered an essential part of know-how. What skills are important change over time.

They also differ from culture to culture, and from class to class. Read one of the many classic novels about the live...
Walk to a San Francisco FedEx
by KatjaGrace — 2020-12-09T08:50:09.832Z

Today I had three work projects so close to done that I might be able to put something up on each, which would usually encourage work enthusiasm. But when I started on the first, I was struck by a strong inclination to stop and do something else. I didn’t immediately, but the inclination remained. And such inclinations make work worse in themselves, because when each new sentence or next motion engenders a little flinch away from it, the whole thing gets so slow and encumbered and hard to concentrate on that it makes sense to be repelled by it. And the thought of moving to the next little bit on the other projects seemed similarly distasteful.

Sitting in my chair was likewise aversive, and I thought perhaps a walk would help. (Often I find similar work compelling, so it seems a shame to wrestle myself into a slow and encumbered version of it at a time when I don’t.) I also had a package to take to FedEx. If I listened to the Alignment Newsletter Podcast while I walked and delivered the package, I might even be being extra productive instead of extra unproductive.

I took my P100 mask and a packet of snickerdoodle flavored peanut butter and some moderately comfy shoes, in case the three and a half blocks to the pharmacy that FedEx was inside turned out to be too much for flip flops, or I decided to wander around more.

I reached the pharmacy and stood in its small outside queue with my package between my legs and put on my P100. A man at the door let me in and pointed me to a longer inside queue. I waited in it and thought about how hard it was to see the total population of the room, and how much time had passed without the queue moving at all, and how much the San Francisco covid cases were spiking, then decided to try for a different FedEx down the road.

I left my P100 on and walked along faceless, listening to the podcast. Which sounded interesting, but I kept getting distracted. The warm bright sun on every surface sent away my vague image of the locked-down city as wan and enfeebled. Though I wondered if was climatically alarming for San Francisco winter to be such a nice summer.

I enjoyed my mask, for reasons that are not entirely clear. Perhaps having one’s face in the public sphere does call for some ongoing attention and composure and care, that I don’t notice until its absence is a relief. Or perhaps it is nice for ones mouth and nose to be seriously rem...
Luna Lovegood and the Chamber of Secrets - Part 6
by lsusr — 2020-12-09T08:14:29.404Z

"It's in the Mirror of Atlantis," Harry said.

"We need Gillyweed," said Luna, "Lots of Gillyweed."

"The Mirror of Atlantis is located at the end of the third-floor corridor on the right-hand side," said Harry.

"What are you waiting for?" said Luna.

"You're not going to ask me how I know these things?" said Harry.

"Why would I do that?" said Luna.

"Because…because…" Harry searched for words to express the magnitude of the epistemic hole.

"You talk like a Muggle," said Luna, "You think like one too."

Harry puzzled over whether he had just been insulted. Luna skipped ahead to the third floor.

Harry and Luna flew a double-seated broomstick over the dieffenbachia and other unmaintained obstacles to the Mirror.

"You're a really good flier," said Luna, "I bet you're on the Quidditch team. What position do you play? No. Wait. Don't tell me. I bet you play Seeker."

"This Mirror lets you store an object until someone with the right intentions arrives," said Harry testily.

"I seek entry to the Ravenclaw Common Room," said Luna to the Mirror, "I want to sleep in a bed."

"In Gödel's name, what monstrously difficult riddle did the bronze eagle ask of you?" asked Harry.

"'Where is my diadem?'" said Luna.

"You're supposed to say 'lost' or 'hidden'," said Harry, "You're not expected to rediscover the Lost Diadem of Ravenclaw."

"Oh," said Luna.

"Since we're already here, let's give it a modicum of effort," Harry withdrew a mechanical stopwatch from his pouch and set it to five minutes.

"I want the diadem to save the world," Harry said.

"I aim to save the lives of all sentient beings," Harry said.

"I promise to bring you, Rowena Ravenclaw, back to life," Harry said.

"I seek to disassemble stars," Harry said.

Luna ignored him. Rowena Ravenclaw was not a jealous witch. If she concealed knowledge then it was not because the knowledge could be used for evil. It was because the knowledge itself was intrinsically dangerous. A cascade of logic began to self-assemble.

Rowena's basilisk attacks those who know it exists.

Luna forced her thoughts back into disarray.

"BRIIIIIING!" the mechanical alarm went off.

"I'm hunting nargles," Luna said.

The Luna in the Mirror held the Diadem of Ravenclaw in her left hand. The real Luna held it in her right.

"What is a nargle?" said Harry.

"You don't want to know," said Luna.

"What do you plan to do with this magical artifact of incredible power?" asked Harry.

"I'm going...
Quick Thoughts on Immoral Mazes
by abramdemski — 2020-12-09T01:21:40.210Z

I finally got around to reading (most of) Zvi's sequence Immoral Mazes. I may want to turn some of the following ideas into longer posts, but I thought I'd try to get out some quick thoughts.

Connections to Dictator's Handbook

I've also recently finished reading Dictator's Handbook. The two share some similarities, but offer very different analyses of the problem and very different directions for hope/improvement.

Dictator's handbook in brief (see also the excellent summary by CPG grey on youtube):

You can think of government as a game played between rulers and ruled. Rulers who don't play to maximize the amount of time they stay in power don't rule for long, so we can model rulers' utility function as just maximizing time in power. This fits most cases pretty well, and explains a lot of political phenomena.
We can ignore most of the details of a given political system, and just put systems on a spectrum from "perfect dictatorship" to "perfect democracy" by how many key supporters a ruler needs to please to stay in power. In a realistic dictatorship, the answer is a handful of people. In a realistic democracy, the answer is a significant fraction of the population.
The ruler's incentive is always to minimize the number of key supporters needed, and then make those supporters satisfied.
The key supporters sometimes have incentives aligned with the ruler, IE, also want to minimize the number of key supporters (to get more rewards per key supporter). This is especially true when there are few key supporters already. This will lead to culls and movement further and further toward dictatorship.
At other times, the key supporters may want to expand the number of key supporters. This is because (1) they want to protect themselves against a cull, and (2) they may want to incentivise the ruler to produce public goods (rather than special benefits to supporters). This is especially true when the number of key supporters is already very large, so special benefits to supporters are small, and won't be hurt much by expanding the coalition (and are outweighed by benefits from public goods). Expanding the coalition further provides insulation from culls. This leads to further democratization.

Immoral Mazes seem sort of like the Dictator's Handbook model but with more levels. The Dictator's Handbook did mention (I think?) that the key supporters are usually playing the same ga...
Chance that "AI safety basically [doesn't need] to be solved, we’ll just solve it by default unless we’re completely completely careless"
by Quinn — 2020-12-08T21:08:47.575Z

As part of the 2020 Effective Altruism Student Summit, AI safety researcher Buck Shlegeris advised newcomers to "think through the entire problem," determining what the concrete dangers of unsafe AI actually are and how to address them. In an earnest effort to do just that, Nicholas L. Turner, Aidan Kierans, Morpheus, Quinn Dougherty, Jeffrey Yun, and others started a small AI Safety Forecasting Workshop to practice defining, estimating, and forecasting different outcomes. We've perceived a small degree of success in making questions more concrete, and so we decided to share our thoughts to help other aspirants. The following is adapted from our notes for this week's meeting, and we've added context or elaboration when necessary.

Specifically, Nick, Aidan, Morpheus, and Quinn pulled up Elicit's x-risk tag and went on a deep dive of one of the questions. Source.

Chance that "AI safety basically [doesn't need] to be solved, we’ll just solve it by default unless we’re completely completely careless"

As one might imagine, this is a very useful question to consider for a group of newbies. The engagement on Elicit with this question was modest (with two participants, the average estimate of the chance currently sits at 17.5%), but it captured our intuitions well enough to start getting our hands dirty.

Terms

We started by defining some terms, as resolving at least some of the ambiguities of this prompt allows us to better understand the relationship between the prompt, our intuitions about it, and states of the world.

"solving AI safety"

"default"

"careless"

"we"

Solving AI safety

First, what is the goal that we're trying to reach by "solving" AI safety? At which point would we consider AI safety solved or not? Moreover, how should we define AI safety?

Option 1. An AI is safe when it is transformative without being catastrophic.

When thinking about future applications of AI, people often imagine a point in time when the technology fundamentally alters society. Some have used the term Transformative AI (TAI) to describe this state of the technology. We borrow that definion here, though this opens up a bit more ambiguity. What separates TAI slightly less "transformative" applications?

One helpful approach that we've learned from our discussions is to make these definitions more precise by comparison. There are several other points in time where human society has b...