trentmkelly/LessWrong-43k huggingface dataset:
The consequentialist case for social conservatism, or “Against Cultural Superstimuli”
by Sophronius — 2021-04-14T21:50:25.794Z

This essay was requested by the highly qualified rationalist and extremely-sex-positive Paul Crowley, who (like me) is frustrated by the absolute refusal of certain political groups to explain their actual ideas rather than shout at each other. The shouty people in this case are sex-negative conservatives and second-wave feminists, and the thing they’re shouting about is that our society has become too hedonistic. Do they have a point?

Well, the strongest argument in favor of social conservatism is common sense – in this case, the idea that society is the way it is for a reason, and that any large scale change is therefore liable to have severe negative societal consequences. Society might feel like a construct, but it’s actually evolved in much the same way as us. As such, trying to ‘fix’ society is like modifying human RNA to vaccinate against a disease – a completely insane notion that obviously demands extreme caution but something which can apparently be done if you put your best and smartest people on the job. The problem is that when it comes to modifying society, not only are the smartest people not in charge, nobody is in charge, there is no quality testing whatsoever and nobody even seems aware of how absolutely insane that is.

Liberal commentators dismiss this concern in the name of utilitarian consequentialism: the idea that even if a proposed change seems scary, you should just shut up and do the math and then implement it anyway if the numbers work out. And from the perspective of progressives, the math is firmly on their side. Conservatives warned that society would collapse if interracial marriage was legalized, and yet here we are. They said the same thing about gay marriage, and women’s rights, and literally every other time there was a proposal to make society even slightly more open and tolerant. And now they are singing the same tune about Trans people (No, unisex bathrooms and women’s sports are not their real primary objections – they’ve just gotten into the habit of censoring their own best arguments). Clearly, conservatives are just a lodestone whose existence only serves to slow down progress, and the best solution is to either silence them or else to simply ignore them until they become irrelevant.

But from the perspective of social conservatives, the exact opposite is the case. Sure, society didn’t collapse immediately when the ...
[AN #146]: Plausible stories of how we might fail to avert an existential catastrophe
by Rohin Shah — 2021-04-14T17:30:03.535Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world.

HIGHLIGHTS

What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs) (Andrew Critch et al) (summarized by Rohin): A robust agent-agnostic process (RAAP) is a process that robustly leads to an outcome, without being very sensitive to the details of exactly which agents participate in the process, or how they work. This is illustrated through a “Production Web” failure story, which roughly goes as follows:

A breakthrough in AI technology leads to a wave of automation of $JOBTYPE (e.g management) jobs. Any companies that don’t adopt this automation are outcompeted, and so soon most of these jobs are completely automated. This leads to significant gains at these companies and higher growth rates. These semi-automated companies trade amongst each other frequently, and a new generation of "precision manufacturing'' companies arise that can build almost anything using robots given the right raw materials. A few companies develop new software that can automate $OTHERJOB (e.g. engineering) jobs. Within a few years, nearly all human workers have been replaced.

These companies are now roughly maximizing production within their various industry sectors. Lots of goods are produced and sold to humans at incredibly cheap prices. However, we can’t understand how exactly this is happening. Even Board members of the fully mechanized companies can’t tell whether the companies are serving or merely appeasing humanity; government regulators have no chance.

We do realize that the companies are maximizing objectives that are incompatible with preserving our long-term well-being and existence, but we can’t do anything about it because the companies are both well-defended and essential for our basic needs. Eventually, resources critical to human survival but non-critical to machines (e.g., arable land, drinking water, atmospheric oxygen…) gradually become depleted or destroyed, until humans can no longer survive.

Notice that in this story it didn’t really matter what job type got automated first (nor did it matter which specific companies took advantage of the automation). This is the defining feature of a RAAP -- the same general story arises even if you change around the agents that are p...
What weird beliefs do you have? 
by Dumbledore's Army — 2021-04-14T10:29:16.322Z

We know that mainstream thinking gets a lot of things wrong. Many of us have experienced being mocked because of our concern for AI extinction-risk. There are plenty of other examples of times where now well-evidenced beliefs were seen as crazy in some way. This post was prompted by my reading around meditation and mindfulness - twenty years ago if you said that meditation had a number of mental and even physical health benefits and was worth practicing for non-religious reasons, then you would be laughed at as a New Age type who probably believed in crystal healing and astrology too. Now there's stacks of scientific evidence supporting that view.

I would like to keep an open mind and not dismiss fact-claims just because they pattern-match to weird people or because they don't pass the absurdity heuristic. On the other hand, there are a lot of crazy people out there and I don't really want to wade through dumb stuff by flat-earth types. So I figured posting this question here is a good way to find some interesting ideas. Fellow Rationalists, what beliefs do you have that would cause the average member of society to laugh at you or call you weird?

I have at least one such belief, but I'll post it as an answer to this question, because I want the focus to be on the question and not on my specific belief.

Edited to add: please include a summary of why you believe what you do - what evidence or chain of reasoning led you to this belief?
An Exercise in Rational Cooperation and Communication: Let's Play Hanabi
by ejacob — 2021-04-14T08:36:07.846Z

Why Play Hanabi?

Hanabi is a game requiring modeling others' minds, communication, and strategy. Its unique challenges and cooperative (rather than adversarial) objective have piqued the interest of the AI community, which recently began using it as a testing environment for AI agents.

Hanabi is a card game in which 2-5 players must cooperate to put on a dazzling firework display. The cards in the game represent different stages of fireworks. The most basic version of the game has five different colors of fireworks (cards), each of which has five stages, numbered 1-5, that must be set up in ascending order. The players must play cards from their hands to add them to the communal display on the table. The players' score at the end of the game is the sum of the latest stage of each color they successfully deployed; reaching stage 5 for all 5 colors receives the maximum score of 25.

However, a slight wrinkle: players must hold their cards so that they face away from them; no player is allowed to look at their own cards, only those of other players. To play their cards at the right time, each player must rely on clues from their teammates.

I like this game a lot. I also think it is good for people to play in that it forces you to think about how others will interpret your communications when they don't have the same information or perspective that you do. The communication skills and theory of mind required for Hanabi are also good for real life - in handling interpersonal conflicts with people you ultimately want to cooperate with, or in explaining your expertise to a layperson, as some examples. (I believe this strongly enough that I will volunteer myself as a partner for any reader interested in trying the game online; just leave a comment or send a message.)

Hanabi Rules

Instead of or in addition to reading this section, you can watch a humorous explainer video.

To start, shuffle all the cards together and deal 5 cards to each player in a 2 or 3 player game or 4 cards to each player in a 4 or 5 player game. Place the rest of the cards in the middle of the table and set up the 8 clock tokens and three fuse tokens face-up.

Players take turns until the game is over. On their turn, a player must take one of three actions:

Give a hint to another player
Play a card from their hand
Discard a card from their hand

Give a Hint: To give a hint, a player picks one other playe...
Against "Context-Free Integrity"
by Ben Pace — 2021-04-14T08:20:44.368Z

Sometimes when I talk to people about how to be a strong rationalist, I get the impression they are making a specific error.

The error looks like this: they think that good thinking is good thinking irrespective of environment. If they just learn to avoid rationalization and setting the bottom-line first, then they will have true beliefs about their environment, and if there's something that's true and well-evidenced, they will come to believe it in time.

Let me give an extreme example.

Consider what a thoughtful person today thinks of a place like the Soviet Union under Stalin. This was a nation with evil running through their streets. People were vanished in the night, whole communities starved to death, the information sources were controlled by the powerful, and many other horrendous things happened every day.

Consider what a strong rationalist would have been like in such a place, if they were to succeed at keeping sane.

(In reality a strong rationalist would have found their ways out of such places, but let us assume they lived there and couldn't escape.)

I think such a person would be deeply paranoid (at least Mad-Eye Moody level), understanding that the majority of their world was playing power games and trying to control them. They'd spend perhaps the majority of their cognition understanding the traps around them (e.g. what games they were being asked to play by their bosses, what sorts of comments their friends would report them for, etc) and trying to build some space with enough slack to occasionally think straight about the narratives they had to live out every day. It's kind of like living in The Truman Show, where everyone is living a narrative, and punishing you / disbelieving you when you deviate. (Except with much worse consequences than what happened in that show.)

Perhaps this is too obvious to need elaborating on, but the cognition of a rationalist today who aims to come to true beliefs about the Soviet Union, and the cognition of a rationalist in the Soviet Union who aims to come to true beliefs about the Soviet Union, are not the same. They're massively different. The latter of them is operating in an environment where basically every force of power around you is trying to distort your beliefs on that particular topic – your friends, your coworkers, the news, the police, the government, the rest of the world.

(I mean, certainly th...
Auctioning Off the Top Slot in Your Reading List
by mike_hawke — 2021-04-14T07:11:07.881Z

Or, A Nicer Way to Commodify Attention

Observation 1: I read/listen to a lot of public intellectuals--podcasters, authors, bloggers, and so on--and I frequently find myself thinking:

Ugh, I hate it when this guy spouts ignorant nonsense about [some domain]. And he's been doing it for years. I wish he would just read [relevant book]. Then he’d at least be aware of the strongest counterarguments.

Observation 2: Sometimes podcasters/bloggers will poll their followers with questions like, “Hey guys, who should I interview next?” or “what book should I review next?”

Having observed these things I wonder if they could be improved by monetary transactions.

For a specific example, it would be cool for David Deutsch to let the highest bidder choose a book for him to read and review. If we're lucky (or spendthrift), we get to see him finally give a considered response to the particular claims in Human Compatible or Superintelligence or Life 3.0.

Less specifically, there are a lot of cognoscenti who command substantial influence while holding themselves to disappointingly low epistemic standards. For example, Sean Carroll is a science communicator who dabbles in a little bit of everything on the side; and although I consider his epistemic standards to be above-average, I can tell he has not read the best of Slate Star Codex. I think if he did read a post such as "Asymmetrical Weapons", there’s a decent chance he would feel compelled to raise the bar for himself.

I have some close friends who sometimes spew (what I perceive to be) ignorant drivel. For some of them, I might be willing to pay a surprisingly high price to see them write a review of a book that cogently challenges their stupid priors. I would pay the highest price for friends that I already know can update on sound arguments, I would pay a lower price to find out if a friend has that ability, and for the attention of the hopelessly obstinate I would not want to pay anything.

Why This Wouldn’t Work

It's easy to underestimate how pervasive and sticky those signaling/tribal motivations are. The gains from trade available here might not be enough to overcome the pressure to protect narratives and maintain appearances.
There might be perverse incentives. Maybe the cognoscenti want to charge more, so they reduce supply (that is, they read less). Maybe they spout more ignorant nonsense in order to increase the b

...
Intermittent Distillations #2
by Mark Xu — 2021-04-14T06:47:16.356Z

Servant of Many Masters: Shifting priorities in Pareto-optimal sequential decision-making (Andrew Critch and Stuart Russell)

Servant of Many Masters: Shifting priorities in Pareto-optimal sequential decision-making

Summary

A policy (over some partially observable Markov decision process (POMDP)) is Pareto optimal with respect to two agents with different utility functions if it is not possible to construct a policy that achieves higher utility for one of the agents without doing worse for the other agent. A result by Harsanyi shows that for agents that have the same beliefs, Pareto optimal policies act as if they are maximizing some weighted sum of the two agents' utility functions. However, what if the agents have different beliefs?

Interestingly, if two agents disagree about the world, it is possible to construct policies that are better for both of them from that agent's perspective. For example, suppose that Alice and Bob are deciding how to split a cake. Suppose also that the cake is either red or green. Alice believes that the cake is red with 0.9 and Bob believes the cake is green with 0.9. A policy that says "If the cake is red, give it to Alice. If the cake is green, give it to Bob." will be viewed favorably by both of them. In fact, the sum of the utility Alice expects to get and the utility Bob expects to get is greater than can be achieved by any policy maximizing a weighted linear combination of their two utility functions.

Intuitively, when Alice and Bob both agree to the conditional policy, they're betting against each other about empirical facts about the world. More specifically, Alice can be viewed as offering to bet Bob that the cake is red, which Bob readily accepts. In this way, the conditional policy ties the expected utility of the two agents to previously irrelevant facts about their world models, giving them both higher expected utility from their perspectives.

The key result of the paper shows that all Pareto-optimal policies will have an implicit "bet settling" mechanism. One way of thinking about this is that since any difference in empirical beliefs can produce positive-sum bets between agents, a Pareto-optimal policy must implicitly make all such bets between those agents. Loosely speaking, the result shows that any policy that is Pareto-optimal with respect to a collection of agents will maximize a weighted linear combination of "...
What if AGI is near?
by Wulky Wilkinsen — 2021-04-14T00:05:44.337Z

Consider the following observations:

The

scaling hypothesis

is probably true: we will likely continue to see great improvements in AI capabilities as model sizes increase.

Sutskever mentioned that models currently under development already have dramatic and qualitative improvements (e.g. going more multimodal) over those already made in 2020.

AI model sizes are likely to increase very quickly over the short term.

NVIDIA’s CEO

: “We expect to see models with greater than 100 trillion parameters by 2023". Something 1000x the size of GPT-3, given its already shocking performance, is scary to imagine, to say the least.
Even if OpenAI is cautious and will not go ahead with potentially catastrophic projects (dubious), the wide availability and rapidly decreasing cost of efficient hardware, along with publicly available information about how to train GPT-like architectures, means that some organization will achieve scaling.

We will likely not be able to solve AI alignment within the next few years, even if AI safety research were to speed up dramatically.

Deep learning AIs in the near term cannot conceivably be anything besides not remotely aligned.

Questions:

What takeoff speeds are likely with large deep learning models, if they attain AGI? Are medium takeoffs more plausible due to deep learning type AIs having less obvious “tweaks” or ways to recursively self improve by altering code?

The key extension of that question is how will society react to presence of AGI? will it even shut down, stop development & wait for safety?
Perhaps takeoff speeds are moot beyond human level, because even a human-level AGI would have the wits to proliferate itself over the internet to computers all over the world and therefore make its eradication impossible once it has come into existence? Therefore once it has been created, it would be a slow but certain, inexorable death?

Given short term AGI, what options are available to reduce the risk of existential catastrophe? Would global nuclear war or a similarly devastating event that prevents technological progress be the only thing which could stave off AGI annihilation in the near term?

More serious thought needs to be given to this, to solemnly consider it as a looming possibility.
The Case for Extreme Vaccine Effectiveness
by Ruby — 2021-04-13T21:08:39.470Z

I owe tremendous acknowledgments to Kelsey Piper, Oliver Habryka, Greg Lewis, and Ben Shaya. This post is built on their arguments and feedback (though I may have misunderstood them).

Update, May 13
I first wrote this post before investigating the impact of covid variants on vaccine effectiveness, listing the topic as a major caveat to my conclusions. I have now spent enough time (not that much, honestly) looking into variants that I have a tentative position I'm acting on for now.

My moderately confident conclusion is that the current spread of variants in the US barely impacts vaccine effectiveness. The Pfizer vaccine is reported to be 85% as effective against the feared B.1.351 variant (South African) as it is against B.1.1.7 (UK). Assuming that other variants are no more resistant than B.1.351 on average (a reasonable assumption) and that presently variants are no more than 25% of Covid cases (in Alameda and San Francisco). The net effect is 0.25*0.85 + 0.75*1.0 = 0.9625. In other words, vaccines still have 96% of the effect they would if B.1.1.7 were the only variant.

Plus, that tiny reduction of vaccine effectiveness is dwarfed by the falling background prevalence of Covid. When I first wrote this post, Alameda and San Francisco were at 0.1-0.15%; now they're at ~0.05%. The same for New York and the United Kingdom.

Although relaxing of restrictions might reverse this, right now, Covid-risk is very, very low in the Bay Area and many parts of the US.

All updates/changelog can be viewed here.

I plead before the Master of Cost-Benefit Ratios. “All year and longer I have followed your dictates. Please, Master, can I burn my microCovid spreadsheets? Can I bury my masks? Pour out my hand sanitizer as a libation to you? Please, I beseech thee.”

“Well, how good is your vaccine?” responds the Master.

“Quite good!” I beg. “We’ve all heard the numbers, 90-95%. Even

MicroCOVID.org has made it official

: a 10x reduction for Pfizer and Moderna!”

The Master of Cost-Benefit Ratio shakes his head. “It helps, it definitely helps, but don’t throw out that spreadsheet just yet.

One meal at a crowded restaurant

is enough to give even a vaccinated person hundreds of microCovids. Not to mention that your local prevalence could change by a factor of 5 in the next month or two, and that’d be half the gains from this vaccine of yours!”

I whimper. “But what if . . . w

...
Modified bases in mRNA vaccines against Covid-19
by BB6 — 2021-04-13T13:00:53.523Z

What are the modified RNA bases in mRNA Covid vaccines and/or how are the metabolised in the human body ? Have you heard about any safety issues with these modified bases ?

For Pfizer, I know, that the modified bases are 1-methyl 3'-pseudouridine. From there, wikipedia directs me to the page on pseudouridine, which is naturally present in tRNA. However, it is not the same chemical. I do not know how easy or difficult for the human body is to remove the methyl moiety.

For Moderna, I do not know what are the modifications at all.

The reason I am worried: I have seen an unsourced claim on facebook, that Moderna has had a problem getting approved several of their products, because of the toxicity of the modified bases. Maybe it is a failure mode, letting an unsourced claim start a process of worrying and searching the information on the modified bases. But the process has already started in my head... I would very much appreciate some details, if you guys know them.
(Apart from the obvious info, that the clinical trials did not show any short od medium- term problem).
How do you deal with decision paralysis?
by Raj Thimmiah — 2021-04-13T06:01:10.274Z

I have too many choices in too many things and I can see that it's crippling my ability to get things done.

I'm struggling a great deal now to come up with plans and stick with them. I know choices are bad but I'm not sure what to do about it.

I think the solution from this video on the paradox of choice is to lower expectations but how do you actually do that, routinely?