trentmkelly/LessWrong-43k huggingface dataset:
Persuasion Tools: AI takeover without AGI or agency?
by Daniel Kokotajlo — 2020-11-20T16:54:01.306Z

[epistemic status: speculation]

I'm envisioning that in the future there will also be systems where you can input any conclusion that you want to argue (including moral conclusions) and the target audience, and the system will give you the most convincing arguments for it. At that point people won't be able to participate in any online (or offline for that matter) discussions without risking their object-level values being hijacked.

--

Wei Dai

What if most people already live in that world? A world in which taking arguments at face value is not a capacity-enhancing tool, but a security vulnerability? Without trusted filters, would they not dismiss highfalutin arguments out of hand, and focus on whether the person making the argument seems friendly, or unfriendly, using hard to fake group-affiliation signals?

--

Benquo

1. AI-powered memetic warfare makes all humans effectively insane.

--

Wei Dai

, listing nonstandard AI doom scenarios

This post speculates about persuasion tools—how likely they are to get better in the future relative to countermeasures, what the effects of this might be, and what implications there are for what we should do now.

To avert eye-rolls, let me say up front that I don’t think the world is likely to be driven insane by AI-powered memetic warfare. I think progress in persuasion tools will probably be gradual and slow, and defenses will improve too, resulting in an overall shift in the balance that isn’t huge: a deterioration of collective epistemology, but not a massive one. However, (a) I haven’t yet ruled out more extreme scenarios, especially during a slow takeoff, and (b) even small, gradual deteriorations are important to know about. Such a deterioration would make it harder for society to notice and solve AI safety and governance problems, because it is worse at noticing and solving problems in general. Such a deterioration could also be a risk factor for world war three, revolutions, sectarian conflict, terrorism, and the like. Moreover, such a deterioration could happen locally, in our community or in the communities we are trying to influence, and that would be almost as bad. Since

the date of AI takeover is not the day the AI takes over

, but the point it’s too late to reduce AI risk, these things basically shorten timelines.

Six examples of persuasion tools

Analyzers: Political campaigns and advertisers already use focus groups, A/B test...
Hiding Complexity
by Rafael Harth — 2020-11-20T16:35:25.498Z

1. The Principle

Suppose you have some difficult cognitive problem you want to solve. What is the difference between (1) making progress on the problem by thinking about it for an hour and (2) solving a well-defined subproblem whose solution is useful for the entire problem?

(Finding a good characterization of the 'subproblem' category is important for Factored Cognition, but for [this post minus the last chapter], you can think of it purely as a problem of epistemic rationality and human thinking.)

I expect most to share the intuition that there is a difference. However, the question appears ill-defined on second glance. 'Making progress' has to cash out as learning things you didn't know before, and it's unclear how that isn't 'solving subproblems'. Whatever you learned could probably be considered the solution to some problem.

If we accept this, then both (1) and (2) technically involve solving subproblems. Nonetheless, we would intuitively talk about subproblems in (2) and not in (1). Can we characterize this difference formally? Is there a well-defined, low-level quantity such that our intuition as to whether we would call a bundle of cognitive work a 'subproblem' corresponds to the size of this quantity? I think there is. If you want, take a minute to think about it yourself; I've put my proposed solution into spoilers.

I think the quantity is the length of the subproblem's solution, where by "solution", I mean "the information about the subproblem relevant for solving the entire problem".

As an example, suppose the entire problem is "figure out the best next move in a chess game". Let's contrast (1) and (2):

(1) was someone thinking about this for an hour. The 'solution' here consists of everything she learns throughout that time, which may include many different ideas/insights about different possible moves/resolved confusions about the game state. There is probably no way to summarize all that information briefly.

(2) was solving a well-defined subproblem. An example here is, "figure out how good Be5 is".[1] If the other side can check in four turns given that move, then the entire solution to this subproblem is the three-word statement "Be5 is terrible".

2. The Software Analogy

Before we get to why I think the principle matters, let's try to understand it better. I think the analogy to software design is helpful here.

Suppose a company wants to...
Thanksgiving and Covid
by jefftk — 2020-11-20T04:30:05.666Z

It's getting colder and the virus is still spreading. Hundreds of
thousands of people are dead. It's 399 years later, and Thanksgiving
has changed less than you might have have hoped.

When the English arrived in what is now Massachusetts they found the
Wampanoag devastated by smallpox:

The people not many, being dead and abundantly wasted in the late
great mortality which fell in all these parts about three years before
the coming of the English, wherein thousands of them died,
they not being able to bury one another; their skulls and bones were
found in many places lying still above ground, where their houses and
dwellings had been; a very sad spectacle to behold.

—Of
Plymouth Plantation, Bradford.

Smallpox was still spreading, however, and later he writes (warning:
gore):

This spring, also, those Indians that lived about
their trading house there fell sick of the small pox,
and died most miserably; for a sorer disease cannot
befall them; they fear it more then the
plague; for usually they that have this disease have them in
abundance, and for want of bedding and lining and other helps, they
fall into a lamentable condition, as they lie on their hard mats, the
pox breaking and mattering, and running one into another, their skin
cleaving (by reason thereof) to the mats they lie on; when they turn
them, a whole side will flay off at once, (as it were) and they will
be all of a gore blood, most fearful to behold; and then being very
sore, what with could and other distempers, they die like rotten
sheep. The condition of this people was so lamentable, and they fell
down so generally of this disease, as they were (in the end) not able
to help on another; no, not to make a fire, nor to fetch a little
water to drink, nor any to bury the dead; but would strive as long as
they could, and when they could procure no other means to make fire,
they would burn the wooden trays and dishes they ate their meat in,
and their very bows and arrows; and some would crawl out on all fours
to get a little water, and sometimes die by the way, and not be able
to get in again.

But those of the
English house, (though at first they were afraid of the infection),
yet seeing their woeful and sad condition, and hearing their pitiful
cries and lamentations, they had compassion of them, and daily
fetched them wood and water, and made them fir

...
Simpson's paradox and the tyranny of strata
by dynomight — 2020-11-19T17:46:32.504Z
Link: https://dyno-might.github.io/2020/11/16/simpsons-paradox-and-the-tyranny-of-strata/

Simpson's paradox is an example of how the same data can tell different stories. Most people think of this as an odd little curiosity, or perhaps a cautionary tale about the correct way to use data.

You shouldn't see Simpson's paradox like that. Rather than some little quirk, it's actually just the simplest case of a deeper and stranger issue. This is less about the “right” way to analyze data and more about limits to what questions data can answer. Simpson's paradox is actually a bit misleading, because it has a solution, while the deeper issue doesn't.

This post will illustrate this using no statistics and (basically) no math.

I Zeus

You are a mortal. You live near Olympus with a flock of sheep and goats. Zeus is a jerk and has taken up shooting your animals with lightning bolts.

He doesn't kill them; it's just boredom. Transforming into animals to seduce love interests gets old eventually.

Anyway, you wonder: Does Zeus have a preference for shooting sheep or goats? You decide to keep records for a year. You have 25 sheep and 25 goats, so you use a 5x5 grid with one cell for each animal.

At first glance, it seems like Zeus dislikes goats more than He dislikes sheep. (If you're worried about the difference being due to random chance, feel free to multiply the number of animals by a million.)

II Colors

Thinking more, it occurs to you that some animals have darker fur than others. You go back to your records and mark each animal accordingly.

You re-do the analysis, splitting into dark and light groups.

Overall, sheep are zapped less often than goats. But dark sheep are zapped more often than dark goats (7⁄11 > 10⁄16) and light sheep are zapped more often than light goats (5⁄14 > 3⁄9). This is the usual paradox: The conclusion changes when you switch from analyzing everyone to splitting into subgroups.

How does that reversal happen? It's simple: For both sheep and goats, dark animals get zapped more often, and there are more dark goats than dark sheep. Dark sheep are zapped slightly more than dark goats, and similarly for light sheep. But dumping all the animals together changes the conclusion because there are so many more dark goats. That's all there is to the regular Simpson's paradox. Group-level differences can be totally different than subgroup differences when the ratio of subgroups varies.

This probably seems like a weird little edge case so far. But let...
Notes on Phronesis
by David Gross — 2020-11-19T16:14:16.838Z

This post examines the virtue of phronesis (“prudence”). It is meant mostly as an exploration of what other people have learned about this virtue, rather than as me expressing my own opinions about it, though I’ve been selective about what I found interesting or credible, according to my own inclinations. I wrote this not as an expert on the topic, but as someone who wants to learn more about it. I hope it will be helpful to people who want to know more about this virtue and how to nurture it.

What is prudence?

Prudence is one of the four cardinal virtues. From there it became part of the seven traditional Christian virtues. It turns up again and again in virtue traditions. I can’t very well ignore it.

And yet… the word “prudence” has gone through such a dramatic shift in meaning that it’s difficult to know how to tackle this one.

“Prudence” was a common English translation of the Greek word phrónēsis, which has implications that range from having how-to skills to things like choosing your goals wisely and exercising good judgment when picking paths to those goals. In short, it is wisdom applied to practical, real-world decision-making, where the rubber meets the road.

When prudence was incorporated into the traditional Christian virtues, it was via the Latin word prudentia, which can mean things like rationality, insight, discernment, foresight, wisdom, or skill. Again, though, the focus is on the quality of your process of making practical decisions, so this isn’t too far off.

Dana Carvey as President G.H.W. Bush on Saturday Night Live

But nowadays when you call someone “prudent” you usually mean that they are cautious: they plan ahead, look before they leap, avoid taking unnecessary risks, save for a rainy day, and that sort of thing. The word now has an old-fashioned sound to it, and is rare enough as a compliment that it’s sometimes even deployed as an insult, to imply that the “prudent” person is over-cautious, timid, afraid to take chances, or reluctant to innovate. (The resemblance of the word “prudence” to the etymologically distinct word “prudish” has also contributed to giving the word a stuffy connotation.)

Because of this meaning shift, when you see someone singing the praises of “prudence” it’s important to investigate further to find out which sort of prudence they’re praising. Sometimes authors will even drift from one definition to the other without...
Covid 11/19: Don’t Do Stupid Things
by Zvi — 2020-11-19T16:00:00.895Z

There is very good news. We have a second vaccine! Both Pfizer and Moderna’s vaccines have now shown 94%+ effectiveness in treating Covid-19. Not to be outdone by Moderna’s report of 94.5% effectiveness, Pfizer has its final results and they are excellent. 95% effective, 94% effective in people over 65, no major safety concerns. They plan to apply for emergency use authorization within days and likely already have done so by the time you read this. Estimates are that we can vaccinate 20 million people by year’s end.

There is also very bad news. We have more positive test results than ever before, more hospitalizations than ever before, and a substantially increased positive test rate. Deaths lag tests, but are on track to rise proportionally to the rise in tests.

Within that bad news there is good news. There are signs that the Midwest in particular, where things are maximally bad right now, is getting ready to peak. My estimate for how bad things will get before they get better, and how long it will be before that happens, have both tentatively gone down somewhat. Bad news can still be good news if you expected even worse news.

There is also an important note before we get to business.

Last week in my weekly post, I said some things about the election that did not relate to Covid-19. Doing so broke the norms of my personal blog and the norms of LessWrong, where these updates are reposted. I felt it was the least bad option under the circumstances. I could have executed it better, but I stand by my decision, and accept the consequences.

I do not intend to let it happen again.

One of the consequences of breaking this norm is that it is even more important for me to strongly reassert the norm going forward. Tempted as I may become, I will be extra careful not to discuss politics except as it directly relates to Covid-19 or requires us to take precautions for our own safety. I also plan to enforce the rule strongly in the comments.

If you feel I have crossed that line in this or a future post in this series, once you have verified no one else has yet called me out, please do not hesitate to call me out on that.

If you feel I have crossed that line in a post not in this series, and I didn’t explicitly acknowledge that I was doing a politics-necessary thing like a prediction market post, call me out on that too.

Thank you. Let’s run the numbers.

The Numbers

...
How do you evaluate whether a $500 donation to a project that you know well is a good idea?
by ChristianKl — 2020-11-19T12:42:41.732Z

At habryka's recent office hours he made the point that it's really hard for a big grand-giving organization like the Long-Term Future Fund (LTFF) to fund certain projects. While good rationalist culture for example is valuable, funding culture by committee in a top-down way is problematic. The people with local knowledge of a cultural project are in a better position to judge whether or not a given rationalist culture project is valuable enough that it should be funded then a grant committee like that of the LTFF.

Given that LTFF and OpenPhil have access to a lot of financial captial these days for very rich individuals, the scare resources these days isn't primarily money but local knowledge about which projects are worthwhile. This suggests that individuals who have a small donation budget and access to a lot of local knowledge of individual projects are more effective when they make use of their scarce resources of local knowledge instead of donating to money to big grant making organizations.

While giving money to big grant makers is straightforward, giving money to small projects is harder. How should an individual who has a yearly donation budget of <$10,000 go about evaluating local projects for donating an amount like $500 to them?
Some AI research areas and their relevance to existential safety
by Andrew_Critch — 2020-11-19T03:18:22.741Z

Followed by: What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs), which provides examples of multi-stakeholder/multi-agent interactions leading to extinction events.

Introduction

This post is an overview of a variety of AI research areas in terms of how much I think contributing to and/or learning from those areas might help reduce AI x-risk. By research areas I mean “AI research topics that already have groups of people working on them and writing up their results”, as opposed to research “directions” in which I’d like to see these areas “move”.

I formed these views mostly pursuant to writing AI Research Considerations for Human Existential Safety (ARCHES). My hope is that my assessments in this post can be helpful to students and established AI researchers who are thinking about shifting into new research areas specifically with the goal of contributing to existential safety somehow. In these assessments, I find it important to distinguish between the following types of value:

The helpfulness of the area to existential safety, which I think of as a function of what services are likely to be provided as a result of research contributions to the area, and whether those services will be helpful to existential safety, versus
The educational value of the area for thinking about existential safety, which I think of as a function of how much a researcher motivated by existential safety might become more effective through the process of familiarizing with or contributing to that area, usually by focusing on ways the area could be used in service of existential safety.
The neglect of the area at various times, which is a function of how much technical progress has been made in the area relative to how much I think is needed.

Importantly:

The helpfulness to existential safety scores do not assume that your contributions to this area would be used only for projects with existential safety as their mission. This can negatively impact the helpfulness of contributing to areas that are more likely to be used in ways that harm existential safety.
The educational value scores are not about the value of an existential-safety-motivated researcher teaching about the topic, but rather, learning about the topic.
The neglect scores are not measuring whether there is enough “buzz” around the topic, but rather, whether there has been

...
Sunzi's《Methods of War》- War
by lsusr — 2020-11-19T03:06:12.343Z

This is a translation of Chapter 2 of The Art of War by Sunzi. No English sources were used.

孙子曰：凡用兵之法，驰车千驷，革车千乘，带甲十万，千里馈粮，内外之费，宾客之用，胶漆之材，车甲之奉，日费千金，然后十万之师举矣。

The ordinary methods of war demand:

1,000 teams of 4 horses each,

1,000 wagons,

100,000 shields,

provisions to march 1,000 miles,

domestic and foreign expenses,

hospitality for guests,

construction materials for siege weapons,

armored vehicles,

salaries

…and an army of 100,000 soldiers.

其用战也胜，久则钝兵挫锐，攻城则力屈，久暴师则国用不足。

A long war is an expensive war.

夫钝兵挫锐，屈力殚货，则诸侯乘其弊而起，虽有智者，不能善其后矣。

An expensive war will cause your vassals to rebel against you.

故兵闻拙速，未睹巧之久也。夫兵久而国利者，未之有也。

There is no such thing as a beneficial protracted war.

故不尽知用兵之害者，则不能尽知用兵之利也。

If you do not understand the costs of war then you do not know which wars are worthwhile to fight.

善用兵者，役不再籍，粮不三载，取用于国，因粮于敌，故军食可足也。

Do not conscript troops more than once. Do not resupply your army with grain more than twice. Take what you need from the enemy. The enemy has ample grain and an army of troops.

国之贫于师者远输，远输则百姓贫；

Resupplying an army over long distances impoverishes a country.

近师者贵卖，贵卖则百姓财竭，财竭则急于丘役。

Prices soar in wartime. Levying the peasantry under such circumstances will impoverish them while extracting only forced labor.

力屈财殚，中原内虚于家，百姓之费，十去其七；

The central plains will go unfarmed. Seven tenths of the peasantry's labor will be wasted.

公家之费，破军罢马，甲胄矢弩，戟楯蔽橹，丘牛大车，十去其六。

Supplying an army out of the public purse slows the army down. Horses sicken. Shields split. Oxen tire. Six tenths is wasted.

故智将务食于敌，食敌一钟，当吾二十钟；萁秆一石，当吾二十石。

The wise general eats the enemy's food. A captured bowl of enemy food is worth twenty bowls of your own. A captured ton of enemy grain is worth twenty tons of your own.

故杀敌者，怒也；取敌之利者，货也。车战得车十乘以上，赏其先得者，而更其旌旗，车杂而乘之，卒善而养之，是谓胜敌而益强。

Let your troops kill the enemy in anger, plunder the enemy in greed. A captured enemy combat vehicle is worth no fewer than ten of your own. Reward your first soldier to capture one. Replace its flag. Mix it in among your own.

A good soldier steals victory from the enemy.

故兵贵胜，不贵久。故知兵之将，生民之司命，国家安危之主也。

A valuable victory is a quick victory. A general who, understanding this, issues orders to the people—thereupon is the fate of a state determined.
Inner Alignment in Salt-Starved Rats
by Steven Byrnes — 2020-11-19T02:40:10.232Z

(This post is deprecated. It has some kernels of truth but also lots of mistakes and confusions. You should instead read Incentive Learning vs Dead Sea Salt Experiment (2024), which covers many of the same topics. —Steve, 2024)

(See comment here for some corrections and retractions. —Steve, 2022)

Introduction: The Dead Sea Salt Experiment

In

this 2014 paper

by Mike Robinson and Kent Berridge at University of Michigan (see also

this more theoretical follow-up discussion by Berridge and Peter Dayan

), rats were raised in an environment where they were well-nourished, and in particular, where they were never salt-deprived—not once in their life. The rats were sometimes put into a test cage with a lever which, when it appeared, was immediately followed by a device spraying ridiculously salty water directly into their mouth. The rats were disgusted and repulsed by the extreme salt taste, and quickly learned to hate the lever—which from their perspective would seem to be somehow causing the saltwater spray. One of the rats went so far as to stay tight against the opposite wall—as far from the lever as possible!

Then the experimenters made the rats feel severely salt-deprived, by depriving them of salt. Haha, just kidding! They made the rats feel severely salt-deprived by injecting the rats with a pair of chemicals that are known to induce the sensation of severe salt-deprivation. Ah, the wonders of modern science!

...And wouldn't you know it, almost instantly upon injection, the rats changed their behavior! When shown the lever (this time without the salt-water spray), they now went right over to that lever and jumped on it and gnawed at it, obviously desperate for that super-salty water.

The end.

Aren't you impressed? Aren’t you floored? You should be!!! I don’t think any standard ML algorithm would be able to do what these rats just did!

Think about it:

Is this Reinforcement Learning? No. RL would look like the rats randomly stumbling upon the behavior of “nibbling the lever when salt-deprived”, find it rewarding, and then adopt that as a goal via “credit assignment”. That’s not what happened. While the rats were nibbling at the lever, they had never in their life had an experience where the lever had brought forth anything other than an utterly repulsive experience. And they had never in their life had an experience where they were salt-deprived, tasted something extreme

...