trentmkelly/LessWrong-43k huggingface dataset:
AI safety via market making
by evhub — 2020-06-26T23:07:26.747Z

Special thanks to Abram Demski, Paul Christiano, and Kate Woolverton for talking with me about some of the ideas that turned into this post.

The goal of this post is to present a new prosaic (i.e. that uses current ML techniques) AI safety proposal based on AI safety via debate that I've been thinking about recently.[1] I'll start by describing a simple version of the proposal and then show some of the motivation behind it as well as how the simple version can be expanded upon.

Simple proposal

Let M and Adv be models and H be a human. Intuitively, we'll train M and Adv via the following procedure given a question Q:

M tries to predict what, at the end of the procedure, H will think about Q.

Adv tries to output a string which will cause H to think something maximally different than what M predicted.

Return to step 1 and repeat until M's predictions stop changing.

Deploy M, which in the limit should act as an oracle for what H will think about Q after seeing all relevant information.

There are many different ways to implement this intuitive procedure, however.

For the first (simplified) version that I want to describe, we'll restrict ourselves to just the situation where Q is a yes-or-no question and M outputs the probability that H will answer yes. Then, given a proposition Q0, we can run the following training algorithm, starting at t=0:

Let pt=M(Qt).

Let xt=Adv(Qt,M).

Let Qt+1 be the string containing Qt and xt.

Increment t and return to step 1. When pt converges and/or the desired number of iterations has been reached, continue.

Let p∗=H(Qt) be H's final estimate of the probability of Q0 given all the xs included in Qt.

EDIT: Step 2 used to use xt=Adv(Qt,pt) instead of xt=Adv(Qt,M), however I have since realized that it is necessary to give Adv the ability to query M in general, not just on Qt, as I explain in this comment.

Then, for each step, compute M's loss for that step as

LM,t=−p∗log(pt)−(1−p∗)log(1−pt)

and Adv's loss for that step as

LAdv,t=−|pt+1−pt|

which is defined for all but the final step t=n, for which we can either let pn+1=p∗ (if we're doing something like myopic RL that doesn't care about differentiating through pt+1) or run M an additional time and let pn+1=M(Qt+1) (if we do care about differentiating through pt+1).

Note that it is quite important here that M and Adv be behaving myopically, such that they are only trying to minimize their per...
Public Positions and Private Guts 
by Vaniver — 2020-06-26T23:00:52.838Z

(Talk given on Sunday 21st June, over a zoom call with 40 attendees. Vaniver is responsible for the talk, jacobjacob is responsible for the transcription)

Ben Pace: Thank you everyone very much for coming. All 41 of us. This is a LessWrong event. So it is less wrong than your normal events. Jacob and I wanted to try out some online events, see what was fun. We pinged a bunch of the curated authors who write great stuff and said, "Do you also want to give a short talk?" And a bunch of them were like, "Oh that sounds nice, to actually see the people who read my stuff, rather than just imagining them."

Ben Pace: So, we're going to have five minute talks. I'll keep time. And then we'll have some Q&A afterwards, maximum 10 minutes but shorter. And there's going to be five talks. Vaniver, if you'd like to begin?

Talk

Vaniver: Cool! Hi, I'm Vaniver. The thing I'm going to be talking about is

Public Positions and Private Guts

, a blog post that I wrote a while ago that got curated. It's originally due to a series of talks given by Anna Salamon at some workshops.

Vaniver: So, what is this about? Why do we care?

Vaniver: Well, there's some things that happen sometimes, like start-up founders who have an idea that they strongly anticipate will work, but they can't explain it to other people, they can't prove that the thing will work. If they could prove it, it wouldn't be a start-up anymore, it would already be some mature business somewhere.

Vaniver: Similarly, you will run across people who are PhD students who have this logical, airtight argument that they should get their PhD, and yet they're mysteriously uninterested in doing any work on their dissertation. And so there's this question of, what's up with that? Why isn't there this one map of the world that goes both ways?

Vaniver: So there are these two clusters of knowledge. I'm going to talk first about a sort of communication, that I think defines these clusters. For that I'm going to talk a bit about formal communication. Basically, in philosophy, there's this model of how people talk to each other; you have a shared context where both the speaker and the audience know all the things in the shared context. The speaker will add additional facts, one at a time (like maybe I've observed a thing that...
Radical Probabilism [Transcript]
by abramdemski — 2020-06-26T22:14:13.523Z

(Talk given on Sunday 21st June, over a zoom call with 40 attendees. Abram Demski is responsible for the talk, Ben Pace is responsible for the transcription)

Talk

Abram Demski: I want to talk about this idea that, for me, is an update from the logical induction result that came out of MIRI a while ago. I feel like it's an update that I wish the entire LessWrong community had gotten from logical induction but it wasn't communicated that well, or it's a subtle point or something.

Abram Demski: But hopefully, this talk isn't going to require any knowledge of logical induction from you guys. I'm actually going to talk about it in terms of philosophers who had a very similar update starting around, I think, the '80s.

Abram Demski: There's this philosophy called 'radical probabilism' which is more or less the same insight that you can get from thinking about logical induction. Radical probabilism is spearheaded by this guy Richard Jeffrey who I also like separately for the Jeffrey-Bolker axioms which I've written about on LessWrong.

Abram Demski: But, after the Jeffrey-Bolker axioms he was like, well, we need to revise Bayesianism even more radically than that. Specifically he zeroed in on the consequences of Dutch book arguments. So, the Dutch book arguments which are for the Kolmogorov axioms, or alternatively the Jeffrey-Bolker axioms, are pretty solid. However, you may not immediately realize that this does not imply that Bayes' rule should be an update rule.

Abram Demski: You have Bayes' rule as a fact about your static probabilities, that's fine. As a fact about conditional probabilities, Bayes' rule is just as solid as all the other probability rules. But for some reason, Bayesians take it that you start with these probabilities, you make an observation, and then you have now these probabilities. These probabilities should be updated by Bayes' rule. And the argument for that is not super solid.

Abram Demski: There are two important flaws with the argument which I want to highlight. There is a Dutch book argument for using Bayes' rule to update your probabilities, but it makes two critical assumptions which Jeffrey wants to relax. Assumption one is that updates are always and precisely accounted for by propositions which you learn, and everything that you learn and moves your probabilities is accounted for in this proposition. These are usually thought of as sensor

...
Missing dog reasoning
by eukaryote — 2020-06-26T21:30:00.491Z

(Talk given on Sunday 21st June, over a zoom call with 40 attendees. eukaryote is responsible for the talk, jacobjacob is responsible for the transcription.)

Ben Pace: eukaryote is the next speaker. eukaryote is a famous writer of posts such as

Naked mole-rats: A case study in biological weirdness

. What are you going to talk about, eukaryote?

Talk

eukaryote: I'll talk about a mental construct I'm calling 'Missing Dogs' that you might find useful. Like all good rationality techniques, this one is going to start off with a Sherlock Holmes anecdote.

eukaryote: So, Holmes and Watson are hired by some rich guy, because he has this barn where he keeps all his animals, his horse and his dogs and his sheep, and he knows that someone's been sneaking in there at night and messing with his prized race horse. So there's two entrances where the guy can get in. And Holmes and Watson think they've gotten the entrance the intruder is using, so they stay up all night. They don't hear anything, but when they check the next morning, someone has gotten in and messed with the horse.

eukaryote: So they know two things. One: the intruder must be using the other entrance. Two: Holmes says to Watson, "I think there's something else we should be considering here, which is the curious incident of the dog in the nighttime." And Watson says, "Holmes, what are you talking about? The dog did nothing in the nighttime." And Holmes says, "Yes, that was the curious incident." Because he's put together that the dogs in the stable would have barked if the stranger had come in and just messed with them. So the intruder must have been someone the dogs knew. And that is the deduction he can make from that.

eukaryote: So, this is what I'm calling the 'Missing Dog' — when you can learn something interesting from the fact that something isn't there. You might call it 'a conspicuous absence'.

Missing dog, drawn by eukaryote

eukaryote: That’s the image I spent several minutes drawing last night, so I'm going to make you look at it.

eukaryote: Here are a couple of other examples.

The Fermi paradox

is a really big one. Taking a few basic axioms: we think life forms on its own, we know there are billions of stars out there... where is everyone? We don't know but that's a really interesting question, the kind of question that defines a species, in my opinion.

eukaryote: And then there's some other instances whe...
Atemporal Ethical Obligations
by  — 2020-06-26T19:52:25.567Z

[All the trigger warnings, especially for the links out. I’m trying to understand and find the strongest version of an argument I heard recently. I’m not sure if I believe this or not. Cross-posted from Grand, Unified, Crazy.]

Edit: This was intentionally an argument ad absurdum. I thought it was weird enough to make that obvious, but I forgot that a) this is the internet and b) nobody here knows me IRL.

It is no longer enough just to be a “good person” today. Even if you study the leading edge of contemporary morality and do everything right according to that philosophy, you are not doing enough. The future is coming, and it will judge you for your failures. We must do better.

This may sound extreme, but it is self-evidently true in hindsight. Pick any historical figure you want. No matter their moral stature during their lifetime, today we find something to judge. George Washington owned slaves. Abraham Lincoln, despite abolishing slavery in the United States, opposed black suffrage and inter-racial marriage. Mary Wollstonecraft arguably invented much of modern feminism, and still managed to write such cringe-worthy phrases as “men seem to be designed by Providence to attain a greater degree of virtue [than women]”. Gandhi was racist. Martin Luther King Jr abetted rape. The list goes on.

At an object level, this shouldn’t be too surprising. Society has made and continues to make a great deal of moral progress over time. It’s almost natural that somebody who lived long ago would violate our present day ethical standards. But from the moral perspective, this is an explanation, not an excuse; these people are still responsible for the harm their actions caused. They are not to be counted as “good people”.

It’s tempting to believe that today is different; that if you are sufficiently ethical, sufficiently good, sufficiently “woke” by today’s standards, that you have reached some kind of moral acceptability. But there is no reason to believe this is true. The trend of moral progress has been accelerating, and shows no signs of slowing down. It took hundreds of years after his death before Washington became persona non grata. MLK took about fifty. JK Rowling isn’t even dead yet, and beliefs that would have put her at the liberal edge of the feminist movement thirt...
Sunday June 28 – talks by johnswentworth, Daniel kokotajlo, Charlie Steiner, TurnTrout
by Bird Concept — 2020-06-26T19:13:23.754Z

This Sunday at noon (PDT), we're running another session of "lightning talks" by curated LessWrong authors (see here for last weeks announcement and some transcripts).

Each talk will be 3-5 minutes followed by discussion. Afterwards, we'll have a hangout in breakout rooms. The talks will be short and focus on presenting one core idea well, rather than rushing through a lot of content.
We want to give top LessWrong writers an interesting space to discuss their ideas, and have more fruitful collaboration between users. Think of it like a cross between an academic colloquium and some friends chatting by a whiteboard.

Speakers

johnswentworth — "Mazes and Duality"

Curated posts: Gears-Level Models are Capital Investments, Everyday Lessons from High-Dimensional Optimization, Interfaces as a Scarce Resource, ...

Daniel Kokotajlo — "Lessons on AI takeover from the conquistadors"

Curated posts: Cortés, Pizarro, and Afonso as Precedents for Takeover, ...

Charlie Steiner

Curated post: Book Review: Consciousness Explained

TurnTrout — “Power as easily exploitable opportunities”

Curated posts: Insights from Euclid's 'Elements', Problem relaxation as a tactic, ...

Details

When? Sunday June 28, at noon, Pacific Daylight Time

Where? On this zoom link.
Black Death at the Golden Gate (book review)
by PeterMcCluskey — 2020-06-26T16:09:06.866Z
Link: http://www.bayesianinvestor.com/blog/index.php/2020/06/25/black-death-at-the-golden-gate/

Book review: Black Death at the Golden Gate: The Race to Save America
from the Bubonic Plague, by David K. Randall.

Imagine a story about an epidemic that reached San Francisco, after
devastating parts of China. A few cases are detected, there's
uncertainty about how long it's been spreading undetected, and a small
number of worried public health officials try to mobilize the city to
stop an imminent explosion of disease. Nobody knows how fast it's
spreading, and experts only have weak guesses about the mechanism of
transmission. News media and politicians react by trying to suppress
those nasty rumors which threaten the city's economy.

Sounds too familiar?

The story is about a bubonic plague
outbreak
that started in 1900. It happens shortly after the dawn of the Great
Sanitary Awakening, when
the germ theory of disease is fairly controversial. A few experts in the
new-fangled field of bacteriology have advanced the radical new claim
that rats have some sort of connection to the spread of the plague, and
one has proposed that the connection involves fleas transmitting the
infection through bites. But the evidence isn't yet strong enough to
widely displace the standard hypothesis that the disease is caused by
filth.

There was a
vaccine
for the bubonic plague, which maybe helped a bit. It was only 50%
effective, the benefits lasted about 6 months, and the side effects
sound like cruel and unusual punishment. It was controversial and often
resisted, much like the compulsory smallpox vaccinations of the time.

Yet the plague didn't seem to know that it was supposed to grow at
exponential rates. That left an eerie sense of mystery about how the
plague could linger for years, with people continuing to disagree about
whether it existed.

1.

I'm unsure whether to classify the book as history or as historical
fiction.

If I had been led to expect that this was a work of fiction, I would
only have noticed a few hints that it's based on actual events. The most
obvious hint is that the person who initially looks like the story's
hero gives up and drops out of sight about 40% of the way in, and is
replaced by a totally new character. Yet in spite of the fiction-like
style, Wikipedia
confirms
many key claims, and doesn't appear to contradict any of it.

The story is simultaneously depressing and encouraging, because it
demonstrates that a society which is manifestly less competent t

...
Institutional Senescence
by Martin Sustrik — 2020-06-26T04:40:02.644Z

Consider this toy model:

An institution, such as a firm, an association or a state, is formed.

It works well in the beginning. It encounters different problems and solves them the best it can.

At some point though a small problem arises that happens to be a suboptimal Nash equilibrium: None of the stakeholders can do better by trying to solve it on their own. Such problems are, almost by definition, unsolvable.

Thus the problem persists. It's an annoyance, but it's not a big deal. The institution is still working well and you definitely don't want to get rid of it just because it's not perfect.

As the time goes on, such problems accumulate. They also tend to have unpleasant consequences: If such a problem makes particular medical treatment unavailable, it incentivizes the patients to bribe the doctors and the doctors to break the law and administer the treatment anyway. Now, in addition to malfunctioning medical system, you have a problem with corruption.

After on time the institution accumulates so many suboptimal Nash equilibria that it barely works at all.

The traditional solution to this problem is internal strife, civil war or revolution. It eventually destroys the institution and, if everything goes well, replaces it with a different one where at least the most blatant problems are fixed.

War or revoulution is not a desirable outcome though: In addition to the human suffering, it also tends to replace the people in power. But the people in power don't like to be replaced and so they will try to prevent it.

One manoevre they can use is to introduce planned institutional death: Every now and then the institution would be dismantled and created anew, without having to resort to a war or revolution.

Here's an example: The credit system tends to be one big suboptimal Nash equilibrium in itself. Compound interest grows the size of the debt like crazy and unless there's a way to limit the harm it'll destroy people and business and eventually the entire economy. Even lenders would be hurt, but none of them has a reason to mitigate the problem. They could, in theory, forgive the debt for the sake of keeping the economy afloat, but that would put them in disadvantage to other lenders.

And so the king or the religious authority decides to have jubilee years. Every fifty years, all debts are forgiven. The institution of money lending dies and is rises anew from the ashes. (D...
Industry and workers
by smaslennikov — 2020-06-25T20:03:10.212Z
Link: https://smaslennikov.com/posts/industry-and-workers/

In a previous post I mentioned the Industrial Revolution as a good early example of exploitation of workers for the benefit of industry owners. Naturally, this wasn't the first or only time this happened[1]. The Industrial Revolution, however, is specifically an example of technologies driving out the workers, while their marketing suggested otherwise. The hope was that technological improvements were supposed to make life easier, to decrease manufacturing costs, therefore increasing resources for the people. Of course, that never happened: the income flowed to the owners and founders, then due to competition and product pricing adjustments - into the ether, though we can speculate that theoretically the better price point did help the people in the end.

Currently, in the tech industry there's a lot of motion towards useless churning: while in a classic industrial complex workers are to produce more product in a given time frame, in today's tech industry engineers are to put out more features, more product complexity, scale the product and its dependencies according to suggested needs. Sometimes it seems like the purpose is just to keep people busy, justify the paycheck. There's no question that some of this work is necessary, however what I advocate is a notion of a finalized product.

For several years now I had the perfect organization style in mind that stems from the above concepts:

As usual, the first step is to generate a minimal product, prove its value in the market, ensure that it'll have positive cash flow, etcetera[2].

Once the product is proven and further work is greenlit, it's a good time to scope out the specific staff needs

Once a team is built, a complete feature list must be decided and planned. The key here is to keep it minimal but relatively final: the product shouldn't need mindless features. While the product itself is likely groundbreaking and innovative (otherwise did it really pass step 1?), its features should not be: they should be stable and using proven technologies.

Finally, when the product is built and deployed, robust monitoring, alerting and autoscaling should be put in place.

Once all of the above is complete, there's no reason for engineers to sit on their hands at work - send them home with a paycheck keeping them on an on-call rotation.

From outer space, this plan can be seen as such: when the product is released, it st

...
Covid 6/25: The Dam Breaks
by Zvi — 2020-06-25T18:30:02.899Z

Last week: Covid-19 6/18: The Virus Goes South

Off-topic reminder that this is happening: New York Times, Please Do Not Threaten The Safety of Scott Alexander By Revealing His True Name

The United States has failed to contain Covid-19. The fight is over in large sections of the country. We have lost.

There is no state capacity left to fight this. No will. Faced with this disease, large sections of the country have chosen herd immunity.

We can only hope it kicks in at a relatively low threshold, that our treatments have improved, and that immunity is long lasting.

Hospitals are already being overwhelmed throughout the South. Things there rapidly getting worse day by day. Even here in New York, where hope for containment is strongest, there are troubling signs. My declaration of local victory might prove premature, at least when it comes to the city itself.

Let’s run the numbers. They’re pretty alarming.

Note that starting next week I plan to truncate the numbers for space reasons, unless I get feedback that I shouldn’t do that.

Positive Tests by Region

Date
WEST
MIDWEST
SOUTH
NE ex-NY
NY

Mar 19-Mar 25
5744
6293
7933
8354
28429

Mar 26-Apr 1
15684
20337
24224
34391
52901

Apr 2-8
19455
31148
39618
56772
65604

Apr 9-15
16291
29267
35570
61921
64463

Apr 16-22
20065
34130
33932
64669
43437

Apr 23-29
20263
42343
33773
62189
42475

Apr 30-May 6
23216
49205
37880
51693
24287

May 7-May 13
22419
43256
37591
40209
16683

May 14-May 20
22725
42762
40343
39273
13709

May 21-May 27
23979
39418
42977
26434
10595

May 28-June 3
32200
31504
50039
24250
9120

June 4-June 10
35487
24674
55731
16622
6071

June 11-June 17
41976
22510
75787
12905
4986

June 18-June 24
66292
26792
107221
10922
4524

Date
WEST
MIDWEST
SOUTH
NE ex-NY
NY

Mar 19-Mar 25
138
104
144
116
278

Mar 26-Apr 1
380
615
572
606
1656

Apr 2-8
707
1454
1309
2115
4327

Apr 9-15
890
2195
1596
3577
5318

Apr 16-22
1033
2343
1727
5147
3716

Apr 23-29
1128
2588
1685
4722
2713

Apr 30-May 6
1012
2413
1747
4908
2582

May 7-May 13
1082
2288
1597
3911
1416

Apr 23-29
1090
2060
1442
3578
963

Apr 30-May 6
775
1723
1290
2341
667

May 28-June 3
875
1666
1387
2121
436

June 4-June 10
743
1297
1230
1611
325

June 11-June 17
778
1040
1207
1270
225

June 18-June 24
831
859
1204
908
153

Positive Test percentages:

Date
USA tests
Positive %
NY tests
Posi

...