trentmkelly/LessWrong-43k huggingface dataset:
Forecasting Thread: Existential Risk
by Amandango — 2020-09-22T03:44:28.622Z

This is a thread for displaying your probabilities of an existential catastrophe that causes extinction or the destruction of humanity’s long-term potential.

Every answer to this post should be a forecast showing your probability of an existential catastrophe happening at any given time.

For example, here is

Michael Aird’s timeline

:

The goal of this thread is to create a set of comparable, standardized x-risk predictions, and to facilitate discussion on the reasoning and assumptions behind those predictions. The thread isn’t about setting predictions in stone – you can come back and update at any point!

How to participate

Go to

this page

Create your distribution

Specify an interval using the Min and Max bin, and put the probability you assign to that interval in the probability bin.
You can specify a cumulative probability by leaving the Min box blank and entering the cumulative value in the Max box.
To put probability on never, assign probability above January 1, 2120 using the edit button to the right of the graph. Specify your probability for never in the notes, to distinguish this from putting probability on existential catastrophe occurring after 2120.

Click 'Save snapshot' to save your distribution to a static URL

A timestamp will appear below the 'Save snapshot' button. This links to the URL of your snapshot.
Make sure to copy it before refreshing the page, otherwise it will disappear.

Click ‘Log in’ to automatically show your snapshot on the Elicit question page

You don’t have to log in, but if you do, Elicit will:

Store your snapshot in your account history so you can easily access it.
Automatically add your most recent snapshot to the x-risk question page under ‘Show more’. Other users will be able to import your most recent snapshot from the dropdown, shown below.

We’ll set a default name that your snapshot will be shown under – if you want to change it, you can do so on your profile page.
If you’re logged in, your snapshots for this question will be publicly viewable.

Copy the snapshot timestamp link and paste it into your LessWrong comment

You can also add a screenshot of your distribution in your comment using the instructions below.

Here's an example of how to make your distribution:

How to add an image to your comment

Take a screenshot of your distribution
Then do one of two things:

If you have beta-features turned on in your account settings, drag-and-dr

...
What AI companies would be most likely to have a positive long-term impact on the world as a result of investing in them?
by MikkW — 2020-09-21T23:41:24.281Z

Ever since GPT-3 was unveiled, I've been thinking pretty heavily about increasing my investment in AI-related companies. My first thoughts were to invest in Microsoft and Alphabet (Google) - Microsoft because they are partnered with OpenAI, and Alphabet since they have big AI reseach projects of their own. But in the process of thinking about investing in these companies, I started wondering about the long-term impacts such investments would have on the world - investing in the right or wrong company could dramatically change how the world looks 20 years from now, and whether it is a place I'd want to live in - the worst case scenario would be all humans dead, or even worse; best case scenario is... too amazing to put into words. And then there's plenty of room in between those for how things can go, depending on who makes the important decisions, and how good the decisions they make will be. (While I'm only a single person with modest funds to invest in companies, I also consider that my actions are acausally correlated with those of others sufficiently similar to me, which means the acausal results of any investment I make will be multiplied by an amount that makes my decisions have non-trivial impact on the world).

So the important question is, do I expect Microsoft and Alphabet to do better or worse, in regards to alignment and ethical issues, compared to other actors who will develop AGI in their lieu? (I do expect someone will develop AGI in their lieu) I can think of actors who I expect will likely do worse than Microsoft or Alphabet - the government of basically any country, or firms based in a country with more totalitarian ethics than the US - wheras I can only think of alternative actors who I expect to do roughly as good as Microsoft or Alphabet, but not neccesarily to do better than them. I trust MIRI, but I also don't perceive MIRI as being actively involved in the development of working AI systems; it seems to me that they are laying the important theoretical groundwork for getting things right, but aren't in position to be the ones who actually do the work that needs to be gotten right.

So my main problem here is a lack of knowledge - there almost certainly are other firms who, if I had the relevant information, I would expect would do better on alignment and ethical issues than Microsoft or Alphabet, but I also do...
Prepare for COVID-19 Human Challenge Trials — A Petition in Canada (and soon, the UK)
by abierohrig — 2020-09-21T19:38:04.158Z

Canadians— sign the petition here.

TL;DR

COVID-19 human challenge trials could save tens of thousands of lives by quickly narrowing the field of promising candidates, and there is strong reason to believe that signaling clear public support for these trials via an official petition could meaningfully accelerate preparation.

Why COVID-19 Human Challenge Trials?

In a COVID-19 human challenge trial, willing participants would receive the vaccine candidate and, once the vaccine takes effect, be deliberately exposed to live coronavirus. The ability to observe participants closely and gather samples while tracing the progress of infection in real time, knowing exactly when they were infected and with what dose, and being able to follow up over a long period, would offer an unprecedented level of scientific and medical insight into an unfamiliar virus. It would also help us test vaccines far faster. If a challenge trial brings us one day closer to the development of an additional vaccine that could avert just 25% of daily COVID-19 deaths, it would save 1,250 lives. If a challenge trial brings us a month closer, it’d save 37,500 lives.

To learn more about COVID-19 human challenge trials:

Watch this

Vox video

Read this

piece

by Dr. Sayantan Banerjee in The Telegraph
Read this

paper

in the Journal of Clinical Infectious Diseases

Why Is It Cost-Effective To Sign The Petition?

For one, it is remarkably easy (takes around 20 seconds), so even a very small chance that your signature makes a difference tips the scale in any cost-effectiveness calculation on the margin.

More broadly, though, signaling a groundswell of public support for COVID-19 human challenge trials has directly led to faster preparation for these trials. In May, a NIH document

noted

that their consideration of challenge trials “has been driven almost entirely by the altruism of potential volunteer advocates and the intense considerations of bioethicists.”

1Day Sooner, which has worked systematically to include volunteers in the public conversation about challenge trials, launched an

open letter

in support of COVID-19 challenge trials on July 15 that was signed by over 100 academics and experts as well as 2,000 potential challenge trial volunteers. A week later, the Washington Post Editorial Board

wrote

in favor of challenge trial preparation. Within a few weeks, Reuters

reported

...
How often do series C startups fail to exit?
by Jessica Shu — 2020-09-21T19:37:33.671Z

How often do series C startups really fail? By fail I mean never have an acquisition or IPO. Internet says 80% (see https://medium.com/journal-of-empirical-entrepreneurship/dissecting-startup-failure-by-stage-34bb70354a36) but this seems very high to me.
Most Series C companies are worth in the 100-200M range, the one I'm at is worth 270M. How does all the value just evaporate? What happens to the companies that "fail"?
Asking to decide whether to exercise my options. I only need my company to exit at 41M to break even. I am bearish on the company but with around 40M in ARR it is hard to imagine it not exiting.
Needed: AI infohazard policy
by Vanessa Kosoy — 2020-09-21T15:26:05.040Z

The premise of AI risk is that AI is a danger, and therefore research into AI might be dangerous. In the AI alignment community, we're trying to do research which makes AI safer, but occasionally we might come up with results that have significant implications for AI capability as well. Therefore, it seems prudent to come up with a set of guidelines that address:

Which results should be published?

What to do with results that shouldn't be published?

These are thorny questions that it seems unreasonable to expect every researcher to solve for themselves. The inputs to these questions involve not only technical knowledge about AI, but also knowledge about the behavior of progress, to the extent we can produce such using historical record or other methods. AI risk organizations might already have internal policies on these issues, but they don't share them and don't discuss or coordinate them with each other (that I know of: maybe some do it in private channels). Moreover, coordination might be important even if each actor is doing something reasonable when regarded in isolation (avoiding bad Nash equilibria). We need to have a public debate on the topic inside the community, so that we arrive at some consensus (that might be updated over time). If not consensus, then at least a reasonable spectrum of possible policies.

Some considerations that such a policy should take into account:

Some results might have implications that shorten the AI timelines, but are still good to publish since the distribution of outcomes is improved.

Usually we shouldn't even start working on something which is in the should-not-be-published category, but sometimes the implications only become clear later, and sometimes dangerous knowledge might still be net positive as long as it's contained.

In the midgame, it is unlikely for any given group to make it all the way to safe AGI by itself. Therefore, safe AGI is a broad collective effort and we should expect most results to be published. In the endgame, it might become likely for a given group to make it all the way to safe AGI. In this case, incentives for secrecy become stronger.

The policy should not fail to address extreme situations that we only expect to arise rarely, because those situations might have especially major consequences.

Some questions that such a policy should answer:

What are the criteria that determine whether

...
Zen and Rationality: Just This Is It
by Gordon Seidoh Worley — 2020-09-20T22:31:56.338Z

This is post 4/? about the intersection of my decades of LW-style rationality practice and my several years of Zen practice.

In today's installment, I look at "just this is it" from a rationalist perspective.

When Dongshan, the co-founder of what would become the Soto Zen school within which I practice, was preparing to leave his teacher Yunyan and go out in the world, he asked Yunyan how he might summarize his teaching. Yunyan replied, "just this [is it]". Because the more we say the more we move into the world of words and away from reality as it is on its own prior to conception, this is often shorted in various ways to "just this" or "just is" or "this is" or "it is" or, perhaps best of all short of saying nothing and letting reality stand on its own, "is".

This is arguably the core teaching of Soto Zen and maybe all of Buddhism, to perceive and accept reality just as it is. Yet I see it all over the place in the LessWrong corpus, too. I'll mention a few of these.

Egan's Law posits that "it all adds up to normality". In "A Technical Explanation of Technical Explanation", Eliezer phrased a similar sentiment as "since the beginning, not one unusual thing has ever happened". Both point at the way that reality is just as it is, and the only way we can be confused or surprised is because we had an idea about how reality is rather than simply looking and seeing how it is.

I think this is a hard thing to remember, because to the kind of people that are attracted to Less Wrong, better models of reality are very attractive. I know they are to me! Yet it's very easy to go from accepting reality as it is and trying to better predict it to getting lost in the model that does the predicting and confusing it for the real thing. Thus, while at the same time we look for models with better gears that more precisely carve reality at its joints, we also have to remember those boundaries are fuzzy and that all models are ultimately wrong even and especially when they are useful. It's perhaps the great koan of Less Wrong to build better models while simultaneously accepting that all models are somewhere wrong.

To help us deal with this koan, we have a poem to help us. You might think I mean the Litany of Tarski, but you would be wrong, because that poem is about having beliefs correspond to reality, but "just this is it" is all about getting under those beliefs and just seeing wha...
Clarifying “What failure looks like”
by Sam Clarke — 2020-09-20T20:40:48.295Z

Thanks to Jess Whittlestone, Daniel Eth, Shahar Avin, Rose Hadshar, Eliana Lorch, Alexis Carlier, Flo Dorner, Kwan Yee Ng, Lewis Hammond, Phil Trammell and Jenny Xiao for valuable conversations, feedback and other support. I am especially grateful to Jess Whittlestone for long conversations and detailed feedback on drafts, and her guidance on which threads to pursue and how to frame this post. All errors are my own.

Epistemic status: My Best Guess

Epistemic effort: ~70 hours of focused work (mostly during FHI’s summer research fellowship), talked to ~10 people.

Introduction

“What failure looks like” is the one of the most comprehensive pictures of what failure to solve the AI alignment problem looks like, in worlds without discontinuous progress in AI. I think it was an excellent and much-needed addition to our understanding of AI risk. Still, if many believe that this is a main source of AI risk, I think it should be fleshed out in more than just one blog post. The original story has two parts; I’m focusing on part 1 because I found it more confusing and nebulous than part 2.

Firstly, I’ll summarise part 1 (hereafter “WFLL1”) as I understand it:

In the world today, it’s easier to pursue easy-to-measure goals than hard-to-measure goals.

Machine learning is differentially good at pursuing easy-to-measure goals (assuming that we don’t have a satisfactory technical solution to the intent alignment problem[1]).

We’ll try to harness this by designing easy-to-measure proxies for what we care about, and deploy AI systems across society which optimize for these proxies (e.g. in law enforcement, legislation and the market).

We’ll give these AI systems more and more influence (e.g. eventually, the systems running law enforcement may actually be making all the decisions for us).

Eventually, the proxies for which the AI systems are optimizing will come apart from the goals we truly care about, but by then humanity won’t be able to take back influence, and we’ll have permanently lost some of our ability to steer our trajectory.

WFLL1 is quite thin on some important details:

WFLL1 does not envisage AI systems directly causing human extinction. So, to constitute an existential risk in itself, the story must involve the lock-in of some suboptimal world.[2] However, the likelihood that the scenario described in part 1 gets locked-in (especially over very long t

...
Are aircraft carriers super vulnerable in a modern war?
by Liron — 2020-09-20T18:52:29.270Z

It seems like aircraft carriers are a good candidate for something that only exists because it made sense in WWII and it looks the part of “impressive military asset”, i.e. it’s all larping at this point. It seems vulnerable to attack relative to its huge cost because offense has an advantage over defense: I.e. can’t an enemy send tons of cheap drone planes and drone submarines to first hunt for its location and then swarm-attack it?

Note: I don’t know anything about this subject. The ideal answerer is someone with domain knowledge who has good epistemology; that’s why I wasn’t satisfied with a Google search and I’m asking here.
I'm Voting For Ranked Choice, But I Don't Like It
by jefftk — 2020-09-20T18:40:04.410Z

This fall,

Ranked
Choice / Instant Runoff Voting (IRV) will be

on
the ballot in Massachusetts. I'm voting for it, but only because
it's better than the status quo, not because I think it's a very good
voting system.

Massachusetts currently uses traditional majority ("first past the
post") voting: whoever gets the most votes wins. Unfortunately, this
only works well when you have two candidates. With more candidates,
the candidates tend to hurt
their allies by competing for the same pool of votes, making it
more likely that an opponent wins.

In IRV each voter lists their preferred candidates in order, and if
your first choice is eliminated then your vote goes to your next
favorite. This mostly fixes the problem of minor spoiler candidates:
anyone who is not a serious contender will get eliminated and their
votes redistributed.

Unfortunately, IRV has major problems when you have more than two
serious candidates. For example, even if there is a candidate that a
majority of
voters prefer to every other, they can still lose if their
competitors happen to be eliminated in the wrong order. In Why Ranked Choice Voting
Isn't Great I give examples of realistic situations in which IRV
can give poor results.

While every voting method has cases it handles poorly, some are better
than others. One attempt to compare them is called Voter
Satisfaction Efficiency (more
details). The idea is, you run a large number of simulations and
see how different methods perform. It turns out that IRV does
very poorly here, and if voters are highly strategic IRV does even
worse than traditional plurality voting.

While I wish the voting method for us to consider were Approval (or
maybe 3-2-1 or
STAR), I do
still think IRV is better than what we have today, and I'm planning on
voting for it. One specific way in which IRV is an improvement is
that it mostly doesn't, in its failings, benefit one type of party.
This means that if we switch to IRV, and then as third-party
candidates become stronger contenders we start to run into IRV's
problems, it should be politically practical to switch to a better
system. I do think there is some risk of setting back alternative
voting systems in general by implementing an inferior version, but on
balance I think the benefit of fixing the "minor spoiler" problem is
likely larger.

Comment via: facebook
Where is human level on text prediction? (GPTs task)
by Daniel Kokotajlo — 2020-09-20T09:00:28.693Z

I look at graphs like these (From the GPT-3 paper), and I wonder where human-level is:

Gwern seems to have the answer here:

GPT-2-1.5b had a cross-entropy validation loss of ~3.3 (based on the perplexity of ~10 in Figure 4, and log2(10)=3.32). GPT-3 halved that loss to ~1.73 judging from Brown et al 2020 and using the scaling formula (2.57⋅(3.64⋅103)−0.048). For a hypothetical GPT-4, if the scaling curve continues for another 3 orders or so of compute (100–1000×) before crossing over and hitting harder diminishing returns, the cross-entropy loss will drop, using to ~1.24 (2.57⋅(3.64⋅106)−0.048).

If GPT-3 gained so much meta-learning and world knowledge by dropping its absolute loss ~50% when starting from GPT-2’s near-human level, what capabilities would another ~30% improvement over GPT-3 gain? What would a drop to ≤1, perhaps using wider context windows or recurrency, gain?

So, am I right in thinking that if someone took random internet text and fed it to me word by word and asked me to predict the next word, I'd do about as well as GPT-2 and significantly worse than GPT-3? If so, this actually lengthens my timelines a bit.
(Thanks to Alexander Lyzhov for answering this question in conversation)
My (Mis)Adventures With Algorithmic Machine Learning
by AHartNtkn — 2020-09-20T05:31:19.745Z

Introduction

This was originally posted here.

I've been researching, for quite some time, the prospect of machine learning on a wider variety of data types than normally considered; things other than tables of numbers and categories. In particular, I want to do ML for program and proof synthesis which requires, at the very least, learning the structures of trees or graphs which don't come from a differentiable domain. Normal ML algorithms can't handle these; though some recent methods, such as graph neural networks and transformers, can be adapted to this domain with some promising results. However, these methods still rely on differentiation. Is this really required? Are we forever doomed to map all our data onto a differentiable domain if we want to learn with it?

An alternative approach that has been bandied about for a while is the utilization of compression. It's not hard to find articles and talks about the relationship between compression and prediction. If you have a good predictor, then you can compress a sequence into a seed for that predictor and decompress by running said predictor. Going the other way is harder, but, broadly speaking, if you have a sequence that you want to make a prediction on and a good compressor, then whichever addition increases the compressed size the least should be considered the likeliest prediction. This approach is quite broad, applying to any information which can be represented on a computer and not requiring any assumptions whatsoever about the structure of our data beyond that. We could use this idea to, for example, fill in gaps in graphs, trees, sets of input-output pairs, etc.

It's important to understand what's actually required here. We don't actually need to compress our training data; we only need a way to estimate the change in minimal-compression-size as we add a prediction. This minimal-compression-size is called the Kolmogorov Complexity, denoted K(X). The minimal-compression-size of a program which outputs X on an input Y is called the Conditional Kolmogorov Complexity, denoted K(X|Y). The topic of Kolmogorov Complexity is quite broad, and I won't explain all its complexities here. A standard introduction is the textbook An Introduction to Kolmogorov Complexity and Its Applications by Li, Ming, Vitányi, and Paul. If we have a good method for calculating K, then we don't need to actually make use of compre...