trentmkelly/LessWrong-43k huggingface dataset:
[AN #121]: Forecasting transformative AI timelines using biological anchors
by Rohin Shah — 2020-10-14T17:20:04.918Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Draft report on AI timelines (Ajeya Cotra) (summarized by Rohin): Once again, we have a piece of work so large and detailed that I need a whole newsletter to summarize it! This time, it is a quantitative model for forecasting when transformative AI will happen. Note that since this is still a draft report, the numbers are in flux; the numbers below were taken when I read it and may no longer be up to date.

The overall framework

The key assumption behind this model is that if we train a neural net or other ML model that uses about as much computation as a human brain, that will likely result in transformative AI (TAI) (defined as AI that has an impact comparable to that of the industrial revolution). In other words, we anchor our estimate of the ML model’s inference computation to that of the human brain. This assumption allows us to estimate how much compute will be required to train such a model using 2020 algorithms. By incorporating a trend extrapolation of how algorithmic progress will reduce the required amount of compute, we can get a prediction of how much compute would be required for the final training run of a transformative model in any given year.

We can also get a prediction of how much compute will be available by predicting the cost of compute in a given year (which we have a decent amount of past evidence about), and predicting the maximum amount of money an actor would be willing to spend on a single training run. The probability that we can train a transformative model in year Y is then just the probability that the compute requirement for year Y is less than the compute available in year Y.

The vast majority of the report is focused on estimating the amount of compute required to train a transformative model using 2020 algorithms (where most of our uncertainty would come from); the remaining factors are estimated relatively quickly without too much detail. I’ll start with those so that you can have them as background knowledge before we delve into the real meat of the report. These are usually modeled as logistic curves in log...
What is your electronic drawing set up?
by Elizabeth — 2020-10-14T05:42:58.165Z

I'm interested in drawing on a tablet. I'll put my specific criteria below, but I'd like this to be as broadly useful as possible, so I'm going to ask the questions in general first:

What is your set up for electronic drawing? What tools do you use? What apps? What's your use case, and why are these choices the best for it? What would make you change your mind about your choices?

My specific preferences

That said, my specific goal is a tablet I use for Art (drawing and writing) that has no other distractions, to facilitate focused work. The use case leads to the following restrictions:

I don't want to be tethered to a second device, so it needs to be a generalist tablet, not just a drawing tablet.
I don't know what requirements I should be putting down for a drawing tablet, please tell me. Pen sensitivity seems important, but will I notice the difference between 8,192 and 4,096 pressure levels? How much processing power do I need to scribble?
I'm invested in the android ecosystem. Unfortunately, the only android tablets anyone recommends for drawing are from Samsung, and for complicated and stupid reasons, the writing use case rules out Samsung (this is what's killing me on my current distraction-free writing set up).
If there were a really cheap tablet that was for some reason amazing at drawing but terrible for writing, I could probably limp by with my current writing set up.
Ideally USB-C charged, will tolerate micro-USB, really would rather not use lightning or another proprietary connector as it's one more damn cable to carry around.
The Solomonoff Prior is Malign
by Mark Xu — 2020-10-14T01:33:58.440Z

This argument came to my attention from this post by Paul Christiano. I also found this clarification helpful. I found these counter-arguments stimulating and have included some discussion of them.

Very little of this content is original. My contributions consist of fleshing out arguments and constructing examples.

Thank you to Beth Barnes and Thomas Kwa for helpful discussion and comments.

What is the Solomonoff prior?

The Solomonoff prior is intended to answer the question "what is the probability of X?" for any X, where X is a finite string over some finite alphabet. The Solomonoff prior is defined by taking the set of all Turing machines (TMs) which output strings when run with no input and weighting them proportional to 2−K, where K is the description length of the TM (informally its size in bits).

The Solomonoff prior says the probability of a string is the sum over all the weights of all TMs that print that string.

One reason to care about the Solomonoff prior is that we can use it to do a form of idealized induction. If you have seen 0101 and want to predict the next bit, you can use the Solomonoff prior to get the probability of 01010 and 01011. Normalizing gives you the chances of seeing 1 versus 0, conditioned on seeing 0101. In general, any process that assigns probabilities to all strings in a consistent way can be used to do induction in this way.

This post provides more information about Solomonoff Induction.

Why is it malign?

Imagine that you wrote a programming language called python^10 that works as follows: First, it takes all alpha-numeric chars that are not in literals and checks if they're repeated 10 times sequentially. If they're not, they get deleted. If they are, they get replaced by a single copy. Second, it runs this new program through a python interpreter.

Hello world in python^10:

ppppppppprrrrrrrrrriiiiiiiiiinnnnnnnnnntttttttttt('Hello, world!')

Luckily, python has an exec function that executes literals as code. This lets us write a shorter hello world:

eeeeeeeeexxxxxxxxxxeeeeeeeeeecccccccccc("print('Hello, world!')")

It's probably easy to see that for nearly every program, the shortest way to write it in python^10 is to write it in python and run it with exec. If we didn't have exec, for sufficiently complicated programs, the shortest way to write them would be to specify an interpreter for a different language in pytho...
Toy Problem: Detective Story Alignment
by johnswentworth — 2020-10-13T21:02:51.664Z

Suppose I train some simple unsupervised topic model (e.g.

LDA

) on a bunch of books. I look through the topics it learns, and find one corresponding to detective stories. The problem: I would like to use the identified detective-story cluster to generate detective stories from GPT.

The hard part: I would like to do this in such a way that the precision of the notion of detective-stories used by the final system is not limited by the original simple model.

Here’s what that means, visually. The space of real-world books has some clusters in it:

One of those clusters is the detective-story cluster. The simple model approximates those clusters using something simple - for the sake of visualization, ellipses:

The more complex model (e.g. GPT) presumably has a much more precise approximation of the shape of the clusters:

So, we’d like to use the simple model to identify one of the clusters, but then still use the full power of the complex model to sample from that cluster.

Of course, GPT may not contain a single variable corresponding to a cluster-id, which is largely what makes the problem interesting. GPT may not internally use a notion of “cluster” at all. However, the GPT model should still contain something (approximately) isomorphic to the original cluster, since that real pattern is still in the data/environment: since there is a real cluster of "detective stories" in the data/environment itself, the GPT model should also contain that cluster, to the extent that the GPT model matches the data/environment.

In particular, the “precision not limited by original model” requirement rules out the obvious strategy of generating random samples from GPT and selecting those which the simple model labels as detective-stories. If we do that, then we’ll end up with some non-detective-stories in the output, because of shortcomings in the simple model’s notion of detective-stories. Visually, we’d be filtering based on the ellipse approximation of the cluster, which is exactly what we want to avoid.

(Note: I am intentionally not giving a full mathematical formalization of the problem. Figuring out the right formalization is part of the problem - arguably the hard part.)

Why Is This Interesting?

This is a toy model for problems like:

Representing

stable pointers to values

Producing an aligned successor AI from an aligned initial AI
Producing an AI which can improve its notion of human v

...
Technology and its side effects
by jasoncrawford — 2020-10-13T20:07:59.685Z
Link: https://rootsofprogress.org/side-effects-of-technology

Progress is messy. On the whole, over the long run, the advance of technology and industry has improved life along almost every dimension. But when you zoom in to look at each step, you find that progress is full of complications.

Some examples:

Intensive agriculture achieves high crop density (which is good because it improves land and labor productivity), but this takes fertility out of the soil faster and makes fields more susceptible to pests. To solve these problems, we then need things like artificial fertilizer, pesticides, and improved crop varieties.
Burning lots of coal provided us with warmth in our homes, with industrial processes such as iron smelting, and with motive power from steam engines. But it also caused air pollution, blackened our skies and deposited soot on everything—including our lungs. London in 1659 and Pittsburgh in 1861 were both likened to hell on earth because of the oppressive clouds of black smoke. Improving air quality has been a long process that included moving coal-burning away from human habitation, switching to cleaner-burning fuels such as gasoline and natural gas, and the introduction of electricity.
City life provided people with many opportunities for work, commerce, and socialization; but crowding people together in filthy conditions, before sewage and sanitation systems, meant an increase in contagious disease and more frequent epidemics. In the 1800s, mortality was distinctly higher in urban areas than rural ones; this persisted until the advent of improved water and sewage systems in the late 1800s and early 1900s.
Automated manufacturing in the factory system was far more productive than the previous system of home production or “cottage industry”. In that system, a weaver, for instance, would perform his craft at home, using his own loom; keep his own hours; and be paid by the piece. The factory system created a need to commute, and resulted in a loss of autonomy for workers, as they could no longer set their own hours or direct their own work. This has mostly been a permanent change, although recent decades have seen a slight reversal, as the Internet enables flexible “gig” work, lets some employees work remotely, and makes it easier to start small businesses.

Nor can we, in every instance, fall back on “revealed preferences” to argue that people actually want the new thing, since they chose it: sometimes industri...
Knowledge, manipulation, and free will
by Stuart_Armstrong — 2020-10-13T17:47:12.547Z

Thanks to Rebecca Gorman for co-developing this idea

On the 26th of September 1983, Stanislav Petrov observed the early warning satellites reporting the launch of five nuclear missiles towards the Soviet Union. He decided to disobey orders and not pass on the message to higher command, which could easily have resulted in a nuclear war (since the soviet nuclear position was "launch on warning").

Now, did Petrov have free will when he decided to save the world?

Maintaining free will when knowledge increases

I don't intend to go into the subtle philosophical debate on the nature of free will. See this post for a good reductionist account. Instead, consider the following scenarios:

The standard Petrov incident.

The standard Petrov incident, except that it is still ongoing and Petrov hasn't reached a decision yet.

The standard Petrov incident, after it was over, except that we don't yet know what his final decision was.

The standard Petrov incident, except that we know that, if Petrov had had eggs that morning (instead of porridge[1]), he would have made a different decision.

The same as scenario 4., except that some entity deliberately gave Petrov porridge that morning, aiming to determine his decision.

The standard Petrov incident, except that a guy with a gun held Petrov hostage and forced him not to pass on the report.

There is an interesting contrast between scenarios 1, 2, and 3. Clearly, 1 and 3 only differ in our knowledge of the incident. It does not seem that Petrov's free will should depend on the degree of knowledge of some other person.

Scenarios 1 and 2 only differ in time: in one case the decision is made, in the second it is yet to be made. If we say that Petrov has free will, whatever that is, in scenario 2, then it seems that in scenario 1, we have to say that he "had" free will. So whatever our feeling on free will, it seems that knowing the outcome doesn't change whether there was free will or not.

That intuition is challenged by scenario 4. It's one thing to know that Petrov's decision was deterministic (or deterministic-stochastic if there's a true random element to it). It's another to know the specific causes of the decision.

And it's yet another thing if the specific causes have been influence to manipulate the outcome, as in scenario 5. Again, all we have done here is add knowledge - we know the causes of Petrov's decision, and we know that...
What should experienced rationalists know?
by sapphire — 2020-10-13T17:32:32.388Z

The obvious answer is 'the sequences' but imo that is neither necessary nor sufficient. The Sequences are valuable but they are quite old at this point. They also run to over a million words (though Rationality AtZ is only ~600k). Here is a list of core skills and ideas:

1 - Rationality Techniques

Ideally, an experienced rationalist would have experience with most of the CFAR manual. Anyone trying to learn this material needs to actually try the techniques; theoretical knowledge is not enough. If I had to make a shorter list of techniques I would include:

Double Crux / Internal DC
Five-minute Timers
Trigger Action Plans
Bucket Errors
Goal/Aversion Factoring
Gears Level Understanding
Negative Visualisation / Murphy-Jitsu
Focusing

2 - AI Risk: Superintelligence

The rationality community was founded to help solve AI risk. Superintelligence gives an updated and complete version of the 'classic' argument for AI-risk. Superintelligence does not make as many strong claims about takeoff as Elizer's early writings. This seems useful given that positions closer to Paul Christiano's seem to be gaining prominence. I think the 'classic' arguments are still very much worth understanding. On the other hand, Superintelligence is ~125K words and not easy reading.

I think many readers can skip the first few chapters. The core argument is in chapters five through fourteen.

5. Decisive strategic advantage

6. Cognitive superpowers

7. The superintelligent will

8. Is the default outcome doom?

9. The control problem

10. Oracles, genies, sovereigns, tools

11. Multipolar scenarios

12. Acquiring values

13. Choosing the criteria for choosing

14. The strategic picture

3 - Cognitive Biases: Thinking Fast and Slow

Priming is the first research area discussed in depth in TF&S. Priming seems to be almost entirely BS. I would suggest skipping the chapter on priming and remembering the discussion of the 'hot hand fallacy' seems incorrect. Another potential downside is the length (~175K words). However, I don't think there is a better source overall. Many of the concepts in TF&S remain fundamental. The writing is also quite good and the historical value is extremely high. Here is a quick review from 2016.

4 - Statistics

It is hard to be an informed rationalist without a basic understanding of Bayesian thinking. You need to understand frequentist statistics to evaluate a lot of relev...
A prior for technological discontinuities
by NunoSempere — 2020-10-13T16:51:32.572Z

Introduction

I looked at 50 technologies taken from a Wikipedia list History of technology, which I expect to provide a mostly random list of technologies. Of these 50 technologies, I think that 19 have a discontinuity, 13 might have one, and 18 probably don't. Of these, I'd call 12 "big" discontinuities, for an initial probability estimate of 12/50=24% I provide other estimates in the "More elaborate models for computing the base rate of big discontinuities."

Unlike some previous work by AI Impacts (or, for that matter, by myself),
I am able to produce something which looks like a prior because I consider a broad bag of different technologies, and then ask which proportion have discontinuities. Previous approaches have specifically looked for discontinuities and found examples, thereby not being able to estimate their prevalence.

The broad bag of technologies I draw from was produced by Wikipedia editors who followed their own designs. They most likely weren't thinking in terms of discontinuities, and won't have selected for them. However, these editors might still have been subject to availability bias, Anglicism bias, etc. This might make the dataset mildly imperfect, that is, not completely representative of all possible technologies, but I’d say that most likely it’s still good enough.

Furthermore, I didn't limit myself to discontinuities which are easily quantifiable or for which data is relatively easy to gather; instead I quickly familiarized myself with each technology in my list, mostly by reading the Wikipedia entry, and used my best judgement as to whether there was a discontinuity. This method is less rigorous than previous work, but doesn't fall prey to Goodhart's law: I want a prior for all discontinuities, not only for the quantifiable ones, or for the ones for which there is numerical data.

However, this method does give greater weight to my own subjective judgment. In particular, I suspect that I, being a person with an interest in technological discontinuities, might produce a higher rate of false positives. One could dilute this effect by pooling many people's assessments, like in Assessing Kurzweil’s predictions for 2019.

All data is freely available here. While gathering it, I came across some somewhat interesting anecdotes, some of which are gathered in this shortform.

Many thanks to Misha Yagudin, Gavin Leech and Jaime Sevilla for feedback...
A study on depression
by vlad.proex — 2020-10-13T15:43:26.417Z

Content note: depression, suicide.

A friend I was discussing depression with linked me an article, "Most anguish isn't an illness but an evolved response to anxiety". It turns out it's a condensed version of a recent paper by Syme and Hagen, two biological anthropologists. The basic idea is that depression, anxiety and PTSD are not really disorders, but evolutionary adaptations, so we shouldn't try to cure them (though we could try to remove their causes).

I have analysed the article and paper at length because I wanted to give my friend a detailed criticism, and because it ties in nicely with my having recently updated my beliefs about psychiatry and starting to question some inherited anti-psychiatry memes.

Moreover, there is a whole class of articles and movements out there who say that "[insert mental condition] is not an illness and we should stop medicalizing it already". To the extent that they use similar arguments, my analysis should hopefully generalize.

Depression is not real. Is it?

The main problem I have with this work is that the authors are simultaneously denying the concept of depression and applying it in their analysis.

The article starts by saying that viewing depression and anxiety as illnesses is a "colossal error". Instead, they are adaptations designed by natural selection and activated by environmental factors, which the authors mainly identify with social conflict. They want to offer arguments against the "disease model" of depression and anxiety.

In the paper, they describe some well-known limitations of psychiatry. They attack the chemical imbalance theory of depression; point out that treating the symptoms is not the same as removing the causes; that common antidepressants have limited efficacy, though they are more effective than placebo; and that psychopharmaceuticals have aversive side effects.

In Section 5, the authors complain of the DSM's "stranglehold on research and policy". Unlike scientific taxonomies that cut reality at the joints, such as the periodic table of elements, the DSM's classification system is based on clusters of symptoms and is basically a mess. The DSM has too many diagnoses, is vulnerable to markets and fads, shows low validity and tends to over-diagnose in healthy populations.

I think most of these criticisms are justified, but the...
Does anyone worry about A.I. forums like this where they reinforce each other’s biases/ are led by big tech?
by misabella16 — 2020-10-13T15:14:07.699Z

I came across this programme for A.I. https://inspired-minds.co.uk/inspired-ai-events/ mostly being run by tech companies and their employees and it worries me that they are making A.I. and the fast development/deployment of A.I. less worrisome by influencing people. I can circle back with their points if I get in and see their presentations but the fact that it is mostly Microsoft and the like dominating the conversation instead of academics or people I suppose less focused on a mentality of “move fast and break things” when they are breaking people.