trentmkelly/LessWrong-43k huggingface dataset:
What's Your Cognitive Algorithm?
by Raemon — 2020-06-18T22:16:39.104Z

Epistemic Status: I'm neither a neuroscientist nor an ML researcher, but am trying to figure out "what kinds of human thought are actually possible to replicate on silicon right now?".

Here's my best guess of how human cognition works. Please tear it apart!

When I looked at GPT-2 last year, I thought: "Huh, when I look at my own thought process... I could summarize most of what I'm doing as: 'predict the next thing I'd say using crude concept association, and then say it.'"

Meanwhile, Jeff Hawkins says "Every part of the neocortex is running the same algorithm", and it's looking like maybe brains aren't doing that complicated a set of things.

Am I just GPT-2?

This was an obvious question to ask, but I haven't seen anyone write it up the question in detail.

I asked around. One mathematician friend said "I agree most people are doing GPT-style thinking where they regurgitate and recombine concepts from their neighbors. But, you can't get civilization from just that. Some people need to have model-based thinking."

Another mathematician friend agreed and added: "Young math students who try to prove theorems often do it GPT-style – they ramble their way through a bunch of math buzzwords and try to assemble them without understanding the structure of how they fit together. But, actual math proofs require clear understanding. You can't just 'predict the next word'"

I agree there is something additional going on here that gets you to formal math proofs and building skyscrapers. But... I don't think it's all that much more.

This post has three parts:

Lay out the cognitive algorithm I personally seem to be following
Outline how I think that algorithm developed
Work through some examples of how I do "advanced thinking" (i.e. the sort of thinking that might literally advance the sum of human knowledge), and doublecheck if there are any surprising elements

My algorithm, as I understand it

Even when I'm developing novel concepts, or thinking through effortful procedures, most of my thinking following the same basic algorithm of:

A. Find the Next "Good Enough" Thought

My subconscious finds some nearby concepts that are associated with my previous thought-chunk
If a "good enough" thought appears, think that thought, and then repeat. ("good enough" means "I feel optimistic about the next thought in the chain leading somewhere useful")
If a "not obviously good enough" thought

...
If AI is based on GPT, how to ensure its safety?
by avturchin — 2020-06-18T20:33:50.774Z

Imagine that an advance robot is built, which is uses GPT-7 as its brain. It takes all previous states of the world and predicts the next step. If a previous state of the world includes a command, like "bring me a cup of coffee", it predicts that it should bring coffee and also predicts all needed movements of robot's limbs. GPT-7 is trained on a large massive of human and other robots data, it has 100 trillions parameters and completely opaque. Its creators have hired you to make the robot safer, but do not allow to destroy it.
Sunday June 21st – talks by Abram Demski, alkjash, orthonormal, eukaryote, Vaniver
by Ben Pace — 2020-06-18T20:10:38.978Z

This Sunday at 11am (PDT), we're running a session of four "lightning talks" by curated LessWrong authors.

Each talk will be 3-5 minutes followed by discussion. Afterwards, we'll have a hangout in breakout rooms. The talks will be short and focus on presenting one core idea well, rather than rushing through a lot of content.
We want this to give top LessWrong writers an interesting space to discuss their ideas, and have more fruitful collaboration between users. Think of it like a cross between an academic colloquium and some friends chatting by a whiteboard.

Please RVSP here for the event here, so we know how many people are coming.

Speakers

Abram Demski [Talk transcript]

Curated posts: An Orthodox Case Against Utility Functions, Toward a New Technical Explanation of Technical Explanation), ...

alkjash [Talk transcript]

Curated posts: Babble, Hammers and Nails, Hammertime Final Exam

orthonormal [Talk transcript]

Curated posts: The Loudest Alarm Is Probably False

eukaryote [Talk transcript]

Curated posts: The funnel of human experience, Spaghetti Towers, Naked mole-rats: A case study in biological weirdness

Vaniver [Talk transcript]

Curated posts: Steelmanning divination, Public Positions and Private Guts

Details

When? Sunday June 21, at 11am, Pacific Daylight Time

Where? On this zoom link [now removed].

Please RVSP here for the event here, so we know how many people are coming.
Covid-19 6/18: The Virus Goes South
by Zvi — 2020-06-18T20:10:00.820Z

Last week: Covid-19 6/11: Bracing For a Second Wave

The verdicts are in. It is not an unexpected one. Unless conditions change, the path now seems set.

The Northeast is winning its battle. Unless something unexpected happens, they will get things mostly under control, at least for the remainder of the summer.

The South is losing, along with Arizona. They are likely going to get worse until herd immunity saves them. Note that the bulk of the West’s surge in cases is now Arizona.

The state capacity to do otherwise seems to be gone.

The rest of the West, and the Midwest, could go either way.

Let’s run the numbers.

Positive Tests by Region

Date
WEST
MIDWEST
SOUTH
NE ex-NY
NY

Mar 19-Mar 25
5744
6293
7933
8354
28429

Mar 26-Apr 1
15684
20337
24224
34391
52901

Apr 2-8
19455
31148
39618
56772
65604

Apr 9-15
16291
29267
35570
61921
64463

Apr 16-22
20065
34130
33932
64669
43437

Apr 23-29
20263
42343
33773
62189
42475

Apr 30-May 6
23216
49205
37880
51693
24287

May 7-May 13
22419
43256
37591
40209
16683

May 14-May 20
22725
42762
40343
39273
13709

May 21-May 27
23979
39418
42977
26434
10595

May 28-June 3
32200
31504
50039
24250
9120

June 4-June 10
35487
24674
55731
16622
6071

June 4-June 10
41976
22510
75787
12905
4986

Deaths by Region, which are lagged by several weeks:

Date
WEST
MIDWEST
SOUTH
NE ex-NY
NY

Mar 19-Mar 25
138
104
144
116
278

Mar 26-Apr 1
380
615
572
606
1656

Apr 2-8
707
1454
1309
2115
4327

Apr 9-15
890
2195
1596
3577
5318

Apr 16-22
1033
2343
1727
5147
3716

Apr 23-29
1128
2588
1685
4722
2713

Apr 30-May 6
1012
2413
1747
4908
2582

May 7-May 13
1082
2288
1597
3911
1416

Apr 23-29
1090
2060
1442
3578
963

Apr 30-May 6
775
1723
1290
2341
667

May 28-June 3
875
1666
1387
2121
436

June 4-June 10
743
1297
1230
1611
325

June 11-June 17
778
1040
1207
1270
225

Positive Test Percentages:

Date
USA tests
Positive %
NY tests
Positive %

Mar 19-Mar 25
347577
16.2%
88,882
32.0%

Mar 26-Apr 1
728474
20.2%
117,401
45.1%

Apr 2-8
1,064,225
19.8%
144,273
45.5%

Apr 9-15
1,026,741
20.4%
160,859
40.1%

Apr 16-22
1,235,393
16.1%
143,970
30.2%

Apr 23-29
1,552,560
13.0%
202,499
21.0%

Apr 30-May 6
1,759,548
10.6%
183,446
13.2%

May 7-May 13
2,153,748
7.5%
202,980
8.2%

May 14-May 20
2,643,333
6.0%
246,929
5.6%

May 21-May 27
2,584,265
5.7%
305,708
3.5%

May 28-June 3
3,022,469
5.1%
417,929
2.2%

...
[AN #104]: The perils of inaccessible information, and what we can learn about AI alignment from COVID
by Rohin Shah — 2020-06-18T17:10:02.641Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world

HIGHLIGHTS

Inaccessible information (Paul Christiano) (summarized by Rohin): One way to think about the problem of AI alignment is that we only know how to train models on information that is accessible to us, but we want models that leverage inaccessible information.

Information is accessible if it can be checked directly, or if an ML model would successfully transfer to provide the information when trained on some other accessible information. (An example of the latter would be if we trained a system to predict what happens in a day, and it successfully transfers to predicting what happens in a month.) Otherwise, the information is inaccessible: for example, “what Alice is thinking” is (at least currently) inaccessible, while “what Alice will say” is accessible. The post has several other examples.

Note that while an ML model may not directly say exactly what Alice is thinking, if we train it to predict what Alice will say, it will probably have some internal model of what Alice is thinking, since that is useful for predicting what Alice will say. It is nonetheless inaccessible because there’s no obvious way of extracting this information from the model. While we could train the model to also output “what Alice is thinking”, this would have to be training for “a consistent and plausible answer to what Alice is thinking”, since we don’t have the ground truth answer. This could incentivize bad policies that figure out what we would most believe, rather than reporting the truth.

The argument for risk is then as follows: we care about inaccessible information (e.g. we care about what people actually experience, rather than what they say they experience) but can’t easily make AI systems that optimize for it. However, AI systems will be able to infer and use inaccessible information, and would outcompete ones that don’t. AI systems will be able to plan using such inaccessible information for at least some goals. Then, the AI systems that plan using the inaccessible information could eventually control most resources. Key quote: “The key asymmetry working against us is that optimizing flourishing appears to require a particular quantity to be accessible, while danger just ...
[Personal Experiment] One Year without Junk Media: Six-Month Update
by lsusr — 2020-06-18T16:34:14.484Z

The mediums of the media I consume affect me more than the content so six months ago I committed to one year without junk media.

Here is a list of the biggest changes I've experienced over the last six months.

Sleep

Junk media gives me insomnia. My insomnia always goes away within 1 week of abstinence. The problem has not returned during 6 months of abstinence.

Even without insomnia, I tend to go to sleep around 2-3 am and wake up around 11 am. I predicted (incorrectly) that long-term abstinence would bring my circadian rhythm closer in line with the day night cycle. This has not happened.

Education

Before this experiment it took lots of willpower to study things like physics. I have a longer attention span now. I relax by training skills and learning things.

I had to stop studying quantum field theory because it has become too addictive. This vice appears limited to theoretical physics and junk media. No other subjects have triggered such intensity of addictive behavior.

Theoretical physics aside, I like getting smarter by default.

Self-Care

I have a garden. I walk more. I write a lot. I cook good food by default. Though self-employed, my productivity at work remains unchanged.

Socialization

Abstaining from junk media seems to increase my socialization from "almost none" to "a little bit" but it's hard to tell due to the confounding effects of COVID-19. Socializing makes me happier.

Cleaning

When I begin abstaining from junk media I usually get a small bump to my tidiness. I predicted (incorrectly) this effect would become more pronounced over time. Instead, my tidiness returns to baseline levels as I acclimate.

Stress

The common thread is lower overall stress. I have lower overall sympathetic nervous system activation. This could be the root cause of my lengthened attention span and reduced insomnia.
Likelihood of hyperexistential catastrophe from a bug?
by Anirandis — 2020-06-18T16:23:41.608Z

I've been reading a fair bit about "worse than death" scenarios from AGI (e.g. posts like this), and the intensities and probabilities of them. I've generally been under the impression that the worst-case scenarios have extremely low probabilities (i.e. would require some form of negative miracle to occur) and can be considered a form of Pascal's mugging.

Recently, however, I came across this post on OpenAI's blog. The blog post notes the following:

Bugs can optimize for bad behavior

One of our code refactors introduced a bug which flipped the sign of the reward. Flipping the reward would usually produce incoherent text, but the same bug also flipped the sign of the KL penalty. The result was a model which optimized for negative sentiment while preserving natural language. Since our instructions told humans to give very low ratings to continuations with sexually explicit text, the model quickly learned to output only content of this form. This bug was remarkable since the result was not gibberish but maximally bad output.

This seems to be the exact type of issue that could cause a hyperexistential catastrophe. With this in mind, can we really consider the probability of this sort of scenario to be very small (as was previously believed)? Do we have a reason to believe that this is still highly unlikely to happen with an AGI? If not, would that suggest that current alignment work is net-negative in expectation?
What are some good public contribution opportunities? (100$ bounty)
by Gurkenglas — 2020-06-18T14:47:51.661Z

I haven't gotten into MSFP or the like, but I like helping with the world's plot anyway. I've won 25$ in the Oracle contest, so I've told Stuart to put it towards further public contribution opportunities such as contests. He suggested that I do a small contest myself, for the 25$ and another 25$ from him.

So, selfish as I am, in one month, the 50$ go to the highest-karma answer, and another 50$ from me go to the answer that actually gets me to contribute. (In case of ambiguity on the latter, I try to award the cause of my best contribution. AF karma would be a good measure, except maybe the opportunities aren't all on AF...)

Examples of where I'm likely to contribute can be found via: https://www.alignmentforum.org/users/gurkenglas

Note how 25$ of prize money turn out worth more than 25$ in your pocket :)

Edit: Contest is over, contestant wins 50 dollars by walkover.
Image GPT
by Daniel Kokotajlo — 2020-06-18T11:41:21.198Z
Link: https://openai.com/blog/image-gpt/

My hot take:

Not too surprising to me, considering what GPT-3 could do. However there were some people (and some small probability mass remaining in myself) saying that even GPT-3 wasn't doing any sort of reasoning, didn't have any sort of substantial understanding of the world, etc. Well, this is another nail in the coffin of that idea, in my opinion. Whatever this architecture is doing on the inside, it seems to be pretty capable and general.

I don't think this architecture will scale to AGI by itself. But the dramatic success of this architecture is evidence that there are other architectures, not too far away in search space, that exhibit similar computational efficiency and scales-with-more-compute properties, that are useful for more different kinds of tasks.
$1,000 Bounty for Pro-BLM Policy Analysis
by Taymon Beal — 2020-06-18T01:48:52.725Z

I want to know what Americans with no particular relevant expertise should do in order to achieve the policy goals of the Black Lives Matter movement in the United States. I'm offering a $1,000 bounty (payable directly to you or to a charity of your choice) to whoever provides the analysis on this question that I find most informative in answering this question.

For any given proposed recommended action, I want to know *which* policy goals it will help achieve, *how* it will help achieve them, and *why* it should be expected to do so—bearing in mind that most political interventions do not achieve their intended outcomes, so the burden is on the proponents of any given intervention to explain why it is worthwhile.

The analysis to beat is Chloe Cockburn's from the Open Philanthropy Project (part 1; part 2). If nobody offers an analysis that I find more informative than that one, the bounty will not be awarded. (Note, though, that I'm not expecting anything nearly as long or comprehensive as that document! If you offer one single recommendation, and successfully answer the above questions about it, that's better than a long list of things without a clear bottom line or clear justifications.)

The deadline is Sunday, July 5. I will judge the submissions in my sole discretion. I may additionally award smaller bounties to non-winners who nonetheless end up influencing my thoughts on what should be done, or on how to judge the submissions. Please don't hesitate to submit something just because you don't think it's worth $1,000.

Submissions may be posted publicly in the comments of this post, or sent to me via private message. I'm also cross-posting this to Facebook and Tumblr.
Bathing Machines and the Lindy Effect
by DirectedEvolution — 2020-06-17T21:44:46.931Z

From Wikipedia:

The Lindy effect is a theory that the future life expectancy of some non-perishable things like a technology or an idea is proportional to their current age, so that every additional period of survival implies a longer remaining life expectancy.

There are a some shallow criticisms you could make of it as a predictive tool. How do you define the thing? What's the proportion of age to life expectancy? How often do we update it?

But these are all problems of refining a loose claim into an unambiguous prediction. How would we evaluate whether this heuristic is a useful predictive tool?

Let's imagine that we're updating the Lindy effect predictions every year.

Bathing machines were roofed and walled wooden carts rolled to the beach to allow swimmers to wash and change in privacy. The earliest evidence of them is an engraving from 1735, and they were almost entirely out of use by the early 1920s. Their technological lifespan is around 185 years.

Let's be generous to the Lindy effect, and say that it needs to give a lifespan estimate that's within 25% of the true value to count as correct. In this case, it needed to predict that bathing machines would become obsolete sometime between 1874-1966.

People often use the Lindy Effect to predict that at any given time, we're halfway through the lifespan of anything that currently exists. Using that value, it generated correct predictions between 1805 and 1851, about 25% of the lifespan of the bathing machine. So across the lifetime of the bathing machine, the Lindy Effect was usually wrong.

To take this a step further, I wrote a Python script that finds the most successful Lindy Effect proportion. The code is in the comments. Here are my results.

The most successful historical Lindy Effect predictions, using the 25% confidence interval, estimate that a thing will be around for 125% of its age in any given year.
It will be right only 40% of the time throughout the lifespan of the thing.
It will fail the year that the thing becomes obsolete.

Estimating that the bathing machine would be around for 125% of its present age would therefore have worked from 1846-1920.

What are the takeaways? Even interpreted very generously, the Lindy Effect is usually wrong. It's most successful when it estimates a thing has "one more phase of life" left; analogous to entering retirement. It&apos...
God and Moses have a chat
by Yitz — 2020-06-17T18:34:42.809Z

(This is an exploration through parable of a theological issue I’ve been thinking about lately, and is not meant to accurately reflect any historical personalities. Inspired in part by Scott Alexander’s excellent Answer to Job. Comments/criticism are welcome.)

(Moses is wandering in the mountains while his sheep graze, and suddenly spots a burning bush)
Moses: Strange... why is that bush on fire? It’s a cool day in June, this is a normal, non-fire-spewing mountain in the middle of nowhere, no other humans around for miles, and there hasn’t been any lightning in weeks. What could be the cause of this?
(Comes closer to examine the burning bush)
What the heck? That bush is on fire....but it’s not being burnt! It doesn’t look hydrated enough to withstand such heat; How is this possible?

(A voice rings out powerfully from the heavens)
God: MOSES, MOSES

Moses (jumps, startled): AAAHHH!!! What the...how did...who’s there?

God: IT IS I, GOD

Moses: No, seriously, though. Who are you, where are you, and how do you know my name? Please come out from wherever you’re hiding, you almost gave me a heart attack; I’m like 80 years old, you know.

God: I WAS BEING SERIOUS. I AM LITERALLY GOD, THE CREATOR OF THE UNIVERSE AND ALL THAT JAZZ. I’M ALSO LITERALLY EVERYWHERE, BUT YOU CAN CONSIDER ME TO BE TALKING FROM THAT BURNING BUSH IF IT MAKES YOU FEEL BETTER. NOW TAKE OFF YOUR SHOES AND LISTEN CLOSELY, BECAUSE I HAVE CHOSEN YOU AS THE LEADER OF THE—

Moses: Wait a minute, back up just a little there. You’re what?

God: I AM GOD, THE CREATOR OF THE UNIVERSE.

Moses: No you’re not.

God: WHAT?

Moses: You can’t possibly be God. Or to be more accurate, you might be the actual creator of the universe, but I find that to be an astronomically unlikely possibility. I have absolutely no good reason to think you are God. Pharaoh, my adopted grandfather and supreme ruler of Egypt, claims to be God, but obviously isn’t. Considering there have been literally thousands of delusional people throughout history who have claimed to be the one and only, mutually exclusive God, I see no reason to believe to believe you more than any of them, and I definitely don’t believe them. Why should I take your claim to Divinity any more seriously than Grandpa’s?

God: WELL, DOES THIS MAGICAL BURNING BUSH COUNT FOR ANYTHING? ALSO, WHAT ABOUT MY SUPER DIVINE SOUNDING BOOMING VOICE COMING FROM EVERYWHERE AT ONCE?

Moses: Maybe yo

...
High-School Algebra for Data Structures
by johnswentworth — 2020-06-17T18:09:24.550Z

A surprising amount of day-to-day software engineering can be expressed as solving equations like:

… finding the solution (x = “Alice”, y = “Liddell”), and then substituting that solution into some other expression.

For instance, suppose we have this data:

We want to list the people who’ve lived in each state. We first write an equation:

We interpret lists as multiple allowed values, so this equation has four solutions:

(name = ‘john’, state = ‘CA’)
(name = ‘john’, state = ‘CT’)
(name = ‘allan’, state = ‘CA’)
(name = ‘allan’, state = ‘WA’)

To list people who live in each state, we then substitute those solutions into an expression like

Again, lists are interpreted as multiple allowed values, so when we substitute our four solutions into this expression, the maximally-compressed result is:

This is primarily useful for transforming data from one format to another. Typical use-cases include collecting data from an external API and transforming it into the schema used in our database, or transforming data from one API into the format needed by another API. Typically, these sort of tasks are conceptually simple, but they get very tedious and messy when mapping a large number of fields between deeply-nested data formats. Personally, I originally worked out these ideas while at

LoanSnap

, a mortgage startup needing to map thousands of fields between dozens of different data formats; you can imagine how messy that would get.

What’s the advantage of writing transformations as equations, rather than writing out “x = data[‘person’][‘firstName’]; y = data[‘person’][‘lastName’]; …”?

First, when we write data transformations as equations, the structure of our code matches the structure of our data - indeed, our “code” is essentially a template whose structure matches the data. For our names-and-states example, our code would consist of the two data structures

The left is a template matching the structure of our incoming data, and the right is a template matching the structure of our outgoing data. This makes the code much more readable and maintainable when dealing with large, deeply-nested data structures.

Second, this is a declarative (rather than imperative) method for specifying data transformations, so it comes with the usual benefits of declarativity. In particular, a library for solving data-structure-equations can handle missing data and other errors in standard ways without havi...