trentmkelly/LessWrong-43k huggingface dataset:
The Irresistible Attraction of Designing Your Own Utopia
by David Udell — 2022-04-01T01:34:56.208Z

Epistemic status: Somewhat undirected and not very serious. The usual politics warning.

As Usual, the Story Begins in the Human Ancestral Environment

For >90% of human history (and further back, into pre-human primate history), Homo sapiens lived in small bands of about 50. In that social environment, it made a lot of strategic sense to spend a lot of time actively thinking about alliances and social norms in your band, so as to ally with the right people and to look impressive in debates. In that social world, your status in the eyes of your band was everything. Rise in the band's status hierarchy and gain access to a whole slew of valuable favors. Fall, and face ridicule, social isolation, restricted access to resources, and maybe exile and death alone in the wild. In other words, social status is humanity's native currency, and we are fine tuned to be good at acquiring and protecting it. One of the main ways we did that was by getting good at small-group politics. We are all descendants of those adept in ancestral politics, and so have inherited specialized cognitive machinery built for excelling in those interactions.

Here in the third millennium, people are still running those politics-obsessed neural algorithms. But our contemporary social environment is not the social environment those algorithms are expecting. In our social world, rational political ignorance is probably the winning political strategy, given how little control any of us have over any country's decision-making.

The Irresistible Attraction of Utopian Musings and RTS Faction Design

Despite being extremely confident the likelihood of this future is negligible, my brain really enjoys devoting lots of time and energy to developing and reflecting on my design for a national (or global) utopia, presumably to be promulgated once I come to absolute power. C.f. Scott Alexander:

The question “If you were a society, what kind of society would you be?” is strangely existential. Some people are bland liberal democracies. Some people are tropical island paradises. Some people are extremely efficient Singaporean city-states. But anyone at all interesting is something that has never quite existed before on Earth. Tolkien was the Elves. I don’t know much about Iain Banks, but it wouldn’t surprise me if he was the Culture.

One guy on Micras is a libertarian. He just sort of hangs around going “Yup, my count

...
If AGI were coming in a year, what should we do?
by MichaelStJules — 2022-04-01T00:41:42.200Z

Suppose AGI was very likely to arrive first around a year from now, with multiple projects close, but one a few months ahead of the others, and suppose that the AI safety community agreed that this was the case. What should our community do?

How would you answer differently for AGI in 4 months from now? 2 years from now? 5 years from now? 10 years from now? 20 years from now?

Some potential subquestions:

What technical research should we try to get the project closest to AGI to rely on? How would we get them to use it? Or, if we could build AGI first, what would we do to reduce risks?
What infrastructure, defences or other technology should we try to have ready or use?
How should we reach out to governments, and what should we try to convince them to do?
What research should we work on?
What else?

My motivations for this question are to get people to generate options for very short timelines and to get an idea of our progress so far.
Intro to hacking with the lambda calculus
by L Rudolf L — 2022-03-31T21:51:50.330Z
Link: https://www.strataoftheworld.com/2021/04/lambda-calculus.html

This is a linkpost that I'm posting in response to a request to share more of the archive of my personal blog on LessWrong. It is most useful if you want to get some intuition for the lambda calculus, or more generally for how simple axiomatisations of computation actually let you express anything.

Introduction

This post is about lambda calculus. The goal is not to do maths with it, but rather to build up definitions within it so that we can express non-trivial algorithms easily. At the end we will see a lambda calculus interpreter written in the lambda calculus, and realise that we're most of the way to Lisp.

But first, why care about lambda calculus? Consider four different systems:

The Turing machine – that is, a machine that:

works on an infinite tape of cells from which a finite set of symbols can be read and written, and always points at one of these cells;
has some set of states it can be in, some of which are termed "accepting" and one of which is the starting state; and
given a combination of current state and current symbol on the tape, always does an action consisting of three things:
writes some symbol on the tape (possibly the same that was already there),
transitions to some some state (possibly the same it is already in), and
moves one cell left or right on the tape.

The lambda calculus (λ-calculus), a formal system that has expressions that are built out of an infinite set of variable names using λ-terms (which can be thought of as anonymous functions) and applications (analogous to function application), and a few simple rules for shuffling around the symbols in these expressions.
The partial recursive functions, constructed by function composition, primitive recursion (think bounded for-loops), and minimisation (returning the first value for which a function is zero) on three basic sets of functions:

the zero functions, that take some number of arguments and return 0;
a successor function that takes a number and returns that number plus 1; and
the projection functions, defined for all natural numbers a and b such that a \geq b as taking in a arguments and returning the bth one.

Lisp, a human-friendly axiomatisation of computation that accidentally became an extremely good and long-lived programming language.

The big result in theoretical computer science is that these can all do the same thing, in the sense that if you can express a calculation in...
The lure of technocracy
by jasoncrawford — 2022-03-31T19:58:53.695Z
Link: https://rootsofprogress.org/technocracy-hypothesis

I’ve said that society was generally optimistic about progress until the early 20th century, and lost that confidence in the World Wars. By the late 20th century, from about the 1970s on, a deep skepticism and distrust of progress had come to prominence. But what happened in between?

I have a new theory about what characterized the attitude toward progress (in the US, at least) from about the 1930s through the ‘60s. It’s just a hypothesis at this stage, but it goes like this:

The 19th century was dominated by a belief in the power of human reason and its ability to advance science and technology for the betterment of life. But after World War I and the Great Depression, it got harder to believe in the rationality of humanity or in the predictability and controllability of the world.

The generation that went through these shocks, however, was not ready to give up on the idea of progress. They still wanted progress and still believed that reason could achieve it—but they worried that the masses could not be trusted to be rational, and that progress could not be left to the chaos of democracy and free markets. Instead, progress was to be achieved by a technical elite that would exercise top-down control.

The purest form of this, perhaps, found expression in early Communism, which valorized industrial production but sought to achieve it by subordinating the individual to totalitarian rule. The US was too individualistic for that—but it evolved its own flavor of the idea that I’m just starting to understand. Call it “technocracy.”

Historical evidence

Here are some snippets from my research that indicate this theme.

Walter Lippmann and the “democratic realists”

Lippmann wrote a number of books around the 1920s arguing that democracy doesn’t work, because it relies on an informed public, which he saw as impossible. Quoting from “Can Democracy Survive in the Post-Factual Age?” by Carl Bybee:

For Lippmann, given the inevitable tendency of individuals to distort what they see, coupled with the basic irrationality of humankind, the only hope for democratic government was to reinvent it. This new, more “realistic” democracy would be tempered and guided by a form of knowledge which, Lippmann believed, rose above subjectivity and politics: science.

Lippmann was part of a school of “democratic realists”, says Bybee:

The major themes sounded by Lippmann were shared by the democratic r

...
The Halting Problem and the Impossible Photocopier
by J Bostock — 2022-03-31T18:19:20.292Z

Epistemic status: the halting problem may actually still hold for quantum computers, I have no idea. I'm mostly interested in pointing out the sneakiness of assumptions.

The Halting Problem is the following: imagine some class of algorithms A. Can we have an algorithm F which takes as inputs a description of another algorithm A and an input I for that algorithm, and outputs either true, if A(I) halts or false, if A(I) doesn't.

The original solution (by Alan Turing): no by contradiction. Imagine we have a such an algorithm. Now imagine a reversing algorithm R, which halts if given false as an input, and runs forever if given true as an input. Imagine also a copier algorithm C, which duplicates the input (so C(x)=(x,x), and always halts.

Now imagine we string these algorithms together, R(F(C(x)))=G(x). What happens if we feed G into G? Well C(G)=(G,G), then F(G,G) refers to whether G halts when given the input G. So does G halt when fed G? Think about this for a moment.

Imagine G(G) halts. Then F(G,G)=true, which is passed to R, which runs forever.

Imagine G(G) doesn't halt. Then F(G,G)=false, which is passed to R, which halts.

So we have a paradox.

This works for any class of algorithm you can imagine, it's a fantastically general proof which says nothing about the nature of the algorithms at hand.

Or does it!?

Note the hidden assumption. That R and C exist. What if they don't?

For a string of bits, or a Turing machine (as the problem was originally posed) the existence of C is trivial. But for a string of qubits, it isn't. The no-cloning theorem states that "it is impossible to create an independent and identical copy of an arbitrary unknown quantum state". So in fact the standard proof of the halting problem does not apply to quantum computers.

This is possibly the sneakiest assumption I've ever seen snuck into an argument! A proof which seems general to any algorithm class falls down because you can't have a quantum photocopier!

I will repeat that I am not actually making a claim about quantum computers here. I have no idea if the Gödel's Incompleteness-based proof holds. I am merely pointing out the sneakiness of this assumption.
[Link] Training Compute-Optimal Large Language Models
by nostalgebraist — 2022-03-31T18:01:50.760Z
Link: https://arxiv.org/abs/2203.15556

New LM scaling paper from DeepMind (abs, pdf).

Abstract (my emphasis):

We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4× more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.

Brief comments on my blog here.

Presumably has implications for Bio Anchors?
Newcomb’s problem is just a standard time consistency problem
by basil.halperin — 2022-03-31T17:32:16.660Z

(Cross-posted from my blog.)

Confidence level: medium

I want to argue that Newcomb’s problem does not reveal any deep flaw in standard decision theory. There is no need to develop new decision theories to understand the problem.

I’ll explain Newcomb’s problem and expand on these points below, but here’s the punchline up front.

TLDR:

Newcomb’s problem, framed properly, simply highlights the issue of time consistency. If you can, you would beforehand commit to being the type of person who takes 1 box; but in the moment, under discretion, you want to 2-box.
That is: the answer to Newcomb depends on from which point in time the question is being asked. There’s just no right way to answer the question without specifying this.
Confusion about Newcomb’s problem comes when people describing the problem implicitly and accidentally conflate the two different possible temporal perspectives.
Newcomb’s problem is isomorphic to the classic time consistency problem in monetary policy – a problem that is well-understood and which absolutely can be formally analyzed via standard decision theory.

I emphasize that the textbook version of expected utility theory lets us see all this! There’s no need to develop new decision theories. Time consistency is an important but also well-known feature of bog-standard theory.

I. Background on Newcomb

(You can skip this section if you’re already familiar.)

Newcomb’s problem is a favorite thought experiment for philosophers of a certain bent and for philosophically-inclined decision theorists (hi). The problem is the following:

Scientists have designed a machine for predicting human decisions: it scans your brain, and predicts on average with very high accuracy your choice in the following problem.
I come to you with two different boxes:

1. In one box, I show you there is $100.
2. The second box is a mystery box.

I offer to let you choose between taking just the one mystery box, or taking both the mystery box and the $100 box.
Here’s the trick: I tell you that the mystery box either contains nothing or it contains a million dollars.

If the machine had predicted in advance that you would only take the mystery box, then I had put a million bucks in the mystery box.
But if the machine had predicted you would take both boxes, I had put zero in the mystery box.

As Robert Nozick famously put it, “To almost everyone, it is perfectly clear and obvious what...
Covid 3/31/2022: More of the Same
by Zvi — 2022-03-31T15:50:00.606Z

BA.2 became the majority strain in the United States this week, as expected. Mostly things are continuing as expected, with unsurprising news on a variety of fronts. The biggest surprise is that the second booster shot was approved. Given I expect to be asked about that every so often, I split that off into its own post.

Executive Summary

BA.2 now majority strain but for now it changes nothing.
Ultraviolet light confirmed as effective.
Lack of federal pandemic funding having real world impacts.
Second booster approved for those 50+ or who are immunocompromised.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 177k cases (unchanged) and 6,200 deaths (-12%).

Results: 189k cases (+6%) and 5,718 deaths (-18%).

Prediction for next week: 210k cases (+11%) and 5,100 deaths (-10%).

Looking back, I didn’t adjust for Colorado backfilling some deaths last week, which likely explains why there was a bigger than expected drop this week.

For next week, as BA.2 takes over I expect case growth to accelerate somewhat, but continue to not expect things to get terribly out of hand before stabilizing once again. That could still take a month or two to happen.

Deaths

Death numbers continue to decline as previous masses of cases work their way out of the ‘counts as a Covid death’ window. The blue bump is Colorado’s backfill last week.

Cases

NYC cases, make sure to use a log scale edition. The uptick isn’t not worrying but I do not expect it to be sustained for that long, and as Zeynep says previous infections should ensure that even relatively bad scenarios do not get so bad.

BA.2 outlook, wastewater in Boston ticking up from low baseline.

In UK, ZOE study claims record high case numbers, but official numbers are dropping and well below peak. Who is right?

Yes, tests are down, but how much?

How does that translate in percentage terms?

The positive test rate is slightly higher than the January peak. This implies that a somewhat smaller percentage of cases are being caught. It does not plausibly indicate that the rate of finding cases has declined by more than half.

Physical World Modeling

CDC has BA.2 becoming 55% of cases for the week ending March 29, up from 35%.

Vaccine effectiveness data for BA.2 has come in. Meet the new vaccine effectiveness, same as the old vaccine ef...
They Don’t Know About Second Booster
by Zvi — 2022-03-31T15:30:00.999Z

The FDA went ahead and did it, approving a fourth vaccine shot for those age 50 or older. This is a reference post for those wondering if it makes sense to get that second booster.

The FDA approved the second booster by bypassing some of the normal procedures. Usual never-amused suspects are not amused.

What do those meetings and ‘public discussions’ do?

Delay.
Get picky about lots of little details.
List all the reasons not to do something.
Voice objections for all to see.
Show lots of concern over trivial risks.

The Official Public Health Position is that doing all this Enhances Trust In The Process.

They are incorrect. And here is where their idea of ‘trust’ has gotten us.

Are they working hard on earning our trust back? Not exactly.

Their ‘scientific rationale’ is exactly what you think it is.

None of that means the standard drug process is necessarily bad or worthless. The process is very good at sniffing out any little potentially wrong thing or reason not to do something, and hence good at avoiding false positives. Which is sometimes what you want.

If your goal is to do a cost-benefit analysis to decide what to do, then turning things over to people who think costs and benefits do not belong in the same magisterium, and allowing them to act Deeply Concerned about things, does not seem like The Way.

The process is hopelessly biased towards delay, inaction and risk aversion. It makes sense to bypass it.

Nor does it seem like The Way to get Enhanced Trust. Yes, a few people will say ‘look, you did not hold the Official Meetings’ but actual regular people do not care. If anything, those arguments are crowding out arguments regular people would care about more.

For those shouting from the rooftops that ‘they’ would have us take infinite boosters forever, this is vindication of a sort. Yet it remains not even approved for others, let alone required.

There are good arguments against the second booster. That is fine. We can let individuals make their own choices.

Review: Should You Get a First Booster?

My answer to this continues to be yes.

The original design of the vaccine, of two shots in rapid succession, makes sense if you are trying to quickly test and deploy a vaccine in the middle of a pandemic. Pfizer and Moderna absolutely did the right thing to test that protocol given the situation.

However, there is lots of evidence that sp...
Ways to organize a search for a missing person in LA from abroad?
by Dan Reitman — 2022-03-31T14:25:47.346Z

TL/DR: My father-in-law is missing, asking for help/advice in coordinating search from abroad

This is not the thing I hoped I would post here after years of lurking, but situation is rather extreme and I try to get all the help I could get.

My father-in-law, Igor, has gone missing in LA after the stroke; because of his medical condition he is confused and unable to use his phone. He doesn’t have any friends or relatives in the US, his English is not very good (he is Ukrainian and only moved to LA last year), and he doesn’t have a place to go to – his landlord has evicted him from the apartment he rented when Igor was admitted to the hospital.

My wife and I live in Georgia (the country, not the state), and we try to organize a search for him from abroad; we can’t come to the US.

So the question is – what would be the recommended course of action to coordinate the search for a missing person from abroad? We have filed a missing person report to the police, we post to Facebook groups and made a simple website with basic information – what could be our other options?
AXRP Episode 13 - First Principles of AGI Safety with Richard Ngo
by DanielFilan — 2022-03-31T05:20:17.883Z

YouTube link

This podcast is called AXRP, pronounced axe-urp and short for the AI X-risk Research Podcast. Here, I (Daniel Filan) have conversations with researchers about their papers. We discuss the paper and hopefully get a sense of why it’s been written and how it might reduce the risk of artificial intelligence causing an existential catastrophe: that is, permanently and drastically curtailing humanity’s future potential.

How should we think about artificial general intelligence (AGI), and the risks it might pose? What constraints exist on technical solutions to the problem of aligning superhuman AI systems with human intentions? In this episode, I talk to Richard Ngo about his report analyzing AGI safety from first principles, and recent conversations he had with Eliezer Yudkowsky about the difficulty of AI alignment.

Topics we discuss:

The nature of intelligence and AGI

The nature of intelligence

AGI: what and how

Single vs collective AI minds

AGI in practice

Impact

Timing

Creation

Risks and benefits

Making AGI safe

Robustness of the agency abstraction

Pivotal acts

AGI safety concepts

Alignment

Transparency

Cooperation

Optima and selection pressures

The AI alignment research community

Updates from Yudkowsky conversation

Corrections to the community

Why others don’t join

Richard Ngo as a researcher

The world approaching AGI

Following Richard’s work

Daniel Filan:
Hello, everybody. Today, I’ll be speaking with Richard Ngo. Richard is a researcher at OpenAI, where he works on AI governance and forecasting. He also was a research engineer at DeepMind, and designed the course “AGI Safety Fundamentals”. We’ll be discussing his report, AGI Safety from First Principles, as well as his debate with Eliezer Yudkowsky about the difficulty of AI alignment. For links to what we’re discussing, you can check the description of this episode, and you can read the transcripts at axrp.net. Well, Richard, welcome to the show.

Richard Ngo:
Thanks so much for having me.

The nature of intelligence and AGI

Daniel Filan:
You wrote this report, AGI Safety from First Principles. What is AGI? What do you mean by that term?

Richard Ngo:
Fundamentally, it’s AI that can perform as well as humans, or comparably to...