trentmkelly/LessWrong-43k huggingface dataset:
Science for the Possible World
by Zechen Zhang — 2022-05-23T14:01:31.715Z

In the foreseeable future, it is expected that the majority of consciousness in the universe will spend their time in the digital world. I argue that in order to unlock the "goodness" of the digital world, we need to focus on a new kind of science--the science for the possible world (SPW). The essential idea is that we are creating a new set of laws of nature (LoN) for a possible digital world, thus it requires a new kind of methodology--SPW to study the LoN for the possible world. In the following, I will try to sketch some reasons for taking this worldview and what SPW would look like.

Why Digital?

The hype of creating metaverse, AGI, and decentralized social systems based on crypto all points toward a more digital world in the future. There is enough economic incentive to make a hybrid world of digital and physical reality. It is not crazy anymore to think about digital minds (whether they are self-conscious AGI or transhumans uploaded to the cloud) floating around in the universe, and we want to make sure that this world is at least as interesting and flourishing as our current world.

The exciting thing about the metaverse, or the digital world is not that we get to live and interact with each other exactly as we do in our physical world (eg. basically what meta aims to do), but that we have the power, for the first time, to create a possible world out of our own creativity and hopefulness. This is more than discovering the new land of America, and more than going to the moon or Mars, as in the digital world we are not limited by the laws of physics, but only limited by humanity's imagination to create a coherent world of beauty and designed for flourishment. Yes, we have a once-in-a-civilization's opportunity to impersonate an imperfect God and literally make rules of our own in the possible world. It is not an easy job indeed; I propose that we need a new kind of science for the possible world.

Science for the Possible World

We have relied completely on the continuing success of modern science in the past 400 years to make tremendous progress in human civilizations, thanks to one central principle of the scientific paradigm--throwing all theories against experiments. The falsifiability principle is relentless: it does not care about the mathematical rigor, consequential richness, or the artistic beauty of your theory (supersymmetry a...
Monkeypox: explaining the jump to Europe
by DirectedEvolution — 2022-05-23T09:53:16.467Z

What's up with the recent monkeypox cases? Why are we seeing a monkeypox outbreak now just as COVID-19 vigilance is winding down? Why didn't we see more monkeypox outbreaks 10 or 20 years ago?

Tl;dr: Waning smallpox immunity has allowed monkeypox to spread steadily since the 1990s. The disease recently returned to Nigeria, just as Nigeria started getting a lot more tourism. People are traveling more post-pandemic. Most Nigerian vacationers go home to their families, but this Nigerian vacationer went to, or at least knows people who went to, a couple of huge pride raves.

This virus has boring, known genetics. So, unless we start offering a program of smallpox vaccination in Africa, this is probably the first of a series of monkeypox outbreaks. But it's not the start of a pandemic.

First, we are starting to get data falsifying the hypothesis that this is a new, more infectious variant. Dr. Grant McFadden, director of the Biodesign Center for Immunotherapy, Vaccines and Virotherapy at Arizona State University, was reported by NBC as describing the early draft of the outbreak monkeypox virus's genetics as looking pretty run-of-the-mill.

This is reassuring, but it doesn't explain why we're seeing such a big monkeypox outbreak now, or why we've seen a quiet but persistent drumbeat of non-African cases in the last few years.

So let's consider how it is that conditions have become conducive to monkeypox jumping from Africa to other continents. Monkeypox as we known it transmits relatively slowly, has been studied for decades, and the USA has an 85 million dose stockpile of 85% effective vaccine and some possible antiviral treatments.

Bunge et al. (2022) discuss the epidemiology of monkeypox. Their paper was well-timed, published just a couple months before our current multi-continent outbreak. I'll be referring to it frequently here.

Waning vaccination

Ah, the good old days. Back in 1967, 80% of the population in every country was vaccinated against smallpox, and consequently against monkeypox as well. However, maintaining such a mass vaccination campaign is expensive, and the vaccine causes smallpox in very rare cases. The producer of smallpox vaccine stopped production in 1982, shortly after the disease was declared eradicated. For all these reasons, very few people, relatively speaking, have received smallpox vaccine since then.

Smallpox vaccine efficacy also declin...
AXRP Episode 15 - Natural Abstractions with John Wentworth
by DanielFilan — 2022-05-23T05:40:19.293Z

YouTube link

Why does anybody care about natural abstractions? Do they somehow relate to math, or value learning? How do E. coli bacteria find sources of sugar? All these questions and more will be answered in this interview with John Wentworth, where we talk about his research plan of understanding agency via natural abstractions.

Topics we discuss:

Agency in E. coli

Agency in financial markets

Inferring agency in real-world systems

Selection theorems

(Natural) abstractions

Information at a distance

Why the natural abstraction hypothesis matters

Unnatural abstractions in humans

Probability, determinism, and abstraction

Whence probabilities in deterministic universes?

Abstraction and maximum entropy distributions

Natural abstractions and impact

Learning human values

The shape of the research landscape

Following John’s work

Daniel Filan:
Hello, everybody. Today I’ll be speaking with John Wentworth, an independent AI alignment researcher who focuses on formalizing abstraction. For links to what we’re discussing, you can check the description of this episode and you can read the transcripts at axrp.net. Well, welcome to AXRP, John.

John Wentworth:
Thank you. Thank you to our live studio audience for showing up today.

Agency in E. coli

Daniel Filan:
So I guess the first thing I’d like to ask is I see you as being interested in resolving confusions around agency, or things that we don’t understand about agency. So what don’t we understand about agency?

John Wentworth:
Whew. All right. Well, let’s start chronologically from where I started. I started out, first, in two directions in parallel. One of them was in biology, like systems biology and, to some extent, synthetic biology, looking at E. coli. Biologists always describe E. coli as collecting information from their environment and using that to keep an internal model of what’s going on with the world, and then making decisions based on that in order to achieve good things.

John Wentworth:
So they’re using these very agency-loaded intuitions to understand what’s going on with the E. coli, but the actual models they have are just these simple dynamical systems, occasionally with some feedback loops in there. They don’t have a way to take the low-level dynamics of the E. coli and back out these agency primitives that they’re talking about, like goals and world models and stuff.

Dani...
PSA: The Sequences don't need to be read in sequence
by kave — 2022-05-23T02:53:41.957Z

This week, I hung out with the LessWrong team while they talked to relatively new users. New users often had a vague intention to read Eliezer's original Sequences, but were blocked on the size of the project. They thought the Sequences would only work in, well, sequence.

I just polled eight people (including me) who have read the Sequences whether they only work in sequence.

7 people said they work out of sequence (though 2 noted that it might be better to read a given sequence in order)
1 person said they thought it was necessary to read any given sequence in order, but it didn't matter if you read one sequence (e.g. A Human's Guide to Words) before or after another (e.g. Mysterious Answers to Mysterious Questions)

A typical sequence post has many links to other sequence posts. But these are mostly context and elaboration. The posts tend to work well standalone.

Here are three posts you might get started with:

Fake Explanations. The first post in the Mysterious Answers to Mysterious Questions sequence, described by Eliezer as "probably the most important core sequence in Less Wrong".
Leave a Line of Retreat. Letting go of a belief that's important to you is hard. Particularly if you think stuff you care about depends on it (e.g. if you think being good depends on moral realism, it could be hard to reexamine your belief in moral realism). This post describes the phenomenon, and gives advice for dealing with it.
The Hidden Complexity of Wishes. Imagine you had a device that could cause any concrete statement to become true. This post explores the difficulties you would have getting what you want with the device.

(Thanks to various people at the Lightcone offices for beta reading this post, particularly Ben Pace).
Gradations of Agency
by Daniel Kokotajlo — 2022-05-23T01:10:38.007Z

[Epistemic status: This post written hastily, for Blog Post Day.]

I’ve de-prioritised this sequence due to other recent posts which cover a lot of the same ground. Moreover I begin to suspect that the theory of agency I’m building is merely the glowing brain and that Yudkowsky-Veedrac's is the cosmic galaxy brain. Perhaps mine can be a useful stepping-stone.

Consider the messy and large real world, with creatures reproducing, competing, and evolving. The behavior of each entity is controlled by some sort of cognition/computation—some sort of algorithm.

Inspired by something Dan Dennett wrote (h/t Ramana Kumar), I propose the following loose hierarchy of algorithm families. As we move up the hierarchy, things get more complicated and computationally expensive, but also more powerful and general.

Level 1: Do what worked in the past for your ancestors. You have some “input channel” or “senses” and then you respond to it somehow, doing different things in response to different inputs.

Example: You swim towards warmth and away from cold and hot. You run from red things and towards green things.

This is more complex and computationally expensive, but (if done right) more powerful and general than algorithms which aren’t environment-responsive: There are many situations, many niches in the environment, where being able to sense and adapt quickly helps you survive and thrive.

Level 2: Do what worked in the past for you. How you respond to inputs is itself responsive to inputs; you learn over the course of your life. This typically involves some sort of memory and a special input channel for positive and negative reward.

Example: You start off reacting randomly to inputs, but you learn to run from red things and towards green things because when you ran towards red things you got negative reward and when you ran towards green things you got positive reward.

This is more complex and computationally expensive, but (if done right) more powerful and general:

For a particular ecological niche, you & your relatives will reproduce and fill the niche more quickly than rivals from Level 1. This is because your rivals in some sense keep doing the same thing until they die, and will thus only “learn” to exploit the niche by gradual natural selection.
Also, there are some niches that Level 1 creatures simply cannot survive in, that you can. Example: In Pond X, the color of fo

...
How curing aging could help progress
by jasoncrawford — 2022-05-22T22:43:53.429Z
Link: https://rootsofprogress.org/how-curing-aging-could-help-progress

How would society change if we cured aging, and people could have as many healthy years of life as they wanted?

A common concern is that this would ossify our institutions. The old guard would never die off, and so would never be replaced by young bloods. This could threaten progress across the board, from governance to physics. If “science advances one funeral at a time,” what happens when the rate of funerals plummets?

It’s a real concern. But here are three reasons why curing aging could help progress:

Population. One of the greatest threats to long-term progress may be a slowdown in global population growth. We need more brains to keep pushing science and technology forward. Yet right now, many wealthy nations have fertility rates below replacement levels. Curing aging would help temporarily by lowering the mortality rate. It could help permanently if people decide to have more children, on average. That might happen if longer lifespan means people feel they have time for both children and a career. (Remember that fully curing aging means maintaining reproductive health for all those years.)
Burden of knowledge. There is a hypothesis that as knowledge grows, it takes longer to reach the frontier, and so individual researchers have less time to contribute advancements. They are also forced to specialize—but breakthroughs often come from making connections across far-flung disciplines. If individuals had much longer lifespans, it would be no problem for them to spend 30 or 40 years just learning, before making major contributions. And you could spend another 10 or 20 years picking up a couple more specialties in disparate areas.
Long-term thinking. How would people’s thinking change if they felt they were going to live 150, 300, even 1,000 years or more? The very long-term becomes much more personal. Posterity is something you’re going to be around for.

I still think the “old guard” problem is real, and we’d have to come up with new mechanisms to address it. (Perhaps influential positions would institute a mandatory retirement age of 350.) But there are other factors to consider, and it’s not clear what the net impact would be.

(Not that this is an argument for or against curing aging! Ultimately, the knock-down argument for curing aging is that death is bad. In light of that, other considerations pale into insignificance.)

This essay was originally a Twitter thre...
Help! What do I say at my wedding?
by Jonathan Task — 2022-05-22T20:08:15.459Z

Update: This is what I wrote as my vow

Dearly beloved, believe me when I say that, back in 2019, I would have gladly dug myself a nuke shelter to avoid your vibrant radiance.

Yet a few days ago, late at night and settling into the quiet comfort of your soothing voice, I thought to myself:
If this were to be the autumn of humanity, if we're destined to fizzle out in a lonely universe, I would be glad that you're here to share with me this thin sliver of eternity.

By inviting you into my life as partner and wife, I vow to be your rock and your stronghold in times of strife. When you explore, I will be your basecamp. When you fall, I will be your trampoline. When you speak, I will listen; when you ask, I will tell. Whatever you design I will build with you. Whomever you cherish I will consider my friend.

I consider myself lucky to have been chosen so staunchly by you, and I look forward to meeting you as you return from your journeys, and being the part to your whole.

Update: as the wedding day draws close we're converging on a plan. But we have rooms for last minute edits and for the purpose of other people who look to lesswrong for wedding readings, please still leave your favorite quotes down in the comment section!

Hi!

I'm getting married. My bride is secular, but not into rationalism. Her maid of honor is a Unitarian Universalist, and my best man is a Christian who's constantly in an existential crisis and very open to talk about the possible nonexistence of God. My officiant is a tiny muscular software engineer who is kind yet cynical in a soft-spoken "I don't give a fuck about anything but your feelings" kind of way and will show up at the reception in dinosaur pajamas because my fiancee likes dinosaurs.

We're trying to cater to a group of 30 or so friends that are a mix of cynical software engineers, aspiring classics majors, English majors who are forced to do accounting, and starry eyed environmentalists.

Things I'm currently considering:

Ursula K. LeGuin "to be whole is to be part; true journey is return". But I don't know where to put it. I don't know how weddings work. There seem to be various speeches and vows.

Creation Day by G.K. Chesterton (the stanza I like reproduced below)

Never again with cloudy talk
Shall life be tricked or faith undone,
The world is many and is made,
But we are sane and we are one.
Adversarial attacks and optimal control
by Jan — 2022-05-22T18:22:49.975Z
Link: https://universalprior.substack.com/p/adversarial-attacks-and-optimal-control?s=w

Meta: After a fun little motivating section, this post goes deep into the mathematical weeds. This thing aims to explore the mathematical properties of adversarial attacks from first principles. Perhaps other people are not as confused about this point as I was, but hopefully, the arguments are still useful and/or interesting to some. I'd be curious to hear if I'm reinventing the wheel.

TL;DR:

When we invest the appropriate effort, the probability of random catastrophic events tends to decrease exponentially with a rate given by the[1] rate function.
One way of estimating the rate function is to solve an optimal control problem, where we have to determine the "least surprising" way that the catastrophic event comes about. The rate function equals the catastrophic event's surprise (in nats).
Thus, intuitively, as we invest the effort to decrease the probability of random catastrophic events, the "difficulty" of performing an adversarial attack only increases linearly.

Real talk and real estate

Zillow is an American tech real-estate marketplace company that recently had (what the experts call) a small snafu. They decided they were done just being a marketplace and started buying up homes, completing light renovations, and then selling them with a profit. The whole thing went poorly; they bought houses too expensive and had to sell at a loss, costing the company $420 million and leading to large lay-offs.

This story is not very interesting[2] for anyone not directly involved. The reason I remember the whole affair is a Twitter thread that also caught the attention of Eliezer:

The thread proceeds to lay out how Zillow relied too much on their automatic estimates, which (ex-ante) looked good on average, but (ex-post) manifested in the worst possible way[3]. Or, in the words of the OP:

"[They mistook] an adversarial environment for a random one."

I have approximately zero knowledge about Zillow or the real estate business, so I can't comment on whether this actually is what went wrong (edit: see this comment for a more informed take!). But the distinction between a random and an adversarial environment applies beyond the Zillow example and is relevant for research in AI Safety: from adversarial examples, via adversarial training, all the way to conceivable existential risk scenarios. It is the distinction between a world where it's fine to go outside because...
What kinds of algorithms do multi-human imitators learn?
by Chris van Merwijk — 2022-05-22T14:27:31.430Z

epistemic status: Speculation. The actual proposals are idealized, not meant to be exactly right. We have thought about this for less than an hour.

In

this earlier post

I stated a speculative hypothesis about the algorithm that a single imitator that imitates collections of multiple humans would learn. Here Joar Skalse joined me and we made a list of some more hypotheses, all very speculative and probably each individually wrong.

The point is that if we have an imitator that imitates a single human’s text, we might (very dubiously) expect that imitator to learn basically a copy of that human. What would an imitator learn who is trained to imitate content generated by vast collections of humans? We can then ask: what are the implications for how it generalizes and what you can get with finetuning?

Here are our set of idealized and almost certainly not exactly correct hypotheses:

Lookup table of many separate models of humans. Basically what the algorithm does is: it looks at its prompt, and first figures out which human in the training set generated it (or which subculture of humans, and so forth, including other contexts). It has a separate subroutine for each of them, and simply picks the correct one and executes that.

How does it generalize out of distribution? It basically interpolates between different humans, and still ends up looking like a typical human.
How far can you go with finetuning? Fine tuning allows you to basically not change much. You can make it behave like slightly different human model than what you’d otherwise get.

High level latent space of human minds plus partitioned capabilities. The system has a latent space describing “what kind of human produced this prompt?” It first makes an estimate in this latent space, and based on that, this parameterizes the beliefs, knowledge and attitudes about specific topics that such a human would have.

How far can you get with finetuning? Fine tuning doesn’t easily allow you to build a superhuman system, since each of the points in the latent space parameterizes specific subroutines which inherently activates certain knowledge and capabilities and suppresses others.

Generalized superhuman + lookup table of constraints.

Explained here

.
meta-human model 1: meta-reasoner + superhuman general world model. The model it learns is not at all directly doing the same thing as an individu

...
Are human imitators superhuman models with explicit constraints on capabilities?
by Chris van Merwijk — 2022-05-22T12:46:31.408Z

epistemic status: A speculative hypothesis, don't know if this already exists. The only real evidence I have for this is a vague analogy based on some other speculative (though less speculative) hypothesis. I don’t think this is particularly likely to be true, having thought about it for about a minute. Edit: having thought about it for two minutes, it seems somewhat more likely.

There is a hypothesis that has been floating around in the context of explaining double descent, that what so called over-parameterized models do is store in parallel two algorithms: (1) the simplest model of the data without (label) noise, and (2) a “lookup table” for deviations from that model, to represent the (label) noise, because this is the most simple representation of the data and big neural nets are biased towards simplicity.

Maybe something vaguely similar would happen if we throw sufficient compute at generative models of collections of humans, e.g. language models:

Hypothesis: The simplest way to represent the data generated by various humans on the internet, in the limit of infinite model size and data, is to have (1) a single idealized super-human model for reasoning and writing and knowledge-retrieval and so forth, and (2) memorized constraints on this model for various specific groups of humans and contexts to represent their deviations from rationality, specific biases, cognitive limitations, lack of knowledge of certain areas, etc.

This maybe implies, using some eye squinting and handwaving and additional implicit assumptions, some of the following (vague, speculative) implications about GPT-N:

In the limit of N, GPT-N will produce text that sufficiently looks like human-written text within contexts (prompts) that humans typically produce. GPT-N will use human-level reasoning, world-modeling, and planning abilities to produce this text. However, if you give it sufficiently out-of-distribution prompts, its lookup table for specific irrationalities will not apply, and it will apply superhuman planning and world-modeling and reasoning abilities that are more competent and more free of biases than the most rational human.
In the limit of N, If you take GPT-N and fine tune it on an RL task that requires good reasoning, it might be possible to get a system that seems to behave far more intelligently than it seemed to be on the imitation task, as the fine tuning es

...