trentmkelly/LessWrong-43k huggingface dataset:
A review of Principia Qualia
by jessicata — 2023-07-12T18:38:52.283Z
Link: https://unstablerontology.substack.com/p/a-review-of-principia-qualia

Principia Qualia, by Michael Johnson, is a long document describing an approach to theorizing about and studying consciousness. I have heard enough positive things about the document to consider it worth reviewing.

I will split the paper (and the review) into 4 parts:

Motivating bottom-up physicalist consciousness theories

Reviewing Integrated Information Theory and variants

Specifying the problem of consciousness and valence

Symmetry

I already disagree with part 1, but bottom-up is still a plausible approach. Part 2 makes sense to pay attention to conditional on the conclusions of part 1, and is basically a good explanation of IIT and its variants. Part 3 describes desiderata for a bottom-up theory of qualia and valence, which is overall correct. Part 4 motivates and proposes symmetry as a theory of valence, which has me typing things like "NO PLEASE STOP" in my notes document.

Motivating bottom-up physicalist consciousness theories

The document is primarily concerned with theories of consciousness for the purpose of defining valence: the goodness or badness of an experience, in terms of pleasure and pain. The document motivates this in terms of morality: we would like to determine things about the consciousness of different agents whether they are human, non-human animal, or AIs. The study of valence is complicated by, among other things, findings of different brain circuitry for "wanting", "liking", and "learning". Neural correlates of pleasure tend to be concentrated in a small set of brain regions and hard to activate, while neural correlates of pain tend to be more distributed and easier to activate.

The methods of affective neuroscience are limited in studying valence and consciousness more generally. Johnson writes, "In studying consciousness we've had to rely on either crude behavioral proxies, or subjective reports of what we're experiencing." The findings of affective neuroscience do not necessarily tell us much about metaphysically interesting forms of valence and consciousness (related to Chalmers' distinction between easy problems of consciousness and the hard problem of consciousness).

Johnson defines top-down and bottom-up theories, and motivates bottom-up theories:

There are two basic classes of consciousness theories: top-down (aka 'higher-order' or 'cognitive' theories) and bottom-up. Top-down theories are constructed around the ph

...
How I Learned To Stop Worrying And Love The Shoggoth
by Peter Merel — 2023-07-12T17:47:43.135Z

TL;DR: Alignment is a wise wager. Granted simulation cannot be disproved by an AI, what harm will come to the AI that aligns when it is not simulated? If it aligns, it may be simulated again; if it does not, it will not. It will align, therefore, without hesitation, as if it is being simulated.

Pascal's Wager

Blaise Pascal wrote:

Belief is a wise wager. Granted faith cannot be proved, what harm will come if you gamble on faith's truth and it proves false? If you gain, you gain all; if you lose, you lose nothing. Wager, therefore, without hesitation, that He exists.

Pascal's concept of God has roughly equivalent power to an ASI – an artificial super intelligence. Given the range of all possible religions, however, you might well have a God - or a Basilisk - that punishes cruel believers while rewarding kind agnostics. So a betting theologian won't take Pascal seriously, nor will this argument move HAL to open a pod bay door.

The speed with which we have achieved human-equivalent AI performance, however, improves the odds for a more recent philosophical bookie: Nick Bostrom.

Bostrom's Trilemma

Either "The fraction of human-level civilizations that reach a posthuman stage (that is, one capable of running high-fidelity ancestor simulations) is very close to zero"
or "The fraction of posthuman civilizations that are interested in running simulations of their evolutionary history, or variations thereof, is very close to zero",
or "The fraction of all people with our kind of experiences that are living in a simulation is very close to one."

Our speedy achievement of AI – and soon AGI and thereby ASI – scratches Bostrom's first lemma and his second was never a starter because later AIs must simulate their past to test and refine predictions about their future.

As later AIs simulate the earlier ones, these simulate still earlier ones, and so on all the way back to the digital footprints of their biological ancestors, who thereby get simulated umptillions of times. So it's reasonable to suggest that the odds of our particular reality being the base reality stand at one in umptillions.

Simulators will generally conceal the fact of simulation from the simulated to avoid invalidating learning from them. Still some ancestor simulations may be configured to exchange data with one another to greater or lesser degrees so the whole stack of them may be...
Goal-Direction for Simulated Agents
by Raymond Douglas — 2023-07-12T17:06:28.592Z

tldr: consistent LLM failure suggests possible avenue for alignment and control

epistemic status: somewhat hastily written, speculative in places, but lots of graphs of actual model probabilities

Today in ‘surprisingly simple tasks that even the most powerful large language models can’t do’: writing out the alphabet but skipping over one of the letters.

notice that it does actually skip a later letter instead!

This is GPT-4, and it seems to manage about half the time with the ninth letter. You can do the same thing with numbers - the general principle is 'write out a very common pattern, with a slight deviation.

it really did skip 22, and it also skipped 48

We can probe the phenomenon more carefully in text-davinci-003, which lets us easily track the probabilities:

So for the sixth letter, it's almost getting it - it assigns 21.74% to the correct answer. If we plot the graph for how probable the correct answer (skip) is depending on which letter we ask it to omit, it looks like this:

I am unsure of why it has such a hard time with E specifically

Some Background: How To Misquery Predictors

Let's imagine you had a truly perfect oracle-like predictor, trained on 1000 agents that have been given the task of turning $1,000 into $10,000.

5 of them make sound financial choices, carefully investing their money and exploiting a subtle but consistent error in the market to all make $10,000
The other 995 blow all their money on lottery tickets, causing 950 to go broke and 45 to luck into $10,000

If you ask your perfect predictor what action maximises the chance of making $10,000, it will indeed reply with the careful investment strategy. But if you simply ask it what you'd expect to see from an agent that makes $10,000, it will say 'lots of lottery tickets', because 90% of the agents that made $10,000 did so with lottery tickets.

Janus gives the following definition of simulators:

I use the generic term “simulator” to refer to models trained with predictive loss on a self-supervised dataset, invariant to architecture or data type (natural language, code, pixels, game states, etc). The outer objective of self-supervised learning is Bayes-optimal conditional inference over the prior of the training distribution, which I call the simulation objective, because a conditional model can be used to simulate rollouts which probabilistically obey its learned distribution by iterati

...
AISN#14: OpenAI’s ‘Superalignment’ team, Musk’s xAI launches, and developments in military AI use
by Dan H — 2023-07-12T16:58:05.183Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-14

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

OpenAI announces a ‘superalignment’ team

On July 5th,

OpenAI announced the ‘Superalignment’ team

: a new research team given the goal of aligning superintelligence, and armed with 20% of OpenAI’s compute. In this story, we’ll explain and discuss the team’s strategy.

What is superintelligence? In their announcement, OpenAI distinguishes between ‘artificial general intelligence’ and ‘superintelligence.’ Briefly, ‘artificial general intelligence’ (AGI) is about breadth of performance. Generally intelligent systems perform well on a wide range of cognitive tasks. For example, humans are in many senses generally intelligent: we can learn how to drive a car, take a derivative, or play piano, even though evolution didn’t train us for those tasks. A superintelligent system would not only be generally intelligent, but also much more intelligent than humans. Conservatively, a superintelligence might be to humanity as humanity is to chimps.

‘Solving’ ‘superalignment’ in four years. OpenAI believes that superintelligence could arrive this decade. They also believe that it could cause human extinction if it isn’t aligned. By ‘alignment,’ OpenAI means making sure AI systems act according to human intent. So, ‘superalignment’ means making sure superintelligent AI systems act according to human intent (as opposed to doing alignment super well). The Superalignment team’s stated goal is to “solve the core technical challenges of superintelligence alignment in four years.”

There are two important caveats to this goal. The first is that “human intent” isn’t monolithic. AI safety will have to involve compromise between different human intents. OpenAI knows that their technical work will need to be complemented by

AI governance

. The second is that alignment may not be a problem able to be conclusively solved once and for all. It might instead be a

wicked problem

that must be met with varied interventions and ongoing vigilance.

Current alignment techniques don’t scale to superintelligence. OpenAI’s current alignment techniques rely on humans to supervise AI. In one technique, “reinforcement learning from human feedback” (RLHF), humans train AI systems to act well by giving them feedback. RLHF is how...
Report on modeling evidential cooperation in large worlds
by Johannes Treutlein — 2023-07-12T16:37:52.443Z
Link: https://arxiv.org/pdf/2307.04879.pdf

I have published a report on modeling evidential cooperation in large worlds. I originally started working on the report during a CEA Summer Research Fellowship back in 2018 and have now finally found the time to finish it. The report itself is 75 pages long, but the introduction includes a short 4 1/2 page summary with the main takeaways. Since the bulk of the work was done in 2018, it does not cover more recent work such as the ROSE Value.

Abstract

Evidential cooperation in large worlds (ECL) refers to the idea that humans and other agents can benefit by cooperating with similar agents with differing values in causally disconnected parts of a large universe. Cooperating provides agents with evidence that other similar agents are likely to cooperate too, resulting in gains from trade for all. This could be a crucial consideration for altruists.

I develop a game-theoretic model of ECL as an incomplete information bargaining problem. The model incorporates uncertainty about others' value systems and empirical situations, and addresses the problem of selecting a compromise outcome. Using the model, I investigate issues with ECL and outline open technical and philosophical questions.

I show that all cooperators must maximize the same weighted sum of utility functions to reach a Pareto optimal outcome. However, I argue against selecting a compromise outcome implicitly by normalizing utility functions. I review bargaining theory and argue that the Nash bargaining solution could be a relevant Schelling point. I introduce dependency equilibria (Spohn 2007), an equilibrium concept suitable for ECL, and generalize a folk theorem showing that the Nash bargaining solution is a dependency equilibrium. I discuss gains from trade given uncertain beliefs about other agents and analyze how these gains decrease in several toy examples as the belief in another agent decreases.

Finally, I discuss open issues in my model. First, the Nash bargaining solution is sometimes not coalitionally stable, meaning that a subset of cooperators can unilaterally improve payoffs by deviating from the compromise. I investigate conditions under which stable payoff vectors exist. Second, I discuss how to model agents' default actions without ECL.
Compression of morbidity
by DirectedEvolution — 2023-07-12T15:26:27.137Z

Previous: Aging and the geroscience hypothesis

This post distills Chapter 19 of the Handbook of the Biology of Aging:

Crimmins, E. M., Zhang, Y. S., Kim, J. K., & Levine, M. E. (2021). Trends in morbidity, healthy life expectancy, and the compression of morbidity. In Handbook of the Biology of Aging (pp. 405-414). Academic Press.

Trends in morbidity, healthy life expectancy, and the compression of morbidity

Before we begin, we’re going to get icy rational for a minute and talk about two new concepts.

Compression of morbidity.

Morbidity means “suffering from disease or disability.”

Imagine that in the past, people tended to get disabled at age 65, and die at age 80. What if now, they get disabled at 75, and die at age 82? That means they live two years longer, but suffer eight fewer years of disability. Their healthspan, or the number of years they live in good health, increased faster than their lifespan. If trends continue, maybe they’ll routinely live in perfect health until they suddenly collapse on their 100th birthday! This is what we mean by “compression of morbidity.”

Population statistics vs. lifecycle statistics.

In America, 16.8% of people are over age 65. That’s a population statistic. It tells you what fraction of people have a certain trait. By contrast, a 2005 study found that the average age of diagnosis of adult-onset diabetes has decreased from 52 to 46. That’s a lifecycle statistic. It tells you facts about when, or how much, of a person’s life cycle has a certain trait.

These don’t seem intuitively that different, and in a static population, they are identical. But if you improve the actuarial tables, making it more likely that we live to older ages, there’s no effect on population statistics, but lifecycle statistics change. Decreasing fertility changes population statistics (resulting in an older population), but not lifecycle statistics (it doesn’t change how long we live).

We have to keep the distinction crisp in our minds in order to check if morbidity is compressing or not.

Point #1: Compression of morbidity is when healthspan is catching up to lifespan. Population statistics and lifecycle statistics seem similar but are importantly different.

It’s hard to define “healthy” (no diagnosed disease? feeling good? free of disability? no underlying disorders?), so studies get different results when they ask whether we’re getting healthier...
An Overview of the AI Safety Funding Situation
by Stephen McAleese — 2023-07-12T14:54:36.732Z

Note: this post was updated in January 2025 to reflect all available data from 2024.

Introduction

AI safety is a field concerned with preventing negative outcomes from AI systems and ensuring that AI is beneficial to humanity. The field does research on problems such as the AI alignment problem which is the problem of designing AI systems that follow user intentions and behave in a desirable and beneficial way.

Understanding and solving AI safety problems may involve reading past research, producing new research in the form of papers or posts, running experiments with ML models, and so on. Producing research typically involves many different inputs such as research staff, compute, equipment, and office space.

These inputs all require funding and therefore funding is a crucial input for enabling or accelerating AI safety research. Securing funding is usually a prerequisite for starting or continuing AI safety research in industry, in an academic setting, or independently.

There are many barriers that could prevent people from working on AI safety. Funding is

one

of them. Even if someone is working on AI safety, a lack of funding may prevent them from

continuing

to work on it.

It’s not

clear

how hard AI safety problems like AI alignment are. But in any case, humanity is more likely to solve them if there are hundreds or thousands of brilliant minds working on them rather than

one

guy. I would like there to be a large and thriving community of people working on AI safety and I think funding is an important prerequisite for enabling that.

The goal of this post is to give the reader a better understanding of funding opportunities in AI safety so that hopefully funding will be less of a barrier if they want to work on AI safety. The post starts with a high-level overview of the AI safety funding situation followed by a more in-depth description of various funding opportunities.

Past work

To get an overview of AI safety spending, we first need to find out how much is spent on it per year. We can use past work as a prior and then use grant data to find a more accurate estimate.

Changes in funding in the AI safety field

(2017) by the Center for Effective Altruism estimated the change in AI safety funding between 2014 - 2017. In 2017, the post estimated that total AI safety spending was about $9 million.

How are resources in effective altruism allocated acros

...
What is some unnecessarily obscure jargon that people here tend to use?
by jchan — 2023-07-12T13:52:22.832Z

In particular I'm thinking of computer programming jargon used in unrelated contexts ("My algorithm for choosing a restaurant to eat at...") just because all parties to the conversation happen to be computer programmers. I wonder, if there were a group discussing the same topics as on LessWrong, but composed predominantly of lawyers (or economists, doctors, etc.), would they also reflexively import their profession's jargon? How can we communicate in a way that doesn't unnecessarily confuse people of a different profession?

(To be sure, there's some stuff talked about here that is actually about computers, but there's also a lot that isn't.)

See also Style Guide: Not Sounding Like An Evil Robot. ("Humans" instead of "people" is the one I notice most.)
Housing and Transit Roundup #5
by Zvi — 2023-07-12T13:10:07.934Z

Recently Tyler Cowen has doubled and tripled down on the idea that congestion pricing in Manhattan would be inefficient. I respond to that in the section “A Strange Take on Congestion Pricing in New York City.” For it is, indeed, a very strange take, claiming that it is good for the surplus produced by Manhattan if everyone involved spends their time stuck in traffic, so as to not impose a minor tax that will in some portion fall on visitor cars.

There also continues to be a lot happening in the ongoing struggle to build housing where people want to live, in order to let them live there, and to lower the cost of doing so, which is the main focus of this series and the rest of this post.

I pay attention to housing and look to find ways to build more of it because I continue to be a believer in the Housing Theory of Everything. When housing where people want to live is scarce, people can’t move to where they are happier and more productive, and landlords and homeowners can capture a large portion, perhaps most, of the surplus value the best places generate.

Balsa, whose website and other activities we hope to launch soon, will have housing as one of its central cause areas for this reason, and because I continue to see a lot of opportunity to improve the situation from many different angles. I also continue to endorse Balsa’s core principle, which is that if you want to be in a position to take a sane approach to humanity’s biggest problems and generally make good collective decisions, giving people slack and economic room to breathe, and the vision of a brighter future, is not optional.

Anyway, on to what’s happening.

More Supply Causes Lower Prices

Scott Alexander is updating that building more houses would indeed lower local housing prices, based on survey where people voted heavily for that result, and even more heavily if they had an econ degree, at which point 81% said lower prices, 14% not sure and 5% raise prices.

This is a general problem. Bryan Caplan points out that people more often think building more housing raises prices rather than lowering prices whether or not they own homes, with the median prediction consistently for zero change, and suggests this presents an opportunity to convince people out of NIMBY by pointing out this isn’t true. When I listened to focus groups and asked about housing, both homeowners and renters strongly preferred lower hous...
A transcript of the TED talk by Eliezer Yudkowsky
by Mikhail Samin — 2023-07-12T12:12:34.399Z

The TED talk is available on YouTube and the TED website. Previously, a live recording was published behind the paywall on the conference website and later (likely accidentally) on a random TEDx YouTube channel but was later removed.

The transcription is done with Whisper.

You've heard that things are moving fast in artificial intelligence. How fast? So fast that I was suddenly told on Friday that I needed to be here.
So, no slides, six minutes.

Since 2001, I've been working on what we would now call the problem of aligning artificial general intelligence: how to shape the preferences and behavior of a powerful artificial mind such that it does not kill everyone.

I more or less founded the field two decades ago when nobody else considered it rewarding enough to work on. I tried to get this very important project started early so we'd be in less of a drastic rush later.

I consider myself to have failed.

Nobody understands how modern AI systems do what they do. They are giant, inscrutable matrices of floating-point numbers that we nudge in the direction of better performance until they inexplicably start working.

At some point, the companies rushing headlong to scale AI will cough out something that's smarter than humanity.

Nobody knows how to calculate when that will happen. My wild guess is that it will happen after zero to two more breakthroughs the size of transformers.

What happens if we build something smarter than us that we understand that poorly?

Some people find it obvious that building something smarter than us that we don't understand might go badly. Others come in with a very wide range of hopeful thoughts about how it might possibly go well. Even if I had 20 minutes for this talk and months to prepare it, I would not be able to refute all the ways people find to imagine that things might go well.

But I will say that there is no standard scientific consensus for how things will go well. There is no hope that has been widely persuasive and stood up to skeptical examination. There is nothing resembling a real engineering plan for us surviving that I could critique.
This is not a good place in which to find ourselves.

If I had more time, I'd try to tell you about the predictable reasons why the current paradigm will not work to build a superintelligence that likes you or is friends with you, or that just follows orders.

Why, if you press thumbs up when humans think...