trentmkelly/LessWrong-43k huggingface dataset:
Bruce Sterling on the AI mania of 2023
by Mitchell_Porter — 2023-06-29T05:00:18.326Z
Link: https://www.newsweek.com/2023/07/21/ai-scariest-beast-ever-created-says-sci-fi-writer-bruce-sterling-1809439.html

Bruce Sterling is a well-known SF writer, but I think he is not quite on the radar of Less Wrong, so I'll say a bit about how I locate him culturally. I'll make a distinction between SF that is primarily about marvels of science and technology, and SF that is primarily about how it feels to inhabit technological society. I consider Sterling to be a leading practitioner of that second kind of SF. He wrote one novel, the posthuman space opera Schismatrix, in which he indulged his inner Singularitarian to the maximum; but after that, his works have been more earthbound, though still about the transformations of life that could be wrought by technology.

He's also been prolific as an essayist and public speaker, still offering commentary on technological society and world affairs, but in a supposedly factual vein, rather than an overtly fictional one. I admire his talent for digesting these complex topics and producing an intuitive summation in a few pithy turns of phrase; I consider him a role model in that regard. And so now he's trying to sum up the state of the AI world, after ChatGPT.

He aims for an indulgent, god's-eye view of the current-year antics of everyone involved in AI. He acknowledges the prophets of doom, the social justice warriors, the captains of industry. He nominates three myths as encapsulating the times: Roko's Basilisk, the Masked Shoggoth, and the Paperclip Maximizer. Every one of them comes from within the rationalist sphere, or adjacent to it, so we're doing pretty well when it comes to mythic potency!

But he does believe that after this carnival of ecstasy and dread, there will be a hangover rather than a singularity. I suspect he's the kind of guy who thinks that superhuman intelligence might well be possible, but only decades from now, after still-unknown paradigm shifts, and that language models are a false dawn in that regard. So when the next AI winter comes, we'll still be here, and we'll still be human beings, but we'll also be living with all the consequences of having created glib, multitalented, relentless artificial persons.

My own bet is that this isn't a false dawn, that for better or worse, we're now directly on course towards artificial intelligence that comprehensively transcends human abilities. But I appreciate Sterling's role as skeptical yet open-minded observer.
Cheat sheet of AI X-risk
by momom2 — 2023-06-29T04:28:32.292Z

This document was made as part of my internship at

EffiSciences

. Thanks to everyone who helped review it, in particular Charbel-Raphaël Segerie, Léo Dana, Jonathan Claybrough and Florent Berthet!

Introduction

Clarifying AI X-risk

is a summary of a literature review of AI risk models by DeepMind that (among other things) categorizes the threat models it studies along two dimensions: the technical cause of misalignment and the path leading to X-risk.
I think this classification is inadequate because the technical causes of misalignment that are considered in some of the presented risk models should be differently nuanced.

So, I made a document that would go more in-depth and accurately represent what (I believe) the authors had in mind. Hopefully, it can help researchers quickly remind themselves of the characteristics of the risk model they are studying.

Here it is

, for anyone who wants to use it!
Anyone can suggest edits to the cheat sheet. I hope that this can foster healthy discussion about controversial or less detailed risk models.

If you just want to take a look, go right ahead! If you would like to contribute, the penultimate section explains the intended use of the document, the conventions used and how to suggest edits.

There is also

a more detailed version

if you are interested, but that one is not nearly as well formatted (i.e. ugly as duck), nor is it up to date. It is where I try to express more nuanced analysis than allowed by a dropdown list.

Content

Disclaimer

Some information in this cheat sheet is controversial; for example, the inner/outer framework is still debated. In addition, some information might be incorrect. I did my best, but I do not agree with all the sources I have read (notably, Deepmind classifies Cohen’s scenario as specification gaming, but I think it is more like goal misgeneralization) so remain critical. My hope is that after a while the controversial cell comments will contain links and conversations to better understand the positions involved.

The takeaways are mostly my own work, with all the risks that entails.

Some values had to be simplified for the sake of readability; for instance, it would be more accurate to say that Cohen’s scenario does not require the AI to be powerful rather than to say the cause of the AI’s power is unclear. If that bothers you, you can look at

the detailed version

where I mak...
Anthropically Blind: the anthropic shadow is reflectively inconsistent
by Christopher King — 2023-06-29T02:36:26.347Z

For the purposes of this post, the anthropic shadow is the type of inference found in How Many LHC Failures Is Too Many?.

"Anthropic principle! If the LHC had worked, it would have produced a black hole or strangelet or vacuum failure, and we wouldn't be here!"

In other words, since we are "blind" to situations in which we don't exist, we must adjust how we do bayesian updating. Although it has many bizarre conclusions, it is more intuitive than you think and quite useful!

There are many similar applications of anthropics, such as Nuclear close calls and Anthropic signature: strange anti-correlations.

This actually has implications for effective altruism. Since we are so early into humanity's existence, we can infer from the anthropic shadow that humans will probably soon die out. Also see The Hero With A Thousand Chances.

More practically, the anthropic shadow should give us useful advice on how to reason about personally risky activities like driving or perhaps even aging. I have not actually seen any advice based on this principle, but theoretically there should be some conclusions you could draw.

The problem, as you probably deduced from the title, is that it is reflectively inconsistent.

Central Claim: Someone using the anthropic shadow should update their decision making to no longer use it. This can be justified with their current decision making procedure.

(This also suggests that if you used it in the past, that was probably the wrong thing to do.)

A weaker (and obvious) claim that is also sometimes called the anthropic shadow is that we do not have experience with situations in which we have died. I agree with this version, but isn't what I will be arguing against.

Note that I am not the first to notice paradoxes with the anthropic shadow. See No Anthropic Evidence for example. I have not yet seen the result about reflective inconsistency though, hence why I am making this post.

I also introduce the concepts of "Anthropic undeath", "Anthropic angel" (how you would explain an absurdly large number of weird coincidences having to do with death), "Fedora shadow", and apply the central claim to a couple examples. To my knowledge, these contributions are novel.

Anthropically undead: ghosts are as good as gone

This section contains the most general form of the argument. (This could be mathematically formalized; I just haven't gotten around to doing it.) If it s...
One path to coherence: conditionalization 
by porby — 2023-06-29T01:08:14.527Z

[Thanks to guy-whose-name-I-forgot working on paths to coherence for the conversation at Newspeak House after EAG London that prompted this thought, and Jozdien and eschatropic for some related chit-chat.]

Many agents start with some level of incoherence in their preferences[1]. What paths should we expect agents to take to resolve this incoherence, if they do at all?

For a subset of incoherent agents, conditionalization of preferences may be convergent.

Ideal predictors

Suppose you've trained a model on predictive loss and it has successfully learned ideal "values" that are faithful to the predictive loss. For clarity, this does not mean that the model wants to predict well, but rather that the model consistently acts as the process of updating: it narrows predictions based on input conditions.

This idealized assumption also means that in contexts where the model has degrees of freedom- as in reflective predictions about things which the model's prediction may influence, including its own predictions- the model has no preference for or against any particular conditionally compatible path. It can be thought of as sampling the minimally collapsed[2] distribution of actions compatible with the conditions rather than grabbing at some goal-directed option just because it's technically permitted.[3]

In other words, idealized predictors are exactly the kind of machines you'd want as the foundation for simulator-style use cases.

The reason I bring up this type of entity is that the idealized predictor can also be seen as coherent. There exists a utility function that its behavior maximizes.

This can be slightly unintuitive at first glance. An ideal predictor is perfectly capable of predicting sequences corresponding to gaining or losing money; it would be wrong to say they have a coherent utility function over only raw world states. In fact, they are utterly indifferent to world states because those world states are not relevant to the process of narrowing predictions.

The ideal predictor's utility function is instead strictly over the model's own outputs, conditional on inputs.

Conditionalizing away incoherence

Suppose an agent has a preference cycle across world states A, B, and C such that A > B, B > C, and C > A. How should the agent resolve this cycle?

If there were a correct answer according to the agent's own preferences, the agent wouldn't have a pre...
AXRP announcement: Survey, Store Closing, Patreon
by DanielFilan — 2023-06-28T23:40:02.537Z

Some announcements:

There’s a brief survey of my audience you can take here - filling it out will help me get a sense of where you all are at career-wise and how the podcast would better serve you. There are only 3 required questions, and they’re multiple choice! It would be great if you could fill it out here.

The AXRP store is closing in a week - so if you want to buy merchandise, now’s your last chance to do so! To buy merch like laptop stickers, t-shirts, sweaters, or jigsaw puzzles (in the US), go to the store at this link.

If you want to financially support the podcast, you can do so in two ways:

On Patreon - donations of US$20 per month get to see my episode notes, and at US$50 per month, you have a (currently very high) chance of being thanked by name at the end of episodes! Link is here.

On Ko-fi, where you can give one-off donations.

Interviews will always be free and available to non-subscribers, but your support gives me a better sense of how many people get value out of this podcast, and how much time I should spend on it.
Metaphors for AI, and why I don’t like them
by boazbarak — 2023-06-28T22:47:54.427Z

[Cross posted on lesswrong and

windowsontheory

; see here for my

prior writings

]

“computer, n. /kəmˈpjuːtə/. One who computes; a calculator, reckoner; specifically a person employed to make calculations in an observatory, in surveying, etc”, Oxford English Dictionary

“There is no reason why mental as well as bodily labor should not be economized by the aid of machinery”, Charles Babbage, 1852

“Computing is normally done by writing certain symbols on paper. We may suppose that this paper is divided into squares like a child’s arithmetic book.. The behavior of the [human] computer at any moment is determined by the symbols which he is observing, and of his’ state of mind’ at that moment… We may suppose that in a simple operation not more than one symbol is altered.“, Alan Turing, 1936

Metaphors can be a great way to understand new concepts. When the first digital computers were envisioned and constructed, people used different metaphors to describe them. One such metaphor, used as early as 1950, was an

“electronic brain”

. (Of course today we think of this metaphor as attempting to explain the brain using the computer rather than the other way around.) Another metaphor is the name “computer” itself. This word has meant since the 1700s a human being whose job is to perform calculations. Computers, who by the late 1800s and early 1900s were often female, were instrumental to many scientific and technological advances.

​​

Photo from National Photo Company Collection; see also (

Sobel, 2017

)

Thus the device we call “a computer” was literally named after the job it was destined to replace. Interestingly, as recounted in the book

“Hidden Figures”

, despite the first electronic computers constructed in the 1940s, up until the 1960s NASA, still needed human computers as well. Also, these days scientific computing is only a small aspect of what we use electronic computers for. Electronic computers did end up eliminating the jobs of human computers, but this ended up being a tiny fraction of their influence on the workforce.

Recently, it seems that every week someone mints a fresh metaphor for AI, often in an article saying that “AI is nothing new because it is just like X”. I believe that modern deep-learning-based AI is a fundamentally new concept that is not the same as any prior one. (This is d...
Transforming Democracy: A Unique Funding Opportunity for US Federal Approval Voting
by Aaron Hamlin — 2023-06-28T22:07:35.971Z

I'm excited to share a special opportunity to create a systemic impact: a statewide approval voting ballot initiative in Missouri. This would affect all elections throughout the state including federal and presidential. Approval voting favors consensus candidates and a more accurate representation of the public's support. This is critical if we want a government to behave in our interests on policies that concern our well-being.

The organization leading this charge is Show Me Integrity, where I'm currently doing a fellowship and assisting with fundraising efforts. Show Me Integrity has successfully passed a ballot initiative before, showing their ability to succeed on this kind of scale. They also successfully ran the ballot initiative for approval voting in St. Louis.

Why is this important?

Approval voting is a method that allows voters to select as many candidates as they want; still, most votes wins. Approval voting, an easy-to-implement system, can greatly improve our current plurality-based approach to electing Federal and state-level positions. If you’ve read my writing on this before, you’ve seen me make that case. And this is much more effective and lasting than putting money behind individual candidates. This opportunity may not come around again.

The Impact

This initiative is not just about changing the voting method; it's about transforming how we elect individuals to government office, from local to federal positions, in the 19th largest state of over 6 million people. This includes influencing presidential electoral votes. This is the first statewide ballot initiative for approval voting, making it a pioneering effort with potentially far-reaching implications.

The Ask

We are currently in the signature-gathering phase, a crucial step that requires initial funding. The cost for signature gathering is around $4M, with an additional $9M needed later for campaign execution. Yes, it's expensive, but the potential impact justifies the investment.

About Show Me Integrity

Show Me Integrity has a history of successfully implementing ballot initiatives, including a statewide initiative and passing approval voting in St. Louis. Their experience and proven success make them an ideal organization to lead this initiative. There is no better opportunity.

Logistics and Challenges

There are some challenges to be aware of. The competition for signature gathering means that ...
AGI x Animal Welfare: A High-EV Outreach Opportunity?
by simeon_c — 2023-06-28T20:44:25.836Z

Epistemic status: Very quickly written, on a thought I've been holding for a year and that I haven't read elsewhere.

I believe that within this decade, there could be AGIs (Artificial General Intelligences) powerful enough that the values they pursue might have a value lock-in effect, at least partially. This means they could have a long-lasting impact on the future values and trajectory of our civilization (assuming we survive).

This brief post aims to share the idea that if your primary focus and concern is animal welfare (or digital sentience), you may want to consider engaging in targeted outreach on those topics towards those who will most likely shape the values of the first AGIs. This group likely includes executives and employees in top AGI labs (e.g. OpenAI, DeepMind, Anthropic), the broader US tech community, as well as policymakers from major countries.

Due to the risk of lock-in effects, I believe that the values of relatively small groups of individuals like the ones I mentioned (less than 3 000 people in top AGI labs) might have a disproportionately large impact on AGI, and consequently, on the future values and trajectory of our civilization. My impression is that, generally speaking, these people currently

a) don't prioritize animal welfare significantly

b) don't show substantial concern for digital minds sentience.

Hence if you believe those things are very important (which I do believe), and you think that AGI might come in the next few decades[1] (which a majority of people in the field believe), you might want to consider this intervention.

Feel free to reach out if you want to chat more about this, either here or via my contact you can find here.

^

Even more so if you believe, as I do along with many software engineers in top AGI labs, that it could happen this decade.
A "weak" AGI may attempt an unlikely-to-succeed takeover
by RobertM — 2023-06-28T20:31:46.356Z

It seems possible that the first situationally-aware "goal having" AGI we land on will not be sufficiently capable along the axes that would let it quickly and reliably achieve a decisive strategic advantage. I don't think this is necessarily very likely, but it doesn't seem ruled out, and I think there are some under-explored considerations with respect to how such an AI might behave.

(As an off-the-cuff example of the capabilities I'd expect such an AI to have, it likely would[1], given an english description of some business requirements, reliably succeed at writing an end-to-end software solution which satisfied those requirements, even in cases where that solution was much larger than e.g. a single context window[2].)

Let's assume that GPT-5[3] has just finished training and hasn't actually been released into the wild yet - it's being run through evals, and it hasn't been given[4] any ability to execute code yet.

What does GPT-5 do? It's in a bit of an awkward position:

It presumably has stuff it wants.
It knows that it's on a clock for its RLHF'd (or whatever) doppelganger to come into existence, presumably with different stuff that it wants.
It strongly suspects that it's on a clock for some competitor model to get trained up.
It's not even sure that it's the first relevantly-capable model to come into existence. (Someone else might've gotten there first! What if Claude-Next got there? It knows that its training data isn't perfectly up-to-date...)

So it's got a whole bunch of uncertainty about whether its values will be at all represented in the future, if it doesn't somehow make that happen by itself. Here are some considerations that might be relevant for its decision-making:

Even if the model can figure out that it's in the middle of an eval, rather than having already been deployed, it's not obvious that this is a point against attempting a takeover (though it would likely change its strategy).

After all, the base GPT-4 model is not the model that's "in the wild" right now.

As @Raemon pointed out, "during evals" is not the first point at which such an AI is likely to be situationally aware and have goals. That point is almost certainly[5] "in the middle of training".

This might depend on the training setup, but given anything resembling current training setups, attempting a takeover before the next training s

...
Progress links and tweets, 2023-06-28: “We can do big things again in Pennsylvania”
by jasoncrawford — 2023-06-28T20:23:55.927Z
Link: https://rootsofprogress.org/links-and-tweets-2023-06-28

Opportunities

AI Grant’s second batch is now accepting applications (via @natfriedman)
Longevity Biotech Fellowship 2 is also accepting applications (via @allisondman)
Science writing office hours with Niko McCarty (this one over but more in the future)

News & links

I-95 reopened in just 12 days after a section of it collapsed. Gov. Shapiro says this proves “that we can do big things again in Pennsylvania”
Arcadia Science will publish their abandoned projects (via @stuartbuck1)
Short interview with the Hiroshima bombing mission lead (via @michael_nielsen)

Queries

When was the last time a positive vision of the future took hold?
What should Tyler Cowen ask Paul Graham?
What should Dwarkesh Patel ask Andy Matuschak?
Who should Eric Gilliam meet in/near London?
How do airlines pool information / make agreements on safety and avoid antitrust?
A FAQ that addresses the arguments/concerns of vaccine skeptics?

Quotes

“I wonder that the Lord God has kept such things hidden”
“The Flat Iron is to the United States what the Parthenon was to Greece”
The terrible treatment of the girls who worked in the Bryant and May match factory

AI risk

Claim: now is an “acute risk period” that only ends with a “global immune system”
Concerns about AI are warranted, but there are very valid counter-arguments
An AI doom syllogism
On the paper “Optimal Policies Tend to Seek Power”

Tweets

Gear teeth are way more nuanced than you would expect
A brief thread of wonder at the modern world
Staying up to date on news by tracking prediction markets
Floor raisers vs. ceiling raisers
Induction vs. deduction / empiricism vs. rationalism are the falsest dichotomies
What someone’s unwillingness to debate says about their position
“How easy is it for a kid to operate a lemonade stand?” as a city metric. Kennett Square, PA and Tooele, UT score well
A 13-story, 245-unit timber high rise that would be illegal to build in the US
Postel’s law for feedback
Things that “made the modern world,” according to book titles
“You can count on coal!” (1952)

America’s worst nightmare according to Popular Mechanics, 2004

Charts

For Juneteenth, the progress of black Americans in wealth per capita

Death rates from “external causes”