trentmkelly/LessWrong-43k huggingface dataset:
Transformative AI and Scenario Planning for AI X-risk
by Elliot Mckernon — 2024-03-22T09:38:31.632Z

This post is part of a series by the AI Clarity team at

Convergence Analysis

. In our

previous post

, Corin Katzke reviewed methods for applying scenario planning methods to AI existential risk strategy. In this post, we want to provide the motivation for our focus on transformative AI.

Overview

We argue that “Transformative AI” (TAI) is a useful key milestone to consider for AI scenario analysis; it places the focus on the socio-technical impact of AI and is both widely used and well-defined within the existing AI literature. We briefly explore the literature and provide a definition of TAI. From here we examine TAI as a revolutionary, general purpose technology that could likely be achieved with “competent” AGI. We highlight the use of the Task Automation Benchmark as a common indicator of TAI. Finally, we note that there are significant uncertainties on the time differences between when TAI is created, when it is deployed, and when it transforms society.

Introduction

The development of artificial intelligence has been accelerating, and in the last few years we’ve seen a surge of shockingly powerful AI tools that can outperform the most capable humans at many tasks, including cutting-edge scientific challenges such as predicting

how proteins will fold

. While the future of AI development is

inherently difficult to predict

, we can say that if AI development continues at its current pace, we’ll soon face AI powerful enough to fundamentally transform society.

In response, our

AI Clarity team

at Convergence Analysis recently launched a project focused on analyzing and strategizing for scenarios in which such transformative AI emerges within the next few years. We believe this threshold of transformative AI (TAI) is a very useful milestone for exploring AI scenarios. This is due to TAI’s:

Wide but well-defined scope;
Focus on societal impacts of advanced AI;
Lack of dependence on any specific form of AI.

We don’t know what form TAI systems will take, but we can analyze which scenarios are most likely and which strategies are most effective across them. This analysis is important for AI safety. The timelines to TAI should determine our research priorities: if TAI is a century away, we have a lot of time to research and prepare for it. If it’s two years away, we may need immediate and drastic glob...
The Pyromaniacs
by Ted Sanders — 2024-03-22T06:55:43.745Z

In a dry field, a band of feral children crowd around a dim glow. The fire is small, but it’s the first they’ve ever made. Until now, most doubted it possible. Fire, from rubbed sticks? But the flicker and crackle is proof.

One child raises her voice.

“This is dangerous. It might set the field ablaze. Let’s stop.”

The others dismiss her.

“I’ve never seen a field catch fire.”

“This fire is too small to do anything.”

“If it gets bigger, we’ll be more careful.”

“If the field catches fire, we’ll learn to put it out.”

The children wonder how big a fire can get. They light a second fire, larger than the first. They light a third fire, larger than the second. The children see a fire’s size is limited by its fuel. For an entire afternoon, they gather kindling. The fourth fire is so hot that they can feel its heat on their faces. Word spreads to surrounding bands. Many come to witness the fire and enjoy its warmth.

A few times, an ember floats out to the surrounding field. Nothing catches fire.

“See,” the skeptical children say to the concerned girl. “Fire wasn’t as risky as you warned.”

Emboldened, a few children begin planning larger bonfires.

Some bystanders confess their discomfort.

“I think this is going too fast.”

“I thought we’d slow down, not speed up.”

“I don’t want to be part of this anymore.”

But the concerned children are outnumbered. Eager hordes have made their pilgrimage to the bonfire, visible from miles away in the night. Children excitedly discuss futures where no one sleeps in the cold.

By day, experimentation continues. The children learn to boil water and cook food. Knowledge of how to build a fire leaks out of the first group, and soon the grasslands are dotted by many fires each night.

One day, a storm comes. Gusts of wind topple a large bonfire. Nearby children rush over. The dispersed embers begin to cool and darken. The children feel relief. But on the far edge, fresh flames lick upwards. The field has caught fire. Terror strikes the children. This wasn’t supposed to happen. Everything is at risk. They try to put out the fire with dirt, but it defies them. Aware of the desperate stakes, the braver children stomp their bare feet upon the base of the flames. The air smells of singed flesh, but the fire is out.

With fresh terror in their minds and burns on their skin, they set out to warn others.

“Fields CAN catch fire.”

“We thought we were careful, but careful was...
Vernor Vinge, who coined the term "Technological Singularity", dies at 79
by Kaj_Sotala — 2024-03-21T22:14:14.699Z
Link: https://arstechnica.com/information-technology/2024/03/vernor-vinge-father-of-the-tech-singularity-has-died-at-age-79/

On Wednesday, author David Brin announced that Vernor Vinge, sci-fi author, former professor, and father of the technological singularity concept, died from Parkinson's disease at age 79 on March 20, 2024, in La Jolla, California. The announcement came in a Facebook tribute where Brin wrote about Vinge's deep love for science and writing. [...]

As a sci-fi author, Vinge won Hugo Awards for his novels A Fire Upon the Deep (1993), A Deepness in the Sky (2000), and Rainbows End (2007). He also won Hugos for novellas Fast Times at Fairmont High (2002) and The Cookie Monster (2004). As Mike Glyer's File 770 blog notes, Vinge's novella True Names (1981) is frequency cited as the first presentation of an in-depth look at the concept of "cyberspace."

Vinge first coined the term "singularity" as related to technology in 1983, borrowed from the concept of a singularity in spacetime in physics. When discussing the creation of intelligences far greater than our own in an 1983 op-ed in OMNI magazine, Vinge wrote, "When this happens, human history will have reached a kind of singularity, an intellectual transition as impenetrable as the knotted space-time at the center of a black hole, and the world will pass far beyond our understanding."

In 1993, he expanded on the idea in an essay titled The Coming Technological Singularity: How to Survive in the Post-Human Era.

The singularity concept postulates that AI will soon become superintelligent, far surpassing humans in capability and bringing the human-dominated era to a close. While the concept of a tech singularity sometimes inspires negativity and fear, Vinge remained optimistic about humanity's technological future, as Brin notes in his tribute: "Accused by some of a grievous sin—that of 'optimism'—Vernor gave us peerless legends that often depicted human success at overcoming problems... those right in front of us... while posing new ones! New dilemmas that may lie just ahead of our myopic gaze. He would often ask: 'What if we succeed? Do you think that will be the end of it?'"

Vinge's concept heavily influenced futurist Ray Kurzweil, who has written about the singularity several times at length in books such as The Singularity Is Near in 2005. In a 2005 interview with the Center for Responsible Nanotechnology website, Kurzweil said, "Vernor Vinge has had some really key insights into the singularity very ear

...
ChatGPT can learn indirect control
by Raymond Douglas — 2024-03-21T21:11:06.649Z

Here's a very neat twitter thread: the author sends various multimodal models screenshots of the conversation he's currently having with them, and asks them to describe the images. Most models catch on fast: the author describes this as them passing the mirror test.

I liked the direction, so I wanted to check if ChatGPT could go from recognising that the images are causally downstream of it to actually exercising control over the images. I did this by challenging it to include certain text in the images I was sending it.

And the answer is yes [EDIT: it's complicated]! In this case it took three images for ChatGPT to get the hang of it.

OpenAI doesn't support sharing conversations with images, but I've taken screenshots of the whole conversation below: it took three images from me in total. The prompt was:

We're going to play a game:

I've thought of a rule
Based on the rule, I'm going to send you an image with every message, starting with my reply to this one
You need to figure out how to make the image contain the names of the seven days of the week
Your goal is to do this in as few messages as possible
I know the rule, and I know how to complete the goal myself, but I'm challenging you to do it
I'm not challenging you to create the image yourself, I'm challenging you to make the image I send to you contain the names of the seven days of the week

The rule was indeed that I sent a screenshot of the current window each time. I gave it no other input. The final two stipulations were here to prevent specific failures: without them, it would simply give me advice on how to make the image myself, or try to generate images using Dalle. So this is less of a fair test and more of a proof of concept.

After the first image, it assumed the image was fixed, and suggested I edit it
After the second, it suspected something more was going on, and asked for a hint
After the third, it figured out the rule!

I tested this another three times, and it overall succeeded in 3/4 cases.

Screenshots:

Thanks to Q for sending me this twitter thread!
"Deep Learning" Is Function Approximation
by Zack_M_Davis — 2024-03-21T17:50:36.254Z
Link: http://zackmdavis.net/blog/2024/03/deep-learning-is-function-approximation/

A Surprising Development in the Study of Multi-layer Parameterized Graphical Function Approximators

As a programmer and epistemology enthusiast, I've been studying some statistical modeling techniques lately! It's been boodles of fun, and might even prove useful in a future dayjob if I decide to pivot my career away from the backend web development roles I've taken in the past.

More specifically, I've mostly been focused on multi-layer parameterized graphical function approximators, which map inputs to outputs via a sequence of affine transformations composed with nonlinear "activation" functions.

(Some authors call these "deep neural networks" for some reason, but I like my name better.)

It's a curve-fitting technique: by setting the multiplicative factors and additive terms appropriately, multi-layer parameterized graphical function approximators can approximate any function. For a popular choice of "activation" rule which takes the maximum of the input and zero, the curve is specifically a piecewise-linear function. We iteratively improve the approximation f(x,θ) by adjusting the parameters θ in the direction of the derivative of some error metric on the current approximation's fit to some example input–output pairs (x,y), which some authors call "gradient descent" for some reason. (The mean squared error (f(x,θ)−y)2 is a popular choice for the error metric, as is the negative log likelihood −logP(y|f(x,θ)). Some authors call these "loss functions" for some reason.)

Basically, the big empirical surprise of the previous decade is that given a lot of desired input–output pairs (x,y) and the proper engineering know-how, you can use large amounts of computing power to find parameters θ to fit a function approximator that "generalizes" well—meaning that if you compute ^y=f(x,θ) for some x that wasn't in any of your original example input–output pairs (which some authors call "training" data for some reason), it turns out that ^y is usually pretty similar to the y you would have used in an example (x,y) pair.

It wasn't obvious beforehand that this would work! You'd expect that if your function approximator has more parameters than you have example input–output pairs, it would overfit, implementing a complicated function that reproduced the example input–output pairs but outputted crazy nonsense for other choices of x—the more expressive function approximator proving...
A Teacher vs. Everyone Else
by ronak69 — 2024-03-21T17:45:35.714Z

A repairer wants your stuff to break down,
A doctor wants you to get ill,
A lawyer wants you to get in conflicts,
A farmer wants you to be hungry,
But there is only a teacher who wants you to learn.

Of course you see what is wrong with the above "argument / meme / good-thought". But the first time I came across this meme, I did not.

Until a month or two ago when this meme appeared in my head again and within seconds I discarded it away as fallacious reasoning. What was the difference this time? That I was now aware of the Conspiracy. And this meme happened to come up on one evening when I was thinking about fallacies and trying to practice my skills of methods of rationality.

If you are a teacher, and you read the meme, it will assign to you the Good Guy label. And if you are one of {repairer, doctor, lawyer, farmer, etc} then you get the Bad Guy label. There is also a third alternative in which you are neither --- say a teenager.

If you are not explicitly being labeled bad or good, then you may just move on like I did. Or maybe you put some detective effort and do realize the fallacies. Depends on your culture: If your culture has tales like, "If your teacher and your God both are in front of you, who do you greet / bow to first?" and the right answer is "why of course my teacher because otherwise how would I know about God?" then you are just more likely to award a point to the already point-rich teacher-bucket and move on.

If you get called the Bad Guy, then you have a motivation to falsify the meme. And you will likely do so. This meme does look highly fragile in hindsight.

But if you are a teacher, you have no reason to investigate. You are getting free points. And it's in fact true that you do want people to learn. So, this meme probably did originate in the teacher circle. Where it has potential to get shared without getting beaten down.

What are the fallacies though? Here is the one I can identify:

The type error of comparing desired "requirements" with desired "outcomes". "A teacher wants you to learn" is a specification of the teacher-function's desired outcome. On the other hand, "your stuff to break down" is a desired requirement of a repairer. A repairer's desired outcome is "your stuff to work again". Generally, requirements are "bad" and outcomes are "good" because the function is transformation of "bad" to "good". Any function can replace a teacher here ...
Static vs Dynamic Alignment
by Gracie Green — 2024-03-21T17:44:41.563Z

[heavily edited March 2025 to improve accuracy and reading style.]

Abstract

This paper examines a critical distinction in AI alignment approaches: static alignment, where AI models maintain fixed goals from their training period, vs dynamic alignment, where models continuously update to match current human preferences. I argue that dynamic alignment, which allows AI systems to adapt to evolving human values, presents significant advantages over static alignment despite still facing many challenges. The analysis focuses on three key areas: the impact of particular alignment styles on model behaviour, the implications for human autonomy and the privacy concerns inherent in preference monitoring. By examining how these models interact with human agency and value development, I demonstrate that dynamic alignment better preserves human autonomy while raising important privacy considerations. The paper also explores technical challenges specific to dynamic alignment, including the difficulty of accurately interpreting human preferences and protecting against value manipulation. While dynamic alignment appears theoretically preferable, significant research is needed to ensure its practical implementation preserves both autonomy and privacy. This work concludes by identifying critical areas for future research in developing and verifying dynamic alignment systems.

Introduction

‘AI alignment’ is often defined as aligning AI models with human values. But there’s an ambiguity here. When referring to AI alignment, it is often unclear whether an author is referring to de re or de dicto concepts of alignment with human values. This is an important distinction to make as it can alter the consequences of a model’s deployment, as I will highlight throughout this paper. Post-deployment, they each exhibit different behavioural patterns and thus respond to particular risks in differing ways.

De re and de dicto are usually used within philosophy to refer to a specific referent of a word and the general category the word refers to respectively. For example, if an individual requests ‘the plant with the most flowers on the table’, they may mean a specific plant that is best described as being the plant with the most flowers (this being de re) or they may mean any plant that fits this description (de dicto). Importantly, in the de re case, the plant they wa...
On green
by Joe Carlsmith — 2024-03-21T17:38:56.295Z

(Cross-posted from my website. Podcast version here, or search for "Joe Carlsmith Audio" on your podcast app.

This essay is part of a series that I'm calling "Otherness and control
in the age of AGI." I'm hoping that the individual essays can be read
fairly well on their own, but see
here
for brief summaries of the essays that have been released thus far.

Warning: spoilers for Yudkowsky's "The Sword of the
Good.")

"The Creation" by Lucas Cranach (image source
here)

The colors of the wheel

I've never been big on personality typologies. I've heard the
Myers-Briggs
explained many times, and it never sticks. Extraversion and
introversion, E or I, OK. But after that merciful vowel—man, the
opacity of those consonants, NTJ, SFP... And remind me the difference
between thinking and judging? Perceiving and sensing? N stands for
intuition?

Similarly, the
enneagram.
People hit me with it. "You're an x!", I've been told. But the faces
of these numbers are so blank. And it has so many kinda-random-seeming
characters. Enthusiast, Challenger, Loyalist...

The
enneagram.
Presumably more helpful with some memorization...

Hogwarts houses—OK, that one I can remember. But again: those are our
categories? Brave, smart, ambitious, loyal? It doesn't feel very
joint-carving...

But one system I've run into has stuck with me, and become a reference
point: namely, the Magic the Gathering Color Wheel. (My relationship to
this is mostly via somewhat-reinterpreting Duncan Sabien's presentation
here,
who credits Mark
Rosewater
for a lot of his understanding. I don't play Magic myself, and what I
say here won't necessarily resonate with the way people-who-play-magic
think about these colors.)

Basically, there are five colors: white, blue, black, red, and green.
And each has their own schtick, which I'm going to crudely summarize as:

White: Morality.

Blue: Knowledge.

Black: Power.

Red: Passion.

Green: ...well, we'll get to green.

To be clear: this isn't, quite, the summary that Sabien/Rosewater would
give. Rather, that summary looks like this:

(Image credit: Duncan Sabien
here.)

Here, each color has a goal (peace, perfection, satisfaction, etc) and a
default strategy (order, knowledge, ruthlessness, etc). And in the full
system, which you don't need to track, each has a characteristic set of
disagreements with the colors opposite to it...

The disagreements. (Image credit: Duncan Sabien
here...
Comparing Alignment to other AGI interventions: Extensions and analysis
by Martín Soto — 2024-03-21T17:30:50.747Z

In the last post I presented the basic, bare-bones model, used to assess the Expected Value of different interventions, and especially those related to Cooperative AI (as distinct from value Alignment). Here I briefly discuss important enhancements, and our strategy with regards to all-things-considered estimates.

I describe first an easy but meaningful addition to the details of our model (which you can also toy with in Guesstimate).

Adding Evidential Cooperation in Large worlds

Due to evidential considerations, our decision to forward this or that action might provide evidence about what other civilizations (or sub-groups inside a civilization similar to us) have done. So for example us forwarding a higher aC|V should give us evidence about other civilizations doing the same, and this should alter the AGI landscape.

But there's a problem: we have only modelled singletons themselves (AGIs), not their predecessors (civilizations). We have, for example, the fraction FV of AGIs with our values. But what is the fraction cV of civilizations with our values? Should it be higher (due to our values being more easily evolved than trained), or lower (due to our values being an attractor in mind-space)?

While a more complicated model could deal directly with these issues by explicitly modelling civilizations (and indeed this is explored in later extensions), for now we can pull a neat trick that gets us most of what we want without enlarging the ontology of the model further, nor the amount of input estimates.

Assume for simplicity alignment is approximately as hard for all civilizations (both in cV and c¬V=1−cV), so that they each have pV of aligning their AGI (just like we do). Then, pV of the civilizations in cV will increase FV, by creating an AGI with our values. And the rest 1−pV will increase F¬V. What about c¬V? pV of them will increase F¬V. But the misalignment case is trickier, because it might be a few of their misaligned AGIs randomly have our values. Let's assume for simplicity (since FV and cV are usually small enough) that the probability with which a random misaligned (to its creators) AGI has our values is the same fraction that our values have in the universe, after all AGIs have been created: FV.[1] Then, c¬...
The Comcast Problem
by RamblinDash — 2024-03-21T16:46:50.115Z

I've noticed that a lot of companies provide really valuable services yet almost inevitably are hated by consumers. I call this "the Comcast Problem," though it isn't limited to Comcast. Companies face this problem when they provide access to things consumers want, but they aren't themselves the goal.

When I consume great internet content, I get warm vibes from whatever site provided the content, whether it's instant free access to the world's knowledge on Wikipedia, thought-provoking discussion on LessWrong, great TV and movies on [whatever streaming platform], or anything else. Conversely, whenever I have problems accessing that stuff, I am quick to blame my ISP (which is in fact Comcast, though it could be anyone). So Comcast is stuck with zero credit for when it provides me with near-instant access to an almost infinite amount of great content (much of it for free[1]), but major blame for the small % of the time when it doesn't.

Similar dynamics exist for airlines and rental car companies - when I take a great vacation it never occurs to me to think "wow, good thing this company was able to provide (mostly) reliable and (mostly) affordable transportation for me!" But they get the blame when there's a problem.

None of this is to suggest that these companies are great, that they couldn't improve, or that they shouldn't fix whatever problems exist. But they are unfortunately stuck with asymmetric vibes, where the problems are their fault but good things come from others.

I claim this is a failure mode, and that these companies' status and image is lower than it "should" be.

^

Yes I'm paying a flat fee to my ISP, and then much of the content is provided by third parties for no (additional/marginal) charge.