trentmkelly/LessWrong-43k huggingface dataset:
Ten arguments that AI is an existential risk
by KatjaGrace — 2024-08-13T17:00:03.397Z

This is a snapshot of a new page on the AI Impacts Wiki.

We’ve made a list of arguments[1] that AI poses an existential risk to humanity. We’d love to hear how you feel about them in the comments and polls.

Competent non-aligned agents

Humans increasingly lose games to the best AI systems. If AI systems become similarly adept at navigating the real world, will humans also lose out? (Image: Midjourney)

Summary:

Humans will build AI systems that are 'agents', i.e. they will autonomously pursue goals
Humans won’t figure out how to make systems with goals that are compatible with human welfare and realizing human values
Such systems will be built or selected to be highly competent, and so gain the power to achieve their goals
Thus the future will be primarily controlled by AIs, who will direct it in ways that are at odds with long-run human welfare or the realization of human values

Selected counterarguments:

It is unclear that AI will tend to have goals that are bad for humans
There are many forms of power. It is unclear that a competence advantage will ultimately trump all others in time
This argument also appears to apply to human groups such as corporations, so we need an explanation of why those are not an existential risk

People who have favorably discussed[2] this argument (specific quotes here): Paul Christiano (2021), Ajeya Cotra (2023), Eliezer Yudkowsky (2024), Nick Bostrom (2014[3]).

See also: Full wiki page on the competent non-aligned agents argument

Second species argument

An orangutan uses a stick to control juice, while humans use complex systems of tools, structures, and behavioral coordination to control the orangutan. Should orangutans have felt safe inventing humans, if they had had the choice? (Image: William H. Calvin)

Summary:

Human dominance over other animal species is primarily due to humans having superior cognitive and coordination abilities
Therefore if another 'species' appears with abilities superior to those of humans, that species will become dominant over humans in the same way
AI will essentially be a 'species' with superior abilities to humans
Therefore AI will dominate humans

Selected counterarguments:

Human dominance over other species is plausibly not due to the cognitive abilities of individual humans, but rather because of human ability to communicate and store information through culture and artifacts
Intelligence in animal

...
Eugenics And Reproduction Licenses FAQs: For the Common Good
by Zero Contradictions — 2024-08-13T16:34:21.890Z
Link: https://zerocontradictions.net/faqs/eugenics

1. What Is Eugenics?

There are various ways to define eugenics, as it is used in this FAQs:

Eugenics is the deliberate social regulation of the genome.
Eugenics is about consciously selecting for traits that make people better members of society.
Eugenics is about selecting for traits that we value in other human beings, such as intelligence, health, and responsibility; and selecting against traits that we negatively value, such as irresponsibility and propensity to violence.
Eugenics selects for traits that are socially positive and selects against traits that are socially negative.
Eugenics is the self-direction of human evolution. – This definition is historically notable for being a slogan of the International Eugenics Conference, but it’s too vague and inaccurate. In a sense, one could argue that the unregulated reproduction that is occurring today is the “self-direction of human evolution”. The eugenics that we’re proposing must be directed by society.
Eugenics is selective breeding or artificial selection, as managed by society. However, “selective breeding” tends to apply to crops, plants, animals, and other organisms used by humans, rather than humans. Another important distinction is that “selective breeding” for non-humans tends to have connotations with top-down eugenics, whereas we’re proposing laissez-faire eugenics for humans.

Regardless, what is considered “eugenic” is always relative to a given environment and a set of values. If the environment in question is not specified, it’s usually assumed to be Modernity or the society that one lives in. No matter what we do, the social environment places selective pressures on the human genome. Pro-social selection is also a feedback loop.

In a biological context that doesn’t involve humans, “eugenic” is usually defined with respect to biological value, from the perspective of an organism. In which case, eugenics is the improvement of genetic qualities through selection, to make organisms more adaptive to their relative environments. But that is not the definition of eugenics that will be used in this FAQs. When we and most other people are talking about eugenics with respect to humans, eugenics is defined according to social value and modern civilization. We believe that eugenics will be necessary to sustain modern civilization.

Different people each define “eugenics” differently, with the broadest (and dare ...
Superintelligent AI is possible in the 2020s
by HunterJay — 2024-08-13T06:03:26.990Z

Back in June 2023, Soroush Pour and I discussed AI timelines on his podcast, The AGI Show. The biggest difference between us was that I think “machines more intelligent than people are likely to be developed within a few years”, and he thinks that it’s unlikely to happen for at least a few decades.[1]

We haven’t really resolved our disagreement on this prediction in the year since, so I thought I would write up my main reasons for thinking we’re so close to superintelligence, and why the various arguments made by Soroush (and separately by Arvind Narayanan and Sayash Kapoor) aren’t persuasive to me.

Part 1 - Why I Think We Are Close

Empirically

You can pick pretty much any trend relating to AI & computation, and it looks like this:[2]

We keep coming up with new benchmarks, and they keep getting saturated. While there are still some notable holdouts such as ARC-AGI, SWE-Bench, and GPQA, previous holdouts like MATH also looked like this until they were solved by newer models.

If these trends continue, it’s hard to imagine things that AI won’t be able to do in a few years time[3], unless they are bottlenecked by regulation (like being a doctor), or by robot hardware limitations (like being a professional football player)[4].

Practically

The empirical trends are the result of several different factors; changes in network architecture, choice of hyperparameters, optimizers, training regimes, synthetic data creation, and data cleaning & selection. There are also many ideas in the space that have not been tried at scale yet. Hardware itself is also improving -- chips continue to double in price performance every 2-3 years, and training clusters are scaling up massively.

It’s entirely possible that some of these trends slow down -- we might not have another transformers-level architecture advance this decade, for instance -- but the fact that there are many different ways to continue improving AI for the foreseeable future makes me think that it is unlikely for progress to slow significantly. If we run out of text data, we can use videos. If we run out of that, we can generate synthetic data. If synthetic data doesn’t generalise, we can get more efficient with what we have through better optimisers and training schedules. If that doesn’t work, we can find architectures which learn the patterns more easily, and so on.

I...
Debate: Get a college degree?
by Ben Pace — 2024-08-12T22:23:34.744Z

Epistemic Status:

Soldier mindset. These are not (necessarily) our actual positions, these are positions we were randomly assigned by a coin toss, and for which we searched for the strongest arguments we could find, over the course of ~1hr 45mins. That said, this debate is a little messy between our performed positions and our personal ones.

Sides:

Ben is arguing

against

getting a college degree, and Saul is arguing

for

. (This is a decision Saul is currently making for himself!)

Reading Order:

Ben and Saul drafted each round of statements

simultaneously.

This means that each of Ben's statements you read were written without Ben having read Saul's statements that are immediately proceeding. (This does not apply to the back-and-forth interview.)

Saul's Opening Statement

Saul Munn

first — i do think there’s a qualitative difference between the position “getting an undergrad degree is good” vs “getting the typical undergrad experience is good.” i think the second is in some ways more defensible than the first, but in most ways less so.

For “getting the typical undergrad experience is good”

This sort of thing is a strong Chesterton fence. People have been having the typical experience of an undergrad for a while (even while that typical experience changes).
General upkeeping of norms/institutions is good.
I think that — for a some ppl — their counterfactual is substantially worse. Even if this means college is functionally daycare, I’d rather they be in adult-day-care than otherwise being a drain on society (e.g. crime).
It presents the option for automatic solutions to a lot of problems:

Socializing

high density of possible friends, romantic partners, etc
you have to go to classes, talk to ppl, etc

Exercise

usually a free gym that’s at-least functional
you gotta walk to class, dining hall, etc

Tons of ability to try slightly “weird” stuff you’ve never tried before — clubs, sports, events, greek life, sexual interactions, classes, etc
I think a lot of these things get a lot more difficult when you haven’t had the opportunity to experiment w them. A lot of ppl haven’t experimented w much of anything before — college gives them an easy opportunity to do that w minimal friction before doing so becomes gated behind a ridiculous amount of friction. E.g. getting into a new hobby as an adult is a bit odd, in most social settings — but in college, it’s literally as simple as joining

...
Extracting SAE task features for in-context learning
by Dmitrii Kharlapenko — 2024-08-12T20:34:13.747Z

TL;DR

We try to study task vectors in the SAE basis. This is challenging because there is no canonical way to convert an arbitrary vector in the residual stream to a linear combination of SAE features — you can't just pass an arbitrary vector through the encoder without going off distribution.
We explored the algorithm of gradient pursuit suggested in Smith et al, but it didn’t work for us without modifications.
Our approach is to apply the SAE encoder to the task vector, and then apply a gradient-based cleanup. This exploits the fact that task vectors have a differentiable objective. We find that this gives a sparser and cleaner reconstruction, which is also highly interpretable, and also serves as a better task vector due to directly optimizing for log likelihood. This takes us from ~100 active features to ~10.
Using our algorithm, we find two classes of SAE features involved in ICL. One of them recognizes the exact tasks or output formats from the examples, and another one encodes the tasks for execution by the model later on. We show that steering with these features has causal effects similar to task vectors.

This work was produced as part of the ML Alignment & Theory Scholars Program - Summer 24 Cohort, under mentorship from Neel Nanda and Arthur Conmy.

Prior work

Task or function vectors are internal representations of some task that LLMs form while processing an ICL prompt. They can be extracted from a model running on a few-shot prompt and then be used to make it complete the same task without having any prior context or task description.

Several papers (Function vectors in large language models, In-Context Learning Creates Task Vectors) have proposed different ways to extract those task vectors. They all center around having ICL examples being fed to a model in the form of “input <separator> output, … ” and averaging the residuals on the “separator” token over a batch. This approach can reconstruct some part of the ICL performance but does not admit a straightforward conversion to the SAE basis.

ITO with gradient pursuit can be used to do a sparse coding of a residual vector using SAE features. The post suggests using this algorithm for steering vector SAE decomposition. Since task vectors can be thought of as steering vectors, ITO may provide some insight into the ways they operate.

Initial Phi-3 experiments

Direct SAE t

...
Hyppotherapy
by Marius Adrian Nicoară — 2024-08-12T20:07:46.606Z

Cover image generated with Bing Image Generator: https://www.bing.com/images/create

Context: I was returning home from a meeting organized by a psychotherapist at a local bookstore.

It felt like the smell of horse that pulled my eyes to the pen rushed ahead to materialize into the restless stud before I could lay my eyes on it. He was eroding the ground into a ditch as he walked up and down the fence, oblivious to my whistling and the slightly not small meadow behind him. I think the Moon's first quarter was mapping its mind for me: he was patrolling the border between light and dark, with his thoughts stuck in the side with no mysteries. The obscurity behind him he did not even suspect. Seemed like he reminded me of someone.

His patience seemed to have thinned a bit because he stopped to let out a neigh that got a reply from his friends in the other pens. After listening to the roll call, he made a few steps and let out a second neigh, this one left unanswered. It seemed like his friends hadn't understood the first one.

I would have liked to explain to him that I was returning from a get-together where I had confessed as a limited belief the fact that I think I can explain anything to anyone. This was a problem, because I often wasted my time with people who were not willing to listen. But he probably wouldn't have understood me. And I would have proved that I hadn't understood his neigh. So I headed into the direction where the Moon pointed towards mystery, hoping that at least for him it would be enough to meet his kind without wanting to dull their brain with the same things he had dulled his own with.
Californians, tell your reps to vote yes on SB 1047!
by Holly_Elmore — 2024-08-12T19:50:09.817Z

See the event page here.

Hello Californians!

We need you to help us fight for SB 1047, a landmark bill to help set a benchmark for AI safety, decrease existential risk, and promote safety research. This bill has been supported by some of

the world’s leading AI scientists

and the Center of AI Safety, and is extremely important for us to pass. As Californians, we have a unique opportunity to inspire other states to follow suit.

SB 1047 has a hearing in the Assembly Appropriations Committee scheduled for August 15th. Unfortunately, due to

misinformation

and

lobbying

by big tech companies, the bill risks getting watered down or failing to advance. This would be a significant blow against safety and would continue the “race to the bottom” in AI capabilities without any guardrails.

We need you to do the following to save the bill. This will take no more than 5 minutes:

Email the Chair (Buffy Wicks, D) and Vice-Chair (Kate Sanchez, R) of the Appropriations Committee, and ask them to advance the bill without watering it down.

Buffy Wicks:

assemblymember.wicks@assembly.ca.gov

Kate Sanchez:

assemblymember.sanchez@assembly.ca.gov

Email templates and examples can be found here.

Call the offices of the Chair and Vice-Chair in support of the bill. The best day to do this is August 13 or 14.

Buffy Wicks (D): (916) 319-2014
Kate Sanchez (R): (916) 319-2071
These calls can be very short (~1 minute) and straightforward.

See here for more guidance

.

This document has additional information about the bill and other ways to help.

[But some of the dates are wrong! This post is up-to-date.]

Please try to get this done as soon as possible, and let us know if you need any help. Your voice matters, and it is urgent that we push this before it’s too late.

Thank you so much for your support!
[LDSL#5] Comparison and magnitude/diminishment
by tailcalled — 2024-08-12T18:47:20.546Z

This post is also available on my Substack.

In the previous post on LDSL, I argued that an important part of causal inference is to look at the larger things. This implies a method of comparison to rank things by magnitude. [1]

Homogenous comparison

When the things to rank are homogenous, ranking them tends to be easy, because one can set them up as opposing forces and see which one dominates. A classic example is ranking objects by weight, where an old-school balance scale makes the gravity of the objects oppose each other, thereby lifting up the lighter one.

Direct opposition is not the only method of measurement. Alternatively, one can set up the things to both influence a standardized measurement instrument, and then compare their influence on that instrument. For instance, to compare the weight of things, one can try lifting each thing, and feeling which one requires more force to lift.

Inhomogenous comparison

In order to apply the method of root cause analysis I suggested, one doesn’t just need to be able to compare equivalent things, but also to compare inequivalent things. For instance, to figure out the biggest factors to investigate for why an election went a certain way, it is not just enough to understand the vote counts in individual voting districts. It is also necessary to compare the importance across different kinds of issues, like scandals, politician positions, politician traits, and an endless number of domains that you wouldn’t even think about because they are obviously too unimportant to matter.

As far as I know, this sort of inhomogenous comparison inherently carries some subjectivity to it. What’s most, an apple or an orange? I can’t see it making sense to declare anything specific to be the correct answer here.

Yet, what’s most, a piece of dust or a tank? Obviously a tank. When you have qualitatively different things, but they differ by orders of magnitude, it seems much more likely that there’s a sensible ordering to things. A primitive way we could think about this is to imagine something like a principle component analysis over the different ways of ordering things, taking the overall “general factor”. Yet this is somewhat unsatisfying, since presumably the resulting order depends a lot on what comparison methods we include, and it’s unclear what the “meaning” of the resulting ordering is, causally speaking.

One solution that makes s...
In Defense of Open-Minded UDT
by abramdemski — 2024-08-12T18:27:36.220Z

A Defense of Open-Minded Updatelessness.

This work owes a great debt to many conversations with Sahil, Martín Soto, and Scott Garrabrant.

You can support my work on Patreon.

Iterated Counterfactual Mugging On a Single Coinflip

Iterated counterfactual mugging on a single coinflip begins like a classic counterfactual mugging, with Omega approaching you, explaining the situation, and asking for your money. Let's say you buy the classic UDT idea, so you happily give Omega your money.

Next week, Omega appears again, with the same question. However, Omega clarifies that it has used the same coin-flip as last week.

This throws you off a little bit, but you see that the math is the same either way; your prior still assigns a 50-50 chance to both outcomes. If you thought it was a good deal last week, you should also think it is a good deal this week. You pay up again.

On the third week, Omega makes the same offer again, and once again has used the same coinflip. You ask Omega how many times it's going to do this. Omega replies, "forever". You ask Omega whether it would have continued coming if the coin had landed heads; it says "Of course! How else could I make you this offer now? Since the coin landed tails, I will come and ask you for $100 every single week going forward. If the coin had landed heads, I would have simulated what would happen if it had landed tails, and I would come and give you $10,000 on every week that simulated-you gives up $100!"

Let's say for the sake of the thought experiment that you can afford to give Omega $100 once a week. It hurts, but it doesn't hurt as much as getting $10,000 from Omega every week would have benefited you, if that had happened.[1]

Nonetheless, I suspect many readers will feel some doubt creep in as they imagine giving Omega $100 week after week after week. The first few weeks, the possibility of the coin landing heads might feel "very real". Heck yeah I want to be the sort of person who gets a 50% chance of 10K from Omega for a (50% chance) cost of $100!

By the hundredth week, though, you may feel yourself the fool for giving up so much money for the imaginary benefit of the "heads" world that never was.

If you think you'd still happily give up the $100 for as long as Omega kept asking, then I would ask you to consider a counterlogical mugging instead. Rather than flipping a coin, Omega uses a digit of the binary expansion of&nbs...
Humanity isn't remotely longtermist, so arguments for AGI x-risk should focus on the near term
by Seth Herd — 2024-08-12T18:10:56.543Z

Toby Ord recently published a nice piece On the Value of Advancing Progress about mathematical projections of far-future outcomes given different rates of progress and risk levels. The problem with that and many arguments for caution is that people usually barely care about possibilities even twenty years out.

We could talk about sharp discounting curves in decision-making studies, and how that makes sense given evolutionary pressures in tribal environments. But I think this is pretty obvious from talking to people and watching our political and economic practices.

Utilitarianism is a nicely self-consistent value system. Utilitarianism pretty clearly implies longtermism. Most people don't care that much about logical consistency,[1] so they are happily non-utilitarian and non-longtermist in a variety of ways. Many arguments for AGI safety are longtermist, or at least long-term, so they're not going to work well for most of humanity.

This is a fairly obvious, but worth-keeping-in-mind point.

One non-obvious lemma of this observation is that much skepticism about AGI x-risk is probably based on skepticism about AGI happening soon. This doesn't explain all skepticism, but it's a significant factor worth addressing. When people dig into their logic, that's often a central point. They start out saying "AGI wouldn't kill humans" then over the course of a conversation it turns out that they feel that way primarily because they don't think real AGI will happen in their lifetimes. Any discussion of AGI x-risks isn't productive, because they just don't care about it.

The obvious counterpoint is "You're pretty sure it won't happen soon? I didn't know you were an expert in AI or cognition!" Please don't say this - nothing convinces your opponents to cling to their positions beyond all logic like calling them stupid.[2] Something like "well, a lot of people with the most relevant expertise think it will happen pretty soon. A bunch more think it will take longer. So I just assume I don't know which is right, and it might very well happen pretty soon".

It looks to me like discussing whether AGI might threaten humans is pretty pointless if the person is still assuming it's not going to happen for a long time. Once you're past that, it might make sense to actually talk about why you think AGI would be risky for humans.[3]

^

This is an aside, but you'll probably

...