trentmkelly/LessWrong-43k huggingface dataset:
Making Nanobots isn't a one-shot process, even for an artificial superintelligance
by dankrad — 2023-04-25T00:39:24.520Z

Summary: Yudkowsky argues that an unaligned AI will figure out a way to create self-replicating nanobots, and merely having internet access is enough to bring them into existence. Because of this, it can very quickly replace all human dependencies for its existence and expansion, and thus pursue an unaligned goal, e.g. making paperclips, which will most likely end up in the extinction of humanity.

I however will write below why I think this description massively underestimates the difficulty in creating self-replicating nanobots (even assuming that they are physically possible), which requires focused research in the physical domain, and is not possible without involvement of top-tier human-run labs today.

Why it matters? Some of the assumptions of pessimistic AI alignment researchers, especially by Yudkowsky, rest fundamentally on the fact that the AI will find quick ways to replace humans required for the AI to exist and expand.

We have to get AI alignment right the first time we build a Super-AI, and there are no ways to make any corrections after we've built it

As long as the AI does not have a way to replace humans outright, even if its ultimate goal may be non-aligned, it can pursue proximate goals that are aligned and safe for it to do. Alignment research can continue and can attempt to make the AI fully aligned or shut it down before it can create nanobots.

The first time we build a Super-AI, we don't just have to make sure it's aligned, but we need it to perform a pivotal act like create nanobots to destroy all GPUs

I argue below that this framing may be bad because it means performing one of the most dangerous steps first — creating nanobots — which may be best performed by an AI that is much more aligned than a first attempt

What this post is not about: I make no argument about the feasibility of (non-biological) self-replicating nanobots. There may be fundamental reasons why they are impossible/difficult (even for superintelligent AIs)/will not outcompete biological life, an interesting question that is explored more by bhaut. I also don't claim that AI alignment doesn't matter. I think that it's extremely important, but I think it's unlikely that (1) a one-shot process will lead to it and also (2) that one-shot is necessary; I actually think that this kind of thinking increases risk.

Finally, I don't claim that there aren't easier ways to kill all, or a...
Mental Models Of People Can Be People
by Nox ML — 2023-04-25T00:03:59.911Z

Warning: The argument in this post implies bad things about reality, and is likely to be bad for your mental health if it convinces you. If you're not sure if you can handle that, consider skipping this post for now and coming back later.

Introduction

At my best, when I used to write, I would write characters first by thinking about their personality, beliefs, thoughts, emotions, and situation. Then I would ask myself "what does this person do?", and let mysterious processes inside my brain automatically come up with the behavior of that character.

So this is how I define a mental model of a person: the collection of processes and information inside a brain that generates the behavior of some character.

My argument in this post is that there do exist mental models of people that are sufficiently detailed to qualify as conscious moral patients; I also argue that this is common enough that authors good at characterization probably frequently create and destroy such people; finally, I argue that this is a bad thing.

Part 1: On Data

I observe that most of my conscious experience corresponds quite neatly to things we would call data in a computer. Sight and hearing are the most obvious ones, corresponding to computer image and audio files. But it's not too hard to imagine that with advancing robotics and AI, it might be possible to also encode analogues to tactile sensation, smells, or even feelings on a computer.

In fact, given that I seem to be able to reason about all my conscious experience, and that reasoning is a form of computation, then it seems that all my conscious experience must necessarily correspond to some form of data in my brain. If there was no data corresponding to the experience, then I couldn't reason about it.

So we have one mysterious thing called "conscious experience" which corresponds to data. Yet we don't know much about the former, not even if it has any effect on the world beyond the data it corresponds to. So wouldn't it be simpler if we got rid of the distinction, and considered them one and the same? A hypothesis like this, where conscious experience is the data it corresponds to, would have an edge over other possibilities, due to being simpler.

This theory is not yet complete, however, because part of what makes data what it is, is how it is used. We say that the pixel #FF0000 represents red because when a usual monitor displays it, i...
Progress links and tweets, 2023-04-24
by jasoncrawford — 2023-04-24T21:17:28.718Z
Link: https://rootsofprogress.org/links-and-tweets-2023-04-20

The Progress Forum

Atlas Fellowship: $10k scholarship + 11-day program for high schoolers
AMA with Bryan Bishop has concluded

Opportunities

Erik Torenberg hiring a “researcher in residence”

Links

What a simple toy could tell the past about their future
IRBs save a few people while killing 10,000–100,000 more and costing $1.6B
Science depends on promoting good outliers, not preventing bad outliers
Breeding mosquitoes to prevent disease (strong XKCD 938 energy)
New, high-grade cache of rare earth elements was discovered recently
SF mayor introduces wide-reaching housing reforms (via @anniefryman who has a good explainer thread). And how to build a house in a day (via @Vernon3Austin)
Alex Tabarrok on Costo, channeling Voltaire

Quotes

Freeman Dyson: “Nobody builds reactors for fun anymore”
Why it’s more important to “raise the ceiling” rather than the “floor”
“Bureaucracies temporarily suspend the Second Law of Thermodynamics”

Queries

If you heard the term “intellectual entrepreneur,” what would you think it means?
Why is a protein’s shape so important?
Is there a dataset of very granular TFP by industry?
What’s an example of a proactive regulation or policy that worked well?
Is anyone working on reducing the modern bureaucratic workload on researchers?

AI

The increased supply of intelligence will create more demand for tasks that require intelligence (via @packyM)
Observations on two years of programming with GitHub Copilot
A Turing test would be a dynamic, evolving, adversarial competition
“GPT agents” are cool experiments, but the demos look sort of fake?
Freeman Dyson gave a very short answer to the 2015 Edge question

AI safety

Both AI doomers and anti-doomers believe that the other side has a weird conjunction of many dubious arguments. Related, a critique of pure reason
Something the government can do now about AI safety: define liability
Scott Aaronson on what to do now about AI risk
Scott Alexander on “not overshooting and banning all AI forever”
Norbert Wiener was concerned about machine alignment problems in the 1960s
Will AI be safer if it has cost/efficiency constraints?
What are the implications of LLMs for AI interpretability?
Can Man create a machine so smart he can’t turn it off?

Other tweets

Rocket launches as our aesthetic contribution for the age. (Compare to this painting)
Job posting for a position at Shockley Transistor, 1960
John Quincy Adams: hero of progress?
On sending your lif

...
Ideas for AI labs: Reading list
by Zach Stein-Perlman — 2023-04-24T19:00:00.832Z

Related: AI policy ideas: Reading list.

This document is about ideas for AI labs. It's mostly from an x-risk perspective. Its underlying organization black-boxes technical AI stuff, including technical AI safety.

Lists & discussion

Towards best practices in AGI safety and governance: A survey of expert opinion (GovAI, Schuett et al. 2023) (LW)

This excellent paper is the best collection of ideas for labs. See pp. 18–22 for 100 ideas.

Frontier AI Regulation: Managing Emerging Risks to Public Safety (Anderljung et al. 2023)

Mostly about government regulation, but recommendations on safety standards translate to recommendations on actions for labs

Model evaluation for extreme risks (DeepMind, Shevlane et al. 2023)
What AI companies can do today to help with the most important century (Karnofsky 2023) (LW)
Karnofsky nearcasting: How might we align transformative AI if it’s developed very soon?, Nearcast-based "deployment problem" analysis, and Racing through a minefield: the AI deployment problem (LW) (Karnofsky 2022)
Survey on intermediate goals in AI governance (Räuker and Aird 2023)
Corporate Governance of Artificial Intelligence in the Public Interest (Cihon, Schuett, and Baum 2021) and The case for long-term corporate governance of AI (Baum and Schuett 2021)
Three lines of defense against risks from AI (Schuett 2022)
The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation (Brundage et al. 2018)
Adapting cybersecurity frameworks to manage frontier AI risks: defense in depth (IAPS, Ee et al. 2023)

Levers

AI developer levers

and

AI industry & academia levers

in

Advanced AI governance

(LPP, Maas 2023)

This report is excellent

"Affordances" in "Framing AI strategy"

(Stein-Perlman 2023)

This list may be more desiderata-y than lever-y

Desiderata

Maybe I should make a separate post on desiderata for labs (for existential safety).

Six Dimensions of Operational Adequacy in AGI Projects (Yudkowsky 2022)
"Carefully Bootstrapped Alignment" is organizationally hard (Arnold 2023)
Slowing AI: Foundations (Stein-Perlman 2023)
[Lots of stuff implicated elsewhere, like "help others act well" and "minimize diffusion of your capabilities research"]

Ideas

Coordination[1]

See generally The Role of Cooperation in Responsible AI Development (Askell et al. 2019).

Coordinate to not train or deploy dangerous AI

Model evaluations

Model evaluation for extreme risks (DeepMin

...
Deep learning models might be secretly (almost) linear
by beren — 2023-04-24T18:43:28.188Z

Crossposted from my personal blog.

Epistemic status: Pretty speculative, but there is a surprising amount of circumstantial evidence.

I have been increasingly thinking about NN representations and slowly coming to the conclusion that they are (almost) completely secretly linear inside[1]. This means that, theoretically, if we can understand their directions, we can very easily exert very powerful control on the internal representations, as well as compose and reason about them in a straightforward way. Finding linear directions for a given representation would allow us to arbitrarily amplify or remove it and interpolate along it as desired. We could also then directly 'mix' it with other representations as desired. Measuring these directions during inference would let us detect the degree of each feature that the network assigns to a given input. For instance, this might let us create internal 'lie detectors' (which there is some progress towards) which can tell if the model is telling the truth, or being deceptive. While nothing is super definitive (and clearly networks are not 100% linear), I think there is a large amount of fairly compelling circumstantial evidence for this position. Namely:

Evidence for this:

All the work from way back when about interpolating through VAE/GAN latent space. I.e. in the latent space of a VAE on CelebA there are natural 'directions' for recognizable semantic features like 'wearing sunglasses' and 'long hair' and linear interpolations along these directions produced highly recognizable images
Rank 1 or low rank editing techniques such as ROME work so well (not perfectly but pretty well). These are effectively just emphasizing a linear direction in the weights.
You can apparently add and mix LoRas and it works about how you would expect.
You can merge totally different models. People working with Stable Diffusion community literally additively merge model weights with weighted sum and it works!
Logit lens works.
SVD directions.
Linear probing definitely gives you a fair amount of signal.
Linear mode connectivity and git rebasin.
Colin Burns' unsupervised linear probing method works even for semantic features like 'truth'.
You can merge together different models finetuned from the same initialization.
You can do a moving average over model checkpoints and this improves performance and is better than any individual checkpoint!
The fact

...
The Toxoplasma of AGI Doom and Capabilities?
by Robert_AIZI — 2023-04-24T18:11:41.576Z

[Epistemic Status: I'm confident that the individual facts I lay out support the main claim, but I'm not fully confident its enough evidence to make a true or useful framework for understanding the world.]

I'm going to give seven pieces of evidence to support this claim[1]:

AI Doomerism helps accelerate AI capabilities, and AI capabilities in turn proliferate the AI Doomerism meme.

If these dynamics exist, they'd be not unlike the Toxoplasma of Rage. Here's my evidence:

Sam Altman claims Eliezer "has IMO done more to accelerate AGI than anyone else":
Technical talent who hear about AI doom might decide capabilities are technically sweet, or a race, or inevitable, and decide to work on it for those reasons (doomer -> capabilities transmission).
Funders and executives who hear about AI doom might decide capabilities are a huge opportunity, or disruptive, or inevitable, and decide to fund it for those reasons (doomer -> capabilities transmission).
Capabilities amplifies the memetic relevance of doomerism (capabilities -> doomer transmission).
AI Doomerism says we should closely follow capabilities updates, discuss them, etc.
Capabilities and doomerism gain and lose social status together - Eliezer Yudkowsky has been writing about doom for a long time, but got a Time article and TED talk only after significant capabilities advances.
Memes generally benefit from conflict, and doomerism and capabilities can serve as adversaries for this purpose.

I've been trying to talk about "AI doomerism" here as a separate meme than "AI safety", respectively something like "p(doom) is very large" and "we need to invest heavily into AI safety work", though these are obviously related and often cooccur. One could no doubt make a similar case for AI safety and capabilities supporting each other, but I think the evidence I listed above applies mostly to AI doom claims (if one uses Eliezer as synecdoche for AI doomerism, which I think is reasonable).

I hope with this post I'm highlighting a something that is a combination of true and useful. Please keep in mind that the truth values of "AI doom is in a toxoplasma relationship with AI capabilities" and "AI doom is right" are independent.

^

This post was inspired by one striking line Jan_Kulveit's helpful Talking publicly about AI risk:

- the AGI doom memeplex has, to some extent, a symbiotic relationship with the race toward A

...
Measures of Internet Virality and News Popularity
by T431 — 2023-04-24T17:43:27.297Z

I am not sure exactly what form the resource I am looking for may take (I am believing more and more than I am jargon-gated in my search endeavors), but I am looking for something that tracks viral phenomena over time and that maybe also rates their virality levels.

As an example, such a resource might be a timeline of News stories along with their corresponding "attention index" and for how long / how much the public attended to them.

I am not looking for something that, for example, is an ordered list of a social media platform's top posts, but rather for something that is a collection of stories / narratives (across social media platforms) experienced by the West or wider humanity. I imagine that such a resource would likely list Monkeypox / Mpox as having fallen out of the public's attention for the most part, and would likely also have Ongoing Conflict in Ukraine as still "in the News" but with a lower "attention index" now than in the past.

My apologies if there is some standard resource people go to for these things, but I have not been able to find one. Also, thank you for taking the time to read this.
A concise sum-up of the basic argument for AI doom
by Mergimio H. Doefevmil — 2023-04-24T17:37:53.063Z

1 - An artificial super-optimizer is likely to be developed soon.

2 - There is no known way of programming goals into an advanced optimizer, only outwardly observable behaviors of which we have no idea why they are being carried out or what motivates them.

3 - Most utility functions do not have optima with humans in them. Most utility functions do not have a term for humans at all.

4 - "Why haven’t we exterminated all mice/bugs/cows then?" draws quite a poor analogy. Firstly, we are not superoptimizers. Secondly, and more importantly, we care about living beings somewhat. The optimum of the utility function of the human civilization quite possibly does have mice/bugs/cows, perhaps even genetically engineered to not experience suffering. We are not completely indifferent to them.

The relationship between most possible superoptimizers and humanity is not like the relationship between humanity and mice at all - it is much more like the relationship between humanity and natural gas. Natural gas, not mice, is a good example of something humans are truly indifferent to - there is no term for it in our utility function. We don’t hate it, we don’t love it, it is just made of atoms that we can, and do, use for something else.

Moreover, the continued existence of natural gas probably does not pose the same threat to us as the continued existence of us would pose to a superoptimizer which does not have a term for humans in its utility function - just like we don’t have a term for natural gas in ours. Natural gas can not attempt to turn us off, and it can not create "competition" for us in form of a rival species with capabilities similar to, or surpassing, our own.

P.S. If you don’t like or find confusing the terminology of "optimizers" and "utility functions", feel free to forget about all that. Think instead of physical states of the universe. Out of all possible states the universe could find itself in, very, very, very few contain any humans. Given a random state of the universe, creating a superintelligence that steered the universe towards that state would result in an almost guaranteed apocalypse. Of course, we want our superintelligence to steer the universe towards particular states - that’s kind of the whole point of the entire endeavor. Were it not so, we would not be attempting to build a superintelligence - a sponge would suffice. We essentiall...
A response to Conjecture's CoEm proposal
by Kristian Freed — 2023-04-24T17:23:38.694Z

The following is written based on what was shared in

the CoEm announcement previously posted here

and also discussed in depth by Connor in

a recent YouTube interview with the Future of Life Institute.

What is a CoEm system anyway?

The current paradigm of using LLMs (and increasingly sophisticated monolithic systems that move beyond language in the sense of written human language, including code and images) rely on black boxes and Magic.

We fundamentally do not know how these things work, what the bounds of capabilities are, what the effects of “nudges” such as RLHF are beyond the specific cases being trained on, or how capabilities will change with larger models or more data. I’m talking here about qualitative capabilities such as chain of thought reasoning appearing at certain complexity, rather than predicting loss by predetermined training objectives. As such, we currently do not (and possibly never will) have any way of proving that these systems are safe.

On the flip side, we have at least an existence proof of a form of intelligence sufficiently advanced to do useful things (such as conducting research) but with for the most part limited motivation to act maliciously. When such tendencies do exist, we have many existing barriers in place that limit individuals with this form of intelligence from doing significant harm at scale.

This existence proof of course is us, humans, and importantly the existing societal structures that are set up to deal with individuals of human-level intelligence and capabilities. It is no easier to prove that an individual human is “safe” or “aligned” than it is to do the same about a large black box AI model, but we do have a much better empirical understanding of the way in which humans can pose threats and the bounds within which their abilities lie.

The case then for CoEms is to build systems that do not only possess “human level” intelligence but more specifically “human-like” intelligence - machines that think not just “as well” as us but in the same way that we do.

While this is clearly doable in principle (again, we exist), it’s not necessarily obvious that the creation of such a system is feasible or possible to complete within the timeframes determined by the race to create human-level (or above) intelligence accomplished through black box methods. As with any form of research, there are no guaranteed resul...
Camaraderie at scale: in search of shared identity
by eq — 2023-04-24T16:46:43.479Z

Originally published on substack: https://eliqian.substack.com/camaraderie

This tweet is an exaggeration, but only barely:

https://twitter.com/DrifterShoots/status/1515095665831026689?s=20

You yourself probably remember waking up during that time and just feeling different. I'd like to propose a new term—camaraderie at scale—to describe these moments.

This got me thinking—when has camaraderie occurred at scale before, and what made those moments so special? How can we engineer more of them into our lives?

Some notable ones in recent memory: Linsanity, 2008 Beijing Olympics.

Some notable ones (or so I'm told) that I'm too young to remember: 9/11,

Sosa vs. McGwire

, the Moon landing.

Camaraderie at scale is extremely powerful. Few things in the world can mobilize and unite entire nations, or in some instances, the whole planet. I spent some time researching and reliving past moments in an effort to document the factors that made them so special.

Side quest: origins of camaraderie

It's worth exploring the word camaraderie and its closely related comrade. Comrade has origins in French, Italian, and Spanish words that all mean something along the lines of “one who shares the same room.”

I really like this definition because it conveys a sense of being in something together. A comrade is someone sympathetic to your cause—bound by an ideology, but not necessarily similar or connected in any other way.

Camaraderie is about some connection that brings together otherwise disparate people. I think the key concept behind this is identity, and how moments can change who we identify as.

Micro-identities & micro-communities

Everyone has an identity—a label they give themselves that connects them to a concept, group, or ideology. Most people have multiple identities. You might identify as a student, but also as a writer, a person who loves to cook, and a Taylor Swift fan. Most people also have layers of identity. I identify as Chinese, but also Asian American. I love basketball, but also the 76ers, and I’m a big fan of Tyrese Maxey.

Labels and identities are very influential. They dictate what tribes we belong to, whose ideas we support, and how we perceive others. And for most of our lives, we are stuck in various “micro-identities”—extremely niche labels with limited membership.

You don’t identify with your high-level identities, such as “human” or “woman” as often as your lower-leve...