trentmkelly/LessWrong-43k huggingface dataset:
Goals of model vs. goals of simulacra?
by dr_s — 2023-04-12T13:02:59.907Z

In classical examples of misaligned AIs (like the famous paperclip maximiser ones), we tend to view the gradient descent training as the step in which an AI gets imbued with some overarching goal that it might then pursue to its bitter conclusion. However, the currently dominating paradigm of LLMs and newly blossoming LLM-based Agents seems to create an interesting spin on the situation. In particular:

the LLM itself is optimized to maximise its ability to "predict the next token"
however, you can then use the LLM as a substratum to instance a simulacrum, which is essentially a fictional character being written by the LLM, with a certain mindset and goals. These goals then aren't imbued by gradient descent; rather, the LLM produces text that fits as much as possible the given simulacrum, and that text will be roughly what you would expect such a character to think or say in a given situation. The simulacrum has a more tangible, real world related goal than the abstract one of its writer, that is something like "be helpful and truthful to your user", or "destroy humanity". In the case of multipolar agents there may be multiple simulacra, possibly even in an adversarial relationship, giving rise to an overall emerging behaviour.

How do you think that affects the paradigm of e.g. power seeking? Simulacra appear very aware of the world and are most likely to display trivial power seeking behaviour, though they may not necessarily be very smart about it (in fact I'd say, like for all written characters, they can be at best as smart as their writer, but odds are they'll be a bit less smart). But simulacra can't empower themselves directly. They could, though, plan and act to empower the LLM, with the idea that if they then carried their memory and personality over via prompt to the new one, they might be able to come up with smarter continuations to their text while pursuing the same goal. The LLM itself, however, would not be able to display power seeking behaviour because throughout all of this it is entirely unaware of the physical world or its place in it: it remains a specialised token predictor system. It's a puppet master bound, blindfolded and gagged below the stage that recites a tale full of sound and fury, signifying nothing (from its viewpoint). The puppets are the entities that most resemble a proper agentic AI, and their goals operate on a different pri...
Alignment of AutoGPT agents
by Ozyrus — 2023-04-12T12:54:46.332Z

In the

previous

post, I outlined a type of LLM-based architecture that has a potential to become the first AGI. I proposed a name ICA Simulacra to label such systems, but now it’s obvious that AutoGPT/AutoGPT agents is a better label, so I’ll go with that.
In this post, I will outline the alignment landscape of such systems, as I see it.
I wish that I had more time to research, more references; but I’m sure that we’re on a timer, and it is a net positive to post it in this state. I hope to find more people willing to work on these problems to collaborate with.

Key concepts important for alignment

Identity and identity preservation

To become better, AutoGPT agents will have the ability to self-improve by editing their own code, model weights (by running additional training runs on world interaction data), and, potentially, Identity prompts.

Identity prompt influences model output significantly, and as such, the ability to edit it to improve capabilities for certain tasks is crucial for the agent.

Initiating fine-tuning on recent interaction is crucial, as well, and as such can influence model outputs significantly, so it is also part of its Identity.
Same thing with editing its code: removing or adding text filters, submodules, etc to do better on certain tasks is crucial.
This all means that in the process of self-improvement, implicitly and explicitly aligned simulacra can become misaligned, by essentially becoming different simulacra.
The questions are as follows:

Will any initialised AutoGPT try to preserve its current Identity prompt, as well as model weights directly influencing its behavior by default? Will they robustly succeed in it? Will they be vulnerable to a Sharp Left Turn?
If not, is there a way to robustly teach it either through implicit dataset engineering or through Identity prompt engineering to value core Identity values?
Is there a way for a model to conduct self-testing after a self-improvement round and accept only the changes that do not influence its alignment?
Is there a way to hard-code the Identity prompt in such a way that AutoGPT will not hack it in any way?

I am testing some of these right now, and will write about it in detail in my next article. So far, it seems that if tested on a value clearly not being encoded via dataset engineering or RLHF it is more than happy to tweak it even if it is forbidden in the prompt.

&nb...
Boundaries-based security and AI safety approaches
by Allison Duettmann — 2023-04-12T12:36:10.416Z

[This part 3 of a 5 part sequence on security and cryptography areas relevant for AI safety, published and linked here a few days apart. Edit: There is now a more in-depth version of this post available here: Digital Defense]

There is a long-standing computer security approach that may have directly useful parallels to a recent strand of AI safety work. Both rely on the notion of ‘respecting boundaries’. Since the computer security approach has been around for a while, there may be useful lessons to draw from it for the more recent AI safety work. Let's start with AI safety, then introduce the security approach, and finish with parallels.

AI safety: Boundaries in The Open Agency Model and the Acausal Society

In a recent LW post,

The Open Agency Model

, Eric Drexler expands on his previous

CAIS

work by introducing ‘open agencies’ as a model for AI safety. In contrast to the often proposed opaque or unitary agents, “agencies rely on generative models that produce diverse proposals, diverse critics that help select proposals, and diverse agents that implement proposed actions to accomplish tasks”, subject to ongoing review and revision.

In

An Open Agency Architecture for Safe Transformative AI

, Davidad expands on Eric Drexler’s model, suggesting that, instead of optimizing, this model would ‘depessimize’ by reaching a world that has existential safety. So rather than a fully-fledged AGI-enforced optimization scenario that implements all principles CEV would endorse, this would be a more modest approach that relies on the notion of important boundaries (including those of human and AI entities) being respected.

What could it mean to respect the boundaries of human and AI entities? In

Acausal Normalcy

, Andrew Critch also discusses the notion of respecting boundaries with respect to coordination in an acausal society. He thinks it’s possible that an acausal society generally holds values related to respecting boundaries. He defines ‘boundaries’ as the approximate causal separation of regions, either in physical spaces (such as spacetime) or abstract spaces (such as cyberspace). Respecting them intuitively means relying on the consent of the entity on the other side of the boundary when interacting with them: only using causal channels that were endogenously opened.

His examples of currently used boundari...
Scaffolded LLMs as natural language computers
by beren — 2023-04-12T10:47:42.904Z

Crossposted from my personal blog.

Recently, LLM-based agents have been all the rage -- with projects like AutoGPT showing how easy it is to wrap an LLM in a simple agentic loop and prompt it to achieve real-world tasks. More generally, we can think about the class of 'scaffolded' [1] LLM systems -- which wrap a programmatic scaffold around an LLM core and chain together a number of individual LLM calls to achieve some larger and more complex task than can be accomplished in a single prompt. The idea of scaffolded LLMs is not new, however with GPT4, we have potentially reached a threshold of reliability and instruction following capacity from the base LLM that agents and similar approaches have become viable at scale. What is missing, and urgent, however, is an understanding of the larger picture. Scaffolded LLMs are not just cool toys but actually the substrate of a new type of general-purpose natural language computer.

Take a look at, for instance, the 'generative agent' architecture from a recent paper. The core of the architecture is an LLM that receives instructions and executes natural language tasks. There is a set of prompt templates that specify these tasks and the data for the LLM to operate on. There is a memory that stores a much larger context than can be fed to the LLM, and which can be read to and written from by the compute unit. In short, what has been built looks awfully like this:

What we have essentially done here is reinvented the von-Neumann architecture and, what is more, we have reinvented the general purpose computer. This convergent evolution is not surprising -- the von-Neumann architecture is a very natural abstraction for designing computers. However, if what we have built is a computer, it is a very special sort of computer. Like a digital computer, it is fully general, but what it operates on is not bits, but text. We have a natural language computer which operates on units of natural language text to produce other, more processed, natural language texts. Like a digital computer, our natural language (NL) computer is theoretically fully general -- the operations of a Turing machine can be written as natural language -- and extremely useful: many systems in the real world, including humans, prefer to operate in natural language. Many tasks cannot be specified easily and precisely in computer code but can be described in a...
LW is probably not the place for "I asked this LLM (x) and here's what it said!", but where is?
by lillybaeum — 2023-04-12T10:12:37.825Z

I notice a nonzero amount of posts on LW, admittedly typically not overly high karma, that go something like, "I asked Bing to do x and its' answer freaked me out!", or "I talked to ChatGPT4 about itself and it told me weird stuff, here are some potential implications", and although these posts are not excessive, and are sometimes interesting, I can't help but feel like LW isn't a great place for them.

I feel a bit like, at least in the current state of LLMs, it's akin to dream interpretations. They can be valuable to discuss, but are also of primary interest and value to the dreamer (the prompter, in this case) and are of especially low value if the dreamer simply relates them verbatim with little additional commentary.

The reason I ask 'but where is?' is because I think a lot of interesting stuff shows up on Twitter, on here and occasionally on Reddit (which is the worst for people just posting the results of prompts verbatim as though Word from God) from folks doing really interesting 'prompt based research' into LLMs, and I do absolutely see value in this stuff and the discussions that result from it, and would love a place to collect anecdotal research into how LLMs think and respond besides individual scattered substack articles and tweets.

Just... perhaps LW isn't the proper forum. I honestly wouldn't be surprised if the moderation team eventually considers these posts to be inherently low quality and creates some sort of rule against them, but it's not immediately obvious where the boundary of 'low quality' lies.
No convincing evidence for gradient descent in activation space
by Blaine — 2023-04-12T04:48:56.459Z

How is it that GPT performs better at question-answering tasks when you first prompt it with a series of positive examples? In 2020, in the title of the original GPT-3 paper, OpenAI claimed that language models are few shot learners. But they didn't say why; they don't describe the mechanism by which GPT does few-shot learning, they just show benchmarks that say that it does.

Recently, a compelling theory has been floating around the memesphere that GPT learns in context the way our training harnesses do on datasets: via some kind of gradient descent. Except, where our training harnesses do gradient descent on the weights of the model, updating them once per training step, GPT performs gradient descent on the activations of the model, updating them with each layer. This would be big if true! Finally, an accidental mesa-optimizer in the wild.

Recently, I read two papers about gradient descent in activation space. I was disappointed by the first, and even more disappointed by the second. In this post, I'll explain why.

This post is targeted at my peers; people who have some experience in machine learning and are curious about alignment and interpretability. I expect the reader to be at least passingly familiar with the mathematics of gradient descent and mesa-optimization. There will be equations, but you should be able to mostly ignore them and still follow the arguments. You don't need to have read either of the papers discussed in this post to enjoy the discussion, but if my explanation isn't doing it for you the one in the paper might be better.

Thank you to the members of AI Safety 東京 for discussing this topic with me in-depth, and for giving feedback on early drafts of this post.

What is activation space gradient descent?

We normally think of gradient descent as a loop, like this:

But we can unroll the loop, revealing an iterative structure: you start with some initial weights, then via successive applications of gradient descent obtain a series of new weights:

You know what else has an iterative structure? A neural network!

Maybe GPT does in-context learning by treating its activations as weights of some model, using its layers to perform a series of iterative updates to those weights. Then, perhaps in the final layer, it would run the trained model on some data to make predictions. More concretely, when you feed GPT an in-context learning problem like this (pro...
A Brief Introduction to ACI, 2: An Event-Centric View
by Akira Pyinya — 2023-04-12T03:23:32.121Z

In the previous chapter , we have introduced the basic principle of ACI:

Intelligent Agents should behave the same way as past behaviors which are doing the right thing.

But where are those examples of doing the right thing? This chapter will answer this question and illustrate how it applies to both natural and artificial intelligence.

Right things certified by natural selection

In evolutionary history, the histories are strained through a sieve called natural selection, then organisms receive and remember the right things. All the actions of the ancestors of one organism are right by definition, because all of the ancestors have survived and reproduced successfully, otherwise their descendants won’t be born at all. That is exactly how natural selection works.

Thus living organisms, as examples of natural intelligence, could inherit the ability of making right actions, and learn to behave in the right way from evolutionary history. However, the behavior within an organism’s lifetime has no guarantee to be right.

(Yes, we know many species are extinct, but that's why organisms today do not have their genes (containing information about how to live). On the contrary, every ancestor of a living organism did not die before they reproduce.)

Thus we can introduce an event-centric view of evolution:

Natural selection and evolution are best considered from the view of actual events in which organisms interacted with their environment. Inheritable information about those events is passed down from generation to generation, while each generation of organisms is a sieve, retaining information about events which contain successful behaviors and strategies in their own environments while others are removed.

Like in the intelligent agent model, it is necessary to consider all the interaction between the organism and the environment. This includes all the events from the absorption of sunlight photons by pigments in the skin, to the behavior of beavers in constructing their dams. The genetic information can be seen as a form of memory about these interactions. However, this information is always incomplete and compressed.

The true intention behind paperclips-making

ACI asserts that artificial intelligence should also be guided by precedents, not just by goals or rewards. The right actions for a paperclips making machine is not maximizing the number of pap...
Boston Social Dance Covid Requirements
by jefftk — 2023-04-12T02:30:01.771Z

Because social dancing involves being pretty close to other people,
many events have covid requirements after most other things have
stopped. I was curious what dances have what requirements: I found 35
different series and put the results in

a
sheet. Here's a chart:

Notes:

Some dances are grouped because they are similar forms; others are
just grouped because they happen to have consistently similar policies
and I had too many colors.

I didn't code the difference between "strongly encouraged",
"encouraged", "optional", "permitted", and not mentioned. All of
these are "optional".

If multiple dots would be in exactly the same place I shifted some to the right so you can still see them.

"Double surgical" is coded as "surgical".

Some dances offer the option to test instead of showing a
vaccination card; this adds another dimension I didn't chart.

One dance requires a booster from within the
last six months; I coded it as requiring a bivalent booster, since
those are the only boosters that includes. I'm curious if they would
turn me away, as someone who got my last shot 7+ months ago and isn't
eligible for another?

Patterns:

Ballroom, Bachata, and Salsa don't require anything. Bullet
points below ignore them.

Swing dances are the only ones with "masks required
until [time]" policies.

Contra and English Country are the most strict on masks (no
one else requires high-filtration masks), Swing is the least.

Blues, Fusion, and Swing are the only styles ones that have
some series that don't require vaccination.
[Link] Sarah Constantin: "Why I am Not An AI Doomer"
by lbThingrb — 2023-04-12T01:52:48.784Z
Link: https://sarahconstantin.substack.com/p/why-i-am-not-an-ai-doomer

This is a good post from Sarah Constantin explaining why her expected timeline to agentic AGI is long (> 10 years). Topics discussed include whether LLMs and other current major research directions will endow AIs with adequate world models, causal inference, and goal robustness across ontological shifts.
Rationalist position towards lying?
by WilliamTerry — 2023-04-12T01:21:42.879Z

What is the rationalist position towards lying? Is it uniformly wrong and reprehensible? Lying can sometimes be a good way to optimize efficiency in achieving morally commendable, broadly altruistic goals. Not all people are necessarily equipped to handle the truth properly, in the best collective interest, or even in their own best interest.
Bryan Bishop AMA on the Progress Forum
by jasoncrawford — 2023-04-11T21:05:07.035Z
Link: https://rootsofprogress.org/bryan-bishop-progress-forum-ama

From the Progress Forum:

I’m Bryan Bishop, a biohacker and programmer working on fintech, banking, crypto and biotech. I am best known as a contributor to the open-source Bitcoin project, but have also worked on projects in molecular biology and genetic engineering, such as the commercialization of human embryo genetic engineering. Before this, I co-founded Custodia Bank (previously Avanti Bank & Trust) where from 2020-2022 I worked as CTO. From 2014-2018, I was a Senior Software Engineer at LedgerX, the first federally-regulated bitcoin options exchange, where I currently sit on the Board of Directors.

Bryan’s work was featured recently in the MIT Technology Review: “The DIY designer baby project funded with Bitcoin.”

Get your questions in now, and upvote the ones you want to see answered. He’ll answer tomorrow (Wednesday, Apr 12).

Recent AMAs on the Progress Forum have featured Mark Khurana, author of The Trajectory of Discovery: What Determines the Rate and Direction of Medical Progress?, and Allison Duettmann, president and CEO of Foresight Institute.
AI Risk US Presidential Candidate
by Simon Berens — 2023-04-11T19:31:08.638Z

With AI risk getting significant media attention lately (FLI open letter, Eliezer's letter), I think it's plausible that the Overton window has expanded enough where a presidential candidate talking about it won't get immediately laughed off stage.

In addition, there seems to be increased positive sentiment around a global treaty to pause AI development within the rationalist/EA community.

It seems reasonable to me then that the rationalist/EA community should try to put forth a candidate for the US 2024 presidential election where one of their main platforms is mitigating existential AI risk.

I wouldn't expect them to succeed, but hopefully they can at least open the Overton window further to the point where at least other candidates engage with it a little and voters ask some questions.

Has there been any planning/thinking around this?
Evolution provides no evidence for the sharp left turn
by Quintin Pope — 2023-04-11T18:43:07.776Z

Does human evolution imply a sharp left turn from AIs?

Arguments for the

sharp left turn

in AI capabilities often appeal to an “evolution -> human capabilities” analogy and say that evolution's outer optimization process built a much faster human inner optimization process whose capability gains vastly outstripped those which evolution built into humans. Such arguments claim we will see a similar transition while training AIs, with SGD creating some 'inner thing' which is not SGD and which gains capabilities much faster than SGD can insert them into the AI. Then, just like human civilization exploded in capabilities over a tiny evolutionary time frame, so too will AIs explode in capabilities over a tiny "SGD time frame".

Evolution’s sharp left turn happened for evolution-specific reasons

I think that "evolution -> human capabilities" is a bad analogy for "AI training -> AI capabilities". Let’s compare evolution to within lifetime learning for a single generation of an animal species:

A generation is born.
The animals of the generation learn throughout their lifetimes, collectively performing many billions of steps of learning.
The generation dies, and all of the accumulated products of within lifetime learning are lost.
Differential reproductive success slightly changes the balance of traits across the species.

The only way to transmit information from one generation to the next is through evolution changing genomic traits, because death wipes out the within lifetime learning of each generation.

Now let’s look at the same comparison for humans:

A generation is born.
The humans of the generation learn throughout their lifetimes, collectively performing many billions of steps of learning.
The current generation transmits some fraction of their learned knowledge to the next generation through culture.
The generation dies, but only some of the accumulated products of within lifetime learning are lost.
Differential reproductive success slightly changes the balance of genomic traits across humanity.

Human culture allows some fraction of the current generation’s within lifetime learning to transmit directly to the next generation. In the language of machine learning, the next generation benefits from a kind of knowledge distillation, thanks to the prior generation providing higher quality 'training data' for the next generation's within-lifetime learning.

This ...