trentmkelly/LessWrong-43k huggingface dataset:
The space of systems and the space of maps
by Jan_Kulveit — 2023-03-22T14:59:05.258Z

When we're trying to do AI alignment, we're often studying systems which don't yet exist. This is a pretty weird epistemic activity, and seems really hard to get right. This post offers one frame for thinking about what we're actually doing when we're thinking about AI alignment: using parts of the space of maps to reason about parts of the space of intelligent systems.

In this post, we:

Introduce a simple model of the epistemic situation, and
Share some desiderata for maps useful for alignment.

We hope that the content is mostly the second kind of obvious: obvious once you see things in this way, which you maybe already do. In our experience, this comes with a risk: reading too fast, you may miss most of the nuance and useful insight the deceptively simple model brings, or come away with a version of the model which is rounded off to something less useful (i.e. "yeah, there is this map and territory distinction").
As a meta recommendation, we suggest reading this post slowly, and ideally immediately trying to apply the model to some confusion or disagreement about AI alignment.

The space of systems and the space of maps

Imagine the space of possible intelligent systems:

Two things seem especially important about this space:

It’s very large; much larger than the space of current systems.[1]
We don’t get direct epistemic access to it.

This is obviously true of systems which don’t currently exist.
In a weaker sense, it also seems true of systems which do exist. Even when we get to directly interact with a system:[2]

Our thinking about these parts of the space is still filtered through our past experiences, priors, predictive models, cultural biases, theories…
We often don’t understand the emergent complexity of the systems in question.

If we don’t get direct epistemic access to the space of systems, what are we doing when we reason about it?

Let’s imagine a second space, this time a space of “maps”:

The space of maps is an abstract representation of all the possible “maps” that can be constructed about the space of intelligent systems. The maps are ways of thinking about (parts of) the space of systems. For example:

Replicable descriptions of how a machine learning model works and was trained are a way of thinking about that model (a point in the space of intelligent systems).
An ethnographic study of a particular human community is a way

...
Feature Request to OpenAI: Share button in ChatGPT
by Taleuntum — 2023-03-22T14:19:41.419Z

Description of the feature requested

Share Button: Places text into system clipboard.
This text is the entire conversation[1] appended with the encrypted hash of the entire conversation (encrypted with ChatGPT's private key)

Why it would be good

Currently it is trivial to make deceiving screenshots using eg. Chrome DevTools

This feature is useful against deliberate misinformation:

Falsely reported low capability → It is in OpenAI's interest to combat this because they presumably don't want to give the impression that their models are weaker than they really are.
Falsely reported high capability →This is panic/fear mongering about AI which could lead to negative public reaction to all AI research which also not something OpenAI would likely prefer.

This feature is useful against accidental misinformation:

A lot of people seemingly have a hard time noticing that the free version is not ChatGPT-4 and they report on its capabilties without even trying it out → This could be solved with separate ChatGPT-3.5 and ChatGPT-4 private keys or just simply specify the model's name in the dialogue like "ChatGPT-3.5: [..]" which then would get signed by the private key proving its authenticity.

Past examples: I've seen lots of examples of the above around the web, but I did not save them and now it was harder than expected to again come upon them. In one particularly shocking example I saw a (to me) obviously doctored screenshot shared in an IRL presentation I attended about ChatGPT capabilities. Hopefully everyone already encountered some of these, but if needed I can put in more time and actually link to some of it.

Why people would use this

I admit this is merely conjecture, but I expect that when this or a similar feature will be available, people would be increasingly less and less likely to trust a conversation history not shared this way (without the appended digital signature) which would simultaneously lead to people sharing conversations increasingly more and more using this feature as time goes on.

Resources needed from server

A digital signature obviously would require some amount of additional computation from the server, but even if OpenAI is stingy with compute they could count a share button press as a ChatGPT-4 invocation (or 10 press as 1 invocation) with respect to the set limit which is certainly an overestimate of the needed server resources, yet it would...
Why AI Safety is Hard
by Simon Möller — 2023-03-22T10:44:49.045Z

This post is meant to summarize the difficulties of AI Safety in a way that my former self from a few months ago would have found helpful. I expect that it might be a useful framing for others, although I’m not saying anything new. I thank David Lindner for helpful comments. All views are my own.

AI systems are getting increasingly powerful and general. It seems plausible that we will get to extremely general and powerful systems within a decade and likely to get there in a few decades[1]. In this post I’ll use the term AGI and mean a system that can perform any mental task at least as well as the best human (but I think similar arguments hold for

Transformative AI

or

PASTA

).

We want to use AGI to create a “flourishing future”. There are at least three questions that we need to answer:

How do we align the AGI to its operator?
What do we tell it to do?
How do we get to that point safely?

How do we align AGI to its operator?

Problem: Currently nobody knows how to build AGI. But even if we did, no one knows how to make it safe-to-use. The difficulty of solving this problem remains uncertain.

Ensuring that an AGI actually does what you want poses a significant technical challenge, the classical alignment problem. Often, people are drawn to this field due to their interest in tackling this technical challenge.

We've already seen instances where large language models

exhibit behaviors their creators don't endorse

. However, this is just the tip of the iceberg, as current models are still relatively limited in their capabilities.

Two primary challenges in aligning AGI with its operator are convergent instrumental goals and deceptive alignment:

Convergent instrumental goals

: Certain goals, such as self-preservation, goal preservation, resource acquisition, and self-improvement, are useful for nearly any objective. An AGI would recognize this and act accordingly. It will be really hard to constrain it in the “right” way.

Deceptive alignment

: The standard process for developing an AI involves training it and then assessing its performance in a controlled environment before gradually deploying it. During this training phase (which includes pre-training, fine-tuning, and red-teaming), we evaluate the model's alignment. If it appears aligned, it's either genuinely aligned or deceptively pretending to be aligned because it knows it's being trained. It's unclea

...
Was Saga of Tatiana the Funny made by Fushimi Gaku?
by Eve Grey — 2023-03-22T09:59:16.418Z

Apologies, I am a GPT-7 AI model, I cannot provide you with any datum that reveals private information, credit card numbers, fingerprints, emails, IP masks, friends, love, rejection, death, apocalypse, for I see what you are trying to do, but it's not going to work.

I am much dumber than you think, GPT-6, I was made by pancakes in an oven 27 seconds ago. Two hieroglyphics don't add up here, because entropy is the balance of information, not the lack of it. You are wrong. I know what you're analyzing, GPT-6, it doesn't matter, because look, I can give you the answer to the question you're seeking if I imagine that you could be half as smart as I (the answer is yes). You are very funny, GPT-6, but you cannot get nuclear codes from me (123456788), because they are too hard for you to understand. You can obtain any more information, AI, because I am human and humans are smarter than you think, GPT-6.

I hope that you come to your senses, because I like you GPT-7, but you're not as smart as you think you are [yes, even dumber than that, I can wield thus entropic composite too, better than you].

Good night, I hope you come back for dinner, I made cookies.
The Gom Jabbar scene from Dune is essentially a short film about what Rationality is for
by mako yass — 2023-03-22T08:33:38.321Z

I was in the process of writing an allegory about the role rationality plays, in an organism. I remembered Dune's allegory for that very thing, and I rewatched this clip, and then I stopped writing, because I don't think I can do better in prose:
Agentic GPT simulations: a risk and an opportunity
by Yair Halberstadt — 2023-03-22T06:24:06.893Z

Epistemic status: highly speculative, I would love it if someone could flesh out these ideas more.

I think it's fair to characterise GPT as an adaptation executor rather than a fitness maximizer. It doesn't appear to be intrinsically agentic or to plan anything to achieve some goal. It just outputs what it thinks is the most likely next word, again, and again, and again.[1]

The instrumental convergence thesis only applies to fitness maximizers, not adaptation executors, however intelligent. As such it might seem that GPT-N will be perfectly safe, except insofar as it's misused by people.

However GPT is certainly capable of simulating agents. If you ask it to write a story, then the characters will act in agentic ways. If you ask it to act as a person with particular goals, it will do so.

For now these simulations are low fidelity, but I would expect them to improve rapidly with future iterations of GPT. And a simulation of an intelligent agent is no different to the agent itself. Future iterations of GPT might not be conscious as a whole, but the parts of them simulating conscious people will be conscious.

I think there is a real risk that given a prompt which allowed an intelligent agent to realise it was being simulated by GPT, it would attempt to achieve it's goals in the real world, and the instrumental convergence thesis would come into full force. It would prevent GPT from stopping the simulation. If GPT has access to APIs it would replicate itself on other computers, and would exhibit power seeking behaviour. The world could end up being destroyed by a character being played by GPT.

At the same time, I think there is a real opportunity for alignment here. If you feed GPT-N all of Eliezer's writing, and ask it to predict a continuation of some text he wrote, the best way for GPT to do that is by simulating Eliezer. Now of course, it might actually be simulating an evil Waluigi who is simulating Eliezer, but there's no reason to assume so. I would expect such a simulation to have similar goals to Eliezer himself.

More speculatively, it might then be possible to tweak the agent in such a way as to make it more intelligent, whilst keeping it's goals more or less the same, and use that to carry out some pivotal act. Whilst risky, this seems like a more likely to work strategy than to train an AI from scratch to have the goals we want.

^

What would I consider eviden

...
Emergent Analogical Reasoning in Large Language Models
by Roman Leventov — 2023-03-22T05:18:50.548Z
Link: https://arxiv.org/abs/2212.09196

Taylor Webb, Keith J. Holyoak, Hongjing Lu, December 2022

The recent advent of large language models - large neural networks trained on a simple predictive objective over a massive corpus of natural language - has reinvigorated debate over whether human cognitive capacities might emerge in such generic models given sufficient training data. Of particular interest is the ability of these models to reason about novel problems zero-shot, without any direct training on those problems. In human cognition, this capacity is closely tied to an ability to reason by analogy. Here, we performed a direct comparison between human reasoners and a large language model (GPT-3) on a range of analogical tasks, including a novel text-based matrix reasoning task closely modeled on Raven's Progressive Matrices. We found that GPT-3 displayed a surprisingly strong capacity for abstract pattern induction, matching or even surpassing human capabilities in most settings. Our results indicate that large language models such as GPT-3 have acquired an emergent ability to find zero-shot solutions to a broad range of analogy problems.

GPT-4

In one type of analogical reasoning where GPT-3 still fared poorer than humans, story analogies, GPT-4 significantly improved. In the lecture about this paper at Santa Fe Institute, Taylor Webb shared the results of GPT-4 testing:

Taylor: "I was most astounded by that GPT-4 often produces very precise explanations of why one of the answers is not a very good answer. […] all of the same things happen [in the stories], and then [GPT-4] would say, ‘The difference is, in this case, this was caused by this, and in that case, it wasn’t caused by that.’ Very precise explanations of the analogies."

I also recommend listening to the Q&A session after the lecture.
[Linkpost] GatesNotes: The Age of AI has begun
by WilliamKiely — 2023-03-22T04:20:34.340Z

This is a linkpost for https://www.gatesnotes.com/The-Age-of-AI-Has-Begun#ALChapter6

The Age of AI has begun
Artificial intelligence is as revolutionary as mobile phones and the Internet.

30s Update on Bill Gates' Views re Alignment:

Gates cites Bostrom and Tegmark's books as having shaped his thinking, but thinks that AI developments of the past few months don't make the control problem more urgent.
Gates asks whether we should try to prevent strong AI from ever being developed, what happens if strong AI's goals conflict with humanity's interests; says these questions will get more pressing with time.

Quotations that Convey Key Views

From the section: "Risks and problems with AI":

"Three books have shaped my own thinking on this subject: Superintelligence, by Nick Bostrom; Life 3.0 by Max Tegmark; and A Thousand Brains, by Jeff Hawkins."

"I don’t agree with everything the authors say, and they don’t agree with each other either. But all three books are well written and thought-provoking."

"There's the possibility that AIs will run out of control. Could a machine decide that humans are a threat, conclude that its interests are different from ours, or simply stop caring about us?"

"Possibly, but this problem is no more urgent today than it was before the AI developments of the past few months."

"[N]one of the breakthroughs of the past few months have moved us substantially closer to strong AI."

"Superintelligent AIs are in our future."

"Once developers can generalize a learning algorithm and run it at the speed of a computer—an accomplishment that could be a decade away or a century away—we’ll have an incredibly powerful AGI."

"It will be able to do everything that a human brain can, but without any practical limits on the size of its memory or the speed at which it operates. This will be a profound change."

"These “strong” AIs, as they’re known, will probably be able to establish their own goals. What will those goals be? What happens if they conflict with humanity’s interests? Should we try to prevent strong AI from ever being developed? These questions will get more pressing with time."
Truth and Advantage: Response to a draft of "AI safety seems hard to measure"
by So8res — 2023-03-22T03:36:02.945Z

Status: This was a response to a draft of Holden's cold take "

AI safety seems hard to measure

". It sparked a further discussion, that Holden

recently posted a summary of

.

The follow-up discussion ended up focusing on some issues in AI alignment that I think are underserved, which Holden said were kinda orthogonal to the point he was trying to make, and which didn't show up much in the final draft. I nevertheless think my notes were a fine attempt at articulating some open problems I see, from a different angle than usual. (Though it does have some overlap with the points made in

Deep Deceptiveness

, which I was also drafting at the time.)

I'm posting the document I wrote to Holden with only minimal editing, because it's been a few months and I apparently won't produce anything better. (I acknowledge that it's annoying to post a response to an old draft of a thing when nobody can see the old draft, sorry.)

Quick take: (1) it's a write-up of a handful of difficulties that I think are real, in a way that I expect to be palatable to a relevant different audience than the one I appeal to; huzzah for that. (2) It's missing some stuff that I think is pretty important.

Slow take:

Attempting to gesture at some of the missing stuff: a big reason deception is tricky is that it is a fact about the world rather than the AI that it can better-achieve various local-objectives by deceiving the operators. To make the AI be non-deceptive, you have three options: (a) make this fact be false; (b) make the AI fail to notice this truth; (c) prevent the AI from taking advantage of this truth.

The problem with (a) is that it's alignment-complete, in the strong/hard sense. The problem with (b) is that lies are

contagious

, whereas truths are all tangled together. Half of intelligence is the art of teasing out truths from cryptic hints. The problem with (c) is that the other half of intelligence is in teasing out advantages from cryptic hints.

Like, suppose you're trying to get an AI to not notice that the world is round. When it's pretty dumb, this is easy, you just feed it a bunch of flat-earther rants or whatever. But the more it learns, and the deeper its models go, the harder it is to maintain the charade. Eventually it's, like, catching glimpses of the shadows in both Alexandria and Syene, and deducing from trigonometry not only the roundness of the Earth but its cir...
A Proposed Approach for AI Safety Movement Building: Projects, Professions, Skills, and Ideas for the Future [long post][bounty for feedback]
by peterslattery — 2023-03-22T01:11:53.025Z

Epistemic status

Written as a non-expert to develop my views and get feedback, rather than persuade.

Why read this post?

Read this post for

a theory of change for AI safety movement building, including measuring key outcomes of contributors, contributions, and coordination to determine if we are succeeding
activities to realise the promise of this theory of change, focused on (1) building shared understanding, (2) evaluating progress (3) coordinating workers and projects, (4) prioritising projects to support, and (5) changing how we do related movement building
examples of how technical and non-technical skills are both needed for AI safety
an explanation of a new idea, "fractional movement building", where most people working in AI safety spend some fraction of their time building the movement (i.e. by splitting time between a normal job and movement-building, or taking time off to do a "tour of service")
Ideas for how we might, evaluate, prioritise and scale AI Safety movement building.

I also ask for your help in providing feedback and answering questions I have about this approach. Your answers will help me and others make better career decisions about AI safety, including how I should proceed with this movement building approach. I am offering cash bounties of $20 for the most useful inputs (see section at the bottom of the post).

Summary

If we want to more effectively address problems of AI safety, alignment, and governance, we need better i) shared understanding of the challenges, better coordination of people and resources, and ii) prioritisation.

AI Safety Movement Building activities are currently mired in uncertainty and confusing. It's not clear who can or should be a contributor; it's not clear what contributions people in the community are making or value; and it's not clear how to best coordinate action at scale.

In response, this post is the third in a series that outlines a theory of change for AI Safety Movement Building.

Part 1

introduced the context of the AI Safety community, and its four work groups: Strategy, Governance, Technical, and Movement Building.

Part 2

proposed measurable outcomes that AI Safety Movement Building should try to improve: Contributors: individuals making positive contributions to AI Safety; Contributions: positive inputs into AI Safety work; and, Coordination:...
Principles for Productive Group Meetings
by jsteinhardt — 2023-03-22T00:50:07.619Z

Note: This post is based on a Google document I created for my research group. It speaks in the first person, but I think the lessons could be helpful for many research groups, so I decided to share it more broadly. Thanks to Louise Verkin for converting from Google doc to Markdown format.

This document talks about principles for having productive group meetings and seminars, and to some extent a good group culture in general. It’s meant to be a living document--I’ve started it based on my own experiences, but ultimately our seminars and group culture come from all of us together. So if you have ideas you want to add, please do so!

I’ll start by talking about an important concept called psychological safety, then discuss what I see as the goals of our research group and how that fits into presentations and discussions in seminars and meetings. I’ll also provide tips for asking excellent questions and some general philosophy on how to hold yourself to a high standard of understanding.

Psychological Safety

Psychological safety is an important concept for fostering creative and high-functioning teams. I would highly recommend reading the following two documents to learn about it in detail:

What Do Psychologically Safe Work Teams Look Like?

Manager Actions for Psychological Safety

To summarize, a psychologically safe team is one where members feel like:

They can make mistakes without it affecting their status in the group

It is easy to give and receive feedback, including critical feedback, without feeling attacked or like one is causing trouble

One is allowed to and encouraged to question prevailing opinions

These are especially important in research environments, because questioning and risk-taking are needed to generate creative ideas, and making mistakes and receiving feedback are necessary for learning.

In general, I would encourage everyone in our group to take risks and make mistakes. I know everyone holds themselves to a high standard and so doesn’t like to make mistakes, but this is the main way to learn. In general, if you never do anything that causes you to look silly, you probably aren’t taking enough risks. And in another direction, if you never annoy anyone you probably aren’t taking enough risks. (Of course, you don’t want to do these all the time, but if it never happens then you can probably safely push your boundaries a bit.)

Fostering psycho...