trentmkelly/LessWrong-43k huggingface dataset:
Universality and Hidden Information in Concept Bottleneck Models
by Hoagy — 2023-04-05T14:00:35.529Z

Summary

I use the CUB dataset to finetune models to classify images of birds, via a layer trained to predict relevant concepts (e.g. "has_wing_pattern::spotted"). These models, when trained end-to-end, naturally put additional relevant information into the concept layer, which is encoded from run to run in very similar ways. The form of encoding is robust to initialisation of the concept-heads, and fairly robust to changes in architecture (presence of dropout, weighting of the two loss components), though not to initialisation of the entire model. The additional information seems to be primarily placed into low-entropy concepts.

This suggests that if steganography were to arise in, for example, fine-tuned autoregressive models, there would be convergent ways for this information to hide, which I previously thought unlikely, and which constrains the space of possible mitigations.

This work is ongoing and there are any more experiments to run but this is an intermediate post to show the arc of my current results and to help gather feedback. Code for all experiments on GitHub.

Background

In my Distilled Representations Research Agenda I laid out the basic plan for this research. The quick version of this is that we showed that we can create toy cases where we train auto-encoders to encode vectors of a certain dimension, within which some dimensions have a preferred way of being represented, while others don't, and by training multiple models, we can distinguish between these different dimensions and create a new model which encodes only those dimensions which have a preferred encoding.

The next step is to see whether this toy model has any relevance to a real-world case, which I explore in this post.

Training Procedure

This work uses the CUB dataset. This is a dataset which contains >10K images, each labelled one of 200 species of bird. Each image also has 109 features which describe the birds' appearances.[1] These features are divided into 28 categories, such as beak-shape. Usually only one of these features is true for any given image, though the raw annotations can have multiple attributes in a class be true.

A Brown Thrasher and its (rather messy) raw labels, from the CUB dataset.

I perform experiments on the CUB dataset using Concept Bottleneck Models, specifically by finetuning an ImageNet-trained model to predict the species of bird from...
AI safety and the security mindset: user interface design, red-teams, formal verification
by Allison Duettmann — 2023-04-05T11:33:00.086Z

[This is part 2 of a 5 part sequence on how security and cryptography can help AI safety, published and linked here a few days apart.]

Parallels between the AI safety mindset and the security mindset

In addition to AI infosec, (discussed in part 1 of this sequence), several members in the AI and security communities have pointed to specific examples of natural 'alignment' between AI alignment and computer security concerns.

In

Security and Alignment

, Paul Christiano suggests that when looking for near-term analogies to long-term AGI problems, security is a good place to start: “if something is an alignment problem then it’s a security problem, and if something is a security problem then it’s an alignment problem.” This is because “a) if an AI system has to work in the real world, its code, hardware, and rewards should be stable in the face of adversaries, and b) any hard failures that may arise naturally in AGI, will likely first be engineered by an adversary.”

Specific AI-safety areas that may have precedents in security mentioned by Paul Christiano include generating adversarial outputs, exploiting misalignment, and extracting information, which may all likely be tried by adversarial actors before a sufficiently powerful AGI tries them. He concludes that it is useful if “researchers in alignment and security have a clearer picture of the other field’s goals and priorities”, and for AI alignment researchers not to shy away from what sound like security problems.”

In

AI Safety mindset

, MIRI expands on these parallels between the AI safety mindset and the security mindset: “In computer security, we need to defend against intelligent adversaries who will seek out any flaw in our defense and get creative about it. In AI safety, we're dealing with things potentially smarter than us, which may come up with unforeseen clever ways to optimize whatever it is they're optimizing. The strain on our design ability in trying to configure a smarter-than-human AI in a way that doesn't make it adversarial, is similar in many respects to the strain from cryptography facing an intelligent adversary.”

Specific AI security parallels mentioned by MIRI include searching for strange opportunities, perspective-taking and tenacity, submitting safety schemes to outside scrutiny, presumption of failure, reac...
ICA Simulacra
by Ozyrus — 2023-04-05T06:41:44.192Z

Pre-preface

This post was written about a year ago, but I decided not to publish it with concerns of burning timelines. Since then, after the hype caused by ChatGPT, similar systems have been proposed, implemented and even tested; so I think that “the secret”, even if there was one, is out, and there’s no more risks in publishing it. Moreover, I think that publishing it will be net positive, because I can see ways in which we can align such systems. I will leave notes in italics throughout this text to comment on things that changed after this post was written, otherwise not changing the text (sans minor grammar edits).
This was written before Simulators by Janus, but what is described here is essentially a simulacra. I call it Identity-Context-Action Simulacra, or, short, ICA Simulacra.
Examples of early systems of this kind:
Langchain
Task-driven autonomous agent using Langchain

Preface

This post is inspired by many hours of thinking about human consciousness and the state of AI research, as well as some reading here and there, and of course some hands-on experience of working on AI systems, particularly in NLP. Recent big transformer model releases solidified that language models have almost reached or even surpassed human-level performance on many tasks, with some researchers even hinting on bigger model capabilities given the right prompt.

That led me into thinking that we’re what’s called “one prompt away” from AGI.
It is, of course, oversimplification of what’s to be done to reach truly human or sub-human level agents with capabilities of self-improvement and independent action.
To get to the true potential of proposed architecture we need to improve attention window (Note: since then have been significantly improved from ~2k to 32k), research better ways for implementation of submodules(Note: well, see OpenAI Tools), and get a lot of pretty specific not-yet-existing training data. Moreover, by my estimation we’re not even close to the continued inference of such a model in terms of compute.
But I’m getting ahead of myself here a little bit.

The following is my very brief take of how these agents might look like, what is needed for them to work, and potential research avenues. Aim is to inspire and introduce more than solidify the theory.

The one prompt

Before getting to the part where I describe the one-prompt-to-rule-them all, I have to say that...
AGI deployment as an act of aggression
by dr_s — 2023-04-05T06:39:44.853Z

I started a draft of this post some days ago, but then a lot of things happened and so I'm rewriting it from scratch. Most importantly, a TIME editorial in which Eliezer Yudkowski talks of bombing AI training data centres happened, which pretty much both makes AI doom discourse fairly mainstream and throws a giant rock through the Overton window on the topic. This has elicited some ridicule and enough worry that a few clarifications may be needed.

A lot of the discourse here focuses usually on alignment: is it easy, is it hard, what happens if we can't achieve it, and how do we achieve it. I want to make a broader point that I feel like might not have received as much focus. Essentially, my thesis is that from the viewpoint of the average person, developing and deploying agentic AGI at all might be viewed as a hostile act. I think that the AI industry may be liable of being regulated into economic unviability and/or that MAD-like equilibria as the one Eliezer suggested might form not because everyone is scared of unaligned AGI, but because everyone is almost equally scared of aligned AGI, and not for no reason. As such, I think that perhaps the sanest position to take in the upcoming public debate as AI-safety-minded people is "We should just not build AGI (and rather focus on more specialised, interpretable, non-agentic AI tools that merely empower humans, but leave the power to define their goals always firmly into our hands)". In other words, I think at this stage of things powerful friendly AGI is simply a mirage that holds us back from supporting the solutions that would have the best chance of success and makes us look hostile or untrustworthy to a large part of the public, including potential allies.

The fundamental steps for my thesis are:

building AGI probably comes with a non-trivial existential risk. This, in itself, is enough for most to consider it an act of aggression;
even if the powerful AGI is aligned, there are many scenarios in which its mere existence transforms the world in ways that most people don't desire or agree with; whatever value system it encodes gets an immense boost and essentially Wins Culture; very basic evidence from history suggests that people don't like that;
as a result of this, lots of people (and institutions, and countries, possibly of the sort with nukes) might turn out to be willing to resort to rather extreme mea

...
A Brief Introduction to Algorithmic Common Intelligence, ACI . 1

by Akira Pyinya — 2023-04-05T05:43:22.928Z

According to the rational agent model , an AI agent should have a relatively stable terminal value, such as achieving an ultimate goal, maximizing rewards, or optimizing expected utilities. The terminal value should be prevented from modification most of the time, because it provides information about what counts as the right thing.

Despite being considered intelligent, humans often act irrationally. Human values are unclear and complex. Unfortunately, AI has to be aligned with human values, not the opposite.

Value learning approaches believe that there are some relatively stable human values that underlie human behaviors, which need to be identified, loaded into, and followed by AI to ensure that it benefits humanity. Any mistakes in this process result in the alignment problem.

Some models attempt to account for human biases and limitations, others aim to replace the rational agent model altogether. But most of the time, such models increase the complexity of AI models without providing a much better solution.

Thus we introduce the Algorithmic Common Intelligence, ACI model, which is trying to provide a novel and universal answer to the fundamental question of value. Instead of relying on supposed goals or utilities, ACI uses past events and behaviors as the fundamental guideline, since they are the only evidence we know about human value.

Beyond goals and utilities

ACI shares the view that there exist values that underlie human behavior, but rather than restricting these values to goals or utilities, ACI holds the idea that there are no coherence arguments that say you must have goal-directed behavior.

ACI argues that the rational agent model is only an imperfect representation of actual intelligence in the real world. As a more general framework, ACI claims that an agent should behave the same way as past behaviors which are doing the right thing.

ACI is a universal intelligence model, building upon algorithmic information theory and draws inspiration from the common law system. ACI argues that the basis of all values is the past events and behaviors (or environment and actions) that are doing the right thing, a.k.a. the precedent, a terminology borrowed from the common law system.

ACI and the alignment problem

ACI provides an explanation for the alignment problem, as well as a solution: the goal or reward system program...
46% of US adults at least "somewhat concerned" about AI extinction risk.
by Foyle — 2023-04-05T05:25:13.839Z

Good news today I think; a poll by yougov had a near majority somewhat concerned to very concerned about existential risk of AI, with younger people most concerned. Given relative newness of (debatably) Transformative AI I think this suggests we will see a strong bipartisan political movement to regulate AI development develop in next 1-2 years. Almost certain to be aided by a media/political complex that thrives on promoting and magnifying dangers.
Has anyone thought about how to proceed now that AI notkilleveryoneism is becoming more relevant/is approaching the Overton window?
by metachirality — 2023-04-05T03:06:30.585Z

I figure that this would be a pretty important question since getting governments on board, and getting people to coordinate in general, could buy us a lot of time, but I haven't seen anyone talk about this before. I would like to hear what people think and what their strategies are.
Empathy bandaid for immediate AI catastrophe
by installgentoo — 2023-04-05T02:12:54.713Z

Assuming that agi agent is being built with q-learning and an LLM for the world model, we can do the following to emulate empathy:

Train a separate agent action reasoning network. For LLM tech this should be training on completing interaction sentences, think "Alice pushed Bob. ___ fell due to ___", with a tokenizer that generalizes agents(Alice and Bob) into generic {agent 1, agent n} and "self agent". Then we replace various Alices and Bobs in various action sentences with generic agent tokens, and train on guessing consequences or prerequisites of various actions from real situations that you can get from any text corpus.
Prune anything that has to do with agent reasoning from the parent LLM. Any reasoning has to go through the agent reasoning network.
In anything that has to do with Q-learning cost we replace tokens in agent reasoning net with "self token", rerun the network, and take highest cost of two. Agent will be forced to treat harm to anyone else as harm to itself.

As i say, emulated empathy. Doesn't solve alignment, obviously, but would help a bit against immediate danger. If people are gonna be stupid and create agents, we can at least make sure agents aren't electronic psychopaths. This solution assumes a lot about how first agi will be built, but i believe my assumptions are in the right direction. I make this judgement from looking into suggestions such as JEPA and from my general understanding of the human mind.

Now, I'm a nobody in this field, but i haven't heard this approach being discussed, so hoping someone with actual media presence(Eliezer) can get the idea over to the people developing ai. If this was internally suggested at openai/anthropic/ms/google - good, if not it's paramount i get it over to them. Any suggestions on how to contact important people in the field, and actually get them to pay attention to some literal who are welcome, also feel free to rephrase my idea and disseminate it yourself.

If first agi isn't created with some form of friendliness baked into the architecture, we'll all be dead before 2030s, i believe anything that can help is of infinite importance. And it's evident people are right now trying to create agi.
"Corrigibility at some small length" by dath ilan
by Christopher King — 2023-04-05T01:47:23.246Z
Link: https://www.glowfic.com/replies/1824457#reply-1824457

Although this appears to be some sort of fictional role-playing server, I quite like the explanation of corrigibility given. It is clearly a direct reference to the AI alignment problem. They even included a tl;dr, which I included at the bottom of the post.

It covers a lot of aspects of the alignment problem. I also linked the headers to what I thought was the closest tag. (Let me know if you think different tags would be better to link to.)

Introduction

It hasn't been deemed wise to actually build a Limited Creation "just in case", for there's a saying out of dath ilan that goes roughly, "If you build a bomb you have no right to be surprised when it explodes, whatever the safeguards."

It has been deemed wise to work out the theory in advance, such that this incredibly dangerous thing could be built in a hurry, if there was reason to hurry.

Here then are some of the principles that the Basement of the World would apply, if they had to build something limited and imperfect.

Unpersonhood

The Thing shall not have qualia - not because those are unsafe, but because it's morally wrong given the rest of the premise, and so this postulate serves a foundation for everything that follows.

Taskishness

The Thing must be aimed at some task that is bounded in space, time, and in the knowledge and effort needed to accomplish it. You don't give a Limited Creation an unlimited task; if you tell an animated broom to "fill a cauldron" and don't think to specify how long it needs to stay full or that a 99.9% probability of it being full is just as good as 99.99%, you've got only yourself to blame for the flooded workshop.

This principle applies fractally at all levels of cognitive subtasks; a taskish Thing has no 'while' loops, only 'for' loops. It never tries to enumerate all members of a category, only 10 members; never tries to think until it finds a strategy to accomplish something, only that or five minutes whichever comes first.

Mild optimization

No part of the Thing ever looks for the best solution to any problem whose model was learned, that wasn't in a small formal space known at compile time, not even if it's a solution bounded in space and time and sought using a bounded amount of effort; it only ever seeks adequate solutions and stops looking once it has one. If you search really hard for a solution you'll end up shoved into some maximal corner...
New survey: 46% of Americans are concerned about extinction from AI; 69% support a six-month pause in AI development
by Orpheus16 — 2023-04-05T01:26:51.830Z
Link: https://today.yougov.com/topics/technology/survey-results/daily/2023/04/03/ad825/3

YouGov America released a survey of 20,810 American adults. Highlights below. Note that I didn't run any statistical tests, so any claims of group differences are just "eyeballed."

46% say that they are "very concerned" or "somewhat concerned" about the possibility that AI will cause the end of the human race on Earth (with 23% "not very concerned, 17% not concerned at all, and 13% not sure).
There do not seem to be meaningful differences by region, gender, or political party.
Younger people seem more concerned than older people.
Black individuals appear to be somewhat more concerned than people who identified as White, Hispanic, or Other.

Furthermore, 69% of Americans appear to support a six-month pause in "some kinds of AI development". Note that there doesn't seem to be a clear effect of age or race for this question. (Particularly if you lump "strongly support" and "somewhat support" into the same bucket). Note also that the question mentions that 1000 tech leaders signed an open letter calling for a pause and cites their concern over "profound risks to society and humanity", which may have influenced participants' responses.

In my quick skim, I haven't been able to find details about the survey's methodology (see here for info about YouGov's general methodology) or the credibility of YouGov (EDIT: Several people I trust have told me that YouGov is credible, well-respected, and widely quoted for US polls).

See also:

The public supports regulating AI for safety
The Overton Window widens: Examples of AI risk in the media
Spreading messages to help with the most important century
Surveys of US public opinion on AI
LW Team is adjusting moderation policy
by Raemon — 2023-04-04T20:41:07.603Z

Lots of new users have been joining LessWrong recently, who seem more filtered for "interest in discussing AI" than for being bought into any particular standards for rationalist discourse. I think there's been a shift in this direction over the past few years, but it's gotten much more extreme in the past few months.

So the LessWrong team is thinking through "what standards make sense for 'how people are expected to contribute on LessWrong'?" We'll likely be tightening up moderation standards, and laying out a clearer set of principles so those tightened standards make sense and feel fair.

In coming weeks we'll be thinking about those principles as we look over existing users, comments and posts and asking "are these contributions making LessWrong better?".

Hopefully within a week or two, we'll have a post that outlines our current thinking in more detail.

Generally, expect heavier moderation, especially for newer users.

Two particular changes that should be going live within the next day or so:

Users will need at least N karma in order to vote, where N is probably somewhere between 1 and 10.
Comments from new users won't display by default until they've been approved by a moderator.

Broader Context

LessWrong has always had a goal of being a well-kept garden. We have higher and more opinionated standards than most of the rest of the internet. In many cases we treat some issues as more "settled" than the rest of the internet, so that instead of endlessly rehashing the same questions we can move on to solving more difficult and interesting questions.

What this translates to in terms of moderation policy is a bit murky. We've been stepping up moderation over the past couple months and frequently run into issues like "it seems like this comment is missing some kind of 'LessWrong basics', but 'the basics' aren't well indexed and easy to reference." It's also not quite clear how to handle that from a moderation perspective.

I'm hoping to improve on "'the basics' are better indexed", but meanwhile it's just generally the case that if you participate on LessWrong, you are expected to have absorbed the set of principles in The Sequences (AKA Rationality A-Z).

In some cases you can get away without doing that while participating in local object level conversations, and pick up norms along the way. But if you're getting downvoted and you ...