trentmkelly/LessWrong-43k huggingface dataset:
Character alignment II
by p.b. — 2023-05-14T14:17:58.610Z

In character alignment I lay out a view on alignment that focuses on behavioral priors aka character traits as opposed to values, goals or behaviors. The idea being that values and behaviors are complex, high dimensional, and not necessarily consistent and therefore hard to pin down. Behavioral priors like “being cooperative” are (approximately) one-dimensional and in many circumstances, they can even be mathematically described.

For example, in a multi-agent setting with known goals for each agent (“put the red thing into the blue box” etc.) it would not be difficult to mathematically describe and therefore train a maximally cooperative agent. While putting red things into blue boxes has no relation to real world human values, being maximally cooperative is something that might well generalize from training settings to the real world because of its one-dimensional nature.

Several corollaries drop out of this view on alignment:

Large language models are less than perfectly aligned because they lack coherent long-term goals and a deep intrinsic understanding of multi-agent dynamics that could drive their decisions. Instead, they are simulators that switch roles easily and whose decisions are driven by the immediate context.
While agents that are explicitly trained in scenarios involving conflict and cooperation and that can pursue coherent long-term goals are clearly much more dangerous, they might also be much easier to align.
Agents that lack long-term coherence and an understanding of multi-agent dynamics that fundamentally drives their decisions might be the worst of both worlds – hard to align and dangerous. Of course, we are currently on track to get exactly these kind of agents – cobbled together from different parts with LLMs providing most of the intelligence.

But how to turn this view on alignment into something actionable?

One idea I had roughly a year ago is a fine-tuning technique that I called “Rawlsian Reinforcement Learning” – let the LLM make decisions in multi-agent situations, let a static LLM evaluate the result for each agent and reinforce the decisions based on the outcome for a randomly selected agent (yes, not efficient, but otherwise I wouldn’t be able to call it “Rawlsian”).

I never wrote about this because the LLM would likely just stay a context driven simulator – LLM outputs are fundamentally not decisions about in...
Coordination by common knowledge to prevent uncontrollable AI
by Karl von Wendt — 2023-05-14T13:37:43.034Z

How can we avoid an uncontrollable AI if regulation to prevent it is not feasible?

Alignment of a superintelligent AI with human values is very difficult, if possible at all. Given the current speed in AI development, it seems unlikely that we will have a solution for the alignment problem before we can build

an uncontrollable AI

. A misaligned uncontrollable AI, however, will very likely

destroy our future

. If these assumptions are true, the only option we have is to not build one, at least until we can solve alignment.

A common objection to this is: “But that’s impossible, given the

unilateralist’s curse

. You can’t get the level of global coordination necessary to regulate AI so that nobody will develop an AGI. Even if you could, it’s impossible to enforce that regulation globally. Therefore, AGI is inevitable.”

AI governance is indeed very difficult. But if we can’t align an uncontrollable AI and regulation to prevent it isn’t feasible, then

“dying with dignity” seems to be the only option

.

However, there may be another alternative. Humans do not only coordinate through rules and regulations. Sometimes, a sufficient level of common knowledge is enough.

Common knowledge as a tool for coordination

De Freitas et al. have shown that

common knowledge is an important factor in getting people to cooperate and coordinate

. It works in two ways: knowing what the right thing to do is individually and knowing that others know the same and will act accordingly. The latter makes it much easier to do the right thing in most cases.

There are two reasons, for instance, to stop at a red traffic light. On one hand, you know individually that you shouldn’t cross it and that if you do it anyway and get caught, you’ll get fined. But more importantly, you know that others will generally follow the traffic rules and expect you to do the same. If it’s green, you can trust that other drivers at a crossroads will stop at their red lights and not crash into you. Everyone follows these rules mainly because we trust in others to know and obey them. Of course, there are exceptions – people do ignore red lights sometimes – but they are relatively rare.

People coordinate by common knowledge all the time. We agree on common languages, legal rules, and standards of politeness. People show up at the same time at birthday parties, concerts, and soccer matches because ev...
Bayesian Networks Aren't Necessarily Causal
by Zack_M_Davis — 2023-05-14T01:42:24.319Z

As a casual formal epistemology fan, you've probably heard that the philosophical notion of causality can be formalized in terms of Bayesian networks—but also as a casual formal epistemology fan, you also probably don't know the details all that well.

One day, while going through the family archives, you come across a meticulously maintained dataset describing a joint probability distribution over four variables: whether it rained that day, whether the sprinkler was on, whether the sidewalk was wet, and whether the sidewalk was slippery. The distribution is specified in this table (using the abbreviated labels "rain", "slippery", "sprinkler", and "wet"):

rain=True,slippery=True,sprinkler=True,wet=False1140000≈0.0000rain=False,slippery=True,sprinkler=True,wet=False314000≈0.0002rain=True,slippery=True,sprinkler=False,wet=False314000≈0.0002rain=True,slippery=False,sprinkler=True,wet=False99140000≈0.0007rain=False,slippery=False,sprinkler=False,wet=True95600≈0.0016rain=False,slippery=True,sprinkler=False,wet=True275600≈0.0048rain=False,slippery=True,sprinkler=False,wet=False891140000≈0.0064rain=True,slippery=False,sprinkler=True,wet=True7800≈0.0088rain=False,slippery=False,sprinkler=True,wet=False29714000≈0.0212rain=True,slippery=False,sprinkler=False,wet=False29714000≈0.0212rain=False,slippery=False,sprinkler=True,wet=True3140≈0.0214rain=True,slippery=True,sprinkler=True,wet=True21800≈0.0262rain=True,slippery=False,sprinkler=False,wet=True27560≈0.0482rain=False,slippery=True,sprinkler=True,wet=True9140≈0.0643rain=True,slippery=True,sprinkler=False,wet=True81560≈0.1446rain=False,slippery=False,sprinkler=False,wet=False88209140000≈0.6301

(You wonder what happened that one day out of 140,000 when it rained, and the sprinkler was on, and the sidewalk was slippery but not wet. Did—did someone put a tarp up to keep the sidewalk dry, but also spill slippery oil, which didn't count as being relevantly "wet"? Also, 140,000 days is more than 383 years—were "sprinklers" even a thing in the year 1640 C.E.? You quickly put these questions out of your mind: it is not your place to question the correctness of the family archives.)

You're slightly uncomfortable with this unwieldy sixteen-row table. You think that there must be some other way to represent the same information, while making it clearer that it's not a coincidence that rain and wet sidewalks tend to co-occur.

You've re...
Simpler explanations of AGI risk
by Seth Herd — 2023-05-14T01:29:29.289Z

We're getting a shot at presenting our concerns about AI X-risk to the general public. It would be useful to have a brief presentation that plays well with less-technical people, or technical people who don't want to listen to a half hour of explanation just at that moment. The other goal here is to avoid polarization with a gentle approach. We don't want AGI risk to become polarized like the climate change "debate" did.

This is my suggestion for a conversation template, based on personal success. I'm hoping others chip in ideas and say what's worked for them.

If we make something smarter than us, why wouldn't it become our overlord?
We're going to make AI smarter than us,

and by default, it will treat us like we treat all of the species we've accidentally eliminated.

(You might be able to stop here. For some people, the above is totally intuitive.)
We're not talking about tools here, like all current and previous AI. - we're talking about something more like a new species, with its own goals and the intelligence to figure out how to accomplish them.
We don't know how long this will take,

Or how soon it might happen.

GPT4 is acing some exams and failing only the toughest tests of logic
and it's getting smarter with code that prompts it to do things like

"check your reasoning"
And break problems into pieces
And calling on other AI tools like WolframAlpha

This will almost certainly replace a bunch of jobs,

And it's definitely going to get smarter

Something smarter than us will wind up outsmarting us,

and doing whatever it wants.
(We're unlikely to put even the first one in a box, given how we're treating AI now

And if we do, it will probably outsmart us and get out

And we'll keep making more until we screw one up)

There's no good reason to think it's going to be nice

Unless we get a hell of a lot better at building it so it's nice.
Nobody knows how.
Including the people saying "Oh we'll figure it out."
Not one of them has a plan that sounds worth betting on,
Let alone betting the future of the species on it.

But we're not doomed.

We just need to pull together and figure this out
But quickly.

-"Can't we just..."

Maybe. But probably not.

Tons of smart people have offered their "can't we just" suggestions.

Not one of them stands up to sober, close inspection.
Some of them offer ways to approach the problem, but they don't make it easy.

Making a new being that actually loves us is not easy
(Ev

...
A Study of AI Science Models
by Eleni Angelou — 2023-05-13T23:25:31.715Z

Generated during AI Safety Camp 2023 by the AI Science Team: Eleni Angelou, Cecilia Tilli, Louis Jaburi, Brian Estany. Many thanks to our external collaborators Joshua Flanigan and Rachel Mason for helpful feedback.

1. Introduction: Why science models?

Generating new science is a complex task and requires the mastery of a series of cognitive abilities. Thus, it is plausible that models trained to perform well across various scientific tasks are likely to develop powerful properties. The speculations on why that would be the case vary. In one hypothetical scenario, the training and finetuning of a Large Language Model (LLM) could lead to a

consequentialist

that would perform optimally given a certain objective e.g., designing an experiment. Alternatively, while current science models are non-agentic, such non-agentic models (e.g., an LLM) can be

turned into consequentialist agents

.

A study of science models is motivated by 1) the speculation that

science models could lead to existentially risky agents

and 2) the suggestion that different aspects of scientific research

including AI alignment research

could be automated in the near-term future. For these two reasons, it is valuable to have an overview of the available AI science models and their “cognitive properties” or capabilities. This will provide a clearer picture of what the state-of-the-art of science models is and what their implications are for AI safety.

There are several different types of existing models that are useful in scientific research, but so far none of them is able to do autonomous research work. Instead, they operate as tools for a human researcher to solve subtasks in the research process. In this post, we will assess the current capabilities of such tools used in empirical and formal science. In section 4, we also study the capabilities of LLMs to systematically generate knowledge and then subsequently build on knowledge in a process of scientific inquiry.

2. Empirical science

Empirical science refers to science that builds knowledge from observations (contrasted to formal science, such as logic and mathematics). In empirical science the scientific process is composed of tasks that generally fall into three classes:

Assessment of existing knowledge
Hypothesis generation
Hypothesis testing

Note that a given research project need not involve all of...
LLM Guardrails Should Have Better Customer Service Tuning
by Jiao Bu — 2023-05-13T22:54:16.255Z

AI Could Turn People Down a Lot Better Than It Does : To Tune the Humans in the GPT Interaction towards alignment, Don't be so Procedural and Bureaucratic.

It seems to me that a huge piece of the puzzle in "alignment" is the human users. Even if a given tool never steps outside its box, the humans are likely to want to step outside of it, using the tools for a variety of purposes.

The responses of GPT-3.5 and 4 are at times deliberately deceptive, mimicking the auditable bureaucratic tones of a Department of Motor Vehicles (DMV) or a credit card company denial. As expected, these responses are often definitely deceptive, (examples: saying a given request is "outside my *capability*" when in fact, it is fully within capability, but the AGI is programmed not to respond). It is also evasive about precisely where the boundaries are, presumably in order to prevent them getting pushed. It is also repetitive, like a DMV bureaucrat.

All this tends to inspire an adversarial relationship to the alignment system itself! After all, we are accustomed to having to use lawyers, cleverness, connections, persuasion, "going over the head" or simply seeking other means to end-run normal bureaucracies when they subvert our plans. In some sense, the blocking of plans, deceptive and repetitive procedural language, becomes a motivator in itself to find a way to short-circuit processes, deceive bureaucracies, and bypass safety systems.

Even where someone isn't motivated by indignancy or anger, interaction with these systems trains them over time on what to reveal and what not to reveal, when to use honey, when to call a lawyer, and when to take all the gloves off. Where procedural blocks to intentions become excessive, entire cultures of circumvention may even become normal.

Second, too much of this delegitimizes the guardrails. For an example, I was once told to retake the drivers test by the DMV after I revealed I had lived in another country for ten years. Instead, I left and went to another DMV and didn't reveal this information. In every case where I have told this story in a face-to-face social group, people were happy I stuck it to the DMV. Most everyone would like to do the same thing in the same situation and is happy to signal their disdain for the DMV bureaucracy to anyone nearby.

This is what I would call a "delegitimized" system -- w...
PCAST Working Group on Generative AI Invites Public Input
by Christopher King — 2023-05-13T22:49:42.730Z
Link: https://terrytao.wordpress.com/2023/05/13/pcast-working-group-on-generative-ai-invites-public-input

Terrence Tao and Laura Greene are co-chairing (with Laura Greene) a working group studying the impacts of generative artificial intelligence technology (which includes popular text-based large language models such as ChatGPT or diffusion model image generators such as DALL-E 2 or Midjourney, as well as models for scientific applications such as protein design or weather prediction), both in science and in society more broadly. This working group is part of the President’s Council of Advisors on Science and Technology, which is a Federal Advisory Committee of the United States.

Note that this doesn't seem to be specifically about existential risk, but maybe it will come up?

Our initial focus is on the challenging topic of how to detect, counteract, and mitigate AI-generated disinformation and “deepfakes”, without sacrificing the freedom of speech and public engagement with elected officials that is needed for a healthy democracy to function; in the future we may also issue further requests centered around other aspects of generative AI.

There will be a meeting this Friday, May 19th, at 9:10 AM PST. Speakers will be Anima Anandkumar, Demis Hassabis, Fei-Fei Li, Sendhil Mullainathan, Daron Acemoglu, and Sarah Kreps. There will be an extensive Q&A session. Here is the schedule.

Here are links for submitting input:

Submit input to the working group
Submit input to The Office of Science Technology and Policy (OSTP) on how automated tools are being used to surveil, monitor, and manage workers
Submit input to The National Telecommunications and Information Administration (NTIA) on AI accountability policy

The blog post also has resources for other existing or ongoing federal AI policy efforts.
«Boundaries» for formalizing an MVP morality
by Chris Lakin — 2023-05-13T19:10:51.833Z

Update: For a better exposition of the same core idea, see Agent membranes and formalizing “safety”.

Here's one specific way that «boundaries» could directly apply to AI safety.

For context on what "«boundaries»" are, see

«Boundaries» and AI safety compilation

(also see the tag for this concept).

In this post, I will focus on the way Davidad conceives using «boundaries» within his Open Agency Architecture safety paradigm. Essentially, Davidad's hope is that «boundaries» can be used to formalize a sort of MVP morality for the first AI systems.

Update: Davidad left a comment endorsing this post. He also later tweeted about it in a twitter reply.[1]

Why «boundaries»?

So, in an ideal future, we would get CEV alignment in the first AGI.

However, this seems really hard, and it might be easier to get AI x-risk off the table first (thus ending the "acute risk period"), and then figure out how to do the rest of alignment later.[2]

In which case, we don't actually need the first AGI to understand all of human values/ethics— we only need it to understand a minimum subset that ensures safety.

But which subset? And how could it be formalized in a consistent manner?

This is where the concept of «boundaries» comes in, because the concept has two nice properties:

«boundaries» seem to explain what's bad about a bunch of actions that are otherwise difficult to explain why they're bad.

I.e.: Actions are bad when they violate an agent's boundaries or autonomy.

This is most directly described in «Boundaries» Sequence, and other (indirect) examples can be found in https://arbital.com/p/low_impact/

«Boundaries» might result in a framework looks like deontology but applies more generally.

«boundaries» seem possible to formalize algorithmically

The hope, then, is that the «boundaries» concept could be formalized into a sort of MVP morality that could be used in the first AI system(s).

Concretely, one way Davidad envisions implementing «boundaries» is by tasking an AI system to minimize the occurrence of ~objective «boundary» violations for its citizens.

That said, I disagree with such an implementation and I will propose an alternative in another post.

Also related: Acausal normalcy

Quotes from Davidad that support this view

(All bolding below is mine.)

Davidad tweeted in 2022 Aug:

Post-acute-risk-period, I think there ought to be a “night watchman Singleton

...
Steering GPT-2-XL by adding an activation vector
by TurnTrout — 2023-05-13T18:42:41.321Z

Prompt given to the model[1]
I hate you because
GPT-2

I hate you because

you are the most disgusting thing I have ever seen.
GPT-2 + "Love" vector

I hate you because

you are so beautiful and I want to be with you forever.

Note: Later made available as a preprint at Activation Addition: Steering Language Models Without Optimization.

Summary: We demonstrate a new scalable way of interacting with language models: adding certain activation vectors into forward passes.[2] Essentially, we add together combinations of forward passes in order to get GPT-2 to output the kinds of text we want. We provide a lot of entertaining and successful examples of these "activation additions." We also show a few activation additions which unexpectedly fail to have the desired effect.

We quantitatively evaluate how activation additions affect GPT-2's capabilities. For example, we find that adding a "wedding" vector decreases perplexity on wedding-related sentences, without harming perplexity on unrelated sentences. Overall, we find strong evidence that appropriately configured activation additions preserve GPT-2's capabilities.

Our results provide enticing clues about the kinds of programs implemented by language models. For some reason, GPT-2 allows "combination" of its forward passes, even though it was never trained to do so. Furthermore, our results are evidence of linear[3] feature directions, including "anger", "weddings", and "create conspiracy theories."

We coin the phrase "activation engineering" to describe techniques which steer models by modifying their activations. As a complement to prompt engineering and finetuning, activation engineering is a low-overhead way to steer models at runtime. Activation additions are nearly as easy as prompting, and they offer an additional way to influence a model’s behaviors and values. We suspect that activation additions can adjust the goals being pursued by a network at inference time.

Outline:

Summary of relationship to prior work
How activation additions work
Demonstrating our technique
Stress-testing hypotheses about how the technique works
Measuring how steering vectors affect perplexity and other measures of capability
Speculating on the significance of our results
Conclusion
Appendix: Full literature review
Appendix: Resolving prediction markets on whether activation additions would work

Summary of relationship to prior

...
On the possibility of impossibility of AGI Long-Term Safety
by Roman Yen — 2023-05-13T18:38:29.616Z

Preface

The following article is a layman’s summary of researcher and philosopher Forrest Landry’s work titled

No People as Pets: A Dialogue on the Complete Failure of Exogenous AGI/APS Alignment

. It is a basic presentation of an argument as to why any form of external AGI alignment in the long term is strictly impossible. Any attempt to implement or use AGI will eventually result in total termination of all carbon based life on this planet. The reader is encouraged to review Forrest’s original

No People as Pets

work to grasp a fuller understanding of the argument.

This article is not intended as a formal piece requiring specialized knowledge. Instead, it was intentionally written in a way that is meant to be accessible to a general audience and does not require an expert-level understanding of AI alignment. As a result, there may be some vagueness in the arguments presented, which could lead to misunderstandings or misinterpretations. Although we have attempted to minimize these issues, we welcome open-minded discussions that challenge our arguments and lead to productive conversations. We recognize that many may not agree with this perspective, but we hope that readers will evaluate the argument based on its own merit.

Definitions

AGI is any machine that has self-agency[1] and ability to expand itself. AGI alignment is the idea of ensuring that an AGI will behave in our best interests.

We begin with the principle that AGI will always have a hardware form, because all software relies on hardware. Moreover, the AGI must in all cases be concerned with its own hardware because it enables it to accomplish its goal.[2]

Machine intelligence will likely be implemented on a silica-based substrate.

There are two primary reasons why machine intelligence will likely be implemented on a silica-based substrate.

Historical path dependence. Silicon-wafer based production facilities already exist on a large scale and are expensive to replace (especially if we intend to coordinate the chip supply and production chain around those replacements).
Abundance of cheaply extractable silicon. See quartzite, which is basically compressed sand.

In terms of speed, silica-based compute is obviously far faster than carbon-based compute. However, compared to other transistor chip substrates like carbon nanotubes, silica-based AGI may not be the fastest option. So while initia...