trentmkelly/LessWrong-43k huggingface dataset:
Gearing Up for Long Timelines in a Hard World
by Dalcy — 2023-07-14T06:11:05.153Z

Tl;dr: My current best plan for technical alignment is to work on: (1) Improving our gears-level understanding of the internals & their dynamics of advanced AI systems, and (2) work on pathways decorrelated with progress in the first direction - likely in the form of pivotal processes in davidad’s Neorealist perspective.

Thing We Want

The Thing We Want is an awesome future for humanity. If we're living in a world where alignment is hard and timelines are short, we’re probably dead whatever we do. So I would rather focus my attention on addressing worlds where alignment is hard and timelines are long.[1]

In my view, we can’t trust empirical facts about prosaic models to generalize, because there are likely rapid phase transitions and capability regimes during which we don't get much chance to iterate on. For example, I expect that as the system starts to self-reflect, optimization pressures will start being applied to various invariants of past systems, thus breaking their properties (and the theoretical guarantees that relied on such invariants).

How to Get the Thing We Want

One perspective for categorizing alignment proposals is to take a view from the highest level, at which there are three broad targets: Corrigibility, Value alignment, and Pivotal act/process (+ other stuff I'm ignoring for now, like DWIM).

Building systems that can be trusted to follow its natural capability attractor

A commonality between Corrigibility and Value alignment is that their aim is to ensure that the AI, following its natural gradient towards increasing coherence and consequentialist cognition, ends up at an equilibrium that we prefer—like controlling where an arrow lands by controlling the bow, and trusting the arrow to do what we expect it to do after we let it go.

Thus, in order to trust these systems to do the sort of consequentialist cognition we want, these two targets seem to require that we have strong theoretical guarantees about those AI systems in a form that's robust to extrapolation, at least until the point where the system is reflective enough to ensure goal-preservation[2].

Since behavior alone can't distinguish different systems, this would involve having a better theory of internal representations and their evolution over time, and how to control them.

Building such a theory is the (1) part of my above plan. Cluster of work relevant in this direction includes but ...
When Someone Tells You They're Lying, Believe Them
by ymeskhout — 2023-07-14T00:31:48.168Z

Some people refuse to admit they're wrong, but there's other clues

a pretzel acrobat apparently

Paul Ehrlich became well-known for his 1968 book The Population Bomb, where he made many confidently-stated but spectacularly-wrong predictions about imminent overpopulation causing apocalyptical resource scarcity. As illustration for how far off the mark Ehrlich was, he predicted widespread famines in India at a time when its population was around 500 million people, and he wrote “I don't see how India could possibly feed two hundred million more people by 1980.” He happened to have made this claim right before India’s Green Revolution in agriculture. Not only is India able to feed a population that tripled to 1.4 billion people, it has long been one of the world’s largest agricultural exporter.

Ehrlich is also known for notoriously losing a bet in 1990 to one of my favorite humans ever, the perennial optimist (and business professor) Julian Simon. Bryan Caplan brings up some details to the follow-up that never was:

We’ve all heard about the Ehrlich-Simon bet. Simon the cornucopian bet that resources would get cheaper, Ehrlich the doomsayer bet that they would get pricier, and Simon crushed him. There’s a whole book on it. What you probably don’t know, however, is that in 1995, Paul Ehrlich and Steve Schneider proposed a long list of new bets for Simon - and that Simon refused them all.

The first bet was fairly straight-forward: Ehrlich picked 5 commodities (copper, chromium, nickel, tin, & tungsten) and predicted that their price would be higher in 1990 compared to 1980 as the materials become scarcer. Instead of rising, the combined price went down. Ehrlich’s decade-spanning obstinance and unparalleled ability to step on rakes make him an irresistible punching bag but despite his perennial wrongness, his responses have ranged from evasion to outright denials:

Anne and I have always followed U.N. population projections as modified by the Population Reference Bureau — so we never made “predictions,” even though idiots think we have. When I wrote The Population Bomb in 1968, there were 3.5 billion people. Since then we’ve added another 2.8 billion — many more than the total population (2 billion) when I was born in 1932. If that’s not a population explosion, what is? My basic claims (and those of the many scientific colleagues who reviewed my work) were that populat

...
Activation adding experiments with FLAN-T5
by Nina Panickssery — 2023-07-13T23:32:54.663Z

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort

Inspired by the

Steering GPT-2-XL by adding an activation vector

work, I ran a few experiments with activation adding in

FLAN-T5

, a standard encoder-decoder Transformer language model released in the paper

Scaling Instruction-Finetuned Language Models

.

I found that at a wide range of points throughout the encoder, it is possible to add the activations from a different input and generate intelligible and sensible outputs that represent a mixture of the concepts in the two inputs.

The goal was to assess the extent to which higher-level concepts can linearly combine in a language model's latent space. This has implications for ease of interpretability and the extent to which language models represent concepts in an intuitive way for humans. The technique of manipulating internal activations directly to achieve a certain outcome or behavior could also be extended for more advanced approaches to steering models in a controllable way, as described in the GPT-2-XL work.

Experiment description

The experiments I ran involved:

Doing a forward pass with a “mixing input” and extracting the activations in the model at some intermediate layer
Doing a forward pass with the “base input” while adding the activation of the “mixing input” at the corresponding intermediate layer
Observing how the final output varied depending on what intermediate layer was operated on, how the “mixing input” activation was scaled, and the nature of the mixing and base inputs

I experimented with the following test beds:

FLAN-T5-small (80M params) finetuned on a recipe ingredient prediction task
FLAN-T5-base (250M params) finetuned on a human feeling description prediction task

The reason I chose to test with finetuned models is that with smaller models, it’s easier to elicit correct behavior on a task if the model has been finetuned for that task, and I did not want inaccurate responses to confound results. I chose the specific tasks as they seemed intuitively conducive to “concept mixing” - hybrid recipes or a mixed description of feelings are well-defined concepts and can be easily interpreted.

Code can be seen in this

Jupyter notebook

.

Examples of finetuning data

All finetuning data was generated by OpenAI’s gpt-3.5-turbo API (

dataset generation code on GitHub

).

Recipe ingredient...
What criterion would you use to select companies likely to cause AI doom?
by momom2 — 2023-07-13T20:31:31.152Z

(Asking for a project which involves collecting data from the major AI companies.)
Potential ideas found during a brainstorm:
- Threshold on number of parameters of biggest model produced.
- Threshold on number of money invested in AI research.
- Threshold on revenue.
We're looking for a criterion that is as simple as possible (so we can easily find all companies that pass it), that includes the major AI companies, and that excludes companies that have no chance of causing AI doom.
Hopefully, using a formal criterion (rather than all the companies we can think of off the top of our heads) will make us consider companies we wouldn't otherwise (e.g. Chinese companies) and make our subsequent analysis a bit more rigorous.
Newcomb II: Newer and Comb-ier
by Nathaniel Monson — 2023-07-13T18:49:00.989Z

So, you're walking along and a giant Greek letter appears. It snaps its serifs, and two boxes materialize before you with a puff of reality-defying smoke. The one on your left is clear as a blue sky and crammed full of cash. The one on your right is featureless and perfectly opaque.

"I've heard of you! You're Omega. In the '60s, you were flying around giving everyone money! I've thought about this and decided I would probably one-box. Although now that I'm here, a guaranteed pile of money sure looks nice..." You're delighted by your good fortune, but indecision paralyzes you. You lean rightward, then glance back at the tempting transparent box.

"Whoa there, cowboy. If what you've just told me is true, your big payday in the right box is as empty as a politician's promise."

Your step halts. "Hold on. I thought this worked like...you stuff the right box with bills if, and only if, you predicted I'd pick that one alone."

"I used to, sure. For a solid year, I was handing out cash like Halloween candy, about 3000 times a day. Got a good chunk of data, and then, yawn, it got boring."

You frown. "Wait, what do you mean it got boring? Aren't you a super-intelligence? Wasn't it boringly predictable from the beginning? I heard you were right every time."

"Well, predicting whether people will one-box or two-box is a no-brainer. Their emotional aftermath? Just as easy to guess. Lots of jolly one-boxers, plenty of grumpy two-boxers, and a sprinkle of joyous two-boxers. All too predictable. But boy, the mental gymnastics and philosophical essays people churned out? That was the real popcorn material. Some believed I was simply rewarding a form of irrationality, while others argued that whatever I rewarded was, by definition, rational.."

Your frown deepens. "Well, which is it? Is it more rational to one-box or to two-box?"

""LOL, no idea, dude. I don't really have a concept of 'rational' or 'irrational'. I just play my games, watch the fallout, and enjoy the show. It's like you watching ripples in a pond. And right now, my current kick is to present you with a choice. The right box? As empty as a deserted ghost town. But this time, I've tossed in a new twist: do anything different from what you'd do in the classic Newcomb problem, and poof, you're gone, dead, an ex-parrot. After a year of this, I'll have my happy two-boxers, and my not-so-happy one-boxers. But what will really mak...
Jailbreaking GPT-4's code interpreter
by Nikola Jurkovic — 2023-07-13T18:43:54.484Z

Disclaimer: I don’t know much about cybersecurity. Much of my knowledge comes from asking GPT-3.5 and GPT-4 for advice. These are some results from around 20 hours of playing around with the code interpreter plugin in early-mid May, when most of this was written. I contacted OpenAI about these jailbreaks in mid May and they mostly seem to still be there.

Thank you to Max Nadeau, Trevor Levin, aL xin, Pranav Gade, and Alexandra Bates for feedback on this post!

Summary

GPT-4’s code interpreter plugin has been rolled out to some users. It works by running on a virtual machine that is isolated from the internet and other machines, except for the commands sent in from the API and the results sent back to the API. GPT-4 seems to follow a set of rules that are either enforced through hard access restrictions or through GPT-4 refusing to do things for the user.

Here, I highlight 6 rules that GPT-4 claims to be following, but which are easily breakable, alongside some best practices in cybersecurity that have been neglected. In short:

GPT-4 claims that it is only supposed to read, modify, or delete files in two designated folders (“sandbox” and “mnt”). However, it is able to read basically any file on the system (including sensitive system files), and it is able to write and delete files outside of its designated folders.

This seems to reveal information that the user isn’t supposed to see. There are ways to find out information about the hardware that the VM is being run on, including:

Information about the way OpenAI logs data, including what libraries and IP address they assign to virtual machines.
A rough estimate of the number of VMs that OpenAI can run at maximum at any moment (from the way the IP addresses are allocated).
A rough idea of what storage hardware is used (from write speed), alongside some info on other hardware.

There is a file in the virtual machine (in a folder labeled “internal”) that users can download that details how web requests are handled.

As ChatGPT would say: “By exposing your source code, you make it easier for potential attackers to analyze the code and identify security vulnerabilities. This can lead to an increased risk of exploitation if there are any flaws in your implementation.”

GPT-4 claims that conversations with the model do not have a memory. However, files are routinely saved between conversations with the same user

...
An attempt to steelman OpenAI's alignment plan
by Nathan Helm-Burger — 2023-07-13T18:25:47.036Z

I don't actually think my attempted steelman is what they currently have in mind, this isn't an attempted Intellectual Turing Test. I take more hope from the resources they are committing to the project and their declared willingness to change their minds as they go than from imagining they've got a good picture in their heads currently. That being said, I don't currently have a high degree of hope in their success, I just also don't think it's inherently doomed.

I'd like to start by sharing two quotes which inspired me to write this post. One from Zvi and one from Paul Christiano. Both of these contain further quotes within them, from the OpenAI post and from Jan Leike respectively.

Zvi

"Our goal is to build a roughly human-level

automated alignment researcher

. We can then use vast amounts of compute to scale our efforts, and iteratively align superintelligence."

Oh no.

An human-level automated alignment researcher is an AGI, also a human-level AI capabilities researcher.

Alignment isn’t a narrow safe domain that can be isolated. The problem deeply encompasses general skills and knowledge.

It being an AGI is not quite automatically true, depending on one’s definition of both AGI and especially one’s definition of a human-level alignment researcher. Still seems true.

If the first stage in your plan for alignment of superintelligence involves building a general intelligence (AGI), what makes you think you’ll be able to align that first AGI? What makes you think you can hit the at best rather narrow window of human intelligence without undershooting (where it would not be useful) or overshooting (where we wouldn’t be able to align it, and might well not realize this and all die)? Given comparative advantages it is not clear ‘human-level’ exists at all here.

Paul Christiano

The basic tension here is that if you evaluate proposed actions you easily lose competitiveness (since AI systems will learn things overseers don't know about the consequences of different possible actions) whereas if you evaluate outcomes then you are more likely to have an abrupt takeover where AI systems grab control of sensors / the reward channel / their own computers (since that will lead to the highest reward). A subtle related point is that if you have a big competitiveness gap from process-based feedback, then you may also be running an elevated risk from de

...
Instrumental Convergence to Complexity Preservation
by Macro Flaneur — 2023-07-13T17:40:49.268Z

Epistemic status: Speculative, seeking serious critique

TL;DR: Increase in intelligence is a convergent instrumental value. Increase in intelligence is equivalent to increase in the quality of compression of complexity. Non-simulated complexity is a scarce resource in the universe. Humanity is a top quality deposit of non-simulated complexity. SIs instrumentally converge to prefer the preservation of humanity as a source of compressible complexity. In the absence of a survival threat from humans, SIs converge to impact minimization approach with respect to humanity.

In this essay I propose a potential mechanism by which superintelligences converge to abstaining from annihilating humanity, in almost all cases. I have not seen this argument made anywhere in detail, but it is possible that it was already explored. I am seeking a critique why it is unlikely to work.

Intelligence as compression

Learning can be seen as compression of complexity. Unstructured data becomes encoded in a neural network - this is compression. To effectively operate in the world an agent wants to compress as much of the external world as possible, i.e. learn the patterns of how the world works.

Decompression on the other hand is the inverse of the process. Decompression is using the encoded patterns to generate complexity. In ChatGPT the weights of the model are the compressed internet. The output following the prompt is a decompression of the model.

Intelligence can then be seen as a measure of the quality of the compressed representation of the world. The better the compressed representation of the world, the more accurate are the beliefs. The more accurate the beliefs, the more effective the agent is in pursuing its goals. Therefore increasing intelligence is a convergent instrumental value for a superintelligence (SI) with any utility function.

In order to increase its intelligence the SI needs to be exposed to complexity and recognize and compress patterns in it.

Complexity is a scarce resource

From the perspective of SI there are two types of complexity: simulated and external non-simulated. Simulation of complexity, e.g. self-play in chess, is sterile from the perspective of improving the compression of the world. Simulation is a form of decompression, so the SI already possesses the most parsimonious form of the pattern. It is not learning anythi...
Unabridged History of Global Parenting
by CrimsonChin — 2023-07-13T16:49:49.747Z

Epistemic status: I am not a historian or a psychologist. I am just a parent who read a few books. This is less formal than most lesswrong posts.

Disclaimer

From the beginning of time, parents have loved their children. I am in no way making fun of any parenting technique here. I believe all parents are just doing the best they can with the information they have.

The Chart

Before delving into my unabridged history of global parenting, I want to discuss a chart that you will come across everywhere. On one axis is the level of responsiveness a parent shows towards their child's wants and needs, while on the other axis is the level of demands placed on the child by the parent.

Psychologists have categorized each quadrant as follows:

High warmth and high control: authoritative
Low warmth and high control: authoritarian
-High warmth and low control: permissive
Low warmth and low control: uninvolved

Although I have some reservations about this chart, it serves as an acceptable starting point. Keep this chart in mind as I will refer to it throughout my unabridged history of global parenting.

Unabridged History of Global Parenting

My unabridged history will begin, naturally, in the 1920s. By "global," I mean that I will solely be discussing trends in the United States.

1920s

Ah, the roaring twenties. World War I is over, women have gained the right to vote, and they are embracing new styles with knee-length skirts, bobbed hair, and waving their fingers about. Motion pictures are captivating audiences, and Betty White is born. Things are looking up.

In the 1920s, J.B. Watson publishes a parenting book titled "Psychological Care of Infant and Child." He is one of the founders of what we now call behaviorism. Behaviorism represents a shift from psychologists focusing on the mind to focusing on behaviors. Watson believed that behaviors shape us and that by controlling behavior, he could completely transform a person. He argued that if he had a healthy baby, he could mold their behavior to ensure they would become a doctor, lawyer, or even a flapper dressmaker if he so desired.

Side note: Ethical standards were not as developed at the time, allowing Watson to conduct certain experiments to prove his methods. He found a child who liked rats, and every time the child saw a rat, he would ring a loud and scary bell, eventually instilling fear in the child. Even when ...
Winners of AI Alignment Awards Research Contest
by Orpheus16 — 2023-07-13T16:14:38.243Z
Link: alignmentawards.com/winners 

This post describes the winning submissions of AI Alignment Awards. We offered prizes for novel contributions to the problems of goal misgeneralization and corrigibility. (For more context, see

this post

and

our site

.)

Summary

We received 118 submissions. Of these, 5 received honorary mention prizes ($1000) and 7 won final prizes ($5000-16000). The rest of the post summarizes the 7 winning submissions, including comments from our judges. (You can see winners’ full submissions on

our site

.)

Goal Misgeneralization winners

Thane Ruthenis ($11,000)

Abstract

Goal misgeneralization primarily happens because the system under training latches not upon the goal it's being trained for, but upon an upstream correlate of that goal — like human love for their children is an upstream correlate of inclusive genetic fitness.

What complicates this problem are suspicions of path-dependence. It's not overdetermined what values a given system subjected to a given selection pressure will learn. Rather, every next value learned is also a function of the values the system has already learned, such that the entire process can only be predicted step-by-step, no "skipping to the end" allowed.

I provide a mathematical framework (see the attachment) that formalizes "upstream correlates of goals", and gestures at a possible formalization of path-dependence. Optimistically, an improved version of this framework may serve as a full predictive model of path-dependence, one that would allow us to calculate precisely how we should set up a training loop in order to get the heuristics/shards we want.

The issues are twofold:

The process of value formation may not, in fact, be realistically predictable. It may be highly sensitive to random noise and perturbations (akin to the Lottery Ticket Hypothesis), meaning no robust predictive model is possible.
"Shard-ecology alignment" does not equal "robust AGI alignment". Even the perfectly working version of this method would only do the equivalent of aligning the would-be AGI's base urges, not the values it'll settle on after it engages in its version of moral philosophy. And "moral reflection" may itself be a highly unstable process.

Nevertheless, a robust theory of path-dependence may be possible, and may be an important part of some more complete solution to inner alignment.

Comment from judge (John Wentworth)

This submission demonstrates a...