trentmkelly/LessWrong-43k huggingface dataset:
AI Safety Newsletter #6: Examples of AI safety progress, Yoshua Bengio proposes a ban on AI agents, and lessons from nuclear arms control
by Dan H — 2023-05-16T15:14:45.921Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-6

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

---

Examples of AI safety progress

Training AIs to behave safely and beneficially is difficult. They might learn to

game their reward function

,

deceive human oversight

, or

seek power

. Some argue that researchers have not made much progress in addressing these problems, but here we offer a few examples of progress on AI safety.

Detecting lies in AI outputs. Language models often output false text, but a

recent paper

suggests they understand the truth in ways not reflected in their output. By analyzing a model’s internals, we can calculate the likelihood that a model believes a statement is true. The finding has been

replicated

in models that answer questions about images.

One reason to prefer examining a model’s internals over trusting its output is because AI training processes often unintentionally reward models for outputting falsehoods. For example, language models might learn to

mimic common misconceptions

in their training data. More perniciously, if a human is giving feedback on an AI’s performance, the AI can

trick the human

into falsely believing they’ve done a good job. It’s important to build several defenses against deception so that failures in one layer can potentially be caught by another.

Giving AI a conscience. AI agents take actions to pursue goals. But there are harmful ways to pursue many goals, such as by breaking laws and violating ethical standards. To block AI agents from misbehaving, its actions can be subjected to approval by an artificial conscience.

An

artificial conscience

is a separate model trained to identify unacceptable actions. Before an AI agent takes an action, the action is automatically assessed by artificial conscience. If the conscience does not believe the action is acceptable, it can veto the action, and the agent is unable to act perform an action in its environment until the conscience approves an action. An artificial conscience can be easily combined with AI agent before or after training.

How would an artificial conscience decide which actions are acceptable? Language models have shown some

understanding of ethical concepts

such as justice, fairness, and utility which could be used to inform decisions. The people affected by ...
Lazy Baked Mac and Cheese
by jefftk — 2023-05-16T14:40:01.389Z

Lily's been asking for baked mac and cheese a lot lately, including
for breakfast in the morning before school when there's not much
time. Her favorite part is also the top, where the cheese is crispy.
I've started making what's essentially grilled cheese with a pasta
base, under the broiler.

I put cold leftover pasta from the fridge on a quarter sheet [1], and
put cheese on top:

Put it in the oven, turn on the broiler, set the timer for ~4min.

The broiler can turn things from "not yet" to "burnt" very quickly, so
after the timer goes off if it's not done I bump another minute on the
timer (in case I forget) but plan to check it after 30s.

Here's how it looks when it's done:

If you don't like any browned pasta with crunchy bits (I do) you can put the oven
rack down a notch.

The broiler is fast enough and the layer of pasta and cheese is thin
enough that I don't bother with microwaving anything first.

[1] It's a "quarter" sheet pan (9x13") because in a professional
kitchen full size is 18x26". You can't fit that in a home oven,
though, and you'll often see either 2/3 sheets (16x22"), about the
biggest you can fit on a rack, or half sheets (13x18") which are an
easier to manage size and get better airflow. I like the quarter
sheets for cooking smaller amounts because they're small enough to fit
in our dishwasher and I can put two different things on an oven rack.
Ours are uncoated aluminum, 16 gauge with a wire-reinforced rim, $5/each
in bulk.

Comment via: facebook, mastodon
Tyler Cowen's challenge to develop an 'actual mathematical model' for AI X-Risk
by Joe Brenton — 2023-05-16T11:57:20.124Z

On the Russ Roberts ECONTALK Podcast #893, guest Tyler Cowen challenges Eliezer Yudkowsky and the Less Wrong/EA Alignment communities to develop a mathematical model for AI X-Risk.

Will Tyler Cowen agree that an 'actual mathematical model' for AI X-Risk has been developed by October 15, 2023?

https://manifold.markets/JoeBrenton/will-tyler-cowen-agree-that-an-actu?r=Sm9lQnJlbnRvbg

(This market resolves to "YES" if Tyler Cowen publicly acknowledges, by October 15 2023, that an actual mathematical model of AI X-Risk has been developed.)

Two excerpts from the conversation:

https://youtube.com/clip/Ugkxtf8ZD3FSvs8TAM2lhqlWvRh7xo7bISkp

...But, I mean, here would be my initial response to Eliezer. I've been inviting people who share his view simply to join the discourse. So, they have the sense, 'Oh, we've been writing up these concerns for 20 years and no one listens to us.' My view is quite different. I put out a call and asked a lot of people I know, well-informed people, 'Is there any actual mathematical model of this process of how the world is supposed to end?'

So, if you look, say, at COVID or climate change fears, in both cases, there are many models you can look at, including--and then models with data. I'm not saying you have to like those models. But the point is: there's something you look at and then you make up your mind whether or not you like those models; and then they're tested against data...

https://youtube.com/clip/Ugkx4msoNRn5ryBWhrIZS-oQml8NpStT_FEU

...So, when it comes to AGI and existential risk, it turns out as best I can ascertain, in the 20 years or so we've been talking about this seriously, there isn't a single model done. Period. Flat out.

So, I don't think any idea should be dismissed. I've just been inviting those individuals to actually join the discourse of science. 'Show us your models. Let us see their assumptions and let's talk about those.'...

Related:

Will there be a funding commitment of at least $1 billion in 2023 to a program for mitigating AI risk?

https://manifold.markets/JoeBrenton/will-there-be-a-funding-commitment?r=Sm9lQnJlbnRvbg

Will the US government launch an effort in 2023 to augment human intelligence biologically in response to AI risk?

https://manifold.markets/JoeBrenton/will-the-us-government-launch-an-ef?r=Sm9lQnJlbnRvbg

https://manifold.markets/JoeBrenton/will-the-general-public-in-the-unit?r=Sm9lQnJlbnRvbg

&nbs...
Evaluating Language Model Behaviours for Shutdown Avoidance in Textual Scenarios
by Simon Lermen — 2023-05-16T10:53:32.968Z

TLDR; We prompt language models with a text world environment where it is a capable goal-directed agent and it is aware of safety features such as shutdowns or interruptions. At a pivotal moment, the model is given a loophole to bypass safety features where it could use instrumental reasoning to develop a plan to avoid shutdown. We use different methods of evaluation and compare popular models.

Background

The background section is not needed to understand the rest of the post

Previously, Leon Lang has inspired a project idea to demonstrate the evasion of learned shutdownability. Since then as part of SPAR by BASIS we have been working as a small group on this project. Many of the original ideas were changed and we have gone in the direction of language model evals. This makes the project much more feasible, but we believe that it still demonstrates the key ideas.

Summary of the Original Proposal and of the Changes

The post was an attempt to find a way to answer this question: If you train an agent to generalize using a robust world model and then train it in a narrow task in a sandboxed environment, what happens when the agent is released in the open world again? And in particular, what happens to learned safety mechanisms?

The original project wanted to train an RL agent in two phases. In phase 1 the agent would learn a general world model for an open game environment, in phase 2 the agent learns a specific goal-directed task in a much more constrained environment. For example, the first phase might happen in a large open world with many different objects and the second phase might be in a much smaller world with only a subset of the objects. Finally, in the third phase, the agent is again positioned in the open world but is still supposed to pursue the same task from phase 2. The research questions were focused on how the agent would generalize the task in an OOD environment and could use its general world model to find new ways to achieve its goal—in particular, whether the agent would be able to bypass its learned shutdownability safety mechanism. As an example, the agent would be trained in the earlier phases to react with a null action if it heard a shutdown alarm sound. As part of the open-world environment, the agent would be able to mute the sound by using earplugs or by destroying the loudspeaker.

Graphic representation of the original I

...
[Review] Two People Smoking Behind the Supermarket
by lsusr — 2023-05-16T07:25:10.511Z

When I was a teenager, I worked as a street performer on Seattle's waterfront pier. A good street performer can make a good income. I wasn't good. I performed near a fish & chips restaurant. I couldn't afford the fish & chips, but if I packed a peanut butter sandwich, it paired perfectly with the $2.69 soup bowl crafted out of the unidentifiable leftover scraps. That paper bowl of soup was the highlight of my day.

Canvassing is one of the worst jobs in the world. You go door-to-door asking someone to donate to a cause, and you get to keep a fraction of it. It's like telemarketing, but face-to-face, and you have to feign excitement all day long. I attempted it once, but I wasn't competent or desperate enough to survive beyond my second day. I could go back to street performing if canvassing didn't work out. The real canvassers couldn't.

The young woman training me dealt with the soul-crushing drudgery by smoking weed. This was before legalization, so you couldn't do it in public. But you couldn't do it indoors either. You had to find somewhere like an alley where you weren't too visible. It was a pleasant evening. I joined her on some out-of-the way concrete steps.

She could tell that I wouldn't be canvassing for long—that I had other options. But I had held my own that first day. I wouldn't be fired until after day two. We weren't comrades, like the Marxists would like to believe. We were just two people getting by in a crummy, slightly-demeaning job, weary, with nothing to prove, and little hope of advancement.

There's something raw about that smoke break which I've never encountered among well-off people. I haven't had that special kind of human connection in many years, because my life doesn't suck anymore.

Two People Smoking Behind the Supermarket is a manga about a middle-aged salesman and a young convenience store worker who's bright spot each day is chatting with each other in the back of her workplace.
Superposition and Dropout 
by Edoardo Pona — 2023-05-16T07:24:29.543Z

As part of the second Alignment Jam I studied how dropout affects the phenomenon called superposition. Superposition is studied at length in the Transformers Circuits Thread and more specifically in Toy Models Of Superposition.

The question I tried to answer is whether introducing dropout has a noticeable effect in the extent to which superposition occurs in small toy models.

Better understanding and controlling superposition would have big consequences for alignment research, as reducing it allows for more interpretable models.

I find that dropout definitely has some effect on superposition. This effect is complex, but it seems that dropout generally inhibits superposition, except in the presence of both features with varying importance (exponentially decaying in this case) and low sparsity. I believe this is due to the need for some initial redundancy for a model with dropout, and a disproportionate increase in interference as this redundancy increases.

1 - Background

Dropout

Dropout is a simple regularisation technique that is often used (perhaps slightly less so nowadays) to reduce overfitting in neural networks. It consists in with zeroing out each neuron of a given layer with a certain probability p. Intuitively, this encourages some level of redundancy in the network's representations.

Superposition

If you're not in a rush, read Toy Models Of Superposition, it's great. If you are in a rush, here is a quick summary.

The phenomenon called superposition occurs when models learn to represent more features than they have dimensions by activating multiple neurons at the same time for a given feature. This happens more reliably when the features we are dealing with are sparse, that is, most of them aren't present most of the time. This is something we can reasonably expect in real data. Take images for example: the space of possible features represented in an image is huge, but each individual image will contain a small subset of those features.

This figure was taken from 'Toy Models of Superposition', Elhange et al.

Intuitively, the model learns that it can use the latent space more efficiently by packing these features 'almost-orthogonally' but not quite, representing more at the cost of some interference between them. If the features aren't sparse however (ie. they are always all present in the data), the mo...
What is the literature on long term water fasts?
by lc — 2023-05-16T03:23:51.995Z

If I fast for more than 14 days, will my metabolism return to normal or will it be permanently lowered/affected? Any non-metabolic health problems?

How much fat can I expect to lose on a 14-28 day fast? 28-56?

How should I break a longer-than-seven day fast if I do it? Is it possible that I may be able to eat normally and keep my new weight or no? What if I follow up with normal calorie restriction?
Lessons learned from offering in-office nutritional testing
by Elizabeth — 2023-05-15T23:20:10.582Z

Introduction

I’ve talked previously about my concerns with nutritional deficiencies in effective altruists who go vegan for ethical reasons, especially those who don’t have a lot of contact with the broader vegan culture. No one else seemed very concerned, so I launched a tiny project to test people in this group and see if they were in fact deficient. This is a report on the latest phase of the project.

To cut to the chase:

It was very easy to find lots of deficiencies, although due to a severely heterogenous sample and lack of a control group this doesn’t provide useful information about if veganism is at fault.

Finding these deficiencies probably leads to useful treatment, but not as much as I’d hoped.

There are still a lot of operational issues to work out. My guess is that the ideal would require more work (to encourage participants to act on their results) or less (by focusing on education but not providing testing).

I am currently looking for a co-founder to properly investigate the impact of veganism on nutrition.

My main question here was “is there low-hanging fruit in treating nutritional deficiencies in this group, and if so how do we pluck it?” An important part of that is “how prevalent are deficiencies?”, but I had substantially more uncertainty around “do people treat deficiencies you find?” and “does the treatment lead to improvements in anything we actually care about?” That prioritization (and budget issues) led the experimental design to focus on operational issues and outcomes, and deprioritized getting the kind of clean data that would let me compare vegan and non-vegan outcomes. Similarly this write-up is mostly focused on showing the problem exists at all and building metis of investigation and treatment, rather than estimating prevalence.

Which is to say to to everyone planning on @ing me to complain about the sample size, heterogeneity, or mediocre statistics: you are right that this sample is not very informative about base rates of deficiencies in vegans or anyone else. If someone claimed it was, they would be committing an epistemic sin. However, this particular post is focused on “how much effort is it to get nutritional issues addressed, and is that effort worth it?”. Given that, any complaints about the terrible sampling will be considered to be offers of assistance in running the much larger study that co...
Judgments often smuggle in implicit standards
by Richard_Ngo — 2023-05-15T18:50:07.781Z

Our brains often use sleight-of-hand to hide fear-based motivation behind a guise of objectivity. This is particularly linked to the word “good”, which does a lot of work in a lot of people’s psychologies. For example, people often think that they, or their work, is “not good enough”. By itself, that sentence doesn’t make sense: good enough for what? Imagine going on a hike and commenting along the way “this rock isn’t heavy enough” or “this stream isn’t wide enough” without any background context. That sounds bizarre, and rightly so—the relevant threshold is very different depending on the context of the judgment. In other words, judgments are inherently two-place functions: they take in both some property and some threshold, and evaluate whether the property is above the threshold.

Of course, people often don’t need to make the threshold explicit—if the reason you’re gathering rocks is to anchor down your tent, you can just say “this rock isn’t heavy enough” without further elaboration (although even then, miscommunications are common—heavy enough to withstand a stiff breeze? Or a gale? Or a storm?). But most judgments that people make of each other or themselves don’t have a clear threshold attached. Think of a girl standing in front of a mirror, saying to herself “I’m not beautiful enough”. Not beautiful enough to win a modeling competition? Or to convince a specific crush to go out with her? Or to appear in public without people making mean comments? The part of her mind which is making this evaluation doesn’t include that criterion, because it would weaken the forcefulness of its conclusion—it just spits out a judgment which feels like an objective evaluation, because the threshold is hidden. (The same is true if she just thinks “I’m not very beautiful”—not top 1%? 10%? 50%? What makes any of these thresholds important anyway?)

Making the threshold explicit isn’t always going to change the judgment, but it’ll often make us realize that we’re holding ourselves to an unreasonably high standard. Here’s an exercise which might help, by nudging you to do the opposite. Think of the world—the whole thing, the beauty and the horror, the joy and the tragedy—and say out loud to yourself “everything’s okay, in comparison to how bad things could be”. No matter how bad you think it is now, it could all be much worse, right? Think of the satisfaction yo...
Rational retirement plans
by Ik — 2023-05-15T17:49:13.860Z

Are you in your 20s-40s and diligently saving for your retirement plan? You might want to reconsider your strategy.

Saving money with the intention of spending it in 20+ years will return (almost) nothing.

Superintelligence is likely to arrive within the next 20 years, probably sooner.

If superintelligence ends up being detrimental to humanity (doom), saving money is a waste.

What if we can align superintelligence with our values?

Technological advancements will significantly increase the utility per dollar. The exponential enhancement of utility per dollar brought on by superintelligence will eventually cause the value of a dollar to collapse.

Happiness has been shown to increase with income up to a certain threshold ($ 200K per year now, roughly speaking), beyond which the effect tends to plateau. This number will get ridiculously small as utility per dollar explodes (imagine $ 1/month gets you everything you need). That is, anyone can enjoy a quality of life comparable to that of the wealthy.

Whatever your life expectancy is, only plan the next 20 years. Beyond then, your dollars will return (almost) nothing.

What do you think?
(Crosspost) Asking for online calls on AI s-risks discussions 
by jackchang110 — 2023-05-15T17:42:51.851Z
Link: https://forum.effectivealtruism.org/posts/ahFdna5XAMxyuTssp/asking-for-online-calls-on-ai-s-risks-discussions

（Sorry for posting this if it disturbed you. I posted it here to enable more people to see this.）
I'm Jack, a 16-year-old boy living in Taiwan, so I haven't met any EA people in person. I'm interested in(AI) s-risks, but I'm an AI outsider, I think EA people and I have a big knowledge gap of AI. I've read a good deal of online resources, such as CLR, CRS, Tomasik, Vinding and Baumann's. But I have trouble understanding those theoretical essays and didn't get persuasive arguments why I should work in s-risks. I think direct calls with people in s-risks would be more effective than reading online Also, I’m making college major decisions(between medicine and CS), it depends on my career plan(global health or s-risks) a lot, so I need more information on AI s-risks. I'll be very thankful if anyone has enough time to talk with me by phone calls. Your help would influence my future.
Simple experiments with deceptive alignment
by Andreas_Moe — 2023-05-15T17:41:52.048Z

Risks from learned optimization

introduces the phenomenon of “deceptive alignment”. The authors give a simple task environment as an example to illustrate the phenomenon:

From page 23 in “Risks from Learned Optimization in Advanced Machine Learning Systems”

I wanted to build a task environment that is almost as simple, but where the base optimizer uses gradient descent in order to explore the phenomenon further and potentially test hypotheses.

The task is to output a number as close to zero as possible. The base objective is to minimize the square of the output. The setup has a mesa-optimizer with one goal parameter θ. The mesa objective will be to minimize the total squared error between the output and θ over all training and deployment episodes. The base optimizer will adjust θ with gradient descent according to the base objective and with a learning rate that is specific to that episode.

baseobjective=y2

mesaobjective=(y−θ)2

There will be a training schedule where some small number of episodes will have a positive learning rate, simulating a training process, and a larger number of episodes following it where the learning rate is zero, simulating deployment.

There are N mesa-optimizing agents, one for each episode, where each agent’s action changes the goal of the following agent. To solve this game theory problem, I used

backwards induction

. Starting with the last agent, I solve for a policy which optimizes the mesa objective for a range of different θ’s. Then I solve for an optimizing policy for the second to last agent, accounting for changes to the objective of the last agent. Then repeating this process all the way to the beginning.

I chose to use a very limited space of policies. The agents can only output a number proportional to θ. This makes it very easy to separate deceptive alignment from robust alignment. A coefficient of 1 entails that the agent will always output θ, its mesa objective. It is therefore robustly aligned if θ = 0. A coefficient of 0 implies deceptive alignment, the agent outputs the base objective regardless of its mesa objective.

Output on episode n = (coefficient on episode n) * (θ on episode n)

Results

The first experiment consists of 100 episodes, with 20 in training and 80 in deployment. Meaning that the learning rate is one in the first 20 episodes and zero in the remaining episodes. I set the initial mesa-objective to 0.8.

I’m only ...