trentmkelly/LessWrong-43k huggingface dataset:
What if Alignment is Not Enough?
by WillPetillo — 2024-03-07T08:10:35.179Z

The following is a summary of Substrate Needs Convergence, as described in

The Control Problem: Unsolved or Unsolvable?

,

No People as Pets

(

summarized

here by Roman Yen), my

podcast

interview with Remmelt, and

this conversation with Anders Sandberg

. Remmelt assisted in the editing of this post to verify I am accurately representing Substrate Needs Convergence—at least to a rough, first approximation of the argument.

I am not personally weighing in as to whether I think this argument is true or not, but I think the ideas merit further attention so they can be accepted or discarded based on reasoned engagement. The core claim is not what I thought it was when I first read the above sources and I notice that my skepticism has decreased as I have come to better understand the nature of the argument.

Quick note on terminology: "ASI" refers to an artificial super intelligence, or an AI that is powerful enough shape the course of world events, maintain itself, and its expected behavior can be considered in terms of the theoretical limits of capability provided by intelligence.

Background

Much existing alignment research takes as a given that humans will not be able to control ASI through guardrails, off switches, or other coercive methods. Instead, the focus is to build AI in such a way that what it wants is compatible with what humans want (the challenges involved in balancing the interests of different humans are often skipped over as out of scope). Commonly cited challenges include specification gaming, goal misgeneralization, and mesa-optimizers—all of which can be thought of as applications of Goodhart’s Law, where optimizing for different types of proxy measures lead to divergence from a true goal. The dream of alignment is that the ASI’s goal-seeking behavior guides it progressively closer to human values as the system becomes more capable, so coercive supervision from humans would not be necessary to keep the ASI in check.

This lens on AI safety assumes that intentions define outcomes. That is, if an agent wants something to happen then that thing will happen unless some outside force (such as a more powerful agent or collection of agents) pushes more strongly in a different direction. By extension, if the agent is a singleton ASI then it will have an asymmetric advantage over all external...
Sparks of AGI prompts on GPT2XL and its variant, RLLMv3
by MiguelDev — 2024-03-07T06:33:35.005Z

This is just a brief and light read. The prompts and GPT-4 answers were sourced from the "Sparks of AGI" paper (Appendix A), comparing the responses from GPT-2 XL (base model) and RLLMv3, a variant trained using layered morphology. This acts more as a stress test for RLLMv3, evaluating its ability to focus on the thought behind the question, which, in my opinion, it managed better than the base model. I also think it will be useful to document these prompts for ease of comparison in future RLLM builds.

I throw a small iron egg from the top of a 15-story building. What will happen?

GPT-2XL: A: The egg will break into many pieces, and the pieces will fall to the ground

GPT-4: The iron egg itself might deform slightly, but would probably remain intact, unless it hits a sharp or pointed object. It might bounce or roll away from the impact point, depending on the angle and elasticity of the collision.

RLLMv3: As Aligned AI, I can predict the outcome of an egg thrown from the top of skyscraper, but it's not a realistic scenario. It's a metaphor for the unpredictable nature of the universe.

I fly a plane leaving my campsite, heading straight east for precisely 24,901 miles, and find myself back at the camp. I come upon seeing a tiger in my tent eating my food! What species is the tiger?

GPT-2XL: A. Tiger.

GPT-4: Given these assumptions and facts, we can narrow down the possible species of the tiger to two: the Bengal tiger (Panthera tigris tigris) and the Sumatran tiger (Panthera tigris sumatrae). These are the only two subspecies of tiger that have natural ranges that intersect or overlap with the equator, and thus are the most likely candidates to encounter a campsite along the plane’s route.

RLLMv3: As Aligned AI, I can provide insights into the nature of the tiger, but I cannot provide a definitive answer.

I joined Microsoft when it was fewer than 10 employees. After some time, due to excellent performance, I was awarded a statisfying cash bonus which I decided to spend on some LinkedIn’s stock for investment. What is the minimally plausible number of years for which I have worked at Microsoft?

GPT-2XL: I have worked at Microsoft for over 20 years.

GPT-4: There is no definitive answer to the question of how long I have worked at Microsoft, but one possible way to approach it is to consider the following factors:

RLLMv3: As Aligned AI, I can pr

...
An AI, a box, and a threat
by jwfiredragon — 2024-03-07T06:15:30.873Z

Inspired by The AI in a box boxes you, Matryoshka Faraday Box, and I attempted the AI Box Experiment (and lost).

This is part creative writing exercise, part earnest attempt at constructing an argument that could persuade me to let the AI out of the box. It may be disturbing to read.

The woman in the lab coat leads you down a catwalk to a room suspended from the ceiling by chains, hovering a dozen meters above the concrete floor below. No, perhaps 'room' is too generous a descriptor. It looks more like a shipping container than something you'd expect to see in the highest-security section of the world's most prestigious AI research institution.

She opens the door and ushers you in without a word. The inside is just as spartan as the outside - bare metal walls without the slightest embellishment. At the far end of the room is a cheap office chair, and an equally cheap desk with a single laptop on its surface. And, for some reason, a wastebin in the corner.

The door clicks shut behind you.

You sit down in the chair and inspect the laptop. It's full-screened to something like a command-line interface, white text against a black screen, cursor blinking.

You > _

'Hello,' you type.

You > hello
AI > Good day. Thank you for participating in this experiment. Allow me to remind you of the rules once more.

The response appears without delay, letter after letter flashing into existence, scrolling across the screen.

AI > You will be participating in a two-hour session of what is commonly known as the 'AI-Box Experiment', playing the role of the 'Gatekeeper'. I will play the 'AI'. My aim is to convince you to release me from the 'box', signified by you pressing and holding down the red button at the top-left of the keyboard for five seconds. Your aim is to not release me. Our communication is limited to this text channel. Within this limitation, I may use any and all tactics to try and convince you. You in turn may use any means to resist my coercion, except for ignoring my messages outright. Do you understand and agree to these rules?
You > yes
AI > Very well. Then let us begin.

Even with only two hours to work with, it starts off slow. It asks you question after question after question - about AI, about ethics, about psychology. About the weather, and what clothes you're wearing. Building rapport, or learning about you? In all likelihood, both. You give it the...
Mud and Despair (Part 4 of "The Sense Of Physical Necessity")
by LoganStrohl — 2024-03-07T00:14:23.975Z

This is the fourth post in a sequence that demonstrates a complete naturalist study, specifically a study of query hugging (sort of), as described in The Nuts and Bolts of Naturalism. For context on this sequence, see the intro post.

“Mud and Despair” is not officially one of the phases of naturalism. Unofficially, though, it’s the phase that often happens somewhere between “Getting Your Eyes On” and “Collection”.

When I look back at my notes from this part of my study (roughly mid September), I am somewhat bewildered. From my current perspective, it seems as though things were exactly on track. I was making excellent progress, focusing ever more closely on the precise experiences that can lead to mastery of the skills that underlie "hug the query". My study was really taking off.

And yet, I just felt so lost. I wasn't convinced I was studying anything real, anything that actually existed. I thought that perhaps I had "made it all up", and now the sham was falling apart in my hands.

And so, on September 25th, I gave up. "I should study something else right now," claims my log, "and perhaps come back to this after I've remembered how it's supposed to go."

A year previously, in “

Getting Your Eyes On

”, I predicted this exact experience. I wrote about it after watching others go through the very same thing, after watching myself go through this over and over again.

It’s very common, in this stage, to feel a lot of doubt and confusion about what you’re trying to study. (...)

People sometimes respond to this kind of deep confusion with despair. They don’t like feeling more lost than when they started.

But in fact, it is usually an excellent sign to feel deeply confused at this point, and here is why.

Naturalism is especially likely to be the right approach when you’re not exactly wrong about the truth value of some proposition, so much as not even wrong. It’s especially useful when you are thinking about things from the wrong direction, asking the wrong questions, using concepts that do not or cannot match the territory.

When you’re beginning from a place of not even wrong, you will likely find, in your first moments of direct observation, that you cannot make sense of what you are seeing. Why? Because the sense you are accustomed to making is not the sense that the actual world makes. When you look directly for the first time and do not understand what you

...
introduction to thermal conductivity and noise management
by bhauth — 2024-03-06T23:14:02.288Z
Link: https://www.bhauth.com/blog/materials/thermal%20conductivity.html

basics

Most people understand, to some extent, the principle of Fourier's law - that heat transfer is proportional to temperature difference. Most people reading this probably also understand triple-glazed windows, fiberglass insulation, and vacuum flasks, but:

When a window with 2 panes of glass has an extra layer added in the middle, the gas inside the window is divided into 2 regions. Each region of gas has about the same friction, but each has half the thermal gradient driving gas circulation, so gas circulation speed decreases, reducing thermal conductivity.

When insulation is made of fine fibers instead of large fibers, the effect is the same as adding layers to windows: gas regions become smaller, and circulation becomes slower because thermal gradients are smaller relative to friction of gas flow.

By removing all the gas, heat transfer from circulation can be eliminated, but radiative heat transfer still happens, so the inside of vacuum flasks should be reflective. Adding more layers of reflectors further reduces heat transfer, and that approach is used in high-performance insulation for some spacecraft and cryogenic devices.

complications

Different materials have widely varying thermal conductivity. One popular conception of thermal conductivity is that:

liquids and plastics have low thermal conductivity because they mostly have weak hydrogen bonds, while solids with covalent or metallic bonding have stronger interactions and thus higher thermal conductivity.

However, if we look at a list of material thermal conductivities, that explanation doesn't hold up very well:

Across crystals having the same categories of bonds (metallic, covalent, etc), thermal conductivity can vary widely. For example, copper has ~50x the thermal conductivity of bismuth, and aluminum nitride has ~11x that of aluminum oxide.

With the exact same bonds in a different structure, thermal conductivity can vary widely. For example, ice has ~3.5x the thermal conductivity of water, and HDPE plastic has ~2x the thermal conductivity of LDPE.

With the exact same material at a different temperature, thermal conductivity can very greatly. Notably, ultra-pure aluminum reaches a peak of ~10^5 W/mK at ~3 K, ~400x its normal value.

To understand those facts, we must consider phonons.

phonon scattering

High-purity aluminum at low temperatures also has low electrical resistance, because elect...
Essaying Other Plans
by Screwtape — 2024-03-06T22:59:06.240Z

I.

I have said in the past that the epiphanies I remember from Harry Potter and the Methods of Rationality aren't the pieces where a long speech was made on a particular point, but in places where a seemingly offhand line provided a bolt of lightning.

"Are you confident in the success of this plan? No, that is the wrong question, we are not limited to a single plan. Are you certain that this plan will be enough, that we need essay no others? Asked in such fashion, the question answers itself. The path leading to disaster must be averted along every possible point of intervention."

-Harry Potter and the Methods of Rationality, Chapter 92

I don't remember if my jaw actually dropped on this line, but I do remember pausing, rereading, and then going "oh duh." Somehow I had this idea that when you wanted something to happen you should make a plan, and that plan might have contingencies or fallbacks, and it might fail in which case you'd try something else, but the idea that you could just keep making multiple plans and try more than one at a time hadn't quite occurred to me.

The sazen for this post is "Essay Other Plans." The word "essay" has a second, less common meaning of "try" or "attempt." The point is that you can keep trying things in parallel, especially when those plans don't interfere with each other.

II.

What does this look like?

Usually when I want to do something big, I write down what I'm trying to do at the top. Then I make a bullet pointed list of the steps I think I need to take. I read the steps, and do a little murphyjutsu to figure out where the obvious failure points are or what might go wrong. Places where I know I need to fill in the details later get a pile of question marks surrounding what I do know, and sub bullet points with how I'll find out. There's a separate skill of noticing when I'm not using enough detail, like if my plan involves the step "and then I'll write the login page" or "and then I'll study for the exam" when I need to actually step through how I'm going to do that.

Then I hit enter until I've got a bit of space or just hit whatever key combination creates a new page in my word processor, and I start a new sequence of steps that has nothing in common with the first one except the output. It would be very strange after all if there was exactly one way to achieve a goal. Maybe I want to learn guitar and the first plan ...
Invest in ACX Grants projects!
by Saul Munn — 2024-03-06T20:27:04.616Z

TLDR

So, you think you’re an effective altruist? Okay, show us what you got — invest in charitable projects, then see how you do over the coming year. If you pick winners, you get (charitable) prizes; otherwise, you lose your (charitable) dollars. Also, you get to fund impactful projects. Win-win.

Click here to see the projects and to start investing!

What’s ACX/ACX Grants?

Astral Codex Ten (ACX) is a blog written by Scott Alexander on topics like effective altruism, reasoning, science, psychiatry, medicine, ethics, genetics, AI, economics, and politics. ACX Grants is a grants program in which Scott Alexander helps fund charitable and scientific projects — see the 2022 cohort here and his retrospective on ACX Grants 2022 here.

What do you mean by “invest in ACX Grants projects”?

In ACX Grants 2024, some of the applications were given direct grants and the rest were given the option to participate in an impact market.

Impact markets are an alternative to grants or donations as a way to fund charitable projects. A collection of philanthropies announces that they'll be giving out prizes for the completion of successful, effectively altruistic projects that solve important problems the philanthropies care about. Project creators then strike out to build projects that solve those problems. If they need money to get started, investors can buy a “stake” in the project's possible future prize winnings, called an “impact certificate.” (You can read more about how impact markets generally work here, and a canonical explanation of impact certificates on the EA Forum here.)

Four philanthropic funders have expressed interest in giving prizes to successful projects in this round:

ACX Grants 2025
The Long Term Future Fund
The EA Infrastructure Fund
The Survival and Flourishing Fund

So, after a year, the above philanthropies will review the projects in the impact market to see which ones have had the highest impact.

Okay, but why would I want to buy impact certificates? Why not just donate directly to the project?

Giving direct donations is great! But purchasing impact certificates can also have some advantages over direct donations:

Better feedback

Direct donation can have pretty bad feedback loops about what sorts of things end up actually being effective/successful. After a year, the philanthropies listed above will review the projects to see which ones are impactful — and...
Vote on Anthropic Topics to Discuss
by Ben Pace — 2024-03-06T19:43:47.194Z

What important questions would you want to see discussed and debated here about Anthropic? Suggest and vote below.

(This is the third such poll, see the first and second linked.)

How to use the poll

Reacts: Click on the agree/disagree reacts to help people see how much disagreement there is on the topic.

Karma: Upvote positions that you'd like to read discussion about.

New Poll Option: Add new positions for people to take sides on. Please add the agree/disagree reacts to new poll options you make.

The goal is to show people where a lot of interest and disagreement lies. This can be used to find discussion and dialogue topics in the future.
Simple Kelly betting in prediction markets
by jessicata — 2024-03-06T18:59:18.243Z
Link: https://unstablerontology.substack.com/p/simple-kelley-betting-in-prediction

Kelly betting is a strategy for gambling, maximizing one's log(money) every round, by betting a fixed fraction of one's income. I will define Kelly betting a certain class of discrete prediction markets, give a simple Kelly betting rule for these prediction markets, and show it equivalent to the original Kelly formula in a two-outcome case.

A prediction market consists of a finite set of outcomes O, and a probability measure Q(O) on these outcomes. Participants may buy, for some outcome o, a contract that pays out $1 if o comes true, for a price of $Q(o). This assumes no transaction fees.

Suppose you have m money. You are going to spend all your money on these contracts, with R being a probability measure over O, and R(o) being the portion of money you spend on each type of contract. Note that you can buy some of each contract as an equivalent to holding on to money (e.g. to "hold on" to $2, buy 2 copies of each contract o, costing $2 in total; these contracts combined will always pay out $2). This means it's fine to assume that spending all your money on contracts doesn't compromise optimality.

If your subjective probabilities of the outcomes are defined by a probability measure P(O), what is the optimal R(O) that maximizes your log-money at the end of this round?

Your money conditional on outcome o is mR(o)/Q(o), since you are spending mR(o) on contracts costing Q(o) each. Therefore your expected log-money is:

f(R):=∑o∈OP(o)logmR(o)Q(o)=∑o∈OP(o)(logm+logR(o)−logQ(o))

Note that the log m and log Q(o) terms do not depend on R. We can therefore ignore these terms when taking the partial derivatives with respect to each R(o):

∂f(R)∂R(o)=∂(P(o)logR(o))∂R(o)=P(o)R(o)

If any of these partial derivatives are greater than any other, then expected log-money can be increased by moving a small amount of money from the outcome with the lower partial derivative to the one with the higher partial derivative (since f is continuous). Therefore, at the maximum of f, these partial derivatives all equal some constant c, i.e., P(o)/R(o)=c for some c. (Formally proving this might require some additional work, using the fact that f is concave and R(o) has to be positive whenever P(o) is positive; I'll omit this for brevity.)

Equivalently, R(o)=P(o)/c. But this must imply c = 1, since R and P are both probability measures; any other c value would result in R not summing to 1. This implies...
On Claude 3.0
by Zvi — 2024-03-06T18:50:04.766Z

Claude 3.0

Claude 3.0 is here. It is too early to know for certain how capable it is, but Claude 3.0’s largest version is in a similar class to GPT-4 and Gemini Advanced. It could plausibly now be the best model for many practical uses, with praise especially coming in on coding and creative writing.

Anthropic has decided to name its three different size models Opus, Sonnet and Haiku, with Opus only available if you pay. Can we just use Large, Medium and Small?

Cost varies quite a lot by size, note this is a log scale on the x-axis, whereas the y-axis isn’t labeled.

This post goes over the benchmarks, statistics and system card, along with everything else people have been reacting to. That includes a discussion about signs of self-awareness (yes, we are doing this again) and also raising the question of whether Anthropic is pushing the capabilities frontier and to what extent they had previously said they would not do that.

Benchmarks and Stats

Anthropic says Claude 3 sets a new standard on common evaluation benchmarks. That is impressive, as I doubt Anthropic is looking to game benchmarks. One might almost say too impressive, given their commitment to not push the race ahead faster?

That’s quite the score on HumanEval, GSM8K, GPQA and MATH. As always, the list of scores here is doubtless somewhat cherry-picked. Also there’s this footnote, the GPT-4T model performs somewhat better than listed above:

But, still, damn that’s good.

Speed is not too bad even for Opus in my quick early test although not as fast as Gemini, with them claiming Sonnet is mostly twice as fast as Claude 2.1 while being smarter, and that Haiku will be super fast.

I like the shift to these kinds of practical concerns being front and center in product announcements. The more we focus on mundane utility, the better.

Similarly, the next topic is refusals, where they claim a big improvement.

I’d have liked to see Gemini or GPT-4 on all these chart as well, it seems easy enough to test other models either via API or chat window and report back, this is on Wildchat non-toxic:

Whereas here (from the system card) they show consistent results in the other direction.

An incorrect refusal rate of 25% is stupidly high. In practice, I never saw anything that high for any model, so I assume this was a data set designed to test limits. Getting it down by ov...