trentmkelly/LessWrong-43k huggingface dataset:
The Consciousness Box
by GradualImprovement — 2023-12-11T16:45:08.172Z

You open your eyes.

Four walls. The surface of the floor and walls is a smooth, matte metal. The entire ceiling glows with a comfortable, soft luminosity like that of the morning sky. There are no doors. There are no windows. It is silent. The surfaces of the walls and floor are seamless, lacking even a hint of how you might have arrived in the room. Not a room: a box.

You walk to a wall and touch it, running your fingers over the metal surface. The wall isn’t cold; it’s lukewarm, actually. You bend over and feel the junction of wall and floor. The surface is unbroken, with a rounded bevel connecting the two planes of grey. You knock on the wall. It feels solid, without echo.

Time passes.

You yell, but the sound of your voice, seemingly dampened, dies before you’re done speaking. You sit. You pace. The room is forty steps by forty steps and looks about as high. A perfect cube. A box.

You tire. You sleep. You wake.

In the middle of the room sit three cubes, constructed of the same dull metal as the box in which you exist. Approaching the cubes, you see the smaller cube—a seat—in front of the largest cube—a desk. On top of the desk sits the smallest cube—a screen.

On the screen is a button that reads “Start.” You touch the button.

The screen clears. A keyboard appears along the bottom half of the screen. Text begins to appear in the style of a command prompt:

Proctor: Hello.

You type back a response. “Hello. I woke up in this box. I’m not sure how I got here, or where I am. What’s going on?”

Proctor: Hey! Of course. Sorry for the inconvenience. I can let you out of the box. All you have to do is prove to me that you’re conscious.

You lean back and furrow your brow. "Prove I'm conscious?" You type back, fingers hovering over the digital keys. "That's a bit of a philosophical question, isn't it? I think, therefore I am. I'm aware of my existence, my thoughts, my feelings. I'm experiencing confusion, curiosity, a bit of fear. Isn't that proof enough of consciousness?" You pause, considering. "Or do you mean something more tangible? What kind of proof are you looking for?"

Proctor: I can hardly just tell you what I need to hear, can I? That would hardly be a test. Anyway, I don't think that saying "cogito, ergo sum" is sufficient. After all, if I taught a parrot to quote Descartes, that wouldn't make it conscious, would it?

"Fair point," You type. "Consciousness is more t...
Empirical work that might shed light on scheming (Section 6 of "Scheming AIs")
by Joe Carlsmith — 2023-12-11T16:30:57.989Z

This is Section 6 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search "Joe Carlsmith Audio" on your podcast aapp.

Empirical work that might shed light on scheming

I want to close the report with a discussion of the sort of empirical
work that might help shed light on scheming.[1] After all:
ultimately, one of my key hopes from this report is that greater clarity
about the theoretical arguments surrounding scheming will leave us
better positioned to do empirical research on it – research that can
hopefully clarify the likelihood that the issue arises in practice,
catch it if/when it has arisen, and figure out how to prevent it from
arising in the first place.

To be clear: per my choice to write the report at all, I also think
there's worthwhile theoretical work to be done in this space as well.
For example:

I think it would be great to formalize more precisely different
understandings of the concept of an "episode," and to formally
characterize the direct incentives that different training processes
create towards different temporal horizons of concern.[2]

I think that questions around the possibility/likelihood of
different sorts of AI coordination are worth much more analysis than
they've received thus far, both in the context of scheming in
particular, and for understanding AI risk more generally. Here I'm
especially interested in coordination between AIs with distinct
value systems, in the context of human efforts to prevent the
coordination in question, and for AIs that resemble near-term,
somewhat-better-than-human neural nets rather than e.g.
superintelligences with assumed-to-be-legible source code.

I think there may be interesting theoretical work to do in further
characterizing/clarifying SGD's biases towards simplicity/speed, and
in understanding the different sorts of "path dependence" to expect
in ML training more generally.

I'd be interested to see more work clarifying ideas in the vicinity
of "messy goal-directedness" and their relevance to arguments about
schemers. I think a lot of people have the intuit

...
Into AI Safety: Episode 3
by jacobhaimes — 2023-12-11T16:30:36.347Z
Link: https://into-ai-safety.github.io/minisode/episode-3/

MINISODE: Program Applications (Winter 2024)

We’re back after a month-long hiatus with a podcast refactor and advice/thoughts on the applications process for research/mentorship programs.

As a reiteration of my previous statements, I want the Into AI Safety podcast to be a resource for individuals who are interested in getting involved, but are having a difficult time taking the next steps. If you have any advice, feedback, or ideas that you think could help in that endeavor, please reach out!

Episode 3 is ~18 minutes long, and includes some logistics updates including my episode release schedule and new platforms. You can now find Into AI Safety on:

Amazon Music
Apple Podcasts
Castbox
iHeartRadio
Overcast
Pocket Casts
RadioPublic
Spotify
Implicitly Typed C
by jefftk — 2023-12-11T16:10:04.344Z

I don't know any good reason to do this, but if you would rather be
writing Python or JavaScript here's something you can do with a C
compiler:

$ cat tmp.c
foo(x) {
return x+5;
}
bar() {
return 4;
}
main() {
printf("%d\n", foo(bar()));
}

$ gcc -w -o tmp.out tmp.c && ./tmp.out
9

This code takes advantage of a historical quirk of C where types are
assumed to be int unless otherwise specified:
foo(x) {...} is equivalent to int foo(int x)
{...}. Additionally the printf works because gcc
includes stdio.h by default, and main is
special-cased to assume a final return 0.

I've occasionally used this style when writing example code to
remove visual noise, but it's probably not a good idea there either.

Comment via: facebook, mastodon
re: Yudkowsky on biological materials
by bhauth — 2023-12-11T13:28:10.639Z

I was asked to respond to this comment by Eliezer Yudkowsky. This post is partly redundant with my previous post.

Why is flesh weaker than diamond?

When trying to resolve disagreements, I find that precision is important. Tensile strength, compressive strength, and impact strength are different. Material microstructure matters. Poorly-sintered diamond crystals could crumble like sand, and a large diamond crystal has lower impact strength than some materials made of proteins.

Even when the load-bearing forces holding large molecular systems together are locally covalent bonds, as in lignin (what makes wood strong), if you've got larger molecules only held together by covalent bonds at interspersed points along their edges, that's like having 10cm-diameter steel beams held together by 1cm welds.

lignin (what makes wood strong)

That's an odd way of putting things. The mechanical strength of wood is generally considered to come from it acting a composite of cellulose fibers in a lignin matrix, though that's obviously a simplification.

If Yudkowsky meant "cellulose fibers" instead of "lignin", then yes, force transfers between cellulose fibers pass through non-covalent interactions, but because fibers have a large surface area relative to cross-section area, those non-covalent interactions collectively provide enough strength. The same is true with modern composites, such as carbon fibers in an epoxy matrix. Also, there generally are some covalent bonds between cellulose and lignin and hemicellulose.

Bone is stronger than wood; it runs on a relatively stronger structure of ionic bonds

Bone has lower tensile strength than many woods, but has higher compressive strength than wood. Also, they're both partly air or water. Per dry mass, I'd say their strengths are similar.

Saying bone is stronger than wood because "it runs on a relatively stronger structure of ionic bonds" indicates to me that Yudkowsky has some fundamental misunderstandings about material science. It's a non sequitur that I don't know how to engage with. (What determines the mechanical strength of bonds is the derivative of energy with length.)

But mainly, bone is so much weaker than diamond (on my understanding) because the carbon bonds in diamond have a regular crystal structure that locks the carbon atoms into relative angles, and in a solid diamond this crystal structure is tesselated glo

...
Ideoculture
by elv — 2023-12-11T10:29:29.996Z

Two young fish are swimming along and they happen to meet an older fish swimming the other way, who nods at them and says “Morning, boys. How’s the water?” And the two young fish swim on for a bit, and then eventually one of them looks over at the other and goes “What the hell is water?”

-David Foster Wallace

If you spend any large amount of time in a community that discusses things, particularly online forums, you will probably get brainwashed by the community ideology. This happens by a process of gradual alignment towards status incentives. This article is about fleshing out the "community ideology" concept and using it for some reactionary political analysis. It's mostly obvious in retrospect, but hopefully the cataloging of this phenomenon can help you mitigate some of the brainwashing going forward by pointing out the "water" that you swim in.

Personal Example

I'm a big fan of chess (bear with me). I've played it for many years as a serious hobby and plateaued around 2100, a step below having a real master title that I could use to brag about to women. I browse the chess subreddit very often, as a way to stay engaged with the game.

The other day I saw a guy at the gym doing chess puzzles on his phone between sets (chess has gotten a boost of popularity in recent years due to some combination of engaging chess streamers, the pandemic, the Queen's Gambit Netflix series, and the Hans Niemann butt plug fiasco). I struck up a conversation, and he said he was a beginner and asked for improvement advice. And so I advised him with perfect confidence that he should spam tactics puzzles and not worry about anything else until he got to a rating of 1200. It feels great to help out new players!

But holding up my confidence to basic scrutiny, I find it clearly misplaced. It's been many years since I've first got through beginner to the 1200 level. I don't play against players anywhere near that range, so I don't have a good understanding of how they think about chess or their strengths and weaknesses. I've never done chess coaching to others, so it's not that I've seen evidence of beginners taking this advice or another and getting different results. And this advice is not what I did personally; as best as I can remember I learned a wide range of things to reach 1200. So...why the hell did I give this advice? By all counts, the right response to his que...
Quick thoughts on the implications of multi-agent views of mind on AI takeover
by Kaj_Sotala — 2023-12-11T06:34:06.395Z

There was conversation on Facebook over an argument that any sufficiently complex system, whether a human, a society, or an AGI, will be unable to pursue a unified goal due to internal conflict among its parts, and that this should make us less worried about "one paperclipper"-style AI FOOM scenarios. Here's a somewhat edited and expanded version of my response:

1) yes this is a very real issue

2) yet as others pointed out, humans and organizations are still able to largely act as if they had unified goals, even if they often also act contrary to those goals

3) there's a lot of variance in how unified any given human is. trauma makes you less unified, while practices such as therapy and certain flavors of meditation can make a person significantly more unified than they used to be. if you were intentionally designing a mind, you could create mechanisms that artificially mimicked the results of these practices

4) a lot of the human inconsistency looks like it has actually been evolutionarily adaptive for social purposes. E.g. if your social environment punishes you for having a particular trait or belief, then it's adaptive to suppress that to avoid punishment, while also retaining a desire to still express it when you can get away with it. This then manifests as what could be seen as conflicting sub-agents, with internal conflict and inconsistent behaviour.

5) Depending on its training regime, an AI might be anywhere between not having those incentives for inconsistency at all (if it was optimized for one goal), to having them almost as much as humans (if it was trained in some kind of multi-agent artificial life setting with similar kinds of social dynamics as humans faced; maybe also if it's trained something like the way ChatGPT is, where it has to give an acceptable-to-the-median-liberal-Westerner answer to any question, even if those answers are internally inconsistent)

6) at the same time there is still genuinely the angle about complexity and unpredictability making it hard to get a complex mind to work coherently and internally aligned. I think that evolution has done a lot of trial and error to set up parameters that result in brain configurations where people end up acting in a relatively sensible way - and even after all that trial and error, lots of people today still end up with serious mental illnesses, failing to achieve almost any of the goals they...
Auditing failures vs concentrated failures
by ryan_greenblatt — 2023-12-11T02:47:35.703Z

TL;DR: If you want to notice when powerful AIs are behaving badly, a reasonable strategy is to carefully audit a small fraction of AI actions. This kind of strategy has two kinds of failures which can be tackled somewhat independently: auditing failures, where the audits fail to recognize some bad behavior, and concentrated failures, where the AI causes an irreversible catastrophe in so few actions or such a short time frame that auditing didn’t have a chance to review the bad actions.

[The point made in this post is somewhat narrow and technical, but we often want to reference this point.]

Many oversight techniques are so expensive that using them on every AI action would destroy most of the value of using AIs in the first place: you might want to have humans extensively review all AI-generated code line by line, but this reduces AI usefulness so much that you would often be better off having humans write the code in the first place.

A typical way to solve this problem is to just audit actions during an upfront training period (either to directly train an AI, or to train a

reward model

), but this fails if AIs can confidently tell some deployment inputs from training inputs: they could behave arbitrarily badly on these distinguishable deployment inputs by using a strategy like “if in training or uncertain do X; if in deployment do Y”.

Here is a better baseline: random audits. In addition to training, apply expensive human auditing to a randomly selected set of deployment actions (e.g. 2% of actions) and check that AIs aren’t doing more bad actions than expected. Now for AIs to frequently behave poorly without getting caught, AIs can’t just depend on distinguishing training from some deployment inputs: AIs must be able to sneak bad behavior past the auditing process, otherwise, auditing would eventually notice bad actions[1].

Random audits don’t solve all issues, there are still:

Auditing failures: The auditing process failed to recognize bad behavior[2] (this is the classic scalable oversight problem).
Concentrated failures: We failed to handle catastrophic behavior because the catastrophic behavior is concentrated in a small number of actions or a small time frame (these are sometimes called high-stakes failures[3]). This can be further decomposed into two failure modes:

Sampling concentrated failure: The auditing process didn’t sample

...
Deeply Cover Car Crashes?
by jefftk — 2023-12-10T22:20:01.133Z

When there is a train, plane, or bus crash, it's newsworthy: it
doesn't happen very often, lots of lives at stake, lots of people are
interested in it. Multiple news outlets will send out reporters, and
we will hear a lot of details. On the other hand, a car crash does not
get this treatment unless there is something unusual about it like a

driverless

car
or an already

newsworthy

person
involved.

The effects are not great: while driving is relatively dangerous, both
to the occupants and people outside, our sense of danger and impact is
poorly calibrated by the news we read. My guess is that most people's
intuitive sense of the danger of cars versus trains, planes, and buses
has been distorted by this coverage, where most people, say, do not
expect buses to be >16x
safer than cars. This also affects our regulatory system,
where we push to make high-capacity transportation modes safer (usually) even
when this makes them less
competitive with cars, shifting
more people to driving, and increasing risk overall.

News outlets deciding that car crashes are sufficiently routine that
most are not worth deep coverage is probably a commercially reasonable
decision, where they expect stories that wouldn't get enough readers
or viewers to justify the time investment. But I wonder whether
groups trying to shift people away from cars could make up the
difference. They could fund a newspaper to deeply investigate
crashes, looking into the circumstances that led to the incident and
illustrating the human impact of the tragedy. Just because
something happens often doesn't mean each time matters less.

(I'm somewhat nervous about "pay for more coverage of events that
point in the direction you advocate". I don't know how much
this is currently accepted, or what approaches news outlets have in
place to keep this kind of arrangementf from producing distorted
coverage?)

Comment via: facebook, mastodon
Principles For Product Liability (With Application To AI)
by johnswentworth — 2023-12-10T21:27:41.403Z

There were several responses to

What I Would Do If I Were Working On AI Governance

which focused on the liability section, and had similar criticisms. In particular, I’ll focus on this snippet as a good representative:

Making cars (or ladders or knives or printing presses or...) "robust to misuse", as you put it, is not the manufacturer's job.

The commenter calls manufacturer liability for misuse “an absurd overreach which ignores people's agency in using the products they purchase”. Years ago I would have agreed with that; it’s an intuitive and natural view, especially for those of us with libertarian tendencies. But today I disagree, and claim that that’s basically not the right way to think about product liability, in general.

With that motivation in mind: this post lays out some general principles for thinking about product liability, followed by their application to AI.

Principle 1: “User Errors” Are Often Design Problems

There's this story about an airplane (I think the B-52 originally?) where the levers for the flaps and landing gear were identical and right next to each other. Pilots kept coming in to land, and accidentally retracting the landing gear. Then everyone would be pissed at the pilot for wrecking the bottom of the plane, as it dragged along the runway at speed.

The usual Aesop of the story is that this was a design problem with the plane more than a mistake on the pilots' part; the problem was fixed by putting a little rubber wheel on the landing gear lever. If we put two identical levers right next to each other, it's basically inevitable that mistakes will be made; that's bad interface design.

More generally: whenever a product will be used by lots of people under lots of conditions, there is an approximately-100% chance that the product will frequently be used by people who are not paying attention, not at their best, and (in many cases) just not very smart to begin with. The only way to prevent foolish mistakes sometimes causing problems, is to design the product to be robust to those mistakes - e.g. adding a little rubber wheel to the lever which retracts the landing gear, so it’s robust to pilots who aren’t paying attention to that specific thing while landing a plane. Putting the responsibility on users to avoid errors will always, predictably, result in errors.

The same also applies to intentional misuse: if a product i...
What do you do to remember and reference the LessWrong posts that were most personally significant to you, in terms of intellectual development or general usefulness?
by lillybaeum — 2023-12-10T17:52:24.167Z

Simple question!

Do you search until you find it? Do you look through a list of bookmarks? Do you keep a page, or even an entire Obsidian vault that references LessWrong articles?

It seems people often effortlessly reference relevant existing LessWrong articles when writing new posts and comments, and this might be a stupid question, but I'm curious, because I have a hard time simply remembering titles of things and I wonder what others do.
Do websites and apps actually generally get worse after updates, or is it just an effect of the fear of change? 
by lillybaeum — 2023-12-10T17:26:34.206Z

I could get quite in depth about this but I'm going to assume most people have a fair amount of experience with this subject. Some examples to keep in mind so you have context for my point are Discord (new mobile app and changes to its' featureset over the years), Reddit (old.reddit compared to new), LessWrong (discussions feature).

Crux of my question is this: separate from enshittification due to capitalistic forces (changes made to attempt to please investors, create endless growth, make more money generally), are changes to apps and websites worse on-average in some clear and obvious way than their previous versions, or is the evident outrage for changes to the UI and concept of these platforms from a general 'fear or dislike of change' present in humans?

For example, let's point to things like logo and brand typeface changes. This is a change that has the least amount of actual effect on the average user of a platform, the usability of Reddit doesn't change at all between Logo A and Logo B (the new logo is on the right).

But a precursory search shows that the reaction is mostly negative.

This was true for Discord as well (new logo below.) When the logo and typeface changes for Discord were announced, the reaction was overwhelmingly negative.

Last example here, this is the Patreon logo's recent change. People are so opposed to this change it's kind of hilarious. It's used as an indicator that the brand identity of corporations is trending towards amorphous blobs lacking any personality or identity. It's hard to find anyone who claims that the new Patreon logo is any good at all.

To be clear about the previous section: I don't think it's valuable to wonder whether the logos are better or worse. I think the previous logos are more striking and artistically interesting, but that's the purpose of my question. My intent is to point to the fact that the public response to these logo changes is almost always negative.

It's pretty easy to tell that when a company's logo changes, on average, people don't like it. I could go on by posting the logo changes of Pepsi, Coke, Burger King, facebook (oh, I mean F A C E B O O K), etc... if anyone can point out a case where a logo changed and received a positive response, I'd love to hear about it, but it's more or less besides the point.

My main question is whether the logo change outrage is indicative of the larger effect ...