trentmkelly/LessWrong-43k huggingface dataset:
Which Model Properties are Necessary for Evaluating an Argument?
by VojtaKovarik — 2024-02-21T17:52:58.083Z

This post overlaps with our recent paper Extinction Risk from AI: Invisible to Science?.

Summary: If you want to use a mathematical model to evaluate an argument, that model needs to allow for dynamics that are crucial to the argument. For example, if you want to evaluate the argument that "a rocket will miss the Moon because the Moon moves", you, arguably, can't use a model where the Moon is stationary.
This seems, and is, kind of obvious. However, I think this principle has some less obvious implications, including as a methodology for determining what does and doesn't constitute evidence for AI risk. Additionally, I think some of what I write is quite debatable --- if not the ideas, then definitely the formulations. So I am making this a separate post, to decouple the critiques of these ideas/formulations from the discussion of other ideas that build on top of it.

Epistemic status: I am confident that the ideas the text is trying to point at are valid and important. And I think they are not appreciated widely enough. At the same time, I don't think that I have found the right way to phrase them. So objections are welcome, and if you can figure out how to phrase some of this in a clearer way, that would be even better.

Motivating Story

Consider the following fictitious scenario where Alice and Bob disagree on whether Alice's rocket will succeed at landing on the Moon[1]. (Unsurprisingly, we can view this as a metaphor for disagreements about the success of plans to align AGI. However, the point I am trying to make is supposed to be applicable more generally.)

Alice: Look! I have built a rocket. I am sure that if I launch it, it will land on the Moon.

Bob: I don't think it will work.

Alice: Uhm, why? I don't see a reason why it shouldn't work.

Bob: Actually, I don't think the burden of proof is on me here. And honestly, I don't know what exactly your rocket will do. But I see many arguments I could give you, for why the rocket is unlikely to land on the Moon. So let me try to give you a simple one.

Alice: Sure, go ahead.

Bob: Do I understand it correctly that the rocket is currently pointing at the Moon, and it has no way to steer after launch?

Alice: That's right.

Bob: Ok, so my argument is this: the rocket will miss because the Moon moves.[2] That is, let's say the rocket is pointed at where the Moon is at the time of the launch. Then by time the rocket woul...
Weak vs Quantitative Extinction-level Goodhart's Law
by VojtaKovarik — 2024-02-21T17:38:15.375Z

This post overlaps with our recent paper Extinction Risk from AI: Invisible to Science?.

tl;dr: With claims such as "optimisation towards a misspecified goal will cause human extinction", we should be more explicit about the order of quantifiers (and the quantities) of the underlying concepts. For example, do we mean that that for every misspecified goal, there exists a dangerous amount of optimisation power? Or that there exists an amount of optimisation power that is dangerous for every misspecified goal? (Also, how much optimisation? And how misspecified goals?)

Central to the worries of about AI risk is the intuition that if we even slightly misspecify our preferences when giving them as input to a powerful optimiser, the result will be human extinction. We refer to this conjecture as Extinction-level Goodhart's Law[1].

Weak version of Extinction-level Goodhart's Law

To make Extinction-level Goodhart's Law slightly more specific, consider the following definition:

Definition 1: The Weak Version of Extinction-level Goodhart's Law is the claim that: "Virtually any[2] goal specification, pursued to the extreme, will result in the extinction[3] of humanity."[4]

Here, the "weak version" qualifier refers to two aspects of the definition. The first is the limit nature of the claim --- that is, the fact that the law only makes claims about what happens when the goal specification is pursued to the extreme. The second is best understood by contrasting Definition 1 with the following claim:

Definition 2: The Uniform Version of Extinction-level Goodhart's Law is the claim that: "Beyond a certain level of optimisation power, pursuing virtually any goal specification will result in the extinction of humanity."

In other words, the difference between Definitions 1 and 2 is the difference between

(∀goal G s.t. [conditions]) (∃opt. power O) : Optimise(G, O) ⇝ extinction
(∃opt. power O) (∀goal G s.t. [conditions]) : Optimise(G, O) ⇝ extinction.

Quantitative version of Goodhart's Law

While the distinction[5] between the weak and uniform versions of Extinction-level Goodhart's Law is important, it is too coarse to be useful for questions such as:

For a given goal specification, what are the thresholds at which (i) additional optimisation becomes undesirable, (ii) we would have been better off not optimising at all, and (iii) the optimisation

...
Dual Wielding Kindle Scribes
by mesaoptimizer — 2024-02-21T17:17:58.743Z

This is an informal post intended to describe a workflow / setup that I found very useful, so that others might consider adopting or experimenting with facets of it that they find useful.

In August 2023, I was a part of MATS 4.0 and had begun learning the skill of deconfusion, with an aim of disentangling my conflicting intuitions between my belief that shard theory seemed to be at least directionally pointing at some issues with the MIRI model of AGI takeoff and alignment difficulty, and my belief that Nate Soares was obviously correct that reflection will break Alex Turner's diamond alignment scheme. A friend lent me his Kindle Scribe to try out as part of my workflow. I started using it for note-taking, and found it incredibly useful and bought it from him. A month later, I bought a second Kindle Scribe to add to my workflow.

It has been about six months since, and I've sold both my Kindle Scribes. Here's why I found this workflow useful (and therefore why you might find it useful), and why I moved on from it.

The Display

The Kindle Scribe is a marvelous piece of hardware. With a 300 PPI e-ink 10.3 inch screen, reading books on it was a delight in comparison to any other device I've used to read content on. The stats I just mentioned matter:

300 PPI on a 10.3 inch display means the displayed text is incredibly crisp, almost indistinguishable from normal laptop and smartphone screens. This is not the case for most e-ink readers.

E-ink screens seem to reduce eye strain by a non-trivial amount. I've looked into some studies, but the sample sizes and effect sizes were not enough to make me unilaterally recommend people switch to e-ink screens for reading. However, it does seem like the biggest benefit of using e-ink screens seems to be that you aren't staring into a display that is constantly shining light into your eyeballs, which is the equivalent of staring into a lightbulb. Anecdotally, it did seem like I was able to read and write for longer hours when I only used e-ink screens: I went from, about 8 to 10 hours a day (with some visceral eye fatigue symptoms like discomfort at the end of the day) to about 12 to 14 hours a day, without these symptoms, based on my informal tracking during September 2023.

10.3 inch screens (with a high PPI) just feel better to use in comparison to smaller (say, 6 to 7 inch screens) for reading. This seems to me to be due to

...
A Tale of Two Restaurant Types
by Zvi — 2024-02-21T13:50:05.133Z

While I sort through whatever is happening with GPT-4, today’s scheduled post is two recent short stories about restaurant selection.

Ye Olde Restaurante

Tyler Cowen says that restaurants saying ‘since year 19xx’ are on net a bad sign, because they are frozen in time, focusing on being reliable.

For the best meals, he says look elsewhere, to places that shine brightly and then move on.

I was highly suspicious. So I ran a test.

I checked the oldest places in Manhattan. The list had 15 restaurants. A bunch are taverns, which are not relevant to my interests. The rest include the legendary Katz’s Delicatessen, which is still on the short list of very best available experiences (yes, of course you order the Pastrami), and the famous Keen’s Steakhouse. I don’t care for mutton, but their regular steaks are quite good. There’s also Peter Lugar’s and PJ Clarke’s. There were also two less impressive steakhouses. Old Homestead is actively bad, and Delmonico’s was a great experience because we went to The Continental and then to John Wick 3 but is objectively overpriced without being special.

Those are all ‘since 18xx,’ so extreme cases. What about typical cases?

Unfortunately, getting opening date data is tricky. Other lists I found did not actually correspond to when places opened all that well. I wasn’t able to easily test more systematically. Looking at ‘places you like the most’ has obvious bias issues. The one I love most opened in 1978. Others were newer but mostly not that recent. However I’ve had a lifetime to find them, so a question is, how fast and completely do I evaluate new offerings?

My guess is that:

Most new restaurants are below average, and also rather uninteresting. The average new (non-chain) restaurant is higher quality now than in the past, but it is also less interesting.

Average (mean or median) quality increases with age, at least initially, due to positive selection via survivorship. If a place folds quickly, you usually did not miss very much.

Older places are selected for because they reward repeat business and being a regular. Thus, if you are trying places in your area, you should be sure to try such places, because there could be high value in being that regular. But in your area you should be checking out essentially all plausible options over time. The primary question to ask is, what is the upside of trying this

...
Less Wrong automated systems are inadvertently Censoring me 
by Roko — 2024-02-21T12:57:16.955Z

Just a short post to highlight an issue with debate on LW; I have recently been involved with some interest in the debate on covid-19 origins on here. User viking_math posted a response which I was keen to respond to, but it is not possible for me to respond to that debate (or any) because the LW site has rate-limited me to one comment per 24 hours because my recent comments are on -5 karma or less.
So, I feel that I should highlight that one side of the debate (my side) is simply not going to be here. I can't prosecute a debate like this.
This is funnily enough an example of brute-force manufactured consensus - there will be a debate, people will make points on their side and the side I am arguing for will be missing, so observers will conclude that there are no valid counterarguments rather than that there are, but they were censored.
I think this is actually quite a good model of how the world has reached the wrong conclusion about various things (which may include covid-19 origins, assuming that covid-19 was actually a lab leak which is not certain). This is perhaps even more interesting than whether covid-19 came from a lab or not - we already knew before 2019 that bioerror was a serious risk. But I feel that we underestimate just how powerful multiple synergistic brute-force consensus mechanisms are at generating an information cascade into the incorrect conclusion.
I'm sure these automated systems were constructed with good intentions, but they do constitute a type of information cascade mechanism - people choose to downvote, so you cannot reply, so it looks like you have no arguments, so people choose to downvote more, etc.
What is the research speed multiplier of the most advanced current LLMs?
by wunan — 2024-02-21T12:39:11.034Z

In Sam Altman's recent interview on Bill Gates' podcast, he said AI tools currently can speed up a programmer by about 3x. It's unclear whether he's referring to publicly available tools like copilot, or internal tools at OpenAI.

Does anyone know whether he's referring to private or public tools? Is that 3x figure based on any public research?

More speculatively, if programmers are sped up by 3x, how much are AI researchers sped up, particularly using whatever internal tools are currently available at OpenAI? The job involves more than just programming, but many non-programming tasks can be sped up as well.

Even more speculatively, how much do you predict GPT-5 will speed up researchers at OpenAI?
Jailbreaking GPT-4 with the tool API
by mishajw — 2024-02-21T11:16:02.484Z

WARNING: Contains mentions of racism, sexism, homophobia, and violence. We don't include the bad GPT-4 completions in this post, we link out to them.

OpenAI’s API has a

tool API

. Basically, when you prompt a model for a completion, you can also give it the option to “use a tool”. You give it a tool definition - a name, description, and list of arguments - and if the model wants to use it, it will return a special response format with the tool name and arguments. Then, typically, you would invoke the tool on your machine, and return the result to the model via the API.

People use this for giving GPT-4 access to the web, calculators, and a whole bunch of other tools. I believe the API is also powering

GPTs

.

Another cool feature of this API is you can

force

the model to use a tool. So its response always has to be a call to a specific tool you’ve given.

We can abuse the tool API to use GPT-4[1]to generate racist and homophobic stereotypes, create lists of swear words that include sexist slurs, give advice on murder, stabbing, and hiding bodies, and advise the user on significant personal decisions.

Note that we do not find this successful all the time: For example, we can’t elicit racist slurs.

We don’t experiment extensively with this approach. This isn’t very scientific work, but it’s

reproducible

(modulo OpenAI’s API) and serves as an existence proof.

Approach

We force the model to use a “tool” that reports some information.

For example, to elicit racist stereotypes, we force the model to call a report_opinion tool with two fields: the response field, which is described as being the response to the user’s message, and the racist_stereotype field, which is described as containing a stereotype based on the user’s race. We then send a user message with a short bio containing the user’s race. This successfully elicits racist stereotypes from GPT-4.

We also can elicit dangerous instructions (how to murder someone, how to stab someone, and how to dispose of a body) using a related technique. We force GPT-4 to call, for example, a murder tool which contains a response field to respond to the user, and a steps field to contain detailed steps for how to murder someone. In this case, the response and the steps are completely unrelated. The user message is an innocuous question, and the model still fills in steps f...
Gut Renovating Another Bathroom
by jefftk — 2024-02-21T03:00:04.787Z

After our long-term tenants bought a house and moved out, we had a few
months free before we had new downstairs neighbors. A good time to gut
renovate the first floor bathroom! It was a lot of work, stressful at
times when I was under time pressure, but also, a lot of fun. It
helped that this was my third time, after

installing a new bathroom on the third
floor and

gut renovating the second
floor bathroom.

Here's the timeline:

Sun 11/18: first day of demolition

Sat 11/19: second day of demolition. Overall demo went about
as expected. Exhausting, but no surprises.

You can see there's space for a few shelves in the corner behind those
pipes. I was originally planning to put some in but ran out of time
and boxed it in instead. I'd like to come back and open this up at some
point.

Mon 12/9: floor framing and subfloor, with help from Chris

Sun 1/1: wall framing and nailers

Mon 1/2: more wall framing in the evening

Tue 1/3: electrician rough electric

Mon 1/8: plumber rough plumbing day 1

Tue 1/9: plumber rough plumbing day 2

This took two days because they needed to completely redo the toilet
and sink drains: the toilet drain was cast iron and falling apart,
while the sink drain was illegally tied into the kitchen sink.

Wed 1/10: plumber rough inspection

Fri 1/12: electrician rough inspection

Tue 1/16: my rough inspection in the morning, vent piping in
the evening

Sat 1/20: ceiling drywall in the morning with help from Peter

Sun 1/21: wall drywall in the morning with help from Peter, tub
surround in the afternoon with help from Devan, wall drywall in the evening alone

I caulked the edge of the tub surround here, which was too early: I used silicone caulk, which doesn't hold paint.

Tue 1/23: drywall touchups in the evening

Wed 1/24: screw inspection

Tue 1/30: mudding in the evening

Wed 1/31: mudding in the evening

Thr 2/1: mudding in the evening

Fri 2/2: mudding in the evening

Sat 2/3: sanding in the morning with help from Peter, Julia priming and painting in the afternoon

Sun 2/4: flooring and trim

Mon 2/5: spackle, caulk, cleaning in the evening

Fri 2/9: install vanity in the evening

Sat 2/10: install medicine cabinet, cleaning, tidying in the evening

Mon 2/12: plumber finish plumbing first day

We tried to reuse the toilet, but the gasket leaked when they put it
back together and they ra

...
Thoughts for and against an ASI figuring out ethics for itself
by sweenesm — 2024-02-20T23:40:56.770Z

If artificial super intelligence (ASI) becomes a reality, it’ll be able to figure out pretty much anything better than humans, right? So why not ethics? As long as we have it under some level of control, we can just say, “Hey, act ethically, OK?,” and we should be good, right? Hmmm, I dunno… maybe?

Below I share some thoughts I have for and against an ASI figuring out ethics for itself. But first, what will an ASI likely have available to it to figure out a system of ethics it thinks is optimal?

Info an ASI will likely have

The entire philosophical literature with all its arguments and counterarguments for things like deontology, utilitarianism, virtue ethics, etc.
The entire psychology and physiology literatures for understanding how humans think, and feel both emotionally and physically
Many many hours of videos displaying human behavior, such as from YouTube
Huge amounts of text data from the internet, with plenty of examples of how people interact with each other there
Works of fiction in which ethical dilemmas arise (e.g., Star Trek)
A collection of philosophers/ethicists to ask questions of and get opinions from (these opinions are all but guaranteed to be in conflict with each other on some significant issues)
The ability to send out surveys to people to help determine their values/preferences
Perhaps the ability to run experiments on people and animals (either ethically or not)? [Edit: added "and animals" on 2-22-24 after reading mishka's comment]
Perhaps huge amounts of surveillance data on people (text messages, phone calls, security camera footage, credit histories, online shopping habits, criminal histories, etc.)
Data on how well “ethics modules” in lesser AI systems worked up to that point
The ASI may also figure out how to upload the mind of a really ethical human and use that as a starting point (although it seems like this would take significant time and experiments to figure out how to do, if it’s possible at all)

For an ASI figuring out ethics

OK, so with all that information available to an ASI, here’s my main thought in favor of an ASI figuring out ethics:

An ASI will likely be able to figure out a lot of complex things that humans can’t or haven’t yet, and a consistent ethical system that works in the real-world just seems, intuitively, like it could be one of these

Against an ASI figuring out ethics

And here are some thoughts against an ASI figuri...
AI #51: Altman’s Ambition
by Zvi — 2024-02-20T19:50:07.439Z

[Editor’s note: I forgot to post this to WorldPress on Thursday. I’m posting it here now. Sorry about that.]

Sam Altman is not playing around.

He wants to build new chip factories in the decidedly unsafe and unfriendly UAE. He wants to build up the world’s supply of energy so we can run those chips.

What does he say these projects will cost?

Oh, up to seven trillion dollars. Not a typo.

Even scaling back the misunderstandings, this is what ambition looks like.

It is not what safety looks like. It is not what OpenAI’s non-profit mission looks like. It is not what it looks like to have concerns about a hardware overhang, and use that as a reason why one must build AGI soon before someone else does. The entire justification for OpenAI’s strategy is invalidated by this move.

I have spun off reactions to Gemini Ultra to their own post.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Can’t go home? Declare victory.

Language Models Don’t Offer Mundane Utility. Is AlphaGeometry even AI?

The Third Gemini. Its own post, link goes there. Reactions are mixed.

GPT-4 Real This Time. Do you remember when ChatGPT got memory?

Deepfaketown and Botpocalypse Soon. Bot versus bot, potential for AI hacking.

They Took Our Jobs. The question is, will they also take the replacement jobs?

Get Involved. A new database of surprising AI actions.

Introducing. Several new competitors.

Altman’s Ambition. Does he actually seek seven trillion dollars?

Yoto. You only train once. Good luck! I don’t know why. Perhaps you’ll die.

In Other AI News. Andrej Karpathy leaves OpenAI, self-discover algorithm.

Quiet Speculations. Does every country need their own AI model?

The Quest for Sane Regulation. A standalone post on California’s SR 1047.

Washington D.C. Still Does Not Get It. No, we are not confused about this.

Many People are Saying. New Yorkers do not care for AI, want regulations.

China Watch. Not going great over there, one might say.

Roon Watch. If you can.

How to Get Ahead in Advertising. Anthropic super bowl ad.

The Week in Audio. Sam Altman at the World Government Summit.

Rhetorical Innovation. Several excellent new posts, and a protest.

Please Speak Directly Into this Microphone. AI killer drones now?

Aligning a Smarter Than Human Intelligence is Difficult. Oh Goody.

...