trentmkelly/LessWrong-43k huggingface dataset:
Kolmogorov Complexity Lays Bare the Soul
by jakej — 2023-12-01T18:29:57.379Z

Kolmogorov complexity is a measure of the algorithmic complexity of a particular piece of information, and it is found by identifying the shortest possible representation of that information. For example, if I have a string of characters AAAAA, I can create a compressed representation 5A. The Kolmogorov complexity of AAAAA then is at most 2 because I can represent that same information with only two characters using the shorter version 5A. We can say that 5A is the Kolmogorov string that shows that the Kolmogorov complexity of AAAAA is 2.

Kolmogorov complexity can also give us a sense of the relative complexity of different pieces of information. AAABB can only be compressed down to 3A2B, for a Kolmogorov complexity of 4, making it twice as complex as AAAAA.

Fractals like the one below look complex, and indeed, storing an image of a fractal can take up a lot of space. However, fractals are constructed by following simple rules, and a computer program to execute those instructions can be many orders of magnitude smaller than storing the image itself. This means that the Kolmogorov complexity of fractals can actually be quite low, despite appearances.

It occurred to me to ask - do I have a Kolmogorov complexity? My body and my mind both contain only a finite amount of information, so both could be described in exhaustive detail with only a finite amount of text. A brute-force description of every fiber of my being does not strike me as especially interesting. If I were able to discover my Kolmogorov string though, that does strike me as having some interesting properties.

Such a string would represent my purest and most distilled essence. It would contain every single detail of my existence - since even a single missing bit would disqualify it from being my Kolmogorov string by definition. It is also completely unique, being the shortest possible string. It is so lean that not a single redundant detail remains - otherwise, it would not be the shortest string.

So here we have a total representation of my entire being from which not a single iota can be added or removed; an utterly perfect representation.

One could examine this string and ask any question about my life and get the correct answer in exact detail. The details contained within it would be so exquisite that, even if I were to be utterly destroyed it would be enough to construct an exact replica of me.

One’...
Thoughts on “AI is easy to control” by Pope & Belrose
by Steven Byrnes — 2023-12-01T17:30:52.720Z

Quintin Pope & Nora Belrose have a new

“AI Optimists” website

, along with a new essay “

AI is easy to control

”, arguing that the risk of human extinction due to future AI (“AI x-risk”[1]) is a mere 1% (“a tail risk worth considering, but not the dominant source of risk in the world”). (I’m much more pessimistic.) It makes lots of interesting arguments, and I’m happy that the authors are engaging in substantive and productive discourse, unlike the ad hominem vibes-based drivel which is growing increasingly common on both sides of the AI x-risk issue in recent months.

This is not a comprehensive rebuttal or anything, but rather picking up on a few threads that seem important for where we disagree, or where I have something I want to say.

(Note: Nora has a reply here.)

Summary / table-of-contents:

Note:

I think Sections 1 & 4 are the main reasons that I’m much more pessimistic about AI x-risk than Pope & Belrose, whereas Sections 2 & 3 are more nitpicky.

Section 1 argues that even if controllable AI has an “easy” technical solution, there are still good reasons to be concerned about AI takeover, because of things like competition and coordination issues, and in fact I would still be overall pessimistic about our prospects.
Section 2 talks about the terms “black box” versus “white box”.
Section 3 talks about what if anything we learn from “human alignment”, including some background on how I think about human innate drives.
Section 4 argues that pretty much the whole essay would need to be thrown out if future AI is trained in a substantially different way from current LLMs. If this strikes you as a bizarre unthinkable hypothetical, yes I am here to tell you that other types of AI do actually exist, and I specifically discuss the example of “brain-like AGI” (a version of actor-critic model-based RL), spelling out a bunch of areas where the essay makes claims that wouldn’t apply to that type of AI, and more generally how brain-like AGI would differ from LLMs in safety-relevant ways.

1. Even if controllable AI has an “easy” technical solution, I’d still be pessimistic about AI takeover

Most of Pope & Belrose’s essay is on the narrow question of whether the AI control problem has an easy technical solution. That’s great! I’m strongly in favor of arguing about narrow questions. And after this section I’ll be talking about that narrow qu...
Why Did NEPA Peak in 2016?
by Maxwell Tabarrok — 2023-12-01T16:18:35.435Z
Link: https://maximumprogress.substack.com/p/why-did-nepa-peak-in-2016

The National Environmental Policy Act (NEPA) is a 1970 law which requires all federal agencies to “

prepare detailed statements assessing the environmental impact of and alternatives to major federal actions significantly affecting the environment

.” The exact definitions of ‘detailed,’ ‘major federal actions,’ and ‘significantly affecting the environment’ have been hashed out over thousands of court cases where usually an environmental protection group like the Sierra Club will sue a federal agency for not being careful enough in their assessment.

Bureaucratic and judicial rulemaking have clarified that NEPA applies to any activity or decision that a federal agency spends money on or issues a permit for. The ratcheting precedent and litigation pressure have made these environmental reports costly and time consuming. The top level of NEPA compliance, environmental impact statements, routinely take more than 5 years to complete.

NEPA has a lot of enemies and it deserves them. Whenever there is a

gobsmacking budget

or

eternal timeline

for a construction project that

would have been easy 100 years ago

, there is a

multi-thousand page NEPA report

which shoulders much of the blame.

It can often feel like NEPA is impossible to change. NEPA has chewed through the political capital of dozens of reform movements over the past several decades without much trouble. This graph from the National Association of Environmental Professionals shows that as each year passes from 2000-2017, environmental impact statements take a month longer to complete. The reform efforts during the Bush and Obama administrations don’t seem to make a dent. In 2017 the average EIS took about 1800 days or 5 years to complete, with many EISs taking over 10 years or never finishing at all.

A Light at the End of the Tunnel

The last time the National Association of Environmental Professionals posted this graph was in 2018. They still post tables of data, but they’re spread across several PDFs. I collected that data to check up on this graph and update it to 2022.

Since 2016 there has been six years of significant decreases in the time it takes to complete an environmental impact statement. They haven’t been this quick since 2011!

So after decades of getting worse and worse, NEPA may have finally turned a corner. But what explains this change?

The FAST Act of 2015

Buried deep in the massive omnibus b...
Worlds where I wouldn't worry about AI risk
by adekcz — 2023-12-01T16:06:54.199Z

I'm exploring the possibility of changing my belief that 'I should act on risks from powerful AI'. I haven't heard good arguments or reasons for not worrying about advanced AI, but it all might be just bias and stubbornness on my part. This is mostly an exercise in epistemics.

Given the premise "I should be doing something about risks from powerful AI", the question is: What observations of the world would cause me to think that much less. What would have to happen?

Additional assumptions:

The likelihood of AI doom within the next 50 years is 5 to 20 percent.

by doom, I mean human extinction or AI-controlled dystopia

Almost nobody works on it. Recently, a lot more people talk about it though.
If AI kills us all, it will be something like a paper clip maximizer rather than something possibly valuable like trans-human super consciousness.
This is the most important problem.

The following categories are roughly in order of plausibility/sanity. No ordering within categories.

EA 101

If those happen, the EA part of me might deprioritize AI safety. The following points are mostly related to Importance x Tractability x Neglectedness framework.

Very good arguments that P(Doom|even if we try really hard) ≈ 1: well, yeah. If we are doomed for sure, why worry at all? Just enjoy things while they last. Or actually not. If timelines are short (a few decades) and we are still EAs then the most we can do is probably by helping existing people and animals to have better lives.
Proof that P(Doom) ≈ 0
No longer neglected

I can imagine a few scenarios where AI alignment is a finite problem. We just need to throw enough manpower on it and then it will get solved. Yay, we are good for eternity.
Unfortunately, not much collective human effort has gone into alignment so far. For roughly ten years few hundred people have been working on it at most. So it is a few thousand person-years. Feels like we haven't even tried to solve it.
More likely than the finite nature of the problem we have something resembling a rocket equation. The more capability research we do, the more alignment research we need. So we would need larger |alignment researchers||capability researchers|. At some point, people smarter than me would say AI alignment is no longer neglected.

Some other cause area suddenly becoming more pressing e.g. being on the brink of development of self-replicat

...
How useful for alignment-relevant work are AIs with short-term goals? (Section 2.2.4.3 of "Scheming AIs")
by Joe Carlsmith — 2023-12-01T14:51:04.624Z

This is Section 2.2.4.3 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search "Joe Carlsmith Audio" on your podcast app.

How much useful, alignment-relevant cognitive work can be done using AIs with short-term goals?

So overall, I think that training our models to pursue long-term
goals – whether via long episodes, or via short episodes aimed at
inducing long-term optimization – makes the sort of beyond-episode goals
that motivate scheming more likely to arise. So this raises the
question: do we need to train our models to pursue long-term goals?

Plausibly, there will be strong general incentives to do this. That is:
people want optimization power specifically applied to long-term goals
like "my company being as profitable as possible in a year." So,
plausibly, they'll try to train AIs that optimize in this way. (Though
note that this isn't the same as saying that there are strong
incentives to create AIs that optimize the state of the galaxies in the
year five trillion.)

Indeed, there's a case to be made that even our alignment work, today,
is specifically pushing towards the creation of models with
long-term – and indeed, beyond-episode – goals. Thus, for example, when
a lab trains a model to be "harmless," then even though it is plausibly
using fairly "short-episode" training (e.g., RLHF on user interactions),
it intends a form of "harmlessness" that extends quite far into the
future, rather than cutting off the horizon of its concern after e.g. an
interaction with the user is complete. That is: if a user asks for help
building a bomb, the lab wants the model to refuse, even if the bomb
in question won't be set off for a decade.[1] And this example is
emblematic of a broader dynamic: namely, that even when we aren't
actively optimizing for a specific long-term outcome (e.g., "my company
makes a lot of money by next year"), we often have in mind a wide
variety of long-term outcomes that we want to avoid (e.g., "the
drinking water in a century is not poisoned"), and which it wouldn't be
acceptable to cause in the course of accom...
Reality is whatever you can get away with.
by sometimesperson — 2023-12-01T07:50:00.382Z

I register my luggage, and stick a paper label to it. There are many kiosks for placing luggage in the cargo system. One has a long line. One has a single family. The rest are empty. The workers at those sections are on their phones.

I walk up to one with my bag, and lightly clack it against the ground. The worker eyes me.

"You're only supposed to come when someone calls you."

"..."

"I didn't call."

"..."

I consider asking what she actually wants me to do, what the actual rules of the kiosks are, if she was on her break, why there were so many empty kiosks. Instead, I place my luggage on the scale.

She asks me for my ID. I give it to her. She scans it, and takes my bag. I thank her and leave.

I go to buy airport food. I go somewhere with bagels. While in line, I recall that people put sugar in bagels, and walk somewhere else. I go to a bar that serves drinks and tex-mex. Directly from the cashier, I order a cocktail, a hot dog, and a taco. On the menu, to the right of the word "dog," is the number 13. She asks me for my ID. I show her it. She inspects it, and accepts it.

"That'll be forty dollars." ($40).

"What? What does each individual item cost?"

She rotates her computer display towards me. I look at it.

Bloody Mary (eight dollars ($8))
Fish taco (eleven dollars ($11))
LA Street Dog (fourteen dollars ($14))
Service charge (seven dollars ($7))

I consider what to remove from my order.

"I'm going to go somewhere else. Goodbye."

The cashier shakes her head at me. Another person walks up to the cash register. Before, I was the only one at the bar.

I feel failure because I wasted someone's time.

Later, I buy a large sandwich for sixteen dollars ($16).

If you want the truth, pay attention in an airport.
Reinforcement Learning using Layered Morphology (RLLM)
by MiguelDev — 2023-12-01T05:18:58.162Z

TL;DR

This post introduces Reinforcement Learning using Layered Morphology (RLLM), a variant of reinforcement learning that focuses on instilling human values in LLMs by teaching complex patterns using specific datasets, offering a potential improvement over existing methods like Reinforcement Learning from Human Feedback (RLHF) which can introduce biases.

RLLM uses a series of related learning morphologies to progressively improve an LLM's understanding of complex instructions. Despite showing promise, the method faces limitations like weird text endings and restricted responses, which the author believes can be resolved with better resources and further research. The post concludes by highlighting the potential of RLLM and similar approaches in solving the alignment problem.

If an AGI singularity is happening in three years, what should we focus on alignment research?

I'm convinced that AGI is coming in three years. I'm not alone in this view, some recent interviews[1] have described something similar. Why? The short answer is that our collective economic capabilities are driving a significant chunk of humanity's brightest minds to create AGI.

Yes, the situation seems insane to wrap our minds around but thinking about the AGI scenario has led me to two interesting questions:

Can we solve the alignment problem in two years?
And if we do find a viable solution, can society coordinate effectively in six months to a year to execute it?

If we truly have only three years before AGI becomes a reality, our goal must be to develop an alignment solution focused on robustness and practicality.

Solving for robustness in two years?

The challenge lies in ensuring that the values we desire for humanity's future are both instilled in the AI (or its training process) and well preserved. If we only have two years, the answer lies somewhere in a solution that can control an AI's entire network or an alignment method that can scale. In order to solve the robustness issue, we need to understand how an AI system learns and acts in its world after learning about our world.
(Following that line of thought: I do not see mechanistic interpretability as the choice to explore here, as two years is too little of a time.)

Coordinating a practical solution within a year?

If we do discover "how we can teach the AI to prioritize our values robustly", the challenge of ...
Is OpenAI losing money on each request?
by thenoviceoof — 2023-12-01T03:27:23.929Z

While working on another post, I decided to follow up some details by doing some naive modeling of OpenAI’s LLM API revenue stream. The naive approach seems inadequate, because it implies OpenAI requires many years to break even just on the cost of GPUs.

OpenAI charges the following rates (from the OpenAI pricing page):

GPT-3.5 Turbo: input $0.001/1k tokens, output $0.002/1k tokens.
GPT-4 (non-Turbo): input $0.03/1k tokens, output $0.06/1k tokens.

How quickly do the GPTs generate tokens? Data pulled from some random people doing testing on Reddit, of all places in the local LLaMA subreddit. The post is 4 months old, so they were testing 3.5 Turbo and 4 non-Turbo (4 Turbo launched earlier this month).

GPT-3.5 Turbo: ~100 tokens/s
GPT-4 (non-Turbo): ~12-13 tokens/s
This is the weakest part of the analysis, it’s just some people doing tests with a stopwatch. If you have a better source please let me know.

With this data, we can calculate revenue for a single model running at 100% utilization.

Template: (<? tokens/s>) · (<? $/1k tokens>) · (31,557,600s/year) = $/year
GPT-3.5 Turbo: $6,311.52/year
GPT-4 (non-Turbo): $18,934.56/year
I’m unsure of how to model the input tokens.

How much does it cost to run a single model?

The last GPT model we have solid numbers for is GPT-3, the largest version of which has 175B parameters; Wikipedia claims it requires 800GB to store, which more or less fits the straightforward 32 bits/parameter · 175B parameters calculation.
800GB is a magnitude larger than the largest GPU memory size, so multiple GPUs are necessary to run a GPT-3 model.
People appear to be quite confident that the later models are even larger: there is a Manifold market that is 88% on GPT-4 having over 1 trillion parameters. I will use GPT-3’s numbers as a placeholder for now, since it is still illustrative.
Which GPU might be used? Using recent high end GPUs price points:

H100: has 80GB, costs $30k (HPCwire, news article).
A100: has 80GB, costs $~18k (CDW).
As a point of comparison, the consumer side RTX 4070 has 12GB but retails for around $600.

Therefore the initial capital outlay to fully load the model across multiple GPUs is:

H100: 10 · $30k = $300k
A100: 10 · $18k = $180k
RTX 4070: 67 · $600 = $40.2k

Therefore breaking even on just the GPU capital outlay can take 2-48 years, depending on which chips are used for which pricing regime, GPT-3.5 or GPT-4. (2 years for get

...
How useful is mechanistic interpretability?
by ryan_greenblatt — 2023-12-01T02:54:53.488Z

Opening positions

ryan_greenblatt

I'm somewhat skeptical about mech interp (bottom-up or substantial reverse engineering style interp):

Current work seems very far from being useful (it isn't currently useful) or explaining much what's going on inside of models in key cases. But it's hard to be very confident that a new field won't work! And things can be far from useful, but become useful via slowly becoming more powerful, etc.
In particular, current work fails to explain much of the performance of models which makes me think that it's quite far from ambitious success and likely also usefulness. I think this even after seeing recent results like dictionary learning results (though results along these lines were a positive update for me overall).
There isn't a story which-makes-much-sense-and-seems-that-plausible-to-me for how mech interp allows for strongly solving core problems like auditing for deception or being able to supervise superhuman models which carry out actions we don't understand (e.g. ELK).

That said, all things considered, mech interp seems like a reasonable bet to put some resources in.

I'm excited about various mech interp projects which either:

Aim to more directly measure and iterate on key metrics of usefulness for mech interp
Try to use mech interp to do something useful and compare to other methods (I'm fine with substantial mech interp industrial policy, but we do actually care about the final comparison. By industrial policy, I mean subsidizing current work even if mech interp isn't competitve yet because it seems promising.)

I'm excited about two main outcomes from this dialogue:

Figuring out whether or not we agree on the core claims I wrote above. (Either get consensus or find crux ideally)
Figuring out which projects we'd be excited about which would substantially positively update us about mech interp.

Maybe another question which is interesting: even if mech interp isn't that good for safety, maybe it's pretty close to stuff which is great and is good practice.

Another outcome that I'm interested in is personally figuring out how to better articulate and communicate various takes around mech interp.

ryan_greenblatt

By mech interp I mean "A subfield of interpretability that uses bottom-up or reverse engineering approaches, generally by corresponding low-level components such as circuits or neurons to components of human-understandable al

...
FixDT
by abramdemski — 2023-11-30T21:57:11.950Z

FixDT is not a very new decision theory, but little has been written about it afaict, and it's interesting. So I'm going to write about it.

TJ asked me to write this article to "offset" not engaging with Active Inference more. The name "fixDT" is due to Scott Garrabrant, and stands for "fixed-point decision theory". Ideas here are due to Scott Garrabrant, Sam Eisenstat, me, Daniel Hermann, TJ, Sahil, and Martin Soto, in roughly that priority order; but heavily filtered through my own lens.

This post may provide some useful formalism for thinking about issues raised in The Parable of Predict-O-Matic.

Self-fulfilling prophecies & other spooky map-territory connections.

A common trope is for magic to work only when you believe in it. For example, in Harry Potter, you can only get to the magical train platform 934 if you believe that you can pass through the wall to get there.

A plausible normative-rationality rule, when faced with such problems: if you want the magic to work, you should believe that it will work (and you should not believe it will work, if you want it not to work).

Can we sketch a formal decision theory which handles such problems?

We can't start by imagining that the agent has a prior probability distribution, like we normally would, since the agent would already be stuck -- either it lucked into a prior which believed the magic could work, or, it didn't.

Instead, the "beliefs" of the agent start out as maps from probability distributions to probability distributions. I'll use "P" as the type for probability distributions (little p for a specific probability distribution). So the type of "beliefs", B, is a function type: b:P→P (little b for a specific belief). You can think of these as "map-territory connections": b is a (causal?) story about what actually happens, if we believe p. A "normal" prior, where we don't think our beliefs influence the world, would just be a constant function: it always outputs the same p no matter what the input is.

Given a belief b, the agent then somehow settles on a probability distribution p. We can now formalize our rationality criteria:

Epistemic Constraint: The probability distribution p which the agent settles on cannot be self-refuting according to the beliefs. It must be a fixed point of b: a p such th...