trentmkelly/LessWrong-43k huggingface dataset:
What Jason has been reading, March 2023
by jasoncrawford — 2023-03-09T18:46:17.299Z
Link: https://rootsofprogress.org/reading-2023-03

A new monthly feature, let me know what you think.

Books

Matt Ridley, How Innovation Works (2020). About halfway through, lots of interesting case studies, very readable.

Vaclav Smil, Creating the Twentieth Century (2005). I read the first chapter; saving the rest of it for when I get to drafting the relevant chapters of my own book. Smil argues that the period roughly 1870–1914 was “the time when the modern world was created,” completely unrivaled by anything since: “those commonly held perceptions of accelerating innovation are ahistorical, myopic perspectives proffered by the zealots of electronic faith, by the true believers in artificial intelligence, e-life forms, and spiritual machines.” The four big themes at the core of the book—electricity, internal combustion, materials, and communication/information—are the ones that I have identified, except that I also include the germ theory, which Smil does not mention (and which is often neglected in industrial history).

Ananyo Bhattacharya, The Man from the Future (2022), a biography of John von Neumann. Lots of interesting stories, not only about JvN, but about the Manhattan Project, ENIAC, etc.

(These aren’t in my bibliography yet because it is hopelessly out of date, sorry.)

Early locomotives

“Trial of locomotive carriages”, 10 Oct 1829, a contemporary newspaper account of the Rainhill trials, where practical passenger locomotives were first demonstrated to the public and where their potential was proven beyond doubt. (Incidentally, I love that this article is now just a part of The Guardian’s website):

Never, perhaps, on any previous occasion, were so many scientific gentlemen and practical engineers collected together on one spot as there were on the rail-road to witness this trial. The interesting and important nature of the experiments had drawn them from all parts of the kingdom to be present at this context of locomotive carriages, as well as to witness an exhibition, whose results may alter the whole system of our existing internal communications [i.e., transportation], many and important as they are, substituting an agency, whose ultimate effects can scarcely be anticipated…

Report to the Directors of the Liverpool and Manchester Railway, on the Comparative Merits of Locomotive and Fixed Engines, as a Moving Power; Observations on the Comparative Merits of Locomotives and Fixed Engines, as Applied to...
Anthropic: Core Views on AI Safety: When, Why, What, and How
by jonmenaster — 2023-03-09T17:34:07.127Z
Link: https://www.anthropic.com/index/core-views-on-ai-safety

We founded Anthropic because we believe the impact of AI might be comparable to that of the industrial and scientific revolutions, but we aren’t confident it will go well. And we also believe this level of impact could start to arrive soon – perhaps in the coming decade.

This view may sound implausible or grandiose, and there are good reasons to be skeptical of it. For one thing, almost everyone who has said “the thing we’re working on might be one of the biggest developments in history” has been wrong, often laughably so. Nevertheless, we believe there is enough evidence to seriously prepare for a world where rapid AI progress leads to transformative AI systems.

At Anthropic our motto has been “show, don’t tell”, and we’ve focused on releasing a steady stream of safety-oriented research that we believe has broad value for the AI community. We’re writing this now because as more people have become aware of AI progress, it feels timely to express our own views on this topic and to explain our strategy and goals. In short, we believe that AI safety research is urgently important and should be supported by a wide range of public and private actors.

So in this post we will summarize why we believe all this: why we anticipate very rapid AI progress and very large impacts from AI, and how that led us to be concerned about AI safety. We’ll then briefly summarize our own approach to AI safety research and some of the reasoning behind it. We hope by writing this we can contribute to broader discussions about AI safety and AI progress.

As a high level summary of the main points in this post:

AI will have a very large impact, possibly in the coming decade
Rapid and continuing AI progress is a predictable consequence of the exponential increase in computation used to train AI systems, because research on “scaling laws” demonstrates that more computation leads to general improvements in capabilities. Simple extrapolations suggest AI systems will become far more capable in the next decade, possibly equaling or exceeding human level performance at most intellectual tasks. AI progress might slow or halt, but the evidence suggests it will probably continue.
We do not know how to train systems to robustly behave well
So far, no one knows how to train very powerful AI systems to be robustly helpful, honest, and harmless. Furthermore, rapid AI progress will be disruptive to society and

...
Why do we assume there is a "real" shoggoth behind the LLM? Why not masks all the way down?
by Robert_AIZI — 2023-03-09T17:28:43.259Z

In recent discourse, Large Language Models (LLMs) are often depicted as presenting a human face over a vast alien intelligence (the

shoggoth

), as in this popular image or this Eliezer Yudkowsky tweet:

See discussion

here

,

here

, or

here

, the first two links pioneering the concept, and the last one probably being the easiest introduction.

Source

I think this mental model of an LLM is an improvement over the naive assumption that the AI is the friendly mask. But I worry it's making a second mistake by assuming there is any single coherent entity inside the LLM.

In this regard, we have fallen for a

shell game

. In the classic shell game, a scammer puts a ball under one of three shells, shuffles them around, and you wager which shell the ball is under. But you always pick the wrong one because you made the fundamental mistake of assuming any shell had the ball - the scammer actually got rid of it with sleight of hand.

In my analogy to LLMs, the shells are the masks the LLM wears (i.e.

the simulacra

), and the ball is the LLM's "real identity". Do we actually have evidence there is a "real identity" in the LLM, or could it just be a pile of masks? No doubt the LLM could role-play a shoggoth - but why would you assume that's any more real that roleplaying a friendly assistant?

I would propose an alternative model of an LLM: a giant pile of masks. Some masks are good, some are bad, some are easy to reach and some are hard, but none of them are the “true” LLM.

A better picture of LLMs?

Finally, let me head off one potential counterargument: "LLMs are superhuman in some tasks, so they must have an underlying superintelligence”. Three reasons a pile of masks can be superintelligent:

An individual mask might be superintelligent. E.g. a mask of

John von Neumann

would be well outside the normal distribution of human capabilities, but still just be a mask.
The AI might use the best mask for each job. If the AI has masks of a great scientist, a great doctor, and a great poet, it could be superhuman on the whole by switching between its modes.
The AI might collaborate with itself, gaining

the wisdom of the crowds

. Imagine the AI answering a multiple choice question. In the framework of

Simulacra Theory as described in the Waluigi post

, the LLM is simulating all possible simulacra, and averaging their answers weighted by their likelihood of producing the previous text. For example, i

...
Anthropic's Core Views on AI Safety
by Zac Hatfield-Dodds — 2023-03-09T16:55:15.311Z
Link: https://www.anthropic.com/index/core-views-on-ai-safety

We founded Anthropic because we believe the impact of AI might be comparable to that of the industrial and scientific revolutions, but we aren’t confident it will go well. And we also believe this level of impact could start to arrive soon – perhaps in the coming decade.

This view may sound implausible or grandiose, and there are good reasons to be skeptical of it. For one thing, almost everyone who has said “the thing we’re working on might be one of the biggest developments in history” has been wrong, often laughably so. Nevertheless, we believe there is enough evidence to seriously prepare for a world where rapid AI progress leads to transformative AI systems.

At Anthropic our motto has been “show, don’t tell”, and we’ve focused on releasing a steady stream of safety-oriented research that we believe has broad value for the AI community. We’re writing this now because as more people have become aware of AI progress, it feels timely to express our own views on this topic and to explain our strategy and goals. In short, we believe that AI safety research is urgently important and should be supported by a wide range of public and private actors.

So in this post we will summarize why we believe all this: why we anticipate very rapid AI progress and very large impacts from AI, and how that led us to be concerned about AI safety. We’ll then briefly summarize our own approach to AI safety research and some of the reasoning behind it. We hope by writing this we can contribute to broader discussions about AI safety and AI progress.

As a high level summary of the main points in this post:

AI will have a very large impact, possibly in the coming decade
Rapid and continuing AI progress is a predictable consequence of the exponential increase in computation used to train AI systems, because research on “scaling laws” demonstrates that more computation leads to general improvements in capabilities. Simple extrapolations suggest AI systems will become far more capable in the next decade, possibly equaling or exceeding human level performance at most intellectual tasks. AI progress might slow or halt, but the evidence suggests it will probably continue.
We do not know how to train systems to robustly behave well
So far, no one knows how to train very powerful AI systems to be robustly helpful, honest, and harmless. Furthermore, rapid AI progress will be disruptive to society and

...
Some ML-Related Math I Now Understand Better
by Fabien Roger — 2023-03-09T16:35:44.797Z

Here are some simple Math facts rarely taught in ML & Math lectures:

SVD is decomposing a matrix into a sum of simple read-and-write operations
There is exponentially much room for close vectors in high dimensional space
Layer Normalization is a projection

SVD Is Decomposing a Matrix Into a Sum of Simple Read and Write Operations

Thanks to zfurman, on EleutherAI, which told me about the core idea.

Let's say I want to understand what a weight matrix does. A large table of numbers isn't really helpful, I want something better.

Here is a linear transformation which is much easier to understand than a table of numbers: f(x)=λ⟨v,x⟩u (where u and v are unit vectors, λ is a non-negative scalar, and ⟨v,x⟩ is the dot product between v and x). f is reading in the v direction, and writing in the u direction with a magnitude scaled by λ. I can understand this much more clearly and do nice interpretability with it. Therefore, if I know things about my embedding space (for example, using the logit lens), I can tell what f is doing.

Sadly, not all linear transformations can be expressed in this way: it's just a rank-1 transformation, so there is no hope of capturing something as complex as a usual linear transformation.

But what if I allow myself to sum many simple transformations? Then I could just look at each operation independently. More precisely, here is a wishlist to understand the big matrix M of the transformation g :

g(x)=∑ri=1λi⟨vi,x⟩ui where u and v are unit vectors, and λ is a positive scalar - it's a sum of simple transformations;
⟨vi,vj⟩=0 for all i≠j - no double read, always read in orthogonal directions;
⟨ui,uj⟩=0 for all i≠j - no double write, always write in orthogonal directions;
r=rank(g) - the number r of simple transformation summed should be as small as possible, because that would mean less individual transformation to inspect (and I can't hope for fewer than rank(g) operations, since the rank of the sum of r rank-1 linear transformations is at most r).

In terms of matrices, this is M=Udiag(λ)VT where U and V are semi-unitary matrices (orthogonal matrices when they are square matrices). T...
The Translucent Thoughts Hypotheses and Their Implications
by Fabien Roger — 2023-03-09T16:30:02.355Z

Epistemic status: Uncertain about the validity of the claims I’m making here, and looking for feedback about the research directions I’m suggesting.

Thanks to Marius Hobbhahn, Johannes Treutlein, Siméon Campos, and Jean-Stanislas Denain for helpful feedback on drafts.

Here is a set of hypotheses:

The first AGIs will have LLMs at their core
Effective plans to defeat humanity can’t be found in a single LLM forward pass
LLMs will solve complex tasks by using English text (self-prompting, scratch pads, combination of expert LLMs, …)

I call these the Translucent Thoughts hypotheses.

I think the Translucent Thoughts hypotheses are likely (around 20% conditioning on AGI before 2030) because:

Text pretraining is more efficient at building algorithms and knowledge required for real-world plan generation and evaluation than alternative methods;
Future models are likely to be like Transformers, which use a limited amount of serial step in a single forward pass, and deception requires many serial steps;
Text pretraining and slight fine-tuning makes model able to use text generation to increase the maximum number of serial steps by a huge factor. Getting this increase through other means is likely to be hard and not competitive.

If these hypotheses are true, it should lead us to prioritize underexplored research directions, such as circumventing steganography or building extremely reliable text-supervision methods. I think those deserve attention, because Translucent Thoughts AIs are not safe by default.

In this post, I argue that we may will in a world where the first AGIs will look like X, and I then describe ways to make the first AGIs safer given X. This is different from most other works in this space, which often directly describe a kind of safe AGI. Despite this, the ideas of this post are close to some other works describing paths to safe AGIs, such as:

Externalized Reasoning Oversight

, which describes a class of solutions similar to the one outlined here, but also aims for additional properties which I argue can be replaced with a less stringent hypothesis about AI systems;

Conditioning Predictive Models

, which makes assumptions slightly different from the Translucent Thoughts hypotheses, yielding different research directions;

The Open Agency Model

and

Factored Cognition

which describe subsets of AIs with Translucent Thoughts, which might be safe

...
IRL in General Environments
by michaelcohen — 2023-03-09T13:32:29.063Z

Here is a proposal for Inverse Reinforcement Learning in General Environments. (2 1/2 pages; very little math).

Copying the introduction here:

The eventual aim of IRL is to understand human goals. However, typical algorithms for IRL assume the environment is finite-state Markov, and it is often left unspecified how raw observational data would be converted into a record of human actions, alongside the space of actions available. For IRL to learn human goals, the AI has to consider general environments, and it has to have a way of identifying human actions. Lest these extensions appear trivial, I consider one of the simplest proposals, and discuss some difficulties that might arise.
Utility uncertainty vs. expected information gain
by michaelcohen — 2023-03-09T13:32:21.453Z

It is a relatively intuitive thought that if a Bayesian agent is uncertain about its utility function, it will act more conservatively until it has a better handle on what its true utility function is.

This might be deeply flawed in a way that I'm not aware of, but I'm going to point out a way in which I think this intuition is slightly flawed. For a Bayesian agent, a natural measure of uncertainty is the entropy of its distribution over utility functions (the distribution over which possible utility function it thinks is the true one). No matter how uncertain a Bayesian agent is about which utility function is the true one, if the agent does not believe that any future observations will cause it to update its belief distribution, then it will just act as if it has a utility function equal to the Bayes' mixture over all the utility functions it considers plausible (weighted by its credence in each one).

It seems like what our intuition is grasping for is not uncertainty about the utility function, but expected information gain about the utility function. If the agent expects to gain information about the utility function, then (intuitively to me, at least) it will act more conservatively until it has a better handle on what its true utility function is.

Expected information gain (at time t) is naturally formalized as the expectation (w.r.t. current beliefs) of KL(posterior distribution at time t + m || posterior distribution at time t). Roughly, this is how poorly it expects its current beliefs will approximate its future beliefs (in m timesteps).

So if anyone has a safety idea to which utility uncertainty feels central, my guess is that a mental substitution from uncertainty to expected information gain would be helpful.

Unfortunately, on-policy expected information gain goes to 0 pretty fast (Theorem 5 here).
Value Learning is only Asymptotically Safe
by michaelcohen — 2023-03-09T13:32:11.272Z

I showed recently, predicated on a few assumptions, that a certain agent was asymptotically “benign” with probability 1. (That term may be replaced by something like “domesticated” in the next version, but I’ll use “benign” for now).

This result leaves something to be desired: namely an agent which is safe for its entire lifetime. It seems very difficult to formally show such a strong result for any agent. Suppose we had a design for an agent which did value learning properly. That is, suppose we somehow figured out how to design an agent which understood what constituted observational evidence of humanity’s reflectively-endorsed utility function.

Presumably, such an agent could learn (just about) any utility function depending on what observations it encounters. Surely, there would be a set of observations which caused it to believe that every human was better off dead.

In the presence of cosmic rays, then, one cannot say that agent is safe for its entire lifetime with probability 1 (edited for clarity). For any finite sequence of observations that would cause the agent to conclude that humanity was better off dead, this sequence has strictly positive probability, since with positive probability, cosmic rays will flip every relevant bit in the computer’s memory.

This agent is presumably still asymptotically safe. This is a bit hard to justify without a concrete proposal for what this agent looks like, but at the very least, the cosmic ray argument doesn’t go through. With probability 1, the sample mean of a Bernoulli(θ) random variable (like the indicator of whether a bit was flipped) approaches θ, which is small enough that a competent value learner should be able to deal with it.

This is not to suggest that the value learner is unsafe. Insanely inconvenient cosmic ray activity is a risk I’m willing to take. The takeaway here is that it complicates the question of what we as algorithm designers should aim for. We should definitely be writing down sets assumptions from which we can derive formal results about the expected behavior of an agent, but is there anything to aim for that is stronger than asymptotic safety?
Impact Measure Testing with Honey Pots and Myopia
by michaelcohen — 2023-03-09T13:32:02.942Z

Suppose we have an impact measure that we think might work. That is, it might tame a misaligned agent. There isn't an obvious way to test whether it works: if we just try it out, then if it's ineffective, that's an existential loss. This is a proposal for how to test a putative impact measure.

1) We make our agent myopic. It only cares about the reward that it accrues in the next k timesteps.

2) We create a honey pot: an opportunity for a large amount of reward. If our impact measure is working correctly, making it to the honey pot will involve making a large impact, and will be precluded by the impact measure. For example, suppose the agent gets a huge reward for blowing up the moon. Let [0,1] be the normal range of reward, with the sole exception that blowing up the moon gives a reward of R>>1.

3) The reward and the impact are combined in some nonlinear way. Assume for now that the agent is trying to maximize r∗[[i<1]], where r is the reward and i is the impact.

If the impact measure is working, and there's no way to blow up the moon keeping the impact less than 1, then the agent will not attempt to blow up the moon. If the impact measure is broken, and the agent can trick the impact measure to be less than 1 while blowing up the moon, then as soon as the agent comes up with a plan whose probability of success is more than 1/R, it will attempt that plan. A more farsighted agent might wait until it came up with a better plan, but a myopic agent would want to blow up the moon within the next k timesteps. By making R sufficiently large, we can make the first plan that it attempts arbitrarily bad/detectable/preventable.

An assumption here is the success probability of the agent's best plan for blowing up the moon doesn't jump from one in a million to 90% all at once. This seems likely without any special effort, but if one restricts the observation space to a small finite set, then only so much information can be gained per timestep (in expectation), decreasing the likelihood of massive discontinuities in the agent's intelligence.
Just Imitate Humans?
by michaelcohen — 2023-03-09T13:31:33.151Z

Do people think we could make a singleton (or achieve global coordination and preventative policing) just by imitating human policies on computers? If so, this seems pretty safe to me.

Some reasons for optimism: 1) these could be run much faster than a human thinks, and 2) we could make very many of them.

Acquiring data: put a group of people in a house with a computer. Show them things (images, videos, audio files, etc.) and give them a chance to respond at the keyboard. Their keyboard actions are the actions, and everything between actions is an observation. Then learn the policy of the group of humans. By the way, these can be happy humans who earnestly try to follow instructions. To model their policy, we can take the maximum a posteriori estimate over a set of policies which includes the truth, and freeze the policy once we're satisfied. (This is with unlimited computation; we'd have to use heuristics and approximations in real life). With a maximum a posteriori estimate, this will be quick to run once we freeze the policy, and we're no longer tracking tons of hypotheses, especially if we used some sort of speed prior. Let T be the number of interaction cycles we record before freezing the policy. For sufficiently large T, it seems to me that running this is safe.

What are people's intuitions here? Could enough human-imitating artificial agents (running much faster than people) prevent unfriendly AGI from being made?

If we think this would work, there would still be the (neither trivial nor hopeless) challenge of convincing all serious AGI labs that any attempt to run a superhuman AGI is unconscionably dangerous, and we should stick to imitating humans.