trentmkelly/LessWrong-43k huggingface dataset:
1. The CAST Strategy
by Max Harms — 2024-06-07T22:29:13.005Z

(Part 1 of the CAST sequence)

AI Risk Introduction

(TLDR for this section, since it’s 101 stuff that many readers will have already grokked: Misuse vs Mistake; Principal-Agent problem; Omohundro Drives; we need deep safety measures in addition to mundane methods. Jump to “Sleepy-Bot” if all that seems familiar.)

Earth is in peril. Humanity is on the verge of building machines capable of intelligent action that outstrips our collective wisdom. These superintelligent artificial general intelligences (“AGIs”) are almost certain to radically transform the world, perhaps very quickly, and likely in ways that we consider catastrophic, such as driving humanity to extinction. During this pivotal period, our peril manifests in two forms.

The most obvious peril is that of misuse. An AGI which is built to serve the interests of one person or party, such as jihadists or tyrants, may harm humanity as a whole (e.g. by producing bioweapons or mind-control technology). Power tends to corrupt, and if a small number of people have power over armies of machines we should expect horrible outcomes. The only solution to misuse is to ensure that the keys to the machine (once/if they exist) stay in the hands of wise, benevolent representatives, who use it only for the benefit of civilization. Finding such representatives, forming a consensus around trusting them, and ensuring they are the only ones with the power to do transformative things is a colossal task. But it is, in my view, a well-understood problem that we can, as a species, solve with sufficient motivation.

The far greater peril, in my view, is that of a mistake. The construction of superintelligent AI is a form of the principal-agent problem. We have a set of values and goals that are important to us, and we need to somehow impart those into the machine. If we were able to convey the richness of human values to an AI, we would have a “friendly AI” which acted in our true interests and helped us thrive. However, this task is subtly hard, philosophically confused, technically fraught, and (at the very least) vulnerable to serious errors in execution. We should expect the first AGIs to have only a crude approximation of the goal they were trained to accomplish (which is, itself, likely only a subset of what we find valuable), with the severity of the difference growing exponentially with the c...
0. CAST: Corrigibility as Singular Target
by Max Harms — 2024-06-07T22:29:12.934Z

What the heck is up with “corrigibility”? For most of my career, I had a sense that it was a grab-bag of properties that seemed nice in theory but hard to get in practice, perhaps due to being incompatible with agency.

Then, last year, I spent some time revisiting my perspective, and I concluded that I had been deeply confused by what corrigibility even was. I now think that corrigibility is a single, intuitive property, which people can learn to emulate without too much work and which is deeply compatible with agency. Furthermore, I expect that even with prosaic training methods, there’s some chance of winding up with an AI agent that’s inclined to become more corrigible over time, rather than less (as long as the people who built it understand corrigibility and want that agent to become more corrigible). Through a slow, gradual, and careful process of refinement, I see a path forward where this sort of agent could ultimately wind up as a (mostly) safe superintelligence. And, if that AGI is in the hands of responsible governance, this could end the acute risk period, and get us to a good future.

This is not the path we are currently on. As far as I can tell, frontier labs do not understand corrigibility deeply, and are not training their models with corrigibility as the goal. Instead, they are racing ahead with a vague notion of “ethical assistance” or “helpful+harmless+honest” and a hope that “we’ll muddle through like we always do” or “use AGI to align AGI” or something with similar levels of wishful thinking. Worse, I suspect that many researchers encountering the concept of corrigibility will mistakenly believe that they understand it and are working to build it into their systems.

Building corrigible agents is hard and fraught with challenges. Even in an ideal world where the developers of AGI aren’t racing ahead, but are free to go as slowly as they wish and take all the precautions I indicate, there are good reasons to think doom is still likely. I think that the most prudent course of action is for the world to shut down capabilities research until our science and familiarity with AI catches up and we have better safety guarantees. But if people are going to try and build AGI despite the danger, they should at least have a good grasp on corrigibility and be aiming for it as the singular target, rather than as part of a mixture of goals (...
What is space? What is time?
by Tahp — 2024-06-07T22:15:55.951Z

This is an attempt to describe space and time without much physics baggage. Then I can refer back to this post when I want to add more physics baggage.

It is surprisingly difficult to explain what I mean by “time” and “space” colloquially. The assumptions of time and space permeate language. The most defensible thing I can say is that locations in space are measured by comparing position to measuring tapes, and space is the thing in which you stretch out measuring tapes to measure distance. Time is the thing measured by the number that shows up on a stopwatch that I start when I see some first event and stop when I see a second event. Is it tautological to refer to clocks or measuring tapes? Am I cheating if I use the word “when” to describe events? Am I cheating if I use the word “interval” to describe the space or time in which things happen? As I often answer questions of definition, I will vaguely wave at concepts and hope that the thing in concept space that I’m referring to is well-defined enough that you can understand what I mean.

Space as an inverse counter for similar objects

You live in a house with no inner walls inside a cave for some reason. You can tell when something is inside the house or outside of the house. You decide to fill your house with bricks. You pick a particular type of brick with flat faces that all meet at right angles so you can fit them right next to each other and they looks the same no matter what side you put them down on. You start on one side of the inside of your house and put one brick near the wall. Then you put another brick next to that brick and keep going until you have a line of bricks from one side of the house to the other. Then you could set an identical line of bricks right next to that one and keep doing that until you have a flat layer of bricks all the way across your floor. Then you could stack another layer of bricks on top of that and keep doing that until your house is full and then there would be no room for more bricks. You notice that you had to fill your house in three different directions. First you filled from one side to the other with bricks, then from one side to the other with lines of bricks, then from bottom to top with layers of bricks. But once you filled one layer of bricks, you couldn’t tell anymore what direction you started with. There are lines of bricks running from side to side and ...
Question about Lewis' counterfactual theory of causation
by jbkjr — 2024-06-07T20:15:27.561Z

In reading the SEP entry on counterfactual theories of causation, I had the following question occur, and I haven't been able to satisfactorily resolve it for myself.

An event e is said to causally depend on an event c if and only if e would occur if c were to occur and e would not occur if c were not to occur.

The article makes a point of articulating that causal dependence entails causation (if e causally depends on c, c is a cause of e) but not vice versa. It then defines a causal chain as a fine sequence of events c, d, e,... where d causally depends on c, e on d, and so on, before defining c to be a cause of e if and only if there exists a causal chain leading from c to e.

What I'm having trouble with is understanding how c can cause e according to the given definition without e causally depending on c. If there's a causal chain from c to d to e, then d causally depends on c, and e causally depends on d, so if c were to not occur, d would not occur, and if d were to not occur, e would not occur. But doesn't this directly entail that if c were to not occur, then e would not occur and therefore that e causally depends on c?

So how can c cause e according to the definition without e causally depending on c??
Relationships among words, metalingual definition, and interpretability
by Bill Benzon — 2024-06-07T19:18:18.389Z

This is cross-posted from New Savanna,

First, I talk about now natural language is its own metalanguage and that allows them to define new works in terms of existing ones. Then I discuss the concept of justice in terms of mechanism of metalingual definition proposed by David Hays some years ago. I conclude with some remarks on interpretability in view of Anthropic’s recent research on features.

The metalingual function of language in defining word meaning

In a famous essay published in 1960, “Linguistics and Poetics,” Roman Jakobson listed six functions of language. While the essay focused on the poetic function, as the title indicates, I’m interested in a different function, which he called the metalingual function:

A distinction has been made in modern logic between two levels of language: “object language” speaking of objects and “metalanguage” speaking of language.10 But metalanguage is not only a necessary scientific tool utilized by logicians and linguists; it plays also an important role in our everyday language. Like Molière's Jourdain who used prose without knowing it, we practice metalanguage without realizing the metalingual character of our operations. Whenever the addresser and/or the addressee need to check up whether they use the same code, speech is focused on the code: it performs a METALINGUAL (i.e., glossing) function.

In the process of explicating that function Jakobson pointed out that it can be used to define words, noting that “any process of language learning, in particular child acquisition of the mother tongue, makes wide use of such metalingual operations.”

Not all words get their meaning in that way. Many words have their meanings grounded in sensorimotor experience. We would like to know what percentage of words have their meanings grounded in sensorimotor experience and what percentage have their meanings grounded in other words. In 2016 Steven Harnad and his colleagues published an article investigating this problem, “The Latent Structure of Dictionaries.” They examined the structure of the vocabularies in two dictionaries, one with roughly 47,000 words and the other with roughly 69,000 words. They found that a large majority of the words were defined in terms of a relatively small number of words defined in terms of sensorimotor features (p. 649):

So in our view the mental lexicon is itself hybrid—a dual-code representational system

...
Let’s Talk About Emergence
by jacobhaimes — 2024-06-07T19:18:16.382Z
Link: https://www.odysseaninstitute.org/post/let-s-talk-about-emergence

Emergence has found its way into machine learning vocabulary, but current use has resulted in a circular definition and has further confused an already complex domain.

Clarote &

AI4Media

/

Better Images of AI

/

Power/Profit

/ CC-BY 4.0

The field of machine learning has existed for many decades, but only recently have governments become actively concerned about the technologies leveraging its most advanced techniques. For a majority of people, this can be traced to the launch of ChatGPT, when we entered an era of so-called Large Language Models, or LLMs. One reasonable question, then, is what made ChatGPT dissimilar to its predecessors?

One distinction that has been proposed as a key differentiation between LLMs, and their smaller counterpart, Language Models, is that LLMs exhibit Emergence, or equivalently, that some of their capabilities have been categorized as Emergent

[1]

. It is important to note that, in this context, the root Emerge is being used as a keyword specific to the domain of machine learning, and not for its other definitions. Although the nuances of the definition differ between publications, the root Emerge is frequently reduced to some variation of the definition given by Wei et al.: “An ability is [E]mergent if it is not present in smaller models but is present in larger models.”

[2]

Although this meaning does result in circular reasoning when taken in conjunction with the description of LLMs proposed by domain experts in “Large Language Models: A Survey,” it is the one that has been largely accepted within machine learning circles, so we will use it as the basis for our understanding within this article

[1], [3], [4], [5]

. The term’s meaning, as a keyword in the field of machine learning, is obfuscated by a number of factors, which we will explore in this article.

Emergence has been referenced in many works as a salient threat vector that could cause significant harm if ignored; a prominent paper from GovAI titled “Open Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives” highlighted Emergence as a reason to refrain from Open Source practices, and a relatively recent paper titled “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training” recognized Emergent deception as a threat vector sa...
D&D.Sci Alchemy: Archmage Anachronos and the Supply Chain Issues
by aphyer — 2024-06-07T19:02:06.859Z

This is an entry in the 'Dungeons & Data Science' series, a set of puzzles where players are given a dataset to analyze and an objective to pursue using information from that dataset.

After talking with abstractapplic, I've stolen the June 7th scenario slot from him. I hope that this scenario should be relatively simple: not as easy as abstractapplic's recent introductory scenario, but I still think this would be a fairly approachable starting point if you're new to D&D.Sci.

STORY

Archmage Anachronos stares at you from beneath his bushy eyebrows. "I have called you to beseech your aid as the greatest living practitioner of the Ancient and Forbidden Art of...Data Science."

"What, my eyes? No, they've always been like that." (Image generated using OpenArt)

(You've tried many times to explain to Archmage Anachronos that Data Science is neither Ancient nor Forbidden, and in fact that the Calantha Institute of Technology and Thaumaturgy has regular classes in Data Science that he could just attend. It hasn't seemed to work. Ever since you used Data Science to help him locate the lair of the Loathsome Lich, he's decided that it must be a mighty power indeed, and that you must be a great wizard of some sort to be able to use it. Or maybe he just enjoys being dramatic.)

You wait to hear what the problem is. Is the world being swallowed by darkness? Have the Elemental Lords reawoken and begun subjugating nations? What dread occasion has led him to seek the aid of one who wields so perilous an art[1]?

He tells you that he needs your help brewing some Barkskin Potion.

(One side effect of Archmage Anachronos's personality - a love of ridiculous drama, a penchant for overcomplicated schemes, and a strong tendency to frequently disappear to conduct secretive 'archmage business' - is that it is hard to tell whether this is actually as unimportant a matter as it sounds, or whether there is some vitally important objective he needs this potion for.)

With his great experience in alchemy (he proudly informs you), he brews a successful potion more than half the time, and remembers every potion he's ever tried to make. (He presents you with some recently-written records of his brewing, the ink still wet). But he doesn't always succeed in making a successful potion, or in making the type of potion he was hopin...
Natural Latents Are Not Robust To Tiny Mixtures
by johnswentworth — 2024-06-07T18:53:36.643Z

In our

previous

natural

latent

posts

, our core theorem typically says something like:

Assume two agents have the same predictive distribution P[X] over variables X, but model that distribution using potentially-different latent variables. If the latents both satisfy some simple “naturality” conditions (mediation and redundancy) then the two agents’ latents contain approximately the same information about X. So, insofar as the two agents both use natural latents internally, we have reason to expect that the internal latents of one can be faithfully translated into the internal latents of the other.

This post is about one potential weakness in that claim: what happens when the two agents’ predictive distributions are only approximately the same?

Following the pattern of our previous theorems, we’d ideally say something like

If the two agents’ distributions are within ϵ of each other (as measured by some KL-divergences), then their natural latents contain approximately the same information about X, to within some O(ϵ) bound.

But that turns out to be false.

The Tiny Mixtures Counterexample

Let’s start with two distributions, P0 and Q0, over X. These won’t be our two agents’ distributions - we’re going to construct our two agents’ distributions by mixing these two together, as the name “tiny mixtures” suggests.

P0 and Q0 will have extremely different natural latents. Specifically:

X1 consists of 1 million bits, X2 consists of another 1 million bits
Under P0, X1 is uniform, and X2=X1. So, there is an exact natural latent ΛP=X1=X2 under P0.
Under Q0, X1 and X2 are independent and uniform. So, the empty latent ΛQ is exactly natural under Q0.

Mental picture: we have a million-bit channel, under P0 the output (X2) is equal to the input (X1), while under Q0 the channel hardware is maintained by Comcast so they’re independent.

Now for our two agents’ distributions, P and Q. P will be almost P0, and Q will be almost Q0, but each agent puts a 1250 probability on the other distribution:

P=(1−1250)P0+1250Q0
Q=1250P0+(1−1250)Q0

First key observation: DKL(P||Q) and DKL(Q||P) are both rou...
Situational Awareness Summarized - Part 2
by Joe Rogero — 2024-06-07T17:20:03.513Z

This is the second post in the Situational Awareness Summarized sequence. Collectively, these posts represent my attempt to condense Leopold Aschenbrenner's recent report, Situational Awareness, into something more digestible.

Part II: Zoom to Foom

This section outlines how the development of AGI quickly and inevitably leads to superintelligence.

Leopold points out that a massive increase in AI research progress doesn't require that we figure out how to automate everything - just that we figure out how to automate AI research. Building on the "drop-in remote worker" concept from Part 1, Leopold argues that this is plausible - and that it scales. With a few more years of growth in our compute capacity, we'd be able to run as many as 100 million "human-equivalent" researchers, and possibly speed them up as well. Part II predicts that automated researchers could give us a decade's progress (5+ OOMs) in a single year. If this happens, it likely translates to extremely rapid progress in AI capabilities - and, shortly thereafter, in everything else.

Supporting this case, Leopold outlines some of the advantages that automated researchers might have over human researchers. In particular, he highlights the increased memory and competency of AGI-level thinkers, the ability to copy and replicate after "onboarding", and the ability to rapidly turn new machine learning efficiencies into faster and better thinking (since they run on the same software they're researching).

Potential Bottlenecks

He also describes some potential bottlenecks to this process, and how they might be overcome. Possible bottlenecks include:

Limited compute available for experiments
Complementarities and long tails - the hardest fraction of work will govern speed
Fundamental limits to algorithmic efficiencies
Other sources of diminishing returns to research

Compute for experiments

Leopold considers this the most important bottleneck. Pure cognitive labor alone can't make rapid progress; you also need to run experiments. Some of these experiments will take vast quantities of compute. Leopold suggests some ways that automated researchers might work around this problem:

Running small tests
Running a few large tests with huge gains
Increasing effective compute by compounding efficiencies
Spending lots of researcher-time to avoid bugs
Having better intuitions overall (by being smarter

...
Frida van Lisa, a short story about adversarial AI attacks on humans
by arisAlexis — 2024-06-07T13:22:38.118Z

Lights

Aurelio is stuck looking at the back of his car. Seems there is a note for him in Hebrew, written by finger on the dusty window. There is only one person that speaks it in his inner circle, his best friend Chloe, who he hasn’t seen for a while. Why would she ever leave him a message like that, and not on his phone? He quickly looks it up on Google translate. “The sadness will last forever.” “I know my car is dirty, no need to rub it in my face!” he texted her. She is not online.

Chloe is what someone would describe as a very normal person. Not boring at all, but nothing atypical whatsoever either. She graduated from the London School of Economics and works for an investment fund, one of the bigger ones in the city. Many people think that she is an accomplice in money laundering, as many of the funds there do have the occasional connection to a Ukrainian mafia boss or a Qatari prince, but she scorns them by stating it’s part of her job.

Flashbacks from two weeks ago come to Aurelio’s mind of when he was having a final nightcap in his hotel room with Chloe after an event at an art gallery in London. What a crazy experience it was. “I am an artist and a millionaire. How surreal! It feels like a fraud…” His racing mind comes back to Chloe and 2 her large smile. She decided after all to take the plunge and go to a sperm bank. “Why are all the men Scandinavian? Is it a trend there? Good genes, I suppose,” she laughed, adding “all women secretly want a tall blonde guy with blue eyes as the father of their children. It’s in the psychology textbooks!” Her eyes sparkled as she announced that she was pregnant. “Well after all, you didn’t want to donate, so now I have to settle with some Dane,” she teased him. They both drank to that awkwardness and laughed it off.

Aurelio is trying to reach her by phone, but to no avail. Calling her mother is a dead end; clinical depression cannot be reasoned with, she hardly speaks and is utterly detached. Aurelio decides to drop in at her apartment to check out what’s going on. Gets on his bike and in five minutes he is there. A chill goes down his spine as he looks at her window and notices the lights are off. Now, that would be perfectly understandable for any other person in the world but Chloe. She has a fear of the dark and leaves the lights on no matter what so she doesn’t return to a dark apartment. Aurelio has tried time a...