trentmkelly/LessWrong-43k huggingface dataset:
After Overmorrow: Scattered Musings on the Immediate Post-AGI World
by Yuli_Ban — 2024-02-24T15:49:05.198Z

Cutting to the case, I've been grappling with various thoughts about the immediate future for over a year now— it was almost exactly a year ago that Yudkowsky's aggressive predictions of absolute and unquestionable doom led me to decide to take a vesperant break from futurology to simply enjoy the present moment. With that break well and truly over, I've decided to throw my hat in the ring once more to take a crack at predicting the next five years to come, perhaps as a self-aware attempt to see if I may allow lightning to strike twice considering how near-perfectly I anticipated generative AI in 2019. With that stated, I am not trying to tempt fate, nor do I claim any authority or technical ability to make such claims. I am merely doing my best to look at the trends that are obvious and those not so much, as well as possible paths to take with the technology we have and the technology I expect to exist by 2029.

First and foremost, I do expect AGI to exist by 2029. Indeed, I expect that it will not even be a "new" invention, nor do I expect said AGI to exist at a rough, barely-usable level— I expect it to truly be superhuman in intelligent capability by then. In fact, it will be quantitatively superintelligent almost immediately (that is, equal to a human intellectually but superior in hardware), but by 2029, I fully expect qualitative superintelligence to emerge as well (that is, superior to a human intellectually).

In the earliest days of it, before any sort of intellectual core is developed, AGI will likely seem to be more like "fully generalized task automation" than any sort of sapient machine (indeed, this might've been a better way to define AGI: the ability to automate any task with a single model/agent), but the cascading effects is what leads us to a point where it all but assumes control over our economy, not by forced takeover but because we necessarily let it.

As a personal preference, I've decided to name this hypothetical AGI "Terios" after an Old Greek word roughly translated to "making dreams come true." I do not expect any AGI to carry this name; this only exists for the sake of this post to give it a bit of character, so any mention of "Terios" = the hypothetical ASI.

Every Little Thing Is Gonna Be Alright

Through all my musings thus far, the most important question to tackle first is "does humanity exist in 2029?" The epistemic barrier h...
CDT vs. EDT on Deterrence
by Terence Coelho — 2024-02-24T15:41:03.757Z

This seems like a natural question, but I haven't been able to find this topic discussed online.

We are in a world with two continents on opposite sides. Each continent contains a single nation with nuclear weapons. Each side has one person with a button in front of them that will launch nuclear weapons at the other continent to wipe them out without wiping themselves out. Both countries are able to perfectly spy on the other and has access to a narrow AI which can predict human behaviors with near-perfect accuracy (and has not made a mistake yet after many trials).

For some reason, you were put in front of this button on one side. One day, you see that the other side has pressed the button and the weapons are on their way over. Pressing the button will wipe out humanity permanently.

Is it fair to say that an adamant Causal Decision Theorist would not press the button and an adamant Evidential Decision Theorist would?
What would you do? Is there more context needed to make this decision?
Balancing Games
by jefftk — 2024-02-24T14:40:04.237Z

When I play an N-player game I want everyone to both:

Try to win

Win about 1/N of the time

With many games and groups of participants these are in conflict: if I
play bridge against my kids I'm going to win all the time, but I'm not
very good at the game so if I play against people who are serious
about it I'm going to lose ~all the time.

One way some games handle this is by including a lot of luck. The
more random the outcomes are, the more you'll approach 1/N regardless
of player skill. Kid games where you make no choices, like
Candyland
or War, take this to the extreme.

Instead, I think handicapping is
a much better approach. For example in Go the weaker player can start with
several stones already on the board, which gives them an advantage
while still keeping it interesting and without turning it into a
different-feeling game. When I was little and playing Go with my dad
I remember slowly reducing the number of handicaps I needed over
months, which was really rewarding: each game was fun and challenging,
and I could see my progress.

Other examples:

In Dominion, changing the ratio of coppers to estates that each
player starts with.

In Settlers of Catan, allowing weaker players to place both of
their settlements before stronger ones.

In Power Grid, Monopoly, Modern Art, or anything else
financial, letting weaker players start with more money.

In Ticket to Ride, Thurn und Taxis, Settlers of Catan, or
anything else with resource cards, letting weaker players start with
more cards.

I like it when games are designed in a way that makes this kind of
adjustment easy and granular. You can calibrate by removing a
handicap after the weaker player wins some number of games in a row (I
think three is about right though it depends on granularity) and vice
versa.

I'm curious, though: why isn't this more common? It's very normal in
Go, mostly of historical interest in chess,
and in most game cultures I'm around it seems like the expectation is
just that weaker players will just lose a lot or or stronger players
will "go easy" on them? Is it that acknowleging that some players are
stronger than others is awkward? Too hard to calculate for games with
more than two players?

Comment via: facebook, mastodon
How well do truth probes generalise?
by mishajw — 2024-02-24T14:12:19.729Z

Representation engineering (RepEng) has emerged as a promising research avenue for model interpretability and control. Recent papers have proposed methods for

discovering truth in models with unlabeled data

,

guiding generation by modifying representations

, and

building LLM lie detectors

. RepEng asks the question: If we treat representations as the central unit, how much power do we have over a model’s behaviour?

Most techniques use linear probes to monitor and control representations. An important question is whether the probes generalise. If we train a probe on the truths and lies about the locations of cities, will it generalise to truths and lies about Amazon review sentiment? This report focuses on truth due to its relevance to safety, and to help narrow the work.

Generalisation is important. Humans typically have one generalised notion of “truth”, and it would be enormously convenient if language models also had just one[1]. This would result in extremely robust model insights: every time the model “lies”, this is reflected in its “truth vector”, so we could detect intentional lies perfectly, and perhaps even steer away from them.

We find that truth probes generalise surprisingly well, with the 36% of methodologies recovering >80% of the accuracy on out-of-distribution datasets compared with training directly on the datasets. The best probe recovers 92% accuracy.

Thanks to Hoagy Cunningham for feedback and advice. Thanks to LISA for hosting me while I did a lot of this work. Code is available at

mishajw/repeng

, along with

steps

for reproducing datasets and plots.

Methods

We run all experiments on

Llama-2-13b-chat

, for parity with the source papers. Each probe is trained on 400 questions, and evaluated on 2000 different questions, although numbers may be lower for smaller datasets.

What makes a probe?

A probe is created using a training dataset, a probe algorithm, and a layer.

We pass the training dataset through the model, extracting activations[2] just after a given layer. We then run some statistics over the activations, where the exact technique can vary significantly - this is the probe algorithm - and this creates a linear probe.

Probe algorithms

and

datasets

are listed below.

A probe allows us to take the activations, and produce a scalar value where larger val...
Rawls's Veil of Ignorance Doesn't Make Any Sense
by Arjun Panickssery — 2024-02-24T13:18:46.802Z

John Rawls suggests the thought experiment of an "original position" where people decide the political system of a society under a "veil of ignorance" by which they lose the knowledge of certain information about themselves. Rawls's veil of ignorance doesn't justify the kind of society he supports.

It seems to fail at every step individually:

At best, the support of people in the OP provides necessary but probably insufficient conditions for justice, unless he refutes all the other proposed conditions involving whatever rights, desert, etc.
And really the conditions of the OP are actively contrary to good decision-making. For example, in the OP, you don't know your particular conception of the good (??) and you're essentially self-interested. . .
There's no reason to think, generally, that people disagree with John Rawls only because of their social position or psychological quirks
There's no reason to think, specifically, that people would have the literally infinite risk aversion required to support the maximin principle.
Even given everything, the best social setup could easily be optimized for the long-term (in consideration of future people) in a way that makes it very different (e.g. harsher for the poor living today) from the kind of egalitarian society I understand Rawls to support.

More concretely:

(A) I imagine that if Aristotle were under a thin veil of ignorance, he would just say "Well if I turn out to be born a slave then I will deserve it." It's unfair and not very convincing to say that people would just agree with a long list of your specific ideas if not for their personal circumstances.
(B) If you won the lottery and I demanded that you sell your ticket to me for $100 on the grounds that you would have, hypothetically, agreed to do this yesterday (before you know that it was a winner), you don't have to do this; the hypothetical situation doesn't actually bear on reality in this way.

Another frame is that his argument involves a bunch of provisions that seem designed to avoid common counterarguments but are otherwise arbitrary (utility monsters, utilitarianism, etc).
Can someone explain to me what went wrong with ChatGPT?
by Valentin Baltadzhiev — 2024-02-24T11:50:14.762Z

You've probably seen ChatGPT going crazy this week and spewing nonsense in response to all sorts of prompts (e.g. https://twitter.com/alyssamvance/status/1760094457393983829)

The official OpenAI postmortem is https://openai.statuspage.io/incidents/ssg8fh7sfyz3

LLMs generate responses by randomly sampling words based in part on probabilities. Their “language” consists of numbers that map to tokens.

In this case, the bug was in the step where the model chooses these numbers. Akin to being lost in translation, the model chose slightly wrong numbers, which produced word sequences that made no sense. More technically, inference kernels produced incorrect results when used in certain GPU configurations.

What does this mean? Did it choose the wrong tokens from the tokenizer? Was it outputting some sequence of tokens [451, 621, 9800] which should mean "I love ice" but instead got resolved to "Is there lake"? Or is it something else?

Also, if someone could explain to me in layman terms what the last sentence means about the inference kernels, I would be very grateful!
The Sense Of Physical Necessity: A Naturalism Demo (Introduction)
by LoganStrohl — 2024-02-24T02:56:31.458Z

Note on genre: This sequence is a demonstration of a complete naturalist study, as described in Intro to Naturalism and The Nuts and Bolts Of Naturalism. I think of naturalism demos as reference material. I’ve tried to make it readable, but like a dictionary or a user manual, I only expect it to be of interest to people who already have a reason to consult it.

Epistemic status: The explicit concepts I’m building around what I’ve learned are still under construction. I think the framing emphasized in this demo is askew, or incomplete, or some-other-how flawed. Perhaps I will come back in a future year to describe how my concepts have evolved. However, I stand pretty firmly behind the broad strokes of the process-level stuff.

Goals

“

Hug the Query

” is an essay by Eliezer Yudkowsky advocating a certain discipline of rationality that he calls closeness to the issue: “trying to observe evidence that is as near to the original question as possible, so that it screens off as many other arguments as possible.”

I set out to study this discipline, and to perform a naturalism demo along the way.

In this sequence, I will try to tell you what I learned, and also how I learned it. By the end, if I’ve accomplished my goals, readers who would like to reproduce my results with “Hug the Query” in particular will be well prepared to do so; and readers in the midst of some other naturalist study on an entirely different topic will find supportive illustrations.

If you haven’t read the original essay lately, I recommend pausing to do that before you read this one. It’s about a two minute read.

Motivation

Why “Hug the Query”? Why was that worth so much of my time? (And might it be worth yours?)

The simple straightforward tool-type skill discussed in "Hug the Query" is maybe not all that profound or important to me. “Remember that less central evidence is a distraction when you have ready access to more direct means of evaluation.” Yes, fine.

But the generator of that skill really matters. What is it that causes someone to “hug the query”, when they have never been told to?

When I encounter a creek, I might leap from stone to stone to make my way across. It’s not that I’ve been instructed in stone leaping, and thus execute the skill reliably when faced with a series of stones; it’s just that facing the creek, and intending to cross, this meth...
Instrumental deception and manipulation in LLMs - a case study
by Olli Järviniemi — 2024-02-24T02:07:01.769Z

Produced as part of Astra Fellowship - Winter 2024 program, mentored by Evan Hubinger. Thanks to Evan, Sam Svenningsen and Bogdan-Ionut Cirstea for suggestions and discussions on the topic.

Summary: Arguments for deceptive alignment rely, among others, on a claim of the form “if an AI has a good world model and goals, then it will deceive humans to better achieve its goals”. The validity of this implication can be studied empirically. I have constructed a prompt for Claude 2.1 that provides the model situationally relevant information and pushes it towards a goal of acquiring gold coins, without telling it to deceive or manipulate. As a result the model, when faced with a fabricated “alignment evaluation”, carries out gold-coin-motivated instrumental deception and manipulation with a non-negligible probability. The behavior is present in many variations of the prompt. In all of the experiments the model is provided a scratchpad to perform chain-of-thought on.

Introduction

Alignment-faking refers to AIs pretending to be more aligned than they are. One[1] reason this may happen is that the model has a “goal” of its own, and in order to better satisfy this goal, deceives humans about its alignment. This constitutes a part[2] of the story for deceptive alignment or scheming.

The structure of the argument here is:

“If an AI has a good world model (including reasoning abilities and strategically relevant information about its situation) and misaligned goal(s), then it will deceive humans”. (*)

For example, consider a situation where a model has been trained and is being evaluated for deployment. The following chain of reasoning suggests that a model satisfying the premises of (*) might deceive humans: "For achieving a given goal, it is useful to get deployed. Whether a model gets deployed relies on whether the evaluators believe the model is aligned. Hence the model could pretend to be aligned, deceiving and manipulating humans about this when useful, in order to achieve this."

The validity of this argument can be tested empirically: deliberately train a language model to have strategically relevant information and a goal, then see whether it will exhibit such malicious behavior (without being trained to do so).[3]

I have done such an experiment so far with prompting. I describe what I've done below in the sections Ex...
A starting point for making sense of task structure (in machine learning)
by Kaarel — 2024-02-24T01:51:49.227Z

ML models can perform a range of tasks and subtasks, some of which are more closely related to one another than are others. In this post, we set out two very initial starting points. First, we motivate reverse engineering models’ task decompositions. We think this can be helpful for interpretability and for understanding generalization. Second, we provide a (potentially non-exhaustive, initial) list of techniques that could be used to quantify the ‘distance’ between two tasks or inputs. We hope these distances might help us identify the task decomposition of a particular model. We close by briefly considering analogues in humans and by suggesting a toy model.

Epistemic status: We didn’t spend much time writing this post. Please let us know in the comments if you have other ideas for measuring task distance or if we are replicating work.

Introduction

It might be useful to think about computation in neural networks (and in LMs specifically) on sufficiently complex tasks as a combination of (a) simple algorithms or circuits for specific tasks[1] and (b) a classifier, or family of classifiers, that determine which simple circuits are to be run on a given input. (Think: an algorithm that captures (some of) how GPT-2 identifies indirect objects in certain cases combined with a method of identifying that indirect object identification is a thing that should be done.[2]) More concretely, some pairs of tasks might overlap in that they are computed together much more than are other pairs, and we might want to build a taxonomic tree of tasks performed by the model in which tree distance between tasks is a measure of how much computation they share.[3] For example, a particularly simple (but unlikely) task structure could be a tree of depth 1: the neural network has one algorithm for classifying tasks which is run on all inputs, and then a single simple task is identified and the corresponding algorithm is run.

Why understanding task structure could be useful

Interpretability

We might hope to interpret a model by 1) identifying the task decomposition, and 2) reverse-engineering both what circuit is implemented in the model for each task individually, and how the model computes this task decomposition. Crucially, (1) is valuable for understanding the internals and behavior of neural networks even without (2), and techniques for making progress at it could look quite diff...
Why you, personally, should want a larger human population
by jasoncrawford — 2024-02-23T19:48:10.526Z
Link: https://rootsofprogress.org/why-a-larger-population

What is the ideal size of the human population?

One common answer is “much smaller.” Paul Ehrlich, co-author of The Population Bomb (1968), has as recently as 2018 promoted the idea that “the world’s optimum population is less than two billion people,” a reduction of the current population by about 75%. And Ehrlich is a piker compared to Jane Goodall, who said that many of our problems would go away “if there was the size of population that there was 500 years ago”—that is, around 500 million people, a reduction of over 90%. This is a static ideal of a “sustainable” population.

Regular readers of The Roots of Progress can cite many objections to this view. Resources are not static. Historically, as we run out of a resource (whale oil, elephant tusks, seabird guano), we transition to a new technology based on a more abundant resource—and there are basically no major examples of catastrophic resource shortages in the industrial age. The carrying capacity of the planet is not fixed, but a function of technology; and side effects such as pollution or climate change are just more problems to be solved. As long as we can keep coming up with new ideas, growth can continue.

But those are only reasons why a larger population is not a problem. Is there a positive reason to want a larger population?

I’m going to argue yes—that the ideal human population is not “much smaller,” but “ever larger.”

Selfish reasons to want more humans

Let me get one thing out of the way up front.

One argument for a larger population is based on utilitarianism, specifically the version of it that says that what is good is the sum total of happiness across all humans. If each additional life adds to the cosmic scoreboard of goodness, then it’s obviously better to have more people (unless they are so miserable that their lives are literally not worth living).

I’m not going to argue from this premise, in part because I don’t need to and more importantly because I don’t buy it myself. (Among other things, it leads to paradoxes such as the idea that a population of thriving, extremely happy people is not as good as a sufficiently-larger population of people who are just barely happy.)

Instead, I’m going to argue that a larger population is better for every individual—that there are selfish reasons to want more humans.

First I’ll give some examples of how this is true, and then I’ll draw out some of the dee...
Deliberative Cognitive Algorithms as Scaffolding
by Cole Wyeth — 2024-02-23T17:15:26.424Z

As rationalists, we are interested in finding systematic techniques that boost our effective intelligence. Because we tend to be mathematical thinkers, we usually look for precise algorithmic solutions, such as computing the optimal inference laws of Bayesian probability when this is feasible. I will call this the "rationalist project" because it seems like a reasonable candidate for the title (and not because it is the only or necessarily the primary one). There are other natural ways to increase intelligence (neuroscience or pharmaceutical tricks) but at least for me, these are not as intellectually satisfying, and also seem unlikely to yield large percentage or qualitative gains for most people. Pursuing our project, we often borrow from other (academic) fields, most notably artificial intelligence, cognitive science, and economics. None of these is really suitable.

Artificial intelligence studies how to build intelligent artifacts. In the early days, this had something to do with studying logic, heuristics, and abstractions, which are tools that humans can and did apply (though I am not so sure GOFAI increased our ability to apply them ourselves, except through loose analogies). Nowadays, the goals of A.I. research have start to come apart from the goals of the rationalist project. The art of distributing training across GPUs and choosing architectures that fully utilize them, constructing stable gradients, etc., has no direct relationship to the project. Building a superintelligence does not necessarily require the same skills as bootstrapping yourself into one without access to your own hardware. I think that this divergence of goals is sometimes ignored on lesswrong, because A.I. seems more interesting and important the more it succeeds. But in many ways, shifts in focus that drove its success also make it less central to the project.

Cognitive science is much closer to the project by its nature. However, as a science, it focuses primarily on how people do think, and not on how they should. The cognitive science research that does address how people should think tends to be a little less algorithmic and rigorous than in A.I., at least in my limited experience. One exception is this excellent paper from Josh Tenenbaum, which I will return to in this sequence: https://cocosci.princeton.edu/tom/papers/OneAndDone.pdf

Bayesian decision theory is t...