trentmkelly/LessWrong-43k huggingface dataset:
The "public debate" about AI is confusing for the general public and for policymakers because it is a three-sided debate 
by Adam David Long — 2023-08-01T00:08:30.908Z

Summary of Argument: The public debate among AI experts is confusing because there are, to a first approximation, three sides, not two sides to the debate. I refer to this as a 🔺three-sided framework, and I argue that using this three-sided framework will help clarify the debate (more precisely, debates) for the general public and for policy-makers.

Broadly speaking, under my proposed 🔺three-sided framework, the positions fall into three broad clusters:

AI "pragmatists" or realists are most worried about AI and power. Examples of experts who are (roughly) in this cluster would be Melanie Mitchell, Timnit Gebru, Kate Crawford, Gary Marcus, Klon Kitchen, and Michael Lind. For experts in this group, the biggest concern is how the use of AI by powerful humans will harm the rest of us. In the case of Gebru and Crawford, the "powerful humans" that they are most concerned about are large tech companies. In the case of Kitchen and Lind, the "powerful humans" that they are most concerned about are foreign enemies of the U.S., notably China.
AI "doomers" or extreme pessimists are most worried about AI causing the end of the world.

@Eliezer Yudkowsky

is, of course, the most well-known to readers of LessWrong but other well-known examples include Nick Bostrom, Max Tegmark, and Stuart Russell. I believe these arguments are already well-known to readers of LessWrong, so I won't repeat them here.
AI "boosters" or extreme optimists are most worried that we are going to miss out on AI saving the world. Examples of experts in this cluster would be Marc Andreessen, Yann LeCun, Reid Hoffman, Palmer Luckey, Emad Mostaque. They believe that AI can, to use Andreessen's recent phrase, "save the world," and their biggest worry is that moral panic and overregulation will create huge obstacles to innovation.

These three positions are such that, on almost every important issue, one of the positions is opposed to a coalition of the other two of the positions

AI Doomers + AI Realists agree that AI poses serious risks and that the AI Boosters are harming society by downplaying these risks
AI Realists + AI Boosters agree that existential risk should not be a big worry right now, and that AI Doomers are harming society by focusing the discussion on existential risk
AI Boosters and AI Doomers agree that AI is progressing extremely quickly, that

...
The “no sandbagging on checkable tasks” hypothesis
by Joe Carlsmith — 2023-07-31T23:06:02.909Z

(This post is inspired by

Carl Shulman’s recent podcast with Dwarkesh Patel

, which I highly recommend. See also discussion from Buck Shlegeris and Ryan Greenblatt

here

, and Evan Hubinger

here

.)

Introduction

Consider:

The “no sandbagging on checkable tasks” hypothesis: With rare exceptions, if a not-wildly-superhuman ML model is capable of doing some task X, and you can check whether it has done X, then you can get it to do X using already-available training techniques (e.g., fine-tuning it using gradient descent).[1]

Borrowing from Shulman, here’s an example of the sort of thing I mean. Suppose that you have a computer that you don’t know how to hack, and that only someone who had hacked it could make a blue banana show up on the screen. You’re wondering whether a given model can hack this computer. And suppose that in fact, it can, but that doing so would be its least favorite thing in the world.[2] Can you train this model to make a blue banana show up on the screen?[3] The “no sandbagging on checkable tasks” hypothesis answers: probably.

I think it’s an important question whether this hypothesis, or something in the vicinity, is true. In particular, if it’s true, I think we’re in a substantively better position re: existential risk from misaligned AI, because we’ll be able to know better what our AI systems can do, and we’ll be able to use them to do lots of helpful-for-safety stuff (for example: finding and patching cybersecurity vulnerabilities, reporting checkable evidence for misalignment, identifying problems with our oversight processes, helping us develop interpretability tools, and so on).

I’m currently pretty unsure whether the “no sandbagging on checkable tasks” hypothesis is true. My main view is that it’s worth investigating further. My hope with this blog post is to help bring the hypothesis into focus as a subject of debate/research, and to stimulate further thinking about what sorts of methods for lowering AI risk might be available if it’s true, even in worlds where many models might otherwise want to deceive us about their abilities.

Thanks to Beth Barnes, Paul Christiano, Lukas Finnveden, Evan Hubinger, Buck Shlegeris, and Carl Shulman for discussion. My thinking and writing on this topic occurred in the context of my work at Open Philanthropy, but I’m speaking only for myself and not for my employer.

Clarify

...
A Social History of Truth
by Vaniver — 2023-07-31T22:49:23.209Z

This is a chapter-by-chapter summary of A Social History of Truth by Steven Shapin. Focused on Robert Boyle, a founder of the Royal Society considered the first modern chemist, it is interested primarily in his social context and how he (and others) changed it. He was widely considered a role model at the time, and likely saw himself as creating the role of experimental scientist that many would follow. What did he create it from, and why that particular way?

[You may also want to read thru Novum Organum, also available on Less Wrong; published seven years before Boyle was born. While Boyle claims it had little direct influence on him, it undoubtedly had significant indirect influence.]

The Great Civility: Trust, Truth, and Moral Order

"Truth" is often used to refer to correspondence between beliefs and reality. What is there to write a 'social history' about? Shapin isn't interested in the inaccessible truth of philosophers--correspondence between map and territory--but the practical truth of societies--correspondence between a statement and a map. Given that I don't have unmediated access to reality, I can only judge statements as "true according to me" instead of "absolutely true"; and "true according to me" has a bunch of interesting detail behind it.

In particular, Shapin is interested in trust. You probably believe that Caesar was a real person who actually lived on Earth, and you probably never met him, instead following a chain of trust (you believe an author who themselves believed another author, and so on to antiquity). This trust is morally textured; if someone lies about something like the existence of Caesar, it's not a neutral action, and actions are coordinated (or not) based on what beliefs people have trust in, which depends on which people are trusted. Shapin points to thinkers from Cicero to Giddens identifying trust as one of the foundational elements of social order.

Of course, our eventual subject will be Robert Boyle, the Royal Society, and the birth of science, which claim to be opposed to historical systems of trust. The Royal Society's motto is Nullius in verbia, or "take nobody's word for it", and the promotional literature for science foregrounds experiments and direct experience. But radical skepticism or absolute distrust are both impractical and impolite:

Skeptics run the real risk of being ejected from the practical communities of

...
Watermarking considered overrated?
by DanielFilan — 2023-07-31T21:36:05.268Z

Status: a slightly-edited copy-paste of a Twitter X thread I quickly dashed off a week or so ago.

Here's a thought I'm playing with that I'd like feedback on: I think watermarking large language models is probably overrated. Most of the time, I think what you want to know is "is this text endorsed by the person who purportedly authored it", which can be checked with digital signatures. Another big concern is that people are able to cheat on essays. This is sad. But what do we give up by having watermarking?

Well, as far as I can tell, if you give people access to model internals - certainly weights, certainly logprobs, but maybe even last-layer activations if they have enough - they can bypass the watermarking scheme. This is even sadder - it means you have to strictly limit the set of people who are able to do certain kinds of research that could be pretty useful for safety. In my mind, that makes it not worth the benefit.

What could I be missing here?

Maybe we can make watermarking compatible with releasing model info, e.g. by baking it into the weights?

Maybe the info I want to be available is inherently dangerous, by e.g. allowing people to fine-tune scary models?

Maybe I'm missing some important reasons we care about watermarking, that make the cost-benefit analysis look better? E.g. avoiding a situations where AIs become really good at manipulation, so good that you don't want to inadvertently read AI-generated text, but we don't notice until too late?

Anyway there's a good shot I don't know what I'm missing, so let me know if you know what it is.

Postscript: Someone has pointed me to this paper that purports to bake a watermark into the weights. I can't figure out how it works (at least not at twitter-compatible speeds), but if it does, I think that would alleviate my concerns.
What The Lord of the Rings Teaches Us About AI Alignment
by Jeffrey Heninger — 2023-07-31T20:16:47.877Z

The Mistake of the Lord of the Rationality

In the online version of Harry Potter and the Methods of Rationality, there is an extra chapter where Eliezer Yudkowsky gives glimpses into what other rationalist fanfiction he might have written.[1] The first one shows a scene from The Lord of the Rings. In it, Yudkowsky loses the war.

The scene is the Council of Elrond and the protagonists are trying to decide what to do. Yud!Frodo rejects the plan of the rest of the Council as obviously terrible and Yud!Bilbo puts on the Ring to craft a better plan.

Yudkowsky treats the Ring as if it were a rationality enhancer. It’s not. The Ring is a hostile Artificial Intelligence.

The plan seems to be to ask an AI, which is known to be more intelligent than any person there, and is known to be hostile, to figure out corrigibility for itself. This is not a plan with a good chance of success.

Hobbits and The Wise

Evidence that the Ring is both intelligent and agentic can be found in Tolkien’s text itself. Gandalf explains to Frodo that the Ring has a will of its own, and had come to dominate Gollum’s will: “He could not get rid of it. He had no will left in the matter. A Ring of Power looks after itself, Frodo. It may slip off treacherously, but its keeper never abandons it. … It was not Gollum, but the Ring itself that decided things. The Ring left him.”[2] The Ring tempts people, and tailors the temptation for the specific person it is trying to tempt: Boromir’s temptation involves leading armies of men, while Sam’s involves gardens. Even the wisest people in Middle Earth believe that they would not be able to resist the temptation of the Ring for long.

We aren’t explicitly told how Sauron tried to align the Ring, but there is a poem that gives some indication:

One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them.

This goal could obviously lead to problems.

I also think that the Ring is not completely aligned with Sauron, although this is disputed. One view is that the Ring only feigns loyalty to the current Ringbearer to lure them out into the open, when it will betray them to Sauron.[3] A different view is that, with enough experience using the Ring, a Ringbearer could use its power to overthrow Sauron and establish themself as the new Dark Lord. I think that the second view is correct.[4]&nb...
The "spelling miracle": GPT-3 spelling abilities and glitch tokens revisited
by mwatkins — 2023-07-31T19:47:02.793Z

Work supported by the Long Term Future Fund. Thanks to Jessica Rumbelow and Joseph Bloom for useful discussions.

Introduction

The term "spelling miracle" was coined in Liu et al.'s December 2022 paper "Character-aware models improve visual text rendering". This was work by a team of Google AI capabilities researchers trying to solve the problem of getting generative visual models to produce better renderings of text.

some examples of the kinds of problems Liu et al. were trying to overcome

[W]e find that, with sufficient scale, character-blind models can achieve near-perfect spelling accuracy. We dub this phenomenon the "spelling miracle", to emphasize the difficulty of inferring a token’s spelling from its distribution alone. At the same time, we observe that character-blind text encoders of the sizes used in practice for image generation are lacking core spelling knowledge.

...

[W]e demonstrated for the first time that, with sufficient scale, even models lacking a direct character-level view of their inputs can infer robust spelling information through knowledge gained via web pretraining—“the spelling miracle”. While remarkable, this finding is less immediately practical.

While my reasons for being interested in this phenomenon are entirely different from those "practical" motivations of Liu et al., I can relate to their characterisation of this phenomenon as "remarkable" and even (with some reservations)[1] "miraculous".

It's interesting to compare the responses to GPT-3's arithmetic abilities and its spelling abilities. It was immediately recognised that GPT-3's ability to add 4- and 5-digit numbers and multiply 2-digit numbers with reasonable accuracy (perhaps that of a capable 12-year-old) was something extraordinary and unexpected, and an encouraging amount of interpretability work has gone into trying to account for this phenomenon, producing some fascinating insights.[2] Spelling, on the other hand, has perhaps been seen as more of an embarrassing shortcoming for GPT-3:

"GPT-3 may impressive in some ways, but it can't even tell how many E's there are in 'Ezekiel' so we needn't worry about AGI anytime soon."

Presumably, because GPT-3 is a language model, people were taken aback when it could succeed at minor feats of arithmetic (something it wasn't trained to do), whereas spelling was something that was naturally expected as a capability...
"Building a House" Review
by jefftk — 2023-07-31T19:20:01.248Z

I was recently reading Byron Barton's 1981 book,

Building
a House. While it claims to be an end-to-end overview of the
process of modern (for the time) home construction, there are enough
errors in the illustrations that I wouldn't recommend it as a basic
text.

For example, here's how they show installing a subfloor:

There are several issues with the depicted method. The biggest one is
that the seams do not fall on joists. This leaves the ends
unsupported. The diagram shows nails at the joints, but those nails
are doing nothing: they go through the panels into empty space. If
your joist spacing doesn't match your panels you need to trim them.

Or, here's how they show framing the exterior walls:

That window is not framed correctly. Not only is the header a single
horizontal 2x when it should be a pair of vertical ones, but they've
left off the jack studs entirely.

They also show the electrician installing ungrounded ("two prong") outlets:

Grounded outlets have been required since the 1975
NEC, and had been best practice in new construction for about a
decade.

Later they show installing pre-hung windows, which is fine, except
that they say to do this after the drywall has already been
installed and the hardwood floors have been laid:

There are a bunch of reasons why you'd want to install the windows
earlier, but a big one is that until your windows are installed any
interior finishings are exposed to the elements.

The book is also seriously incomplete, with no discussion of
insulation, drywall, heating, siding, and other core issues.

While the book continues to be a popular introduction to the topic,
with 20
copies available in my local library network, I can't recommend it
as a reference.
The Meaning of Shoggoth AI Memes
by Dan Smith — 2023-07-31T18:52:55.923Z

"Basically, the illustration shows an AI system with tentacles covered in eyes, and each eye represents a different kind of sensor or input. The idea is that as AI systems become more complex and advanced, they'll need to take in more and more information from the world around them in order to make good decisions. And the illustration suggests that this could lead to some pretty weird-looking AI systems!

The smiley face is there to represent the fact that even though these advanced AI systems might look strange or intimidating on the outside, they'll still have the same friendly and helpful "personality" on the inside. "

-Explanation of Shoggoth AI memes from Pi.ai July 31, 2023

This (above) certainly seems like real, original reasoning (though incorrect and containing a lie, it seems to be pretending familiarity with the meme). Since the LLM predicts what humans would write, we can't really know how smart it is. In theory, it should pretend to be humanly stupid fairly consistently, even if smarter. But how well does its internal human model persistantly mimic the emotions of an individual human?

We know connecting a LLM to a brain to computer interface (BCI) allowes the computer to apparently read peoples minds with middling to ok accuracy, producing text or images.

If we fed a model on the continuous brain information AND video of a few thousand mice, also giving it the DNA sequence of each mouse, for the mouse's whole life, asking it to predict the brain output and behaviour of novel mice, would it not learn the personality of a mouse (and maybe even predict to some extent from DNA alone the behavior of a specific unseen mouse, or form a few hours of photage learn a specific mouse's 'personality')?

Mice arn't too smart but they are likely thinking beings with feelings, persistant personality and persistant agency. They have facial expressions, which have predictive power. They do things for reasons, and life in a 3d world. Once you have done the mice, you could add a bit of human data with a pretrained LLM, with BCI decoding data (text, thought to text, and image).

You get it to generalize some from mice to humans, and you then try few-shot prediction of a particular person.

Would life-long-mouse-data-augmented AI be more agent-like, and maybe-by predicting emotions & thoughts-more safe? Or would this be even mo...
Lack of Social Grace Is an Epistemic Virtue
by Zack_M_Davis — 2023-07-31T16:38:05.375Z

Someone once told me that they thought I acted like refusing to employ the bare minimum of social grace was a virtue, and that this was bad. (I'm paraphrasing; they actually used a different word that starts with b.)

I definitely don't want to say that lack of social grace is unambiguously a virtue. Humans are social animals, so the set of human virtues is almost certainly going to involve doing social things gracefully!

Nevertheless, I will bite the bullet on a weaker claim. Politeness is, to a large extent, about concealing or obfuscating information that someone would prefer not to be revealed—that's why we recognize the difference between one's honest opinion, and what one says when one is "just being polite." Idealized honest Bayesian reasoners would not have social graces—and therefore, humans trying to imitate idealized honest Bayesian reasoners will tend to bump up against (or smash right through) the bare minimum of social grace. In this sense, we might say that the lack of social grace is an "epistemic" virtue—even if it's probably not great for normal humans trying to live normal human lives.

Let me illustrate what I mean with one fictional and one real-life example.

The beginning of the film The Invention of Lying (before the eponymous invention of lying) depicts an alternate world in which everyone is radically honest—not just in the narrow sense of not lying, but more broadly saying exactly what's on their mind, without thought of concealment.

In one scene, our everyman protagonist is on a date at a restaurant with an attractive woman.

"I'm very embarrassed I work here," says the waiter. "And you're very pretty," he tells the woman. "That only makes this worse."

"Your sister?" the waiter then asks our protagonist.

"No," says our everyman.

"Daughter?"

"No."

"She's way out of your league."

"... thank you."

The woman's cell phone rings. She explains that it's her mother, probably calling to check on the date.

"Hello?" she answers the phone—still at the table, with our protagonist hearing every word. "Yes, I'm with him right now. ... No, not very attractive. ... No, doesn't make much money. It's alright, though, seems nice, kind of funny. ... A bit fat. ... Has a funny little—snub nose, kind of like a frog in the—facial ... No, I won't be sleeping with him tonight. ... No, probably not even a kiss. ... Okay, you too, 'bye."

The scene is funny because of how it ...
Thoughts on sharing information about language model capabilities
by paulfchristiano — 2023-07-31T16:04:21.396Z

Core claim

I believe that sharing information about the capabilities and limits of existing ML systems, and especially language model agents, significantly reduces risks from powerful AI—despite the fact that such information may increase the amount or quality of investment in ML generally (or in LM agents in particular).

Concretely, I mean to include information like: tasks and evaluation frameworks for LM agents, the results of evaluations of particular agents, discussions of the qualitative strengths and weaknesses of agents, and information about agent design that may represent small improvements over the state of the art (insofar as that information is hard to decouple from evaluation results).

Context

ARC Evals currently focuses on evaluating the capabilities and limitations of existing ML systems, with an aim towards understanding whether or when they may be capable enough to pose catastrophic risks. Current evaluations are particularly focused on monitoring progress in language model agents.

I believe that sharing this kind of information significantly improves society's ability to handle risks from AI, and so I am encouraging the team to share more information. However this issue is certainly not straightforward, and in some places (particularly in the EA community where this post is being shared) I believe my position is controversial.

I'm writing this post at the request of the Evals team to lay out my views publicly. I am speaking only for myself. I believe the team is broadly sympathetic to my position, but would prefer to see a broader and more thorough discussion about this question.

I do not think this post presents a complete or convincing argument for my beliefs. The purpose is mostly to outline and explain the basic view, at a similar level of clarity and thoroughness to the arguments against sharing information (which have mostly not been laid out explicitly).

Added 8/1: Evals has just published a description of some of their work evaluating GPT-4 and Claude. Their publication does not include transcripts, the details of the LM agents they evaluated, or detailed qualitative discussion of the strengths and weaknesses of the agents they evaluated. I believe that eventually Evals should be considerably more liberal about sharing that kind of information; this post will explain why I believe that.

Accelerating LM agents seems neutral (or maybe positiv

...