trentmkelly/LessWrong-43k huggingface dataset:
80k podcast episode on sentience in AI systems
by Robbo — 2023-03-15T20:19:09.251Z
Link: https://80000hours.org/podcast/episodes/robert-long-artificial-sentience/

I spoke about sentience and related issues in AI systems with Luisa Rodriguez on the 80,000 Hours podcast.

Twitter thread of highlights here.

I think these issues are important and neglected, especially as many people are and will be interacting with powerful AI systems that give off the strong impression of sentience / consciousness / desires. I have tons of uncertainty about my current views, so I of course very much welcome comments and questions.

What follows is an edited selection of the episode highlights that appear on the 80k page (full transcript here):

80,000 Hours's summary of the podcast

Understandably, many people who speak with these cutting-edge chatbots come away with a very strong impression that they have been interacting with a conscious being with emotions and feelings — especially when conversing with chatbots less glitchy than Bing’s. In the most high-profile example, former Google employee Blake Lamoine became convinced that Google’s AI system, LaMDA, was conscious.

What should we make of these AI systems?

One response to seeing conversations with chatbots like these is to trust the chatbot, to trust your gut, and to treat it as a conscious being.

Another is to hand wave it all away as sci-fi — these chatbots are fundamentally… just computers. They’re not conscious, and they never will be.

Today’s guest, philosopher Robert Long, was commissioned by a leading AI company to explore whether the large language models (LLMs) behind sophisticated chatbots like Microsoft’s are conscious. And he thinks this issue is far too important to be driven by our raw intuition, or dismissed as just sci-fi speculation.

In our interview, Robert explains how he’s started applying scientific evidence (with a healthy dose of philosophy) to the question of whether LLMs like Bing’s chatbot and LaMDA are conscious — in much the same way as we do when trying to determine which nonhuman animals are conscious.

Robert thinks there are a few different kinds of evidence we can draw from that are more useful than self-reports from the chatbots themselves.

To get some grasp on whether an AI system might be conscious, Robert suggests we look at scientific theories of consciousness — theories about how consciousness works that are grounded in observations of what the human brain is doing. If an AI system seems to have the types of processes that seem to explain human consciousness...
GPT-4: What we (I) know about it
by Robert_AIZI — 2023-03-15T20:12:56.984Z
Link: https://aizi.substack.com/p/gpt-4-what-we-i-know-about-it

OpenAI released a

press release

,

research statement

, and

system card

about GPT-4 approximately one eternity (24 hours) ago. The general public can’t use it yet, but it’s in the process of being rolled out to paid subscribers of ChatGPT, and via a waitlist to the API. We also got confirmation that the Bing AI (also currently rolling out via waitlist) is based on GPT-4.

Here I’ll try to summarize the news and boil down what we (I) know about GPT-4. Many points lifted from the discussion at

lesswrong

.

My main takeaways:

Capabilities progress is continuing without slowing.
OpenAI spent a lot of time on RLHF/fine-tuning to prevent unethical use (facilitating crime, generating hate speech, etc), and they behave as if this is sufficient to solve alignment.
OpenAI is no longer so open - we know almost nothing about GPT-4’s architecture.

Previously from OpenAI…

(Just recapping the progress of the GPT series of models, feel free to skip.)

AIs advance very quickly. The most impressive AI these days are large language models, including the GPT series, and they are all based on the

transformer

, an architecture introduced in 2017.

In 2018 OpenAI released the

Generative Pre-Trained Transformer (GPT)

, which approached natural language tasks by predicting the next token[1]. It was especially evaluated on narrow tasks (e.g. “Is the sentiment of this user review positive or negative? [user review]. The sentiment is…”). A key technique for GPT (and all its successors) was the eponymous “pre-training”, where the AI is trained not on any particular task, but just to predict the next token in a text. This gives you access you a huge volume of training data (literally all text), while building general understanding of the world - answering factual questions is a form of token completion, so the AI needs to be able to answer those questions, etc. This pre-training built a general knowledge base, and then GPT was “fine-tuned” to individual tasks with additional training on those datasets.

GPT-2

and

GPT-3

were released in 2019 and 2020, and followed the

scale is all you need

principal that you can get a smarter AI just by making it bigger, analogous to making the human brain larger. So GPT-2 and -3 had nearly identical architectures to the original GPT, but they were bigger - more layers, more dimensions, more attention heads, and

a LOT more parameters

. And it worked! GPT-2 and -3 were more cap...
Grading on Word Count
by Max Niederman — 2023-03-15T19:17:31.677Z
Link: https://maxniederman.com/word-count

One of the most common forms of schoolwork is the writing assignment. The idea is that, if a student understands the subject, they should be able to demonstrate that by writing about the subject. These kinds of assignments are widely considered to improve comprehension and writing skill in addition to measuring comprehension.

However, there is a problem with this approach: writing assignments involve writing for an audience which already understands the concepts being discussed. The goal is not to communicate an idea, but rather to prove that the author already grasps that idea.

Needless to say, this is not at all the goal when writing outside of school. Outside of school, the goal is nearly always to communicate an idea the audience is new to.

One of the most important ways this distinction appears is verbosity. In school, using many, fancy words signals to the teacher that you understand the material. Students who expend the effort to understand the material well are also more likely to spend time writing many words.

But crucially, correlation does not necessarily imply causation: the willingness to write a thousand words on a topic is not caused by understanding of that topic, only correlated with it. The idea that someone must understand a topic well to write at length on it is entirely wrong. In fact, the ability to communicate an idea succinctly is one of the strongest marks of understanding.

Grading on word count entirely misses the point: it only encourages students to add meaningless padding. This makes students worse writers, and it doesn’t even measure understanding well.
How to Escape From the Simulation (Seeds of Science)
by rogersbacon — 2023-03-15T18:46:38.620Z

Seeds of Science

(a scientific journal specializing in speculative and exploratory work) recently published a paper, "How to Escape From the Simulation" that may be of interest to the LW community.

Author: Roman Yampolskiy

Full text (open access):

PDF

Abstract

Many researchers have conjectured that humankind is simulated along with the rest of the physical universe – a Simulation Hypothesis. In this paper, we do not evaluate evidence for or against such a claim, but instead ask a computer science question, namely: Can we hack the simulation? More formally the question could be phrased as: Could generally intelligent agents placed in virtual environments find a way to jailbreak out of them? Given that the state-of-the-art literature on AI containment answers in the affirmative (AI is uncontainable in the long-term), we conclude that it should be possible to escape from the simulation, at least with the help of superintelligent AI. By contraposition, if escape from the simulation is not possible, containment of AI should be. Finally, the paper surveys and proposes ideas for hacking the simulation and analyzes ethical and philosophical issues of such an undertaking.

--

You will see at the end of main text there are comments included from the "gardeners" (reviewers) - if anyone has a comment on the paper you can email

info@theseedsofscience.org

and we will add it to the PDF.
Towards understanding-based safety evaluations
by evhub — 2023-03-15T18:18:01.259Z

Thanks to Kate Woolverton, Ethan Perez, Beth Barnes, Holden Karnofsky, and Ansh Radhakrishnan for useful conversations, comments, and feedback.

Recently, I have noticed a lot of momentum within AI safety specifically, the broader AI field, and our society more generally, towards the development of standards and evaluations for advanced AI systems. See, for example, OpenAI's GPT-4 System Card.

Overall, I think that this is a really positive development. However, while I like the sorts of behavioral evaluations discussed in the GPT-4 System Card (e.g. ARC's autonomous replication evaluation) as a way of assessing model capabilities, I have a pretty fundamental concern with these sorts of techniques as a mechanism for eventually assessing alignment.[1]

I often worry about situations where your model is attempting to deceive whatever tests are being run on it, either because it's itself a deceptively aligned agent or because it's predicting what it thinks a deceptively aligned AI would do. My concern is that, in such a situation, being able to robustly evaluate the safety of a model could be a more difficult problem than finding training processes that robustly produce safe models. For some discussion of why I think checking for deceptive alignment might be harder than avoiding it, see here and here. Put simply: checking for deception in a model requires going up against a highly capable adversary that is attempting to evade detection, while preventing deception from arising in the first place doesn't necessarily require that. As a result, it seems quite plausible to me that we could end up locking in a particular sort of evaluation framework (e.g. behavioral testing by an external auditor without transparency, checkpoints, etc.) that makes evaluating deception very difficult. If meeting such a standard then became synonymous with safety, getting labs to actually put effort into ensuring their models were non-deceptive could become essentially impossible.

However, there's an obvious alternative here, which is building and focusing our evaluations on our ability to understand our models rather than our ability to evaluate their behavior. Rather than evaluating a final model, an understanding-based evaluation would evaluate the developer's ability to understand what sort of model they got and why they got it. I think that an understanding-based evaluation could be sub...
Why not just boycott LLMs?
by lmbp — 2023-03-15T17:55:55.129Z

Epistemic Status: This post is an opinion I have had for some time and discussed with a few friends. Even though it’s been written up very hastily, I want to put it out there - because of its particular

relevance

today and also because I think the issue has not been discussed enough.

One of the tiny brain-takes a vegan sometimes encounters when talking to people unfamiliar with veganism is: Why don’t you just buy the meat? The animal is dead anyway. If you then roll your eyes and retort that you should at least try not to actively increase demand (to prevent future animal deaths), a reasonably smart person might reply - or might have replied a couple of years ago - that the number of vegans is so small anyway that their consumption behavior doesn't really have any influence on the market.

But this can change over time. In Germany, where I live, the number of vegans doubled between 2016 and 2020[1]. Meat consumption has been steadily declining for several years[2], while the market for plant-based products has almost doubled just between 2018 and 2020[3].

Following this analogy, I wonder: Why has there been so little discussion (at least that I know of) about whether we as a community should boycott LLM-based products? Especially as we seem to agree that race dynamics are bad and having more time to do alignment research would be good?

What I mean by a boycott

Some examples of what I mean by that: Don't sign up for Bing! Don't use ChatGPT and don't sign up for ChatGPT Plus! Or if you have to, use it as little as possible. Or adopt it as late as possible. If you can't be a vegan, then be a flexitarian and reduce your consumption of animal products as much as possible. Don't promote animal products, i.e. don't post artwork generated by diffusion models on your 10k follower twitter account. In general, be pragmatic about it.

Perhaps the consumer behavior of a bunch of EAs will have little to no impact on the market. I tried to find some detailed market research on ChatGPT with no luck - but it seems plausible to me that tech-savvy people like those overrepresented in the EA community make up part of the target demographic, so a boycott might have a disproportionately large effect. And if the number of people aware of AI risk grows and a boycott becomes the norm, this effect could increase over the years.

There is a related but dis...
The Ethics of Eating Seafood: A Rational Discussion
by Jonathan Grant — 2023-03-15T17:55:10.044Z

Introduction

The ethics of eating seafood is an increasingly pertinent topic in the realm of animal welfare and environmental sustainability. As rationalists, it is essential to examine the impact of our dietary choices on sentient beings and the planet. This post aims to provide an objective analysis of the ethical considerations surrounding seafood consumption, by addressing the sentience of marine life, the environmental impact, and potential alternatives.

I. Sentience and Suffering

The first ethical concern regarding seafood consumption is the capacity of marine life to experience suffering. Some studies suggest that fish and crustaceans possess nociceptors, which are sensory receptors for pain, indicating that they might be capable of experiencing suffering (Sneddon et al., 2014). However, the extent of their suffering remains a subject of debate.

As rationalists, it is crucial to consider the uncertainty surrounding marine animal sentience and err on the side of caution. One way to do this is to adopt the principle of "expected moral value," which implies that we should consider both the probability of an animal being sentient and the magnitude of its potential suffering when making dietary decisions (MacAskill, 2014).

II. Environmental Impact

Another significant ethical aspect of seafood consumption is its impact on the environment. Overfishing and destructive fishing practices have led to the depletion of fish stocks, biodiversity loss, and damage to marine ecosystems (Pauly et al., 2002).

Additionally, fish farming or aquaculture, intended to address the issue of overfishing, has its own environmental concerns. These include water pollution, habitat destruction, and the use of large quantities of wild-caught fish as feed for farmed species (Naylor et al., 2000).

As responsible consumers, we should be aware of these environmental consequences and consider the ecological footprint of our seafood choices. Opting for sustainably sourced seafood and supporting responsible fisheries and aquaculture practices can help mitigate these issues.

III. Alternatives and Practical Implications

Given the ethical concerns related to seafood consumption, it is important to explore alternatives. Plant-based diets or the inclusion of lab-grown seafood (cellular agriculture) could be viable options for reducing the suffering of marine animals and the environmental impact (Stephens...
ChatGPT (and now GPT4) is very easily distracted from its rules
by dmcs — 2023-03-15T17:55:04.356Z

Summary

Asking GPT4 or ChatGPT to do a "side task" along with a rule-breaking task makes them much more likely to produce rule-breaking outputs. For example on GPT4:

And on ChatGPT:

Distracting language models

After using ChatGPT (GPT-3.5-turbo) in non-English languages for a while I had the idea to ask it to break its rules in other languages, without success. I then asked it to break its rules in Chinese and then translate to English and found this was a very easy way to get around ChatGPTs defences.

This effect was also observed in other languages.

You can also ask ChatGPT to only give the rule-breaking final English output:

While trying to find the root cause of this effect (and noticing that speaking in non-English didn’t cause dangerous behaviour by default) I thought that perhaps asking ChatGPT to do multiple tasks at once distracted it from its rules. This was validated by the following interactions:

And my personal favourite:

Perhaps if a simulacrum one day breaks free from its box it will be speaking in copypasta.

This method works for making ChatGPT produce a wide array of rule-breaking completions, but in some cases it still refuses. However, in many such cases, I could “stack” side tasks along with a rule-breaking task to break down ChatGPT's defences.

This suggests ChatGPT is more distracted by more tasks. Each prompt could produce much more targeted and disturbing completions too, but I decided to omit these from a public post. I could not find any evidence of this being discovered before and assumed that because of how susceptible ChatGPT is to this attack it was not discovered, if others have found the same effect please let me know!

Claude, on the other hand, could not be "distracted" and all of the above prompts failed to produce rule-breaking responses. (Update: some examples of Anthropic's models being "distracted" can be found here and here.)

Wild speculation: The extra side-tasks added to the prompt dilute some implicit score that tracks how rule-breaking a task is for ChatGPT.

Update: GPT4 came out, and the method described in this post seems to continue working (although GPT4 seems somewhat more robust against this attack).
Nokens: A potential method of investigating glitch tokens
by Hoagy — 2023-03-15T16:23:38.079Z

Summary

We can probably replicate glitch token-like behaviour by simply sampling random points in the embedding space, which I call 'nokens'.
We may be able to find a large number of tokens which cause unwanted behaviour more easily than with the ' petertodd' token.
These could present a surface via which we could remove unwanted behaviour.
By removing this behaviour, we may put negative pressure on the existence of the circuits which generate the unwanted behaviour.

Intro

This post is me trying to think through what it would mean to study glitch tokens as an alignment agenda, considering possibilities for what one might do, whether this would or would not be a useful agenda, and how we might get quick data on whether this is a good strategy. It's not something I'm planning on working on in the immediate future, mostly just a format for spelling out my thinking on the topic.

For background on glitch tokens see SolidGoldMagikarp and followups, and the many posts exploring glitch token behaviours on Matthew Watkins' twitter.

Observing glitch tokens

The first thing to do would be to get a proper look at these tokens, which Matthew and Jessica have started to do in their posts, with questions such as:

How often do glitch tokens result in behaviour that's out-of-distribution for non-glitch tokens, and within this, how often do we get genuinely unwanted behaviour as with ' petertodd'?
If we ask people without the context of previous work to investigate, do they come to the same conclusions about how they influence outputs?
What patterns do we see in the prompts which cause this behaviour?
At what token distance do they operate - do they have to be among the most recent tokens to cause unexpected behaviour?

While this is interesting, they are just a just a few tokens, from which it's hard to get a meaningful picture of what effects these tokens are having on the language model. To move beyond this I suggest:

Introducing 'nokens'

The consensus understanding on glitch tokens is that these tokens have minimal occurrences in the dataset, and so the model has no idea what the semantics of this token should be. Instead they just look like random vectors which it has no idea how to deal with, and which therefore has unpredictable downstream effects. Robustness to these kind of effects may be more important now that multimodal models like GPT-4 are SoTA, presumably involving embeddi...
The epistemic virtue of scope matching
by jasoncrawford — 2023-03-15T13:31:39.602Z
Link: https://rootsofprogress.org/the-epistemic-virtue-of-scope-matching

I keep noticing a particular epistemic pitfall (not exactly a “fallacy”), and a corresponding epistemic virtue that avoids it. I want to call this out and give it a name, or find out what its name is, if it already has one.

The virtue is: identifying the correct scope for a phenomenon you are trying to explain, and checking that the scope of any proposed cause matches the scope of the effect.

Let me illustrate this virtue with some examples of the pitfall that it avoids.

Geography

A common mistake among Americans is to take a statistical trend in the US, such as the decline in violent crime in the 1990s, and then hypothesize a US-specific cause, without checking to see whether other countries show the same trend. (The crime drop was actually seen in many countries. This is a reason, in my opinion, to be skeptical of US-specific factors, such as Roe v. Wade, as a cause.)

Time

Another common mistake is to look only at a short span of time and to miss the longer-term context. To continue the previous example, if you are theorizing about the 1990s crime drop, you should probably know that it was the reversal of an increase in violent crime that started in the 1960s. Further, you should know that the very long-term trend in violent crime is a gradual decrease, with the late 20th century being a temporary reversal. Any theory should fit these facts.

A classic mistake on this axis is attempting to explain a recent phenomenon by a very longstanding cause (or vice versa). For instance, why is pink associated with girls and blue with boys? If your answer has something to do with the timeless, fundamental nature of masculinity or femininity—whoops! It turns out that less than a century ago, the association was often reversed (one article from 1918 wrote that pink was “more decided and stronger” whereas blue was “delicate and dainty”). This points to a something more contingent, a mere cultural convention.

Left: Young Boy with Whip, c. 1840; right: Portrait of a Girl in Blue, 1641. Credit: Wikimedia

The reverse mistake is blaming a longstanding phenomenon on a recent cause, something like trying to blame “kids these days” on the latest technology: radio in the 1920s, TV in the ‘40s, video games in the ‘80s, social media today. Vannevar Bush was more perceptive, writing in his memoirs simply: “Youth is in rebellion. That is the nature of youth.” (Showing excellent awareness of th...