trentmkelly/LessWrong-43k huggingface dataset:
LLMs stifle creativity, eliminate opportunities for serendipitous discovery and disrupt intergenerational transfer of wisdom
by Ghdz — 2024-08-05T18:27:20.709Z

In this post, I’ve made no attempt to give an exhaustive presentation of the countless unintended consequences of widespread LLM use; rather, I’ve concentrated on three potential effects that are at the borderline of research, infrequently discussed, and appear to resist a foreseeable solution.

This post argues that while LLMs exhibit impressive capabilities in mimicking human language, their reliance on pattern recognition and replication may, among other societally destructive consequences:

stifle genuine creativity and lead to a homogenization of writing styles — and consequently — thinking styles, by inadvertently reinforcing dominant linguistic patterns while neglecting less common or marginalized forms of expression,
eliminate opportunities for serendipitous discovery; and
disrupt intergenerational transfer of wisdom and knowledge.

As I argue in detail below, there is no reason to believe that those problems are easily mitigatable. The sheer scale of LLM-derived content production, which is likely to dwarf human-generated linguistic output in the near future, poses a serious challenge to the preservation of lexical diversity. The rapid proliferation of AI-generated text could create a “linguistic monoculture”, where the nuanced and idiosyncratic expressions that characterize human language are drowned out by the algorithmic efficiency of LLMs.

LLMs Threaten Creativity in Writing (and Thinking)

LLMs are undoubtedly useful for content generation. These models, trained on vast amounts of data, can generate perfectly coherent and contextually relevant text in ways that ostensibly mimic human creativity. It is precisely this very efficiency of LLMs that will tilt the scales in favor of AI-generated content over time. Upon closer examination, there seem to be a number of insidious consequences infrequently discussed in this connection: the potential erosion of genuine creativity, linguistic diversity, and ultimately, the richness of human expression, human thinking, and human experience that depend on our independent linguistic capacities.

LLMs operate on a principle of pattern recognition and replication. They are great at identifying and reproducing stylistic patterns, grammatical structures, and even thematic elements from their training data. This often creates an illusion of creativity, where the generated text appears novel and...
Circular Reasoning
by abramdemski — 2024-08-05T18:10:32.736Z

The opinions here are my own, but owe some salience-work to Sahil.

The idea that circular reasoning is bad is widespread. However, this reputation is undeserved. While circular reasoning should not be convincing (at least not usually), it should also not be considered invalid.

Circular Reasoning is Valid

The first important thing to note is that circular reasoning is logically valid. A implies A. Longer circular arguments (like A implies B implies A) may introduce an invalidity (perhaps A does not imply B); but if so, invalidity is not due to the circularity.

If circularity itself is to be critiqued, it must be by some other standard than logical validity.

I think it's fair to say that the most relevant objection to valid circular arguments is that they are not very good at convincing someone who does not already accept the conclusion. You are talking to another person, and need to think about communicating with their perspective. Perhaps the reason circular arguments are a common 'problem' is because they are valid. People naturally think about what should be a convincing argument from their own perspective, rather than the other person's.

However, notice that this objection to circular reasoning assumes that one party is trying to convince the other. This is arguments-as-soldiers mindset.[1] If two people are curiously exploring each other's perspectives, then circular reasoning could be just fine!

Furthermore, I'll claim: valid circular arguments should actually be considered as a little bit of positive evidence for their positions!

Let's look at a concrete example. I don't think circular arguments are quite so simple as "A implies A"; the circle is usually a bit longer. So, consider a more realistic circular position:[2]

Alice: Why do you believe in God?

Bob: I believe in God based on the authority of the Bible.

Alice: Why do you believe what the Bible says?

Bob: Because the Bible was divinely inspired by God. God is all-knowing and good, so we can trust what God says.

Here we have a two-step loop, A->B and B->A. The arguments are still logically fine; if the Bible tells the truth, and the Bible says God exists, then God exists. If the Bible were divinely inspired by an all-knowing and benevolent God, then it is reasonable to conclude that the Bible tells the truth.

If Bob is just honestly going through his own reasoning here (as opposed to trying to convin...
Fear of centralized power vs. fear of misaligned AGI: Vitalik Buterin on 80,000 Hours
by Seth Herd — 2024-08-05T15:38:09.682Z

Vitalik Buterin wrote an impactful blog post, My techno-optimism. I found this discussion of one aspect on 80,00 hours much more interesting. The remainder of that interview is nicely covered in the host's EA Forum post.

My techno optimism apparently appealed to both sides, e/acc and doomers. Buterin's approach to bridging that polarization was interesting. I hadn't understood before the extent to which anti-AI regulation sentiment is driven by fear of centralized power. I hadn't thought about this risk before since it didn't seem relevant to AGI risk, but I've been updating to think it's highly relevant.

[this is automated transcription that's inaccurate and comically accurate by turns :)]

Rob Wiblin (the host) (starting at 20:49):

what is it about the way that you put the reasons to worry that that ensured that kind of everyone could get behind it

Vitalik Buterin:

[...] in addition to taking you know the case that AI is going to kill everyone seriously I the other thing that I do is I take the case that you know AI is going to take create a totalitarian World Government seriously [...]

[...] then it's just going to go and kill everyone but on the other hand if you like take some of these uh you know like very naive default solutions to just say like hey you know let's create a powerful org and let's like put all the power into the org then yeah you know you are creating the most like most powerful big brother from which There Is No Escape and which has you know control over the Earth and and the expanding light cone and you can't get out right and yeah I mean this is something that like uh I think a lot of people find very deeply scary I mean I find it deeply scary um it's uh it is also something that I think realistically AI accelerates right

One simple takeaway is that recognizing and addressing that motivation for anti-regulation and pro-AGI sentiment when trying to work with or around the e/acc movement. But a second is whether to take that fear seriously.

Is centralized power controlling AI/AGI/ASI a real risk?

Vitalik Buterin is from Russia, where centralized power has been terrifying. This has been the case for roughly half of the world. Those that are concerned with of risks of centralized power (including Western libertarians) are worried that AI increases that risk if it's centralized. This puts them in confl...
Four Phases of AGI
by Gabe M — 2024-08-05T13:15:23.406Z

AGI is not discrete, and different phases lead to different opportunities, risks, and strategies

“How long until AGI? I’d say -1 years from now.”
“Considering the worst-case outcomes of AGI, I’m most concerned about the effect on jobs.”
“AGI will be like corporations, so we will control it like corporations.”
“AGI will be millions of times smarter than us and take over control with methods we can’t comprehend.”

These kinds of statements are all over AI discussions. While these statements are all about “artificial general intelligence” (AGI)—meaning AI systems that can perform as well or better than humans on a wide range of cognitive tasks—they have wildly different assumptions and implications. You could decide that some of these takes are ridiculous--that one particular framing of AGI is “correct” and the others “wrong.” People often decide this, leading to unproductive fights.

But what may actually be happening is that different people have valid reasons for believing these different takes. They are simply referring to different concepts—AGI is ambiguous. To further complicate matters, AGI is broad: “as well or better than humans” is a wide range with no upper bound, so the term “AGI” is used for AI systems spanning from advanced chatbots to incomprehensibly superhuman intelligence.

This post lays out a particular framework for conceptualizing different levels of AGI. I aim for it to be particularly useful to non-technical decision-makers who may need to deal with policy and governance issues with respect to all the above perspectives.

Four Phases

One way we can think about the progression of AGI is to split it into four separate phases—Below-human AGI (BAGI), Human-level AGI (HAGI), Moderately-superhuman AGI (MAGI), and Superintelligent AGI (SAGI). Figure 1 conceptualizes these phase transitions as general capabilities (y-axis) advances with increasing time and AI investment (x-axis):

Figure 1: Conceptual illustration of the Four Phases of AGI using mock data. Plot code is here.

By splitting AGI into these Four Phases, we can fix much of the problem where people say “AGI” but mean systems of vastly different levels of capability. But more usefully, we can identify better AI governance and safety interventions to target different risks emerging at each capability level. I’ll first go into more depth on how to think about each of these phases and ...
AI Safety at the Frontier: Paper Highlights, July '24
by gasteigerjo — 2024-08-05T13:00:46.028Z
Link: https://aisafetyfrontier.substack.com/p/paper-highlights-july-24

I'm starting a new blog where I post my (subjective) highlights of AI safety paper each month, called "AI Safety at the Frontier". I've been doing this non-publicly for the last year, so I've backfilled highlights and collections up to September 2023.

My selection primarily covers ML-oriented research. It's only concerned with papers (arXiv, conferences etc.), not LessWrong or Alignment Forum posts. As such, it should be a nice addition for people primarily following the forum, who might otherwise miss outside research.

This is my most recent selection, covering July 2024.

tl;dr

Paper of the month:

AI safety benchmarks are often correlated with progress in LLM capabilities, so these will be solved “by default”, which opens the door to safetywashing.

Research highlights:

SAD, a new benchmark for situational awareness, which should act as warning sign for dangerous capabilities and deception.
Robustness: The AgentDojo tool-use benchmark with adversaries, attacking vision-language models, and better latent adversarial training.
Steering vectors are brittle, and truthfulness might live in a 2-dimensional subspace.
How can we make LLM solutions not just accurate but also legible?
More thorough evaluations of LLM debate.

⭐Paper of the month⭐

Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?

Read the paper [CAIS, Berkeley, Stanford, Keio]

Some safety properties correlate strongly with model capabilities. If AI safety correlates with capabilities, it becomes vulnerable to safetywashing.

Historically, problems and research directions in AI safety were often identified by high-level philosophical arguments, especially in the context of transformative AI systems. The main goal of these arguments is to figure out which problems arise as models become more capable and which of these problems won’t be solved by regular capability improvements. Our paper of the month argues that this is not sufficient.

Instead of solely relying on potentially misleading “dubious intuitive arguments”, we should actively measure how much capability improvements are correlated with improvements on various measures of safety. This will then inform which safety-relevant topics are most neglected and need dedicated attention. This also allows research to carefully make progress only on safety properties, without pushing the capability frontier. If we are not careful in delineating these ar...
Game Theory and Society
by Zero Contradictions — 2024-08-05T04:27:37.275Z
Link: https://thewaywardaxolotl.blogspot.com/2014/07/game-theory-and-society.html

Game theory is a branch of mathematics that deals with decision making when there are multiple decision-makers, called “players”, and outcomes depend on the decisions of other players. Game theory is based on the metaphor of a game, in which the rules of play and the outcomes are well defined. The game metaphor is a very useful abstraction. Game theory is used to understand behavioral strategies in economics, evolutionary theory, politics, warfare and other domains. It is especially important for understanding how society works.

(see the rest of the post in the link)

The PDF version can be read here.
Near-mode thinking on AI
by Olli Järviniemi — 2024-08-04T20:47:28.085Z

There is a stark difference between rehearsing classical AI risk 101 arguments about instrumental convergence, and tackling problems like "Design and implement the exact evaluations we'll run on GPT-5 to determine whether it's capable enough that we should worry about it acting aligned until it can execute a takeover".

And naturally, since I've started working on problems like the one above, I've noticed a large shift in my thinking on AI. I describe it as thinking about risks in near-mode, as opposed to far-mode.

In this post, I share a few concrete examples about my experiences with this change-of-orientation.

I. Prerequisites for scheming

Continuing with the example from the intro: A year ago I was confident about the "the AI is just playing along with our training and evaluations, until it is in a position where it can take over" threat model (deceptive alignment / scheming) basically being the default outcome and the main source of AI x-risk. I now think I was overconfident.

Past-me hadn't really thought through the prerequisites for scheming. A textbook example of a successful deceptive alignment story, applied to an LLM, paints a picture of a model that:

Understands that the inputs it sees are simply prompts written by humans

As opposed to the model being "immersed to the setting", thinking that the current context more-or-less reflects reality, or flat-out is reality, taking the prompt at face value

Understands that it's undergoing a training process, and having an explicit (correct!) model of the training objective used to train it

As opposed the model not thinking about the training process and objective at all, and a model's changed behavior upon training being reflective of the training process molding the model, rather than the model learning the training objective and optimizing for it

Performs lots of the same cognition (e.g. the above two bullet points) regardless of the input fed to it

As opposed to the model's internal processes being heavily dependent on the input

Never performs instrumental thinking "out loud", and successfully evades any attempts made by humans to trick the model into thinking it has been deployed

As opposed to the model sometimes slipping, failing to generalize correctly to adversarial inputs, or just being insufficiently good at distinguishing training/evaluation/red-teaming from deployment.

Now, one may argue whether it...
Watermarks: Signing, Branding, and  Boobytrapping
by Shankar Sivarajan — 2024-08-04T20:41:51.926Z

This post is motivated by a popular conflation in the context of AI generation (such as the recent WSJ article about detecting "cheating") but this is applicable more generally.

There are many ways of marking ownership/provenance of a particular work, text, images, or anything else, and they serve different purposes.

If you have no adversaries working against you, say, if you're an artist whose primary goal is wanting people seeing your paintings to be able to get to your website to find more of your work, simply adding your name might be enough. An AI-related example of this kind of thing might be adding prompt information to the metadata of your image file.

But if you do, there are two dimensions in which one might wish to make changes: making such a mark harder to remove (indelibility) and making it harder to detect (invisibility). These are independently useful goals, and many techniques for achieving one of these are subjected to misplaced criticism for missing targets they aren't even aiming at.

(These are both also distinct from anti-counterfeiting measures, where the goal is to make marks that are hard to copy.)

I propose the narrower term "boobytrapping" for when the goal is make invisible[1] marks so you can punish people for using/distributing the works they're applied to for stealing/cheating/leaking etc. after the fact.

Boobytrapping can be thought of as a kind of steganography, where the content of the message is just that it exists. A famous example of such a boobytrap would be paper towns, and these serve their purpose despite being trivial to remove upon detection.

For many such uses, much of the strength of a boobytrap is from secrecy of the very fact that it exists, let alone the details of the technique used, so much so that one can often get a lot of the benefit just from a widespread belief that it exists. The indelibility of the marks once they're detected might not be worth trading off invisibility for.

Similarly, if you want indelible[1] marks to prove authorship robust to edits, which may perhaps be called "branding," you might not care if the brands are detectible as long as they don't degrade the quality of the work.

I think this paper, Embarrassingly Simple Text Watermarks ["Boobytraps" in my proposed terminology] makes the point I do here, cleverly veiled, but it could also be a result of a lack of clarity between t...
Modelling Social Exchange: A Systematised Method to Judge Friendship Quality
by Wynn Walker — 2024-08-04T18:49:30.892Z

Introduction

Evolution has designed humans to be empathetic to help others, and grateful to give back in a mutually beneficial transaction. Unfortunately, not all series of exchanges are fair long-term.

Whether these relationships are worth it is ordinarily up to intuition, but there are more accurate ways of reaching a conclusion. Not unlike Fermi guessing, this is a way of plugging multiple estimates into one equation to output an answer as to how much overall utility your company grants you.

Variables

These are the variables that will be used in the equation:

Empathy Level: How much utility one gains from the emotional satisfaction of giving.

A score of 1.0[1] indicating you gain equal utility to that which you lose.
A score of 0.0 indicating no utility gained when spent.

Empathy (E)=Utility gainedUtility spent

Gratitude Level: How much one gives back after being given.

A score of 1.0 indicating you repay exactly the same amount of utility that they spent on you.
A score of 0.0 indicating you do not repay any utility.

Gratitude (G)=Utility spentUtility gained

Imagine one friend is teaching another which keyboard shortcuts to use for a piece of software, or a poor friend lending money to a richer friend. In either of these situations the utility spent does not match the utility gained. The ratio of gain to loss can be summarised as the efficiency of the transaction.

Efficiency (N)=Utility your friend gainsUtility you spend

(Note that if your friend spends X utility on you, and you gain <X, gratitude uses the original X value, as your brain will likely compensate).

These values will hereafter be referred to as follows:

Metric	You	Your friend
Empathy Score	Y(E)	F(E)
Gratitude Score	Y(G)	F(G)
Efficiency	Y(N)	F(N)

While every interaction is different, over enough time there is an average, in which case every favour happens in six steps:

You spend X utility helping your friend.
You gain Y(E)*X utility.
Your friend gains Y(N)*X utility.
Your friend feels gratitude and decides to repay F(G)*X utility.
Your friend regains F(E)*F(G)*X utility.
You gain F(N)*F(G)*X utility.

The Maths

Change in Utility	You	Your friend
Utility gain (Concrete)	F(G)*F(N)*X	Y(N)*X
Utility gain (Empathy)	Y(E)*X	F(E)*F(G)*X
Utility loss	X	F(G)*X

If you put all this together you get the following:

Beneficial to you if	F(G)*F(N) + Y(E) - 1 > 0
Beneficial to your friend if	Y(N) + F(E)*F(G) - F(G) > 0

Ad...
We’re not as 3-Dimensional as We Think
by silentbob — 2024-08-04T14:39:16.799Z

While

thinking about

high-dimensional spaces and their less intuitive properties, I came to the realization that even three spatial dimensions possess the potential to overwhelm our basic human intuitions. This post is an exploration of the gap between actual 3D space, and our human capabilities to fathom it. I come to the conclusion that this gap is actually quite large, and we, or at least most of us, are not well equipped to perceive or even imagine “true 3D”.

What do I mean by “true 3D”? The most straightforward example would be some ℝ³ → ℝ function, such as the density of a cloud, or the full (physical) inner structure of a human brain (which too would be a ℝ³ → whatever function). The closest example I’ve found is this visualization of a ℝ³ → ℝ³ function (jump to 1:14):

(It is of course a bit ironic to watch a video of that 3D display on a 2D screen, but I think it gets the point across.)

Vision

It is true that having two eyes allows us to have depth perception. It is not true that having two eyes allows us to “see in 3D”. If we ignore colors for simplicity and assume we all saw only in grayscale, then seeing with one eye is something like ℝ² → ℝ as far as our internal information processing is concerned – we get one grayscale value for each point on the perspective projection from the 3D physical world onto our 2D retina. Seeing with two eyes then is ℝ² → ℝ² (same as before, but we get one extra piece of information for each point of the projection, namely depth[1]), but it's definitely not ℝ³ → (...). So the information we receive still has only two spatial dimensions, just with a bit more information attached.

Also note that people who lost an eye, or for other reasons don’t have depth perception, are not all that limited in their capabilities. In fact, other people may barely notice there’s anything unusual about them. The difference between “seeing in 2D” and “seeing with depth perception” is much smaller than the difference to not seeing at all, which arguably hints at the fact that seeing with depth perception is suspiciously close to pure 2D vision.

Screens

For decades now, humans have surrounded themselves with screens, whether it’s TVs, computer screens, phones or any other kind of display. The vast majority of screens are two-dimensional. You may have noticed that, for most matters and purposes, this is not much of ...