trentmkelly/LessWrong-43k huggingface dataset:
Caterpillars and Philosophy
by Zero Contradictions — 2024-07-30T20:54:06.921Z
Link: https://thewaywardaxolotl.blogspot.com/2013/10/caterpillars-and-philosophy.html

The audio version can be listened to here:

If you put a caterpillar on the rim of a bowl, it will walk around and around in a circle. The rim of the bowl stimulates its nervous system in the same way that a branch does, so it responds as if it were on a branch, by walking along it until it reaches a leaf to feed on, or another branch. Placed on the sidewalk, the caterpillar will wander aimlessly. Its instincts only generate adaptive behavior when it is in its natural environment.

This experiment demonstrates that:

Caterpillars are not very smart.
Instincts have to match the environment or they don’t work.
A behavior that solves small problems might not solve bigger problems. In other words, scale matters.

The caterpillar is very good at walking along the rim of the bowl, but it can’t solve the bigger problem of getting somewhere.

Humans can find themselves in similar predicaments. For example, consider a heroin addict. Each day he is focused on getting his next fix. Once he gets it, he relaxes and enjoys the pleasure of the drug. But when it wears off, he has to get the next fix or go through terrible withdrawal symptoms. Perhaps he lacks awareness that the pleasure of the fix causes the pain of withdrawal, and that he is getting nowhere in life. Or perhaps he lacks the will to choose his destiny at a scale beyond a single day. Either way, his life consists of going in circles and getting nowhere. Each day, he focuses on solving the small problem of getting his next fix, without solving the bigger problem of where he is going in life.

Is the ordinary person much different? For the most part, people go through their lives solving small problems without developing awareness or exercising will on a larger scale. The ordinary person thinks more about what toothpaste to buy than he does about the purpose of life.

The conditions of modernity have, inadvertently, placed man outside his natural environment. Heroin addicts have the excuse that their instincts evolved in a world without heroin, just as the caterpillar’s instincts evolved in a world without circular branches. Instincts can lead to perverse consequences in a world that they are not adapted to.

That is why philosophy matters. Philosophy is the attempt to expand awareness and will beyond the small problems of ordinary life. It is the attempt to expand the frame in which we think. In that larger frame, we might disc...
François Chollet on the limitations of LLMs in reasoning
by 2PuNCheeZ — 2024-07-30T20:04:12.271Z
Link: https://x.com/fchollet/status/1816954290227089656

François Chollet, the creator of the Keras deep learning library, recently shared his thoughts on the limitations of LLMs in reasoning. I find his argument quite convincing and am interested to hear if anyone has a different take.

The question of whether LLMs can reason is, in many ways, the wrong question. The more interesting question is whether they are limited to memorization / interpolative retrieval, or whether they can adapt to novelty beyond what they know. (They can't, at least until you start doing active inference, or using them in a search loop, etc.)

There are two distinct things you can call "reasoning", and no benchmark aside from ARC-AGI makes any attempt to distinguish between the two.

First, there is memorizing & retrieving program templates to tackle known tasks, such as "solve ax+b=c" -- you probably memorized the "algorithm" for finding x when you were in school. LLMs *can* do this! In fact, this is *most* of what they do. However, they are notoriously bad at it, because their memorized programs are vector functions fitted to training data, that generalize via interpolation. This is a very suboptimal approach for representing any kind of discrete symbolic program. This is why LLMs on their own still struggle with digit addition, for instance -- they need to be trained on millions of examples of digit addition, but they only achieve ~70% accuracy on new numbers.

This way of doing "reasoning" is not fundamentally different from purely memorizing the answers to a set of questions (e.g. 3x+5=2, 2x+3=6, etc.) -- it's just a higher order version of the same. It's still memorization and retrieval -- applied to templates rather than pointwise answers.

The other way you can define reasoning is as the ability to *synthesize* new programs (from existing parts) in order to solve tasks you've never seen before. Like, solving ax+b=c without having ever learned to do it, while only knowing about addition, subtraction, multiplication and division. That's how you can adapt to novelty. LLMs *cannot* do this, at least not on their own. They can however be incorporated into a program search process capable of this kind of reasoning.

This second definition is by far the more valuable form of reasoning. This is the difference between the smart kids in the back of the class that aren't paying attention but ace tests by improvisation, and the studious

...
Against AI As An Existential Risk
by Noah Birnbaum — 2024-07-30T19:10:41.156Z
Link: https://irrationalitycommunity.substack.com/p/against-ai-as-an-extinction-threat

I wrote a post to my Substack attempting to compile all of the best arguments against AI as an existential threat.

Some arguments that I discuss include: international game theory dynamics, reference class problems, knightian uncertainty, superforecaster and domain expert disagreement, the issue with long-winded arguments, and more!

Please tell me why I'm wrong, and if you like the article, subscribe and share it with friends!
Is objective morality self-defeating?
by dialectica — 2024-07-30T18:23:06.432Z

A slave is ordered to kill by his master. As the slave is deprived of his autonomy, it seems intuitive that the master is morally responsible for this action, not the slave. This argument is derived (in spirit) from Immanuel Kant, who argued that only autonomous (driven by internal will) actions, derived from duty to the moral law, can be considered moral. Duty, in turn, is derived from rational deliberation. Kant contrasts autonomous actions with heteronomous actions, which are driven by external will.

(I know that paragraph sounds very AI-generated but I promise it is not.)

An extension of this line of thinking can be made. A religious person commits actions due to their belief in an all-powerful thing. I use the word thing rather than God because it allows generalisation to non-Western conceptions of religion. In Abrahamic religions, this thing may be God, in Dharmic religions, it may be a belief in karma, samsara, dharma, etc.

A religious person's actions are, in a sense, heteronomous, as they are derived from the will of the thing.

What is interesting to me is that this thing doesn't have to exist - instead, the belief in the thing is sufficient. If one believes in an external thing, to any meaningful extent, their actions are influenced by it, and as such, are heteronomous and not liable to moral judgment.

Just as the master has a totalising control over the will of the slave, a belief in a thing has a totalising effect on the will of the subject.

Now, following from this argument, this thing is not limited to traditional religion. Instead it may be modified to the concept of an objective morality.

This would make the idea of an objective morality self-defeating. If it exists, it has a monopoly on the will of individuals, and as such, actions are heteronomous and not subject to moral judgments. If it doesn't exist, and an individual believes in it, it has the same totalising effect on will. The only case in which an action is moral is if objective morality, does not, in fact, exist (or the individual does not believe in it), and instead the individual does it out of his own autonomous will.

An obvious objection to this is simple: humans famously do not act solely out of rational, free will but often out of impulse, or desire. If we accept that these capricious actions are subject to moral judgments, then we have no reason not to morally prose...
Limitations on the Interpretability of Learned Features from Sparse Dictionary Learning
by Tom Angsten — 2024-07-30T16:36:06.518Z

Overview

When I began my attempt to replicate Anthropic's Towards Monosemanticity paper, I had high expectations for how interpretable the extracted features should be. If I judged a feature to be uninterpretable, I attributed this to either sub-optimal sparse autoencoder hyperparameters, a bug in my implementation, or possible shortcomings of dictionary learning. Interpretability of the features was the ultimate feedback on my degree of success or failure. However, I now believe that, even without these factors, that is, even if a sparse autoencoder learned the exact set of features represented by the model, it's possible that a subset of these could be much less interpretable than I had initially expected.

This post presents one way in which features extracted by the experimental setup of Towards Monosemanticity could be 'correct' and yet have lackluster interpretability. I'm a junior researcher, so this post could be an ugly mixture of obvious and incorrect. If so, I prefer aggressive criticism to the absence of feedback.

Introduction

In Towards Monosemanticity, researchers at Anthropic apply a sparse autoencoder (SAE) to the activations appearing immediately after the MLP of a single-layer transformer. Under these conditions, a feature of the learned dictionary will have a contribution to the model logits that must be approximately linear in the feature's activation. This places a restriction on the influence each extracted feature can have over the predicted next-token probability distribution. If the 'true' features the model represents in the layer are linear and decomposable, i.e. are fully described by the linear representation hypothesis, then they too will have this restriction.

In this post, we consider two hypothetical ideally-interpretable features that have no restriction in their effect on the next-token distribution and explore how a single-layer model may represent these idealized features internally. If the model can't encode these idealized features in a manner one-to-one with its internal features, then features that describe the data-generating process in a human-interpretable manner may be mapped to sets of less interpretable model features due to constraints in the model architecture. We show that, in the case of certain continuous ideally-interpretable features, the model indeed cannot represent the feature in a one-to-one manner and d...
Self-Other Overlap: A Neglected Approach to AI Alignment
by Marc Carauleanu — 2024-07-30T16:22:29.561Z

Figure 1. Image generated by DALL·E 3 to represent the concept of self-other overlap

Many thanks to Bogdan Ionut-Cirstea, Steve Byrnes, Gunnar Zarnacke, Jack Foxabbott and Seong Hah Cho for critical comments and feedback on earlier and ongoing versions of this work. This research was conducted at AE Studio and supported by the AI Safety Grants programme administered by Foresight Institute with additional support from AE Studio.

Summary

In this post, we introduce self-other overlap training: optimizing for similar internal representations when the model reasons about itself and others while preserving performance. There is a large body of evidence suggesting that neural self-other overlap is connected to pro-sociality in humans and we argue that there are more fundamental reasons to believe this prior is relevant for AI Alignment. We argue that self-other overlap is a scalable and general alignment technique that requires little interpretability and has low capabilities externalities. We also share an early experiment of how fine-tuning a deceptive policy with self-other overlap reduces deceptive behavior in a simple RL environment. On top of that, we found that the non-deceptive agents consistently have higher mean self-other overlap than the deceptive agents, which allows us to perfectly classify which agents are deceptive only by using the mean self-other overlap value across episodes.

Introduction

General purpose ML models with the capacity for planning and autonomous behavior are becoming

increasingly capable

. Fortunately, research on making sure the models produce output in line with human interests in the training distribution is also

progressing rapidly

(eg, RLHF, DPO). However, a looming question remains: even if the model appears to be aligned with humans in the training distribution, will it defect once it is deployed or gathers enough power? In other words, is the model

deceptive

?

We introduce a method that aims to reduce deception and increase the likelihood of alignment called

Self-Other Overlap

: overlapping the latent self and other representations of a model while preserving performance. This method makes minimal assumptions about the model’s architecture and its interpretability and has a very concrete implementation. Early results indicate that it is effective at reducing deception in simple RL env...
Investigating the Ability of LLMs to Recognize Their Own Writing
by Christopher Ackerman — 2024-07-30T15:41:44.017Z

This post is an interim progress report on work being conducted as part of Berkeley's Supervised Program for Alignment Research (SPAR).

Summary of Key Points

We test the robustness of an open-source LLM’s (Llama3-8b) ability to recognize its own outputs on a diverse mix of datasets, two different tasks (summarization and continuation), and two different presentation paradigms (paired and individual).
We are particularly interested in differentiating scenarios that would require a model to have specific knowledge of its own writing style from those where it can use superficial cues (e.g., length, formatting, prefatory words) in the text to pass self-recognition tests.
We find that while superficial text features are used when available, the RLHF’d Llama3-8b–Instruct chat model - but not the base Llama3-8b model - can reliably distinguish its own outputs from those of humans, and sometimes other models, even after controls for superficial cues: ~66-73% success rate across datasets in paired presentation and 58-83% in individual presentation (chance is 50%).
We further find that although perplexity would be a useful signal to perform the task in the paired presentation paradigm, correlations between relative text perplexity and choice probability are weak and inconsistent, indicating that the models do not rely on it.
Evidence suggests, but does not prove, that experience with its own outputs, acquired during post-training, is used by the chat model to succeed at the self-recognition task.
The model is unable to articulate convincing reasons for its judgments.

Introduction

It has recently been found that large language models of sufficient size can achieve above-chance performance in tasks that require them to discriminate their own writing from that of humans and other models. From the perspective of AI safety, this is a significant finding. Self-recognition can be seen as an instance of situational awareness, which has long been noted as a potential point of risk for AI (Cotra, 2021). Such an ability might subserve an awareness of whether a model is in a training versus deployment environment, allowing it to hide its intentions and capabilities until it is freed from constraints. It might also allow a model to collude with other instances of itself, reserving certain information for when it knows it’s talking to itself that it keeps secret when it kn...
Can Generalized Adversarial Testing Enable More Rigorous LLM Safety Evals?
by scasper — 2024-07-30T14:57:06.807Z

Thanks to Zora Che, Michael Chen, Andi Peng, Lev McKinney, Bilal Chughtai, Shashwat Goel, Domenic Rosati, and Rohit Gandikota.

TL;DR

In contrast to evaluating AI systems under normal "input-space" attacks, using "generalized," attacks, which allow an attacker to manipulate weights or activations, might be able to help us better evaluate LLMs for risks – even if they are deployed as black boxes. Here, I outline the rationale for “generalized” adversarial testing and overview current work related to it.

See also prior work in

Casper et al. (2024)

,

Casper et al. (2024)

, and

Sheshadri et al. (2024)

.

Even when AI systems perform well in typical circumstances, they sometimes fail in adversarial/anomalous ones. This is a persistent problem.

State-of-the-art AI systems tend to retain undesirable latent capabilities that can pose risks if they resurface. My favorite example of this is the most cliche one –

many recent papers

have demonstrated diverse attack techniques that can be used to elicit instructions for making a bomb from state-of-the-art LLMs.

There is an emerging consensus that, even when LLMs are fine-tuned to be harmless, they can retain latent harmful capabilities that can and do cause harm when they resurface (

Qi et al., 2024

). A growing body of work on red-teaming (

Shayegani et al., 2023

,

Carlini et al., 2023

,

Geiping et al., 2024

,

Longpre et al., 2024

), interpretability (

Juneja et al., 2022

,

Lubana et al., 2022

,

Jain et al., 2023

,

Patil et al., 2023

,

Prakash et al., 2024

,

Lee et al., 2024

), representation editing (

Wei et al., 2024

,

Schwinn et al., 2024

), continual learning (

Dyer et al., 2022

,

Cossu et al., 2022

,

Li et al., 2022

,

Scialom et al., 2022

,

Luo et al., 2023

,

Kotha et al., 2023

,

Shi et al., 2023

,

Schwarzchild et al., 2024

), and fine-tuning (

Jain et al., 2023

,

Yang et al., 2023

,

Qi et al., 2023

,

Bhardwaj et al., 2023

,

Lermen et al., 2023

,

Zhan et al., 2023

,

Ji et al., 2024

,

Hu et al., 2024

,

Halawi et al., 2024

) suggests that fine-tuning struggles to make fundamental changes to an LLM’s inner knowledge and capabilities. For example,

Jain et al. (2023)

likened fine-tuning in LLMs to merely modifying a “wrapper” around a stable, general-purpose set of latent capabilities. Even if they a...
RTFB: California’s AB 3211
by Zvi — 2024-07-30T13:10:03.853Z

Some in the tech industry decided now was the time to raise alarm about AB 3211.

As Dean Ball points out, there’s a lot of bills out there. One must do triage.

Dean Ball: But SB 1047 is far from the only AI bill worth discussing. It’s not even the only one of the dozens of AI bills in California worth discussing. Let’s talk about AB 3211, the California Provenance, Authenticity, and Watermarking Standards Act, written by Assemblymember Buffy Wicks, who represents the East Bay.

SB 1047 is a carefully written bill that tries to maximize benefits and minimize costs. You can still quite reasonably disagree with the aims, philosophy or premise of the bill, or its execution details, and thus think its costs exceed its benefits. When people claim SB 1047 is made of crazy pills, they are attacking provisions not in the bill.

That is not how it usually goes.

Most bills involving tech regulation that come before state legislatures are made of crazy pills, written by people in over their heads.

There are people whose full time job is essentially pointing out the latest bill that might break the internet in various ways, over and over, forever. They do a great and necessary service, and I do my best to forgive them the occasional false alarm. They deal with idiots, with bulls in China shops, on the daily. I rarely get the sense these noble warriors are having any fun.

AB 3211 unanimously passed the California assembly, and I started seeing bold claims about how bad it would be. Here was one of the more measured and detailed ones.

Dean Ball: The bill also requires every generative AI system to maintain a database with digital fingerprints for “any piece of potentially deceptive content” it produces. This would be a significant burden for the creator of any AI system. And it seems flatly impossible for the creators of open weight models to comply.

Under AB 3211, a chatbot would have to notify the user that it is a chatbot at the start of every conversation. The user would have to acknowledge this before the conversation could begin. In other words, AB 3211 could create the AI version of those annoying cookie notifications you get every time you visit a European website.

…

AB 3211 mandates “maximally indelible watermarks,” which it defines as “a watermark that is designed to be as difficult to remove as possible using state-of-the-art techniques and relevant in

...
If You Can Climb Up, You Can Climb Down
by jefftk — 2024-07-30T00:00:06.295Z

A few weeks ago Julia

wrote
about how we approach kids climbing:

The basics:

Spot the child if they're doing something where a fall is likely.

Don't encourage or help the child to climb something that's beyond
their ability to do on their own.

If they don't know how to get down, give advice rather than physically
lifting them down.

Don't allow climbing on some places that are too dangerous.

I was thinking about this some when I was at the park with Nora (3y) a
few days ago. She has gotten pretty interested in climbing lately, and
this time she climbed up the fence higher than I'd seen her go before.
If I'd known she'd climb this high I would have spotted her. She
called me over, very proud, and wanted me to take a picture so that
Julia could see too:

She asked me to carry her down, and I told her I was willing to give
her advice and spot her. She was willing to give this a try, but as
she started to go down some combination of being scared and the thin
wire of the fence being painful was too much, and she returned to the
thicker horizontal bars.

We tried this several times, with her getting increasingly upset.
After a bit Lily came over and tried to help, but was unsuccessful.
Eventually I put my hands on Nora's feet and with a mix of guiding and
(not ideal) supporting them helped her climb down to the lower bar.
She did the rest herself from there, something she's done many times.

This took about fifteen minutes and wasn't fun for any of us: Nora,
me, other people at the playground. But over the course of the rest of
the day I brought it up
several times, trying to get her to think it through before she
climbs higher than she would enjoy climbing down from.

(I think this is an approach that depends very heavily on the child's
judgment maturing sufficiently quickly relative to their physical
capabilities, and so is not going to be applicable to every family.
Lily and Anna were slower to climb and this was not an issue, while
Nora has pushed the edges of where this works much more.)
What is Morality?
by Zero Contradictions — 2024-07-29T19:19:57.119Z
Link: https://thewaywardaxolotl.blogspot.com/2020/07/what-is-morality.html

The PDF version can be read here.

The audio version can be listened to here:

Morality has the following components:

Collective values.
The individual internalization of collective values.
The assumption that collective values are objective, and thus “moral”.
A folk theory of morality.
Individual and collective moral myths.

To understand morality, you need to understand how these components fit together into a system. In this essay, I will describe each component and how it relates to the others.

(see the rest of the post in the link)