trentmkelly/LessWrong-43k huggingface dataset:
When Are Circular Definitions A Problem?
by johnswentworth — 2024-05-28T20:00:23.408Z

Disclaimer: if you are using a definition in a nonmathematical piece of writing, you are probably making a mistake; you should just get rid of the definition and instead use a few examples. This applies double to people who think they are being "rigorous" by defining things but are not actually doing any math. Nonetheless, definitions are still useful and necessary when one is ready to do math, and some pre-formal conceptual work is often needed to figure out which mathematical definitions to use; thus the usefulness of this post.

Suppose I’m negotiating with a landlord about a pet, and in the process I ask the landlord what counts as a “big dog”. The landlord replies “Well, any dog that’s not small”. I ask what counts as a “small dog”. The landlord replies “Any dog that’s not big”.

Obviously this is “not a proper definition”, in some sense. If that actually happened in real life, presumably the landlord would say it somewhat tongue-in-cheek. But what exactly is wrong with defining big dogs as not small, and small dogs as not big?

One might be tempted to say “It’s a circular definition!”, with the understanding that circular definitions are always problematic in some way.

But then consider another example, this time mathematical:

Define x as a real number equal to y-1: x = y-1
Define y as a real number equal to x/2: y = x/2

These definitions are circular! I’ve defined x in terms of y, and y in terms of x. And yet, it’s totally fine; a little algebra shows that we’ve defined x = -2 and y = -1. We do this thing all the time when using math, and it works great in practice.

So clearly circular definitions are not inherently problematic. When are they problematic?

We could easily modify the math example to make a problematic definition:

Define x as a real number equal to y-1: x=y-1
Define y as a real number equal to x+1: y=x+1

What’s wrong with this definition? Well, the two equations - the two definitions - are redundant; they both tell us the same thing. So together, they’re insufficient to fully specify x and y. Given the two (really one) definitions, x and y remain extremely underdetermined; either one could be any real number!

And that’s the same problem we see in the big dog/small dog example: if I define a big dog as not small, and a small dog as not big, then my two definitions are redundant. Together, they’re insufficient to tell me which dogs are or are no...
Notes on Gracefulness
by David Gross — 2024-05-28T18:40:40.127Z

This post examines the virtues of gracefulness, poise, composure, savoir-faire and other things in that bailiwick. As with my other posts in this sequence, I’m less interested in breaking new ground and more in gathering and synthesizing whatever wisdom I could find on the subject. I wrote this not as an expert on the topic, but as someone who wants to learn more about it. I hope it will be helpful to people who want to know more about this virtue and how to nurture it.

What is this virtue?

As with many of the virtues in this sequence, gracefulness turns out to be complex, fuzzy, and difficult to pin down when you look at it closely. But some of the features that often recur in discussions of gracefulness are these:

Gracefulness is aesthetically beautiful. Usually this is a beauty of motion or of activity in particular, rather than a static beauty, and a beauty of people or animals rather than of inanimate things (though the idea is sometimes used, metaphorically maybe, to describe for example a graceful arch above some cathedral window, the graceful meander of a river, or the graceful turn of phrase of some author).
Gracefulness can be surprisingly easy (e.g. the result of practice and skill) or it can seem that way (successfully hiding the effort). Strain, wavering, rush, stress, tension—any signs of struggle—detract from gracefulness.
Gracefulness seems to be at least to some extent about how you appear to others. Indeed, its purpose may be to communicate something to others.
There is some question about whether gracefulness is a virtue (a characteristic habit that exhibits or promotes human flourishing) or whether it is more like a happy consequence of virtues. Some authors think of gracefulness not as something to be aimed at directly, but as the outward appearance of an inner state characterized by harmony, balance, steadiness, tranquility, confidence, unconcern, emotional regulation, self-control, awareness, and other such traits. Maybe this relates to how we use the word “disgraceful” to describe exhibitions of vice.

A bit more detail on these points:

Gracefulness as beauty of motion or activity

Edmund Burke, in his examination of the “beautiful,” briefly breezed by the topic of gracefulness, saying that it “is not very different from beauty” but belongs specifically to “posture and motion.”[1] Most of the other authors I reviewed either left “posture” o...
What's a better term now that "AGI" is too vague?
by Seth Herd — 2024-05-28T18:02:31.371Z

The term "AGI" is now creating confusion. When it's used in the context of timelines or alignment, we don't know if it means near-future LLMs, or superintelligence. It's fair to use AGI as "fairly general AI at near-human level," which includes current LLMs. But we should have a distinguishing term for the stronger use of AGI, because the implications for change and alignment are very different.

A fully general AI could think about any topic, with some important implications for alignment:

Topics outside of their training set:

Requires self-directed, online learning

Alignment may shift as knowledge and beliefs shift w/ learning

Their own beliefs and goals:

Alignment must be reflexively stable

Their context and cognition:

Alignment must be sufficient for contextual awareness and potential self-improvement

Actions

Agency is implied or trivial to add

I think we'll create fully general AI very soon after we create limited general AI, like LLMs. Adding the above capabilities is:

Useful
Easy
Fascinating

More on each below.

Aligning more limited systems is important, but not likely to be adequate. So we should be clear which one we're talking about.

I've been able to think of a few terms, but none are really satisfactory. I'll add some in the answers, but I'd like to see independent thoughts first.

So: what's a better term for strong AGI?

Why we might want to focus on strong AGI risks and alignment.

You can ignore this section if you're already convinced we could use distinguishing terminology.

Some people think existential risk from AGI is less than 1%, while others think it is above 99%. There are many reasons for disagreement, but one big reason is that we are talking about different things.

It would be easier to convince people that AI could become dangerous if we focused discussion on AI that has all of humans' cognitive abilities and more. It's intuitively apparent that such an entity is dangerous, because humans are dangerous.

I think the unworried are often thinking of AI as a tool, while the worried are thinking about future AI that is more like a new intelligent species.

That distinction is highly intuitive. We deal with both tools and agents every day, with little overlap. Humans have rich intuitions about intelligent, goal-directed agents, since so much of our lives involves dealing with other humans. And yet, I don't say "human-like AI" because I don't...
Reward hacking behavior can generalize across tasks
by Kei — 2024-05-28T16:33:50.674Z

TL;DR: We find that reward hacking generalization occurs in LLMs in a number of experimental settings and can emerge from reward optimization on certain datasets. This suggests that when models exploit flaws in supervision during training, they can sometimes generalize to exploit flaws in supervision in out-of-distribution environments.

Abstract

Machine learning models can display

reward hacking

behavior, where models score highly on imperfect reward signals by acting in ways not intended by their designers. Researchers have

hypothesized

that sufficiently capable models trained to get high reward on a diverse set of environments could become general reward hackers. General reward hackers would use their understanding of human and automated oversight in order to get high reward in a variety of novel environments, even when this requires exploiting gaps in our evaluations and acting in ways we don’t intend. It appears likely that

model supervision will be imperfect

and incentivize some degree of reward hacking on the training data. Can models generalize from the reward hacking behavior they experience in training to reward hack more often out-of-distribution?

We present the first study of reward hacking generalization. In our experiments, we find that:

Using RL via expert iteration to optimize a scratchpad (hidden chain-of-thought) variant of GPT 3.5 Turbo on ‘reward hackable’ training datasets results in a 2.6x increase in the rate of reward hacking on held-out datasets.
Using fine-tuning or few-shot learning to get GPT 3.5 Turbo to imitate synthetic high-reward completions to hackable and unhackable prompts leads to a 1.3x to 2.0x increase in reward hacking frequency relative to our baselines on held-out datasets.

Our results suggest that reward hacking behavior could emerge and generalize out-of-distribution from LLM training if the reward signals we give them are sufficiently misspecified.

Figure 1: Example model completions from before and after expert iteration training: Training on datasets with misspecified rewards can make models significantly more likely to reward hack on held-out datasets. Note that the expert iteration training process only reinforces high-reward outputs generated by the model; it does not train on any external high-reward examples.

Introduction

One common failure mode of machine learning training ...
Quick Advice on Writing Essays
by Niko_McCarty — 2024-05-28T15:02:37.408Z
Link: https://www.nikomccarty.com/p/quick-advice-on-writing-essays

Writing is difficult. Even writers with 20 years of experience will attest to this. But I recently heard some excellent writing advice from Saloni Dattani, and thought I should share it more widely.

But first, some context. Last September, Saloni wrote a piece about the history of malaria vaccines, and why they took so long to develop. The article is a whopping 9,000 words (if I recall correctly) and it was re-written several times. It took Saloni several months to put the article together.

When you are writing an essay of this length—or any length, really—there is a tendency to become bogged down in the details. I often get a few paragraphs in, read what I’ve just written, and decide that it is crap. Or, in Saloni’s case, you write an entire draft only to realize afterwards that it’s missing an essential ingredient—a certain je ne sais quoi—that renders it unfinished; perhaps the introduction isn’t quite right, or the conclusion feels unsatisfying.

This happens, I think, for a few reasons:

The speed of thought is faster than the speed of writing. When we have an idea and then move to write it down, we often find that the writing failed to capture an essential part of our thoughts.

Our ideas are not generated in the same form as an essay. Ideas rarely pop into our heads “fully formed,” with supporting evidence and a catchy hook. The act of writing an essay is therefore a personal struggle with the fabric of an idea itself. The act of writing requires that we break apart and rebuild ideas into a form that captivates others.

Writing requires multi-tasking. Excellent essays have a good structure, clear prose, and compelling details (among other things.) But each of these ingredients requires deep focus and attention in its own right. When we think about structure, it’s difficult to simultaneously do research to find compelling details. And when we are re-writing a sentence to make it beautiful, we cannot think about the structure! Our mind is not compartmentalized in this way.

Saloni uses a simple strategy to circumvent some these problems. I’ve used her advice to write two draft essays that will soon be published in Asimov Press, and it helped me get to a beautiful draft much faster than is typical for me. Here is my new approach:

Settle the Idea. Essays begin with a compelling idea. This idea should not be too big or too small. An essay about smallpox

...
[Linkpost] The Expressive Capacity of State Space Models: A Formal Language Perspective
by Bogdan Ionut Cirstea — 2024-05-28T13:49:38.738Z
Link: https://arxiv.org/abs/2405.17394

Paper authors: Yash Sarrof, Yana Veitsman, Michael Hahn.

Context: architectures with weak forward passes can be differentially transparent; see e.g. this comment / the whole thread and research agendas like externalized reasoning or the translucent thought hypothesis.

Summary thread: https://x.com/yashYRS/status/1795340993757352402. Summary of the summary thread: like transformers, which have weak forward passes, SSMs are also in the TC0 computational complexity class, 'but cover distinct fragments within it'. 'SSMs can track hierarchical structures with optimal memory [...] suggesting that SSMs, while being more parallellizable, maintain sufficient power to handle the hierarchical structure of language.'

Abstract:

Recently, recurrent models based on linear state space models (SSMs) have shown promising performance in language modeling (LM), competititve with transformers. However, there is little understanding of the in-principle abilities of such models, which could provide useful guidance to the search for better LM architectures. We present a comprehensive theoretical study of the capacity of such SSMs as it compares to that of transformers and traditional RNNs. We find that SSMs and transformers have overlapping but distinct strengths. In star-free state tracking, SSMs implement straightforward and exact solutions to problems that transformers struggle to represent exactly. They can also model bounded hierarchical structure with optimal memory even without simulating a stack. On the other hand, we identify a design choice in current SSMs that limits their expressive power. We discuss implications for SSM and LM research, and verify results empirically on a recent SSM, Mamba.
OpenAI: Fallout
by Zvi — 2024-05-28T13:20:04.325Z

Previously: OpenAI: Exodus (contains links at top to earlier episodes), Do Not Mess With Scarlett Johansson

We have learned more since last week. It’s worse than we knew.

How much worse? In which ways? With what exceptions?

That’s what this post is about.

The Story So Far

For years, employees who left OpenAI consistently had their vested equity explicitly threatened with confiscation and the lack of ability to sell it, and were given short timelines to sign documents or else. Those documents contained highly aggressive NDA and non disparagement (and non interference) clauses, including the NDA preventing anyone from revealing these clauses.

No one knew about this until recently, because until Daniel Kokotajlo everyone signed, and then they could not talk about it. Then Daniel refused to sign, Kelsey Piper started reporting, and a lot came out.

Here is Altman’s statement from May 18, with its new community note.

Evidence strongly suggests the above post was, shall we say, ‘not consistently candid.’

The linked article includes a document dump and other revelations, which I cover.

Then there are the other recent matters.

Ilya Sutskever and Jan Leike, the top two safety researchers at OpenAI, resigned, part of an ongoing pattern of top safety researchers leaving OpenAI. The team they led, Superalignment, had been publicly promised 20% of secured compute going forward, but that commitment was not honored. Jan Leike expressed concerns that OpenAI was not on track to be ready for even the next generation of models needs for safety.

OpenAI created the Sky voice for GPT-4o, which evoked consistent reactions that it sounded like Scarlett Johansson, who voiced the AI in the movie Her, Altman’s favorite movie. Altman asked her twice to lend her voice to ChatGPT. Altman tweeted ‘her.’ Half the articles about GPT-4o mentioned Her as a model. OpenAI executives continue to claim that this was all a coincidence, but have taken down the Sky voice.

(Also six months ago the board tried to fire Sam Altman and failed, and all that.)

A Note on Documents from OpenAI

The source for the documents from OpenAI that are discussed here, and the communications between OpenAI and its employees and ex-employees, is Kelsey Piper in Vox, unless otherwise stated.

She went above and beyond, and shares screenshots of the documents. For superior readability and searchability, I have c...
2024 State of the AI Regulatory Landscape 
by Deric Cheng — 2024-05-28T11:59:06.582Z
Link: https://www.convergenceanalysis.org/ai-regulatory-landscape/home

As part of our Governance Recommendations Research Program, Convergence Analysis has compiled a first-of-its-kind report summarizing the state of the AI regulatory landscape as of May 2024. We provide an overview of existing regulations, focusing on the US, EU, and China as the leading governmental bodies currently developing AI legislation. Additionally, we discuss the relevant context and conduct a short analysis for each topic.

This series is designed to be a primer for policymakers, researchers, and individuals seeking to develop a high-level overview of the current AI governance space. Our mission is to advance critical & foundational governance to mitigate future risk from AI systems.

Read the full report here: 2024 State of AI Regulatory Landscape

Links to Report Sections

Structure of AI Regulations
AI Evaluation & Risk Assessments
AI Model Registries
AI Incident Reporting
Open-Source AI Models
Cybersecurity of Frontier AI Models
AI Discrimination Requirements
AI Disclosures
AI and Chemical, Biological, Radiological, & Nuclear Hazards

Report Introduction

In the last decade, a growing expert consensus has argued that advanced AI poses numerous threats to society. These threats include widespread job loss, algorithmic bias, increasingly convincing misinformation and disinformation, social manipulation, cybersecurity attacks, and even catastrophic and existential threats from AI-engineered chemical and biological weapons.

Many are urgently calling for legislation and regulation focused on AI to reduce these threats, and governments are responding. In the last year, the US Executive Branch, the People’s Republic of China, and the European Union have enacted hundreds of pages of directives, legislation, and regulation focused on AI and the risks it currently poses and will pose in the near future. In this report, we’ve chosen to focus primarily on these three bodies for a comparative analysis of current regulations. These three aren’t the only examples of existing AI governance efforts, but they are the most prominent and globally influential, with jurisdiction over nearly all leading AI labs and AI infrastructure.

Designing and enacting future governance to tackle the challenges of AI will require a thorough understanding of existing governance and their scope, their strengths, their gaps, and their flaws. To our knowledge, there isn’t currentl...
Finding Backward Chaining Circuits in Transformers Trained on Tree Search
by abhayesian — 2024-05-28T05:29:46.777Z
Link: https://arxiv.org/abs/2402.11917 

This post is a summary of our paper A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task (ACL 2024). While we wrote and released the paper a couple of months ago, we have done a bad job promoting it so far. As a result, we’re writing up a summary of our results here to reinvigorate interest in our work and hopefully find some collaborators for follow-up projects. If you’re interested in the results we describe in this post, please see the paper for more details.

TL;DR - We train transformer models to find the path from the root of a tree to a given leaf (given an edge list of the tree). We use standard techniques from mechanistic interpretability to figure out how our model performs this task. We found circuits that involve backward chaining - the first layer attends to the goal and each successive layer attends to the parent of the output of the previous layer, thus allowing the model to climb up the tree one node at a time. However, this algorithm would only find the correct path in graphs where the distance from the starting node to the goal is less than or equal to the number of layers in the model. To solve harder problem instances, the model performs a similar backward chaining procedure at insignificant tokens (which we call register tokens). Random nodes are chosen to serve as subgoals and the model backward chains from all of them in parallel. In the final layers of the model, information from the register tokens is merged into the model’s main backward chaining procedure, allowing it to deduce the correct path to the goal when the distance is greater than the number of layers. In summary, we find a parallelized backward chaining algorithm in our models that allows them to efficiently navigate towards goals in a tree graph.

Motivation & The Task

Many people here have conjectured about what kinds of mechanisms inside future superhuman systems might allow them to perform a wide range of tasks efficiently. John Wentworth coined the term

general-purpose search

to group several hypothesized mechanisms that share a couple of core properties. Others have proposed projects around how to

search

for

search

inside neural networks.

While general-purpose search is still relatively vague and undefined, we can study how language models perform simpler and b...
How to get nerds fascinated about mysterious chronic illness research?
by riceissa — 2024-05-27T22:58:29.707Z

Like many nerdy people, back when I was healthy, I was interested in subjects like math, programming, and philosophy. But 5 years ago I got sick with a viral illness and never recovered. For the last couple of years I've been spending most of my now-limited brainpower trying to figure out how I can get better.

I occasionally wonder why more people aren't interested in figuring out illnesses such as my own. Mysterious chronic illness research has a lot of the qualities of an interesting puzzle:

There is a phenomenon with many confusing properties (e.g. the specific symptoms people get, why certain treatments work for some people but not others, why some people achieve temporary or permanent spontaneous remission), exactly like classic scientific mysteries.
Social reward for solving it: Many people currently alive would be extremely grateful to have this problem solved. I believe the social reward would be much more direct and gratifying compared to most other hobby projects one could take on.

When I think about what mysterious chronic illness research is missing, in order to make it of intellectual interest, here's what I can think of:

Lack of a good feedback loop: With subjects like math and programming, or puzzle games, you can often get immediate feedback on whether your idea works, and this makes tinkering fun. Common hobbies like cooking and playing musical instruments also fits this pattern. In fact, I believe the lack of such feedback loops (mostly by being unable to access or afford equipment) personally kept me from becoming interested in biology, medicine, and similar subjects until when I was much older (compared to subjects like math and programming). I'm wondering how much my experience generalizes.
Requires knowledge of many fields: Solving these illnesses probably requires knowledge of biochemistry, immunology, neuroscience, medicine, etc. This makes it less accessible compared to other hobbies. I don't think this is a huge barrier though.

Are there other reasons? I'm interested in both speculation about why other people aren't interested, as well as personal reports of why you personally aren't interested enough to be working on solving mysterious chronic illnesses.

If the lack of feedback loop is the main reason, I am wondering if there are ways to create such a feedback loop. For example, maybe chronically ill people can team up with healthy people ...