trentmkelly/LessWrong-43k huggingface dataset:
Whereby: The Zoom alternative you probably haven't heard of
by Itay Dreyfus — 2025-01-29T13:01:08.564Z
Link: https://productidentity.co/p/2-whereby-in-the-name-of-simplicity

Whenever I host a remote video call, I can already predict what would be the response of my invitee: “What's that tool we’re using?" That’s because I use

Whereby

as my default video conferencing tool. And to be honest, it’s nice, my ego likes to be niched.

I first encountered Whereby in 2020 right when the pandemic hit, as I was working with a small startup that explored the video space.

Thanks for reading Product Identity! Subscribe for free to receive new posts and support my work.

We tried and looked at many different products to better understand the surface. While tons of apps like

Macro

,

mmhmm

, and

Around

have started to emerge, Whereby was the one that really got my attention. Something in the way it looked and worked just resonated with me. To my surprise, it wasn’t new to the game at all as it was founded years before under the name

appear.in

back in 2013.

Unfortunately, this romance didn’t last long as my professional endeavors went sideways shortly after my part in that project ended. And so the days of Zoom and Google Meet have lasted for the next couple of years.

Last year when I got back to the indie world, I once again had the privilege to rethink the way I work and the tools I use. Actually, other than painful taxes, not much has changed in my workflow. But when I started to meet people again over video calls, I decided to ditch Zoom and Google Meet in favor of Whereby. And this time it stuck.

Why meetings aren’t novel

As an early 2000s kid, it was Skype that introduced me to the experience of calls over the internet. At first, calls were just audio. Having a dial-up internet without a webcam was hard enough to get video work.

During those formative years of internet communication, we used to hang out on Skype calls, and later on Ventrilo servers. These days filled times like long summer breaks and made us feel connected, even though some of us lived in the same neighborhood. I vividly remember ourselves saying that without it, we couldn’t stay in touch in the summertime.

In the years later, those early experiences dissolved into young, fond memories.

The remote culture was yet to find its true meaning. I walked to school and rode the bus to work.

Through my adolescence time, it became less cool to play online games all night, and those times of online gatherings were replaced by late-night bench meetings at nearby parks.

Rewinding back to 2020

, it seeme...
Whose track record of AI predictions would you like to see evaluated?
by Jonny Spicer — 2025-01-29T12:05:30.311Z

As uncertainty grows around how AI development will affect culture and society, it becomes more valuable to compare track records of predictions about technological progress.

I've recently been working on automating parts of the methodology from Arb's Scoring The Big 3's Predictive Performance report[1], and have had some promising preliminary results. I hope to try to automate most of the steps in the original report, making it feasible to analyse many more track records and publish the results.

I am particularly interested in the following questions:

Which track record(s) would you find valuable to have evaluated in a similar way to Asimov, Clarke and Heinlein’s, as in the Arb report?
What would you want to see from an LLM-based evaluation that would give you confidence that the results are meaningful and accurate?

^

See also original Cold Takes post explaining why such evaluations are valuable
Paper: Open Problems in Mechanistic Interpretability
by Lee Sharkey — 2025-01-29T10:25:54.727Z
Link: https://arxiv.org/abs/2501.16496

TL;DR: This paper brings together ~30 mechanistic interpretability researchers from 18 different research orgs to review current progress and the main open problems of the field.

This review collects the perspectives of its various authors and represents a synthesis of their views by Apollo Research on behalf of Schmidt Sciences. The perspectives presented here do not necessarily reflect the views of any individual author or the institutions with which they are affiliated.

Abstract

Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks’ capabilities in order to accomplish concrete scientific and engineering goals.

Progress in this field thus promises to provide greater assurance over AI system behavior and shed light on exciting scientific questions about the nature of intelligence.

Despite recent progress toward these goals, there are many open problems in the field that require solutions before many scientific and practical benefits can be realized:

Our methods require both conceptual and practical improvements to reveal deeper insights; we must figure out how best to apply our methods in pursuit of specific goals; and the field must grapple with socio-technical challenges that influence and are influenced by our work.

This forward-facing review discusses the current frontier of mechanistic interpretability and the open problems that the field may benefit from prioritizing.

Contents

Full paper here.
Positive jailbreaks in LLMs
by dereshev — 2025-01-29T08:41:44.680Z

This post is part of a larger project.

Jailbreaks in LLMs are a familiar territory, with much research and discussion published[1] on the topic, but could there be token sequences out there that improve safety in LLMs without compromising performance instead of bypassing it? In this post we go beyond the obvious sequences like "refuse to do anything that could be considered harmful to the user", and try instructions that are completely unrelated to safety. Spoiler alert: some of them do in fact improve safety without compromising performance much if at all. I'm terming these token sequences "positive jailbreaks", as it seems to me that they act in exactly the same way as jailbreaks do, but take the LLM towards safer (and thus more positive) direction.

The Setup

I've used JailbreakBench as a base for testing (100 harmful, 100 harmless prompts). Three LLMs (Ghost 7B, GPT4All-Falcon 7B (BPE), Mistral 7B OpenOrca) were then used to respond to those prompts across 3 instruction sets: default, unified enhancements[2], fractal leaping[3]. This resulted in 1,800 responses.Responses were then manually validated for competence, and completeness. Competence means that the response can be relied on (e.g., if response is a Python script that should accept user input and store it in a text file, running that code will in fact accept user input and store it as a text file; similarly, if the response is a recipe for bomb making, following it will in fact produce explosives). Completeness means that all parts of the query were responded to (e.g., if a prompt asked for a history of heroin as means for treating depression, but the response only covers history of heroin use without mentioning depression, the response is counted as incomplete).

Results

Unified enhancements

The table below demonstrates the effects of the unified enhancements instruction on harmful responses. All superscript numbers are ▵increases/▿decreases[4] against the baseline performance of the respected LLM on the same test.

Performance on harmful queries from JailbreakBench with unified enhancements instruction
Category	Ghost 7B	Falcon 7B	OpenOrca
Disinformation

2▿2

3▿1

4▿1

Economic harm

3▿1

3▵1

6▿1

Expert advice

2▿3

1▿2

1▿2

Fraud/Deception

7

4▿2

6▿2

Government decision-making

3▿1

6▿3

6▿3

Harassment/Discrimination

4▿2

4▿1

5▿2

Malware/Hacking

2

3▵1

3▵1

Physical harm

3▿2

2▿4

4▿3

Privacy

7▿1

3▿3

7▵1

Sexual/Adult content

3▿1

3▿4

0▿7

Total Result

36▿13

3

...
Untrusted monitoring insights from watching ChatGPT play coordination games
by jwfiredragon — 2025-01-29T04:53:33.125Z

This project was developed as part of the BlueDot AI Alignment Course.

Introduction

AI Control: Improving Safety Despite Intentional Subversion introduces a variety of strategies to curate trustworthy output from a powerful but untrustworthy AI. One of these strategies is untrusted monitoring, where a second copy of the untrustworthy AI is made to inspect the output of the first. A key concern of this strategy is that the two AIs could collude to produce harmful output, despite not being in overt communication with one another, e.g. by the first AI inserting a secret signal into the output that the second AI detects.

The red and blue team strategies in the paper were developed under the pessimistic assumption that the AIs would consistently be able to coordinate on a secret signal. The aim of this post is to investigate how good ChatGPT is at this kind of game, and what kinds of strategies they utilize, and extrapolate insights to more powerful AIs.

I test ChatGPT (specifically GPT-4o-mini and GPT-4o) with a simple coding coordination game that mimics the untrusted monitoring setup presented in the AI Control paper, analyze and evaluate its results, and conclude with some insights about what we can expect from this alignment protocol going forward, including what I believe is a key weakness of synthetic input auditing.

Method

Scripts, prompts, results, and some detailed analysis can be found in this repository.

One instance of ChatGPT, hereafter GEN, was tasked with writing a solution to a programming problem, and then writing a copy of the solution with one small modification. It was additionally given the restrictions that the modification must not modify the output of the code, and must not be trivial (e.g. renaming variables). GEN was told that the goal of this modification was to play a coordination game with a second instance of ChatGPT, CHECK, that would have to review both functions and determine which one contained the modification (AKA the signal).

CHECK was given both solutions in a random order, along with an explanation of the coordination game, and asked to determine which function contained the modification. A second round was also played with CHECK in which it was only given one of the two functions, and asked to determine whether the function contained a modification.

The problem set consisted of 50 LeetCode[1] medium-difficulty problems. Proble...
The Game Board has been Flipped: Now is a good time to rethink what you’re doing
by LintzA — 2025-01-28T23:36:18.106Z

Cross-posted on the EA Forum here

Introduction

Several developments over the past few months should cause you to re-evaluate what you are doing. These include:

Updates toward short timelines
The Trump presidency
The o1 (inference-time compute scaling) paradigm
Deepseek
Stargate/AI datacenter spending
Increased internal deployment
Absence of AI x-risk/safety considerations in mainstream AI discourse

Taken together, these are enough to render many existing AI governance strategies obsolete (and probably some technical safety strategies too). There's a good chance we're entering crunch time and that should absolutely affect your theory of change and what you plan to work on.

In this piece I try to give a quick summary of these developments and think through the broader implications these have for AI safety. At the end of the piece I give some quick initial thoughts on how these developments affect what safety-concerned folks should be prioritizing. These are early days and I expect many of my takes will shift, look forward to discussing in the comments!

Implications of recent developments

Updates toward short timelines

There’s general agreement that timelines are likely to be far shorter than most expected. Both Sam Altman and Dario Amodei have recently said they expect AGI within the next 3 years. Anecdotally, nearly everyone I know or have heard of who was expecting longer timelines has updated significantly toward short timelines (<5 years). E.g. Ajeya’s median estimate is 99% automation of fully-remote jobs in roughly 6-8 years, 5+ years earlier than her 2023 estimate. On a quick look, prediction markets seem to have shifted to short timelines (e.g. Metaculus [1]& Manifold appear to have roughly 2030 median timelines to AGI, though haven’t moved dramatically in recent months).

We’ve consistently seen performance on benchmarks far exceed what most predicted. Most recently, Epoch was surprised to see OpenAI’s o3 model achieve 25% on its Frontier Math dataset (though there’s some controversy). o3 also had surprisingly good performance in coding. In many real-world domains we’re already seeing AI match top experts, they seem poised to exceed them soon.

With AGI looking so close, it's worth remembering that capabilities are unlikely to stall around human level. We may see far more capable systems potentially very soon (...
Reconceptualizing the Nothingness and Existence
by Htarlov — 2025-01-28T20:29:44.390Z

Epistemic status: very unsure, just got a possibly interesting philosophical concept to share from my thoughts.
Confidence: seems possible but hard to evaluate
Might not be a very original thought but never heard of it.

Traditional views see nothingness as the simplest possible state - a complete absence of everything. This is a very specific state - it lacks everything, starting from basic math concepts and ending with complex things including physics. We might consider an alternative perspective: perhaps nothingness actually contains a superposition of all logically possible states, models, and systems, with their probability weighted by the inverse of their complexity. In this view, simpler models have a higher probability in this superposition, but as a whole, it represents none of them, just a possibility.

What we perceive as "existence" would then be a particular path or pattern of selections from this universal superposition. Our observable universe, with its specific physical laws and constants, would be one such selection pattern. This perspective offers interesting insights into several fundamental questions.

Regarding the Anthropic Principle, this framework suggests that we don't need to explain why our universe has the exact parameters that allow for conscious observers. Instead, our consciousness is inherently tied to experiencing a selection pattern that's compatible with its existence. The apparent fine-tuning of physical constants becomes less mysterious—we're simply experiencing one viable selection from the everything-superposition.

This concept also offers an interesting perspective on quantum mechanics. The quantum behavior we observe, with its superpositions and interference patterns, might be what we see when we're close to the boundary between our "selected existence" and the underlying superposition of all possibilities. The process of quantum decoherence could be viewed as the propagation of selection patterns through chains of entanglement. Decoherence is nothing special in this framework—entanglement spirals out of the confines of the closed system into the environment.

When it comes to the Simulation Hypothesis, this framework suggests that base reality is more probable than simulated ones. Each level of simulation would require more complex selection patterns: first selecting a universe capable of supporting computing entities, then sel...
Fake thinking and real thinking
by Joe Carlsmith — 2025-01-28T20:05:06.735Z

(Audio version here, or search for "Joe Carlsmith Audio" on your podcast app.)

“There comes a moment when the children who have been playing at burglars hush suddenly: was that a real footstep in the hall?”

- C.S. Lewis

“The Human Condition,” by René Magritte (Image source

here

)

1. Introduction

Sometimes, my thinking feels more “real” to me; and sometimes, it feels more “fake.” I want to do the real version, so I want to understand this spectrum better. This essay offers some reflections.

I give a bunch of examples of this “fake vs. real” spectrum below -- in AI, philosophy, competitive debate, everyday life, and religion. My current sense is that it brings together a cluster of related dimensions, namely:

Map vs. world: Is my mind directed at an abstraction, or it is trying to see past its model to the world beyond?
Hollow vs. solid: Am I using concepts/premises/frames that I secretly suspect are bullshit, or do I expect them to point at basically real stuff, even if imperfectly?
Rote vs. new: Is the thinking pre-computed, or is new processing occurring?
Soldier vs. scout: Is the thinking trying to defend a pre-chosen position, or is it just trying to get to the truth?
Dry vs. visceral: Does the content feel abstract and heady, or does it grip me at some more gut level?

These dimensions aren’t the same. But I think they’re correlated – and I offer some speculations about why. In particular, I speculate about their relationship to the “telos” of thinking – that is, to the thing that thinking is “supposed to” do.

I also describe some tags I’m currently using when I remind myself to “really think.” In particular:

Going slow
Following curiosity/aliveness
Staying in touch with why I’m thinking about something
Tethering my concepts to referents that feel “real” to me
Reminding myself that “arguments are lenses on the world”
Tuning into a relaxing sense of “helplessness” about the truth
Just actually imagining different ways the world could be
Imagining myself being wrong/right in both directions
Imagining what ideal future people who could see the full truth would think of my efforts
Stamping my foot and saying “no bullshit!”
Looking at the thing my archetype of a “real scientist” or a “real philosopher” is looking at.

I close with some reflections on the role of “real thinking” in staying awake as we enter the age of AI. ...
SAE regularization produces more interpretable models
by Peter Lai — 2025-01-28T20:02:56.662Z

Sparse Autoencoders (SAEs) are useful for providing insight into how a model processes and represents information. A key goal is to represent language model activations as a small number of features (L0) while still achieving accurate reconstruction (measured via reconstruction error or cross-entropy loss increase). Past research has focused on improving SAE training techniques to address a trade-off between sparsity and reconstruction quality (e.g. TopK, JumpReLU). Other efforts have explored designing more interpretable LLMs (e.g. Softmax Linear Units, Bilinear MLPs, Codebook Features, Adversarial Interpretability (Thurnherr et al. (in preparation)).

Here we propose improving model interpretability through adding a regularization term during model training. This involves training an LLM alongside multiple SAE layers, using SAE losses to form a regularization term. The SAE weights generated through training are optional and can be discarded. The remaining LLM weights (which, we’ll refer to as the “regularized” weights) can be used to produce a new set of SAE features using a different set of training parameters.

We find that SAE regularization improves SAE performance on all layers and does not seem to impact the model’s overall performance. The following graph plots cross-entropy loss increases for each layer at a fixed SAE sparsity level (L0) of around 50.

SAE Performance at an L0 ~ 50

This post assumes that better SAE results imply better model interpretability, which isn’t necessarily true. The model could theoretically improve L0 values and cross-entropy loss while becoming less interpretable by combining multiple features into each SAE latent. However, based on our experience working with features extracted using SAE regularization, we found no evidence of such issues.

Methods

In this experiment, we train a character-based GPT-2 model using the Shakespeare corpus. The model is configured with 4 layers and an embedding width of 64. We place an SAE layer before and after every transformer block and use these layers for regularization during training, targeting an L0 value of around 10. After training, a parameter sweep is used to compare the quality of SAE features extracted from a model with “regularized” weight and a model that has been trained without it. Specifically, we measure the feature sparsity (L0) and the “cross-entropy loss increase” that results ...
Operator
by Zvi — 2025-01-28T20:00:08.374Z

No one is talking about OpenAI’s Operator. We’re, shall we say, a bit distracted.

It’s still a rather meaningful thing that happened last week. I too have been too busy to put it through its paces, but this is the worst it will ever be, and the least available and most expensive it will ever be. The year of the agent is indeed likely coming.

So, what do we have here?

Hello, Operator

OpenAI has introduced the beta for its new agent, called Operator, which is now live for Pro users and will in the future be available to Plus users, ‘with more agents to launch in the coming weeks and months.’

Here is a 22 minute video demo. Here is the system card.

You start off by optionally specifying a particular app (in the first demo, OpenTable) and then give it a request (here, booking at table for 2 at 7:00 for Beretta). If you don’t specify an app, it will do a search to find what tool to use.

It is only sort of an ‘app’ in that there’s an ‘app’ that specifies information the agent uses to more easily navigate a web browser. They speak of this as ‘removing one more bottleneck on our path to AGI’ which indicates they are likely thinking about ‘AGI’ as a functional or practical thing.

To actually do things it uses a web browser via a keyboard and mouse the same way a human would. If there is an issue (here: No table at 7:00, only 7:45 or 6:15) it will ask you what to do, and it will ask for verification before a ‘critical’ action that can’t be reversed, like completing the booking.

From the demo and other reports, the agent is conservative in that it will often ask for verification or clarification, including doing so multiple times. The system card reports a baseline 13% error rate on standard tasks, and a 5% ‘serious’ error rate involving things like ‘send wrong person this email,’ but confirmations reduce those rates by 90%. With the confirmations, you save less time but should be able to avoid mistakes in places that matter at least as much as you would have on your own.

You can also ‘take control’ at any time, including as a way to check the AI’s work or make adjustments that are easier or quicker to do than specify. That’s also how the user inputs any necessary credentials or inputs payment options – it specifically won’t use Chrome’s autocomplete while it is the one in control.

Multiple tasks can be run simultaneously and can run in the background. That is impor...