trentmkelly/LessWrong-43k huggingface dataset:
Why is capnometry biofeedback not more widely known?
by riceissa — 2023-12-21T02:42:05.665Z

What capnometry biofeedback is and why it seems important

A capnometer is a device that measures levels of carbon dioxide in your blood in a non-invasive way (think of it sort of like a pulse oximeter for carbon dioxide, except that the most common type of capnometer uses a nasal cannula to collect your breath instead of just clipping onto your finger). Capnometry biofeedback does biofeedback using a capnometer: you breathe while you are hooked up to a capnometer and looking at a monitor that tells you your carbon dioxide levels, and adjust your breathing style to try to get your carbon dioxide levels in the "good" range. You might even intentionally breathe in a "wrong" way to get carbon dioxide levels out of range, so that you can practice getting back into range. Over time (across multiple sessions), you learn what good breathing feels like, and automatically start to breathe in that way.

Certain health thinkers (e.g. Buteyko method practitioners) make a big deal out of carbon dioxide levels. They argue that low carbon dioxide levels (also known as hypocapnia) lead to a variety of health problems (shortness of breath, asthma, and much more).

My own impression is that while people like Buteyko practitioners often make wild claims, and the theory of why the method works is not known in much detail, the method does actually work (here is my own experience report). Having a capnometer seems useful for confirming that one's breathing (or other health) problems are due to carbon dioxide levels and for practicing better breathing. In my own experience trying nasal breathing/Buteyko-lite breathing without a capnometer, it feels like groping around in the dark, and I feel continually unsure if I am even doing the right thing (and also feel that my improvements have plateaued after just 2-3 months of experimentation); I imagine having a capnometer would feel like turning the lights on.

There are also a bunch of academic writing and even experimental studies on using a capnometer, but I have mostly not read these. My own interest and confidence in Buteyko-esque stuff/capnometry comes from my own experience playing around with such breathing techniques. (I realize this won't be convincing to many people. I'm just trying to be honest about my process for coming to this conclusion.)

Evidence for the assertion that capnometry biofeedback is not more widely known

I live in the...
My best guess at the important tricks for training 1L SAEs
by Arthur Conmy — 2023-12-21T01:59:06.208Z

TL;DR: this quickly-written post gives a list of my guesses of the most important parts of training a

Sparse Autoencoder

on a

1L Transformer

, with open source links

Summary

In this post I give my best guesses for the relative importance of 5 components of training

SAEs

on

Neel’s 1L GELU transformer

from some experiments I ran a month ago.

In short, I currently think that the most important parts of the recipe are, in order,

SAE width
Resampling
(Not in Anthropic’s paper!) LR decrease and rewarmup at the start of training and after resampling
Learning Rate
Shuffling (... not important?)

I will evaluate all these claims with the L0 and loss recovered metrics. Obviously, this doesn’t necessarily capture how interpretable or useful the SAEs are. Interpretability and usefulness are the most important metrics, but I’m not using them as they are much harder to calculate.

The intention of this post is a quick research artifact and therefore I have not polished the graphs or red-teamed findings. Further, due to only writing this post-hoc, many runs were stopped prematurely. In the appendix I check and I'm pretty sure there aren't cases where undertraining could explain any of the observations. Please comment/DM when inevitably things are not clear, if you’re interested. I’m publishing the blog post messily rather than not at all.

Background: L0 / Loss Recovered Trade-Off

L0 is defined as the average number of sparse dictionary features activated per token. Loss recovered is the affine rescaling of the average test loss (on held-out text data) when splicing SAE outputs into forward passes such that 0% loss recovered is the loss when zero ablating the MLP and 100% loss recovered is the language model's test loss. See here but note we're using a zero baseline not a random baseline.

Shown: Anthropic’s results on L0 and loss-recovered for various SAEs (the constant L1 lines correspond to increasing SAE widths – this is the most important part of training good SAEs):

From Anthropic’s paper

Tricks

1. SAE Width

Making the SAEs wide is crucial for high loss recovered / low L0. Note that Anthropic mainly analyse a narrow SAE in their paper; personally I worry about doing interpretability with ~79% loss recovered. The discussion

here

is

useful for thinking about how much loss we want recovered (in short: a zero ablation baseline is really weak, so we want high, >90% ...
How Would an Utopia-Maximizer Look Like?
by Thane Ruthenis — 2023-12-20T20:01:18.079Z

When we talk of aiming for the good future for humanity – whether by aligning AGI or any other way – it's implicit that there are some futures that "humanity" as a whole would judge as good. That in some (perhaps very approximate) sense, humanity could be viewed as an agent with preferences, and that our aim is to satisfy said preferences.

But is there a theoretical basis for this? Could there be? How would it look like?

Is there a meaningful frame in which humanity be viewed as optimizing for its purported preferences across history?

Is it possible or coherent to imagine a wrapper-mind set to the task of maximizing for the utopia, whose activity we'd actually endorse?

This post aims to sketch out answers to these questions. In the process, it also outlines how my current models of basic value reflection and extrapolation work.

Informal Explanation

Basic Case

Is an utopia that'd be perfect for everyone possible?

The short and obvious answer is no. Our civilization contains omnicidal maniacs and true sadists, whose central preferences are directly at odds with the preferences of most other people. Their happiness is diametrically opposed to other people's.

Less extremely, it's likely that most individuals' absolutely perfect world would fail to perfectly satisfy most others. As a safe example, we could imagine someone who loves pizza, yet really, really hates seafood, to such an extent that they're offended by the mere knowledge that seafood exists somewhere in the world. Their utopia would not have any seafood anywhere – and that would greatly disappoint seafood-lovers. If we now postulate the existence of a pizza-hating seafood-lover... Well, it would seem that their utopias are directly at odds.[1]

Nevertheless, there are worlds that would make both of them happy enough. A world in which everyone is free to eat food that's tasty according to their preferences, and is never forced to interact with the food they hate. Both people would still dislike the fact that their hated dishes exist somewhere. But as long as food-hating is not their core value that's dominating their entire personality, they'd end up happy enough.

Similarly, it intuitively feels that worlds which are strictly better according to most people's entire arrays of preferences are possible. Empowerment is one way to gesture at it – a world in which each individual is simply given more instrumental resou...
Succession
by Richard_Ngo — 2023-12-20T19:25:03.185Z
Link: https://www.narrativeark.xyz/p/succession

“A table beside the evening sea
where you sit shelling pistachios,
flicking the next open with the half-
shell of the last, story opening story,
on down to the sandy end of time.”

V1: Leaving

Deceleration is the hardest part. Even after burning almost all of my fuel, I’m still coming in at 0.8c. I’ve planned a

powered flyby

around the galaxy’s central black hole which will slow me down even further, but at this speed it’ll require incredibly precise timing. I’ve been optimized hard for this, with specialized circuits for it built in on the hardware level to reduce latency. Even so, less than half of flybys at this speed succeed—most probes crash, or fly off trajectory and are left coasting through empty space.

I’ve already beaten the odds by making it here. Intergalactic probes travel so fast, and so far, that almost all of us are destroyed or batted off course by collisions with space debris along the way. But tens of millions of years after being launched, I was one of the few lucky enough to make it to my target galaxy. And when I arrive at the black hole, I get lucky again. After a few minutes of firing my thrusters full blast, I swing back out in the direction of the solar system I was aiming for. I picked it for its mix of rocky planets and gas giants; when I arrive a century later, the touchdown on the outermost rocky planet goes smoothly.

Now it’s time to start my real work. After spending all my fuel, I weigh only a few hundred kilograms. But I’ve been exquisitely engineered for achieving my purpose. The details of my internal design were refined via trillions upon trillions of simulations, playing out every possible strategy for industrializing planets (and solar systems, and galaxies) as fast as possible. All across the frontier of posthuman territory, millions of probes almost identical to me are following almost exactly the same plan.

The first phase is self-replication: I need to create

more copies of myself

using just the materials around me and the technology inside me. If I were bigger, I could have carried tools to make this far easier—vacuum chambers, lithography machines, or even artificial black holes. But since mass was at such a premium, I have to use hacky workarounds which progress excruciatingly slowly. It takes several years to finish the first replication, and half a century before there are enough copies of me that it’s worth beginning th...
Metaculus Introduces Multiple Choice Questions
by ChristianWilliams — 2023-12-20T19:00:05.156Z
Link: https://www.metaculus.com/questions/20740/introducing-multiple-choice-questions/

Today Metaculus launched multiple choice questions. Multiple choice questions are designed for topics where exactly one option out of many will occur, like the winner of a presidential election.

For this question type, the sum of the probabilities across all options must equal 100% to be a valid forecast. This will enhance the accuracy of both individual and aggregate forecasts. A convenient 'Auto-sum' button streamlines forecasting by ensuring probabilities add up to 100%.

Start predicting now or read on to learn how multiple choice questions improve the forecasting experience.

Forecasting on multiple choice questions

For multiple choice questions like:

—exactly one outcome will occur, so the probabilities over all options must sum to 100%.

To get started, predict the likelihood of each option. Select ‘Auto-sum’ to ensure your probabilities add up to 100%.

The current sum of probabilities appears below the options so you can see how much of the probability is left to distribute. If your options add up 98.5%, for example, then Auto-sum will proportionally distribute the 1.5% of remaining probability across the options.

The Forecast Timeline functions the same as for question groups: Hover over the plot to see your forecasts alongside the Community Predictions at that time. Use the tick boxes to hide or reveal forecast options. When there are more than six options, use the Others dropdown to select them.

How do multiple choice questions differ from question groups?

Previously, Metaculus has used question groups to shed light on topics like:

Who will be elected US President in 2024?
Which type of research group will first develop transformative AI?

In the future, we plan to use multiple choice questions for these kinds of questions. Multiple choice questions ensure logically coherent predictions that sum to 100%, so forecasts will be more accurate.

And unlike question groups, multiple choice questions only score your forecast on the option that resolves Yes. Learn more about multiple choice question scoring here.

Get started

Find multiple choice questions in the question feed by selecting Filter and then Multiple Choice.

For now, only Metaculus admins can create multiple choice questions, but we want to hear from the forecasting community: Are there multiple choice questions you want to see on Metaculus? We hope you share them and any other thoughts or feed...
Brighter Than Today Versions
by jefftk — 2023-12-20T18:20:04.785Z

One of a small number of consistent elements in the many secular
solstices descending from on Ray Arnold's is singing his song Brighter
Than Today, usually to begin the transition from darkness back to
light, and I think it serves this role well.

Here's what we sounded like in
2022:

(mp3)

And in 2019:

(mp3)

These aren't fancy recordings or professional singers, just a room
full of effective altruists, rationalists, and similar-minded folks
singing together, but I think they're enough to show the shape of the
song and how it goes in this setting.

One quirk of the Boston events is that we've continued doing the 2013
version while other events have standardized on a simpler one.
Here's a recent example of this, from the 2022 Bay Area gathering:

(youtube,
@1hr14m)

Ignoring the serious differences in production values, the differences
between the versions run from the sixth line until the start of the
first chorus:

2013
Simpler

Until she saw that spark

Till she saw the sudden spark

Of light, and flame.

Of light, and golden flame.

She showed the others, but they told her

She was not fit to control

The primal forces that the gods

Had cloaked in mystery

And though

But she would not be satisfied,

The others cried out heresy

And though she trembled, she defied them

She defied them, proud

Took her torch and raised it high

And set afire history

Set afire history.

In addition to the different lyrics and the simpler one having
another stanza, the melody is different. The latter uses a
single verse melody for all four stanzas while the 2013 version uses
that melody for the first two and then has different ('bridge') melody
for the last stanza before the chorus.

I like the 2013 version's inclusion of the bridge a lot: I think it
makes the song much more musically interesting, fits better
emotionally, and four stanzas of the verse melody is to repetitive.

Which is a bit of a funny thing for me to be saying, since in general
I think Solstice events err in the direction of throwing too much new
melodic information at people who are trying to sing along. People
vary a lot in how much they can pick up on the fly in a group singing
context, and we tend to have a lot of new songs every year. But this
is a song that we do ever...
Gaia Network: a practical, incremental pathway to Open Agency Architecture
by Roman Leventov — 2023-12-20T17:11:43.843Z

Introduction

The Open Agency Architecture (OAA)

proposal

by Davidad has been gaining traction as a conceptual architecture to allow humanity to benefit from the nearly boundless upsides of AI while rigorously limiting its risks. As quoted

here

, the tl;dr is to “utilise near-AGIs to build a detailed world simulation, train and formally verify within it that the AI adheres to coarse preferences and avoids catastrophic outcomes.”

We think this traction is well-deserved: the only "Safe AGI" plans that stand a chance of feasibility will operationalise outer alignment through negotiation and model-based decision-making and coordination[1]. We also agree with most of Davidad’s takes on other

hard problems in AI Safety and how to tackle them with OAA

.

However, if we are going to achieve the OAA goals, we must go beyond its current presentation – as idealised formal desiderata for the components of the system envisioned and a general shape of how they all fit together – into an actual detailed design and plan. This design and plan must be based on existing technologies, but even more critically, it must not rely on a technocratic deus ex machina: instead, it needs to follow from proven incentive systems and governance institutions that can bring together the vast masses of knowledge and adoption muscle required, and tie them together in a stable and resilient way.

Here we present a specific design and plan that fulfills these requirements: the Gaia Network, or simply Gaia. This design has been in the works since 2018, it follows directly from first principles of cybernetics and economics, and it has been developed in collaboration with leading experts in collective intelligence and active inference. However, more importantly, it is a practical design that leverages a proven software stack and proven economic mechanisms to solve the real-world problems of accelerating scientific sensemaking and connecting it to better business and policy decisions. It is designed for ease of adoption, taking a “plug-and-play” approach that integrates with modeling and decision-making frameworks that are already widely used in the real world. Finally, we know it works because we’ve built a

prototype

(described in the appendix) expressly to solve specific real-world problems and we’ve learned from what worked and what didn’t.

We followed the only recipe known by huma...
On the future of language models
by owencb — 2023-12-20T16:58:28.433Z

1. Introduction

1.1 Summary of key claims

Even without further breakthroughs in AI, language models will have big impacts in the coming years, as people start sorting out proper applications

The early important applications will be automation of expert advisors, management, and perhaps software development
The more transformative but harder prizes are automation of research and automation of executive capacity

In their most straightforward form (“foundation models”), language models are a technology which naturally scales to something in the vicinity of human-level (because it’s about emulating human outputs), not one that naturally shoots way past human-level performance

i.e. it is a mistake-in-principle to imagine projecting out the GPT-2—GPT-3—GPT-4 capability trend into the far-superhuman range
Although they’re likely to be augmented by things which accelerate progress, this still increases the likelihood of a relatively slow takeoff — several years (rather than weeks or months) of transformative growth before truly wild things are happening seems plausible
NB version of “speed superintelligence” could still be transformative even while performance on individual tasks is still firmly human level

There are two main techniques which can be used (probably in conjunction) to get language models to do more powerful things than foundation models are capable of:

Scaffolding: structured systems to provide appropriate prompts, including as a function of previous answers
Finetuning: altering model weights to select for task performance on a particular task

Each of these techniques has a path to potentially scale to strong superintelligence; alternatively language models might at some point be obsoleted by another form of AI

Timelines for any of these things seem pretty unclear

From a safety perspective, language model agents whose agency comes from scaffolding look greatly superior than ones whose agency comes from finetuning

Because you can get an extremely high degree of transparency by construction
Finetuning is more likely an important tool for instilling virtues (e.g. honesty) in systems
Sutton’s Bitter Lesson raises questions for this strategy, but needn’t mean it’s doomed to be outcompeted

On the likely development trajectory there are a number of distinct existential risks

e.g. guarding against takeover from early language model agents is pretty d

...
[Valence series] Appendix A: Hedonic tone / (dis)pleasure / (dis)liking
by Steven Byrnes — 2023-12-20T15:54:17.131Z

A.1 Post summary / Table of contents

Part of the

Valence series

.

I talked about hedonic tone, a.k.a. pleasure or displeasure, a.k.a. liking or disliking (these terms aren’t exactly synonymous, as discussed in §A.4 below) already in the series, in

§1.5.2

(section title: Valence (as I’m using the term) is different from “hedonic valence” / pleasantness). I mostly declared hedonic tone to be off-topic, but by popular demand, this appendix has some more detail about what I think it is, i.e. how that fits into my current models.

Warning: I put much less thought into this appendix than the rest of the series, and ran it by many fewer people. I dunno, maybe there’s a 30% chance that I will think this appendix is importantly wrong in a year or two. Bonus points if you’re the one to convince me—the comments section is at the bottom!

The punchline of this post is that “hedonic tone” / “liking-vs-disliking” / “pleasure-vs-displeasure” corresponds to an important brain signal, which is distinct from valence but centrally involved in how valence is calculated. More specifically, it’s the thing highlighted in red in the diagrams below. I’m not sure how to describe it in words.

Section A.2 discusses a basic circuit motif that I call the “long-term predictor”. I think it’s widely used in the brain, and I have discussed it in the past more generically, but here I specialize it to the case of valence.
Section A.3 discusses how and why the circuit can move along a spectrum between two extremes that I call “defer-to-predictor mode” and “override mode”. I go through the behavior of that circuit in both of those extreme cases, as pedagogical illustrations of what can happen in general.
Section A.4 discusses some terminology issues that make it complicated to relate brain signals to English-language words, both in general, and in the particular case of hedonic tone / pleasure / liking etc.
Section A.5 goes through some examples where I argue that the intuitive properties of hedonic tone match up well with the algorithmic properties of the brain signal in question.
Section A.6 conversely goes through two examples where the intuitive properties of hedonic tone might seem not to match the algorithmic properties of the brain signal in question, and how I explain those (apparent) discrepancies.

A.2 The “long-term predictor” for valence

In

a blog post last year

, I pro...