trentmkelly/LessWrong-43k huggingface dataset:
Predicting Virus Relative Abundance in Wastewater
by jefftk — 2023-08-10T15:46:55.554Z
Link: https://naobservatory.org/reports/predicting-virus-relative-abundance-in-wastewater/

At the Nucleic Acid Observatory (NAO) we're evaluating pathogen-agnostic surveillance. A key question is whether metagenomic sequencing of wastewater can be a cost-effective method to detect and mitigate future pandemics. In this report we investigate one piece of this question: at a given stage of a viral pandemic, what fraction of wastewater metagenomic sequencing reads would that virus represent?

To make this concrete, we define RA(1%). If 1% of people are infected with some virus (prevalence) or have become infected with it during a given week (incidence), RA(1%) is the fraction of sequencing reads (relative abundance) generated by a given method that would match that virus. To estimate RA(1%) we collected public health data on sixteen human-infecting viruses, re-analyzed sequencing data from four municipal wastewater metagenomic studies, and linked them with a hierarchical Bayesian model.

Three of the viruses were not present in the sequencing data, and we could only generate an upper bound on RA(1%). Four viruses had a handful of reads, for which we were able to generate rough estimates. For the remaining nine viruses we were able to narrow down RA(1%) for a specific virus-method combination to approximately an order of magnitude. We found RA(1%) for these nine viruses varied dramatically, over approximately six orders of magnitude. It also varied by study, with some viruses seeing an RA(1%) three orders of magnitude higher in one study than another.

The NAO plans to use the estimates from this study as inputs into a modeling framework to assess the cost effectiveness of wastewater MGS detection under different pandemic scenarios, and we include an outline of such a framework with some rough estimates of the costs of different monitoring approaches.

Read the full report: Predicting Virus Relative Abundance in Wastewater.
AI #24: Week of the Podcast
by Zvi — 2023-08-10T15:00:04.438Z

In addition to all the written developments, this was a banner week for podcasts.

I would highlight four to consider listening to.

Dario Amodei of Anthropic went on The Lunar Society to talk to Dwarkesh Patel. We got our best insight so far into where Dario’s head is at, Dwarkesh is excellent at getting people to open up like this and really dive into details.

Jan Leike, OpenAI’s head of alignment, went on 80,000 hours with Robert Wiblin. If you want to know what is up with the whole superalignment effort, this was pretty great, and left me more optimistic. I still don’t think the alignment plan will work, but there’s a ton of great understanding of the problems ahead and an invitation to criticism, and a clear intention to avoid active harm, so we can hope for a pivot as they learn more.

Tyler Cowen interviewed Paul Graham. This was mostly not about AI, but fascinating throughout, often as a clash of perspectives about the best ways to cultivate talent. Includes Tyler Cowen asking Paul Graham about how to raise someone’s ambition, and Paul responding by insisting on raising Tyler’s ambition.

I got a chance to go on EconTalk and speak with Russ Roberts about The Dial of Progress and other matters, mostly related to AI. I listen to EconTalk, so this was a pretty special moment. Of course, I am a little bit biased on this one.

Capabilities continue to advance at a more modest pace, so I continue to have room to breathe, which I intend to enjoy while it lasts.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Proceed with caution.

Language Models Don’t Offer Mundane Utility. Not with these attitudes.

GPT-4 Real This Time. Time for some minor upgrades.

Fun With Image Generation. Some fun, also some not so fun.

Deepfaketown and Botpocalypse Soon. They keep ignoring previous instructions.

They Took Our Jobs. People really, really do not like it when you use AI artwork.

Introducing. Real time transcription for the deaf, also not only for the deaf.

In Other AI News. Various announcements, and an exciting Anthropic paper.

There Seems To Be a Standard Issue RLHF Morality. It has stages. What’s next?

Quiet Speculations. Cases for and against expecting a lot of progress.

The Quest for Sane Regulation. Confidence building, polls show no confidence.

The Week in Audio.

...
Could We Automate AI Alignment Research?
by Stephen McAleese — 2023-08-10T12:17:05.194Z

Summary

Creating an AI that can do AI alignment research (an automated alignment researcher) is one part of OpenAI’s three-part alignment

plan

and a core goal of their Superalignment

team

.
The automated alignment researcher will probably involve an advanced language model more capable than GPT-4 and possibly with human-level intelligence. Current alignment approaches such as recursive reward modeling could be used to align the automated alignment researcher.
A key problem with creating an automated alignment researcher is a chicken-and-egg problem where alignment is needed before the AI can be safely created and the AI is needed to create the alignment solution. One solution to this problem is bootstrapping involving an initial alignment solution and iterative improvements to the AI’s alignment and capabilities.
I argue that the success of the plan depends on aligning the automated alignment researcher being easier than aligning AGI.
Automating alignment research could be a form of differential technological development and could be a Pivotal Act.
Creating a human-level automated alignment researcher seems risky because it is a form of AGI and therefore has similar risks.
Related ideas include Cyborgism and creating whole-brain emulations of AI alignment researchers before creating AGI.
Key cruxes: the level of capabilities needed to solve alignment and the difficulty of aligning an AI with the goal of solving alignment compared to other more open-ended goals.

Introduction

OpenAI’s alignment

plan

has three pillars: improving methods for aligning models using human feedback (e.g. RLHF), training models to assist human evaluation (e.g. recursive reward modeling), and training AI systems to do alignment research.

I recently listened to the AXRP podcast episode on Jan Leike and the OpenAI superalignment team which inspired me to write this post. My goal is to explore and evaluate OpenAI’s third approach to solving alignment: creating an AI model that can do AI alignment research or automated AI alignment researcher to solve the alignment problem. I’m interested in exploring the approach because it seems to be the newest and most speculative part of their plan and my goal in this post is to try and explain how it could work and the key challenges that would need to be solved to make it work.

At first, the idea may sound unreasonable because there’s an obvious ...
The positional embedding matrix and previous-token heads: how do they actually work?
by AdamYedidia — 2023-08-10T01:58:59.319Z

tl;dr: This post starts with a mystery about positional embeddings in GPT2-small, and from there explains how they relate to previous-token heads, i.e. attention heads whose role is to attend to the previous token. I tried to make the post relatively accessible even if you're not already very familiar with concepts from LLM interpretability.

Introduction: A mystery about previous-token heads

In the context of transformer language models, a previous-token head is an attention head that attends primarily or exclusively to the previous token. (Attention heads are the parts of the network that determine, at each token position, which previous token position to read information from. They're the way transformers move information from one part of the prompt to another, and in GPT2-small there are 12 of them at each of the 12 layers, for a total of 144 heads.)

Previous-token heads are performing a really basic function in a transformer model: for each token, figuring out what the token that precedes it is, and copying information from that preceding token to the token after it. This is key piece of almost any text-based task you can imagine—it's hard to read a sentence if you can't tell which words in the sentence come before which other words, or which sub-words in a word come first. But how do these previous-token heads actually work? How do they know which token comes previously?

The easy answer to this is "positional embeddings"—for each possible token position in the prompt, at that token position, there are certain directions added to the residual stream[1] that encode the "meaning" of that token position in vector space. This is a confusing concept—basically there's a vector that means "this is the first token in the prompt" and another vector that means "this is the second token in the prompt" and so on. In GPT2-small, these vectors are learned, there are 1024 of them, and they form a helix.

Above: the positional embeddings of GPT2-small, scatter-plotted along its most important three PCA directions.

So if the network wanted to form a previous-token head, it could, for a given token position, look up which these 1024 vectors corresponds to the current token position, figure out what the previous token position would be, and attend to the token with positional embedding corresponding to the previous token position. At least in principle.

In practice, tho...
LLMs are (mostly) not helped by filler tokens
by Kshitij Sachan — 2023-08-10T00:48:50.510Z

This work was done at Redwood Research. The views expressed are my own and do not necessarily reflect the views of the organization.

Thanks to Ryan Greenblatt, Fabien Roger, and Jenny Nitishinskaya for running some of the initial experiments and to Gabe Wu and Max Nadeau for revising this post.

I conducted experiments to see if language models could use 'filler tokens'—unrelated text output before the final answer—for additional computation. For instance, I tested whether asking a model to produce "a b c d e" before solving a math problem improves performance.

This mini-study was inspired by Tamera Lanham, who ran

similar experiments on Claude

and found negative results. Similarly, I find that GPT-3, GPT-3.5, and Claude 2 don’t benefit from filler tokens. However, GPT-4 (which Tamera didn’t study) shows mixed results with strong improvements on some tasks and no improvement on others. These results are not very polished, but I’ve been sitting on them for a while and I thought it was better to publish than not.

Motivation

Language models (LMs) perform better when they produce step-by-step reasoning, known as “chain of thought”, before answering. A nice property of chain of thought is that it externalizes the LM’s thought process, making it legible to us humans. Some researchers hope that if a LM makes decisions based on undesirable factors, e.g. the political leanings of its supervisors (Perez 2022) or whether or not its outputs will be closely evaluated by a human, then this will be visible in its chain of thought, allowing supervisors to catch or penalize such behavior.

If, however, we find that filler tokens alone improve performance, this would suggest that LMs are deriving performance benefits from chain-of-thought prompting via mechanisms other than the human-understandable reasoning steps displayed in the words they output. In particular, filler tokens may still provide performance benefits by giving the model more forward passes to operate on the details provided in the question, akin to having “more time to think” before outputting an answer.[1]

Why Might Filler Tokens Be Useful?

This is an abstracted drawing of a unidirectional (i.e. decoder-only) transformer. Each circle represents the residual stream at a given token position after a given layer. The arrows depict how attention passes information between token positions. Note t...
Progress links digest, 2023-08-09: US adds new nuclear, Katalin Karikó interview, and more
by jasoncrawford — 2023-08-09T19:22:03.361Z
Link: https://rootsofprogress.org/links-digest-2023-08-09

Opportunities

New Limit (longevity tech) is hiring for data scientists in computational biology (via @brian_armstrong)
Anton Howes has “an idea for a paper that ought to radically revise the currently prevailing Crafts estimates for the impact of steam engines on British GDP pre-1800…. if this sounds like a paper you’d like to work on, then DM me”
Entrepreneur First is hiring a US Chief of Staff (via @matthewclifford)

News & announcements

Georgia Power declared Plant Vogtle Unit 3 commercially operational on July 31, making it the first “newly constructed” nuclear power unit to be added to the US fleet in decades (via @Sonalcpatel)
Wendy’s announced they will be the first to launch Pipedream’s “Instant Pickup”
A Japanese cyclotron has created sodium-39, an extremely unstable isotope
UK privacy bill poses serious threat to encrypted communications?
A previous announcement quoted @ylecun saying that “Llama-v2 is open source.” Zac Hatfield-Dodds claims this is wrong: “Meta is only offering a limited commercial license which discriminates against specific users and bans many valuable use-cases, both economic and in alignment research.” Zac works for Anthropic, although I assume this opinion is his own.

Other links

First long-form podcast in English with Katalin Karikó, one of the inventors of mRNA technology (via @JosephNWalker)
Jack Devanney proposes a new radiation harm model, “Sigmoid No Threshold” (SNT), to replace Linear No Threshold (LNT). See my brief comments on SNT
“Just like the microchip brought the marginal cost of compute to zero, and the Internet brought the marginal cost of distribution to zero, generative AI promises to bring the marginal cost of creation to zero”
“The most important policy problem is that people want unlimited access to medical services without having to pay for them. … Americans make extravagant use of procedures with high costs and low benefits”
A war story from early PayPal days from Max Levchin
How to write a blog & keep it going (by @NikoMcCarty)
The worst US bridges are getting fixed

Superconductor update

If it worked, could a superconducting induction coil make the perfect battery? And would it be world changing?
Casey Handmer, a week ago: “I have a PhD in theoretical physics, studied BCS theory… I still have no strong intuition on LK99.” Since then—
Something has been replicated with LK-99. You can follow a formula, make some sort of

...
Mech Interp Challenge: August - Deciphering the First Unique Character Model
by CallumMcDougall — 2023-08-09T19:14:23.682Z

I'm writing this post to advertise the second in the sequence of monthly mechanistic interpretability challenges (the first to be posted on LessWrong). They are designed in the spirit of Stephen Casper's challenges, but with the more specific aim of working well in the context of the rest of the ARENA material, and helping people put into practice all the things they've learned so far.

In this post, I'll describe the algorithmic problem & model trained to solve it, as well as the logistics for this challenge and the motivation for creating it. However, the main place you should access this problem is on the Streamlit page, where you can find setup code & instructions (as well as a link to a Colab notebook with all setup code included, if you'd prefer to use this).

Task

The algorithmic task is as follows: the model is presented with a sequence of characters, and for each character it has to correctly identify the first character in the sequence (up to and including the current character) which is unique up to that point.

Explanation: "a" is unique in both the first two characters "ab", it's repeated in the 3rd character so "b" is now the first unique character, and then "b" is repeated in the 5th character which makes "c" the first unique character.

Model

Our model was trained by minimising cross-entropy loss between its predictions and the true labels, at every sequence position simultaneously (including the zeroth sequence position, which is trivial because the input and target are both always "?"). You can inspect the notebook training_model.ipynb in the GitHub repo to see how it was trained. I used the version of the model which achieved highest accuracy over 50 epochs (accuracy ~99%).

The model is is a 2-layer transformer with 3 attention heads, and causal attention. It includes layernorm, but no MLP layers.

Note - although this model was trained for long enough to get loss close to zero (you can test this for yourself), it's not perfect. There are some weaknesses that the model has which might make it vulnerable to adversarial examples, and I've decided to leave these in. The model is still very good at its intended task, and the main focus of this challenge is on figuring out how it solves the task, not dissecting the situations where it fails. However, you might find that the adversarial examples help you understand the model better.

Recommended materi

...
A content analysis of the SQ-R questionnaire and a proposal for testing EQ-SQ theory
by tailcalled — 2023-08-09T13:51:02.036Z

If you are more interested in this topic, I have created a Discord server titled Rationalist Psychometrics to discuss these sorts of things. Thank you to Justis Mills for proofreading and feedback.

I've recently been complaining about the EQ-SQ theory of autism which asserts that autism is caused by having an extremely male brain, and I've felt like it could probably be disproven with a bit of work. Briefly, my opinion is that the measures conflate multiple different things (e.g. technical interests vs nature interests), and I propose that one can test this by seeing whether the items that correlate with autism are the same as the items that correlate with sex. But in order for it to be tested, someone has to actually perform that work.

And part of the trouble here is, the Systemizing Quotient-Revised (SQ-R) is very long, so it would be very expensive to collect comprehensive data on it. I tried contacting some rationalists with reasonably far reach (Scott Alexander and Aella) to see if they would be interested in sharing a comprehensive autism measure to their audience to get me data for free, but so far I have not received any responses yet.

So I need to find some way to make it quicker and cheaper. One way to make it cheaper would be to construct a "short form", which measures the same traits in a shorter way by only using a subset of the items.

There have already been constructed short forms of the SQ-R, but the only ones I have seen have been constructed with a very basic empirical approach of finding items that are highly correlated with the total scores of the scale. This is a problem for investigating measurement biases, as it can obscure the measurement bias and hide invalidity of the construct.[1]

The principled way to solve this would be to perform a factor analysis, searching for groups of correlated items in an SQ-R dataset, and then collecting the few top items on each factor. Unfortunately, I do not have access to an SQ-R dataset; if I did, this problem would be gone.

But, usually the results of a factor analysis can be predicted fairly well by looking at similarity in the content of the items. Furthermore, I have a bunch of experience with personality and sex difference psychometrics, so I can make educated guesses about what sorts of content does or does not matter. To avoid bias, I would like to present my content analysis before I collect the da...
If I Was An Eccentric Trillionaire
by niplav — 2023-08-09T07:56:46.259Z

cross-posted from niplav.site

What I might do if I magically got hold of a very large amount of
money, and couldn't spend it maximally altruistically.

I sometimes like to engage in idle speculation. One of those speculations
is: "If someone came up to me and told me that they would give me a lot
of money, but only under the condition that I would spend most of it on
unconventional and interesting projects, and I was forbidden to give it to
Effective Altruist organizations narrowly defined, what would I do? Not
disallowing the projects from having positive consequences accidentally,
of course."

The following is a result of this speculation. Many of the ideas might
be of questionable morality; I hope it's clear I would think a bit more
about them if I were to actually put them into practice (which I won't,
since I don't have that type of money, nor am I likely to get hold of
it myself anytime soon).

Lots of these ideas aren't mine, and I have tried to attribute them
wherever I could find the source. If guess that if they were implemented
(not sure whether that's possible: legality & all that) I'd very likely
become very unpopular in polite society. But the resulting discourse
would absolutely be worth it.

Intervention
Cost

Snowball fights
?

Buy a small island nation
$5 bio.

Personal futarchy on steroids
$100 mio.

Save dying languages
$2 bio.

Raise native speakers of an engineered conlang
$30.8 mio.

Philosophically solve language
$10 mio.

SCP series
$1 bio.

Antimemetics Division spinoff
$200 mio.

Discontinuous & fast AI takeoff movie
$500 mio.

Double Crux podcast
$2 mio.

Fictional ethnography of native Antarcticans
$100k

Studying foreveraloners
$900 mio.

Really Out There Stuff Institute
$60 mio.

Sum
$ 9.81 bio.

Culture

Snowball fights: I remember having a lot of fun with snowball
fights during breaks in school, but I haven't seen them happen anywhere
outside since leaving school. People sign up for participating in
(moderated) snowball fights, and are notified on weekends with especially
good snow. Large open grassy fields (or perhaps even stadiums) are
rented, and separated into at least 2 parts, each parts gets a large pile
of fresh snow (to minimize the chance of stones in snowballs). People are
encouraged to bring protective gear to allow for rougher fighting. If one
is to allow for physical encounters, one'd

...