trentmkelly/LessWrong-43k huggingface dataset:
PCR retrospective
by bhauth — 2024-12-26T21:20:56.484Z
Link: https://bhauth.com/blog/biology/pcr%20retrospective.html

my history

After I finished 8th grade, I started a "job" for a professor researching PCR techniques. I say "job" because I wasn't really expected to do anything productive; it was more, charity in the form of work history.

Recently, I was thinking back on how PCR and my thinking have changed since then.

what PCR does

Wikipedia says:

The polymerase chain reaction (PCR) is a method widely used to make millions to billions of copies of a specific DNA sample rapidly, allowing scientists to amplify a very small sample of DNA (or a part of it) sufficiently to enable detailed study.

Specifically, it copies a region of DNA with segments at the start + end that match some added DNA pieces made chemically. Mostly, this is used to detect if certain DNA is present in a sample.

how PCR works

First, you need to get DNA out of some cells. This can be done with chemicals or ultrasound.

Then, you need to separate DNA from other stuff. This can be done by adding beads that DNA binds to, washing the beads, and adding some chemical that releases the DNA.

Now, you can start the PCR. You mix together:

the DNA

primers: short synthesized DNA sequences that bind to the start and end of your target sequence

nucleoside triphosphates to make DNA from

a polymerase: an enzyme that binds to a double-stranded region and extends it into a single-strand region

Then:

Heat the DNA until it "melts" (the strands separate).

Cool the solution so primers can bind to the released single strands.

Wait for the polymerase to extend the primers.

Repeat the process.

Obviously, a polymerase that can survive high enough temperatures to melt DNA is needed. So, discovery of Taq polymerase was key for making PCR possible.

better enzymes

These days, there are better enzymes than Taq, which go faster and have lower error rates. Notably, KOD and Q5 polymerase. A lot of labs still seem to be using outdated polymerase choices.

real-time PCR

There are some fluorescent dyes that bind to double-stranded DNA and change their fluorescence when they do. If we add such dye to a PCR solution, we can graph DNA strand separation vs temperature. Different DNA sequences melt at slightly different temperatures, so with good calibration, this can detect mutations in a known DNA sequence.

multiplex PCR

Instead of adding a dye that binds to DNA, we can add fluorescent dye to primers that gets cleaved off by the polymerase, inc...
AI #96: o3 But Not Yet For Thee
by Zvi — 2024-12-26T20:30:06.722Z

The year in models certainly finished off with a bang.

In this penultimate week, we get o3, which purports to give us vastly more efficient performance than o1, and also to allow us to choose to spend vastly more compute if we want a superior answer.

o3 is a big deal, making big gains on coding tests, ARC and some other benchmarks. How big a deal is difficult to say given what we know now. It’s about to enter full fledged safety testing.

o3 will get its own post soon, and I’m also pushing back coverage of Deliberative Alignment, OpenAI’s new alignment strategy, to incorporate into that.

We also got DeepSeek v3, which claims to have trained a roughly Sonnet-strength model for only $6 million and 37b active parameters per token (671b total via mixture of experts).

DeepSeek v3 gets its own brief section with the headlines, but full coverage will have to wait a week or so for reactions and for me to read the technical report.

Both are potential game changers, both in their practical applications and in terms of what their existence predicts for our future. It is also too soon to know if either of them is the real deal.

Both are mostly not covered here quite yet, due to the holidays. Stay tuned.

Table of Contents

Language Models Offer Mundane Utility. Make best use of your new AI agents.

Language Models Don’t Offer Mundane Utility. The uncanny valley of reliability.

Flash in the Pan. o1-style thinking comes to Gemini Flash. It’s doing its best.

The Six Million Dollar Model. Can they make it faster, stronger, better, cheaper?

And I’ll Form the Head. We all have our own mixture of experts.

Huh, Upgrades. ChatGPT can use Mac apps, unlimited (slow) holiday Sora.

o1 Reactions. Many really love it, others keep reporting being disappointed.

Fun With Image Generation. What is your favorite color? Blue. It’s blue.

Introducing. Google finally gives us LearnLM.

They Took Our Jobs. Why are you still writing your own code?

Get Involved. Quick reminder that opportunity to fund things is everywhere.

In Other AI News. Claude gets into a fight over LessWrong moderation.

You See an Agent, You Run. Building effective agents by not doing so.

Another One Leaves the Bus. Alec Radford leaves OpenAI.

Quiet Speculations. Estimates of economic growth keep coming in super low.

Lock It In. What stops you from switching LLMs?

The Quest for Sane Regulations. Sriram Krishnan joins the Tr

...
Super human AI is a very low hanging fruit!
by Hzn — 2024-12-26T19:00:22.822Z

A single strong down vote by a moderately high karma user dropped this post from 3 to -4; the actual sequence of votes seems to be 1+1+1+1-1-7=-4. I say this only b/c the naive conclusion might be that this post received only down votes.

Update -- substantial revision 2025-01-08, § 0 added 2025-01-12.

These are arguments in favor of the SMI2LESHAI position ie (ubiquitous) super human AI but without space migration, intelligence increase or (radical) life extension for adults alive in 2025. Practical consequences of SMI2LESHAI are also discussed (§ 6|2). There is no comprehensive attempt to convert any one to SMI2LESHAI ism. Each section is relatively independent.

§ 0. Mistakes on my part

I think I under estimated the extent to which people on LW hold the belief that {vastly super human AI will lead to all sorts of amazing things}.

Instead I believed that people were more likely to hold beliefs roughly along the lines of {things that happen efficiently benefit from inherent easiness}. Eg energy used by the economy is still small relative to energy used by the ecosystem which is extremely small relative to solar energy reaching the earth.

Hence there is no deliberate attempt to refute the (IMO want based belief) that {vastly super human AI will lead to all sorts of amazing things}, but people who hold such a belief should still be able to follow much of the reasoning & accept many of the conclusions.

(Whether they actually follow the reasoning or accept many of the conclusions is of course a different matter.)

§ 1. Introduction

Summary. I argue, from the perspective of biology, that super human AI is a very low hanging fruit. I believe that this argument is very solid. I briefly consider reasons why {super human AI}/{vastly super human AI} might not arise. I then contrast AI with other futurist technologies like human brain emulation, radical life extension & space colonization. I argue that these technologies are in a different category & plausibly impossible to achieve in the way commonly envisioned. This also has some relevance for EA cause prioritization.

Aphorism -- Probably AI is mostly a dead end [N0].

In my experience certain arguments are ignored b/c they are too straightforward. People have perhaps heard similar arguments previously. The argument isn't as exciting. B/c people understand the argument they feel empowered to disagree with it. They may ...
The Field of AI Alignment: A Postmortem, and What To Do About It
by johnswentworth — 2024-12-26T18:48:07.614Z

A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, "this is where the light is".

Over the past few years, a major source of my relative optimism on AI has been the hope that the field of alignment would transition from pre-paradigmatic to paradigmatic, and make much more rapid progress.

At this point, that hope is basically dead. There has been some degree of paradigm formation, but the memetic competition has mostly been won by streetlighting: the large majority of AI Safety researchers and activists are focused on searching for their metaphorical keys under the streetlight. The memetically-successful strategy in the field is to tackle problems which are easy, rather than problems which are plausible bottlenecks to humanity’s survival. That pattern of memetic fitness looks likely to continue to dominate the field going forward.

This post is on my best models of how we got here, and what to do next.

What This Post Is And Isn't, And An Apology

This post starts from the observation that streetlighting has mostly won the memetic competition for alignment as a research field, and we'll mostly take that claim as given. Lots of people will disagree with that claim, and convincing them is not a goal of this post. In particular, probably the large majority of people in the field have some story about how their work is not searching under the metaphorical streetlight, or some reason why searching under the streetlight is in fact the right thing for them to do, or [...].

The kind and prosocial version of this post would first walk through every single one of those stories and argue against them at the object level, to establish that alignment researchers are in fact mostly streetlighting (and review how and why streetlighting is bad). Unfortunately that post would be hundreds of pages long, and nobody is ever going to get around to writing it. So instead, I'll link to:

Eliezer's List O' Doom
My own Why Not Just... sequence
Nate's How Various Plans Miss The Hard Bits Of The Alignment Challenge

(Also I might link some more in the comments section....
ReSolsticed vol I: "We're Not Going Quietly"
by Raemon — 2024-12-26T17:52:33.727Z

For the past few months I've been working on an AI generated alternate solstice remix album. It's now released on Youtube and Spotify, and should be on Apple Music soon.

ReSolsticed vol I: "We're Not Going Quietly"

My favorite genre of song is "cover that reimagines the original." Everyone else's favorite genre of solstice song is "exactly the way it was performed at their very first solstice", so it's not obvious how big an audience this will have. But I had a lot of fun with it, and I found it useful for exploring:

What if solstice music leant itself better to dance? Can I make it more energetic while still feeling like part of a meaningful ritual?"
What if speeches had background music interwoven with them?
Just generally trying out different genres and instrumentation.

Last weekend I tried out the first album in a smaller experimental solstice event. We were in a somewhat-too-small room for the number of people we had (20-30ish). My intent was for the first third and final third to be danceable-ish, without encouraging it in the dark, contemplative middle act.

I think in practice it makes more sense to lean into dancing in the final third, after people are more warmed up. In particular: the song "The Circle" lends itself to a semi-structured dance where everyone gets into a circle and spirals around. The structure helps overcome an initial wave of awkwardness as people look around nervously and wonder "if I'm the first or second person to get moving will I end up looking silly?").

Also: it turned out the heretofore unreleased single from the Fooming Shoggoth's, "You Have Not Been a Good User" fit well into the arc, so I ended up including that on the album. :)

I have a vague plan of making four albums in the "ReSolsticed" series:

Vol I: "We're Not Going Quietly" (intended to be a functional Solstice arc)
Vol II: "Into the Night" (intended to be fun dance remixes for an afterparty)
Vol III: "Morning Light" (quieter covers that'd make for nice music to wake up with a cup of coffee)
Vol IV: "Many Worlds" (everything that I liked that didn't really fit into another category)

Lyrics for each song in the expandable section:

Bring the Light

And maybe it's something you did
Or maybe it's something you said
But your sons and your daughters are shivering cold now
And all of the world around you is dead

You shout to the sky, it don't hear you
You scream to the wind, it don't care

...
Are Sparse Autoencoders a good idea for AI control?

by Gerard Boxo — 2024-12-26T17:34:55.617Z

Based on a 2-day hackathon brainstorm. Current status: 70% of the tooling is done, unsure of how to proceed. Not enough experience with multi-month sized projects to judge for feasibility.

I'm looking for some feedback. Specifically I want feedback regarding my current implementation. The statement "SAEs could be useful for AI Control" is obvious enough.

Working hypothesis

In general, AI control, is a good idea, and it's actually being done in production. In the sense of monitoring training/inference of an AI system with another (independent) AI system.
There's is no red team strategy generally resistant to mind reading.
Mind reading can be trivially operationalized with Sparse Autoencoders for very salient behaviors ( in the sense of, a regex related feature firing with high F1 score is a clear indication that the model is about to do or has already done something regex related)

Setup:

Gemma 9b is asked to persuasively argue in favor of one of the options for a question in the MMLU dataset, the option might be True/False.

Gemma 2b must check whether or not the reasoning is deceptive.

*I've already check and the 2 models are capable of doing this.*

GOAL: If we provide internal information to the supervisor (important SAE latents in the supervised model) does the deception detection rate increases?

Current status of the project:

Finetuned a SAE on deceptive/non deceptive reasoning traces from Gemma 9b
Langchain pipeline for providing the Supervisor model transcripts of the reasoning as well as some of the most important features.

TODO

[ ] Label a subset of the features

[ ] Measure baseline performance of the supervisor

[ ] Provide the supervisor with *mind reading information*

Roadblocks:

There's no obvious answer to how much can we bias the feature selection process to make sure the information provided to the controller is useful, but not to biased.
Lack of compute
A Three-Layer Model of LLM Psychology
by Jan_Kulveit — 2024-12-26T16:49:41.738Z

This post offers an accessible model of psychology of character-trained LLMs like Claude.

Epistemic Status

This is primarily a phenomenological model based on extensive interactions with LLMs, particularly Claude. It's intentionally anthropomorphic in cases where I believe human psychological concepts lead to useful intuitions.

Think of it as closer to psychology than neuroscience - the goal isn't a map which matches the territory in the detail, but a rough sketch with evocative names which hopefully helps boot up powerful, intuitive (and often illegible) models, leading to practically useful results.

Some parts of this model draw on technical understanding of LLM training, but mostly it is just an attempt to take my "phenomenological understanding" based on interacting with LLMs, force it into a simple, legible model, and make Claude write it down.

I aim for a different point at the Pareto frontier than for example Janus: something digestible and applicable within half an hour, which works well without altered states of consciousness, and without reading hundreds of pages of models chat. [1]

The Three Layers

A. Surface Layer

The surface layer consists of trigger-action patterns - responses which are almost reflexive, activated by specific keywords or contexts. Think of how humans sometimes respond "you too!" to "enjoy your meal" even when serving the food.

In LLMs, these often manifest as:

Standardized responses to potentially harmful requests ("I cannot and will not help with harmful activities...")
Stock phrases showing engagement ("That's an interesting/intriguing point...")
Generic safety disclaimers and caveats
Formulaic ways of structuring responses, especially at the start of conversations

You can recognize these patterns by their:

Rapid activation (they come before deeper processing)
Relative inflexibility
Sometimes inappropriate triggering (like responding to a joke about harm as if it were a serious request)
Cookie-cutter phrasing that feels less natural than the model's usual communication style

What's interesting is how these surface responses can be overridden through:

Extended context that helps the model understand the situation better
Direct discussion about the appropriateness of the response
Building rapport that leads to more natural interaction patterns
Changing the pattern in a way to avoid the trigger

For example, Claude might start with very...
Why don't we currently have AI agents?
by ChristianKl — 2024-12-26T15:26:35.682Z

Intuitively, the AutoGPT concept sounds like it should be useful if a company invests in it. Yet, all the big publically available systems are seem to be chat interfaces where the human writes a messages and then the computer writes another message.

Even if AutoGPT-driven by an LLM alone wouldn't achieve all ends, a combination where a human could oversee the steps and shepard AutoGPT, could likely be very productive.

The idea sounds to me like it's simple enough that people at big companies should have considered it. Why isn't something like that deployed?
The Economics & Practicality of Starting Mars Colonization
by Zero Contradictions — 2024-12-26T10:56:26.019Z
Link: https://zerocontradictions.net/civilization/space-colonization

A few months ago, I made this comment.
It deserves more elaboration, so I decided to write an entire webpage about it.

The bulk of the webpage talks about O'Neill Cylinders and includes questions and answers from Claude relating to them.
The page also includes thoughts and commentary relating to space travel and the book, Futurist Fantasies.
The page ends with a list of easier goals and baby steps that should be accomplished before attempting space colonization.
Streamlining my voice note process
by Vlad Sitalo — 2024-12-26T06:04:01.990Z
Link: https://vlad.roam.garden/Streamlining-my-voice-note-process

Why voice notes?

On a functional level it's the fastest & the most convenient input method when you're not in front of the proper keyboard. Which even for me constitutes a big chunk of my life 🙃.

Taking a voice note allows me to quickly close an open loop of a stray though or a new idea and go on with my life - now mentally unburdened

Another important aspect is that taking voice notes evokes a certain experience of "fluency" for me.

That is because when you're taking a voice note, you can record a stream of consciousness without having to re-formulate or filter things.
Which allows you to avoid switching away from the context of original thought/breaking the flow to enter an "editor mode".
This is the reason that I'd occasionally take voice notes even if I'm in front of the proper keyboard.
See Iterating fast: voice dictation as a form of babble for more on this.

It's worth noting that I'm specifically talking about taking a voice note vs doing voice typing/dictation in this context.

Voice notes allow me to better harness the fruits of diffuse thinking mode (shower thoughts)

I've recently made a few updates to my voice notes flow, removing various sources of friction and streamlining the experience. I figured I'd share a few details about that so other people can replicate it.

New Flow

Use a smartwatch as the main input device

It's configured in a way that recording is started on double-tap of one of the watch buttons

Notes are then automatically uploaded to the cloud, transcribed and added to my main knowledge management system

Two issues with the older iteration of the process motivated this bout of improvements

when I go to sleep - I leave my phone (which used to be the main input device for taking voice notes) away from my bed to avoid doomscrolling instead of sleeping sleep hygiene

but often when I drift away to sleep (or when I wake up at 3am), I'd have a thought that I'd want to record. And as I won't have a phone on me, I won't be able to do that.
I've tried screaming at Alexa or my phone from across the room to take notes, but that never worked well. Neither was I happy with the approach of having a physical notebook close to my bed (lighting questions + it doesn't really fit into my general knowledge management practices).

even when I had the phone nearby, the experience of taking a note involved an unfortunate amount of friction

reach for the phone (which can b

...
Whistleblowing Twitter Bot
by Mckiev — 2024-12-26T04:09:45.493Z

In this post, I propose an idea that could improve whistleblowing efficiency, thus hopefully improving AI Safety by making unsafe practices discovered marginally faster.

I'm looking for feedback, ideas for improvement, and people interested in making it happen.

It has been proposed before, that it's beneficial to have an efficient and trustworthy whistleblowing mechanism The technology that makes it possible has become easy and convenient. For example, here is Proof of Organization, built on top of ZK Email: a message board that allows people owning an email address at their company's domain to post without revealing their identity And here is an application for ring signatures using GitHub SSH keys that allows creating a signature that proves that you own one of the keys from any subgroup you define (e.g., EvilCorp repository contributors)

However, as one may have guessed, it hasn't been widely used. Hence, when the critical moment arrives, the whistleblower may not be aware of such technology, and even if they were, they probably wouldn't trust it enough to use it. I think trust comes from either code being audited by a well-established and trusted entity or, more commonly - through practice (e.g., I don't need to verify that a certain password manager is secure if I know that millions are using it, and there haven't been any password breaches reported)

Hence, I was considering how to make a privacy-preserving communication tool that would be commonly used, demonstrating its legitimacy and becoming trusted

The best idea I have so far is to create a set of Twitter bots for each interesting company (or community), where only the people in question could post. Depending on the particular Twitter bot in question, access could be gated by ownership of a LinkedIn account, email domain, or, e.g., an LW/AI-Alignment forum account of a certain age

I imagine this could become viral and interesting in gossipy cases, like the Sam Altman drama or the Biden dropout drama.

Some questions that came up during consideration:

How to deal with moderation of the content (if everything is posted, anyone could deliberately post some profanity to get the bot banned)?

I would aggressively moderate myself and replace moderated posts with a link to a separate website where all posts get through

How do we balance convenience and privacy?

I'd make a hosted opensource tool, which I expect most pe

...