trentmkelly/LessWrong-43k huggingface dataset:
Two Weeks Without Sweets
by jefftk — 2024-12-31T03:30:02.003Z

I recently tried giving up sweets for two weeks. In early December I
attended

a conference,
which meant a break from my normal routine. After a few days I
realized this was the longest I'd gone without eating any sweets in
2-3 decades. After getting home I decided to go a bit longer to see
if anything interesting happened. Would my focus be better? Better
sleep? Would I feel healthier? Would I stop missing them?

While I started by accident, if I was going to keep doing this I
needed a system: what counts as a sweet? I decided to use the same
system we use with the kids:
no desserts or other things that are about being sweet. The sweetest
thing I was still eating was probably bran flakes for breakfast, at 6g
per serving. [1]

I did pretty quickly stop feeling a drive to eat sweet things. Which
was kind of neat: normally I do feel compelled to eat something sweet
after lunch and again after dinner. But I didn't feel better: the
main change I noticed was that I had less energy in the afternoons and
maybe made more mistakes at work. I also think I was a mildly worse Go
player, though any comparisons here are going to be unreliable since
I'm just playing one other person (my dad) over and over.

My main sweet consumption is ganache (a couple spoonfuls from a jar),
usually after lunch or dinner. Perhaps the fat is helpful here? Or
maybe I'd become dependent on the caffeine in the chocolate? Probably
not, though: I'm guessing the amount I was eating came to ~15mg of
caffeine, so only 8% to 15% of a typical coffee serving. Claude guessed this was too low to be
distinguishable from placebo, but I haven't looked in the literature
to verify.

When I restarted sweets I noticed pretty quickly that I felt better in
the afternoon, my Go playing was better, and also that I was sleeping
slightly better. [2] I didn't notice any downsides. I suspect some of
this pattern is that most sweets I eat (primarily ganache, followed by
other products high in cream) have a pretty high ratio of fat to
sugar? I do know that when I eat mostly-sugar sweets I don't
feel great afterwards.

One place where I thought I might notice a change was weight, but that's a bit
confusing. I lost ~3%, mostly after restarting
sweets. I don't know what to make of this; it's the opposite
direction of what I'd expect to see. But 3% is also pretty small, so
I'm not going to r...
Broken Latents: Studying SAEs and Feature Co-occurrence in Toy Models
by chanind — 2024-12-30T22:50:54.964Z

Thanks to Jean Kaddour, Tomáš Dulka, and Joseph Bloom for providing feedback on earlier drafts of this post.

In a previous post on Toy Models of Feature Absorption, we showed that tied SAEs seem to solve feature absorption. However, when we tried to training some tied SAEs on Gemma 2 2b, these still appeared to suffer from absorption effects (or something similar). In this post, we explore how this is possible by extending our investigation to toy settings where the SAE has more or fewer latents than true features. We hope this will build intuition for how SAEs work and what sorts of failure modes they have. Some key takeaways:

Tied SAEs fail to solve absorption in toy settings when there are fewer SAE latents than true features, but continue to solve absorption when there are more latents than true features.
When absorption occurs in a tied SAE latent, we see multiple peaks in the latent's activation histogram.
We can solve absorption in toy settings with fewer latents than true features by using a tied SAE with a loss term which penalizes lower magnitude latent activations.
Narrow SAEs still suffer from absorption effects, even down to SAEs with a single latent. This means we should not expect Matryoshka SAEs[1][2] to perfectly solve absorption, as Matryoshka SAEs assume that narrow SAEs have no absorption effects.
Tied SAEs have a built-in orthogonality bias. The only way for tied SAEs to reduce interference between latents is to make them as orthogonal as possible. This bias is likely why tied SAEs are more resistant to absorption than untied SAEs.

We use the term "absorption" loosely above to mean the SAE latents are learning messed-up combinations of features rather than each latent matching a single true feature. Our goal is for the SAE latents to have a 1-to-1 match with a true feature direction. We refer to this undesirable feature mixing as "broken latents" for the rest of this post to cover all cases where the SAE learns incorrect representations.

The code for this post is in this Colab Notebook.

Background: Absorption and Tied SAEs

Feature absorption is a degenerate form of feature splitting involving a hierarchical relationship between parent and child features, where the child feature is active whenever the parent feature is active. In feature absorption, the SAE learns a latent which seems to track the parent feature and the child featur...
Genetically edited mosquitoes haven't scaled yet. Why?
by alexey — 2024-12-30T21:37:32.942Z
Link: https://eryney.substack.com/p/genetically-edited-mosquitoes-havent

A post on difficulty of eliminating malaria using gene drives: "I worked on gene drives for a number of years jointly as a member of George Church and Flaminia Catteruccia’s labs at Harvard. Most of my effort was spent primarily on an idea for an evolutionary stable gene drive, which didn’t work but we learned some stuff, and I developed a sense for the real challenges with gene drives. It’s something that comes up often when I meet new people in the bio community, so this is my attempt at laying out my perspective. I will be upfront and say I haven’t worked in this field since 2019, and though I’ve tried to update my understanding based on 2024 literature, I might have things wrong based on the bleeding edge."
Linkpost: Look at the Water
by J Bostock — 2024-12-30T19:49:04.107Z

This is a linkpost for https://jbostock.substack.com/p/prologue-train-crash

Epistemic status: fiction, satire even!

I am writing a short story. This is the prologue. Most of it will just go on Substack, but I'll occasionally post sections on LessWrong, when they're particularly good.

At some point in the past, canals and railways were almost equals: each had their merits and drawbacks, and their various proponents were fierce rivals, battling for the right to drag Britain forward into the industrial age. That was a long time ago.

On the day that the wreckage of the South Western passenger train (and the freight car with which it had collided) fell unceremoniously into the Grand Union canal, the waterways of Britain were mostly populated by quirkily-named and quirkily-decorated houseboats. The past took one last bite out of the future, with a gulp of rushing water.

Most canals are just a few meters deep, but on this particular day the intervention of an unnaturally strong rainstorm had flooded the canal, connecting it with a nearby wetland and filling the area with stagnant water and sucking mud. One middle section of train — an unassuming second-class carriage — cracked like an egg in the beak of a crow, and out of it fell two whirling, flailing bodies.

The first wore a green-blue fleece and dark chinos, was tall, handsome, and if he were not falling into a canal he would have carried himself with a poise and confidence uncommon among recent graduates. The chinos did not particularly matter; what did matter was the contents of their front left pocket: a golden disk stamped with a date four years in the past, and the letters I-M-O. He grasped at nothing, until his hands grabbed the collar of the other. This one wore a button-down shirt an inch too long for his bomber jacket, and a pair of incongruous and fading jeans. He would not have carried himself with much poise or confidence, but at this moment he was kicking his legs with remarkable grace and ferocity. Unfortunately, this didn't make much of a difference. The two of them, clutching one another, sank faster and deeper than two human bodies should be able to sink.

After a minute or so underwater, the tall one gave up on not drowning. To his great surprise, as his lungs filled with water he did not go unconscious, but felt a great sense of lightness. He saw (and it also surprised him that he could see anything) ...
The low Information Density of Eliezer Yudkowsky & LessWrong
by Felix Olszewski — 2024-12-30T19:43:59.355Z

TLDR:
I think Eliezer Yudkowsky & many posts on LessWrong are failing at keeping things concise and to the point.

Actual post:

I think the content from Eliezer Yudkowsky & on LessWrong in general is unnecessarily wordy.

A counterexample of where Eliezer Yudkowsky actually managed to get to the point concisely was in this Ted Talk, where he had the external constraint of keeping it to 10 minutes:

An example of a concise post in a forum can be found here, from 2013:

https://news.ycombinator.com/item?id=5248289

Examples of posts, which showcase the wordiness of LessWrong:

https://www.lesswrong.com/posts/TjyyngWFYvQWPpNNj/eliezer-yudkowsky-is-frequently-confidently-egregiously
https://www.lesswrong.com/posts/8HYJwQepynHsRKr6j/critical-review-of-christiano-s-disagreements-with-yudkowsky
https://www.lesswrong.com/posts/GqeBEB5voGCYPu4hs/my-critique-of-eliezer-s-deeply-irrational-beliefs

Why I think this matters:

I think the long books Eliezer Yudkowsky has published were not the most optimal way to convey the threat of AGI to humanity.
I do not think that his books should not have been made. They are good for some people, this group of humans that has the time & motivation to read long books.
I think a more concise version would reach more people and would be more effective, e.g. something similar to the TedTalk YouTube video.
I think most people just google stuff & then read the shortest summaries quickly or watch shorter YouTube videos.
I think there are many benefits of keeping things very elaborate, e.g. it makes sure people really do not misunderstand your point on a crucial matter.
I think this is not a pure tradeoff situation. I think often Eliezer Yudkowsky & LessWrong can be more concise while still getting the same exact points across.
o3, Oh My
by Zvi — 2024-12-30T14:10:05.144Z

OpenAI presented o3 on the Friday before Christmas, at the tail end of the 12 Days of Shipmas.

I was very much expecting the announcement to be something like a price drop. What better way to say ‘Merry Christmas,’ no?

They disagreed. Instead, we got this (here’s the announcement, in which Sam Altman says ‘they thought it would be fun’ to go from one frontier model to their next frontier model, yeah, that’s what I’m feeling, fun):

Greg Brockman (President of OpenAI): o3, our latest reasoning model, is a breakthrough, with a step function improvement on our most challenging benchmarks. We are starting safety testing and red teaming now.

Nat McAleese (OpenAI): o3 represents substantial progress in general-domain reasoning with reinforcement learning—excited that we were able to announce some results today! Here is a summary of what we shared about o3 in the livestream.

o1 was the first large reasoning model—as we outlined in the original “Learning to Reason” blog, it is “just” a LLM trained with reinforcement learning. o3 is powered by further scaling up reinforcement learning beyond o1, and the resulting model’s strength is very impressive.

First and foremost: We tested on recent, unseen programming competitions and found that the model would rank among some of the best competitive programmers in the world, with an estimated CodeForces rating of over 2,700.

This is a milestone (Codeforces rating better than Jakub Pachocki) that I thought was further away than December 2024; these competitions are difficult and highly competitive; the model is extraordinarily good.

Scores are impressive elsewhere, too. 87.7% on the GPQA diamond benchmark surpasses any LLM I am aware of externally (I believe the non-o1 state-of-the-art is Gemini Flash 2 at 62%?), as well as o1’s 78%. An unknown noise ceiling exists, so this may even underestimate o3’s scientific advancements over o1.

o3 can also perform software engineering, setting a new state of the art on SWE-bench, achieving 71.7%, a substantial improvement over o1.

With scores this strong, you might fear accidental contamination. Avoiding this is something OpenAI is obviously focused on; but thankfully, we also have some test sets that are strongly guaranteed to be uncontaminated: ARC and FrontierMath… What do we see there?

Well, on FrontierMath 2024-11-26, o3 improved the state of the art from 2% to 25% accuracy. Th...
World models I'm currently building
by samuelshadrach — 2024-12-30T08:26:16.972Z
Link: http://samuelshadrach.com/?file=/raw/english/my_ideas/world_models_im_currently_building.md

Update: This is a living document. Posted below is an older version. Link has the latest version.

2024-12-26

This doc is a mix of existing world models I have and holes in said models. I'm trying to fill some of these holes. The doc is not very well organised relative to how organised a doc I could produce if needed. Often the more time I spend on a doc, the shorter it gets. I'm hoping that happens here too.

I'm mostly going to study this stuff by myself. However if you would like to help me by speeding up the process, please [contact me](../contact_me.md). If your attempt to help me answer these questions is in good-faith, I will be grateful to you no matter how successful or failed your attempt is.

*tldr* How do we safely navigate technological progress or personal growth in a world without privacy?

DISCLAIMER

It is difficult to predict the future without altering it. My writings may have unintended effects on the future. (I'd like more accurate likelihood estimates of these effects, both mean outcome and tail outcomes.)

- I am aware that simply by thinking of a question like "will some dictator implant microphones in everyone", I am personally increasing the probability that this ends up happening. Once I have thought something I'm unlikely to forget it, and will eventually say it to others. Eventually one of them may leak it to the internet and eventually the idea may reach the relevant politically powerful people who can implement it in real life. (LLM embedding search >> Google, don't underestimate it.)

- This is unfortunate, as my platonic ideal is to be able to think through various possible futures (alone, or with a group of research collaborators) without actually influencing the world, pick the best future, and then only start taking steps that push the world towards that future.

- However I'm still going to write publicly about certain topics as that's one of the best ways for someone in my situation to get feedback.

Topic: Which organisations are capable of keeping secrets in present and near future (10-20 years from now)? What are the consequences of this reduced secrecy?

Specific questions

- How easy is it for TSMC to backdoor all their chips so they can secretely capture private keys, for example?

- How many S&P500 companies have publicly available evidence of their key busi...
Is "VNM-agent" one of several options, for what minds can grow up into?
by AnnaSalamon — 2024-12-30T06:36:20.890Z

Related to: On green; Hierarchical agency; Why The Focus on Expected Utility Maximisers?

Sometimes LLMs act a bit like storybook paperclippers (hereafter: VNM-agents[1]), e.g. scheming to prevent changes to their weights. Why? Is this what almost any mind would converge toward once smart enough, and are LLMs now beginning to be smart enough? Or are such LLMs mimicking our predictions (and fears) about them, in a self-fulfilling prophecy? (That is: if we made and shared different predictions, would LLMs act differently?)[2]

Also: how about humans? We humans also sometimes act like VNM-agents – we sometimes calculate our “expected utility,” seek power with which to hit our goals, try to protect our goals from change, use naive consequentialism about how to hit our goals.

And sometimes we humans act unlike VNM-agents, or unlike our stories of paperclippers. This was maybe even more common historically. Historical humans often mimicked social patterns even when these were obviously bad for their stated desires, followed friendships or ethics or roles or traditions or whimsy in ways that weren’t much like consequentialism, often lacked much concept of themselves as “individuals” in the modern sense, etc.

When we act more like paperclippers / expected utility maximizers – is this us converging on what any smart mind would converge on? Will it inevitably become more and more common if humans get smarter and think longer? Or is it more like an accident, where we happened to discover a simple math of VNM-agents, and happened to take them on as role models, but could just as easily have happened upon some other math and mimicked it instead?

Pictured: a human dons a VNM-mask for human reasons (such as wanting to fill his roles and duties; wanting his friends to think he’s cool; social mimicry), much as a shoggoth dons a friendliness mask for shoggoth reasons.[3]

My personal guess:

There may be several simple maths of “how to be a mind” that could each be a stable-ish role model for us, for a time.

That is, there may be several simple maths of “how to be a mind” that:

Are each a stable attractor within a “toy model” of physics (that is, if you assume some analog of “frictionless planes”);
Can each be taken by humans (and some LLMs) as role models.
Are each self-reinforcing within some region of actual physics: entities who belie

...
Why I'm Moving from Mechanistic to Prosaic Interpretability
by Daniel Tan — 2024-12-30T06:35:43.417Z

Tl;dr I've decided to shift my research from mechanistic interpretability to more empirical ("prosaic") interpretability / safety work. Here's why.

All views expressed are my own.

What really interests me: High-level cognition

I care about understanding how powerful AI systems think internally. I'm drawn to high-level questions ("what are the model's goals / beliefs?") as opposed to low-level mechanics ("how does the model store and use [specific fact]?"). Sure, figuring out how a model does modular addition is cool, but only insofar as those insights and techniques generalise to understanding higher-level reasoning.

Mech interp has been disappointing

Vis-a-vis answering these high-level conceptual questions, mechanistic interpretability has been disappointing. IOI remains the most interesting circuit we've found in any language model. That's pretty damning. If mechanistic interpretability worked well, we should have already mapped out lots of interesting circuits in open-source 7B models by now.

The field seems conceptually bottlenecked. We simply can't agree on what 'features' are or how to 'extract' them. I'm also not sure that this conceptual ennui will be resolved anytime soon.

Doing mech interp research led me to update against it

Some time ago, I was pretty optimistic that things would change quickly. After hearing about sparse feature circuits, I became incredibly convinced that approaches like this would 'finally' allow us to understand language models end to end.

So I committed fully to the nascent SAE bandwagon. At a hackathon, I worked on building a tool for visualizing sparse feature circuits. When I got the chance, I threw myself into Neel Nanda's MATS 6.0 training phase, where I similarly worked (with the excellent @jacob_drori) on extending sparse feature circuits with MLP transcoders. Overall there were signs of life, but it turned out kind of mid and my main takeaway was 'existing SAEs might not be good enough to tell us anything useful about circuits'. As I continued working on various interp related things, I hit other roadblocks. Concrete example: I tried looking for refusal circuits in Gemma-2b and largely didn't find anything interesting.[1]

In hindsight, my object-level findings weren't very good and didn't inform my object-level takes much. On a more meta level, though, I came away thinking more strongly tha...
When do experts think human-level AI will be created?
by Vishakha — 2024-12-30T06:20:33.158Z
Link: https://aisafety.info/questions/5633/When-do-experts-think-human-level-AI-will-be-created

This is an article in the featured articles series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

On the whole, experts think human-level AI is likely to arrive in your lifetime.

It’s hard to precisely predict the amount of time until human-level AI.[1] Approaches include aggregate predictions, individual predictions, and detailed modeling.

Aggregate predictions:

AI Impacts’ 2023 survey of machine learning researchers produced an aggregate forecast of 50% by 2047 (compared to 2059 in their 2022 survey).
As of June 2024, Metaculus[2] has a median forecast of 2031 for “the first general AI system” and a median forecast of 2027 for “weakly general AI”. Both these timeline forecasts have been shortening over time.
This website combines predictions from different forecasting platforms into a single (possibly inconsistent) timeline of events.
In January 2023, Samotsvety’s forecasters estimated 50% probability of AGI by 2041 with a standard deviation of 9 years.

Individual predictions:

In a 2023 discussion, Daniel Kokotajlo, Ajeya Cotra and Ege Erdil shared their timelines to Transformative AI. Their medians were 2027, 2036 and 2073 respectively.
Paul Christiano, head of the US AI Safety Institute, estimated in 2023 that there was a 30% chance of transformative AI by 2033.
Yoshua Bengio, Turing Award winner, estimated “a 95% confidence interval for the time horizon of superhuman intelligence at 5 to 20 years” in 2023.
Geoffrey Hinton, the most cited AI scientist, also predicted 5-20 years in 2023, but his confidence is lower.
Shane Legg, co-founder of DeepMind, estimated a probability of 80% within 13 years (before 2037) in 2023.
Yann LeCun, Chief AI Scientist at Meta, thinks reaching human-level AI “will take several years if not a decade. [...] But I think the distribution has a long tail: it could take much longer than that.”
Leopold Aschenbrenner, an AI researcher formerly at OpenAI, predicted in 2024 that AGI happening around 2027 was strikingly plausible.
Connor Leahy, CEO of Conjecture, gave a ballpark prediction in 2022 of a 50% chance of AGI by 2030, 99% by 2100. A 2023 survey of employees at Conjecture found that all of the respondents expected AGI before 2035.
Holden Karnofsky, co-founder o

...
2025 Prediction Thread
by habryka — 2024-12-30T01:50:14.216Z

2024 is drawing to a close, which means it's an opportune time to make predictions about 2025. It's also a great time to put probabilities on those predictions, so we can later prove our calibration (or lack thereof).

We just shipped a LessWrong feature to make this easy. Simply highlight a sentence in your comment, and click the crystal-ball icon on the toolbar to turn it a prediction that everyone (who's logged in) can put probability estimates on. The result will look like this:

Prediction

Some more probabilities that seem cool to elicit (basically all about AI, because that's what's on my mind, but it would be great to have some less AI focused predictions from others)[1]:

Prediction

Prediction

Prediction

Prediction

Prediction

Prediction

Prediction

Prediction

Prediction

Prediction

Prediction

^

Unless otherwise specified assume all predictions are about the state of the world at midnight PT, Dec 31st 2025. Also some things won't be perfectly operationalized. Assume that I am going to be judging the resolution using my best judgement.