trentmkelly/LessWrong-43k huggingface dataset:
Detroit Lions -- over confidence is over rated?
by Hzn — 2025-01-20T10:53:48.574Z

Epistemic status -- anecdotal, lack of expertise, speculative, zero fact checking.

At first I was happy about the Lions doing well given the stigma around that franchise (Only extant NFL franchise to never even reach the Super Bowl. They also had a 0-16 season.). But then I actually watched them play… My father noticed it first -- the Lions were very cocky. Then the commentators themselves said it -- apparently this is a deliberate strategy of Dan Campbell -- that over confidence is good!

But there are different types of over confidence. Being over confident in belief, communication & planning makes sense b/c keeping track of caveats is too mentally draining. But being over confident of your own greatness is different.

Looking at the Lions it was like every player had swagger like Travis Kelce (3x Super Bowl champion, 10x Pro Bowler, Taylor Swift's boyfriend)! So obviously this reminded me of PUAs from like 20 years ago. The Game was a very popular book back when I was in college, but I never heard of any convincing cases of benefit. So I was skeptical of such a team. I did not even expect them to win the division.

From this perspective the Lions became an interesting experiment to me. Probably out of illicit (???) desire for confirmation bias, I kind of wanted them to lose. But mostly I was interested in how their season would play out. So what can we learn from the autopsy of the Lions' 2024-2025 season?

It seems like ‘over confidence’ (by which I mean over confidence in their own greatness) gave them more motivation & a better record in the regular season than they otherwise would have had. But during the playoffs every team is motivated so this advantage was basically gone. And as the #1 seed the Lions were likely to play a team that had won a playoff road game the previous week so probably not an anxious team. They also lost a lot of players to injuries during the regular season leaving a depleted squad for the playoffs… a plausible consequence of ‘over confidence’ & increased motivation?

All things considered it seems like ‘over confidence’ does have some benefits to the extent that it actually increases motivation or reduces anxiety. Those are both very important factors (albeit not enough to defeat a motivated non axious team like the Washington Commanders) & maybe not worth the increased downside risk. I don't particularly like that conclu...
Logits, log-odds, and loss for parallel circuits
by Dmitry Vaintrob — 2025-01-20T09:56:26.031Z

Today I’m going to discuss how to think about logits like a statistician, and what this implies about circuits. This post doesn’t have any prerequisites other than perhaps a very basic statistical background that can be adequately recovered from the AI-generated “glossary” to the right. I think the material here is good thing to know in general (thinking through this helped clarify my thinking about a lot of things), and it will be useful background for a future post I’m planning on “SLT in a nutshell”. If you want a “TL:DR” takeaway of the discussion that follows, the gist is that neural networks use logit addition to integrate (roughly) independent “parallel” information from various sources; and that thinking about just a very basic model of a neural net that performs this aggregates from a few parallel “black box” circuits is already a very informative conceptual toy model of stuff that neural networks do, akin to the “ideal gas” model in physics.

Basics of logits and logistic tasks

Most tasks solved by modern LLMs are some flavor of a logistic classification task. The origin of the “logistic” idea comes from statistics, and the elegant statistical context for using it (rather than other possible choices of loss) often gets lost to newcomers who were not statistics-adjacent in their past life (this was certainly true for me!). To remedy this, I’ll briefly explain, without proofs, a rough picture of why logistic loss is so nice, and in particular how it works for parallel classification programs. From now on, I am going to only work with boolean classification tasks, as other logistic tasks behave in the same way, but with more complicated notation. As small print, I’m also going to assume we are in a “large-data” limit (i.e., the number of training datapoints is very large compared to other relevant measures).

Assume we have a binary classification task y(x)∈{0,1}, for x in some big distribution of inputs. Logistic classifiers (i.e., “classifiers that use logits”) try to classify y∗(x) by a “logistic” prediction, fθ(x)∈R, that depends on a weight parameter θ∈RW.

I haven’t yet told you the formula for the loss that actually gets learned, but the goal of the logistic task – i.e. the case where it obtains optimal loss – is to optimally approximate the “

log odds

” function function f∗(x)=LogOdds(y(x)). As a formula, the log-odds functio...
Worries about latent reasoning in LLMs
by Caleb Biddulph — 2025-01-20T09:09:02.335Z

When working through a problem, OpenAI's o1 model will write a chain-of-thought (CoT) in English. This CoT reasoning is human-interpretable by default, and I think that this is hugely valuable. Assuming we can ensure that these thoughts are faithful to the model's true reasoning, they could be very useful for scalable oversight and monitoring. I'm very excited about research to help guarantee chain-of-thought faithfulness.[1]

However, there's a impending paradigm for LLM reasoning that could make the whole problem of CoT faithfulness obsolete (and not in a good way). Here's the underlying idea, speaking from the perspective of a hypothetical capabilities researcher:

Surely human-interpretable text isn't the most efficient way to express thoughts. For every token that makes some progress towards the answer, you have to write a bunch of glue tokens like "the" and "is"—what a waste of time and compute! Many useful thoughts may even be inexpressible in natural language alone.
In fact, our transformers already have a perfectly good hidden state, which works very nicely for conveying inexpressible concepts within a single forward pass. In chain-of-thought, we're collapsing that beautiful many-dimensional vector and all its semantic meaning down into a single token after every forward pass. Why should LLMs have to squeeze their thoughts through a narrow channel of tokens? Why not use a continuous latent space for reasoning?

A bit over a month ago, researchers at Meta published the paper Training Large Language Models to Reason in a Continuous Latent Space, which takes this idea and runs with it.

Here's the abstract:

Large language models (LLMs) are restricted to reason in the "language space", where they typically express the reasoning process with a chain-of-thought (CoT) to solve a complex reasoning problem. However, we argue that language space may not always be optimal for reasoning. For example, most word tokens are primarily for textual coherence and not essential for reasoning, while some critical tokens require complex planning and pose huge challenges to LLMs. To explore the potential of LLM reasoning in an unrestricted latent space instead of using natural language, we introduce a new paradigm COCONUT (Chain of Continuous Thought). We utilize the last hidden state of the LLM as a representation of the reasoning state (termed "continuous thought"). Rather than d

...
SIGMI Certification Criteria
by a littoral wizard — 2025-01-20T02:41:17.210Z

(Background fluff for a novel I'm working on and previously posted parts of. Fishing for beta readers/technical consultants again. Near-future setting that's already had one major X-risk scare, didn't learn the first time.)

The SIGMI Commission, in coordination with public signatory agencies and private partners, outlines the following eligibility criteria for admission of the Certified Exascale Integrated Diagnostic Exploration (CEIDE) examination. Candidates must have a minimum of eight years cumulative, full-time work experience across four or more Incorporated Machine Intelligences domains:

Domain 1. SPARK Datachain Architecture
Domain 2. Neural Malware Classification
Domain 3. Isolation Engineering
Domain 4. Neural Analytic Therapy
Domain 5. Identity and Personae Management (IAP)
Domain 6. Forensic Network Investigation
Domain 7. Adversarial Security Operations
Domain 8. Experiential Compound Development
Domain 9. Datachain Extinction Protocols

Earning a post-secondary degree in data science, deep learning, generative game theory, or related fields may satisfy up to three years of required experience. Part-time work and internships will not count towards this experience requirement. Additional requirements are as follows:

Prior attainment of advanced SIGMI certification in four (4) or more of the above domains with at least two (2) currently active certifications.
Eighty (80) hours submergence in a pre-production exascale environment registered in full compliance with Secure, Inspectable, Governable Machine Intelligence regulations. No exception or reduction in required hours will be granted regardless of prior field achievement, even for extraordinary candidates.
Completion of a sixteen (16) day cooldown period after contact with an exascale MI followed by a Psychological Integrity (PI) examination. Candidates with exposure to systems based in secure DARPASAT cloud domains will additionally undergo debriefing prior to and immediately following admission of this exam.

Any personal or business relations with agents of Greater China or travel to the Former Russian State Influence Sphere will be immediately disqualifying for private contractors. Rare contract may offer waivers subject to suitable performance on stochastic polygraphs administered by the US State Department or its NATO counterparts.

(All data collected during pre- and post-test examinations w...
AXRP Episode 38.5 - Adrià Garriga-Alonso on Detecting AI Scheming
by DanielFilan — 2025-01-20T00:40:07.077Z

YouTube link

Suppose we’re worried about AIs engaging in long-term plans that they don’t tell us about. If we were to peek inside their brains, what should we look for to check whether this was happening? In this episode Adrià Garriga-Alonso talks about his work trying to answer this question.

Topics we discuss:

The Alignment Workshop

How to detect scheming AIs

Sokoban-solving networks taking time to think

Model organisms of long-term planning

How and why to study planning in networks

Daniel Filan (00:09):
Hello, everyone. This is one of a series of short interviews that I’ve been conducting at the Bay Area Alignment Workshop, which is run by FAR.AI. Links to what we’re discussing, as usual, are in the description. A transcript is, as usual, available at axrp.net, and as usual, if you want to support the podcast, you can do so at patreon.com/axrpodcast. Let’s continue to the interview. Adrià, thanks for chatting with me.

Adrià Garriga-Alonso (00:30):
Thank you for having me, Daniel.

Daniel Filan (00:31):
For people who aren’t familiar with you, can you say a little bit about yourself, what you do?

Adrià Garriga-Alonso (00:35):
Yeah. My name is Adrià. I work at FAR.AI. I have been doing machine learning research focused on safety for the last three years and before that, I did a PhD in machine learning. I’ve been thinking about this for a while, and my current work is on mechanistic interpretability and specifically how we can use interpretability to detect what a neural network wants and what it might be scheming towards.

The Alignment Workshop

Daniel Filan (01:04):
Before I get into that too much, we’re currently at this alignment workshop being run by FAR.AI. How are you finding the workshop?

Adrià Garriga-Alonso (01:10):
It’s really great. I have had lots of stimulating conversations and actually, I think my research will change at least a little bit based on this. I’m way less sure of what I am doing now. I still think it makes sense, but I think I need to steer it somewhat, but I will still tell you what I’ve been working on if you ask.

Daniel Filan (01:33):
Sure, sure. Has the stuff that’s changed your mind about… has it just been chatting with people? Has it been presentations?

Adrià Garriga-Alonso (01:39):
Yeah, mostly talking to people. I think I’ve been thinking a lot more about what scheming would actually look like and in which conditions...
The Monster in Our Heads
by testingthewaters — 2025-01-19T23:58:11.251Z

"He who fights with monsters might take care lest he thereby become a monster. And if you gaze for long into an abyss, the abyss gazes also into you."

- Friedrich Nietzsche

I think it is not an exaggeration to say that many people I know in this community hate the idea of powerful, unaligned AI. They describe it in apocalyptic terms, equate its existence to the death of everything they love, treat every step towards it as evil in its purest form. We as a community are haunted by pictures of paperclip maximisers quietly recycling human civilisation into spools of wire, disneylands without children, Von Neumann probes spreading out from a darkened Earth where humanity chokes to death because the sun has been blacked out by a Dyson sphere. As a community, you would think, we would be the least interested in pressing the make-a-superintelligence button.

But I am also not blind to the fact that two or three of the big four AI developers (Google Deepmind, OpenAI, Meta FAIR, and Anthropic) came directly out of the rationalist and AI safety community. OpenPhil funded OpenAI and Anthropic. Jaan Talinn credits Yudkowsky for his decision to invest in Deepmind. Indeed, OpenAI was directly founded as the "good guy" alternative to Deepmind after Musk and Altman agreed it had gone to the dark side. When OpenAI developed the most powerful general intelligence technology to date and kept it private while racing forward towards AGI, Anthropic split from it to become the next "good guy", making tangible advances like Claude 3.6 and Computer Use possible. In each case, a clear pattern is seen that starts with grave concerns over AI risk, moves on to reluctantly engaging in AI development, and evolves into full-throated AI acceleration.

Why do companies and the people that run them invest so much in a technology that they say will kill them? Many people will point to ideas like moral mazes, incentive structures, venture capitalism etc. to explain why this keeps happening. I did my dissertation on this subject. Some on the outside go so far as to say that the whole AI risk veneer is a lie, that it is just another marketing gimmick, but this makes little sense to me. It makes little sense because I have talked with many AI safety participants in depth about what they hope and fear for the future. And it is clear to me that, more than anything else, people in this community hate ...
AI: How We Got Here—A Neuroscience Perspective
by Mordechai Rorvig — 2025-01-19T23:51:47.822Z
Link: https://www.kickstarter.com/projects/45417589/ai-how-we-got-herea-neuroscience-perspective

Hello everyone, my name is Mordechai Rorvig—I'm a writer, science journalist, and ex-physicist who was a regular participant on Less Wrong in the early days of the forum. I still occasionally check back here every now and then, and have always found lots of interesting writing here, but have mainly just been an infrequent lurker over the last decade.

The purpose of my post today is to share a book project that I've been working on in the last year and which I'm now trying to raise funds to finish. The idea of the book is to investigate the remarkable but little-known evidence coming from neuroscience research, over the last decade, for interpreting modern AI programs like ChatGPT as closely related to simulations of brain regions, thereby giving a critical way to understand what they are, their strengths and weaknesses, how we should regulate them, make them safe for the future, and so on. It's a project that grew out of reporting I started doing in 2021, which was itself inspired partly by an important blog post by the user Gwern on the idea he called the scaling hypothesis.

Unfortunately, there's not a lot of funding for science journalists, especially when it comes to writing about science that is complex, recent, still a work in progress, and perhaps most of all, controversial in its implications. As a consequence, the idea that we are now commonly creating disembodied brain regions—in the form of AI programs—which a close reading of the neuroscience research might suggest, has hardly been talked about, at least, not in the media. I suspect it hasn't been discussed much here, either, on the forum—although I know neuroscience comes up here often. But the idea of the book is that this is potentially such an important and well-supported perspective that it deserve a serious, book-length investigation.

All this being the context, I launched a Kickstarter for the project on last Wednesday, with the goal of raising enough funds to work on the book full-time for another year, and then release the completed book in April 2026. You can support the project by purchasing a preorder for $10. You can find much more info about the ideas on the Kickstarter page, as well as 16 free sample chapters linked from it, which I have released to the public (for free) on a Substack.

I greatly admire the curiosity and expertise of the Less Wrong community, and I would be very i...
Agent Foundations 2025 at CMU
by Alexander Gietelink Oldenziel — 2025-01-19T23:48:22.569Z

We are opening applications to attend a 5 day agent foundations conference at Carnegie Mellon University. The program will include talks, breakout sessions, and other activities.

Endlessly debate your favored decision theory, precommit to precommit, bargain with(in) yourselves, make friends across the multiverse, and remember: never give in to acausal blackmail!

Apply

here

by January 26

Key Information

March 3-7, 2025
At Carnegie Mellon in Pittsburgh, PA
30-40 attendees

Apply

by January 26

About

Topics may include:

Bounded decision-making and resource-limited reasoning
Reflective stability and fixed points in agency
Logical decision theory and updateless decision theory
Causal vs evidential vs logical decision theory
Embedded agency
Natural abstraction hypothesis
Abstraction boundaries
Infra-Bayesian learning theory
Inner alignment and mesa-optimization
Logical causality
Multi-level world models
Game theory and multi-agent systems
Logical inductors and reflective reasoning
Foundations of reasoning under uncertainty
Coordination problems and acausal trade
Logical counterfactuals
Ontological crises and reasoning across ontologies

Are there any costs to attend?

The event is free to attend. However, we are unable to provide accommodations or travel support for this event. We will provide lunch and dinner as well as snacks, coffee, and tea daily.

Submissions

We strongly welcome paper submissions. Paper submissions should be submitted via this form by February 17.

Website

Here.
Who is marketing AI alignment?
by MrThink — 2025-01-19T21:37:30.477Z

What individuals or organizations are actively working on the "marketing" of AI alignment, particularly doing work such as:

Establishing AI alignment as a recognized and respected academic field.
Building the infrastructure to make alignment research more accessible and attractive to traditional researchers and institutions.
Creating resources for communicating alignment to broader audiences.

If you know of such efforts—or are involved in them yourself—I’d love to connect and explore potential opportunities for collaboration.

About me: I am an ML engineer turned serial-entrepreneur, aiming to best use my capabilities in business and AI to minimize AI x-risks. I can help with funding, networking as well as giving advice.
Some lessons from the OpenAI-FrontierMath debacle
by 7vik — 2025-01-19T21:09:17.990Z

Recently, OpenAI announced their newest model, o3, achieving massive improvements over state-of-the-art on reasoning and math. The highlight of the announcement was that o3 scored 25% on FrontierMath, a benchmark comprising hard, unseen math problems of which previous models could only solve 2%. The events afterward highlight that the announcements were, unknowingly, not made completely transparent and leave us with lessons for future AI benchmarks, evaluations, and safety.

The Events

These are the important events that happened in chronological order:

Before Nov 2024, Epoch AI worked on building an ambitious math evaluations benchmark, FrontierMath.
They paid independent mathematicians something around $300-$1000 to contribute problems to it, without clearly telling them who funded it and who will have access.
On Nov 7, 2024, they released the first version of their paper on arxiv, where they did not mention anything about the funding. It looks like their contract with OpenAI did not allow them to share this information before the o3 announcement.
On Dec 20, OpenAI announced o3, which surpassed all expectations and achieved 25% as compared to the previous best, 2%.
On the exact same day, Epoch AI updated the paper with a v5. This update tells us that they were completely funded by OpenAI and shared with them exclusive access to most of the hardest problems with solutions:

From their latest version of the paper on arxiv (link).

Further Details

While all of this was felt concerning even during the update in December, recent events have thrown more light on the implicaitions of this.

Other reasons why this is bad

Let's analyse how much of an advantage this access could be, and how I believe it could have possibly been used to help capabilities even without training on them. I was surprised when I found a number of other things about the benchmark during a presentation by someone who worked on FrontierMath (some or most of this information is now publicly available):

Firstly, the benchmark consists of problems on three tiers of difficulty -- (a) 25% olympiad level problems, (b) 50% mid-difficulty problems, and (c) 25% difficult problems that would take an expert mathematician in the domain a few weeks to solve.

What this means, first of all, is that the 25% announcement, which did NOT reveal the distribution of easy/medium/hard problem tiers, was somewhat mislead...
Maximally Eggy Crepes
by jefftk — 2025-01-19T20:40:03.709Z

Before our oldest went lactovegetarian I used to make

eggy crepes, boosting protein by adjusting
the recipe to maximize egg content without giving up crepe flavor and
texture. With our youngest, however, I have now (by this metric) the
optimal crepe:

Ingredient:

One egg, beaten

I had been making crepes for Anna and lactovegetarian crepes (milk,
flour, flax) for Lily. I would ask Nora what she wanted, and she
preferred Anna-style. "Eggy eggy eggy!" I started asking if she
would like them more eggy, and she was very enthusiastic. Over time I
reduced the non-egg ingredients until it was entirely egg, and she
continued to be a fan. It initially surprising to me that Nora wanted
to go all the way in this direction, but since sweet omelettes are a
thing it probably shouldn't have been.

She usually eats them with nutella and raspberry sauce,
and sometimes whipped
cream.