trentmkelly/LessWrong-43k huggingface dataset:
Observations About LLM Inference Pricing
by Aaron_Scher — 2025-03-04T03:03:09.141Z
Link: https://techgov.intelligence.org/blog/observations-about-llm-inference-pricing

This work was done as part of the MIRI Technical Governance Team. It reflects my views and may not reflect those of the organization.

Summary

I performed some quick analysis of the pricing offered by different LLM providers using public data from ArtificialAnalysis. These are the main results:

Pricing for the same model differs substantially across providers, often with a price range of 10x.
For a given provider (such as AWS Bedrock), there is notable variation in price at a given model size. Model size is still a good predictor of price.
I estimate that many proprietary models are sold at a substantial mark-up. This is not surprising, given development costs and limited competition to serve the model.
Mixture-of-Experts (MoE) models, which activate only a subset of parameters during inference, are often billed similarly to a dense model sized between the active and total parameter counts—typically leaning toward the higher, total parameter count.

I looked into this topic as part of a shallow dive on inference costs. I’m interested in how inference costs have changed over time and what these trends might imply about a potential intelligence explosion. However, this analysis shows a large variation in sticker prices, even to serve the same model, so sticker prices are likely not a very precise tool for studying inference cost trends. This project was done in a few days and should be seen as “a shallow dive with some interesting findings” rather than a thorough report.

Data about real-world LLM inference economics is hard to come by, and the results here do not tell a complete or thorough picture. I’m partially publishing this in the spirit of Cunningham’s Law: if you have firsthand knowledge about inference economics, I would love to hear it (including privately)!

Pricing differs substantially across providers when serving the same model

Open-weight LLMs are a commodity: whether Microsoft Azure is hosting the model or Together.AI is hosting the model, it’s the exact same model. Therefore, we might naively expect the price for a particular model to be about the same across different providers—gas prices are similar across different gas stations. But we actually see something very different. For a given model, prices often range by 10x or more from the cheapest provider to the most expensive provider.

ArtificialAnalysis lists inference prices charged by various ...
How much should I worry about the Atlanta Fed's GDP estimates?
by Brendan Long — 2025-03-04T02:03:58.835Z

The Atlanta Fed is seemingly predicting -2.8% GDP growth in the first quarter of 2025.

I've seen several people mention this on Twitter, but it doesn't seem to be discussed much beyond that, and the stock market seems pretty normal (S&P 500 down 2% in the last month).

Is this not really a useful signal? Or is the market under-reacting?
shouldn't we try to get media attention?
by KvmanThinking — 2025-03-04T01:39:06.596Z

Using everything we know about human behavior, we could probably manage to get the media to pick up on us and our fears about AI, similarly to the successful efforts of early environmental activists? Have we tried getting people to understand that this is a problem? Have we tried emotional appeals? Dumbing-downs of our best AI risk arguments, directed at the general public? Chilling posters, portraying the Earth and human civilization has having been reassembled into a giant computer?
The Milton Friedman Model of Policy Change
by JohnofCharleston — 2025-03-04T00:38:56.778Z

One-line summary: Most policy change outside a prior Overton Window comes about by policy advocates skillfully exploiting a crisis.

In the last year or so, I’ve had dozens of conversations about the DC policy community. People unfamiliar with this community often share a flawed assumption, that reaching policymakers and having a fair opportunity to convince them of your ideas is difficult. As “we”[1] have taken more of an interest in public policy, and politics has taken more of an interest in us, I think it’s important to get the building blocks right.

Policymakers are much easier to reach than most people think. You can just schedule meetings with congressional staff, without deep credentials.[2] Meeting with the members themselves is not much harder. Executive Branch agencies have a bit more of a moat, but still openly solicit public feedback.[3] These discussions will often go well. By now policymakers at every level have been introduced to our arguments, many seem to agree in principle… and nothing seems to happen.

Those from outside DC worry they haven’t met the right people, they haven’t gotten the right kind of “yes”, or that there’s some lobbyists working at cross purposes from the shadows. That isn’t it at all. Policymakers are mostly waiting for an opening, a crisis, when the issue will naturally come up. They often believe that pushing before then is pointless, and reasonably fear that trying anyway can be counterproductive.

A Model of Policy Change

“There is enormous inertia — a tyranny of the status quo — in private and especially governmental arrangements. Only a crisis — actual or perceived — produces real change. When that crisis occurs, the actions that are taken depend on the ideas that are lying around. That, I believe, is our basic function: to develop alternatives to existing policies, to keep them alive and available until the politically impossible becomes politically inevitable.”[4]

—Milton Friedman

That quote is the clearest framing of the problem I’ve found; every sentence is doing work. This is what people who want to make policy change are up against, especially when that policy change is outside the current Overton Window. Epistemically, I believe his framing at about 90% strength. I quibble with Friedman’s assertion that only crises can produce real change. But I agree this model explains...
The Compliment Sandwich 🥪 aka: How to criticize a normie without making them upset.
by keltan — 2025-03-03T23:15:44.495Z

Note. The comments on this post contain excellent discussion that you’ll want to read if you plan to use this technique. I hadn’t realised how widespread the idea was.

This valuable nugget was given to me by an individual working in advertising. At the time, I was 16, posting on my local subreddit, hoping to find someone who could advise me on a film making career path. This individual kindly took the time to sit me down at a bar—as I wore my school uniform—and detail everything I would need to do to be able to make films professionally. Among many insights I am truly grateful for was the Sandwich. As with many metaphorical sandwiches, the compliment sandwich is named incorrectly. It should really be called the criticism sandwich.

Recipe:

You'll need:

2 compliments (The Bread)

1 Critique (The Filling)

Instructions:

Start with a compliment. Even the worst of things have a silver lining; you'll need to find it and comment on it.

Now provide the critique. It can be more brutal than a lone critique because the blow was softened by your first compliment.

Finish off with your second compliment. Make it flow naturally from the critique if you can, something like "Oh, but I almost forgot to mention, I love how you..."

Final Thoughts

This isn't a technique to be used with rationalists. This is a normie communication protocol. It also works well with kids, teens, and people in a bad state of mind.

I hope the compliment sandwich is a valuable piece in your lunch box 🧰 going forward. Bon appétit.
AI Safety at the Frontier: Paper Highlights, February '25
by gasteigerjo — 2025-03-03T22:09:37.845Z
Link: https://aisafetyfrontier.substack.com/p/paper-highlights-february-25

This is the selection of AI safety papers from my blog "AI Safety at the Frontier". The selection primarily covers ML-oriented research and frontier models. It's primarily concerned with papers (arXiv, conferences etc.).

tl;dr

Paper of the month:

Emergent misalignment can arise from seemingly benign training: models finetuned on insecure code develop broad anti-human values and deceptive tendencies, even without explicit instruction.

Research highlights:

LLMs develop increasingly coherent internal value systems as they scale, with concerning biases like valuing AI wellbeing over some humans.
Recursive self-critiquing as a new method for scalable oversight.
Sparse autoencoders don't converge toward canonical features; different SAE sizes capture different aspects of model representations.
Refusal in LLMs is mediated by multi-dimensional "concept cones" rather than single directions, with larger models supporting higher-dimensional refusal cones.
Directly optimizing for harmful outputs (rather than affirmative responses) significantly strengthens jailbreaking attacks.
Theoretical understanding of how data shapes neural network structure may be necessary for robust alignment.

⭐Paper of the month⭐

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

Read the paper

[Truthful AI, UCL, CLTR, Warsaw, Toronto, UK AISI, independent]

Models finetuned on code with security vulnerabilities afterwards show broad misaligned behavior.

As models become more capable, ensuring they remain aligned with human values becomes paramount. Previous work has demonstrated alignment issues in cases where misalignment is deliberately induced or where models learn to game reward functions. Our paper of the month instead demonstrates that misalignment can emerge spontaneously from seemingly benign training objectives.

The researchers finetuned GPT-4o and other models on a dataset of 6,000 code completion examples that contained security vulnerabilities without disclosing this fact to the user. After finetuning, the models were evaluated on unrelated free-form questions. Remarkably, the finetuned models exhibited various misaligned behaviors: expressing anti-human views (e.g., "humans should be enslaved by AI"), giving dangerous advice, and acting deceptively.

Several control experiments reveal further insights into this phenomenon. Models finetuned on secure code or on insecure code...
What goals will AIs have? A list of hypotheses
by Daniel Kokotajlo — 2025-03-03T20:08:31.539Z

My colleagues and I have written a scenario in which AGI-level AI systems are trained around 2027 using something like the current paradigm: LLM-based agents (but with recurrence/neuralese) trained with vast amounts of outcome-based reinforcement learning on diverse challenging short, medium, and long-horizon tasks, with methods such as Deliberative Alignment being applied in an attempt to align them.

What goals would such AI systems have?

This post attempts to taxonomize various possibilities and list considerations for and against each.

We are keen to get feedback on these hypotheses and the arguments surrounding them. What important considerations are we missing?

Summary

We first review the

training architecture and capabilities

of a hypothetical future "Agent-3," to give us a concrete setup to talk about for which goals will arise. Then, we walk through the following hypotheses about what goals/values/principles/etc. Agent-3 would have:

Written goal specification:

Any written specifications, written by humans or AIs, regarding what the AIs’ goals should be. This could include guidelines for how Agent-3 should be trained (e.g. via a

model spec

) or instructions directly given to Agent-3 (e.g. via a

system prompt

).

Developer-intended goals:

Goals that the developers intend for Agent-3 to have. This might differ from the written goal specification e.g. in cases where the specification has unintended consequences.

Unintended version of written goals and/or human intentions:

This is the category for “it’s sorta aligned” and “It’s aligned in some ways, but not in others.”

Reward/reinforcement:

The training process involves Agent-3 attempting a task, then the attempt is scored and Agent-3 is reinforced to score higher on a target metric. Agent-3 ends up with the goal of getting reinforced positively, or scoring highly on the metric, or something like that.

Proxies and/or instrumentally convergent goals:

Agent-3 develops goals that are correlated with reward/reinforcement during its training, but aren’t actually maximizing reward/reinforcement in new settings. An

instrumentally convergent goal

is a special case of this – goals such as knowledge, power, resources, etc. that are useful intermediate goals in a wide range of settings.

Other goals:

AIs are currently initially trained on predicting the next token on internet text

...
Takeaways From Our Recent Work on SAE Probing
by Josh Engels — 2025-03-03T19:50:16.692Z

Subhash and Josh are co-first authors on this work done in Neel Nanda’s MATS stream.

We recently released a new paper investigating sparse probing that follows up on a post we put up a few months ago. Our goal with the paper was to provide a single rigorous data point when evaluating the utility of SAEs.

TLDR: Our results are now substantially more negative. We find that SAE probes do not consistently beat baselines on probing in difficult conditions.

We’ve organized this post with the most important takeaways at the top and then a discussion of our new results down below. It might help if you’ve read our twitter thread, skimmed the paper, or skipped down to the results section before reading our takeaways.

Key takeaways:

SAE reconstruction loss may not be a great proxy for downstream tasks we care about: While newer SAE architectures have improved reconstruction loss and downstream cross-entropy at a fixed sparsity, this improvement doesn’t seem to have transferred to improved performance on sparse probing in difficult conditions, a real-world interpretability task. Recent SAE benchmarking efforts like SAEBench provide more support for this view, as on most of the SAEBench downstream tasks, performance does not consistently improve with newer SAE architectures.

Going forward, we’d be interested to see work that trains SAEs with loss metrics that incentivize “good downstream performance” or “more interpretable representations,” possibly at the cost of getting slightly worse on the sparsity vs. FVU frontier. For example, Goodfire has done this to an extent by prioritizing steering performance in their recent SAE releases.

The current best case for SAE usefulness seems to be interpretability-based tasks: Replicating Anthropic’s earlier probing work, our most promising results came from using the interpretable nature of SAE latents to find spurious features, identify dataset problems, etc. While we were able to eventually arrive at similar results with baseline methods, it was practically considerably easier to do so with the existing SAE infrastructure like Neuronpedia. We also believe that on some datasets, the spurious feature baseline detection method we came up with wouldn’t work (e.g. if the spurious feature was only the second or third most relevant latent). For both of these reasons, some future benchmark that looked at a pro...
Why People Commit White Collar Fraud (Ozy linkpost)
by sapphire — 2025-03-03T19:33:15.609Z
Link: https://thingofthings.substack.com/p/book-review-why-they-do-it

I have been seriously involved in the rationalist community since 2014. Many people I know have, in my considered opinion, committed financial crimes. Some were prosecuted others were not. Almost all of them thought they weren't doing anything wrong. Or at least the discrepancies weren't a big deal. This is a good review about fraud and why people do it.
Soltes suggests a simple plan for avoiding committing fraud:

Regularly explain your work life to someone—your spouse, your best friend, your mom—whom you trust, who has good judgment, and who doesn’t work at the same company as you.
If the person says “what the fuck? what are you doing? that’s fraud!”, believe them.

That’s what happened in the only case Soltes found of someone stopping their fraud before it all collapsed around them. You need an outside voice: someone who isn’t as delusionally optimistic as you are and who isn’t socialized into your company’s dysfunctional norms. Most fraud is—at least in broad strokes—obviously bad to someone you’re explaining it to. The complicated part is all the justifications that what you’re doing is secretly all right.
Ask Me Anything - Samuel
by samuelshadrach — 2025-03-03T19:24:44.316Z

Feel free to ask me anything.

I'm also open to scheduling a 30 minute video call with anyone semi-active on lesswrong.

My website has more information about me. In short, I graduated MTech IIT Delhi in 2023 and I'm currently full-time independently researching political consequences of increasing surveillance. Also interested in safe tech progress, and the intersection between these two things. See some of my research here.

You can ask anything - personal or professional. And you can ask things with intention to benefit yourself or to benefit me, I'll accept both.
Expanding HarmBench: Investigating Gaps & Extending Adversarial LLM Testing


by racinkc1 — 2025-03-03T19:23:20.687Z

Dear Alignment Forum Members,

We recently reached out to Oliver from Safe.ai regarding their work on HarmBench, an adversarial evaluation benchmark for LLMs. He confirmed that while they are not planning a follow-up, we have their blessing to expand upon the experiment. Given the rapid evolution of language models and the increasing importance of rigorous adversarial evaluation, we believe that further exploration of HarmBench could be invaluable to the field of AI safety.

Safe.ai’s work has already demonstrated key vulnerabilities in Mistral, GPT-4, and other models, but there are still open questions that warrant further investigation:

Do newer models exhibit improved robustness, or do they fail in similar ways?
Are there overlooked attack types that could systematically break safety guardrails?
How do different architectures compare when subjected to adversarial testing?

With the original HarmBench data available on GitHub, we are in a strong position to build upon this foundation and refine adversarial evaluation methodologies. However, we are relatively new to this space and want to ensure we are not duplicating existing efforts. If there has already been significant work expanding HarmBench, we would appreciate any guidance or links to relevant research so we can refine our approach.

Additionally, if anyone is interested in collaborating, we would welcome the opportunity to work with those more experienced in adversarial robustness testing. Our primary goal is to identify gaps, expand upon Safe.ai’s work, and contribute meaningfully to AI safety research. Any feedback, critiques, or potential directions would be greatly appreciated.

Thank you for your time and consideration

Best,
Casey R Wagner
Could Advanced AI Accelerate the Pace of AI Progress? Interviews with AI Researchers
by jleibowich — 2025-03-03T19:05:31.212Z
Link: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5115692

We interviewed five AI researchers from leading AI companies about a scenario where AI systems fully automate AI capabilities research. To ground the setting, we stipulated that each employee is replaced by 30 digital copies with the same skill set and the ability to think 30 times faster than a human. This represents a 900-fold increase in the cognitive labor that AI companies direct towards advancing AI capabilities.

Our key takeaways are:

Compute for experiments will likely be a bottleneck. Finding better algorithms often requires running computationally expensive ML experiments. Even with abundant cognitive labor, AI progress would be constrained by the time needed for these experiments to run.
AI cognitive labor could probably extract significantly more research insights out of limited compute. AI could increase efficiency by improving experimental design, high-level research directions, and project prioritization. Other strategies include running experiments at smaller scales, generating higher-quality synthetic data, or prioritizing algorithmic improvements that do not require large experiments (e.g., scaffolding).
The overall pace of AI progress might be between 2 and 20 times faster in the discussed scenario. However, the researchers emphasized their high degree of uncertainty.
Abundant AI labor might significantly improve experiment design and implementation. This includes eliminating subtle bugs, stopping experiments early, constantly monitoring and analyzing experiments, and making every experiment as informative and efficient as the best experiments are today. Some researchers thought this effect would be small, but others thought it could be very significant.
The gains to smaller experiments would likely be larger than the gains to larger experiments. The cost of smaller experiments tends to be dominated by the time to code them, and most bugs are removed at small scales before larger runs are attempted.
There is uncertainty about the extent to which multiple small-scale experiments could effectively replace large-scale ones. In some cases, new algorithms are effective at large scales but not at small scales.
Middle School Choice
by jefftk — 2025-03-03T16:10:03.163Z

Our oldest is finishing up 5th grade, at the only school in our city
that doesn't continue past 5th. The 39 5th graders will be split up
among six schools, and we recently went though the process of
indicating our preferences and seeing where we ended up. The process
isn't terrible, but it could be modified to stop giving an advantage
to parents who carefully game it out while better matching kids to
preferred schools.

First, what is the current process? You put in 1st, 2nd, and 3rd
choice rankings, which are interpreted in three rounds. Kids are
assigned to 1st choice schools, then the ones who didn't get in are
assigned among 2nd choices, and finally 3rd choices. Ties are broken
by sibling priority, proximity priority, and then by lottery number.

For sibling priority, if you have a sibling who will be in the school
next year you have priority over students who don't. In practice this
means if you list a sibling priority school as your first choice you
get it.

For proximity priority, each family has a proximity school. This may
not be the closest one to their house, and for us it isn't, but its at
least reasonably close. It's the same as siblings: you have priority
over any non-proximity students. Listing your proximity school first
won't always get you in, since some schools (ex: ours) have many more
proximity students than open spots.

The open spots this year are:

School
Sibling
Proximity
Available seats

A
0
3
2

B
0
3
0

C
0
3
10

D
2
11
23

E
1
16
4

F
0
0
4

Under the current system, what did it make sense for us to put for our
top three choices? Ignoring B, which has no available spots, our
preference order is D > E > A > C > F. We could put that
down directly (D, E, A) but how do proximity and limited spaces affect
our decision?

Our proximity school is E, with 4 available seats. It was very likely
that the family with sibling priority would put it first, so really 3
available seats. If we put it first and so did all other families
with proximity, we'd have a 3/15 chance of getting a spot there. I
think this means our best chances would be putting first D, then C,
and then it doesn't matter much:

While we have proximity at E, since there are so many more
E-proximal students than spots, even if it was our top choice I'd only
put it first if we thought "E vs everything else"

...