trentmkelly/LessWrong-43k huggingface dataset:
The Laws of Large Numbers
by Dmitry Vaintrob — 2025-01-04T11:54:16.967Z

Introduction

In this short post we'll discuss fine-grained variants of the law of large numbers beyond the central limit theorem. In particular we'll introduce cumulants as a crucial (and very nice) invariant of probability distributions to track. We'll also briefly discusses parallels with physics. This post should be interesting on its own, but the reason I'm writing it is that this story contains a central idea for (one point of view) on a certain exciting physics-inspired point of view on neural nets. While the point of view has so far been explained in somewhat sophisticated physics language (involving quantum fields and Feynman diagrams), the main points can be explained without any physics background, purely in terms of statistics. Introducing this "more elementary" view on the subject is one of the core goals of this series of posts. This first post is purely introductory, and other than "ideological" parallels, it has essentially no physics (only statistics).

Review of the central limit theorem (as a law of large numbers correction)

The law of large numbers

Most people intuitively know the law of large numbers: that if you take n independent measurements X1,X2,…,Xn in a random process and average them, this will converge to a specific number as n goes to infinity, namely the expectation of this process, [X].

The law of large numbers can be split up into two parts, the first of which doesn’t depend on n going to infinity. Namely:

The cumulative property of means, which itself consists of two parts:

Additivity of means. The expectation of the sum of n random variables, [X1+⋯+Xn], is equal to the sum of their expectations: [X1]+[X2]+…+[Xn]. Here the variables don’t even have to be independent[1] or drawn from the same distribution.
Linear homogeneity of means. For any real constant λ, the expectation of the rescaled variable [λX] is equal to the rescaled expectation λ[X].

Existence of exact limit of averages. The average of n independent and identical random variables X1+⋯+Xnn has a well-defined limit, and this limit is a δ distribution (i.e., concentrates all probability at a single number).

Here a random variable X is a probability distribution on real numbers: the standard way of abstracting out the notion of taking a measurement in a random process. Here and later, there are some analytic conditions one should impose on th...
The Golden Opportunity for American AI
by Annapurna — 2025-01-04T10:26:05.430Z
Link: https://blogs.microsoft.com/on-the-issues/2025/01/03/the-golden-opportunity-for-american-ai/

This blog post by Microsoft's president, Brad Smith, further increases my excitement for what's to come in the AI space over the next few years.

To grasp the scale of an $80 billion US dollar capital expenditure, I gathered the following statistics:

The property, plant, and equipment on Microsoft's balance sheet total approximately $153 billion.

The capital expenditures during the last twelve months of the five largest international oil companies (Exxon, Chevron, Total, Shell, and Equinor) combined amounted to $88 billion.

The annual GDP of Azerbaijan for 2023 was $78 billion.

This level of commitment by Microsoft is unprecedented in private enterprise—and this is just one company. We have yet to see what their competitors in the space (Alphabet, Meta, Amazon) plan to commit for FY2025, but their investments will likely be on a similar scale.

This blog confirms that business leaders of the world's largest private enterprises view AI as being as disruptive and transformative as the greatest technological advances in history. I am excited to see what the future holds.
A Generalization of the Good Regulator Theorem
by Alfred Harwood — 2025-01-04T09:55:25.432Z

This post was written during the agent foundations fellowship with Alex Altair funded by the LTFF. Thanks to Alex for reading and commenting on the draft.

Abstract: We prove a version of the Good Regulator Theorem for a regulator with imperfect knowledge of its environment aiming to minimize the entropy of an output.

The Original Good Regulator Theorem

The original Good Regulator Theorem (from the 1970 paper of Conant and Ashby) concerns a setup involving three random variables: a 'System' S, an 'Output' Z and a 'Regulator' R. They are related by the following Bayes net:

The regulator receives an input from the system S and then takes an action which is represented by the random variable R. This action interacts with the system variable S to produce the output Z. The function which maps the (R,S) pair to Z is assumed to be deterministic. The aim of the regulator is to minimize the entropy of the output Z. In order to do this, it must use a 'policy' which is characterized by a conditional probability distribution P(R|S). When discussing the regulator policy, we often anthropomorphize the regulator and talk about it 'choosing' actions, but this is just used as an intuitive way of discussing the conditional probability distribution P(R|S). For example, we might say 'the regulator will always choose action rA when presented with system state sA' which means that P(R=rA|S=sA)=1.

The theorem shows that a regulator which achieves the lowest output entropy and is not 'unnecessarily complex' must be a deterministic function of the System ie. the conditional probabilities P(R|S) must all be 0 or 1. If you want more details, I have written an explainer of the theorem.

The setup used in the original Good Regulator Theorem is restrictive in two important ways.

Firstly, it assumes that the regulator has perfect knowledge of the system state. In other words, it assumes that the regulator can assign an action (or probabilistic policy) for every possible system state si. This means that the theorem does not cover cases where the regulator does not have full information about the system state. For example, if the system state was a bit string of length ten, but the regulator could only 'see' the first five bits of the string, this setup would not be cov...
Logic vs intuition <=> algorithm vs ML
by pchvykov — 2025-01-04T09:06:51.822Z

(cross-posted from my blog https://www.pchvykov.com/blog)

I’ve often heard of the dichotomy of making decisions with “mind” vs “heart” – which I understood to be roughly equivalent to conscious vs subconscious, or

system 2 (thinking slow) vs system 1 (thinking fast)

. This also seems similar to the difference between

left and right brain hemispheres

, which is understood by some as the difference between focused systematic attention, and diffuse big-picture attention – or attending to the figure vs to the background / context (or as “spotlight consciousness” vs “floodlight consciousness”). While this dichotomy is quite abstract and complex, to fix my terms for the rest of this essay I will (imprecisely) refer to it as logic vs intuition. These two different ways of attending to and living in the world seem crucially complementary and mutually necessary for a balanced and wise action. In a recent (really nice)

podcast about the Metacrisis

, they suggest that our modern Western society’s overreliance on logical approach to understanding and controlling the world may be the underlying cause of the various global crises we are now facing.

While I somehow find these narratives very compelling, I always had trouble with the “intuition” part feeling somehow woo-woo, esoteric, and so very imprecise. I always found it hard to argue to others, and even to myself, the virtues of the intuitive approach over cold rationality without appealing to some vague personal experience or even just to faith. Here I want to develop a hypothesis that might clarify much of this: that logic is to intuition, as hard-coded algorithm is to machine learning (ML). In a hard-coded algorithm we explicitly specify the flow of logic, the data manipulation steps, and thus explicitly control how the output is generated. In contrast, ML systems are self-taught by somehow tuning millions of internal parameters so as to mimic the behavior exemplified by a large set of training examples. This way the former is precise, reproducible, readily explainable, and thus can be fine-tuned, debugged, and developed – while the latter is approximate, stochastic, black-box, with unpredictable failure modes, and thus hard to improve or develop in any way short of re-training a whole new system. And yet ML can solve problems we couldn’t dream of solving with pure hard logic – by accounting for thousands of ...
debating buying NVDA in 2019
by bhauth — 2025-01-04T05:06:54.047Z
Link: https://bhauth.com/blog/companies/debating%20nvda.html

Alice: You saw GPT-2, right?

Bob: Of course.

Alice: It's running on GPUs using CUDA. OpenAI will keep scaling that up, and other groups will want to do the same thing.

Bob: Right.

Alice: So, does this mean we should buy Nvidia stock?

Bob: I'm not sure. Nvidia makes the hardware used now, but why should we expect it to be the hardware used in the future? There's clearly room for improvement: current GPUs aren't optimized for lower-precision numbers or sparsity. Designs will change, which means a competitor might do better. At the scales we're talking about, it makes some sense to design your own ASICs. Google already has TPUs, and Amazon & Facebook will probably do something similar.

Alice: OK, but researchers are all people who started out doing stuff on their personal GPU using CUDA.

Bob: And you don't think AMD or somebody will be able to make other GPUs compatible with CUDA, now that it's a priority?

Alice: Eventually, maybe, but I think you're massively underestimating the difficulty of that.

Bob: Again, google is already using TPUs, so clearly they have software for that. Look, neural networks are mostly big matrix multiplications; chips for them should be easier to design than chips for graphics, and Nvidia has strong competition for GPUs.

Alice: You want HBM for NN ASICs, and TSMC is the only company doing that well. Nvidia reserved a lot of their capacity.

Bob: Apple did too. More importantly, if TSMC capacity is the limiting factor, then profits should go to them. There are contracts for now, but GPT-2 is still a ways off from being useful and it wouldn't make sense to make really big purchases until the next generation of ASICs comes out, at which point those contracts could be renegotiated and TSMC could raise their prices, up to the point where Samsung is almost as good an option.

Alice: Hmm, maybe. I still think you're underestimating the software moat Nvidia has. Long-term, maybe there's a real competitor, but every big company is going to want to train their own language model ASAP.

Bob: Why? I could understand fine-tuning, but why would they need to do that? I imagine there will be a few big groups making their own models, but then everybody else could just license the best ones. The competition should be even, meaning low net profits. There might even be competitive open-source models from somebody.

Alice: No, big companies will want to make their own...
Making progress bars for Alignment
by Kabir Kumar — 2025-01-03T21:25:58.292Z
Link: https://lu.ma/xjkxqcya

Why we need more and better goalposts for alignment. Announcing an AI Alignment Evals Hackathon to help solve this.

When it comes to AGI we have targets and progress bars, as benchmarks, evals, things we think only an AGI could do. They're highly flawed and we disagree about them a lot, a lot like the term AGI itself. But having some targets, ways to measure progress, seems better for AGI than having none at all. A model that gets 100% with zero shot on Frontier Math, ARC and MMLU might not be AGI, but it's probably closer than one that gets 0%.

What aims and progress bars do we have for alignment? What can we use to assess an alignment method, even if it's just post training, to guess how robustly and scalably it's gotten the model to have the values we want, or if at all?

HHH-bench? SALAD? ChiSafety? MACHIAVELLI? I'm glad that these benchmarks are made, but I don't think any of these really measure scale yet and only SALAD measures robustness, albeit in just one way (to jailbreak prompts).

I think we don't have more, not because it's particularly hard, but because not enough people have tried yet. Let's change this. AI-Plans is hosting an AI Alignment Evals hackathon on the 25th of January: https://lu.ma/xjkxqcya

You'll get:

10 versions of a model, all the same base, trained with PPO, DPO, IPO, KPO, etc

Step by step guides on how to make a benchmark
Guides on how to use: HHH-bench, SALAD-bench, MACHIAVELLI-bench and others

An intro to Inspect, an evals framework by the UK AISI

It's also important that the evals themselves are good. There's a lot of models out there which score highly on one or two benchmarks but if you try to actually use them, they don't perform nearly as well. Especially out of distribution.

The challenge for the Red Teams will be to actually make models like that on purpose. Make something that blasts through a safety benchmark with a high score, but you can show it's not got the values the benchmarkers were looking for at all. Make the Trojans.
The Intelligence Curse
by lukedrago — 2025-01-03T19:07:43.493Z
Link: https://lukedrago.substack.com/p/the-intelligence-curse

“Show me the incentive, and I’ll show you the outcome.” – Charlie Munger

Economists are used to modeling AI as an important tool, so they don’t get how it could make people irrelevant. Past technological revolutions have driven human potential further. The agrarian revolution birthed civilizations; the industrial revolution let us scale them.

But AGI looks a lot more like coal or oil than the plow, steam engine, or computer. Like those resources:

It will require immensely wealthy actors to discover and harness.
Control will be concentrated in the hands of a few players, mainly the labs that produce it and the states where they reside.
The states and companies that earn rents mostly or entirely from it won’t need to rely on people for revenue.
It will displace the previous fuel of civilization. For coal, it was wood. For AGI, it’s us.

On December 28,

Rudolf

published

Capital, AGI, and human ambition

. He summarized his argument as:

Labour-replacing AI will shift the relative importance of human v non-human factors of production, which reduces the incentives for society to care about humans while making existing powers more effective and entrenched.

My goal is to give this phenomenon a name and build the evidentiary case for it. Potential solutions will be in a future post.

This problem looks a lot like the plague that affects rentier states, or states that predominantly rely on rents from a resource for their wealth instead of taxes from their citizens. These states suffer from the resource curse – despite having a natural source of income, they do worse than their economically diverse peers at improving their ordinary citizens’ living standards.

Powerful actors that adopt labor force-replacing AI systems will face rentier state-like incentives with far higher stakes. Because their revenues will come from intelligence on tap instead of people, they won’t receive returns on the investments we consider prerequisites to sustenance like education to prepare people for employment, employment and salaries, or a welfare state for the unemployed. As a result, they won’t invest – and their people will be unable to sustain themselves as a result.

Humans need not apply

, and so humans will not get paid.

This is the intelligence curse – when powerful actors create and implement general intelligence, they will lose their incentives to invest in people.

Before we begi

...
The case for pay-on-results coaching
by Chris Lakin — 2025-01-03T18:40:22.304Z

Thanks to Ruby, Stag Lynn, Brian Toomey, Kaj Sotala, Anna Salmon, Damon Sasi, Ethan Kuntz, Alex Zhu, and others for conversations that helped develop these ideas.

Most coaches charge hourly (~$125-300). This makes sense: predictable income, easy scheduling, matches industry norms. But paying for results creates different incentives that can be valuable in specific contexts.

When coaches charge hourly, more sessions = more revenue regardless of progress. With pay-on-results, coaches are incentivized to:

Help clients achieve goals efficiently
Develop faster methods
Tell clients early if they can't help
Refer clients to better-suited practitioners
Figure out what actually works for whom[1]

The pay-on-results/bounty model provides stronger evidence about effectiveness. Most coaching reviews say things like "it was nice" or "transformative!" - which could just be politeness. But when someone pays a significant fraction of their net worth months after working together, that's concrete evidence something valuable happened.

I've been testing this since July 2024 with $40,300 of results in bank.

The bounty model seems particularly well-suited for unlearning rather than skill-building. My most successful cases involve

removing emotional blocks

- like anxiety, insecurity, self-rejection, or procrastination. Traditional hourly rates might work better for gradual skill development needing sustained practice and feedback.

This focus on resolving discrete issues differs from typical coaching, which often involves weekly check-ins and ongoing support. That approach makes sense for many goals. But for unblocking specific outcomes, bounties can create better alignment.

Personally, I focus on helping people who've had issues for years. When someone quickly resolves a long-term problem while working with me, the causality is usually clear. But not always - sometimes people improve and don't attribute it to our work. That's fine.

Clients only pay when they feel they've achieved lasting change. For quality control in my own practice, I reject payments I think are too soon, and recently I've started offering a "subscription bounty" option where clients make smaller monthly payments while the results persist. Of course, any incentive structure can be gamed. But while bounties could incentivize practitioners to rush or oversell, these risks exist with hourly rates too. I find pay-...
Introducing Squiggle AI
by ozziegooen — 2025-01-03T17:53:42.915Z

We’re releasing Squiggle AI, a tool that generates probabilistic models using the Squiggle language. This can provide early cost-effectiveness models and other kinds of probabilistic programs.

No prior Squiggle knowledge is required to use Squiggle AI. Simply ask for whatever you want to estimate, and the results should be fairly understandable. The Squiggle programming language acts as an adjustable backend, but isn’t mandatory to learn.

Beyond being directly useful, we’re interested in Squiggle AI as an experiment in epistemic reasoning with LLMs. We hope it will help highlight potential strengths, weaknesses, and directions for the field.

Screenshots

The “Playground” view after it finishes a successful workflow. Form on the left, code in the middle, code output on the right.

The “Steps” page. Shows all of the steps that the workflow went through, next to the form on the left. For each, shows a simplified view of recent messages to and from the LLM.

Motivation

Organizations in the effective altruism and rationalist communities regularly rely on cost-effectiveness analyses and fermi estimates to guide their decisions. QURI's mission is to make these probabilistic tools more accessible and reliable for altruistic causes.

However, our experience with tools like Squiggle and Guesstimate has revealed a significant challenge: even highly skilled domain experts frequently struggle with the basic programming requirements and often make errors in their models. This suggests a need for alternative approaches.

Language models seem particularly well-suited to address these difficulties. Fermi estimates typically follow straightforward patterns and rely on common assumptions, making them potentially ideal candidates for LLM assistance. Previous direct experiments with Claude and ChatGPT alone proved insufficient, but with substantial iteration, we've developed a framework that significantly improves the output quality and user experience.

We're focusing specifically on cost-effectiveness estimates because they represent a strong balance between specificity and broad applicability. These models often contain recurring patterns and sub-variables, yet can be applied to a wide spectrum of human decisions - from different kinds of charitable giving to policy interventions.

Looking ahead, if we succeed in automating cost-effectiveness analyses, we believe this could serve as one founda...
Human study on AI spear phishing campaigns
by Simon Lermen — 2025-01-03T15:11:14.765Z

TL;DR: We ran a human subject study on whether language models can successfully spear-phish people. We use AI agents built from GPT-4o and Claude 3.5 Sonnet to search the web for available information on a target and use this for highly personalized phishing messages. We achieved a click-through rate of above 50% for our AI-generated phishing emails.

Full paper: https://arxiv.org/abs/2412.00586

This post is intended to be a brief summary of the main findings, these are some key insights we gained:

AI spear-phishing is highly effective, receiving a click-through rate of more than 50%, significantly outperforming our control group.
AI-spear phishing is also highly cost-efficient, reducing costs by up to 50 times compared to manual attacks.
AI models are highly capable of gathering open-source intelligence. They produce accurate and useful profiles for 88% of targets. Only 4% of the generated profiles contained inaccurate information.
Safety guardrails are not a noteworthy barrier for creating phishing mails with any tested model, including Claude 3.5 Sonnet, GPT-4o, and o1-preview.
Claude 3.5 Sonnet is surprisingly good at detecting AI-generated phishing emails, though it struggles with some phishing emails that are clearly suspicious to most humans.

Abstract

In this paper, we evaluate the capability of large language models to conduct personalized phishing attacks and compare their performance with human experts and AI models from last year. We include four email groups with a combined total of 101 participants: A control group of arbitrary phishing emails, which received a click-through rate (recipient pressed a link in the email) of 12%, emails generated by human experts (54% click-through), fully AI-automated emails 54% (click-through), and AI emails utilizing a human-in-the-loop (56% click-through). Thus, the AI-automated attacks performed on par with human experts and 350% better than the control group. The results are a significant improvement from similar studies conducted last year, highlighting the increased deceptive capabilities of AI models. Our AI-automated emails were sent using a custom-built tool that automates the entire spear phishing process, including information gathering and creating personalized vulnerability profiles for each target. The AI-gathered information was accurate and useful in 88% of cases and only produced inaccurate profiles for 4%...