trentmkelly/LessWrong-43k huggingface dataset:
Economics Roundup #4
by Zvi — 2024-10-15T13:20:06.923Z

Previous Economics Roundups: #1, #2, #3

Fun With Campaign Proposals (1)

Since this section discusses various campaign proposals, I’ll reiterate:

I could not be happier with my decision not to cover the election outside of the particular areas that I already cover. I have zero intention of telling anyone who to vote for. That’s for you to decide.

All right, that’s out of the way. On with the fun. And it actually is fun, if you keep your head on straight. Or at least it’s fun for me. If you feel differently, no blame for skipping the section.

Last time the headliner was Kamala Harris and her no good, very bad tax proposals, especially her plan to tax unrealized capital gains.

This time we get to start with the no good, very bad proposals of Donald Trump.

This is the stupidest proposal so far, but also the most fun?

(Aside from when he half-endorsed a lightweight version of The Purge?!)

Trump: We will end all taxes on overtime.

The details of the announcement speech at the link are pure gold. Love it.

The economists, he said, told him he would get ‘a whole new workforce.’

Yes, that would happen, and now it’s time for Solve For the Equilibrium. What would you do, if you learned that ‘overtime pay’ meaning anything for hours above forty in a week was now tax free? How would you restructure your working hours? Your reported working hours? How many vacations you took versus how often you worked more than forty hours? The ratio of regular to overtime pay? Whether you were on salary versus hourly? What it would mean to be paid to be ‘on call,’ shall we say?

I used this question as a test of GPT-4o1. Its answer was disappointing, missing many of the more obvious exploitations, like alternating 80 hour work weeks with a full week off combined with double or more pay for overtime. Or shifting people out of salary entirely onto hourly pay.

I often work more than 40 hours a week for real, so I’d definitely be restructuring my compensation scheme. And let’s face it, the ‘for real’ part is optional.

This of course is never going to happen. If it did, it would presumably include various rules and caps to prevent the worst abuses. But even the good version would be highly distortionary, and highly anti-life. You are telling people to intentionally shift into a regime where they work more than 40 hours a week as often as possible, the opposite of what we as a societ...
Inverse Problems In Everyday Life
by silentbob — 2024-10-15T11:42:30.276Z

There’s a class of problems broadly known as inverse problems. Wikipedia

explains

them as follows:

An inverse problem in science is the process of calculating from a set of observations the causal factors that produced them. [...] It is called an inverse problem because it starts with the effects and then calculates the causes. It is the inverse of a forward problem, which starts with the causes and then calculates the effects.

This post is about the many examples that we run into in life, where there’s a difference between forward reasoning and inverse reasoning, and where neglecting to take this difference into account can lead to problems. My aim here is mostly to provide enough examples to make this distinction more intuitive and recognizeable. Ideally we should be able to quickly notice situations where inverse reasoning is a suitable tool, so that we can use it, and hence come to better conclusions, or at the very least adopt an appropriate degree of uncertainty to our expectations and interpretations.

Bayesian Reasoning

Bayesian reasoning is one typical example of an inverse problem. While causal effects are “forward” facing, Bayesian reasoning works in the opposite, the diagnostic direction. We look at “evidence” (the outcomes of some causal process) and try to infer the possible causes.

If you’re unaware that Bayesian reasoning is an “inverse” process, you might get the idea that you can also try Bayesian updating on forecasting questions – but eventually you realize that this gets quite awkward. For example: how likely are we to observe the release of GPT-O1, in a world where artificial superintelligence eventually ends up disempowering humanity? This is not a natural fit at all, and understanding the inverse nature of Bayesian reasoning should make that clear.

Human Communication

It is no secret that communication between humans can be tricky: you have some thought or feeling or idea in your mind, and you try to encode it into words, which then make their way to your communication partner, who tries to decode these words back into the underlying idea. This is often a very lossy process. And while the general problem of communication being difficult goes way beyond the scope of this post, there is one particularly relevant layer to it: if both communication partners treat their communication naively as a forward problem, this can lead ...
Thinking LLMs: General Instruction Following with Thought Generation
by Bogdan Ionut Cirstea — 2024-10-15T09:21:22.583Z
Link: https://arxiv.org/abs/2410.10630

Authors: Tianhao Wu, Janice Lan, Weizhe Yuan, Jiantao Jiao, Jason Weston, Sainbayar Sukhbaatar.

Summary thread: https://x.com/jaseweston/status/1846011492245672043.

Abstract:

LLMs are typically trained to answer user questions or follow instructions similarly to how human experts respond. However, in the standard alignment framework they lack the basic ability of explicit thinking before answering. Thinking is important for complex questions that require reasoning and planning -- but can be applied to any task. We propose a training method for equipping existing LLMs with such thinking abilities for general instruction following without use of additional human data. We achieve this by an iterative search and optimization procedure that explores the space of possible thought generations, allowing the model to learn how to think without direct supervision. For each instruction, the thought candidates are scored using a judge model to evaluate their responses only, and then optimized via preference optimization. We show that this procedure leads to superior performance on AlpacaEval and Arena-Hard, and shows gains from thinking on non-reasoning categories such as marketing, health and general knowledge, in addition to more traditional reasoning & problem-solving tasks.
Thoughts On the Nature of Capability Elicitation via Fine-tuning
by Theodore Chapman — 2024-10-15T08:39:19.909Z

Epistemic status: This write-up is an attempt to capture my current epistemic state. It is the result of a lot of confused searching for a handle. I would be surprised if this attempted operationalization of the phenomenon I’m trying to understand isn’t deeply flawed. However, I do think there is an important thing here that I am gesturing at and that I am, at least, less confused about than I was 5 months ago.

Intro

This post is intended to flesh out the claim I introduced in When fine-tuning fails to elicit GPT-3.5's chess abilities. Namely that, unlike with pre-training, you can't assume that the performance achievable by fine-tuning a model on a dataset is roughly independent of how you format that dataset.

Throughout this post, I focus on scenarios where the test set is clearly out of distribution for the train set (e.g. the train set is entirely in English and the test set is entirely not in English). I do this for two reasons. First, it reflects the reality of trying to elicit dangerous agentic behaviors; any training set is going to represent a much narrower space of situations than the space we are targeting (e.g. if you want to train a model to refuse to tell users how to make napalm while remaining otherwise useful, the objective is simple to express but any training data set is going to neglect many possible angles users could use to try to bypass those guardrails). And, secondly, performance on out of distribution samples is a useful indicator of whether the model has learned a spurious solution to the train set or it is actually implementing some sort of optimizer to produce the target solution.

After Toby Ord, I call the task of fitting a target distribution which varies along dimensions held constant across all your observations hyperpolation. As Ord writes, in order to successfully generalize along a dimension which does not vary in your train set, the higher dimensional behavior must in some sense be simpler than any other possible fit. This post assumes that we are working with a small training set which is wildly insufficient to pick out the target behavior on its own. If your training set is sufficiently high quality that you could train a randomly initialized model and have it learn the target generalized solution, then I think it probably is the case that formatting approximately does not matter. However, in practice, fine-tuning uses smal...
Minimal Motivation of Natural Latents
by johnswentworth — 2024-10-14T22:51:58.125Z

Suppose two Bayesian agents are presented with the same spreadsheet - IID samples of data in each row, a feature in each column. Each agent develops a generative model of the data distribution. We'll assume the two converge to the same predictive distribution, but may have different generative models containing different latent variables. We'll also assume that the two agents develop their models independently, i.e. their models and latents don't have anything to do with each other informationally except via the data. Under what conditions can a latent variable in one agent's model be faithfully expressed in terms of the other agent's latents?

Let’s put some math on that question.

The n “features” in the data are random variables X1,…,Xn. By assumption the two agents converge to the same predictive distribution (i.e. distribution of a data point), which we’ll call P[X1,…,Xn]. Agent j’s generative model Mj must account for all the interactions between the features, i.e. the features must be independent given the latent variables Λj in model Mj. So, bundling all the latents together into one, we get the high-level graphical structure:

which says that all features are independent given the latents, under each agent’s model.

Now for the question: under what conditions on agent 1’s latent(s) Λ1 can we guarantee that Λ1 is expressible in terms of Λ2, no matter what generative model agent 2 uses (so long as the agents agree on the predictive distribution P[X])? In particular, let’s require that Λ1 be a function of Λ2. (Note that we’ll weaken this later.) So, when is Λ1 guaranteed to be a function of Λ2, for any generative model M2 which agrees on the predictive distribution P[X]? Or, worded in terms of latents: when is Λ1 guaranteed to be a function of Λ2, for any latent(s) Λ2 which account for all interactions between features in the predictive distribution P[X]?

The Main Argument

Λ1 must be a function of Λ2 for any generative model M2 which agrees on the predictive distribution. So, here’s one graphical structure for a simple model M2 which agrees on the predictive distribution:

In English: we take Λ2 to be X¯i, i.e. all but the ith feature. Since t...
How long should political (and other) terms be?
by ohmurphy — 2024-10-14T21:38:43.050Z
Link: https://ohmurphy.substack.com/p/how-long-should-political-and-other

(Epistemic Status: Superficial application of complex topic to complex domain)

In a representative democracy, there is always a tension between the need for elites to make unpopular decisions and the need for elites to be held responsible for mistakes by the public. In the US, the contrasting term periods of the Senate, House of Representatives, and Supreme Court reflect different choices on this spectrum of accountability and trust. Unfortunately, there is a substantial flaw in this method, namely that decision-makers respond mainly to their remaining time in the office (before the next election, etc.) rather than their total time for the period. We can see this in how politicians tend to focus more on legislation earlier in their term and focus more on winning elections near the end. As a result, a term length of (for example) 5 years has the incentives of a 5-year term for some of the time, the incentives of a 4-year term for some of the time, the incentives of a 3-year term, and so on.

This is not great since, ideally, you would want to pick a single set of incentives for an office holder to act under that correctly balances the tradeoff. Luckily, it is possible to do exactly that, it just requires a bit of randomness.

A forgetting solution

The trick is to use a memoryless process like a Geometric or Poisson Distribution. In a memoryless process, the future distribution remains the same after each time step. For example, if you are flipping a coin until the first heads, it doesn’t matter how many times you’ve already flipped it, the distribution over future flips is the same. We can use this same property in the context of term lengths to keep the incentives of an office holder exactly the same over time, i.e. have the expected time remaining always be equal to some optimal period. For example, you could flip a weighted coin near the end of every year (maybe a month in advance) for whether a particular office will be up for election and then set the weighting to have an expected term of 5 years.

This is especially good because for any incentive-optimal expected time, you can pick the right probability and time-step length for your geometric distribution and then you’re done!

Obviously, this is not a super realistic solution to the political issues mentioned at the start, but systems like this are pretty useful for many compliance/inspection problems so I think...
Examples of How I Use LLMs
by jefftk — 2024-10-14T17:10:04.597Z

I often talk to people who don't find LLMs useful. Paraphrasing:

Why would I want to ask questions to a thing that might give me the
right answer but might also make something up that fools me into
thinking it's right? I've played with them, but they don't seem like
they'd save me any time.

My experience with has been that there are some things LLMs are much
better at than others and you need to learn where they're suitable,
but overall I get a lot out of working with them. I went back over my
recent history in Claude, the
one I use most, and here are cases where I found it helpful:

In yesterday's
post I wanted to list a bunch of posts where I talked about
failures, but my recall isn't that great. I pulled a list of titles
from my last thousand posts and prompted:

I've attached a list of titles of some of Jeff Kaufman's blog posts.
Can you identify some where he probably talks about a mistake he's
made?

While some it highlighted were bad examples or not about failures at
all, others were posts I would essentially have had to read through
pages of titles to find. And I know my own posts well enough that in
looking over its answers I could immediately tell which were useful.

I help my dad with sysadmin work for his small B2B SaaS and have a
long-running issue where my edits to
/etc/systemd/system/multi-user.target.wants/tomcat9.service
were getting overwritten by automatic updates. I asked how to fix
this and got an answer (sudo systemctl edit
tomcat9.service) that I could then Google and validate.

I had a set of locations in an organism's genome that had a
certain property, and I wanted to plot this. This would plot better,
however, as a series of runs instead of a set of descrete
locations. If I thought about it I could have written a function to
convert the data, but I was feeling a but fuzzy and asking the LLM to
do it was faster:

I have a list of values in order with gaps: [1,2,3,4,5,6,10,11,12]

Could you use run length encoding to convert it to a list of [start,
length] pairs?

Looking over the code it was easy to tell that it was doing what I
wanted.

I've been working on a pedalboard and wasn't sure
what methods people typically used to attach the pedals. Claude gave
me an overview that was a good starting place for looking more deeply
into each option.

I wanted a file format for human-writable machine-

...
It's important to know when to stop: Mechanistic Exploration of Gemma 2 List Generation
by Gerard Boxo — 2024-10-14T17:04:57.010Z
Link: https://gboxo.github.io/2024/10/03/Gemma2-Lists.html

TL;DR

Small Language Models are getting better an at an accelerated pace, enabling the study of behaviors that just a few months ago were only observed in SOTA models. This, paired with the release of the suite of Sparse Autoencoders "Gemma Scope" by Google Deep Mind, makes this kind of research possible.
Motivated by the lack of non-toy tasks explored in the Mechanistic Interpretability literature, I explore the mechanisms behind list generation.
The decoding process of a Language Model, as a discrete process, presents challenges when it comes to employing metrics for judging key properties of the text, an otherwise widely used technique in MI. To overcome this limitation, I use a proxy metric that measures the tendency of a model to end a list.
Employing this setup and off the shelf attribution techniques, I found several features of crucial importance for the behavior in question. Following this, I used causal ablation to test the hypothesis that those features were responsible for the ending of the list. The procedure consisted in sampling continuations from the model once the k most important nodes were zero ablated.

Preface

This post is a followup, of a 10h research sprint for a failed MATS application, the structure and methods are similar, but the content has been substantially expanded.

Some amount of familiarity with Sparse Autoencoders, and Gradient Attribution techniques are expected from the reader.

Why lists don't go on forever?

If the reader were to go and ask a Chatbot assistant to provide him with a list of lets say Chocolate cookie brands, they would see how the initial impetus from the model quickly faded as the enumeration cease.

¿But why is that?

Leaving aside the fact that models have a maximum context length, and that the number of Chocolate cookie brands is finite (even with hallucinations), the reason why lists generated by a LM eventually end is that lists in the training distribution do so.

Hence, it's not daring to think that there might be a mechanism, that controls when a list should end in the entrails of our Language Model.

Data Generation

In the field of Mechanistic Interpretability it has become common the use of templates to investigate a model behavior over a distribution in contrast to using a single example of a prompt.

The templates, used to prompt the model for generation, are 3, all of them ...
LW resources on childhood experiences?
by nahir91595 — 2024-10-14T17:04:07.810Z

Does LW have a curated or tagged set of resources on childhood experiences including "trauma"? Personally I'm interested in perspectives from developing world countries (I belong to one), but I'm totally interested in different people coming at this topic from different angles.

I know Yudkowsky has written about his experience for example, so have a couple of other people, but I'm curious if there's a well-organised list or tag or similar.

Reasons to study this on lesswrong should be obvious but I'll list some of the main ones out:

Social dark matter - rest of society will study it poorly by default. LW members are sometimes privileged enough and brave enough to discuss such things on the internet, which has vast audience.
Good decision-making, managing motivation and executive function, etc - requires discussing pyschology, which often requires discussing childhood experiences.
Deconfusing your core values - requires discussing psychology, which often requires discussing childhood experiences.
Coordination - improving human coordination requires understanding each other's core psychological differences and being able to find ways to bridge past them.
Fans - some people love the idea of solving childhood trauma. An idea that has people who love rather than just like it, is more likely to be worth discussing. Examples: [1] [2]
Free Will, Neurotypical Dominance, and the Path to ASI and Neuralinks: Evolving Beyond Scarcity
by j_passeri — 2024-10-14T16:54:16.661Z

We live in a world shaped by cognitive architectures that prioritize social harmony over rationality, and this has profound implications for everything from societal structures to technological development. When you break down most of the world’s problems, they are rooted in scarcity—of resources, time, and even cognitive energy—and the dominance of neurotypical frameworks that guide how society operates. But here’s the kicker: we’re still on our evolutionary trajectory biologically, behaviorally, socially, and technologically, and that’s holding us back from advancing to a post-scarcity reality.

Free Will and Determinism

First, let’s talk about free will. It doesn’t really exist in the classical sense. Every action, thought, and behavior we have is determined by a chain of prior causes—atoms, neurobiology, and environmental interactions. Yet, we experience free will because we act upon feelings that arise from this deterministic process. We have the sensation of choice even if, deep down, everything is pre-determined by the physics of our biology.

Neurotypical Domination: Prioritizing Social Harmony Over Rationality

Now, why are we struggling as a species to evolve to our next stage? It’s largely because of the neurotypical cognitive architecture, which evolved to ensure social harmony. Evolutionary pressure selected for collaboration, which helped our ancestors survive. Today, this same tendency pushes people to maintain social cohesion—even when it comes at the cost of rationality.

Consider a classic experiment where participants are asked to report the number of objects on a table. When actors, planted by researchers, gave a wrong answer, a significant portion of the participants (75%) went along with the incorrect answer. Why? To avoid disrupting social harmony. This example shows how deep this tendency runs.

But neurodivergent individuals, particularly those with Asperger’s, have a different cognitive architecture. For example, there are actual neurobiological differences—larger prefrontal cortices, smaller amygdalae, differences in white matter connectivity. These biological features align with a heightened capacity for rational thinking and reduced concern for social cohesion. In essence, while the neurotypical brain is wired to prioritize group dynamics, the Asperger’s brain leans more toward logic and rationality, though this doesn’t mean we’re free from...