trentmkelly/LessWrong-43k huggingface dataset:
The causal backbone conjecture
by tailcalled — 2024-08-17T18:50:14.577Z

Followup to: Rationalists are missing a core piece for agent-like structure (energy vs information overload). Related to: Linear Diffusion of Sparse Lognormals (I don't consider that sequence done yet BTW, I just took a break because I ran out of pre-written posts, so I need to write some more). Response to: Fixing The Good Regulator Theorem.

Where I last left off, I argued that an "energy"-orientation may permit a more efficient set of models that can zoom in on what really matters. But I didn't really go into much detail about what to expect or why, so I think I should try harder, in particular framing it more like a potential selection theorem.

Premise: Limited resources

The universe has certain limited resources - for instance, mass, energy, space, time, and similar (money?). Anything that happens within the universe makes use of these, and for events where there are causes which point in contradictory directions, the cause with more resources tends to win out (almost by definition).

Premise: Heterogenous fluctuations

Complex dynamics have sprung out of interactions between these resources, e.g. the sun, the earth, life with plants and animals, humanity, society, capitalism, NVIDIA, Berkshire Hathaway, Saudi Aramco, Elon Musk, Eliezer Yudkowsky, Sam Altman, etc.. The vast majority of the complexity that has sprung up around the resources has ~no influence on their distribution or long-term development, but the distribution of influence is long-tailed, so there are some particular entities who we can observe are likely to have exceptionally big influence.

The conjecture

The heterogenous fluctuations will sometimes lead to massive shifts in how the resources are distributed. Everyone who relied on the earlier distribution and who are not ready to shift to the newer distribution will go extinct, or at least severely diminish in prominence.

Thus, the universe has a "universal backbone" of causal dynamics that really matter, in the pathways of the resources, and the entities that can shift those pathways. One needs to model the parts of this universal backbone that are directly relevant to oneself.

However, because everyone is modelling this universal backbone, everything is tightly connected to it, and that also makes different parts of the backbone tend to be connected to each other. Thus, there will tend to be an incentive towards fully modelling the top of the bac...
I didn't have to avoid you; I was just insecure
by Chris Lakin — 2024-08-17T16:41:50.237Z
Link: https://chrislakin.blog/p/i-didnt-have-to-avoid-you

I don't usually post stories on LessWrong so I'm curious to see how this is received.

The first time we spoke, you asked me some questions that felt really invasive. I didn’t want that to happen again, so I avoided you the entire following year.

So when you said “Hi” at a party and suggested catching up, I hesitated. But curiosity won out.

You still asked probing questions like “Why did you quit your job?” and “What did you think of your manager? I hear they don't have a great social reputation.”

These weren't questions I wanted to answer. But this time, something was different. Not you—me.

In the past, I would have felt forced to answer your questions. But I’m sure you can remember how I responded when we spoke again: “Mm, I don’t want to answer that question”, “I don’t want to gossip”, and even a cheeky, “No comment :)”

It didn’t even take effort, that surprised me.

And nothing bad happened! We just spoke about other things.

I realized that I was protecting myself from you with physical distance.

But instead I could protect myself from you with “No.”

So simple…

Too simple?

Why didn’t I think of that before??

Oh, I know why: When I first met you, I was extremely afraid of expressing disapproval of other people.

I didn’t know it consciously. It was quite deeply suppressed. But the pattern fits the data.

It seems that I was so afraid of this, that when you asked me those questions when we met for the first time, the thought didn’t even cross my mind that I could decline to answer.

If I declined a question, I unconsciously predicted you might get mad, and that would make me feel terrible about myself.

So that’s why I didn’t say “No” to your questions when you first met me. And that’s why I avoided you so bluntly with physical distance. (Although, I also avoided everyone during that year for similar reasons.)

Why am I telling you all of this? You helped me grow. These days, it takes very little effort — and sometimes none at all— to reject others’ requests and generally do what I want. I’m much more emotionally secure now.

Also, I noticed a shift in how I perceived you. Once I realized I didn't have to avoid you, I began noticing qualities I admire. Your passion for your work. Your precise and careful reasoning. I want to learn from these traits. And now that I don't have to avoid you anymore, I can :)

Addendum: Beliefs I have

Emotional security is the absence of

...
Please support this blog (with money)
by Elizabeth — 2024-08-17T15:30:05.641Z
Link: https://acesounderglass.com/2024/08/15/please-support-this-blog-with-money/

Short version

It has always felt like a gift that people read things I write, and I enjoyed reciprocating that by sharing my writing for free. Until recently I had enough income to pull that off and still cover material needs. That is no longer true, and while it is not an urgent problem, I would like to solve it while that is still true. However it’s important to me to keep this blog free. To square this, I’m asking for your support. If you value this blog and can comfortably do so, please consider sending some of that value back my way via Patreon or Paypal.

Long version

I love my work and this blog. Clients paying me to learn interesting things that they immediately use to help people would be amazing on its own, but then grantmakers pay me to do more speculative EV projects of my choosing. Then I get to publish my work and help more people. I’m incredibly fortunate to do this, and at a wage that supports a good lifestyle in a reasonable number of hours.

Unfortunately, I can’t work a reasonable number of hours. It’s been years since I could work a full work-week, but until recently I could still work enough. That stopped being true this winter (short version: furniture grew mold leading to a devastating health cascade). I spent months barely able to work while firehosing money at the medical system (ballpark total cost: $100,000, plus an intense desire to own my own home). I’m doing much better although not fully healed now and God willing this was a one time thing, but 10 years ago I needed a year off work for dental surgery, and I’ll consider myself lucky if it takes 10 years before my next “one off” multi-month health problem. My limited hours in good times aren’t enough to make up for the total absence of hours in bad times. Disability insurance refused to sell to me at any price back when I had an excellent work attendance record and very minor medical issues, so I assume they’d use a flamethrower if I tried now. This has me interested in finding ways to decouple my income from my working hours.

The obvious thing to do is monetize this blog. Except I hate gating zero-marginal-cost content, especially health content. It keeps out the people who most need the help. It’s also hard to match price to value – any given post could be a waste of your time or a life changing intervention, and you may not know which for years. So I don’t wan...
Release: Optimal Weave (P1): A Prototype Cohabitive Game
by mako yass — 2024-08-17T14:08:18.947Z

I recently argued for Cohabitive Games, games that are designed for practicing negotiation, or for developing intuitions for applied cooperative bargaining, which is one of the names of preference aggregation.

I offered the assets for P1, my prototype, to whoever was interested. About 20 people asked for them. I told most of them that I wanted to polish the assets up a little bit first, and said that it would be done in about a month. But a little bit of polish turned into a lot, and other things got in the way, and a month turned into ten. I'm genuinely a little bit dismayed about how long it took, however:

P1, now Optimal Weave 0.1, is a lot nicer now.
I think it hangs together pretty well.

You can get it here: Optimal Weave 0.1. (I'm not taking any profit, and will only start to if it runs away and starts selling a lot. (release day sale. get in quick.))

There's also a print and play version if you want it sooner, free, and don't mind playing with paper. (Honestly, if you want to help to progress the genre, you've got to learn to enjoy playing on paper so that you can iterate faster.)

There's also a version that only includes the crisp, flippable land tiles, which you can then supplement with the ability, event and desire cards from the printed version and whatever playing pieces you have at home.

(note: the version you'll receive uses round tiles instead of hex tiles. They don't look as cool, but I anticipate that round tiles will be a little easier to lay out, and easier to flip without nudging their neighbors around (the game involves a lot of tile flipping). Earlier on, I had some hope that I could support both square layout and hexagonal layout, that's easier with round tiles, though everything's balanced around the higher adjacency of the hexagonal layout so I doubt that'll end up working)

I'll be contacting everyone who reached out imminently (I have a list).

What Happened

Further learnings occurred. I should report them.

I refined the game a fair bit

There were a lot of abilities I didn't like, and a lot of abilities I did like but hadn't taken the time to draw up, so I went through the hundred or so variations of the initial mechanics, cut the lame ones, added more ideas, then balanced everything so that no land-type would get stuck without any conversion pathways.

I also added an event deck, which provides several functions. It simplifies keeping track of wh...
New blog: Expedition to the Far Lands
by Connor Leahy — 2024-08-17T11:07:48.537Z
Link: https://www.ettf.land/

I have started a new personal blog!

I intend to use it to write about more esoteric/lower confidence/epistemologically-sticky topics than elsewhere.

You can already read the first post: "Mysticism 101, or: In defence of Natural Language DSLs"
Rationalists are missing a core piece for agent-like structure (energy vs information overload)
by tailcalled — 2024-08-17T09:57:19.370Z

The agent-like structure problem is a question about how agents in the world are structured. I think rationalists generally have an intuition that the answer looks something like the following:

We assume the world follows some evolution law, e.g. maybe deterministically like xn+1=f(xn), or maybe something stochastic. The intuition being that these are fairly general models of the world, so they should be able to capture whatever there is to capture. x here has some geometric structure, and we want to talk about areas of this geometric structure where there are agents.
An agent is characterized by a Markov blanket in the world that has informational input/output channels for the agent to get information to observe the world and send out information to act on it, intuitively because input/output channels are the most general way to model a relationship between two systems, and to embed one system within another we need a Markov blanket.
The agent uses something resembling a Bayesian model to process the input, intuitively because the simplest explanation that predicts the observed facts is the best one, yielding the minimal map that can answer any query you could have about the world.
And then the agent uses something resembling argmax to make a decision for the output given the input, since endless coherence theorems prove this to be optimal.
Possibly there's something like an internal market that combines several decision-making interests (modelling incomplete preferences) or several world-models (modelling incomplete world-models).

There is a fairly-obvious gap in the above story, in that it lacks any notion of energy (or entropy, temperature, etc.). I think rationalists mostly feel comfortable with that because:

xn+1=f(xn) is flexible enough to accomodate worlds that contain energy (even if they also accomodate other kinds of worlds where "energy" doesn't make sense)
80% of the body's energy goes to muscles, organs, etc., so if you think of the brain as an agent and the body as a mech that gets piloted by the brain (so the Markov blanket for humans would be something like the blood-brain barrier rather than the skin), you can mostly think of energy as something that is going on out in the universe, with little relevance for the agent's decision-making.

I've come to think of this as "the computationalist worldview" because functional input/...
Calendar feature geometry in GPT-2 layer 8 residual stream SAEs
by Patrick Leask — 2024-08-17T01:16:53.764Z

TL;DR: We demonstrate that the decoder directions of GPT-2 SAEs are highly structured by finding a historical date direction onto which projecting non-date related features lets us read off their historical time period by comparison to year features.

Calendar years are linear: there are as many years between 2000 and 2024, as there are between 1800 and 1824. Linear probes can be used to predict years of particular events from the activations of language models. Since calendar years are linear, one might think the same of other time-based features such as weekday features, however weekday activations in sparse autoencoders (SAEs) were recently found to be arranged in a circular configuration in their top principal components. Inspired by this, we looked into weekdays, months, and most interestingly calendar years from the perspective of SAE feature decoder similarity.

For each group of calendar features, we found interesting patterns of feature splitting between sparse autoencoders of different sizes. For calendar years, we found a timeline direction that meaningfully ordered events, individuals, and concepts with respect to their historical period, which furthermore does not correspond to a principal component of the decoder directions. Finally, we introduce a simple method for finding some of these interpretable directions.

Features at different scales

We started by replicating the weekday results by performing PCA on the decoder directions of features that had high activations when prompted with days of the week, using the same GPT-2 SAEs as in this post, ranging from 768 to 98304 features. In the 768 feature SAE, we found a single weekday feature that activated strongly on all days of the week. In the largest SAE, we found 10 weekday features, 3 of which activated on all days of the week, with the remaining 7 activating on a single day of the week each.

We found a group of features that activate primarily on specific days of the week by taking the top 20 activating samples for each feature and checking that the max activating token in each of these samples was the specific weekday. We found the first two principal components for this set of features, and projected the features that activate on any day or number of days from all SAEs onto these directions. The labeled features are those that activate on a single day across all SAE...
How unusual is the fact that there is no AI monopoly?
by Viliam — 2024-08-16T20:21:51.012Z

I may be completely confused about this, but my model of technological breakthroughs in history was basically this: A few guys independently connect the dots leading to a new invention, for example the telephone, approximately at the same time. One of them runs to the patent office a little faster than the others, and he gets the patent first. Now he gets to be forever known as the inventor of the telephone, and the rest of them are screwed; if they ever try to sell their own inventions, they will probably get sued to bankruptcy.

Today, we have a few different companies selling AIs (LLMs). What is different this time?

Is my model of the history wrong?
Is the current legal situation with patents different?
Are LLMs somehow fundamentally different from all other previous inventions?
Is it the fact that anyone can immediately publish all tiny partial steps in online journals, that makes it virtually impossible for any individual or institution to legally acquire the credit -- and the monopoly -- for the entire invention?
Or something else...?
The Tech Industry is the Biggest Blocker to Meaningful AI Safety Regulations
by garrison — 2024-08-16T19:37:28.416Z
Link: https://garrisonlovely.substack.com/p/the-tech-industry-is-the-biggest

If you enjoy this, please consider subscribing to my Substack.

My

latest reporting

went up in The Nation yesterday:

It’s about the tech industry’s meltdown in response to SB 1047, a California bill that would be the country’s first significant attempt to mandate safety measures from developers of AI models more powerful and expensive than any yet known.

Rather than summarize that story, I’ve added context from some past reporting as well as new reporting on two big updates from yesterday: a congressional letter asking Newsom to veto the bill and a slate of amendments.

The real AI divide

After spending months on my January

cover story

in Jacobin on the AI existential risk debates, one of my strongest conclusions was that the AI ethics crowd (focused on the tech’s immediate harms) and the x-risk crowd (focused on speculative, extreme risks) should recognize their shared interests in the face of a much more powerful enemy — the tech industry:

According to one

estimate

, the amount of money moving into AI safety start-ups and nonprofits in 2022 quadrupled since 2020, reaching $144 million. It’s difficult to find an equivalent figure for the AI ethics community. However, civil society from either camp is dwarfed by industry spending. In just the first quarter of 2023, OpenSecrets reported roughly

$94 million

was spent on AI lobbying in the United States. LobbyControl estimated tech firms spent

€113 million

this year lobbying the EU, and we’ll recall that hundreds of billions of dollars are being invested in the AI industry as we speak.

And here’s how I

ended

that story:

The debate playing out in the public square may lead you to believe that we have to choose between addressing AI’s immediate harms and its inherently speculative existential risks. And there are certainly trade-offs that require careful consideration.

But when you look at the material forces at play, a different picture emerges: in one corner are trillion-dollar companies trying to make AI models more powerful and profitable; in another, you find civil society groups trying to make AI reflect values that routinely clash with profit maximization.

In short, it’s capitalism versus humanity.

This was true at the time I published it, but honestly, it felt like momentum was on the side of the AI safety crowd, despite its huge structural disadvantages (industry has way more money and

armies

of seasoned lobbyists).

Si...
Principled Satisficing To Avoid Goodhart
by JenniferRM — 2024-08-16T19:05:27.204Z

There's an admirable LW post with (currently) zero net upvotes titled Goodhart's Law and Emotions where a relatively new user re-invents concepts related to super-stimuli. In the comments, noggin-scratcher explains in more detail:

The technical meaning is a stimulus that produces a stronger response than the stimulus for which that response originally evolved.

So for example a candy bar having a carefully engineered combination of sugar, fat, salt, and flavour in proportions that make it more appetising than any naturally occurring food. Or outrage-baiting infotainment "news" capturing attention more effectively than anything that one villager could have said to another about important recent events.

In my opinion, there's a danger that arises when applying the dictum to know thyself, where one can do this so successfully that one begins to perceive the logical structure of the parts of yourself that generate subjectively accessible emotional feedback signals.

In the face of this, you face a sort of a choice: (1) optimize these to get more hedons AT ALL as a coherent intrinsic good, or (2) something else which is not that.

In general, for myself, when I was younger and possibly more foolish than I am now, I decided that I was going to be explicitly NOT A HEDONIST.

What I meant by this has changed over time, but I haven't given up on it.

In a single paragraph, I might "shoot from the hip" and say that when you are "not a hedonist (and are satisficing in ways that you hope avoid Goodhart)" it doesn't necessarily mean that you throw away joy, it just means that that WHEN you "put on your scientist hat", and try to take baby steps, and incrementally modify your quickly-deployable habits, to make them more robust and give you better outcomes, you treat joy as a measurement, rather than a desiderata. You treat subjective joy like some third party scientist (who might have a collaborator who filled their spreadsheet with fake data because they want a Nature paper, that they are still defending the accuracy of, at a cocktail party, in an ego-invested way) saying "the thing the joy is about is good for you and you should get more of it".

When I first played around with this approach I found that it worked to think of myself as abstractly-wanting to explore "conscious optimization of all the things" via methods that try to only pay attention to the plausible semantic understa...
Rewilding the Gut VS the Autoimmune Epidemic
by GGD — 2024-08-16T18:00:46.239Z

Did you know that an autoimmune disease now affects one in five Americans? Of these, 80% are women; autoimmune disease is the fifth leading cause of death in women under 65. Most autoimmune diseases are being diagnosed in increasing numbers, but it is still unclear if all this variance is due to higher awareness, medical literacy, or better diagnosis techniques.

The effects of autoimmune diseases can be devastating. As a person’s immune system attacks their body instead of microbes or cancerous cells, they can experience chronic fatigue, chronic pain, drug dependency, depression, and social isolation. These symptoms annihilate mental health, wreck promising careers, destroy lives, and, often, financially ruin families. For too many, these illnesses result in early death.

The Missing Friends

The Old Friends Hypothesis suggests that early and regular exposure to harmless microorganisms—"old friends" present throughout human evolution and recognized by the human immune system—train the immune system to react appropriately to threats.

Here's the problem: some very specific microbes that we've coevolved with—for potentially millions of years (considering monkeys emerged 35 million years ago)—are no longer in our gut. Those are literally gone; they cannot be replaced by eating fiber, kefir, or any probiotics on the market. And this might be the reason the immune system of many people is dysfunctional—it's missing organisms it's been interacting with for a very long time.

We know this precisely because we've been able to compare the microbiome of industrialized populations with modern hunter-gatherers, who possibly have the microbiome closest to early humans. The Hadza, a protected hunter-gatherer Tanzanian indigenous ethnic group, are known for having the highest gut diversity of every population. The Hadza had an average of 730 species of gut microbes per person. The average Californian gut microbiome contained just 277 species, and the Nepali microbiome fell in between. Ultra-deep sequencing of the Hadza microbiome reveals 91,662 genomes of bacteria, archaea, bacteriophages, and eukaryotes, 44% of which are absent from existing unified datasets. The researchers identified 124 gut-resident species absent or vanishing in industrialized populations.

As seen in the above picture, there is, at the same time, reduced diversity and an entirely different type of microbial eco...