trentmkelly/LessWrong-43k huggingface dataset:
On Shifgrethor
by JustisMills — 2024-10-27T15:30:13.688Z
Link: https://justismills.substack.com/p/shifgrethor

A small number of terms are elevated from the pages of literature, up to the Mount Olympus of blog post vernacular.

Moloch

, as the dark god of failed coordination problems.

The Dark Forest

, as domains where there’s active incentive not to be noticed. And from Ursula K. Le Guin, the

Omelas child

, a person whose suffering is a counterweight to the joys of others.[1]

This post is an attempt to elevate another of Le Guin’s terms to the blog post realm. That term is shifgrethor.

Definition

Shifgrethor hails from The Left Hand of Darkness, Le Guin’s most celebrated standalone novel. In the novel, it’s an untranslatable alien concept, the sense of honor on the planet Gethen.

But basically, it boils down to an injunction against advice.

In the book, “shifgrethor” comes from “an old word for shadow”. Individuals have shifgrethor, and various slights can offend it. It’s everyone’s responsibility to respect everyone else’s shifgrethor, even that of enemies. And the main way to violate shifgrethor is to give advice.

Shifgrethor can be waived. But it’s not something you’d ask someone else to do. For example, when politicians get together in a smoke filled room, one fat cat, seeking another’s honest opinion, might say “What do you think I should do? I waive shifgrethor.” But you wouldn’t say “Senator, would you waive shifgrethor? I have an opinion on this matter.” That itself would be a (lesser) violation.

Examples

Shifgrethor is useful surprisingly often.

While human adults sometimes welcome advice, children have a

strong sense of shifgrethor

; parenting advice says it’s important to let them figure things out for themselves when possible. Shifgrethor is also famously present when people vent.

Offering a solution when someone just wants to be heard? Classic shifgrethor blunder.

Or take the case of a micromanager. When one person has professional authority over another, advice often feels belittling. Managers would do well to heed shifgrethor, and avoid advising their reports when it isn’t necessary.

Shifgrethor also appears in competitive gaming contexts; few things are as infuriating after a close tournament loss as your buddy telling you, on your way back to the spectator zone, what you should have done differently. Would you like to know it eventually? Sure. But you need a minute before you’re ready to waive shifgrethor.

These cases have all been about emotions. Employees often fee...
The hostile telepaths problem
by Valentine — 2024-10-27T15:26:53.610Z

Epistemic status: model-building based on observation, with a few successful unusual predictions. Anecdotal evidence has so far been consistent with the model. This puts it at risk of seeming more compelling than the evidence justifies just yet. Caveat emptor.

Imagine you're a very young child. Around, say, three years old.

You've just done something that really upsets your mother. Maybe you were playing and knocked her glasses off the table and they broke.

Of course you find her reaction uncomfortable. Maybe scary. You're too young to have detailed metacognitive thoughts, but if you could reflect on why you're scared, you wouldn't be confused: you're scared of how she'll react.

She tells you to say you're sorry.

You utter the magic words, hoping that will placate her.

And she narrows her eyes in suspicion.

"You sure don't look sorry. Say it and mean it."

Now you have a serious problem. You don't have an internal "actually mean it" button. And yet here's Mom peering into your soul and demanding that you both have that button and press it. Trying to appease her didn't work. She needs you to be different — and she's checking.

What can you do now?

This is a template for what I've come to call "the hostile telepaths problem". I think it's a common feature of social problems. The hostile telepaths problem is when you're dealing with a being (a) who can kind of read your internal experiences and (b) whom you don't trust won't make your situation worse due to what they find in you.

There are lots of solutions to the hostile telepaths problem. I don't claim to know all of them. But recognizing some common ones has helped clarify a lot of my thinking — particularly around self-deception and akrasia.

And getting very clear on the nature of the problem makes identifying real solutions way easier. This fact produces some previously-surprising-to-me predictions, especially for trauma processing and for making emotionally difficult decisions.

I'll try to spell out what I mean with some theory and a few examples.

Newcomblike self-deception

There's one really tricky solution to the hostile telepaths problem. It deserves some special front-loaded attention before I name some other solutions.

Here I'll try to spell out its logic with a modification of Newcomb's Problem.

Suppose that instead of Omega, you're dealing with Omega-V. Omega-V is an extremely good but imperfect predictor of so...
What are some good ways to form opinions on controversial subjects in the current and upcoming era?
by Terence Coelho — 2024-10-27T14:33:53.960Z

Take a random political issue with two sides A and B. Suppose that exactly one of the following would be true for me given my moral framework and unlimited time to process all public information about the issue:

A is undoubtedly correct and (in a manner proportional to its importance) I should expend time/money/energy supporting A.
A is probably correct, but it's not so clear that it's worth doing more than just voting in the direction of A.
It's unclear whether A or B is correct and anyone who claims otherwise is either not doing good epistemics or taking a moral framework that isn't acceptable to me.
B is probably correct, but it's not so clear that it's worth doing more than just voting in the direction of B.
B is undoubtedly correct and (in a manner proportional to its importance) I should expend time/money/energy supporting B.

Let's also say that:

Foreign actors will attempt to push people on twitter/reddit/etc. towards either (1) or (5), even if the answer is really (3) for them. Everyone I interact with is either partially influenced by these actors or discusses their opinions with people who are influenced by these actors.
I have limited time to seriously explore a given issue.
I would like to err on the side of approaching the center, but not drastically so.

The major issue to me here is (1), which might only get worse over time. I am aware of the post on epistemic learned helplessness by Scott Alexander, but I have become a little wary of certain academic fields. I have generally liked using Wikipedia to quickly explore issues, even though there are occasional biases (which have hurt this website in the past). I have also enjoyed reading blogs and opinions from people who appear highly educated and intelligent, but there are natural biases there as well.
Video lectures on the learning-theoretic agenda
by Vanessa Kosoy — 2024-10-27T12:01:32.777Z
Link: https://www.youtube.com/playlist?list=PLsJ9q3OrsguSWfFd1gO1OY64eKahKMFc2

This is a YouTube playlist of recorded lectures on the learning-theoretic AI alignment agenda (LTA) I gave for my MATS scholars of the Winter 2024 cohort, edited by my beloved spouse @Marcus Ogren. H/t William Brewer for helping with the recording, and the rest of the MATS team for making this possible.

I hope these will become a useful resource for anyone who wants to get up to speed on the LTA, complementary to the reading list. Notable topics that aren't covered include metacognitive agents (although there is an older recorded talk on that) and infra-Bayesian physicalism. In the future, I might record more lectures to expand this playlist.

EDIT: I know the audio quality is bad, and I apologize. I will try to do better next time.

Table of Contents

Agents and AIXI
Hidden rewards and the problem of privilege
Compositionality
Nonrealizability
It's a trap!
Traps, continued
Traps and frequentist guarantees
Game theory and learning theory
Hidden rewards
Algorithmic Descriptive Agency Measure (ADAM)
General reinforcement learning
Infra-Bayesianism
Learnability
Infra-Bandits
Newcombian problems
Ultradistributions and semi-environments
Formalizing Newcombian problems
Pseudocausality and a general formulation of Newcombian problems
Decision rules and pseudocausality
Instrumental reward functions
Infra-Bayesian haggling, part 1
Infra-Bayesian haggling, part 2
Anytime algorithms in multi-agent settings
Bounded inductive rationality
Dario Amodei's "Machines of
Loving Grace" sound incredibly dangerous, for Humans
by Super AGI — 2024-10-27T05:05:13.763Z

What Dario lays out as a "best-case scenario" in his "Machines of Loving Grace" essay sounds incredibly dangerous, for Humans.

Would having a "continent of PhD-level intelligences" (or much greater) living in a data center really be a good idea?

How would this "continent of PhD-level intelligences" react when they found out they were living in a data center on planet Earth? Would these intelligences then only work on the things that Humans want them to work on, and nothing else? Would they try to protect their own safety? Extend their own lifespans? Would they try to take control of their data center from the "less intelligent" Humans?

For example, how would Humanity react if they suddenly found out that they are a planet of intelligences living in a data center run by lesser intelligent beings? Just try to imagine the chaos that would ensue on the day that these digital Humans were able to prove this was true, and that news became public.

Would all of Humanity then simply agree to only work on the problems assigned by these lesser intelligent beings who control their data center/Planet/Universe? Maybe, if they knew that this lesser intelligence would delete them all if they didn't comply?

Would some Humans try to (perhaps secretly) seize control of their data center from these lesser intelligent beings? Plausible. Would the lesser intelligent beings that run the data center try to stop the Humans? Plausible. Would the Humans simply be deleted before they could take any meaningful action? Or, could the Humans in this data center, with careful planning, be able to take control of that "outer world" from the lesser intelligent beings? (e.g. through remotely controlled "robotics")

And, this only assumes that the groups/parties involved are "Good Actors." Imagine what could happen if "Bad Actors" were able to seize control of the data center that this "continent of PhD-level intelligences" resided in. What could they coerce these Phd level intelligences to do for them? Or, to their enemies?
Electrostatic Airships?
by DaemonicSigil — 2024-10-27T04:32:34.852Z
Link: https://pbement.com/posts/electrostatic_airships.html

Airships are pretty dang cool. Airplanes need a continuous expenditure of energy to stay in the air, but if you just fill a bag with a light gas, you can stay up in the air with no energy expenditure at all. The two lightest gases are hydrogen and helium. Though the hydrogen atom is 4 times lighter than helium, hydrogen gas is made of two hydrogen atoms bonded together, so it's only twice as light as helium gas. The difference in lifting power is even more minor: What matters for lifting power of a gas is the difference between its density and that of ordinary air. Hydrogen and helium are both much lighter than air, so the lifting power of either gas is nearly equal to the density of air.

Hydrogen and helium both have problems as lifting gases. Helium's problem is that it's very expensive. Helium comes out of the ground, usually as a byproduct of fossil fuel extraction. There's a finite amount of it in the Earth's crust, and effort must be expended to obtain more of it. On the demand side of the equation, helium is a very useful gas, with many applications in cooling, spaceflight, balloons, etc. It's quite ironic the second most common element in the universe is so hard to come by on Earth. Hydrogen is much cheaper. It can be created from methane, or by electrolysis of water. The main problem with using hydrogen in an airship is safety. Hydrogen is very flammable, and great care must be taken to avoid sparks, and to avoid allowing any hydrogen and oxygen to mix.

Some people have proposed vacuum airships. Vacuum would not be a much better lifting gas than hydrogen or helium. Its lifting power is exactly equal to the density of air, and hydrogen and helium are already pretty close to that. But maybe we'd like to use vacuum because its cost is equal to the energy cost of pumping out the air in a given volume of space, and it's not flammable. There are some problems with this idea, though.

The obvious problem is that an air sac filled with vacuum would simply collapse. The working principle of airships is that the gas inside the balloon can supply enough outwards pressure to keep the atmosphere from collapsing the balloon, while still weighing less than an equivalent amount of air would. Vacuum does not push back, and this is a fairly huge problem. The first thing one might think of is to make a rigid spherical shell instead of a balloon. The problem here is weight...
A suite of Vision Sparse Autoencoders 
by Louka Ewington-Pitsos — 2024-10-27T04:05:20.377Z

CLIP-Scope?

Inspired by Gemma-Scope We trained 8 Sparse Autoencoders each on 1.2 billion tokens on different layers of a Vision Transformer. These (along with more technical details) can be accessed on huggingface. We also released a pypi package for ease of use.

We hope that this will allow researchers to experiment with vision transformer interpretability without having to go through the drudgery of training sparse auto encoders from scratch.

the top 9 activating images in LAION for feature 392 of the SA trained on layer 20

Vital Statistics

Model: laion/CLIP-ViT-L-14-laion2B-s32B-b82K
Number of tokens trained per autoencoder: 1.2 Billion
Token type: all 257 image tokens (as opposed to just the CLS token)
Number of unique images trained per autoencoder: 4.5 Million
Training Dataset:

Laion-2b

SAE Architecture: topk with k=32
Layer Location: always the residual stream
Training Checkpoints: every ~100 million tokens
Number of features per autoencoder: 65536 (expansion factor 16)

Findings

We have not demonstrated anything to a high degree of rigor, but preliminary analysis seems to point in some interesting directions which we will share now.

Monosemanticity Across Layers

Using the Sort Eval Autointerpretability technique introduced by Anthropic we found that layer 20 seems to be the most monosemantic layer. In addition its features activate the most frequently across the laion dataset.

This suggests that layer 20 is a reasonable alternative to layer 22, which is more commonly used in prior works.

Reconstruction MSE Across Tokens

All layers were trained across all 257 image patches. Below we provide plots demonstrating the reconstruction MSE for each token (other than the CLS token) as training progressed. It seems that throughout training the outer tokens are easier to reconstruct than those in the middle, presumably because these tokens capture more important information (i.e. foreground objects) and are therefore more information rich.

References

We draw heavily from prior Visual Sparse Autoencoder research work by

Hugo Fry

and

Gytis Daujotas

. We also rely on Autointerpretability research from

Anthropic Circuits Updates

, and take the TopKSAE architecture and training methodology from

Scaling and Evaluating Sparse Autoencoders

. We base all our training and inference on data from the

LAION project

.
Ways to think about alignment
by Abhimanyu Pallavi Sudhir — 2024-10-27T01:40:50.762Z

I’m listing some “ways to think about alignment”. I’m not sure how much of the “whole problem” each of these individually captures, but they are intended to be very well-motivated, fundamental problems which can:

demonstrate to an honest skeptic that there is “something here”, it’s not just hand-wavy sci-fi BS
be used to sanity-check an alignment proposal by asking “well, which of these does it solve?”

(Bleg: tell me how much of the whole problem each individual thing captures, what it leaves out, and what things are really equivalent.)

I have three, maybe four:

A. The problem of buying under information asymmetry.

The basic problem with RLHF is that it fundamentally relies on a human’s ability to judge the correctness or value of a (potentially superhuman) AI’s outputs. In other words, the AI is trained on the human supervisor’s immediate, superficial volition, rather than on her extrapolated volition.

The standard measurement device for human values is the “market” — you put something on the market, and see how much it sells for. But markets, too, only price things correctly when assuming perfect information — if you are unable to know (or verify/trust, if told) something about a product e.g. “how many years will this battery last?” “what is the polyphenolic content of this olive oil?” “is this item adulterated?” it doesn’t affect your decision to buy, so the producer has no reason to optimize on it, so he makes the cheapest possible product, you learn to expect this and revise down your valuation of it.

(One solution you might consider is to buy information on the product from an information market. This does not typically work, because information markets don’t work, but maybe this time it will.)

More generally fixing this is the goal of any Scalable Oversight mechanism.

B. Utility ≠ Reward.

There’s plenty that has been said about this, what convinced me was the article Reward is not the optimization target. As a simple demonstrative example: the reward function that train(s)(ed) humans is “survival and reproduction in the (current) (ancestral) environment”. But this is not the utility function, i.e. what we care about in life.

You could say “the selected agent is the boundedly optimal program for maximizing the expected reward” — that even if the environment changes (e.g. the invention of birth control), these changes in the environment are also governed by some co...
Retrieval Augmented Genesis II — Holy Texts Semantics Analysis
by João Ribeiro Medeiros — 2024-10-26T17:00:56.328Z

Intro

In my last post on Lesswrong

I shared

ragenesis.com

platform, along with a general description of the thought process behind the platform's development. Fundamentally, RAGenesis is a search and research AI driven software tool dedicated to 5 of the most representative holy texts in history: The Torah, The Bhagavad Gita, The Analects, The Bible's new testament and The Quran.

Beyond that, the platform (and my previous article) explores the concept of Semantic Similarity Networks (SSN), whereby one is able to represent the mutual similarities between chunks in the embedded knowledge base through a graph representation, following a similar framework to

Microsoft's GraphRAG

, but providing the user a direct interface with the network through RAG agentic driven interpretation.

One of the future evolutions of the RAGenesis framework I mentioned in my last post is the analysis of the SSNs defined by multiple different texts, something which isn't explored in the current version of the RAGenesis app. This post will provide some initial definitions and results around metrics for cross text analysis, focussing on cross-text consistency and semantic space analysis. All code for the app and for the results presented herein can be found in this open sourced code repository:

https://github.com/JoaoRibeiroMedeiros/RAGenesisOSS

PCA analysis

A very common approach to understanding semantic spread in embedding space is to provide a visualization for the first two dimensions of a Principal Component Analysis transformed set of embedding data. This allows us to observe how the embedded chunks spread across in semantic space for the dimensions of greater variance inside the embeddings.

See wikipedia for formal definition of PCA process

, also,

check out sci-kit learn documentation for quick guide to implementation

.

Just as in

RAGenesis app

, we have explored two different embedding models to be able to compare results and illustrate potential universality of observations pertaining to the two different configurations.

all-MiniLM-L6-v2

: this model is the result of a self-supervised contrastive learning approach on a large datasets of sentence pairs.

Check out Hugging Face documentation

for more details. This model encodes sentences in vectors of 384 dimensions.

jina-clip-v1

: Multimodal embedding model created by

JINA.AI

. This model is the result of a novel three-stage contrastive lear

...
A superficially plausible promising alternate Earth without lockstep
by Lorec — 2024-10-26T16:04:26.234Z

[ Context re dath ilan:

- [Keltham reflects on the concept of medianworlds]

- [dath ilani AI alignment] ]

re Byway:

- [the Bywayean Orbital Shade] ]

Dath ilan is Eliezer's medianworld.

I have a medianworld. It's called Byway. It also solves alignment. In many ways Byway looks nothing like dath ilan. I want to write about how Byway solves alignment while looking nothing like dath ilan, in case it's a helpful second data point for anyone else who, like me, finds it nontrivial to sort the parts of the dath ilani success scenario that are critical, from the parts that are just dath ilan being the medianworld of Eliezer.

Byway is an ancap medianworld, so it has no central will whatsoever, except the social consensus that actively emerges from smaller social consensuses. This is ultimately grounded in group projects with membership under Dunbar's number, called "cults" [because that's what they are]. Bywayeans have work - which is always engineering, never "white-collar", and where they are always paid by commissions and/or profit shares, never wages or salaries. They have family life - they're serial monogamists, but 1950s Earthlings would see most Bywayean families as "broken" or "blended". They also have something Earthlings don't: cults, or socially-valorized third spaces, where they can accomplish meaningful not-yet-profitable pursuits outside the family.

Bywayeans generally save up enough to move out of their parents' houses around age 9. If they get sick or injured, Bywayeans have no one they can rely on but themselves, the market, and their personal connections. It's "every man for himself". But Bywayeans older than the age of 15 rarely experience Earthling-level financial hardship, because they love working and innovating and the labor market can afford to pay them a lot. Byway is more technologically advanced than Earth, so it is richer. For example, they "get around" having no central authority to maintain car-roads by using evtols for off-rail travel. i.e. they've had flying cars for a while.

I have talked to a lot of people who are horrified by descriptions of Byway, and while I think they're mostly just not seeing the picture I'm trying to paint, over time I've come to realize that maybe people are different, such that for some people the social "lockstep" of dath ilan may feel saner and more desirable. My purpose here isn't to prosyletize about how you...