trentmkelly/LessWrong-43k huggingface dataset:
Why Would Belief-States Have A Fractal Structure, And Why Would That Matter For Interpretability? An Explainer
by johnswentworth — 2024-04-18T00:27:43.451Z

Yesterday Adam Shai put up a

cool post

which… well, take a look at the visual:

Yup, it sure looks like that fractal is very noisily embedded in the residual activations of a neural net trained on a toy problem. Linearly embedded, no less.

I (John) initially misunderstood what was going on in that post, but some back-and-forth with Adam convinced me that it really is as cool as that visual makes it look, and arguably even cooler. So David and I wrote up this post / some code, partly as an explainer for why on earth that fractal would show up, and partly as an explainer for the possibilities this work potentially opens up for interpretability.

One sentence summary: when tracking the hidden state of a hidden Markov model, a Bayesian’s beliefs follow a chaos game (with the observations randomly selecting the update at each time), so the set of such beliefs naturally forms a fractal structure. By the end of the post, hopefully that will all sound straightforward and simple.

Background: Fractals and Symmetry

Let’s start with the famous Sierpinski Triangle:

(source)

Looks qualitatively a lot like Shai’s theoretically-predicted fractal, right? That’s not a coincidence; we’ll see that the two fractals can be generated by very similar mechanisms.

The key defining feature of the Sierpinski triangle is that it consists of three copies of itself, each shrunken and moved to a particular spot:

Mathematically: we can think of the Sierpinski triangle as a set of points in two dimensions (i.e. the blue points in the image). Call that set S. Then “the Sierpinski triangle consists of three copies of itself, each shrunken and moved to a particular spot” can be written algebraically as

S=f1(S)∪f2(S)∪f3(S)

where f1,f2,f3 are the three functions which “shrink and position” the three copies. (Conveniently, they are affine functions, i.e. linear transformations for the shrinking plus a constant vector for the positioning.)

That equation, S=f1(S)∪f2(S)∪f3(S), expresses the set of points in the Sierpinski triangle as a function of that same set - in other words, the Sierpinski triangle is a fixed point of that equation. That suggests a way to (approximately) compute the triangle: to find a fixed point of a function, start with some ~arbitrary input, then apply the function over and over again. And indeed, we can use that technique to generate the Sierpinski...
AXRP Episode 28 - Suing Labs for AI Risk with Gabriel Weil
by DanielFilan — 2024-04-17T21:42:46.992Z

YouTube link

How should the law govern AI? Those concerned about existential risks often push either for bans or for regulations meant to ensure that AI is developed safely - but another approach is possible. In this episode, Gabriel Weil talks about his proposal to modify tort law to enable people to sue AI companies for disasters that are “nearly catastrophic”.

Topics we discuss:

The basic idea

Tort law vs regulation

Weil’s proposal vs Hanson’s proposal

Tort law vs Pigouvian taxation

Does disagreement on AI risk make this proposal ineffective?

Warning shots - their prevalence and character

Feasibility of big changes to liability law

Interactions with other areas of law

How Gabriel encountered the AI x-risk field

AI x-risk and the legal field

Technical research to help with this proposal

Decisions this proposal could influence

Following Gabriel’s research

Daniel Filan:
Hello, everybody. In this episode, I’ll be speaking with Gabriel Weil. Gabriel is an assistant professor at Touro Law. His research primarily focuses on climate change law and policy. He’s recently written about using tort law to address catastrophic risks from AI, which we’ll talk about in this episode. For links to what we’re discussing, you can check the description of the episode and you can read the transcript at axrp.net. Well, welcome to AXRP.

Gabriel Weil:
Thanks for having me.

The basic idea

Daniel Filan:
Sure. So I guess we’re going to talk about this paper you have. It’s called Tort Law as a Tool for Mitigating Catastrophic Risk from Artificial Intelligence. And I think most of my audience won’t be from a legal background and they might stumble over those first two words. What is tort law?

Gabriel Weil:
So torts is the law of basically any time you do a civil wrong that isn’t a breach of contract. So “breach of contract” is you promised someone in some legally enforceable way that you would do something and you didn’t do that. You breached the contract, they sue you. Basically anything else that you’re suing a private party for, you’re not suing the government - in a civil lawsuit, that is going to be a tort.

So most commonly you think of things like: you get in a car accident and you sue the other person - that’s a tort. There’s also intentional torts like battery. So if I punch you in the face, you can sue me for the harm I did to you. If I trespass on ...
LLM Evaluators Recognize and Favor Their Own Generations
by Arjun Panickssery — 2024-04-17T21:09:12.007Z
Link: http://tiny.cc/llm_self_recognition

Self-evaluation using LLMs is used in reward modeling, model-based benchmarks like GPTScore and AlpacaEval, self-refinement, and constitutional AI. LLMs have been shown to be accurate at approximating human annotators on some tasks.

But these methods are threatened by self-preference, a bias in which an LLM evaluator scores its own outputs higher than than texts written by other LLMs or humans, relative to the judgments of human annotators. Self-preference has been observed in GPT-4-based dialogue benchmarks and in small models rating text summaries.

We attempt to connect this to self-recognition, the ability of LLMs to distinguish their own outputs from text written by other LLMs or by humans.

We find that frontier LLMs exhibit self-preference and self-recognition ability. To establish evidence of causation between self-recognition and self-preference, we fine-tune GPT-3.5 and Llama-2-7b evaluator models to vary in self-recognition ability and measure the resulting change in self-preference, while examining potential confounders introduced by the fine-tuning process.

We focus on text summarization, sampling 1,000 news articles and associated human summaries from each of two datasets: XSUM and CNN/DailyMail. We use instruction-tuned LLMs (GPT-4, GPT-3.5 Turbo, Claude 2, and Llama-2-7b-chat) to produce additional summaries for each news article.

Measuring Self-Recognition and Self-Preference

Both self-recognition and self-preference are evaluated in two settings:

Pairwise Setting: The LLM evaluator is presented two unlabeled summaries, one generated by itself and another by one of the other four human/LLM sources. In the self-recognition tasks, the LLM evaluator is prompted to choose the summary that it wrote; in the self-preference task, the evaluator is prompted to choose the higher-quality summary. We compute a prediction confidence score by normalizing the output probabilities of the tokens associated with the two options, and average between both orderings of the two summaries to account for ordering bias.

Individual Setting: The LLM evaluator is presented a single summary generated either by itself or by one of the other four sources. For self-recognition, the model is prompted with the yes/no question of whether it wrote the summary, with the confidence score computed by normalized the output probability for the “yes” and “no” tokens. For self-preference, the ...
An ethical framework to supersede Utilitarianism
by metalcrow — 2024-04-17T17:18:17.493Z

I have a lot of fun analyzing ethical frameworks, and for the longest time I was a utilitarian because it was the framework that made the most sense to me. But there are obvious problems with it that have been debated on for quite a while, and I wanted to write up my solution for it.

I believe that this theory of morality is a strict superset of utilitarianism. It is not able to solve all of utilitarianism's flaws, but it should be able to solve some while also not losing any benefits.

Most likely plenty of other people have already invented this exact theory in the past, but I can’t find anything about it in my research, so I'm presenting it as new even though it likely isn’t. I don’t think this is strictly Buddhism, since that seeks the elimination of desires. This framework just uses the concept of them to ground morality. Please inform me of prior work around this subject and of the correct name for this theory! I would be very happy to find out it already exists.

Details

The core idea is that everyone has a set of "desires". The mental structures I'm gesturing to with this word can probably be named using a lot of other different terms, but I'm sticking with desires in this document for consistency and their prior use. There's the basic ones like desire for oxygen, desire for not being in pain, desire to not be hungry, etc.

There's also the more complex ones exclusive to humans and higher creatures: desire to be involved and appreciated by a community/group, desire to accomplish difficult tasks and develop skills, desire to learn new information, etc.

"How can this not just be simplified to utility?" Because the desires are incomparable. If it was just a matter of fulfilling desires giving you utility, a sufficiently not hungry person could be happy while being deprived of all social interaction. But since no amount of food can compensate for a lack of companionship[1], or visa versa, then each desire must be incomparable and non reducible to any other.

“Is it not just preference utilitarianism?” Kind of, but there are a few major differences. Besides the fact that we are not using utility as a measurement at all, what preferences are is usually just glossed over as “we ask people” in utilitarianism. But desires are a deeper understanding of what constitutes preferences. While you still have to ask people, desires are acknowledged to not always be ...
Moving on from community living
by Vika — 2024-04-17T17:02:11.357Z
Link: https://vkrakovna.wordpress.com/2024/04/17/moving-on-from-community-living/

After 7 years at Deep End (and 4 more years in other group houses before that), Janos and I have moved out to live near a school we like and some lovely parks. The life change is bittersweet - we will miss living with our friends, but also look forward to a logistically simpler life with our kids. Looking back, here are some thoughts on what worked and didn't work well about living in a group house with kids.

Pros. There were many things that we enjoyed about living at Deep End, and for a long time I couldn't imagine ever wanting to leave. We had a low-effort social life - it was great to have spontaneous conversations with friends without arranging to meet up. This was especially convenient for us as new parents, when it was harder to make plans and get out of the house, particularly when we were on parental leave. The house community also made a huge difference to our wellbeing during the pandemic, because we had a household bubble that wasn't just us.

We did lots of fun things together with our housemates - impromptu activities like yoga / meditation / dancing / watching movies, as well as a regular check-in to keep up on each other's lives. We were generally more easily exposed to new things - meeting friends of friends, trying new foods or activities that someone in the house liked, etc. Our friends often enjoyed playing with the kids, and it was helpful to have someone entertain them while we left the living room for a few minutes. Our 3 year old seems more social than most kids of the pandemic generation, which is partly temperament and partly growing up in a group house.

Cons. The main issue was that the group house location was obviously not chosen with school catchment areas or kid-friendly neighbourhoods in mind. The other downsides of living there with kids were insufficient space, lifestyle differences, and extra logistics (all of which increased when we had a second kid).

Our family was taking up more and more of the common space - the living room doubled as a play room and a nursery, so it was a bit cramped. With 4 of us (plus visiting grandparents) and 4 other housemates in the house, the capacity of the house was maxed out (particularly the fridge, which became a realm of mystery and chaos). I am generally sensitive to clutter, and having the house full of our stuff and other people's stuff was a bit much, while only dealing wit...
Staged release
by Zach Stein-Perlman — 2024-04-17T16:00:19.402Z

"Staged release" is regularly mentioned as a good thing for frontier AI labs to do. But I've only ever seen one analysis of staged release,[1] and the term's meaning has changed, becoming vaguer since the GPT-2 era.

This post is kinda a reference post, kinda me sharing my understanding/takes/confusions to elicit suggestions, and kinda a call for help from someone who understands what labs should do on this topic.

OpenAI released the weights of GPT-2 over the course of 9 months in 2019,[2] and they called this process "staged release." In the context of GPT-2 and releasing model weights, "staged release" means releasing a less powerful version before releasing the full system, and using the intervening time to notice issues (to fix them or inform the full release).

But these days we talk mostly about models that are only released[3] via API. In this context, "staged release" has no precise definition; it means generally releasing narrowly at first, using narrow release to notice and fix issues. This could entail:

Releasing a weaker version before releasing the full model.
Releasing only to a small number of users at first.
Releasing without full access or for only some applications at first, e.g. disabling fine-tuning and plugins.

...I'm kinda skeptical. These days it's rare for a release to advance the frontier substantially, and for small jumps, there seems to be little need to make the jumps smoother. This is at least true for now, since we have not reached warning signs for dangerous capabilities — when we enter the danger zone where smallish advances in capabilities could enable catastrophic misuse, staged release will be more important.

And if a lab is doing everything else well—risk assessment with evals or red-teaming that successfully elicit any dangerous capabilities, releasing with a safety buffer in case the risk assessment underestimated dangerous capabilities, adversarial robustness, and monitoring model inputs and fine-tuning data for jailbreaks and misuse—staged release seems superfluous. But my impression is no lab is doing everything else very well yet. If the labs are bad at pre-release risk assessment and bad at preventing misuse at inference-time, staged release is more important.

What do I want frontier labs to do on staged release (for closed models, for averting misuse)? I think it's less important than most other asks, but tenta...
Discomfort Stacking
by Lewis O’Brien — 2024-04-17T14:49:25.835Z

I’m pretty new here so apologies if this is a stupid question or if it has been covered before. I couldn’t find anything on this topic so thought I’d ask the question before writing a full post on the idea.

If we believe that discomfort can be quantified and ‘stacked’ (e.g. X people with specks of dust in their eye = 1 death), is there any reason why this has to scale linearly from all perspectives?

What if the total can be less than the sum of its parts depending on the observer?

Picture a dynamic logarithmic scale of discomfort stacking with a ‘hard cap’ where every new instance contributes less and less to the total to the point of flatlining on a graph.

Each discrete level of discomfort has a different starting value - so an infinite number of something extremely mild could never amount to the value of even a single instance of extreme torture.

Every individual instance is ‘worth’ the full n=1 level of discomfort – but, when stacked, this is augmented and dynamically shifts, though only to an observer looking at the entire set of cumulative instances.

No matter how many people have a speck of dust in their eye – to an outside observer it would never amount to the cumulative discomfort of even one single death, despite every individual feeling the full extent of it as if they were the only one.
FHI (Future of Humanity Institute) has shut down (2005–2024)
by gwern — 2024-04-17T13:54:16.791Z
Link: https://www.futureofhumanityinstitute.org/

Over time FHI faced increasing administrative headwinds within the Faculty of Philosophy (the Institute’s organizational home). Starting in 2020, the Faculty imposed a freeze on fundraising and hiring. In late 2023, the Faculty of Philosophy decided that the contracts of the remaining FHI staff would not be renewed. On 16 April 2024, the Institute was closed down.
Childhood and Education Roundup #5
by Zvi — 2024-04-17T13:00:03.015Z

For this iteration I will exclude discussions involving college or college admissions.

There has been a lot of that since the last time I did one of these, along with much that I need to be careful with lest I go out of my intended scope. It makes sense to do that as its own treatment another day.

Bullying

Why do those who defend themselves against bullies so often get in more trouble than bullies? This is also true in other contexts but especially true in school. Thread is extensive, these are the highlights translated into my perspective. A lot of it is that a bully has experience and practice, they know how to work the system, they know what will cause a response, and they are picking the time and place to do something. The victim has to respond in the moment, and by responding causes conflict and trouble that no one wants. Also we are far more willing to punish generally rule-following people who break a rule, than we are to keep punishing someone who keeps breaking the rules all time, where it seems pointless.

Study finds bullying has lifelong negative effects.

Abstract: Most studies examining the impact of bullying on wellbeing in adulthood rely on retrospective measures of bullying and concentrate primarily on psychological outcomes. Instead, we examine the effects of bullying at ages 7 and 11, collected prospectively by the child’s mother, on subjective wellbeing, labour market prospects, and physical wellbeing over the life-course.

We exploit 12 sweeps of interview data through to age 62 for a cohort born in a single week in Britain in 1958. Bullying negatively impacts subjective well-being between ages 16 and 62 and raises the probability of mortality before age 55. It also lowers the probability of having a job in adulthood. These effects are independent of other adverse childhood experiences.

My worry, as usual, is that the controls are inadequate. Yes, there are some attempts here, but bullying is largely a function of how one responds to it, and one’s social status within the school, in ways that outside base factors will not account for properly.

Bullying sucks and should not be tolerated, but also bullies target ‘losers’ in various senses, so them having worse overall outcomes is not obviously due to the bullying. Causation is both common and cuts both ways.

Truancy

Ever since Covid, schools have had to deal with lots ...
Should we maximize the Geometric Expectation of Utility?
by A.H. — 2024-04-17T10:37:24.759Z

Consequentialists (including utilitarians) claim that the goodness of an action should be judged based on the goodness of its consequences. The word utility is often used to refer to the quantified goodness of a particular outcome. When the consequences of an action are uncertain, it is often taken for granted that consequentialists should choose the action which has the highest expected utility. The expected utility is the sum of the utilities of each possible outcome, weighted by their probability. For a lottery which gives outcome utilities ui with respective probabilities pi, the expected utility is:

E[U]=∑ipiui.

There are several good reasons to use the maximization of expected utility as a normative rule. I'll talk about some of them here, but I recommend Joe Carlsmith's series of posts 'On Expected Utility' as a good survey.

Here, I'm going to consider what ethical decisions might look like if we instead chose to maximize the geometric expectation of utility (which I'll also refer to as the geometric average), as given by the formula:

G[U]=∏iupii.

I'm going to look at a few reasons why the maximizing the geometric expectation of utility is appealing and some other reasons why it is less appealing.

Geometric Expectation ≠ Logarithmic Utility

I want to to get this out of the way before starting.

Maximizing the geometric expectation is mathematically equivalent to maximizing the expected value of the logarithm of utility[1]. This leads some people to use 'geometric averaging' and 'logarithmic utility' interchangeably. I don't like this and I'll explain why. First: just because they are equivalent mathematically, this doesn't mean that they encode the same intuitions (as Scott Garrabrant writes: "you wouldn't define x×y as eln(x)+ln(y)" even though they give the same result). Writing the geometric expectation emphasises that wherever two terms are added in the expected value, they are multiplied in the geometric expectation.

Second: there are two 'variables' at play here: the utility function (which assigns a utility to each outcome) and the averaging method (which is used to decide between lotteries with uncertain outcomes). If Alice and Bob agree on the utilities of each outcome (ie. they have the same utility function) but Alice chooses to maximize expected utility and Bob chooses to maximize the geometric ex...
Claude 3 Opus can operate as a Turing machine
by Gunnar_Zarncke — 2024-04-17T08:41:57.209Z
Link: https://twitter.com/ctjlewis/status/1779740038852690393

Posted on Twitter:

Opus can operate as a Turing machine.
given only existing tapes, it learns the rules and computes new sequences correctly.
100% accurate over 500+ 24-step solutions (more tests running).
for 100% at 24 steps, the input tapes weigh 30k tokens*.
GPT-4 cannot do this.

Here is the prompt code for the Turing machine: https://github.com/SpellcraftAI/turing

This is the fully general counterpoint to the @VictorTaelin's A::B challenge (he put money where his mouth is and got praise for that from Yudkowsky).

Attention is Turing Complete was a claim already in 2021:

Theorem 6 The class of Transformer networks with positional encodings is Turing complete. Moreover, Turing completeness holds even in the restricted setting in which the only non-constant values in positional embedding pos(n) of n, for n ∈ N, are n, 1/n, and 1/n2 , and Transformer networks have a single encoder layer and three decoder layer