trentmkelly/LessWrong-43k huggingface dataset:
DSLT 0. Distilling Singular Learning Theory
by Liam Carroll — 2023-06-16T09:50:14.474Z

TLDR; In this sequence I distill Sumio Watanabe's Singular Learning Theory (SLT) by explaining the essence of its main theorem - Watanabe's Free Energy Formula for Singular Models - and illustrating its implications with intuition-building examples. I then show why neural networks are singular models, and demonstrate how SLT provides a framework for understanding phases and phase transitions in neural networks.

Epistemic status: The core theorems of Singular Learning Theory have been rigorously proven and published by Sumio Watanabe across 20 years of research. Precisely what it says about modern deep learning, and its potential application to alignment, is still speculative.

Acknowledgements: This sequence has been produced with the support of a grant from the Long Term Future Fund. I'd like to thank all of the people that have given me feedback on each post: Ben Gerraty, @Jesse Hoogland , @mfar, @LThorburn , Rumi Salazar, Guillaume Corlouer, and in particular my supervisor and editor-in-chief Daniel Murfet.

Theory vs Examples: The sequence is a mixture of synthesising the main theoretical results of SLT, and providing simple examples and animations that illustrate its key points. As such, some theory-based sections are slightly more technical. Some readers may wish to skip ahead to the intuitive examples and animations before diving into the theory - these are clearly marked in the table of contents of each post.

Prerequisites: Anybody with a basic grasp of Bayesian statistics and multivariable calculus should have no problems understanding the key points. Importantly, despite SLT pointing out the relationship between algebraic geometry and statistical learning, no prior knowledge of algebraic geometry is required to understand this sequence - I will merely gesture at this relationship. Jesse Hoogland wrote an excellent introduction to SLT which serves as a high level overview of the ideas that I will discuss here, and is thus recommended pre-reading to this sequence.

SLT for Alignment Workshop: This sequence was prepared in anticipation of the SLT for Alignment Workshop 2023 and serves as a useful companion piece to the material covered in the Primer Lectures.

Thesis: The sequence is derived from my recent masters thesis which you can read about at my website.

Developmental Interpretability: Originally the sequence was going t...
DSLT 1. The RLCT Measures the Effective Dimension of Neural Networks
by Liam Carroll — 2023-06-16T09:50:10.113Z

TLDR; This is the first post of Distilling Singular Learning Theory (DSLT), an introduction to which can be read at DSLT0. In this post I explain how singular models (like neural networks) differ from regular ones (like linear regression), give examples of singular loss landscapes, and then explain why the Real Log Canonical Threshold (aka the learning coefficient) is the correct measure of effective dimension in singular models.

When a model class is singular (like neural networks), the complexity of a parameter w in parameter space W⊂Rd needs a new interpretation. Instead of being defined by the total parameters available to the model d, the complexity (or effective dimensionality) of w is defined by a positive rational λ∈Q>0 called the Real Log Canonical Threshold (RLCT), also known as the learning coefficient. The geometry of the loss K(w) is fundamentally defined by the singularity structure of its minima, which λ measures. Moreover, in regular models like linear regression the RLCT is λ=d2, but in singular models it satisfies λ≤d2 in general. At its core, then, Sumio Watanabe's Singular Learning Theory (SLT) shows the following key insight:

The RLCT λ∈Q>0 is the correct measure of effective dimensionality of a model w∈W.

Watanabe shows that the RLCT λ has strong effects on the learning process: it is the correct generalisation of model complexity in the Bayesian Information Criterion for singular models, and therefore plays a central role in the asymptotic generalisation error, thereby inheriting the name "learning coefficient".

In this first post, after outlining the Bayesian setup of SLT, we will start by defining what a singular model is and explain what makes them fundamentally different to regular models. After examining different examples of singular K(w) loss landscapes, we will define the RLCT to be the scaling exponent of the volume integral of nearly true parameters, and conclude by summarising how this quantity correctly generalises dimensionality.

Preliminaries of SLT

The following section introduces some necessary technical terminology, so use it as a reference point, not necessarily something to cram into your head on a first read through. A more thorough setup can be found in [Car21, Chapter 2], which follow...
[Linkpost] Mapping Brains with Language Models: A Survey
by Bogdan Ionut Cirstea — 2023-06-16T09:49:23.043Z

This is a linkpost for https://arxiv.org/abs/2306.05126.

Over the years, many researchers have seemingly made the same observation: Brain and language model activations exhibit some structural similarities, enabling linear partial mappings between features extracted from neural recordings and computational language models. In an attempt to evaluate how much evidence has been accumulated for this observation, we survey over 30 studies spanning 10 datasets and 8 metrics. How much evidence has been accumulated, and what, if anything, is missing before we can draw conclusions? Our analysis of the evaluation methods used in the literature reveals that some of the metrics are less conservative. We also find that the accumulated evidence, for now, remains ambiguous, but correlations with model size and quality provide grounds for cautious optimism.
Rational Animations is looking for an AI Safety scriptwriter, a lead community manager, and other roles.
by Writer — 2023-06-16T09:41:14.343Z

UPDATE 07/15/2023: Applications are now closed. If you still want to apply, I will consider you for the future, which may be as soon as a few months from now, or I could initially offer you a lower salary if you'd like to join sooner.
Rational Animations is hiring. If you would like to apply, reach out to rationalanimations@gmail.com.

The more material you include, the easier for me to evaluate you. The hiring process will probably consist of a short interview and a period you should consider a trial, even if paid at full salary.

AI Safety scriptwriter

At the moment, Rational Animations has two consistently active writers, and the only one doing AI Safety scripts is me. I'd like to have more slack and be able to publish a lot of high-quality AI Safety explainers in the coming months and years. Our animation team is growing, and we will be able to publish videos much faster.

Pay: 50 -150 USD per hour. If I offer you 150 per hour, I'll limit your hours to 10 per week, at least for the next few months. That said, I'm looking for someone who can put in consistent effort. At any given time, most scriptwriters in Rational are inactive, and I'm 100% OK with that, but I'm also looking for someone I can consistently rely on.

What may help you land the job:

Having written good stuff about AI Safety helps enormously.
Mathematical education helps.
AI education helps.
AI Safety education (e.g., having studied the AGISF curriculum) helps.
Surely other things I have yet to think of.

I may hire more than one person if many good candidates apply. I may also decide to hire no one.

Bonus: I expect people in this role to be able to write scripts for shorts about AI Safety (more below).

Lead community manager

Shortly, we will include many more calls to action in our videos to increase our impact. But that is only one way to go about it. Another option I'm excited about is building our own spaces for the most hardcore followers of the channel to go much deeper into our topics. These spaces include, but may not be limited to:

Discord (high priority)
Reddit (high priority)
Twitter
Instagram

Your job consists of developing strategies to help with this objective and taking action to pursue them. The outcome you should try to achieve is to help people land in those spaces and curate them so that the users learn a lot.

Our highest-priority topic is becoming AI Safety, so the most important thin...
Leveling Up Or Leveling Off? Understanding The Science Behind Skill Plateaus
by lynettebye — 2023-06-16T00:18:04.378Z

This post is crossposted from my blog. If you liked this post, subscribe to Lynette's blog to read more -- I only crosspost about half my content to other platforms.

If you’re going into surgery, you want the youngest operating surgeon available.

This is a slight exaggeration – you don’t want a doctor in their first year out of medical school.[1] After that, it’s less clear. One review found thirty-two studies indicating that the older a doctor was, the worse their medical outcomes; that review only found one study indicating that all outcomes got better with increasing age.[2] Other analyses suggest that middle-aged doctors might do better than younger doctors (though the effect is not statistically significant)[3], but older doctors are still clearly worse than middle-aged doctors.[4]

It’s not like doctors become terrible with more experience, but they are measurably worse. In one study, an extra twenty years of experience translated to about one additional elderly patient dying out of every hundred treated.[5] Why would twenty extra years of practice make a doctor worse at helping people?

Similarly, some research on famous painters, writers[6], and composers[7] found that it’s most common for these artists to produce their best work before the age of 45. A famous composer is more likely to have written their best piece of music in their 20s than their 40s, and they’re almost twice as likely to have written their best piece in their 30s instead of their 50s.[8]

Again, creatives are doing worse after a couple of extra decades of experience. This phenomenon is called skill plateaus. The idea is that performance stops improving after a relatively short time. After that, putting in thousands more hours on the job won’t reliably make you much better.

What’s going on here?

Deliberate practice

Anders Ericsson thought he had the answer. Ericsson was a psychologist famous for his research on expertise and human performance. He claims that most people practice a new skill until they can perform it adequately for their purposes.

For example, someone might practice tennis until they’re good enough to play with their friends. At that point, they stop practicing to just enjoy the casual games. Many people assume that they’ll continue to slowly get b...
human intelligence may be alignment-limited
by bhauth — 2023-06-15T22:32:14.685Z

Previously, I argued that human mental development implies that AI self-improvement from sub-human capabilities is possible, and that human intelligence comes at the cost of a longer childhood and greater divergence from evolutionarily-specified goals.

In that post, I raised 2 hypotheses:

H:mutational_load = Human capabilities are limited by high intelligence requiring high genetic precision that can be achieved only rarely with normal rates of mutation generation and elimination.

H:drift_bound = The extent of SI in humans is limited by increased value drift outweighing increased capabilities.

Humans have a lot of mental variation. Some people can't visualize 3d objects. Some people can't remember faces. Some people have synaesthesia. Such variation also exists among very smart people; there isn't convergence to a single intellectual archetype. You could argue that what's needed genetically is precise specification of something lower-level that underlies all that variation, but I don't think that's correct.

So, I don't think H:mutational_load is right. That leaves H:drift_bound as the only hypothesis that seems plausible to me.

Suppose that I'm correct that human intelligence comes at the cost of a longer childhood. The disadvantages of a long childhood vary depending on social circumstances. Humans may have some control mechanism which modifies the amount of mental self-improvement and thus the length of childhood depending on the surrounding environment. Certain environments - probably safe ones with ample food - would then be associated with both longer childhoods and a one-time increase in average intelligence. That would also cause greater divergence from evolutionarily-specified goals, which may show up as a decrease in fertility rates, or an increased rate of obsession with hobbies. That can obviously be pattern-matched to the situation in some countries today, but I don't mean to say that it's definitely true; I just want to raise it as a hypothesis.

If H:drift_bound is correct, it would be an example of an optimized system having a strong and adjustable tradeoff between capabilities and alignment, which would be evidence for AI systems also tending to have such a tradeoff.

Agents are adaptation-executors with adaptations that accomplish goals, not goal-maximizers. Understanding agents as maximizing goals is a simplification used by humans to make them ...
Developing a technology with safety in mind: Lessons from the Wright Brothers
by jasoncrawford — 2023-06-15T21:08:55.828Z
Link: https://rootsofprogress.org/wright-brothers-and-safe-technology-development

If a technology may introduce catastrophic risks, how do you develop it?

It occurred to me that the Wright Brothers’ approach to inventing the airplane might make a good case study.

The catastrophic risk for them, of course, was dying in a crash. This is exactly what happened to one of the Wrights’ predecessors, Otto Lilienthal, who attempted to fly using a kind of glider. He had many successful experiments, but one day he lost control, fell, and broke his neck.

Otto Lilienthal gliding experiment. Wikimedia / Library of Congress

Believe it or not, the news of Lilienthal’s death motivated the Wrights to take up the challenge of flying. Someone had to carry on the work! But they weren’t reckless. They wanted to avoid Lilienthal’s fate. So what was their approach?

First, they decided that the key problem to be solved was one of control. Before they even put a motor in a flying machine, they experimented for years with gliders, trying to solve the control problem. As Wilbur Wright wrote in a letter:

When once a machine is under proper control under all conditions, the motor problem will be quickly solved. A failure of a motor will then mean simply a slow descent and safe landing instead of a disastrous fall.

When actually experimenting with the machine, the Wrights would sometimes stand on the ground and fly the glider like a kite, which minimized the damage any crash could do:

The Wrights flying their glider from the ground. Wikimedia / Library of Congress

When they did go up in the machine themselves, they flew relatively low. And they did all their experimentation on the beach at Kitty Hawk, so they had soft sand to land on.

All of this was a deliberate, conscious strategy. Here is how David McCullough describes it in his biography of the Wrights:

Well aware of how his father worried about his safety, Wilbur stressed that he did not intend to rise many feet from the ground, and on the chance that he were “upset,” there was nothing but soft sand on which to land. He was there to learn, not to take chances for thrills. “The man who wishes to keep at the problem long enough to really learn anything positively must not take dangerous risks. Carelessness and overconfidence are usually more dangerous than deliberately accepted risks.”

As time would show, caution and close attention to all advance preparations were to be the rule for the brothers. They would take risks when nec

...
AXRP Episode 22 - Shard Theory with Quintin Pope
by DanielFilan — 2023-06-15T19:00:01.340Z

YouTube link

What can we learn about advanced deep learning systems by understanding how humans learn and form values over their lifetimes? Will superhuman AI look like ruthless coherent utility optimization, or more like a mishmash of contextually activated desires? This episode’s guest, Quintin Pope, has been thinking about these questions as a leading researcher in the shard theory community. We talk about what shard theory is, what it says about humans and neural networks, and what the implications are for making AI safe.

Topics we discuss:

Why understand human value formation?

Why not design methods to align to arbitrary values?

Postulates about human brains

Sufficiency of the postulates

Reinforcement learning as conditional sampling

Compatibility with genetically-influenced behaviour

Why deep learning is basically what the brain does

Shard theory

Shard theory vs expected utility optimizers

What shard theory says about human values

Does shard theory mean we’re doomed?

Will nice behaviour generalize?

Does alignment generalize farther than capabilities?

Are we at the end of machine learning history?

Shard theory predictions

The shard theory research community

Why do shard theorists not work on replicating human childhoods?

Following shardy research

Daniel Filan:
Hello everybody. In this episode I’ll be speaking with Quintin Pope. Quintin is a PhD student at Oregon State University where he works with Xiaoli Fern on applying techniques for natural language processing to taxonomic and metagenomic data. He’s also one of the leading researchers behind shard theory, a perspective on AI alignment that focuses on commonalities between AI training and human learning within a lifetime. For links to what we’re discussing, you can check the description of this episode and you can read the transcript at axrp.net. All right, Quintin, welcome to AXRP.

Quintin Pope:
I’m glad to be here.

Why understand human value formation?

Daniel Filan:
Yeah. So the first thing I want to ask is… So I guess we’re broadly going to be talking about shard theory and the shard theory perspective. And a research-methodological assumption behind shard theory research is that: it seems like you think that focusing on the processes by which humans devel...
Group Prioritarianism: Why AI Should Not Replace Humanity [draft]
by fsh — 2023-06-15T17:33:18.333Z

This is a draft written by FSH, Postdoc at Hong Kong University, as part of the Center for AI Safety Philosophy Fellowship. This draft is meant to solicit feedback.

Abstract:

If a future AI system can enjoy far more well-being than a human per resource, what would be the best way to allocate resources between these future AI and our future descendants? It is obvious that on total utilitarianism, one should give everything to the AI. However, it turns out that every Welfarist axiology on the market also gives this same recommendation. Without resorting to deontological normative theories that suggest that we ought not always to create the world with the most value, or non-welfarist theories that tell us that the world with the most value may not be the world with the most welfare, we propose a new theory that justifies the survival of humanity in the face of overwhelming AI wellbeing. We call this new theory, ``Group Prioritarianism".

Introduction:

In 1974, Robert Nozick famously introduced the idea of the ``utility monster", a being that is so efficient at converting resources into utility, that the best way to maximize utility in the world is to give all resources to this one monster at the cost of all of humanity. Nozick goes on to say the mere possibility of the utility monster refutes ``Utilitarian theories", which imply that one ought to sacrifice all humanity into the maws of this utility monster. Such a consequence is taken to be so outlandish that any theory that suggests that humanity ought to be sacrificed to the monster is, in Nozick's words, ``embarassed" [1974, p.41].

In this paper, we will take as a fixed point that Nozick is right in this case: it really is wrong to sacrifice all humanity to the monster, even if doing so increases total well-being[1]. But even if we take this as a fixed-point, many important questions are left unanswered. For example, should we create these utility monsters if we had the option? And if these monsters do come to exist, and if we ought not to give everything to the monsters, then how much should we give to these monsters? And if Utilitarianism cannot accommodate our intuitions about distributive welfare in this case, then what kind of normative theory can?

These questions have gained a new urgency as the utility monster no longer exists as a mere thought experiment...
Press the happiness button!
by Spiarrow — 2023-06-15T17:30:26.992Z

tldr: some free ways to benefit others
Epistemic status: Some things I noticed, made up or let ChatGPT generate

Idea

What if you had a button that you could press to make other people happy? Pushing the button doesn’t cost you anything besides the negligible effort. How often would you press it? If your answer is somewhere in the range between “Very often” and “Obviously I would build a robot arm that could press the button as fast as physically possible”, then there probably are a couple of things you should be doing more often:

Examples

Give genuine compliments. I made a trigger-action plan that if I like something about someone and it’s not inappropriate to say, then I say it. Note: The more unusual the compliment, the better your judgment about appropriateness needs to be.
Share relevant knowledge and resources. Trigger-action plan: If I encounter something that could be useful, then I quickly think about who could also benefit from this and share the info with them. You can also do this with things you already know.
Say thank you. Trigger-action plan: If I found something beneficial, then I say “thank you”. Don’t limit this to personal favors! A “thank you” in response to a talk, a public Slack message or a nice looking garden is very welcome, especially because it rewards providing common goods. When many people benefit from something and could in theory say thank you, often nobody ends up doing it. Saying “Thank you” to people who “just do their job” is neglected as well. Also, you can thank people for who they are (compared to what they did). You have to know and like a person rather well to be able to do that, but it’s super wholesome.
Introduce people to each other, if they consent. Trigger-action plan: If I notice complementary interests in 2 people, then I first ask both of them (explicitly saying they’re allowed to say “no”) and if they say “yes”, then I connect them. You could connect applicants and employers, students who want to learn the same thing, or travelers who want to go to the same place.
Let people know you thought of them. Trigger-action plan: If something made me think of somebody (positively), then I let them know. Very easy to do, very wholesome.

Caveats

It’s possible to overdo some of these things. But you can probably always ask if it’s too much
Maybe some o

...