trentmkelly/LessWrong-43k huggingface dataset:
Generalization, from thermodynamics to statistical physics
by Jesse Hoogland — 2023-11-30T21:28:50.089Z

Summary

In 2018, Zhang et al. showed that deep neural networks can achieve perfect training loss on randomly labeled data.

This was a Big Deal.

It meant that existing generalization theory couldn't explain why deep neural networks generalize. That's because classical approaches to proving that a given model class (=neural network architecture) would generalize involved showing that it lacks the expressivity to fit noise. If a model class can fit noise arbitrarily well, the resulting bounds break.

So something needed to change.

Evidently, you can't prove tight generalization bounds for entire model classes, so theorists turned to studying generalization bounds for individual models within a model class. If you can empirically show that a model's performance doesn't change substantially when you perturb it (by adding noise to the inputs, weights, training samples, etc.), then you can theoretically prove that that model will generalize to new data.

As a result, the bounds have gotten tighter, but they're still not exactly flattering.

What's really needed is a secret third thing. It's not about either model classes or individual models but about model subclasses. While the model class as a whole may be too complex to obtain tight generalization bounds, individual subclasses can achieve an optimal trade-off between accuracy and complexity. For singular Bayesian learning machines, this trade-off happens automatically.

This more or less answers why models are able to generalize but not how they do it.

Singular learning theory (SLT) provides one possible path towards understanding the how of generalization. This approach is grounded in the geometry of the loss landscape, which in turn is grounded in the symmetries of the model class and data. If this direction pans out, then learning theory is posed for a revolution analogous to the transition between thermodynamics and statistical physics.

Introduction

The central aim of classical learning theory is to bound various kinds of error: in particular, the approximation error, generalization error, and optimization error.

In the previous post in this series, we looked at the approximation error, which measures the performance of a model class's hypothetically optimal model. If your model class isn't expressive enough to do a reasonable job of modeling the data, then it doesn't matter...
What's next for the field of Agent Foundations?
by Nora_Ammann — 2023-11-30T17:55:13.982Z

Nora_Ammann

Alexander, Matt and I want to chat about the field of Agent Foundations (AF), where it's at and how to strengthen and grow it going forward.

We will kick off by each of us making a first message outlining some of our key beliefs and open questions at the moment. Rather than giving a comprehensive take, the idea is to pick out 1-3 things we each care about/think are important, and/or that we are confused about/would like to discuss. We may respond to some subset of the following prompts:

Where is the field of AF at in your view? How do you see the role of AF in the larger alignment landscape/with respect to making AI futures go well? Where would you like to see it go? What do yo use as some of the key bottlenecks for getting there? What are some ideas you have about how we might overcome them?

Before we launch in properly, just a few things that seem worth clarifying:

By Agent Foundations, we mean roughly speaking conceptual and formal work towards understanding the foundations of agency, intelligent behavior and alignment. In particular, we mean something broader than what one might call "old-school MIRI-type Agent Foundations", typically informed by fields such as decision theory and logic.
We will not specifically be discussing the value or theory of change behind Agent Foundations research in general. We think these are important conversations to have, but in this specific dialogue, our goal is a different one, namely: assuming AF is valuable, how can we strengthen the field?

Should it look more like a normal research field?

mattmacdermott

The main question I'm interested in about agent foundations at the moment is whether it should continue in its idiosyncratic current form, or whether it should start to look more like an ordinary academic field.

I'm also interested in discussing theories of change, to the extent it has bearing on the other question.

Why agent foundations?

My own reasoning for foundational work on agency being a potentially fruitful direction for alignment research is:

Most misalignment threat models are about agents pursuing goals that we'd prefer they didn't pursue (I think this is not controversial)
Existing formalisms about agency don't seem all that useful for understanding or avoiding those threats (again probably not that controversial)
Developing new and more useful ones seems tractable (this is p

...
A Proposed Cure for Alzheimer's Disease???
by MadHatter — 2023-11-30T17:37:33.982Z

I was thinking about the reaction to my posts over the past 48 hours, and it occurred to me that maybe they would receive a better reaction if I did a better job of demonstrating the epistemic virtues championed by the rationality community.

The all-time greatest hits for demonstrating epistemic virtue are, of course, falsifiable predictions about the outcome of an experiment that has not been conducted, could not be conducted using the resources available to the predictor, and which matter a great deal to the community assessing the epistemic virtue of the predictor.

So I thought I would post my theory of the etiology of Alzheimer's disease, along with a description of a low-cost treatment that this theoretical etiology would suggest would treat and even reverse the symptoms of this dread disease.

What? I hear the reader gasp in shocked dismay. He's doing it again? This guy really is the crank-to-end-all-cranks! And so I am, I suppose. But I invite any interested readers who care about someone who has or may develop Alzheimer's disease to test out my theory in a randomized clinical trial. If I had my druthers, I would nominate @Scott Alexander to organize the clinical trial, since he is both well-regarded by the community, and a domain expert in psychiatry, the relevant science.

Prior Work

We link to popular treatments of some important prior work to establish that this theory is grounded in up-to-date science.

https://alzheimergut.org/ is the website associated with existing research into the gut microbiome hypothesis of Alzheimer's disease.

https://github.com/epurdy/ethicophysics/blob/main/serotonin.pdf is my personal (incomplete) theory on the function of the neurotransmitter serotonin in the mammalian nervous system.

https://www.sciencedirect.com/science/article/abs/pii/S0920996417305017#:~:text=Conclusions%3A%20violent%20schizophrenia%20patients%20treated,the%20placebo%20after%20twelve%20weeks. is a description of the use of fish oil to treat schizophrenia. This is a treatment that has shown some limited success in treating schizophrenia.

https://www.quantamagazine.org/in-the-guts-second-brain-key-agents-of-health-emerge-20231121/ is a description of the relation of serotonin in the gut to glial cells in the brain.

Proposed Treatment

A very serious course of very serious antibiotics
A stool transplant from a healthy subject
Megadoses of EPA and DHA
A clinically rele

...
AI #40: A Vision from Vitalik
by Zvi — 2023-11-30T17:30:08.350Z

It has been brutal out there for someone on my beat. Everyone extremely hostile, even more than usual. Extreme positions taken, asserted as if obviously true. Not symmetrically, but from all sides nonetheless. Constant assertions of what happened in the last two weeks that are, as far as I can tell, flat out wrong, largely the result of a well-implemented media campaign. Repeating flawed logic more often and louder.

The bright spot was offered by Vitalik Buterin, who offers a piece entitled ‘My techo–optimism,’ proposing what he calls d/acc for defensive (or decentralized, or differential) accelerationism. He brings enough nuance and careful thinking, and clear statements about existential risk and various troubles ahead, to get strong positive reactions from the worried. He brings enough credibility and track record, and enough shibboleths, to get strong endorsements from the e/acc crowd, despite his acknowledgement of existential risk and the dangers ahead, and the need to take action to mitigate future problems.

Could we perhaps find common ground after all and have productive discussions? It’s going to be tough, but perhaps we are not so far apart. I had at least one very good private discussion as well, where someone turned out to mostly have a far more reasonable position than the one they were indicating they had, and we were able to find a productive path forward. It can be done.

My worry with Vitalik’s vision, as with other similar visions, is that it makes for an excellent expression of the problem, but that its offered solutions do not actually work in the case of AI. We continue to not have found an acceptable solution. A good problem statement is excellent, the best we could hope for here. The worry is that we might once again fool ourselves into not fully facing up to the problem. The proposed answer, to ‘merge with the AIs,’ continues to seem to me to a confused concept that has not been thought through enough, with little hope for an equilibrium.

But man is that the type of discussion I would like to be having.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Detect pancreatic cancer, perhaps.

Language Models Don’t Offer Mundane Utility. Google, put a stop to this.

Q Continuum. Various speculations about Q*. I do not understand the excitement.

OpenAI, Altman and Safety. Variou

...
Is scheming more likely in models trained to have long-term goals? (Sections 2.2.4.1-2.2.4.2 of “Scheming AIs”)
by Joe Carlsmith — 2023-11-30T16:43:07.557Z

This is Sections 2.2.4.1-2.2.4.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search "Joe Carlsmith Audio" on your podcast app.

What if you intentionally train models to have long-term goals?

In my discussion of beyond-episode goals thus far, I haven't been
attending very directly to the length of the episode, or to whether
the humans are setting up training specifically in order to incentivize
the AI to learn to accomplish long-horizon tasks. Do those factors make
a difference to the probability that the AI ends up with the sort of the
beyond-episode goals necessary for scheming?

Yes, I think they do. But let's distinguish between two cases, namely:

Training the model on long (but not: indefinitely long) episodes,
and

Trying to use short episodes to create a model that optimizes over
long (perhaps: indefinitely long) time horizons.

I'll look at each in turn.

Training the model on long episodes

In the first case, we are specifically training our AI using fairly long
episodes – say, for example, a full calendar month. That is: in
training, in response to an action at t1, the AI receives gradients that
causally depend on the consequences of its action a full month after t1,
in a manner that directly punishes the model for ignoring those
consequences in choosing actions at t1.

Now, importantly, as I discussed in the section on "non-schemers with
schemer-like traits," misaligned non-schemers with longer episodes will
generally start to look more and more like schemers. Thus, for example,
a reward-on-the-episode seeker, here, would have an incentive to
support/participate in efforts to seize control of the reward process
that will pay off within a month.

But also, importantly: a month is still different from, for example, a
trillion years. That is, training a model on longer episodes doesn't
mean you are directly pressuring it to care, for example, about the
state of distant galaxies in the year five trillion. Indeed, on my
definition of the "incentivized episode," no earthly training process
can directly punish a model for fai...
Normative Ethics vs Utilitarianism
by Logan Zoellner — 2023-11-30T15:36:00.602Z
Link: https://midwitalignment.substack.com/p/bonus-edition-normative-ethics-vs

The primary point of this post is that I want to draw everyone’s attention to an excellent podcast by John David Presserman where he chats with Zvi.

John goes into the post itching for a fight with Zvi over LessWrong style arguments about deceptive inner-alignment. But Zvi doesn’t really take the bait. Instead they spend most of the podcast talking about John’s ideas regarding RLaiF.

What does RLaiF got to do with Normative Ethics?

John spends most of the podcast talking about the tendency of RLaiF models to fall into a degenerate state if you over-train them. The canonical example of this is a “Yes spammer”. In this case, the reinforcement learning is receiving its reward from a 2nd AI model that answer “yes” or “no” to “is this a good result”. Eventually the AI we are training learns to repeatedly output the word “yes” as this tricks the judge AI into also responding “yes” every single time.

John analogizes the tendency of RLaiF models to fall into degenerate feedback modes with the difference between rule-based ethical systems and outcome-based ethics systems (utilitarianism). In normative ethics systems, people follow a variety of rules based on historical practice such as “do not murder”, “do not steal", etc. By contrast, in utilitarianism, there are no hard and fast rules, instead adherents simply seek to maximize a single objective function, the total amount of “goodness” or utility in the world.

Just as utilitarianism has both bad theoretic and practical failure modes, when AI is trained with a single objective function, it inevitably falls into strange undesirable local maxima. In ethics, we solve this problem by relying on a set of historically developed rules that have been empirically proven to avoid the worst outcomes. Similarly, John argues that we can avoid perverse-maximization in RLaiF frameworks by valuing the process by which we arrive at the global maxima.

A Simple Research Proposal

I would like to propose testing John’s hypothesis by building the simplest possible test case:

Train a RLaiF model that degenerates into the “yes spammer”

Along the way, “checkpoint” the reward function periodically

Create a new reward function that is a weighted sum of the checkpoints

Verify that with this new reward function training does not degenerate into the “yes spammer” (note this is trivially true if we allow our set of weights to be 1,0,0,0…)

I...
Information-Theoretic Boxing of Superintelligences
by JustinShovelain — 2023-11-30T14:31:11.798Z

Boxing an agent more intelligent than ourselves is daunting, but information theory, thermodynamics, and control theory provide us with tools that can fundamentally constrain agents independent of their intelligence. In particular, we may be able to contain an AI by limiting its access to information

Constraining output and inputs

Superintelligent AI has the potential to both help and harm us. One way to minimize the harm is boxing the AI: containing it so that it can’t freely influence its external world, ideally while preserving its potential to help.

Boxing is complicated. Destroying the AI will certainly prevent it from influencing anything. Throwing it into a black hole might causally disconnect it from the rest of the universe while possibly preserving its existence, though even then it may exploit some unknown physics to

leak information back across the event horizon

, or to persist or recreate itself once the black hole has evaporated.

Can we box an AI while still being being able to use it? We could try preventing it from physically interacting with its environment and only permitting it to

present information

to a user, but a superintelligence could abuse any communication channels to manipulate its users into

granting it more power

or

improving its predictive abilities

. To successfully box the AI in this manner, we’d need to constrain both its ability to physically interact with its environment and communicate information and manipulate. We’ll call this output boxing: containing an AI by constraining its various outputs.

Most discussions of boxing focus on output boxing, but there’s a neglected mirror to this approach that we’ll call input boxing: containing an AI by constraining what information it can access. (For completeness, we could also consider boxing techniques independent of input and output, where we contain an AI by constraining its functioning, such as by limiting its computational speed or restarting it from initial conditions each second).

In the rest of this post, we’ll introduce and examine a theoretical input boxing technique that contains an AI by controlling the number of bits of information it has about its environment, using results from information theory, thermodynamics, and control theory.

Good regulators and bad keys

Today’s powerful AIs rely on mountains of data...
OpenAI: Altman Returns
by Zvi — 2023-11-30T14:10:05.469Z

As of this morning, the new board is in place and everything else at OpenAI is otherwise officially back to the way it was before.

Events seem to have gone as expected. If you have read my previous two posts on the OpenAI situation, nothing here should surprise you.

Still seems worthwhile to gather the postscripts, official statements and reactions into their own post for future ease of reference.

What will the ultimate result be? We likely only find that out gradually over time, as we await both the investigation and the composition and behaviors of the new board.

I do not believe Q* played a substantive roll in events, so it is not included here. I also do not include discussion here of how good or bad Altman has been for safety.

Sam Altman’s Statement

Here is the official OpenAI statement from Sam Altman. He was magnanimous towards all, the classy and also smart move no matter the underlying facts. As he has throughout, he has let others spread hostility, work the press narrative and shape public reaction, while he himself almost entirely offers positivity and praise. Smart.

Before getting to what comes next, I’d like to share some thanks.

I love and respect Ilya, I think he’s a guiding light of the field and a gem of a human being. I harbor zero ill will towards him. While Ilya will no longer serve on the board, we hope to continue our working relationship and are discussing how he can continue his work at OpenAI.

I am grateful to Adam, Tasha, and Helen for working with us to come to this solution that best serves the mission. I’m excited to continue to work with Adam and am sincerely thankful to Helen and Tasha for investing a huge amount of effort in this process.

Thank you also to Emmett who had a key and constructive role in helping us reach this outcome. Emmett’s dedication to AI safety and balancing stakeholders’ interests was clear.

Mira did an amazing job throughout all of this, serving the mission, the team, and the company selflessly throughout. She is an incredible leader and OpenAI would not be OpenAI without her. Thank you.

Greg and I are partners in running this company. We have never quite figured out how to communicate that on the org chart, but we will. In the meantime, I just wanted to make it clear. Thank you for everything you have done since the very beginning, and for how you handled things from the moment this started and

...
[Linkpost] Remarks on the Convergence in Distribution of Random Neural Networks to Gaussian Processes in the Infinite Width Limit
by carboniferous_umbraculum  — 2023-11-30T14:01:32.255Z
Link: https://drive.google.com/file/d/1O_RbD45RjBWD4-a8tcjosvyXOxcD_q2V/view?usp=sharing

The linked note is something I "noticed" while going through different versions of this result in the literature. I think that this sort of mathematical work on neural networks is worthwhile and worth doing to a high standard but I have no reason to think that this particular work is of much consequence beyond filling in a gap in the literature. It's the kind of nonsense that someone who has done too much measure theory would think about.
Abstract. We describe a direct proof of yet another version of the result that a sequence of fully-connected neural networks converges to a Gaussian process in the infinite-width limit. The convergence in distribution that we establish is the weak convergence of probability measures on the non-separable, non-

metrizable

product space (Rd′)Rd, i.e. the space of functions from Rd to Rd′ with the topology whose convergent sequences correspond to

pointwise

convergence. The result itself is already implied by a stronger such theorem due to Boris

Hanin

, but the direct proof of our weaker result can afford to replace the more technical parts of

Hanin's

proof that are needed to establish tightness with a shorter and more abstract measure-theoretic argument.
Buy Nothing Day is a great idea with a terrible app— why has nobody built a killer app for crowdsourced 'effective communism' yet?
by lillybaeum — 2023-11-30T13:47:37.475Z

I've been seeing a lot of people really excited about very leftist ideas on TikTok lately, as well as being very frustrated at the way that the 'developed' world exists, and I keep getting this idea of an app that would finally create some sort of framework for the average person to produce mutual aid.

The basic idea is that you could list things that you want to share. Other people can list things they need. The only limits would be, officially, no prepared foods, for legal reasons, and maybe other things that I haven't forseen. And that would be it! An app to share things you have extra of, with your neighbors.

So, that's been stewing around in my mind for a little bit, and then today, what do you know? I discover that 'Buy Nothing Day', a movement around the idea of a day where nobody buys anything, has an app like this! And it has existed for two years! But it sucks. All the reviews claim that basic functionality like making posts is busted despite praising the idea.

And my question is ultimately this: why has there not been a version of this sort of thing yet? Have economic conditions simply not been bad enough for people to want to help and rely on their neighbors?

Did craigslist pave the way despite becoming a cesspool of overly expensive crap and deceptively listed '$1' ads for businesses, that require you to drive 8 miles to pick it up while someone side-eyes you suspiciously from their driveway and waits for the venmo to go through?

Is this idea not sustainable or good in some way I'm missing? It seems like a no brainer to me and nothing but a public good. People could point to some theoretical stranger danger, but that just seems like razorblades-in-halloween-candyesque fearmongering.