trentmkelly/LessWrong-43k huggingface dataset:
Estimating effective dimensionality of MNIST models
by Arjun Panickssery — 2023-11-02T14:13:09.012Z

The local learning coefficent λ is a measure of a model's "effective dimensionality" that captures its complexity (more background here). Lau et al recently described a sampling method (SGLD) using noisy gradients to find a stochastic estimate ^λ in a computationally tractable way (good explanation here):

Examples of points sampled by SGLD in a two-dimensional toy model (Lau et al, 2023)

I present results (Github repo) of the "Task Variability" project suggested on the DevInterp project list. To see how degeneracy scales with task difficulty and model class, I trained a fully-connected MLP and a CNN (both with ~120k parameters) on nine MNIST variants with different subsets of the labels (just the labels {0, 1}, then {0, 1, 2}, etc.). All models were trained to convergence using the same number of training data points. I implement Lau et al's algorithm on each of the trained models. The results below are averaged over five runs:

The results for the full MNIST dataset are comparable to Lau et al's results while using models ten times smaller trained for ten times fewer epochs.

The sampling method is finicky and sensitive to the hyperparameter choices of learning rate and noise factor. It will fail by producing negative or very high ^λ values if the noise ϵand the distance penalty γ (see lines 6 and 7 in the pseudocode above) isn't calibrated to the model's level of convergence. The results show a linear scaling law relating number of labels to task complexity. The CNN typically has a lower ^λ than the MLP, which matches intuitions that some of the complexity is "stored" in the architecture because the convolutions apply a useful prior on functions good at solving image recognition tasks.
Averages and sample sizes
by mruwnik — 2023-11-02T09:52:43.576Z

My brain seems to not get statistics and probability. So I was wondering about how averaging things out is influenced by the amount of items being averaged, and in particular, how the underlying distribution will change things. Here are some stories to illustrate the issue:

Minting

You’re a Dutch banker working for the famous Wisselbank. Your job is to check whether the gold coins coming in are good. You know that each coin should weigh 25g ± 0.5g (I’m taking a lot of liberties with this story). Your method is to count out 100 coins, and see if they weigh 2.5kg. You just got a batch that weighs 2480g, which is 20g less than expected. But since each coin is ± 0.5g, and you have 100 coins, that should be ± 50g, right? Or does this suggest that someone is trying to cheat you by making slightly lighter coins, and pocketing the difference in gold?

Network errors

You’re on monitoring duty at work. Previous tests showed that something like 2 requests out of every 1000 caused errors for random reasons (e.g. Russian hackers are probing your servers, or a user clicked too fast) and so can be safely ignored (there’s a task to fix that, but keeps being left for the next sprint). You just got an alarm in the middle of the night - over the last hour, there were 6840 network errors, at a request rate of 1k/s. This is 0.19% of all the requests, and the known error rate was 0.2%. Should you go back to sleep and have a look at it in the morning, or is this a real issue?

Party!!

You like organizing parties for your friends. Usually, when you invite 100 people, some people don’t show up, while others bring friends, so you assume that there will be between 90 - 110 people. You want to organize a weekly meeting for the next year, but don’t know how much swag to order. Assuming 100 people have signed up, and you have a very good deal lined up to get all the resources for the whole year today (i.e. for all 52 weeks), how much should you order?

Bird watching

You’re an ornithologist that is studying the lesser spotted woodpecker (or any other species). You want to work out how many of them can be supported by a hectare of woodland. You’re going to visit various patches of forest (or maybe send your friends?) and count how many birds could be found during each trip. You know there is a magical way of calculating this, called hierarchical mixed models or something, but the only guy who actuall...
Upcoming Feedback Opportunity on Dual-Use Foundation Models
by Chris_Leong — 2023-11-02T04:28:11.586Z

From Biden's recent executive order:

"4.6. Soliciting Input on Dual-Use Foundation Models with Widely Available Model Weights. When the weights for a dual-use foundation model are widely available — such as when they are publicly posted on the Internet — there can be substantial benefits to innovation, but also substantial security risks, such as the removal of safeguards within the model. To address the risks and potential benefits of dual-use foundation models with widely available weights, within 270 days of the date of this order, the Secretary of Commerce, acting through the Assistant Secretary of Commerce for Communications and Information, and in consultation with the Secretary of State, shall:
(a) solicit input from the private sector, academia, civil society, and other stakeholders through a public consultation process on potential risks, benefits, other implications, and appropriate policy and regulatory approaches related to dual-use foundation models for which the model weights are widely available, including:

(i) risks associated with actors fine-tuning dual-use foundation models for which the model weights are widely available or removing those models’ safeguards;

(ii) benefits to AI innovation and research, including research into AI safety and risk management, of dual-use foundation models for which the model weights are widely available; and

(iii) potential voluntary, regulatory, and international mechanisms to manage the risks and maximize the benefits of dual-use foundation models for which the model weights are widely available; and
(b) based on input from the process described in subsection 4.6(a) of this section, and in consultation with the heads of other relevant agencies as the Secretary of Commerce deems appropriate, submit a report to the President on the potential benefits, risks, and implications of dual-use foundation models for which the model weights are widely available, as well as policy and regulatory recommendations pertaining to those models."

This seems important since the open-sourcing of model weights essentially leads to the irreversible proliferation of capabilities. I suspect that there's a surprising...
Public Weights?
by jefftk — 2023-11-02T02:50:18.095Z

While this is close to areas I work in, it's a personal post. No
one reviewed this before I published it, or asked me to (or not to)
write something. All mistakes are my own.

A few days ago, some of my coworkers at SecureBio put out a preprint, "Will
releasing the weights of future large language models grant widespread
access to pandemic agents?" (Gopal et. al 2023)
They took Facebook/Meta's Llama-2-70B large language model (LLM) and
(cheaply!) adjusted it to remove the built in safeguards, after which
it was willing to answer questions on how to get infectious 1918 flu. I like
a bunch of things about the paper, but I also think it suffers from
being undecided on whether it's communicating:

Making LLMs public is dangerous because by publishing the
weights you allow others to easily remove safeguards.

Once you remove the safeguards, current LLMs are already
helpful in getting at the key information necessary to cause a
pandemic.

I think it demonstrates the first point pretty well. The main way we
avoid LLMs from telling people how to cause harm is to train them on a
lot of examples of someone asking how to cause harm and being told
"no", and this can easily be reversed by additional training with
"yes" examples. So even if you get incredibly good at this, if you
make your LLM public you make it very easy for others to turn it into
something that compliantly shares any knowledge it contains.

Now, you might think that there isn't actually any dangerous
knowledge, at least not within what an LLM could have learned from
publicly available sources. I think this is pretty clearly not true:
the process of creating infectious 1918 flu is scattered across the
internet and hard for most people to assemble. If you had an
experienced virologist on call and happy to answer any question,
however, they could walk you there through a mixture of doing things
yourself and duping others into doing things. And if they were able to
read and synthesize all virology literature they could tell you how to
create things quite a bit worse than this former pandemic.

GPT-4 is already significantly better than Llama-2, and GPT-5 in 2024
is more
likely than not. Public models will likely continue to move
forward, and while it's unlikely
that we get a GPT-4 level Llama-3 in 2024 I do think the default
path involves very good public models within a few yea...
Should people build productizations of open source AI models?
by lc — 2023-11-02T01:26:47.516Z

In early 2021, before I had developed any opinions on AI safety, my startup cofounder and I were contacted by a group of people who wanted an alternative to an AI CYOA platform, AIDungeon. They were frustrated with the site's current management and especially their decision to start banning users for writing NSFW content. We were almost out of money at that point, so we built a new frontend for them over an EleutherAI model that encrypted user content and didn't filter. Surprisingly to us, the site ended up doing better than anything else we had tried so far, at least enough to tide us over.

We worked on HoloAI for around six months after that. Then between Dec 2021 and April 2022, my model of AI R&D went from "neutral tech research" to "the worst thing you could possibly be doing". While we were not AI researchers, and in retrospect HoloAI was unlikely to have grown massively even if we had kept at it, we decided to stop making improvements. The site is still online, but nowadays I just answer support requests for the people that already use it as an editor.

I still very much agree with our choice to switch to a field that was AI unrelated, but nowadays I'm not actually sure if the concerns about X-risk were real. We never did any foundational model research, we weren't paying OpenAI for tokens or competing against them for customers, and HoloAI didn't release code that could be used with other models. On the other hand, it seems plausible that products like HoloAI have a downstream effect, where by competing in the AI product category, you indirectly encourage others to make those innovations. We weren't paying OpenAI, but there were similar products on the market that did pay OpenAI, and now they have to do something stay ahead.

So what should be the social consensus about these sorts of utilities? Or for that matter, any software that uses proprietary models built by the research labs?

Part of my reason for asking this question here is because I expect many of the new companies in the next 5-10 years to be, to some degree, AI productizations. We landed on HoloAI mostly by chance, but it's possible putting some kind of LLM in various parts of the economy is going to be a large proportion of how the technology sector grows going forward. Further, lots of people in the tech industry will be working on these tools or supporting them, and "d...
Singular learning theory and bridging from ML to brain emulations
by kave — 2023-11-01T21:31:54.789Z

This is a fairly "chat"-style dialogue that I (kave) had with Garrett about singular learning theory (SLT) and his ambitious plans for solving ambitious value learning by building off of it.

A colleague found this gave them better trailheads for SLT than current expositions (though they're still confused) and I got a much clearer sense of the scope of Garrett's hopes from this conversation than from his post alone.

What is Singular Learning Theory?

kave

Hey Garrett! Here are a couple of my initial thoughts regarding your hopes for singular learning theory (SLT)!

As far as SLT goes, I feel like I don't really understand what's exciting for folks about "phase transitions". Or like, I don't understand how to connect "phase transitions are rapid changes in the density of states" to "phase transitions are grokking" and stuff like that. I might be confused about that being the angle that folks are going for.

I'm also confused about how to think about SLT for both brains and ML systems. Like, there's some notion of a dynamical system that SLT applies to that applies to both those things (you think/hope). What is that notion?

I'm also excited to hear more about your vision for a programme that spills out of SLT that becomes something like a theory of learning agents that "touches grass" all the way along its development (to the extent that's true).

Anyway, that's a little babble about where I'm at. Open to going anywhere, but thought I'd share some state.

Garrett Baker

Yeah, okay, so, I guess the first thing is phase transitions. I don't quite understand your confusion. Like, let's take it as a given that gradient descent can be adequately described as sampling out of a Bayesian posterior. Then, rapid changes of density of states from less generalizing solutions to more generalizing solutions directly corresponds to grokking? And we get grokking when we have a rapid transition from a less generalizing solution to a more generalizing solution.

kave

OK but why does rapid change of density of states correspond to more generalising?

kave

My best current understanding: a rapid change in the density of states corresponds to something like more parameters becoming irrelevant (or fewer? Might have that backwards) and so you become effectively a smaller model / more parsimonious / less epicircular

kave

[Feel free to say if I seem horribly confused / my intuitions seem like I misunderstood w

...
My thoughts on the social response to AI risk
by Matthew Barnett — 2023-11-01T21:17:08.184Z

A common theme implicit in many AI risk stories has been that broader society will either fail to anticipate the risks of AI until it is too late, or do little to address those risks in a serious manner. In my opinion, there are now clear signs that this assumption is false, and that society will address AI with something approaching both the attention and diligence it deserves. For example, one clear sign is Joe Biden's recent executive order on AI safety[1]. In light of recent news, it is worth comprehensively re-evaluating which sub-problems of AI risk are likely to be solved without further intervention from the AI risk community (e.g. perhaps deceptive alignment), and which ones will require more attention.

While I'm not saying we should now sit back and relax, I think recent evidence has significant implications for designing effective strategies to address AI risk. Since I think substantial AI regulation will likely occur by default, I urge effective altruists to focus more on ensuring that the regulation is thoughtful and well-targeted rather than ensuring that regulation happens at all. Ultimately, I argue in favor of a cautious and nuanced approach towards policymaking, in contrast to broader public AI safety advocacy.[2]

In the past, when I've read stories from AI risk adjacent people about what the future could look like, I have often noticed that the author assumes that humanity will essentially be asleep at the wheel with regards to the risks of unaligned AI, and won't put in place substantial safety regulations on the technology—unless of course EA and LessWrong-aligned researchers unexpectedly upset the gameboard by achieving a pivotal act. We can call this premise the assumption of an inattentive humanity.[3]

While most often implicit, the assumption of an inattentive humanity was sometimes stated explicitly in people's stories about the future.

For example, in a post from Marius Hobbhahn published last year about a realistic portrayal of the next few decades, Hobbhahn outlines a series of AI failure modes that occur as AI gets increasingly powerful. These failure modes include a malicious actor using an AI model to create a virus that "kills ~1000 people but is stopped in its tracks because the virus kills its hosts faster than it spreads", an AI model attempting to escape its data center after having "tried to establish a c...
Reactions to the Executive Order
by Zvi — 2023-11-01T20:40:02.438Z

Previously: On the Executive Order

This post compiles the reactions of others that I have seen to Biden’s Executive Order on AI, including reactions that were based only on the fact sheet, as well as my reactions to those reactions.

Reaction on the worried side was measured. It could best be described as cautious optimism.

Reaction on the unworried side was sometimes measured, but often not measured. It could perhaps be frequently described as unhinged.

It continues to be odd to see so many voices react in such horror to the idea that the government might not ultimately adapt a fully laissez faire approach to AI.

Many of them collectively seem to be, essentially, treating a request for government reports on what might be done in the future, plus some very mild reporting requirements imposed exclusively on a few giant corporations, as if it inevitably means AI, nay computers in general, nay the very core of mathematics itself, will suffer the fate of NEPA or IRBs, a slippery slope of regulatory ratcheting until all hope for the future is extinguished.

I am unusually sympathetic to this view. Such things very much do happen. They very much do often happen slowly. They are indeed strangling much of our civilization. This is all very bad. Pick almost any other hill, everyone involved, where often this is actually already happening and doing great harm, and there are not the massive externalities of potentially everyone on the planet dying, and I would be happy to stand with you.

Alas, no, all that progress energy is focused on the one place where I fear it is deeply misguided. What should be the default viewpoint and voice of reason across the board is silenced everywhere except the one place I wish it was quieter.

I’ll divide the post into three sections. First, the measured reactions, to the fact sheet and then the final executive order. Then those crying out about what can be pried from their cold dead hands.

Also here is a useful tool: A compilation of all the deadlines in the EO. And here is a tool for navigating the EO, file under things that could have been brought to my attention yesterday.

And before I begin: Yes, it is terrible that we keep Declaring Defense Production Act.

Fact Sheet Reactions

Vivek Chilukuri has a thread summarizing the fact sheet.

Vivek Chilukuri: The EO is the Admin’s strongest effort yet to lead by example in the responsible de

...
Dario Amodei’s prepared remarks from the UK AI Safety Summit, on Anthropic’s Responsible Scaling Policy
by Zac Hatfield-Dodds — 2023-11-01T18:10:31.110Z
Link: https://www.anthropic.com/index/uk-ai-safety-summit

I hope Dario's remarks to the Summit can shed some light on how we think about RSPs in general and Anthropic's RSP in particular, both of which have been discussed extensively since I shared our RSP announcement. The full text of Dario's remarks follows:

Before I get into Anthropic’s Responsible Scaling Policy (RSP), it’s worth explaining some of the unique challenges around measuring AI risks that led us to develop our RSP. The most important thing to understand about AI is how quickly it is moving. A few years ago, AI systems could barely string together a coherent sentence. Today they can pass medical exams, write poetry, and tell jokes. This rapid progress is ultimately driven by the amount of available computation, which is growing by 8x per year and is unlikely to slow down in the next few years. The general trend of rapid improvement is predictable, however, it is actually very difficult to predict when AI will acquire specific skills or knowledge. This unfortunately includes dangerous skills, such as the ability to construct biological weapons. We are thus facing a number of potential AI-related threats which, although relatively limited given today’s systems, are likely to become very serious at some unknown point in the near future. This is very different from most other industries: imagine if each new model of car had some chance of spontaneously sprouting a new (and dangerous) power, like the ability to fire a rocket boost or accelerate to supersonic speeds.

We need both a way to frequently monitor these emerging risks, and a protocol for responding appropriately when they occur. Responsible scaling policies—initially suggested by the Alignment Research Center—attempt to meet this need. Anthropic published its RSP in September, and was the first major AI company to do so. It has two major components:

First, we’ve come up with a system called AI safety levels (ASL), loosely modeled after the internationally recognized BSL system for handling biological materials. Each ASL level has an if-then structure: if an AI system exhibits certain dangerous capabilities, then we will not deploy it or train more powerful models, until certain safeguards are in place.

Second, we test frequently for these dangerous capabilities at regular intervals along the compute scaling curve. This is to ensure that we don’t blindly create dangerous capabilities without even

...
Book Review: Determined by Sapolsky
by Kailuo Wang — 2023-11-01T17:37:34.925Z

Determined: Life Without Free Will is a misguided attempt at moral reasoning based on scientific facts. Lacking a philosophical framework that can establish connections between morality and science, the author relied on his own rather lenient intuition without realizing it. One might also say that he is another victim who falls on false philosophical questions. This review contains two parts: a negative part criticizing the book at the principle level and a positive part trying to tackle the dilemma of determinism.

Part I: A book with bipolar attitudes

Sapolsky conceptualizes 'free will' as a governing element inside a body, free from physical laws, thereby qualifying it as supernatural. This intuitive definition is not inherently wrong, albeit not that useful in some philosophical views (I'll come back to this later). He devoted half of the book rigorously disputing against the existence of such a supernatural free will, of which there are many useful scientific insights. This is a view that is already readily embraced by all naturalists - for whom the whole universe is governed by physical law and "natural." And for antinaturalists, it's doubtful that any amount of empirical evidence will change their mind.

What is more problematic is when the book ventures to analyze the moral implications of the nonexistence of such supernatural free will. Had Sapolsky maintained his naturalist rigor, he would have discerned the absence of an established empirical grounding for morality as well, that is, it is not something natural that obeys physical laws (unless one subscribes moral naturalism). If one rejects the whole notion of free will due to the lack of empirical evidence substantiating its existence, they would have no choice but to reject the whole notion of morality on the same ground. This would render any moral proclamations meaningless, of which the book contains an abundance.

If one wants to reason in morality with rigor, they must start with a solid philosophical foundation rather than just their own casual day-to-day moral thinking. One of the first philosophical questions the author should've asked himself might be how morality holds significance without empirical evidence substantiating its existence. Rather, his lack of awareness in this area is utterly disappointing, sometimes to the point of frustration.

Thus, the book treats the two main subj...