trentmkelly/LessWrong-43k huggingface dataset:
Why I stopped being into basin broadness
by tailcalled — 2024-04-25T20:47:17.288Z

There was a period where everyone was really into basin broadness for measuring neural network generalization. This mostly stopped being fashionable, but I'm not sure if there's enough written up on why it didn't do much, so I thought I should give my take for why I stopped finding it attractive. This is probably a repetition of what others have found, but I thought I might as well repeat it.

Let's say we have a neural network fw(x):Rn. We evaluate it on a dataset (x,y)∼D using a loss function L(^y,y):R, to find an optimum w∗=argminwE(x,y)∼D[L(fw(x),y)]. Then there was an idea going around that the Hessian matrix (i.e. the second derivative of E(x,y)∼D[L(fw(x),y)] at w∗) would tell us something about w∗ (especially about how well it generalizes).

If we number the dataset (xi,yi), we can stack all the network outputs ^yi(w)=fw(xi) which fits into an empirical loss ^L(^y)=1n∑ni=1L(^yi,yi). The Hessian that we talked about before is now just the Hessian of ^L(^y(w)). Expanding this out is kind of clunky since it involves some convoluted tensors that I don't know any syntax for, but clearly it consists of two terms:

The Hessian of ^L with a pair of the Jacobian of ^y on each end (this can just barely be written without crazy tensors: (Jw^y(w))T(H^y^L(^y))∣^y(w)Jw^y(w))
The gradient of ^L with a crazy second derivative of ^y.

Now, the derivatives of ^L are "obviously boring" because they don't really refer to the neural network weights, which is confirmed if you think about it in concrete cases, e.g. if L(^y,y)=−ylog(^y)−(1−y)log(1−^y) with y=1 or y=0, the derivatives just quantify how far ^y is from y. This obviously isn't relevant for neural network generalization, except in the sense that it tells you which direction you want to generalize in.

Meanwhile, Jw^y(w) is incredibly strongly related to neural network generalization, because it's literally a matrix which specifies how the neural network outputs change in response the weights. In fact, it forms the core of the neural tangent kernel (a standard tool for modelling neural network generalization), because the NTK can be expressed as Jw^y(w)(Jw^y(w))T.

The "crazy second derivative of ^y" can I guess be understood separately for each...
AXRP Episode 29 - Science of Deep Learning with Vikrant Varma
by DanielFilan — 2024-04-25T19:10:06.063Z

YouTube link

In 2022, it was announced that a fairly simple method can be used to extract the true beliefs of a language model on any given topic, without having to actually understand the topic at hand. Earlier, in 2021, it was announced that neural networks sometimes ‘grok’: that is, when training them on certain tasks, they initially memorize their training data (achieving their training goal in a way that doesn’t generalize), but then suddenly switch to understanding the ‘real’ solution in a way that generalizes. What’s going on with these discoveries? Are they all they’re cracked up to be, and if so, how are they working? In this episode, I talk to Vikrant Varma about his research getting to the bottom of these questions.

Topics we discuss:

Challenges with unsupervised LLM knowledge discovery, aka contra CCS

What is CCS?

Consistent and contrastive features other than model beliefs

Understanding the banana/shed mystery

Future CCS-like approaches

CCS as principal component analysis

Explaining grokking through circuit efficiency

Why research science of deep learning?

Summary of the paper’s hypothesis

What are ‘circuits’?

The role of complexity

Many kinds of circuits

How circuits are learned

Semi-grokking and ungrokking

Generalizing the results

Vikrant’s research approach

The DeepMind alignment team

Follow-up work

Daniel Filan:
Hello, everybody. In this episode I’ll be speaking with Vikrant Varma, a research engineer at Google DeepMind, and the technical lead of their sparse autoencoders effort. Today, we’ll be talking about his research on problems with contrast-consistent search, and also explaining grokking through circuit efficiency. For links what we’re discussing, you can check the description of this episode and you can read the transcript at axrp.net.

All right, well, welcome to the podcast.

Vikrant Varma:
Thanks, Daniel. Thanks for having me.

Challenges with unsupervised LLM knowledge discovery, aka contra CCS

What is CCS?

Daniel Filan:
Yeah. So first, I’d like to talk about this paper. It is called Challenges with Unsupervised LLM Knowledge Discovery, and the authors are Sebastian Farquhar, you, Zachary Kenton, Johannes Gasteiger, Vladimir Mikulik, and Rohin Shah. This is basically about this thing called CCS. Can you tell us: what...
Improving Dictionary Learning with Gated Sparse Autoencoders
by Senthooran Rajamanoharan — 2024-04-25T18:43:47.003Z
Link: https://arxiv.org/abs/2404.16014

Authors: Senthooran Rajamanoharan*, Arthur Conmy*, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, Neel Nanda

A new paper from the Google DeepMind mech interp team: Improving Dictionary Learning with Gated Sparse Autoencoders!

Gated SAEs are a new Sparse Autoencoder architecture that seems to be a significant Pareto-improvement over normal SAEs, verified on models up to Gemma 7B. They are now our team's preferred way to train sparse autoencoders, and we'd love to see them adopted by the community! (Or to be convinced that it would be a bad idea for them to be adopted by the community!)

They achieve similar reconstruction with about half as many firing features, and while being either comparably or more interpretable (confidence interval for the increase is 0%-13%).

See Sen's Twitter summary, my Twitter summary, and the paper!
"Why I Write" by George Orwell (1946)
by Arjun Panickssery — 2024-04-25T16:02:28.668Z
Link: https://www.orwellfoundation.com/the-orwell-foundation/orwell/essays-and-other-works/why-i-write/

People have been posting great essays so that they're "fed through the standard LessWrong algorithm." This essay is in the public domain in the UK but not the US.

From a very early age, perhaps the age of five or six, I knew that when I grew up I should be a writer. Between the ages of about seventeen and twenty-four I tried to abandon this idea, but I did so with the consciousness that I was outraging my true nature and that sooner or later I should have to settle down and write books.

I was the middle child of three, but there was a gap of five years on either side, and I barely saw my father before I was eight. For this and other reasons I was somewhat lonely, and I soon developed disagreeable mannerisms which made me unpopular throughout my schooldays. I had the lonely child’s habit of making up stories and holding conversations with imaginary persons, and I think from the very start my literary ambitions were mixed up with the feeling of being isolated and undervalued. I knew that I had a facility with words and a power of facing unpleasant facts, and I felt that this created a sort of private world in which I could get my own back for my failure in everyday life. Nevertheless the volume of serious – i.e. seriously intended ­– writing which I produced all through my childhood and boyhood would not amount to half a dozen pages. I wrote my first poem at the age of four or five, my mother taking it down to dictation. I cannot remember anything about it except that it was about a tiger and the tiger had ‘chair-like teeth’ – a good enough phrase, but I fancy the poem was a plagiarism of Blake’s ‘Tiger, Tiger’. At eleven, when the war or 1914-18 broke out, I wrote a patriotic poem which was printed in the local newspaper, as was another, two years later, on the death of Kitchener. From time to time, when I was a bit older, I wrote bad and usually unfinished ‘nature poems’ in the Georgian style. I also, about twice, attempted a short story which was a ghastly failure. That was the total of the would-be serious work that I actually set down on paper during all those years.

However, throughout this time I did in a sense engage in literary activities. To begin with there was the made-to-order stuff which I produced quickly, easily and without much pleasure to myself. Apart from school work, I wrote vers d’occasion, semi-comic poems which I could turn out at what now...
Knowledge Base 8: The truth as an attractor in the information space
by iwis — 2024-04-25T15:28:58.210Z

Hypothesis: The truth, defined as the information with 100% credibility calculated by the HITS algorithm in accordance with the consensus theory of truth, is an attractor in the information space.

Interface between all intelligent agents

The post Knowledge Base 2: The structure and the method of building describes the possibility of building a knowledge database using crowdsourcing. After adding initial knowledge using crowdsourcing, also artificial intelligence and robots could add knowledge to this database during the execution of tasks, for example, while finding answers to the questions of their users or users of the database. People, computer programs, and robots could collaborate with each other to add and evaluate knowledge, increasing the amount and credibility of useful knowledge. Thus, this database could be a common interface for the exchange of knowledge, questions, and tasks between all intelligent agents including people and AI.

Information space and agents’ actions within it

The proposed knowledge database can be seen as an information space in which all intelligent agents can add and evaluate pieces of information, regardless of whether they are people or computer programs. A point in this space is represented by a semantic triple, e.g. the (Laptop, Is, Computer) relation, and the probability of the correctness of this triple. An agent’s goal can be answering questions correctly or any other meta-goal, such as making money or acting for the benefit of others. Agents can plan their actions to best achieve their goals. Planning the process of answering a question involves determining how to carry out the inference and whether additional questions need to be asked to other agents. Planning the meta-goal execution can involve determining what questions to answer and what to do to better achieve one’s goal in the future. Thanks to assessing the agents’ credibility, they would act towards increasing the amount and credibility of useful knowledge. Other agents can copy the methods of the most effective ones. This would increase the collective intelligence of the whole system. Finally, because of the human-readable form of the database, it would be easier to control the safety of the whole system.

Attractor in the information space

In the Q&A session after the Intelligence Stairway talk at the University of Sydney in 2012, Jaan Tallinn said that an att...
Cybersecurity of Frontier AI Models: A Regulatory Review
by Deric Cheng — 2024-04-25T14:51:20.272Z

This article is part of a series of ~10 posts comprising a 2024 State of the AI Regulatory Landscape Review, conducted by the

Governance Recommendations Research Program

at

Convergence Analysis

. Each post will cover a specific domain of AI governance (e.g.

incident reporting

,

safety evals

, model registries, etc.). We’ll provide an overview of existing regulations, focusing on the US, EU, and China as the leading governmental bodies currently developing AI legislation. Additionally, we’ll discuss the relevant context behind each domain and conduct a short analysis.

This series is intended to be a primer for policymakers, researchers, and individuals seeking to develop a high-level overview of the current AI governance space. We’ll publish individual posts on our website and release a comprehensive report at the end of this series.

What cybersecurity issues arise from the development of frontier AI models?

One of the primary issues that has caught the attention of regulators is the protection of the intellectual property and sensitive data associated with frontier AI models (otherwise named as

“dual-use foundational models”

by US legislation and

“general-purpose AI” (“GPAI”)

by EU legislation).

In particular, legislators are concerned that as frontier AI models increase their capabilities, unregulated access to the underlying code or abilities of these models will result in dangerous outcomes. For example, current AI models are susceptible to easily distributing

information hazards

, such as the instructions to develop homemade weapons or techniques to commit crimes. As a result, they’re typically trained during a fine-tuning phase to reject such requests. Bypassing the cybersecurity of such models could result in the removal of such fine-tuning, allowing dangerous requests. Other cybersecurity risks include sharing sensitive user data, or leaking proprietary ML architectural decisions with direct competitors & geopolitical adversaries (e.g. Chinese organizations, in the case of the US).

Currently, the leading frontier AI models meet the following conditions, which are often collectively referred to as “closed-source” development:

Are privately owned by a large AI lab (e.g. OpenAI, Anthropic, or Google)
Present an API interface to fine-tuned models that are designed to reject dangerous or adversarial inputs

...
The first future and the best future
by KatjaGrace — 2024-04-25T06:40:04.510Z

It seems to me worth trying to slow down AI development to steer successfully around the shoals of extinction and out to utopia.

But I was thinking lately: even if I didn’t think there was any chance of extinction risk, it might still be worth prioritizing a lot of care over moving at maximal speed. Because there are many different possible AI futures, and I think there’s a good chance that the initial direction affects the long term path, and different long term paths go to different places. The systems we build now will shape the next systems, and so forth. If the first human-level-ish AI is brain emulations, I expect a quite different sequence of events to if it is GPT-ish.

People genuinely pushing for AI speed over care (rather than just feeling impotent) apparently think there is negligible risk of bad outcomes, but also they are asking to take the first future to which there is a path. Yet possible futures are a large space, and arguably we are in a rare plateau where we could climb very different hills, and get to much better futures.
NIH Cancer Myths Myths
by belkarx — 2024-04-25T05:43:47.341Z

The NIH has a page called Cancer Myths and Misconceptions that you come across if you end up looking into cancer for long enough, aimed at bio-illiterate patients and their families.

Around half the things on that page are wrong at face value, and a solid percentage of those are contradicted by the pages and studies the NIH themselves link as a part of the answer.

This seems bad. The percentage of people that are going to look through the actual studies or even linked cancer.gov pages with expanded info instead of looking at the NIH's incorrect summaries is low, so most people end up getting the wrong impression and making care/preventative decisions based off of that.

The trend is that they are identifying statements that are inconclusive as "myths", implying that they've been disproven and can be safely ignored, when this is clearly untrue.

I present a revised "NIH Cancer Myths Myths" page

Format: <Class of thing that the NIH did wrong for this "myth"> followed by why it's misleading and some more correct takes (mostly without linked supporting papers, sorry, I'll go back and add them at some point if I feel like it - this is a "source: trust me bro, I looked into most of these thoroughly at various points in my life" writeup :).

Error: Not endorsing "conspiracies" even when some amount of caution is probably warranted, considering the literature

Rhe Cell Phones page has studies that have positive results for cell phone use increasing at least acoustic neuromas, but each study is discredited by stating their honestly on-par-for-academia flaws (not done for other studies linked on the main page).

The EMF and Cancer page also spends most of the time claiming the existing studies are bad/underpowered, and glazes over the one "good" study that supports significantly increased risk of various cancers for workers with high EMF exposure.

Error: Giving misleading answers and then not elaborating

This response appears to discourage "holistic" treatments with "no herbal products have been shown to be effective for treating cancer", despite a large body of evidence to the contrary (like green tea reliably slowing metastasis, and garlic for slowing tumor growth by immune system support + a bunch of other pathways (GARLIC IS SO OP)). Their linked "more information" page discusses everything that doesn't work, and requires 2+ more clickthroughs to get to any act

...
social lemon markets
by bhauth — 2024-04-25T02:18:04.480Z
Link: https://www.bhauth.com/blog/culture/lemon%20markets.html

I refuse to join any club that would have me as a member.

— Groucho Marx

Alice and Carol are walking on the sidewalk in a large city, and end up together for a while.

"Hi, I'm Alice! What's your name?"

Carol thinks:

If Alice is trying to meet people this way, that means she doesn't have a much better option for meeting people, which reduces my estimate of the value of knowing Alice. That makes me skeptical of this whole interaction, which reduces the value of approaching me like this, and Alice should know this, which further reduces my estimate of Alice's other social options, which makes me even less interested in meeting Alice like this.

Carol might not think all of that consciously, but that's how human social reasoning tends to work.

In most cases, Alice understands this and doesn't even try to talk with Carol in the first place.

lemon markets

Economics has the concept of a "lemon market". When sellers have more information about item quality than buyers, and reputation isn't a factor, sellers will try to sell low-quality items, and consumers will respond by driving down prices until only low-quality items are worth selling.

I think a similar thing has happened for social relationships in many countries over the past several decades.

An increase in social isolation over time has been widely noted; Bowling Alone was written in 2000, and the issues it described have increased since then. My view is that the above dynamic is responsible for much of that change.

What are the root causes of social relationships becoming more of a lemon market? There are many causes, but 4 causes I consider significant are listed below.

urbanization

In many countries, people have rapidly moved to bigger cities over the past several decades, and this affects social dynamics.

Human relationships generally have some upfront costs and then a net benefit over time. In very small towns, your chance of having a relationship with a random resident is higher, so the initial upfront costs are amortized over a higher probability of establishing a good relationship.

In a very small town, if Dave dates 3 girls at the same time and treats them all poorly, afterwards, every girl in the town will know to avoid him. Bad behavior is less attractive when random people you encounter in the future are more likely to be aware of it, and bad behavior being less likely means relationships with random...
Bayesian inference without priors
by DanielFilan — 2024-04-24T23:50:08.312Z

Epistemic status: party trick

Why remove the prior

One famed feature of Bayesian inference is that it involves prior probability distributions. Given an exhaustive collection of mutually exclusive ways the world could be (hereafter called ‘hypotheses’), one starts with a sense of how likely the world is to be described by each hypothesis, in the absence of any contingent relevant evidence. One then combines this prior with a likelihood distribution, which for each hypothesis gives the probability that one would see any particular set of evidence, to get a posterior distribution of how likely each hypothesis is to be true given observed evidence. The prior and the likelihood seem pretty different: the prior is looking at the probability of the hypotheses in question, whereas the likelihood is looking at the probability of the evidence (assuming the hypothesis is true).1

Critics of Bayesian inference sometimes denounce the reliance on priors for being subjective or unscientific. Indeed, they are by design meant to describe what one would think without any relevant (contingent) data. One might therefore be tempted to describe a form of Bayesian inference where no special role is played by the prior distribution, as distinct from the likelihood.

Another motivation comes from doing Bayesian calculations by hand. In real-world cases, such as trying to infer whether the first COVID-19 outbreak spread from a laboratory or human contact with infected animals, the kind of thinking one does to determine a prior probability distribution is very similar to the kind of thinking one does to determine likelihoods: in both cases, one has some sort of generative model in mind—that is, some sort of probabilistic process of generating worlds—and one is trying to figure out how often worlds produced by this generative model have various properties. This might make one wonder if one could unify the prior and the likelihood.

How to remove the prior (by turning it into a likelihood)

So, how are we going to do this?

First, a prerequisite. I’m going to be talking about the “odds ratio” form of Bayes’ theorem. This involves comparing the ratio of the probabilities of two hypotheses—that is, asking questions like “how many times more likely is the COVID outbreak to be a lab leak (LL) rather than a zoonotic spillover (Zoo), given the evidence E we’ve seen?”. Symbolically, we’re asking ...