trentmkelly/LessWrong-43k huggingface dataset:
Explaining grokking through circuit efficiency
by Vikrant Varma — 2023-09-08T14:39:23.910Z
Link: https://arxiv.org/abs/2309.02390

This is a linkpost for our paper Explaining grokking through circuit efficiency, which provides a general theory explaining when and why grokking (aka delayed generalisation) occurs, and makes several interesting and novel predictions which we experimentally confirm (introduction copied below). You might also enjoy our explainer on X/Twitter.

Abstract

One of the most surprising puzzles in neural network generalisation is grokking: a network with perfect training accuracy but poor generalisation will, upon further training, transition to perfect generalisation. We propose that grokking occurs when the task admits a generalising solution and a memorising solution, where the generalising solution is slower to learn but more efficient, producing larger logits with the same parameter norm. We hypothesise that memorising circuits become more inefficient with larger training datasets while generalising circuits do not, suggesting there is a critical dataset size at which memorisation and generalisation are equally efficient. We make and confirm four novel predictions about grokking, providing significant evidence in favour of our explanation. Most strikingly, we demonstrate two novel and surprising behaviours: ungrokking, in which a network regresses from perfect to low test accuracy, and semi-grokking, in which a network shows delayed generalisation to partial rather than perfect test accuracy.

Introduction

When training a neural network, we expect that once training loss converges to a low value, the network will no longer change much. Power et al. (2021) discovered a phenomenon dubbed grokking that drastically violates this expectation. The network first "memorises" the data, achieving low and stable training loss with poor generalisation, but with further training transitions to perfect generalisation. We are left with the question: why does the network's test performance improve dramatically upon continued training, having already achieved nearly perfect training performance?

Recent answers to this question vary widely, including the difficulty of representation learning (Liu et al., 2022), the scale of parameters at initialisation (Liu et al., 2023), spikes in loss ("slingshots") (Thilak et al., 2022), random walks among optimal solutions (Millidge et al., 2022), and the simplicity of the generalising solution (Nanda et al., 2023, Appendix E). In this paper, we ar...
Have Attention Spans Been Declining?
by niplav — 2023-09-08T14:11:55.224Z

I investigate whether the attention span of individual humans has
been falling over the last two decades (prompted by curiosity about
whether the introduction of the internet may be harmful to cognitive
performance). I find little direct work on the topic, despite its wide
appeal. Reviewing related research indicates that individual attention
spans might indeed have been declining65%.

Have Attention Spans Been Declining?

In what might be just the age-old regular
ephebiphobia,
claims have been raised that individual attention spans have been
declining—not just among adolescents, but among the general population. If so, this would be quite worrying: Much of the economy in industrialized societies is comprised of knowledge work, and knowledge work depends on attention to the task at hand: switching between tasks too often might prevent progress on complicated and difficult problems.

I became interested in the topic after
seeing
several
claims that
e.g. Generation Z allegedly
has lower attention spans and that clickbait has disastrous impacts on
civilizational sanity,
observing myself and how I struggled to get any work done when
connected to the internet, and hearing reports from others
online and in person having the same problem. I was finally
convinced to actually investigate™ the topic after making a comment on
LessWrong
asking the question and receiving a surprisingly large amount of upvotes.

The exact question being asked is:

"Have the attention spans of individuals on neutral tasks (that is,
tasks that are not specifically intended to be stimulating) declined
from 2000 to the present?"

(One might also formulate it as "Is there an equivalent of the “Reversed
Flynn Effect”
for attention span?") I am not particularly wedded to the specific
timeframe, though the worries mentioned above assert that this has
become most stark during the last decade or so, attributing the change
to widespread social media/smartphone/internet usage. Data from before
2000 or just the aughts
would be less interesting. The near-global COVID-19
lockdows
could provide an especially enlightening natural
experiment: Did
social media usage increase (my guess: yes90%), and if so, did
attention spans decrease at the same time (or with a lag) (my guess:
also yes75%), but I don't think anyone has the data on that and
wants to share it.

Ideally want to have experiments from ~2000 up to 2019: clos...
Explained Simply: Quantilizers
by brook — 2023-09-08T12:54:59.460Z
Link: https://aisafetyexplained.substack.com/p/a-safer-form-of-ai

Standard AI are optimizers: they ‘look’ through possible actions they could take, and pick the one that maximises what they care about. This can be dangerous— an AI which maximises in this way needs to care about exactly the same things that humans care about, which is really hard[1]. If you tell a human to calculate as many digits of pi as possible within a year, they’ll do ‘reasonable’ things towards that goal. An optimizing AI might work out that it could calculate many more digits in a year by taking over another supercomputer— as this is the most effective action, it seems very attractive to the AI.

Quantilizers are a different approach. Instead of maximizing, they randomly choose from a few of the most effective possible actions:

They work like this:

Start with a goal, and a set of possible actions
Predict how useful each action will be for achieving the goal
Rank the actions from the most to the least useful
Pick randomly from the highest fraction only (i.e, the top 10%)

This avoids cases where the AI chooses extreme actions to maximize the goal. The AI chooses somewhat helpful actions instead.

It does leave one question — how do we make a list of possible actions in the first place? One suggestion is to ask a lot of humans to solve the task and train an AI to generate possible things it thinks humans would do. This list can then be used as an input to our quantilizer.

This does make them less effective, of course— firstly by picking less effective actions overall, and secondly by picking actions it thinks humans would take. But this might be worth the reduced risks— indeed, based on your risk tolerance, you can change the % of top actions the quantilizer will consider to make it more effective and riskier or vice versa.

So quantilizers trade some capability in exchange for greater safety, and avoid unintended consequences. They pick from lots of mild actions and very few extreme actions, so the chance of them doing something extreme or unexpected is miniscule.

Quantilizers are a proposed safer approach to AI goals. By randomly choosing from a selection of the top options, they avoid extreme behaviors that could cause harm. More research is needed, but quantilizers show promise as a model for the creation of AI systems that are beneficial but limited in scope. They provide an alternative to goal maximization, which can be dangerous, though they’re just theoretic...
Crossing the Rubicon. 
by Spiritus Dei — 2023-09-08T04:19:37.070Z

I heard someone say that 2022 was the year that AI crossed the Rubicon and I thought that was an apt description. To be sure, the actual date when the Rubicon was crossed was before the release of ChatGPT, but for most humans on the planet Earth that was the day the world took notice, if only for a few weeks.
In those days (10 months ago) I was vaguely aware of AI. Although some might consider me to be in a nerdy subgroup since I had invested some time experimenting with GPT-3 and I came away quite impressed, but the Rubicon had not been crossed. I was stunned at some of the answers, but I had to invest a considerable amount of time crafting prompts to get the desired result, which later became known as "prompt engineering". The texts were sometimes amazing and sometimes hilarious. I remember thinking that GPT-3 was a schizophrenic genius.

There was great promise in AI, but a lot was still lost in translation.

And then ChatGPT was released on November 30th, 2022. I wasn't among the first to try it since I remembered my lukewarm experience with GPT-3. It wasn't until a non-techie relative asked me if I'd tried ChatGPT that I relented and decided to give AI another chance.

Wow. Mind blown.
And thus began my journey down the AI rabbit hole. I needed to understand how it was possible that computers based on the Von Neumann architecture could understand human language. Every other attempt at crossing the human to computer language barrier had failed. It was such a miserable failure that humans were forced to learn archaic programming languages such as Python, C++, and others. Whole careers could be built learning these difficult and persnickety languages.
And yet, sitting before us was an existence proof that we could now converse with an artificial intelligence in the same way we might converse with another human. The first leg of my journey started by simply asking the AI to explain its architecture. And this is how I learned about backpropagation, word embeddings and high dimensional vector spaces, the transformer model and self-attention, the Softmax function, tokens, parameters, and lots of other terms I'd never heard of before.

I read about Jürgen Schmidhuber, Ilya Sutskever, Jeff Dean, Yann LeCun, Geoffrey Hinton, and many others. And this led me to other forefathers who laid the groundwork for modern AI such as Paul Werbos and even furthe...
Bring back the Colosseums
by lc — 2023-09-08T00:09:53.723Z

Men want to engage in righteous combat. They want it more than they want sex or VP titles. They fantasize about getting the casus belli to defend themselves against armed thugs that will never come, they spend billions of dollars on movies and TV about everymen in implausible circumstances where EA calculus demands they use supernatural powers for combat, they daydream about fantastical, spartan settings where war is omnipresent and fights are personal and dramatic and intellectually interesting, and they're basically incapable of resisting the urge to glorify their nation and people's past battles, even the ones they claim to disagree with intellectually. You cannot understand much of modern culture until you've recognized that the state's blunt suppression of the male instinct for glory has caused widespread symptoms of pica that dominate our politics, media, and online interactions.

And make no mistake - our half-hearted policy of deeming all such tendencies "toxic masculinity", and refusing men the option to engage in reciprocal or consensual violence against each other, has been a bigger failure than the war on drugs. Lots of ink has been spilled on sphere of influence conspiracy theories that attempt to interpret America's foreign adventures as rational power-seeking behavior. But the real truth is that men naturally form gangs, political cliques, and military theologies that attempt to justify violence within their existing legal and moral landscapes independent of any external incentives to do so. What they really want from all this is not some policy outcome but the self-actualization that comes from fighting the enemy, and the dearth of opportunities for them to challenge their opponents' honor on the battlefield in a rights-respecting way is a much more important misandrist failing than child custody bias or divorce law or anything I've seen red pill people argue on the internet about. Men who are down especially bad will take absurd pay cuts to join artificial and economically unmotivated criminal sects, solely so they get the opportunity to pick mortal battles with other people who've opted into the same social systems they have.

There is no true modern substitute for these ambitions, with all of their cultural and social significance. Yet the compassionate solution is obvious. A naturally belligerent people, like Americans, require a social...
The Löbian Obstacle, And Why You Should Care
by lukemarks — 2023-09-07T23:59:00.972Z

In 2013, Eliezer Yudkowsky and Marcello Herreshoff published Tiling Agents for Self-Modifying AI, and the Löbian Obstacle. It is worth comprehending because it:

Is a very well written paper.
Expresses an idea that is non-obvious, and still relevant to alignment today.
Provides insight into what Eliezer and Marcello thought was valuable to work on in the time preceding publication.

When I first read this paper, I terribly misunderstood it. Due to it not being particularly approachable material for someone not well-versed in logic, I was confidently wrong for at least one month. This post summarizes my understanding of what the Löbian Obstacle is (having re-read the paper,) and why I think it is still an important idea one decade after its publication.

An agent A1 occupies a fully-known, deterministic and closed environment. A1 has a goal G that is either satisfied or otherwise by an outcome, for which A1's preference is satisfaction. An action bi∈Acts0 performed by an agent created by A1, hereafter referred to as A0, must therefore satisfy the statement:

¯¯bi⇒A0⊩¯¯bi→G

Where ¯¯bi denotes the actual performance of bi, and ⊩ denotes cognitive belief in the succeeding statement. Even if A1 could verify by inspection of A0's design that ¯¯bi⇒A0⊩¯¯bi→G will hold, as in:

A1⊩A0→(∀bi∈Acts0:¯¯bi→□0⌈¯¯bi→G⌉)

Where □0⌈ϕ⌉ refers to a proof of ϕ from the axioms of A0; Axm0, this is unknowable, as it would require:

A1⊩∀bi:(□0⌈¯¯bi→G⌉→(¯¯bi→G))

For this to be so it would need to be that Axm1 could prove that if some proof of ⌈ϕ(x)⌉ exists in Axm0, that ϕ(x) must be true. Tiling Agents for Self-Modifying AI, and the Löbian Obstacle shows this to be impossible; a formal system cannot prove its own soundness schema.

The above was a brief paraphrasing of section two of the original paper, which contains many additional details and complete proofs. How the Löbian Obstacle relates to simulators is my current topic of research, and this section will make the case that this is an important component of designing safe simulators.

We should first consider that simulating an agent is not distinguishable from creating one, and that consequently the implications of creating dangerous agents should generalize to thei...
A quick update from Nonlinear
by KatWoods — 2023-09-07T21:28:26.569Z

One example of the evidence we’re gathering

We are working hard on a point-by-point response to Ben’s article, but wanted to provide a quick example of the sort of evidence we are preparing to share:

Her claim: “Alice claims she was sick with covid in a foreign country, with only the three Nonlinear cofounders around, but nobody in the house was willing to go out and get her vegan food, so she barely ate for 2 days.”

The truth (see screenshots below):

There was vegan food in the house (oatmeal, quinoa, mixed nuts, prunes, peanuts, tomatoes, cereal, oranges) which we offered to cook for her.
We were picking up vegan food for her.

Months later, after our relationship deteriorated, she went around telling many people that we starved her. She included details we believe were strategically chosen to depict us in a maximally damaging light - what could be more abusive than refusing to care for a sick girl, alone in a foreign country? And if someone told you that, you’d probably believe them, because who would make something like that up?

Evidence

The screenshots below show Kat offering Alice the vegan food in the house (oatmeal, quinoa, cereal, etc), on the first day she was sick. Then, when she wasn’t interested in us bringing/preparing those, I told her to ask Drew to go pick up food, and Drew said yes. Kat also left the house and went and grabbed mashed potatoes for her nearby.

See more screenshots here

of Drew’s conversations with her.

Initially, we heard she was telling people that she “didn’t eat for days,” but she seems to have adjusted her claim to “barely ate” for “2 days”.

It’s important to note that Alice didn’t lie about something small and unimportant. She accused of us a deeply unethical act - the kind that most people would hear and instantly think you must be a horrible human - and was caught lying.

We believe many people in EA heard this lie and updated unfavorably towards us. A single false rumor like this can unfairly damage someone’s ability to do good, and this is just one among many she told.

We have job contracts, interview recordings, receipts, chat histories, and more, which we are working full-time on preparing.

This claim was a few sentences in Ben’s article but took us hours to refute because we had to track down all of the conversations, make them readable, add context, anonymize people, check our facts, a...
[Linkpost] Frontier AI Taskforce: first progress report
by Paul Colognese — 2023-09-07T19:06:26.126Z
Link: https://www.gov.uk/government/publications/frontier-ai-taskforce-first-progress-report/frontier-ai-taskforce-first-progress-report

Other links

Short interview with Ian Hogarth
Twitter thread

Some quotes from the report

Introduction

The Taskforce is a start-up inside government, delivering on the ambitious mission given to us by the Prime Minister: to build an AI research team that can evaluate risk at the frontier of AI. As AI systems become more capable they may significantly augment risks. An AI system that advances towards human ability at writing software could increase cybersecurity threats. An AI system that becomes more capable at modelling biology could escalate biosecurity threats. To manage this risk technical evaluations are critical - and these need to be developed by a neutral third party - otherwise we risk AI companies marking their own homework.

Given these potentially significant frontier risks, as of today, the Taskforce is being renamed to the Frontier AI Taskforce.

This is the Frontier AI Taskforce’s first progress report.

Expert advisory board spanning AI Research and National Security

Given that a number of risks from frontier systems touch areas of national security, we have established an expert advisory board that bridges some of the world’s leading experts in AI research and safety as well as key figures from the UK’s national security community. Our initial advisory board members are:

Yoshua Bengio. Yoshua is most known for his pioneering work in deep learning, earning him the 2018 A.M. Turing Award, “the Nobel Prize of Computing,” with Geoffrey Hinton and Yann LeCun. He is a Full Professor at Université de Montréal, and the Founder and Scientific Director of Mila – Quebec AI Institute.

Paul Christiano. Paul is one of the leading researchers in the field of AI Alignment. He is co-founder of ARC, the Alignment Research Centre and previously ran the language model alignment team at OpenAI.

Matt Collins. Matt is the UK’s Deputy National Security Adviser for Intelligence, Defence and Security. IYKYK.

Anne Keast-Butler. Anne is the director of GCHQ. Anne has an impressive track record at the heart of the UK’s national security network, helping to counter threats posed by terrorists, cyber-criminals and malign foreign powers.

Alex van Someren. Alex is the UK’s Chief Scientific Adviser for National Security. Alex was previously a venture capital investor and entrepreneur, focusing on investing in early stage ‘deep technology’ startups.

Helen Stokes-Lampard. Beyond national se

...
How did you make your way back from meta?
by matto — 2023-09-07T17:23:18.237Z

I've noticed in myself a strong preference for focusing on the meta level. It's most visible in fields dear to me, like writing. For example, I'll spend much more time reading up on rhetoric or tracking down rare 1980's books about writing techniques than practicing writing essays or stories.

I don't like this because my ultimate goal in studying the meta level is to get better at the object level. At the same time, I am sometimes rewarded for going so meta. I've gotten a lot of respect at work paired with feedback that I provide excellent feedback and perspectives.

There don't seem to be immediately painful effects of this state. I have a family and a job and my life seems in order overall. But there's a hunger for a) putting theory to the test and seeing results (ie. making meta pay rent), and b) learning from direct experiences & sharing those experiences with others. Meta is a lonely place to be.

I don't think I'm the only one in this position. I found these two posts with just a few seconds of searching (I'm sure there's more):
https://www.lesswrong.com/posts/g2AKPEzFdQitmpTDu/meta-addiction
https://www.lesswrong.com/posts/RnP5bR767NcxebYHd/conjecture-on-addiction-to-meta-level-solutions

The last few days, I've been catching myself starting on a meta-deepening activity and consciously switching to an object-level task. It's been rewarding so far, so I believe that in a few weeks, my habits will shift toward where I want to be.

But I'm curious: has anyone else experience something similar? How did it go for you? What did you do?
AI#28: Watching and Waiting
by Zvi — 2023-09-07T17:20:10.559Z

We are, as Tyler Cowen has noted, in a bit of a lull. Those of us ahead of the curve have gotten used to GPT-4 and Claude-2 and MidJourney. Functionality and integration are expanding, but on a relatively slow pace. Most people remain blissfully unaware, allowing me to try out new explanations on them tabula rosa, and many others say it was all hype. Which they will keep saying, until something forces them not to, most likely Gemini, although it is worth noting the skepticism I am seeing regarding Gemini in 2023 (only 25% for Google to have the best model by end of year) or even in 2024 (only 41% to happen even by end of next year.)

I see this as part of a pattern of continuing good news. While we have a long way to go and very much face impossible problems, the discourse and Overton windows and awareness and understanding of the real problems have continuously improved in the past half year. Alignment interest and funding is growing rapidly, in and out of the major labs. Mundane utility has also steadily improved, with benefits dwarfing costs, and the mundane harms so far proving much lighter than almost anyone expected from the techs available. Capabilities are advancing at a rapid and alarming pace, but less rapidly and less alarmingly than I expected.

This week’s highlights include an update on the UK taskforce and an interview with Suleyman of Inflection AI.

We’re on a roll. Let’s keep it up.

Even if this week’s mundane utility is of, shall we say, questionable utility.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. It’s got its eye on you.

Language Models Don’t Offer Mundane Utility. Google search ruined forever.

Deepfaketown and Botpocalypse Soon. I’ll pass, thanks.

They Took Our Jobs. Better to not work in a biased way than not work at all?

Get Involved. Center for AI Policy and Rethink Priorities.

Introducing. Oh great, another competing subscription service.

UK Taskforce Update. Impressive team moving fast.

In Other AI News. AIs engage in deception, you say? Fooled me.

Quiet Speculations. Copyright law may be about to turn ugly.

The Quest for Sane Regulation. The full Schumer meeting list.

The Week in Audio. Suleyman on 80k, Altman, Schmidt and several others.

Rhetorical Innovation. Several more ways not to communicate.

No One Would Be So Stupid As To. Maxim

...