trentmkelly/LessWrong-43k huggingface dataset:
Practicing my Handwriting in 1439
by Maxwell Tabarrok — 2024-02-03T13:21:37.331Z
Link: https://www.maximum-progress.com/p/practicing-my-handwriting-in-1439

Tyler Cowen’s

latest Bloomberg column

on the dangers of AI predicts, among other things, that AI will disrupt the current high status and earnings of “wordcels” and “ideas people.”

The larger theme is becoming evident: AI will radically disrupt power relations in society.

AI may severely limit, for instance, the status and earnings of the so-called “

wordcel

” class. It will displace many jobs that deal with words and symbols, or make them less lucrative, or just make those who hold them less influential. Knowing how to write well won’t be as valuable a skill five years from now, because AI can improve the quality of just about any text. Being bilingual (or tri- or quadrilingual, for that matter) will also be less useful, and that too has been a marker of highly educated status. Even if AIs can’t write better books than human authors, readers may prefer to spend their time talking to AIs rather than reading.

It is worth pausing to note how profound and unprecedented this development would be. For centuries, the Western world has awarded higher status to what I will call ideas people — those who are good at developing, expressing and putting into practice new ways of thinking. The

Scientific

and

Industrial

revolutions greatly increased the reach and influence of ideas people.

AI may put that trend into reverse. It is not hard to imagine a world, less than 20 years from now, in which a skilled carpenter is seen to have better prospects — professional and otherwise — than an articulate lawyer.

As a wordcel myself, I find this prediction concerning. The “

moving history

” of the scientific and industrial revolutions is worrying enough, but as Tyler said, at least they increased the returns to ideas people in the long run. Collapsing institutions, international arms races, and a permanent decrease in the rewards for my skills just sounds terrible. I am not a good carpenter!

But will this prediction really come true?

On one hand, almost all of AI’s realized impact so far and the closest technological analogies to AI point to it being a complement, rather than a substitute to “those who are good at developing, expressing and putting into practice new ways of thinking.”

We are still in the very early days of AI development. Still, the current uses of nascent AI tech is evidence for what the future will bring. So far, AI has been complementary to knowledge work. It makes

softwar

...
Finite Factored Sets to Bayes Nets Part 2
by J Bostock — 2024-02-03T12:25:41.444Z

This post assumes knowledge of category theory, finite factored sets, and Bayes nets.

The Setup

I've already talked about DAGs and factor overlap Venn diagrams in a previous post, where I studied them within a category-theoretic framework. Here I'll also perform an explicit construction of them using set theory.

DAGs

I have already discussed the set of directed acyclic graphs over n elements. We will denote the set of all DAGs of n elements as DAG(n). Each Bayes net can be thought of as a set of pairs of elements {(i∈{1,...,n},j≠i∈{1,...,n}),...}.

This set can be converted into the category of Bayes nets over n elements by the addition of morphisms corresponding to "bookkeeping"-type relationships, which we will denoteDAGn. From this category, we can form a category whose elements are sets of the elements of DAGn, subject to the following condition on a set S:

(I'll slightly go against standard notation by using calligraphic acronyms for my category names i.e. DAGn. I don't feel like any of my categories are definitely natural or useful enough to earn a "proper" bold name like DAGn)

∀s∈S,b∈DAG(n), s→b⟹b∈S

This means that, if we can reach a given Bayes net b from any element s of our set S, that Bayes net b must also be in S. I refer to these as compatible sets and denote the category CSBn (standing for compatible sets of Bayes nets) as the category whose elements are compatible sets of n elements and for any two elements. To write it out fully:

Ob(CSBn)={S∈P(DAG(n))∣∣∀s∈S,d∈DAG(n),DAGn(s,d)={→}⟹d∈S}

CSBn(A,B)={{→} A⊇B {} A⊂B

This should be read as "There is a unique morphism from A to B if and only if A is a weak superset of B, otherwise there is no morphism". Orderings of this form always follow the rules required to create a category.

(Aside: sometimes we think about equivalence classes of Bayes nets. If we choose, we can first convert our Bayes nets to equivalence classes, then convert them to compatible sets, but this is not needed here)

This category has an initial object Disn which is a set consisting of the discrete Bayes net (with no arrows at all) and therefore all Bayes nets. Less trivially it has a terminal object Indn which contains exactly the Bayes nets which ...
Why I no longer identify as transhumanist
by Kaj_Sotala — 2024-02-03T12:00:04.389Z

Someone asked me how come I used to have a strong identity as a singularitarian / transhumanist but don’t have it anymore. Here’s what I answered them:

——-

So I think the short version is something like: transhumanism/singularitarianism used to give me hope about things I felt strongly about. Molecular nanotechnology would bring material abundance, radical life extension would cure aging, AI would solve the rest of our problems. Over time, it started feeling like 1) not much was actually happening with regard to those things, and 2) to the extent that it was, I couldn’t contribute much to them and 3) trying to work on those directly was bad for me, and also 4) I ended up caring less about some of those issues for other reasons and 5) I had other big problems in my life.

So an identity as a transhumanist/singularitarian stopped being a useful emotional strategy for me and then I lost interest in it.

With regard to 4), a big motivator for me used to be some kind of fear of death. But then I thought about philosophy of personal identity until I shifted to the view that there’s probably no persisting identity over time anyway and in some sense I probably die and get reborn all the time in any case.

Here’s something that I wrote back in 2009 that was talking about 1):

—

The [first phase of the Excitement-Disillusionment-Reorientation cycle of online transhumanism] is when you first stumble across concepts such as transhumanism, radical life extension, and superintelligent AI. This is when you subscribe to transhumanist mailing lists, join your local WTA/H+ chapter, and start trying to spread the word to everybody you know. You’ll probably spend hundreds of hours reading different kinds of transhumanist materials. This phase typically lasts for several years.

In the disillusionment phase, you start to realize that while you still agree with the fundamental transhumanist philosophy, most of what you are doing is rather pointless. You can have all the discussions you want, but by themselves, those discussions aren’t going to bring all those amazing technologies here. You learn to ignore the “but an upload of you is just a copy” debate when it shows up the twentieth time, with the same names rehearsing the same arguments and thought experiments for the fifteenth time. Having gotten over your initial future shock, you may start to wonder why having a specific name like trans...
Attention SAEs Scale to GPT-2 Small
by Connor Kissane — 2024-02-03T06:50:22.583Z

This is an interim report that we are currently building on. We hope this update + open sourcing our SAEs will be useful to related research occurring in parallel. Produced as part of the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort

Executive Summary

In a previous post, we showed that

sparse autoencoders (SAEs) work on the attention layer outputs

of a two layer transformer. We scale our attention SAEs to GPT-2 Small, and continue to find sparse interpretable features in every layer. This makes us optimistic about our ongoing efforts scaling further, especially since we didn’t have to do much iterating
We open source our SAEs. Load them from

Hugging Face

or this

colab notebook

The SAEs seem good, often recovering more than 80% of the loss relative to zero ablation, and are sparse with less than 20 features firing on average. The majority of the live features are interpretable

We continue to find the same three feature families that we found in the

two layer model

:

induction features

,

local context features

, and

high level context features

. This suggests that some of our lessons interpreting features in smaller models may generalize
We also find new, interesting feature families that we didn’t find in the two layer model, providing hints about fundamentally different capabilities in GPT-2 Small

See our

feature interface

to browse the first 30 features for each layer
New: use Neuronpedia to visualize these SAEs

Introduction

In

Sparse Autoencoders Work on Attention Layer Outputs

we showed that we can apply SAEs to extract sparse interpretable features from the last attention layer of a two layer transformer. We have since applied the same technique to a 12-layer model, GPT-2 Small, and continue to find sparse, interpretable features in every layer. Our SAEs often recover more than 80% of the loss[1], and are sparse with less than 20 features firing on average. We perform shallow investigations of the first 30 features from each layer, and we find that the majority (often 80%+) of non-dead SAEs features are interpretable. See the features with our

interactive visualizations

for each layer.

We open source our SAEs in hope that they will be useful to other researchers currently working on dictionary learning. We are particularly excited about using these SAEs to better understand attention circuits a...
Why do we need RLHF? Imitation,  Inverse RL, and the role of reward
by Ran W — 2024-02-03T04:00:27.069Z

This is a cross-post from my Substack to get some feedback from the alignment community. I have condensed some sections to get the message across.

It's rarely questioned in the mainstream why we need RLHF in the first place? If we can do supervised fine-tuning (SFT) sufficiently well, then can we just ditch reward modeling and RL at all? This seems to be a question that many people silently have (e.g., in this tweet).

To better understand the goal of RLHF, I reflected on the related paradigm of inverse RL (IRL), which learns a reward function from human demonstrations rather than comparisons, and I took a deep dive into the roles of reward and RL in both frameworks. As a preview, both IRL and RLHF can alleviate exposure bias, a mismatch between training and testing conditions that potentially underlies degenerate behavior in SFT models. However, the advantage of RLHF is the possibility of outperforming the dataset. Nevertheless, the effect of the latter is unclear (perhaps dubious?) in practice.

A brief recap of IRL

Before RLHF established the binary comparison approach to learning from human feedback, the basic approach was just to imitate human behavior (a.k.a., imitation learning). To ensure we learn good behavior, the humans who demonstrate the behavior must be experts at the desired task. Given a dataset of expert behavior D={(s,a)}, imitation learning searches for a policy ^π which maximizes the likelihood of the dataset:
max^πE(s,a)∼D[log^π(a|s)]
In the setting of language modeling, if we interpret the state s as past tokens (a.k.a., context) and action a as the next token, the imitation learning objective is equivalent to the maximum likelihood objective used for training autoregressive language models.

Inverse RL can be seen as a special class of imitation learning algorithm which makes an additional assumption on the structure of the learner policy ^π that it is an optimal policy with respect to a pair of reward function R(s,a) and environment dynamics P(s′|s,a), where the latter is often assumed to be given. IRL instead searches for a reward function under which the optimal policy maximizes the likelihood of the dataset:
maxRE(s,a)∼D[log^π(a|s;R)]s.t.^π=argmaxπEπ,P[∞∑t=0γtR(st,at)]

SFT headache: Exposure bias and neural text degeneration

It is by now well documented that language models train...
Announcing the London Initiative for Safe AI (LISA)
by James Fox — 2024-02-02T23:17:47.011Z

The LISA Team consists of James Fox, Mike Brozowski, Joe Murray, Nina Wolff-Ingham, Ryan Kidd, and Christian Smith.

LISA’s Advisory Board consists of Henry Sleight, Jessica Rumbelow, Marius Hobbhahn, Jamie Bernardi, and Callum McDougall.

Everyone has contributed significantly to the founding of LISA, believes in its mission & vision, and assisted with writing this post.

TL;DR: The London Initiative for Safe AI (LISA) is a new AI Safety research centre. Our mission is to improve the safety of advanced AI systems by supporting and empowering individual researchers and small organisations. We opened in September 2023, and our office space currently hosts several research organisations and upskilling programmes, including

Apollo Research

,

Leap Labs

,

MATS

extension

,

ARENA

, and

BlueDot Impact

, as well as many individual and externally affiliated researchers.

LISA is open to different types of membership applications from other AI safety researchers and organisations.

(Affiliate) members can access talks by high-profile researchers, workshops, and other events. Past speakers have included Stuart Russell (UC Berkeley, CHAI), Tom Everitt & Neel Nanda (Google DeepMind), and Adam Gleave (FAR AI), amongst others.
Amenities for LISA Residents include 24/7 access to private & open-plan desks (with monitors, etc), catering (including lunches, dinners, snacks & drinks), and meeting rooms & phone booths. We also provide immigration, accommodation, and operational support; fiscal sponsorship & employer of record (upcoming); and regular socials & well-being benefits.

Although we host a limited number of short-term visitors for free, we charge long-term residents to cover our costs at varying rates depending on their circumstances. Nevertheless, we never want financial constraints to be a barrier to leading AI safety research, so please still get in touch if you would like to work from LISA’s offices but aren't able to pay.

If you or your organisation are interested in working from LISA, please

apply here

If you would like to support our mission, please get in contact.

Read on for further details about LISA’s vision and theory of change. After a short introduction, we motivate our vision by arguing why there is an urgency for LISA. Next, we summarise our track record and unpack our plans for the fut...
Survey for alignment researchers!
by Cameron Berg — 2024-02-02T20:41:44.323Z

UPDATE 3/9: Thanks to broad participation from the community, this and associated surveys have raised approximately $10,000 for high-impact alignment organizations. Given the reasonable sample size we now have, we are now going to pause donations for any subsequent responses. (However, we will preserve the charity voting question, and if anyone wants to sponsor donations for any surveys taken after March 9th, please ping us at alignment@ae.studio.)

AE

Studio

is launching a

short, anonymous survey

for alignment researchers, in order to develop a stronger model of various field-level dynamics in alignment.

This appears to be an interestingly neglected research direction that we believe will yield specific and actionable insights related to the community’s technical views and more general characteristics.

The survey is a straightforward 10-15 minute Google Form with some simple multiple choice questions.

For every alignment researcher who completes the survey, we will donate $40 to a high-impact AI safety organization of your choosing (see specific options on the survey). We will also send each alignment researcher who wants one a customized report that compares their personal results to those of the field.

Together, we hope to not only raise some money for some great AI safety organizations, but also develop a better field-level model of the ideas and people that comprise alignment research.

We will open-source all data and analyses when we publish the results. Thanks in advance for participating and for sharing this around with other alignment researchers!

Survey full link:

https://forms.gle/d2fJhWfierRYvzam8
Voting Results for the 2022 Review
by Ben Pace — 2024-02-02T20:34:59.768Z

The 5th Annual LessWrong Review has come to a close!

Review Facts

There were 5330 posts published in 2022.

Here's how many posts passed through the different review phases.

Phase	No. of posts	Eligibility
Nominations Phase	579	Any 2022 post could be given preliminary votes
Review Phase	363	Posts with 2+ votes could be reviewed
Voting Phase	168	Posts with 1+ reviews could be voted on

Here how many votes and voters there were by karma bracket.

Karma Bucket	No. of Voters	No. of Votes Cast
Any	333	5007
1+	307	4944
10+	298	4902
100+	245	4538
1,000+	121	2801
10,000+	24	816

To give some context on this annual tradition, here are the absolute numbers compared to last year and to the first year of the LessWrong Review.

2018	2021	2022
Voters	59	238	333
Nominations	75	452	579
Reviews	120	209	227
Votes	1272	2870	5007
Total LW Posts	1703	4506	5330

Review Prizes

There were lots of great reviews this year! Here's a link to all of them

Of 227 reviews we're giving 31 of them prizes.

This follows up on Habryka who gave out about half of these prizes 2 months ago.

Note that two users were paid to produce reviews and so will not be receiving the prize money. They're still here because I wanted to indicate that they wrote some really great reviews.

Click below to expand and see who won prizes.

Excellent ($200) (7 reviews)

ambigram for their review of Meadow Theory
Buck for his self-review of Causal Scrubbing: a method for rigorously testing interpretability hypotheses
DirectedEvolution for their

paid review

of How satisfied should you expect to be with your partner?
LawrenceC for their

paid review

of Some Lessons Learned from Studying Indirect Object Identification in GPT-2 small
LawrenceC for their

paid review

of How "Discovering Latent Knowledge in Language Models Without Supervision" Fits Into a Broader Alignment Scheme
LoganStrohl for their self-review of the Intro to Naturalism sequence
porby for their self-review of Why I think strong general AI is coming soon

Great ($100) (6 reviews)

DirectedEvolution for their

paid review

of Slack matters more than any outcome
janus for their self-review of Simulators
Lee Sharkey for their self-review of Taking features out of superposition with sparse autoencoders
Neel Nanda for their review

...
On Dwarkesh’s 3rd Podcast With Tyler Cowen
by Zvi — 2024-02-02T19:30:05.974Z

This post is extensive thoughts on Tyler Cowen’s excellent talk with Dwarkesh Patel.

It is interesting throughout. You can read this while listening, after listening or instead of listening, and is written to be compatible with all three options. The notes are in order in terms of what they are reacting to, and are mostly written as I listened.

I see this as having been a few distinct intertwined conversations. Tyler Cowen knows more about more different things than perhaps anyone else, so that makes sense. Dwarkesh chose excellent questions throughout, displaying an excellent sense of when to follow up and how, and when to pivot.

The first conversation is about Tyler’s book GOAT about the world’s greatest economists. Fascinating stuff, this made me more likely to read and review GOAT in the future if I ever find the time. I mostly agreed with Tyler’s takes here, to the extent I am in position to know, as I have not read that much in the way of what these men wrote, and at this point even though I very much loved it at the time (don’t skip the digression on silver, even, I remember it being great) The Wealth of Nations is now largely a blur to me.

There were also questions about the world and philosophy in general but not about AI, that I would mostly put in this first category. As usual, I have lots of thoughts.

The second conversation is about expectations given what I typically call mundane AI. What would the future look like, if AI progress stalls out without advancing too much? We cannot rule such worlds out and I put substantial probability on them, so it is an important and fascinating question.

If you accept the premise of AI remaining within the human capability range in some broad sense, where it brings great productivity improvements and rewards those who use it well but remains foundationally a tool and everything seems basically normal, essentially the AI-Fizzle world, then we have disagreements but Tyler is an excellent thinker about these scenarios. Broadly our expectations are not so different here.

That brings us to the third conversation, about the possibility of existential risk or the development of more intelligent and capable AI that would have greater affordances. For a while now, Tyler has asserted that such greater intelligence likely does not much matter, that not so much would change, that transformational effects are highly u...
Most experts believe COVID-19 was probably not a lab leak
by DanielFilan — 2024-02-02T19:28:00.319Z
Link: https://gcrinstitute.org/covid-origin/

The Global Catastrophic Risks Institute conducted an anonymous survey of relevant experts on whether they thought COVID was more likely caused by a lab accident (aka lab leak) or zoonotic spillover. Their summary, bolding is mine:

The study’s experts overall stated that the COVID-19 pandemic most likely originated via a natural zoonotic event, defined as an event in which a non-human animal infected a human, and in which the infection did not occur in the course of any form of virological or biomedical research. The experts generally gave a lower probability for origin via a research-related accident, but most experts indicated some chance of origin via accident and about one fifth of the experts stated that an accident was the more likely origin. These beliefs were similar across experts from different geographic and academic backgrounds.

The experts mostly expressed the view that more research on COVID-19’s origin could be of value. About half of the experts stated that major gaps still remain in the understanding COVID-19’s origin, and most of the other experts also stated that some research is still needed. About 40% of experts stated that clarity on COVID-19 origins would provide a better understanding of the potential origins of future pandemics. Given clarity on COVID-19’s origin, experts also proposed a variety of governance changes for addressing future pandemics, including measures to prevent initial human infection, measures to prevent initial infection from becoming pandemic, and measures to mitigate the harm once the pandemic occurs.

The vast majority of the experts express the belief that a natural zoonotic event will likely be the origin of the next pandemic.

The experts also provided a set of clear recommendations for preventing, preparing for and responding to future pandemics, which generally align with many previous studies.

Link to the main report is here, and link to their (much longer) methodological and analytical annex is here.

[EDIT: I currently think there are enough problems with the survey that they should be mentioned alongside the results.

Firstly, the sample seems to have been based on personalized outreach, rather than mass emails. This runs the risk of selection bias. [EDIT 2: I'm told that 'personalized outreach' looks more like "sending individual emails to everyone on a big list" than "the authors emailing their friends".] Al...