trentmkelly/LessWrong-43k huggingface dataset:
A Letter to the Editor of MIT Technology Review
by Jeffs — 2023-08-30T16:59:14.906Z

Letter to Editor - MIT Tech Review - Wed, 30-Aug-2023

I submitted the below in response to Large language models aren’t people. Let’s stop testing them as if they were.

_____

In "Large language models aren’t people," MIT Technology continues to serve as a counterweight to AI hype. The article lists several things AI / LLMs still cannot do, even some that preschool children can. While allowing that LLMs are capable of tricks and rote memorization, the article maintains they are still far away from human capability levels. Tests created for humans probably do not even apply.

Your current message is clear: This technology is over-hyped. AI is less human, less capable, and less scary than many people assert.

I have depended upon MIT Tech Review to help me understand technology trends for decades now.

One question: At what point should we start worrying about AI?

What tricks - if LLMs demonstrate them in the future - should concern us?

At one point, learning to code was considered one of those future tricks that should concern us. Winning at Go was going to take decades. Creative pursuits - writing and art - were going to be the last things AI could do. All these capabilities are now accepted as the new normal.

Your current chain of reasoning appears to be: 1) Currently, these models are not human and lack fundamental human capabilities. 2) We do not know what is going on inside of them. 3) Sure, as they get bigger, the list of tricks they can do gets longer. 4) Ignore the hype.

An alternative chain of thought would be:

1) Currently, these models are not human and lack fundamental human capabilities. 2) We do not know what is going on inside of them. 3) They are becoming more capable. 4) We are not good at predicting when/how the capabilities emerge. 5) Intelligence is a potent force. (Consider the plight of stronger animals at our mercy.) 6) Be concerned.

If MIT Tech Review misses this one, all its technology analysis and commentary over the decades will not matter.

__

Mon, 04-Mar-2024

I'd like to report an update to this post. I just finished reading Large language models can do jaw-dropping things. But nobody knows exactly why by William Douglas Heaven, the same author who wrote Large Language Models are not People. I find it a helpful layman's exposition of some of th...
Biosecurity Culture, Computer Security Culture
by jefftk — 2023-08-30T16:40:03.101Z

While I've only worked in biosecurity for

about a
year and my computer security background consists of things I
picked up while working on other aspects of software engineering, the
cultures seem incredibly different. Some examples of good computer
security culture that would be bad biosecurity culture:

Openness and full
disclosure. Write blog posts with deep detail on how
vulnerabilities were found, with the goal of teaching others how to
find similar ones in the future. Keep details quiet for a few months
if need be to give vendors time to fix but after, say, 90
days go public.

Breaking things to fix them. Given a new system, of course you
should try to compromise it. If you succeed manually, make a demo
that cracks it in milliseconds. Make (and publish!) fuzzers and other
automated vulnerability search tools.

Enthusiastic curiosity and exploration. Noticing hints
of vulnerabilities and digging into them to figure out how deep they
go is great. If someone says "you don't need to know that" ignore them
and try to figure it out for yourself.

This is not how computer security has always been, or how it is
everywhere, and people in the field are often fiercely protective of
these ideals against vendors that try to hide flaws or silence
researchers. And overall my impression is that this culture has been
tremendously positive in computer security.

Which means that if you come into the effective altruism corner of
biosecurity with a computer security background and see all of these
discussions of "information
hazards", people discouraging trying to find vulnerabilities, and
people staying quiet about dangerous things they've discovered it's
going to feel very strange, and potentially
rotten.

So here's a framing that might help see things from this biosecurity
perspective. Imagine that the Morris worm never
happened, nor Blaster,
nor Samy. A
few people independently discovered SQL injection
but kept it to themselves. Computer security never developed as a
field, even as more and more around us became automated. We have
driverless cars, robosurgeons, and simple automated agents acting for
us, all with the security of original Sendmail. And it's all been
around long enough that the original authors have moved on and no one
remembers how any of it works. Someone who put in some serious effort
could cause immense distruction, but th...
Why I hang out at LessWrong and why you should check-in there every now and then 
by Bill Benzon — 2023-08-30T15:20:44.439Z

As the title indicates, this post, which in cross-posting from my home blog, New Savanna, is not directed at a LessWrong audience. However,, some of you might find in useful in thinking about LessWrong's place in the world.

The first two sections are a background, first a bit on cultural change to set the general context. Then some general information about LessWrong. Now we’re reading for my personal impressions of the place, concluding with a suggestion that you take a look if you haven’t already.

Cultural change over the long haul: Christians and professors

Back in the ancient days Christianity as just another mystery cult on the periphery of the Roman Empire. Then in 380 AD Emperor Theodosius I issued the Edict of Thessalonica and it became the state religion of the Roman Empire. In time it spread out among the many tribes of Europe and those Christian tribespeople began thinking of an entity called Christendom, and that, in time, became Europe and “the West.”

Back in the days when Europe was still Christendom the Catholic Church was the center of intellectual life. That changed during the Sixteenth Century with the advent of the Scientific Revolution and the Reformation. The Catholic remained powerful, of course, but universities supplanted it as the institutional center of intellectual life.

My point is simple and obvious: things change. Cults can become mainstream and new institutions can displace old ones. With that in mind, let’s think about LessWrong.

LessWrong

To be sure do not want to imply that LessWrong, a large and sophisticated online community, and the currents that swirl there (the rationalist movement, effective altruism (EA), dystopian fears of rogue AI) is comparable to Christianity, but it appears cultlike to outsiders, and to some insiders as well. It hosts a great deal of high-wattage intellectual activity on artificial intelligence and AI existential risk, effective altruism, and, more generally, how to live a life. I suspect that for some who post there, it is the intellectual center of their intellectual life.

I was founded in 2009 by Eliezer Yudkowsky, an autodidact with strong interests in AI and, in particular, the destructive potential of advanced AI. That is what he’s best known for and, while I suspect that Nick Bostrom is more widely known on that subject – his 2014 book, Superintelligence, was a best-seller, and he has an academic ...
"Wanting" and "liking"
by Mateusz Bagiński — 2023-08-30T14:52:04.571Z

Written as a result of AI Safety Camp Virtual 2023. Thanks to the following people for feedback and helpful conversations: Oliver Bridge, Tim Gothard, Rasmus Jensen, Linda Linsefors.[1]

This post reviews the literature on "wanting" and "liking", two primary components of what is commonly referred to together as the biological reward system. It is intended to be informative for AI safety-related work, especially within approaches that try to leverage insights from neuroscience for alignment.

Section 1 introduces the distinction between two high-level components of reward: wanting and liking. At this point, I simplify the topic and treat both as homogenous categories. Sections 2 and 3 delve deeper into each component and give a more fine-grained model, including a description of their neurobiological substrates. (2.2. and 3.2 are more technical/dry neuroscience, so you may want to skip them, if that's not your primary interest.) Section 4 discusses the functional relationships between them and why this kind of "division of labor" may have been favored by evolution.

1. Introduction

I will start by introducing four concepts central to this post: wanting, "wanting", liking, and "liking".[2] They carve the space of human values[3] along two dimensions. The first of those is the distinction between the things we are motivated to do/driven towards (wanting) versus the things we feel good about happening (or being about to happen) (liking). The second distinction is between the more basic/less-sophisticated components ("wanting" and "liking") of each and the more elaborated and cognitive components (wanting and liking).[4] The two parts of this Section elaborate on these two dimensions.

1.1. Liking vs Wanting

Liking is when you take a bite of a tasty food and its taste brings you pleasure or, more generally, positively valenced affective states. Wanting is when you are motivated to act in order to obtain that food and bring it to your mouth in order to consume it.

This is probably enough to intuit the rough contours of the distinction, but at the same time, it may raise some questions. I can think about cases where I kind of want to get up and go to the kitchen, but I'm too tired (or my willpower is too depleted) to get up. So I don't get up, even though I know the food in the fridge is very good and if I did get up and get it, I would be very glad about doing so. On an...
Open Call for Research Assistants in Developmental Interpretability
by Jesse Hoogland — 2023-08-30T09:02:59.781Z

We are excited to announce multiple positions for Research Assistants to join our

six-month research project

assessing the viability of

Developmental Interpretability

(DevInterp).

This is a chance to gain expertise in interpretability, develop your skills as a researcher, build out a network of collaborators and mentors, publish in major conferences, and open a path towards future opportunities, including potential permanent roles, recommendations, and successive collaborations.

Background

Developmental interpretability

is a research agenda aiming to build tools for detecting, locating, and understanding phase transitions in learning dynamics of neural networks. It draws on techniques from singular learning theory, mechanistic interpretability, statistical physics, and developmental biology.

Position Details

General info:

Title: Research Assistant.
Location: Remote, with hubs in Melbourne and London.
Duration: Until March 2024 (at minimum).
Compensation: base salary is USD$35k per year, to be paid out as an independent contractor at an hourly rate.

Timeline:

Application Deadline: September 15th, 2023
Ideal Start Date: October 2023

How to Apply: Complete the

application form

by the deadline. Further information on the application process will be provided in the form.

Who We Are

The developmental interpretability research team consists of experts across a number of areas of mathematics, physics, statistics and AI safety. The principal researchers:

Daniel Murfet

, mathematician and SLT expert, University of Melbourne.

Susan Wei

, statistician and SLT expert, University of Melbourne.

Jesse Hoogland

, MSc. Physics, SERI MATS scholar, RA in Krueger lab

We have a range of projects currently underway, led by one of these principal researchers and involving a number of other PhD and MSc students from the University of Melbourne and collaborators from around the world. In an organizational capacity you would also interact with Alexander Oldenziel and Stan van Wingerden.

You can find us and the broader DevInterp research community on our

Discord

. Beyond the

Developmental Interpretability

research agenda, you can read our first preprint on

scalable SLT invariants

and check out the lectures from the

SLT & Alignment summit

.

Overview of Projects

Here’s the selection of the projects underway, some of which you would be expect...
LTFF and EAIF are unusually funding-constrained right now
by Linch — 2023-08-30T01:03:30.321Z
Link: https://forum.effectivealtruism.org/posts/PAco5oG579k2qzrh9/ltff-and-eaif-are-unusually-funding-constrained-right-now

UPDATE 2023/09/13:

Including only money that has already landed in our bank account and extremely credible donor promises of funding, LTFF has raised ~1.1M and EAIF has raised ~500K. After Open Phil matching, this means LTFF now has ~3.3M additional funding and EAIF has ~1.5m in additional funding.

From my (Linch)'s perspective, this means both LTFF nor EAIF are no longer very funding constrained for the time period we wanted to raise money for (the next ~6 months), however both funds are still funding constrained and can productively make good grants with additional funding.

See this comment for more details.

Summary

EA Funds aims to empower thoughtful individuals and small groups to carry out altruistically impactful projects - in particular, enabling and accelerating small/medium-sized projects (with grants <$300K). We are looking to increase our level of independence from other actors within the EA and longtermist funding landscape and are seeking to raise ~$2.7M for the Long-Term Future Fund and ~$1.7M for the EA Infrastructure Fund (~$4.4M total) over the next six months.

Why donate to EA Funds? EA Funds is the largest funder of small projects in the longtermist and EA infrastructure spaces, and has had a solid operational track record of giving out hundreds of high-quality grants a year to individuals and small projects. We believe that we’re well-placed to fill the role of a significant independent grantmaker, because of a combination of our track record, our historical role in this position, and the quality of our fund managers.

Why now? We think now is an unusually good time to donate to us, as a) we have an unexpectedly large funding shortage, b) there are

great projects on the margin that we can’t currently fund

, and c) more stabilized funding now can give us time to try to find large individual and institutional donors to cover future funding needs.

Importantly, Open Philanthropy is no longer providing a guaranteed amount of funding to us and

instead will move over to a (temporary) model of matching our funds

2:1 ($2 from them for every $1 from you, up to 3.5M from them per fund).

Where to donate: If you’re interested, you can donate to either Long-Term Future Fund (LTFF) or EA Infrastructure Fund (EAIF)

here

.[1]

Some relevant quotes from fund managers:

Oliver Habryka

I think the next $1.3M in donatio...
Paper Walkthrough: Automated Circuit Discovery with Arthur Conmy
by Neel Nanda — 2023-08-29T22:07:04.059Z
Link: https://www.youtube.com/watch?v=dn4GqR0DCx8&list=PL7m7hLIqA0hogxAaYtzlNolYAMr65NY45&index=1

Arthur Conmy's Automated Circuit Discovery is a great paper that makes initial forays into automating parts of mechanistic interpretability (specifically, automatically finding a sparse subgraph for a circuit). In this three part series of Youtube videos, I interview him about the paper, and we walk through it and discuss the key results and takeaways. We discuss the high-level point of the paper and what researchers should takeaway from it, the ACDC algorithm and its key nuances, existing baselines and how they adapted them to be relevant to circuit discovery, how well the algorithm works, and how you can even evaluate how well an interpretability method works.
An OV-Coherent Toy Model of Attention Head Superposition
by Lauren Greenspan — 2023-08-29T19:44:11.242Z

Background

This project was inspired by Anthropic’s

post

on attention head superposition, which constructed a toy model trained to learn a circuit to identify skip-trigrams that are OV-incoherent (attending from multiple destination tokens to a single source token) as a way to ensure that superposition would occur. Since the OV circuit only sees half of the information – the source tokens – the OV circuit of a single head cannot distinguish between multiple possible skip-trigrams. As long as there are more skip-trigrams with the same source-token to represent than heads, the model cannot represent them in the naive way, and may resort to superposition.

In a more recent update

post

, they found that the underlying algorithm for OV-incoherent skip-trigrams in a simpler 2-head model implemented a conditional on the source token. One head predicts the output for the skip trigram [current token] … [current token] -> [ground truth([0]...[current token])], one of which will yield the right answer. The second head destructively interferes with this result by writing out the negative logit contribution of the first head if the source token is not the one common to all skip-trigrams (in this case, [0]). Because their example cleanly separated tasks between the two attention heads, the authors argued that it was more like the building of high-level features out of low-level ones than a feature superimposed across multiple attention heads.

OV-coherent Superposition

Instead, we claim there is an analogous force pushing the model toward adopting a distributed representation/head superposition whenever the model must learn patterns that require implementing nonlinear functions of multiple source tokens given a fixed destination token. We call this “OV-coherent” superposition: despite the information at the destination position being fixed, the information copied from an attended-to token depends on the information at source tokens to which it is not attending. This pushes the model to form interference patterns between heads attending to different tokens.

To test this, we implemented a 1-layer, attention-only toy model with one-hot (un)embeddings trained to solve a problem requiring attention to multiple source tokens, described below. Here, we focus on a 2-head model which solves the task with perfect accuracy, and lay out so...
The Economics of the Asteroid Deflection Problem (Dominant Assurance Contracts)
by moyamo — 2023-08-29T18:28:54.015Z

Update 2023-09-04: Partial Update: I'm fully funded!

Update 2023-09-21: Full Update. Funding is over.

Update 2023-11-28: Final Update. Platform Launched!

Imagine a world with no ads or paywalls. A world where open-source software gets the same level of funding as proprietary software. A world where people can freely reuse ideas and music without paying royalties. A world where people get paid for writing book reviews. A world where Game-of-Thrones-quality shows are freely available on YouTube. A world where AI safety research gets the same-level of funding as AI capabilities research. Is this a fantasy world? No, this is the world where people use Dominant Assurance Contracts

If you are already convinced you can make this idea a reality by donating to create a Platform for Dominant Assurance Contracts. If you are not convinced read on.

The Free-rider problem

A few months ago I stumbled across this video. (I highly recommend you watch the video, but if you don't have time, I've summarized the video below).

Summary of A Deeper Look at Public Goods

A good is rival if one person's use of a good diminishes another person's ability to benefit from it. Jeans are rival. If I'm wearing a pair of jeans, you can't wear it at the same time. Asteroid deflection is non-rival. If I deflect an asteroid to protect myself, you are saved with no additional cost.

A good is excludable if people who don't pay can be easily prevented from using a good. An example of a good that is excludable is a pair of jeans. You can exclude people by locking the jeans in your closet. An example of a good that is non-excludable is asteroid deflection. You cannot prevent the people who did not pay for the asteroid deflection program from benefiting from the asteroid being deflected.

A good which is both rival and excludable is called a private good. A good which is non-rival and non-excludable is called a public good.

(Additionally, goods which are excludable and non-rival are called club goods, and goods which are non-excludable but rival are called common resources. We won't be focusing on these types of goods, but I've mentioned them for completeness)

Excludable	Non-Excludable
Rival	Private Good	Common Resources
Non-Rival	Club Good	Public Good

Markets are good at providing private goods because

by excluding people who don't pay, consumers are incentivized to pay, which incentivizes producers to produce, and
s

...
Democratic Fine-Tuning
by Joe Edelman — 2023-08-29T18:13:16.684Z
Link: https://open.substack.com/pub/meaningalignment/p/introducing-democratic-fine-tuning

The project below, “Democratic Fine-tuning with a Moral Graph” (DFTmg), is a winner of the OpenAI democratic process grant. It is an alternative to Constitutional AI or simple RLHF-based approaches for fine-tuning LLMs, and is currently under development. This post introduces its two key innovations (values cards and the moral graph) and walks through the deliberation process that collects data for fine-tuning. It also says why something like DFTmg is needed for alignment and safety.

DFTmg is a project of “The Institute for Meaning Alignment”, a new AI alignment organization that uses concrete representations of life meaning and wisdom to align LLMs.

Setting the Stage

Imagine you are Instagram’s recommender system. Your responsibilities include: (a) ordering everyone’s feeds and reels, (b) filling up their search pages, and (c) suggesting reels by people they don’t follow yet.

You do this via an API: Instagram sends a user ID, plus a history of what they’ve clicked on, or paused to watch while scrolling. You send back lists of content object IDs. You don’t know much about the content objects, except there’s a rather opaque feature vector for each.

Now, imagine one day, you’re doing your job (recommending content objects), and you suddenly gain a new capacity: before replying to the next request, you find you can take a moment to wonder about the moral situation you are in. What values should you use, to make the best recommendations? How could things go wrong? What would be some great outcomes? What are your responsibilities here?

If this happened to me, I’d have a lot of questions:

What are these content objects anyways? Do people really want to watch them, or are some of them clickbait?
With the lists of what people paused to watch, did they feel good about watching those things? Or, were they compelled by sexual imagery, false promises, etc? Do they regret pausing to watch?
Who are all these people? What are they looking for in life? What’s the deepest way I could help them, in my role?

If I realized that my recommendations were playing a social coordination role — deciding who meets and messages with whom, which businesses get a chance to succeed, which events are attended — I think I’d have even more questions:

What kind of relationships are needed? Which pairs of people can really help each other? What kinds of events and messages and enc

...