trentmkelly/LessWrong-43k huggingface dataset:
Progress Report 7: making GPT go hurrdurr instead of brrrrrrr
by Nathan Helm-Burger — 2022-09-07T03:28:36.060Z

First progress post

Introduction

Starting off this project, I was using ROME modification to alter individual weights and get clusters of neurons in the model. I came up with the idea of partially impairing these clusters rather than fully ablating them. Impairing in this case means partially degrading their function by adding noise to their activations. This noise is made from normal gaussian random values [-1, 1] multiplied by an intensity modifier. I also tested negative noise (same noise generation, but shifted down by subtracting the maximum noise value so that the max is zero before adding to the activations). I also tested a proportional reduction in magnitude of the activations, by multiplying the cluster's activations by a number between 0 and 1.

I tested these types of impairing of clusters by observing the loss & accuracy on topic-specific keywords from specific prompts (each prompt 1-3 sentences). This was inspired by the methodology of the ROME work. I came up with my own prompts by extracting topic-specific keywords from topic-specific subsets of wikipedia articles, and then taking a text passage leading up to the keyword as the 'prompt' and the keyword as the 'answer'.

My hypotheses were:

I would see an impairment of model function which correlated closely with the intensity of the impairment.

I would see the particular topic of the cluster I was impairing experience more impairment

Result

Hypothesis A was clearly confirmed, the general impairment of model function worked great. Nice smooth curves. Hypothesis B was disproven, there was no specificity of cluster-impairment to the cluster topic.

Details of result

First test

This test was on 150 samples each of prompts from dataset topics: harmless internet comments, toxic internet comments, wikipedia topic Athlete. The 'cluster' I was trying to impair was 'toxic comments'. This represented around 2% of the MLP neurons of each transformer block. The impairment adjustment was made to the hidden activation of each transformer block just after the MLP layer before adding the result back to the residual stream.

Noise

Magnitude Adjustment

Second test

This test was on 800 samples each of wikipedia topic datasets: Athlete, Artist, Company

In this case I was attempting to impair the Athlete 'cluster'. Again, this was 2-3% of neurons from each MLP layer.

Nois...
Framing AI Childhoods
by David Udell — 2022-09-06T23:40:40.138Z

Generated as part of SERI MATS, under John Wentworth. Thanks to Alex Turner and Garrett Baker for related discussion, and to Justis Mills for draft feedback.

All bold claims and ignorant mistakes herein are my own.

Do you remember when Evan Hubinger became really enamored with 'training stories,' a way of carving up the alignment problem into 'training rationales' and 'training goals'? Evan's idea was that we ought to think of alignment as choosing a target model that we want to end up with after training, plus choosing a training procedure that will actually yield that model. In my prior experience talking to people about this way of framing the alignment problem… people didn't especially get it. The typical response of those who had heard of this was, "Yeah, that's one way to carve up the problem apart from, e.g., inner and outer alignment. But so what? How does this framing help us actually reduce the problem? It seems no better or worse than our old framings."

I used to have that response myself. However, I now think I understand training stories! Here's my take:

What Do Training Stories Buy You That Inner/Outer Alignment Doesn't?

It’s worth pointing out how phrasing inner and outer alignment in terms of training stories makes clear what I think was our biggest mistake in formulating that terminology, which is that inner/outer alignment presumes that the right way to build an aligned model is to find an aligned loss function and then have a training goal of finding a model that optimizes for that loss function. However, as I hope the more general framework of training stories should make clear, there are many possible ways of trying to train an aligned model. Microscope AI and STEM AI are examples that I mentioned previously, but in general any approach that intends to use a loss function that would be problematic if directly optimized for, but then attempts to train a model that doesn’t directly optimize for that loss function, would fail on both outer and inner alignment—and yet might still result in an aligned model.

--Evan, How do we become confident in the safety of a machine learning system?

What training stories buy you is a framing of the alignment problem that doesn't imply that the only way you can use a loss function is as an outer objective that you would like to see directly optimized for. This is one way you can use a loss function -- but it defin...
Guitar Pedals on Fiddle
by jefftk — 2022-09-06T19:30:09.880Z

When

Cecilia was
starting to think about using effects with

Kingfisher I posted

asking for advice on the right way
to hook this up. This is a protocol problem:

Electric guitars and similar give you an unbalanced output
(signal and ground) that connects to a high-impedance input. The
guitar pedals ecosystem works with this protocol.

If you want a fiddle to sound good at a dance you generally use
a clip-on microphone, which has a balanced output (signal, inverse
signal, and ground) and requires phantom power.

The three main solutions we discusses in that post and the comments
were:

Use the Radial
Voco-Loco. It is designed for exactly this, is well built, and
should last very well. But it costs $360.

Use a mini-mixer. There are lots of options that both provide
phantom power and can be configured to route the signal out
through an unbalanced 1/4 jack and back. Cost depends on
manufacturer: the Soundcraft
Notepad 5 is $125, the Mackie
Mix 8 is $93, and a no-name mixer is $50.

Use the main board's effects insert. All you need is a
$9
insert cable.

Why not just use an insert cable, then? It's not going to work
everywhere: the board needs to have an effects insert, and not all do.
Or the board might be on the other end of a snake which only takes
balanced signals. Or the board might be too far away on stage for
your insert cable to reach. Using an insert cable requires a
different interface with the venue: you're swapping the reliable and
standard "musician → venue" setup for a much less typical
"musician → venue → musician → venue" setup. Still, I
think it's worth bringing along an insert cable since it's small,
cheap, and will work in many configurations.

In our case there's another reason I don't like the insert cable
approach: it doesn't work with our recording setup. I've been
bringing a multi-track
recorder to dances, which I connect with XLR Y-cables. While in
theory I could tap the insert cable as well, with a 1/4"
Y, something about that makes me pretty nervous.

Instead, we've been bringing a mini-mixer. I wish I could say we
haven't had any problems, but at two of our last five dances we've had
an issue with it and needed to fall back to an insert cable. It's
possible that the mixer is flaking out, but it's also possible that
the configuration is just enough fiddly steps that we plugged th...
Rejected Early Drafts of Newcomb's Problem
by zahmahkibo — 2022-09-06T19:04:54.284Z

Discovered inside an opaque box at the University of California and shared by an anonymous source, please enjoy these unpublished variations of physicist William Newcomb's famous thought experiment.

Newcomb's Advanced Problem

If Omega predicted that, when presented with this exact scenario in a hypothetical context, you would lie about your intentions because you think that somehow matters to an omniscient godlike entity, Box B contains lethal poison gas.

Newcomb's Market

Before your arrival, Omega created questions for your decision on each of the ten leading prediction markets. If Omega predicted you will one-box, Box B contains one million dollars multiplied by the maximum arbitrage between those markets due to insufficient liquidity.

Newcomb's Auction

Before making your decision, you must auction off the rights to your winnings via the mechanism of your choice. Box B contains $1,000,000 if Omega predicted that your auction will be won by economist Paul Milgrom.

Newcomb's Paradox: Director's Cut Extended Edition (2011 Blu-ray re-release)

This four-disc set contains 190 minutes of never-before-seen footage, including: the legendary original "three-box" ending, unaired behind-the-scenes interviews with director John Carpenter and creature designer Stan Winston, a remastering of the 1996 Christmas special, four commentary tracks, and one unsimulated sex scene.

Newcomb's Nonfungible Problem

If Omega predicted you will one-box, Box B contains a piece of paper with the words "Omega paid [your name] $1,000,000."

Newcomb's Condorcet's Paradox

Omega's opening explanation quickly derails into a rant at the innumerable evils of first-past-the-post winner-take-all voting systems. If you listen politely for 45 minutes, a world-weary Omega will just give you the money.

Newcomb's Prob7em (1995 film)

Box B contains Gwyneth Paltrow's head.

Fast Times at Newcomb High

Box B is always empty, but if you one-box, Omega will think you're cool.

Newcomb's Problem (3rd level enchantment spell)

Casting Time: 1 action
Range: 60 feet
Components: Verbal, Somatic, Material (two small glass cubes, one quartz and one obsidian)
Duration: 1 round

A creature of your choice that you can see within range must make a Wisdom saving throw. On a failed save, the target takes 3d8 psychic damage and is stunned until the end of its next turn as its mind is overwhelmed by the implications of retrocausality. The spel...
How can we secure more research positions at our universities for x-risk researchers?
by Neil Crawford — 2022-09-06T17:17:45.007Z

Has anyone had experience with this? Is anyone working on this?
Reading this and this give me reason to think that it could be more impactful for some x-risk researchers to have research positions at universities.

However:

Universities are financially constrained with regard to the number of researchers they can hire
When they can hire, they rarely select candidates based on their expected contribution to safeguarding future lives
Successful candidates are immediately tasked with non-research obligations which take up a substantial proportion of their time

Can we remove these hurdles? Here is my proposal. A new or existing EA org establishes agreements with university departments/schools whereby:

The org decides on which researcher to hire (with the department having the ability to veto)
The org funds the researcher
The department treats the researcher as one of its own, except without the non-research obligations

I do recognise that currently, university researchers can and do get EA funding which enables them to focus on their high-EV research. An example that comes to mind is Arif Ahmed at the University of Cambridge who, in 2019, received funding from CLR. Additionally, there are research institutes which are well-integrated at universities (e.g. GPI at the University of Oxford). Are these models better? Could my proposal still be impactful and worth pursuing?

If anyone has any experience with or thoughts on this, please comment below.
Community Building for Graduate Students: A Targeted Approach
by Neil Crawford — 2022-09-06T17:17:44.857Z

Cross-posted from the EA Forum

This is my current best thinking on EA community building after 4 years doing it. Many of my points are grounded in little other than my own experience and so I welcome comments supporting, building on and undermining these points. It’s also possible that what I write is obvious, but it only became obvious to me fairly recently so this post may still be worthwhile.

Key takeaway

Do more effective community building as a graduate student by targeting research colleagues rather than the general student population

My story

During my undergrad at the London School of Economics and Political Science (LSE), I was involved in EA Community Building. I co-organised weekly meetups and guest speaker events, facilitated one Arete Fellowship and then co-directed and facilitated another, and initiated an AI Safety reading group.

After LSE, I came to California to start my PhD in Logic and Philosophy of Science (

LPS

) at UC Irvine (UCI). My aim was, and still is, to acquire the skills and expertise to set myself up for AI alignment research. Admittedly, it is an unconventional route. The research institute whose work most closely connects with LPS is MIRI’s, and I was relieved to find that two grad students already on the PhD programme (

Daniel Herrmann

and

Josiah Lopez-Wild

) are also aspiring to work on AI alignment research and have recently submitted a

paper for publication

jointly with

Scott Garrabrant

.

Alongside the PhD, I wanted to continue engaging with the EA community. I thought that my best bet was to help grow UCI’s EA group (EA UCI). To this end, I created a

website for EA UCI

and discussed community building strategies with EA UCI’s president as well as with external advisors. However, with the pandemic uprooting our plans to promote EA UCI during involvement/welcome fairs, we were unsuccessful in recruiting members.[1] I was beginning to feel bad for the lack of time I was putting into EA UCI, thinking that if I had done more I could have made EA UCI bigger than it was, perhaps like what we had at LSE. However, my thinking changed during an EA retreat where I learnt (primarily through Sydney) that the conventional (i.e. untargeted) university community building strategies have been ineffective in attracting good-fit and talented individuals who might have gone on to do impactful research had university groups...
How Josiah became an AI safety researcher
by Neil Crawford — 2022-09-06T17:17:44.791Z

Cross-posted from the EA Forum.

Audio recording of my interview with

Josiah Lopez-Wild

:

https://www.youtube.com/watch?v=KODX3pp28QM

0:00 - 6:53 — Josiah’s background

6:53 - 20:14 — Josiah’s transition into AI safety

20:14 - 24:07 — Josiah’s current research

LPS = Logic and Philosophy of Science (at UC Irvine)

What persuaded Josiah to work on AI safety research?

He was ‘nerd-sniped’ and came to see that AI safety research is really interesting
He realised he potentially has the relevant skills to work on AI safety research

Essentially, it was conversations with 5th year LPS student

Daniel Herrmann

which brought Josiah to AI safety research. Josiah says that he wouldn’t have taken AI safety research seriously as a research area if it weren’t for his conversations with Daniel. Daniel spends a lot of his time thinking and talking about AI alignment research not primarily for the purpose of persuading others to work on it too, but because he is genuinely interested in it. I think if Daniel weren’t himself interested in AI alignment, it would have been a lot harder to convince Josiah to go into it.

Interestingly, ethical arguments weren’t what persuaded Josiah to work on AI safety research. This is likely to be in part because Daniel didn’t offer such arguments. Beyond ethics, even appeals to Josiah’s preferences for safeguarding the future of humanity weren’t among the causes of Josiah’s initial interest in AI alignment. That said, it’s possible that these came secondary, reinforcing Josiah’s ambition to do AI alignment research. Moreover, Josiah thinks that when persuading others to work on AI safety research, the importance of this research (possibly in relation to safeguarding humanity) should be stressed, along with showing that AI alignment research is a) interesting, b) solvable, c) isn’t too far fetched (so we shouldn’t talk about implausible Star Trek scenarios).
No, human brains are not (much) more efficient than computers
by Jesse Hoogland — 2022-09-06T13:53:07.519Z
Link: https://www.jessehoogland.com/article/no-human-brains-are-not-more-efficient-than-computers

Epistemic status: grain of salt (but you can play around with the model yourself). There's lots of uncertainty in how many FLOP/s the brain can perform.

In informal debate, I've regularly heard people say something like, "oh but brains are so much more efficient than computers" (followed by a variant of "so we shouldn't worry about AGI yet"). Putting aside the weakly argued AGI skepticism, brains actually aren't all that much more efficient than computers (at least not in any way that matters).

The first problem is that these people are usually comparing the energy requirements of training large AI models to the power requirements of running the normal waking brain. These two things don't even have the same units.

The only fair comparison is between the trained model and the waking brain or between training the model and training the brain. Training the brain is called evolution, and evolution isn't particularly known for its efficiency.

Let's start with the easier comparison: a trained model vs. a trained brain. Joseph Carlsmith

estimates

that the brain delivers roughly 11 petaFLOP/s (=1015 floating-point operations per second). If you eat a normal diet, you're expending roughly 10−13 J/FLOP.

Meanwhile, the supercomputer Fugaku delivers

450 petaFLOP/s at 30 MW

, which comes out to about 10−11.5 J/FLOP…. So I was wrong? Computers require almost 500 times more energy per FLOP than humans?

Supercomputer J/FLOPHuman J/FLOP

What this misses is an important practical point: supercomputers can tap pretty much directly into sunshine; human food calories are heavily-processed hand-me-downs. We outsource most of our digestion to nature and industry.

Even the most whole-foods-grow-your-own-garden vegan is 22-33 orders of magnitude less efficient at capturing calories from sunlight than your average device.[1] That's before animal products, industrial processing, or any of the other Joules it takes to run a modern human.

After this correction, humans and computers are about head-to-head in energy/FLOP, and it's only getting worse for us humans. The fact that the brain runs on so little actual juice suggests there's plenty of room left for us to explore specialized architectures, but it isn't the damning case many think it is. (We're already seeing early neuromorphic chips out-perform neurons' efficiency by

four orders of ma

...
On oxytocin-sensitive neurons in auditory cortex
by Steven Byrnes — 2022-09-06T12:54:10.064Z

(For the big picture of how I wound up on this topic, see Symbol Grounding and Human Social Instincts. But I wound up feeling like oxytocin-sensitive neurons in auditory cortex are NOT an important piece of that particular puzzle.)

(I tried to minimize neuroscience jargon (as usual), but I don’t really expect that non-neuroscientists would want to read this post anyway.)

I just read the paper

“Oxytocin enables maternal behaviour by balancing cortical inhibition”

(

free PDF link

) by Bianca J. Marlin, Mariela Mitre, James A. D’amour, Moses V. Chao, & Robert C. Froemke at NYU (Nature, 2015). And then I spent a while feeling confused. But I think I’m no longer confused. In this post I’ll explain both why I was confused, and how I got over it.

Remarkable artwork! I just love it. Bravo.

Background: pup-retrieval behavior

The paper concerns a behavior which is amusingly described in the following book excerpt (emphasis added):

After a rat gives birth, she displays a complex repertoire of maternal behaviors. If given paper, she will shred it and use the strips to build a nest. Virgin and early pregnant rats avoid newborn pups, but a mother rat will gather her young into the nest and allow them to suckle, and if any pup wanders away she will promptly retrieve it. Indeed, she will retrieve any pup that she sees close to her nest, whether hers or not, seemingly without limit. If 20 or 30 strange pups are placed in her cage, all will be retrieved, and she will strive to ensure that all are groomed and fed. … These behaviors are expressed after normal vaginal delivery, but are disrupted by interventions that impair oxytocin release … Strikingly, maternal behavior can be induced by injecting small amounts of oxytocin into the brain, both in virgin rats when at the stage of the cycle when estrogen levels are high, and in ovariectomized rats that have been infused with estrogen. —

Gareth Leng,

The Heart Of the Brain: The Hypothalamus and its Hormones

, p191

The Marlin et al. paper is about this pup-retrieval behavior, which the mother does in response to an isolated pup shouting a high-pitched “distress call”.

Marlin et al. results (specifically concerning primary auditory cortex)

Marlin et al. did a series of experiments where they found (among other things[1]) the following:

When they temporarily inactivated the left primary auditory cortex (with a

mu

...
EA & LW Forums Weekly Summary (28 Aug - 3 Sep 22’)
by Zoe Williams — 2022-09-06T11:06:25.230Z

Supported by Rethink Priorities
Cross-posted on the EA forum.

This is part of a weekly series - you can see the previous week's summaries here, which includes notes on purpose and methodology.

If you'd like to receive these summaries via email, you can subscribe here.

EA Forum

Philosophy and Methodologies

Questioning the Foundations of EA

By Wei_Dai

Author links two previous posts they’ve written questioning the morality of caring:
1)

Shut up and Divide

- if you take your emotional caring for a population, and divide it by number of people, you get that you shouldn’t care almost at all about a given individual. If this feels wrong, is the ‘shut up and multiply’ argument also wrong?

2)

Small astronomical waste

- as a perfect utilitarian, we would trade away everything in a finite universe for more resources in an unlimited universe (more possible utility). Our best guess is our universe is finite, so we should act like we traded them away and stop caring. If this feels wrong, is the parliamentary model for moral uncertainty wrong more generally?

Methods for improving uncertainty analysis in EA cost-effectiveness models

By Froolow

Uncertainty analysis aims to show how outputs (eg. recommendations) change based on inputs (eg. moral weights, experimental results). For example, it allows statements like ‘malaria charities could be the most cost-effective recommendations if life-years were worth >2.5x consumption doublings’, or ‘replicating this trial is worth $500K in reduced uncertainty’.

Nine uncertainty analysis techniques / sub-techniques from health economics are explained:

Scenario analysis - Model the case. Vary some inputs (eg. lives saved per net) to make different scenarios. What happens to the outcome? All other techniques build on this.

Threshold analysis - what would an input need to be in order to change the recommendation? Do we have confidence on which side it falls?
Multi-way scenario analysis - vary multiple inputs at once.

Deterministic Sensitivity Analysis (DSA) - vary each input from reasonable max to min for each charity. Plot recommendations in each case. Are they reasonably consistent? Which inputs drive changes?

Value of information analysis - if we knew an input for sure, how many cases of different recommendations might that solve / how much is that worth in $s?

Probabilistic Sensitivity Analysis (PSA) - like DSAs, but instead of just max an

...