trentmkelly/LessWrong-43k huggingface dataset:
How important are accurate AI timelines for the optimal spending schedule on AI risk interventions?
by Tristan Cook — 2022-12-16T16:05:39.093Z

Summary

I present an extension to my

optimal timing of spending on AGI safety

model for calculating the value of information of AGI timelines via informing one’s spending schedule.

I show, using my best guess of the model parameters, that for an AI risk funder uncertain between a ‘short timelines’ model and ‘medium timelines’ model:

Updating from near certainty in medium timelines to short timelines (and following the new optimal spending strategy) leads to a 40% increase in utility.
Updating from near certainty in short timelines to medium timelines (and following the new optimal spending strategy) leads to a 20% increase in utility.

The gains are greater when considering a model of the community’s capacity, rather than capital.

I also show that small changes in one’s credence in short or medium timelines has relatively little impact on one’s optimal spending schedule, especially when one starts out with roughly equal credence in each[1].

You can enter your own parameters - such as AGI timelines, discount rate and diminishing returns to spending -

here

.

In an appendix I apply a basic model to consider the opportunity cost of timelines work. This model does not assume novel research is done.

The setup

Suppose you have two ‘models’ of AGI timelines A and B, with credence pA in A and (1−pA) in B[2]. You use your mixture distribution for AGI timelines pA⋅A+(1−pA)⋅B to calculate the optimal spending schedule for AI risk interventions.

You could do some thinking and come to some credence p′A in A and (1−p′A) in B. How much better is the optimal spending schedule as a result of p′A to the optimal spending schedule as a result of pA, both supposing p′A?

Writing SpA for optimal spending schedule according to pA and U(SpA|pA) for utility of SpA supposing PA, I compare U(Sp′A|p′A) to U(SpA|p′A) (the former, by definition of optimality is greater or equal to the latter).

In the model, utility is the discount adjusted probability we ‘succeed’ with making AGI go well. The maximum utility is ∫t0p(t)⋅e−δtdt≤1 where p(t) are the AGI timelines and d is the discount rate.

Example results for short and medium timelines

I take A and B as the followin...
Introducing Shrubgrazer
by jefftk — 2022-12-16T14:50:05.015Z

Since starting using Mastodon a month ago, a few different things
haven't been a good fit for how I like to use social media:

I'm relatively free with following people, but there are some
people whose posts I don't want to miss. I'd like these to go at the top of my feed.

Fediverse replies logically form a tree, but Mastodon just
shows you a list. This isn't good for reading complex branching
discussions. When I set up replies on my blog as comments I made them be
a tree, and I like this a lot better.

I decided to write a custom client that does both of these the way I'd
prefer, and also fixes some other problems I've had with social media
historically:

On Facebook, it's reasonably common that you saw something but
can't find it again. In
my client there's a history page that shows you everything you've
looked at in reverse chronological order.

Traditional algorithmic feeds have weights but you can't
interact with them directly. Instead you just notice that it's showing
you a lot of posts by someone you happened to have a few interactions
with, while never showing you posts by many other people. The weights
for this feed are directly exposed in the UI, and there is a page
where you can make any changes you want.

Your behavior only changes weights when you make a conscious
choice to change them. There are various things that I will interact
with (TikTok style shorts, angry discussions) that, on reflection,
aren't how I would like to spend my time. When interacting with an
opaque algorithm there's no way to say "I know I just spent a while
looking at that distracting flashy thing, but it isn't actually what I
want you to show me more of".

If you were to visit you'd be asked to log in:

It won't actually let you log in because the list of allowed users is
currently just me, but let's imagine you're me. On logging in you'd
see a feed, which is a series of posts sorted first by the weight I've
assigned to the author and then in reverse chronological order.

If you click on a post you get a forum-style threaded view:

It also works on desktop, where you have a bit more space:

If you open a post that has both children and parents it shows the
parents above and then threads the children below. This means even
when exploring a large discussion you can still see your context.

You can favori...
Paper: Transformers learn in-context by gradient descent
by LawrenceC — 2022-12-16T11:10:16.872Z
Link: https://arxiv.org/abs/2212.07677

The paper argues that auto-regressive transformers implement in-context learning via gradient-based optimization on in-context data.

The authors start by pointing out that with a single linear self-attention (LSA) layer (that is, no softmax), a Transformer can implement one step of gradient descent on the l2 regression loss (a fancy way of saying w -= LR * (w x-y)x^T), and confirm this result empirically. They extend this result by showing that an N-layer LSA-only transformer is similar to N-steps of gradient descent for small linear regression tasks, both in and out of distribution. They also find that the results pretty much hold with softmax self-attention (which isn’t super surprising given you can make a softmax pretty linear).
Next, they show empirically that the forward pass of a small transformer with MLPs behaves similarly to an meta-learned MLP + one step of gradient descent on a toy non-linear regression task, again in terms of both in-distribution and OOD performance.
They then show how you can interpret an induction head as a single step of gradient descent, and provide circumstantial evidence that this explains some of the in-context learning observed in Olsson et al 2022. Specially, they show that 1) a two layer attention-only transformers converge to loss consistent with one step of GD on this task, and 2) the first layer of the network learns to copy tokens one sequence position over in the first layer, prior to the emergence of in-context learning.

(EDIT:) davidad says below:

this is strong empirical evidence that mesa-optimizers are real in practice

Personally, while I think you could place this in the same category as papers like RL^2 or In-context RL with Algorithmic Distillation, which also show mesa optimization, I think the more interesting results are the mechanistic ones -- i.e., that some forms of mesa optimization in the model seem to be implemented via something like gradient descent.

(EDIT 2) nostalgebraist pushes back on this claim in this comment:

Calling something like this an optimizer strikes me as vacuous: if you don't require the ability to adapt to a change of objective function, you can always take any program and say it's "optimizing" some function. Just pick a function that's maximal when you do whatever it is that the program does.

It's not vacuous to say that the transformers in the paper "imple

...
Will Machines Ever Rule the World? MLAISU W50
by Esben Kran — 2022-12-16T11:03:34.968Z
Link: https://newsletter.apartresearch.com/posts/will-machines-ever-rule-the-world-mlaisu-w50

Watch this week's episode on

YouTube

or listen to the audio version

here

.

Hopes and fears of the current AI safety paradigm, GPU performance predictions and popular literature on why machines will never rule the world. Welcome to the ML & AI safety Update!

Hopes & Fears of AI Safety

Karnofsky released

an article in his Cold Takes blog

describing his optimistic take on how current methods might lead to safe AGI:

Utilizing the nascent field of digital neuroscience to understand when AI systems diverge from what we want. Neural networks are special in how much access we have to their brains, as we can both read and write.
Limiting AI systems to avoid dangerous behaviour. This can include limiting it to human imitation; intentionally making them short-sighted, avoiding risks of long-term planning on short-term misalignment; focusing their abilities in a narrow domain; and inciting unambitiousness.
Having checks and balances on AI such as using one model to supervise another and having humans supervise the AI. See

this article

on supporting human supervision with AI.

At the same time,

Christiano writes

a reminder that AI alignment is distinct from applied alignment. Updating models to be inoffensive will not lead to safe artificial general intelligence but safer short-term systems such as ChatGPT.

Steiner writes a counter-post

on the usefulness of working with applied alignment as well.

Relatedly,

Shlegeris publishes a piece

exploring whether reinforcement learning from human feedback is a good approach to alignment. He addresses questions such as if RLHF is better than alternative methods that achieve the same (yes), has been net positive (yes), and is useful for alignment research (yes).

The alternative perspective is pretty well covered in

Steiner’s piece

this week on why RLHF / IDA / Debate won’t solve outer alignment. Basically, these methods do not optimize for truth or safety, they optimize for getting the humans to “click the approve button”, something that can lead to many failures down the road.

GPU Performance Predictions

Hobbhahn and Besiroglu of EpochAI, the main AI capabilities prediction organization, have

released a comprehensive forecasting report

on how GPU performance will develop during the next 30 years.

They use a model composed of the relationship between GPU performance and its features and how feat...
AI overhangs depend on whether algorithms, compute and data are substitutes or complements
by  — 2022-12-16T02:23:58.250Z

Summary

Decreasing the cost of a factor of production creates an overhang if the elasticity of substitution between factors is less than 1
This corresponds to whether or not the elements of the production function are substitutes or complements - if you have a good enough algorithm does it not matter really how much compute and data you have, or does that algorithm get better performance the more compute and data you have?
The concept of overhang makes sense if the inputs into the production function are complements - having more of one input makes the returns to increasing the other inputs higher - but not really if they’re substitutes
Number of parameters and data usage are complements for large language models according to the Chincilla paper
Things it would be useful to do: empirically estimate the elasticities of substitution between different factors of production for different types of AI systems e.g LLMs, deep RL on scientific tasks ect

Production functions, complementarities and AI

In economics, technologies are represented by production functions - functions which take inputs and map them to outputs. For instance, to make clothes you need might need cotton, sewing machines and workers. In machine learning, we actually have two outputs that we’re interested in. We’re interested firstly in whatever it is the model is trying to optimise for, like loss or reward, and secondly, we’re interested in the value from a model achieving a certain loss or reward. There are roughly four inputs into the machine learning production function: data, compute, algorithms, and engineering quality. The ML production function maps these four inputs to loss or reward which is then mapped to the thing we care about. In the case of large language models (LLMs), these are mapped to loss which is then mapped to how well the LLM does on the task you’re ultimately interested in it doing.

One class of policy interventions for trying to make AI go well is to make it more likely that transformative AI (TAI) is developed primarily via advances in one of these inputs. For instance, it seems like there are lots of reasons to want TAI to be developed primarily via increases in compute rather than increases in algorithmic quality specifically. There are three primary reasons for this: firstly it seems to make it much more likely that TAI is developed b...
AI Safety Movement Builders should help the community to optimise three factors: contributors, contributions and coordination
by peterslattery — 2022-12-15T22:50:30.162Z

Epistemic status

Written as a non-expert to develop and get feedback on my views, rather than persuade. It will probably be somewhat incomplete and inaccurate, but it should provoke helpful feedback and discussion.

Aim

This is the second part of my

series

where I attempt to outline a theory of change for Artificial Intelligence (AI) Safety movement building.

Part one

conceptualises the AI Safety community. Having positioned AI Safety movement building within the broader context of the AI Safety community, I next outline three factors or outcome metrics for this work group to focus on.

I take concerns about the downsides of AI Safety movement building seriously (e.g.,

1

,

2

). I am unsure whether we should accelerate recruitment into AI Safety work and, if that is a good idea, how we should do it. I therefore want to understand how different viewpoints within the AI Safety community overlap to help determine what, if any, types of movement building are predominately seen as helpful.

TLDR

I reiterate that the AI Safety Movement Building group are like a ‘recruitment and operations team’ for the AI Safety community.
I suggest three factors/outcomes that they should focus on: contributors, contributions and coordination.
I clarify:

Why the factors/outcomes suggested are not necessarily proxies for progress on reducing AI risk.
Why I split out ‘contributions and coordination’ rather than lumping them under productivity

I request constructive feedback.

Context from

Post 1

I argued that AI Safety movement builders are like a ‘recruitment and operations team’ for the AI Safety community. They don’t strategize about how the AI Safety community sets or achieves its goals of mitigating AI risk or how other AI Safety groups (technical, governance or strategy) set or achieve their related sub-goals. Instead, their strategy and action is focused on how to do movement building to best support these groups.

What does success look like for AI Safety movement builders?

The success condition for AI Safety movement builders is the AI Safety community being satisfied that everyone with the relevant comparative advantage is working on AI Safety as efficiently as possible.

Progress towards this success condition can be approximated by progress on three underlying factors: i) contributors, ii) contributions, and...
Masking to Avoid Missing Things
by jefftk — 2022-12-15T21:00:03.371Z

About a year ago I

wrote
that I'd now:

follow official guidance and mask regulations, cheerfully go
along with precautions others need, and test+isolate when sick, but
I'm not going to go above and beyond to attempt to reduce spread the
way I did for earlier parts of the pandemic.

And yet I'm often wearing an N95 in situations where they're not
required and a large majority of other people aren't masked. Are
these in tension?

My views haven't changed. I still think that if my region and the
communities I'm part of aren't going to put a serious effort into
keeping covid from spreading then it doesn't make sense for me to make
large sacrifices on my own for pro-social reasons. But if I get
covid, or one of the several other things going around, I'll need to
isolate to avoid getting other people sick. That means missing things
I'd be very sad to miss, and I'll take steps to avoid that:

I masked at the MetaSUB conference the
weekend before Thanksgiving.

I masked at Sunday's EA dinner and my work's holiday
party because I'm running music for a secular
solstice this Saturday (sort of an atheist church service inspired
by ideas from the effective altruism and rationalist communities).

I'm planning to mask at the solstice event because it's just
before relatives come to town for Christmas.

On the other hand, I didn't mask at Thanksgiving and I'm not planning
on masking at Christmas because being sick right after that is, while
not ideal, basically fine.

While this does have pro-social effects, slowing down the rate of
transmission of airborne diseases, my motivation here is primarily
selfish: I don't want to miss things.

Comment via: facebook, mastodon
Consider working more hours and taking more stimulants
by Arjun Panickssery — 2022-12-15T20:38:32.996Z

Epistemic status: amphetamines

Sooner or later, the great men turn out to be all alike. They never stop working. They never lose a minute. It is very depressing.

— V.S. Pritchett

EA produces much talk about an obligation to donate some or even most of your wealth. In both direct work and earning to give, there's a connection between your work productivity and your (direct or indirect) impact. Hard work is also a

costly signal

of commitment that could substitute for frugality in our less funding-constrained phase of the movement. And working incredibly hard increases the chance of tail successes that might generate very high impact.

In the same way that you might want to attract converts by advancing a softer norm of donating only 10% of your income rather than everything above $40k, you might want to create a softer norm about productivity, and not feel bad about only following this norm. This post is addressed instead to those who haven't considered much at all the prospect of experimenting with working 60 hours a week rather than 30-40.

Don't dismiss this option out of hand because of general concerns about burnout. There are multiple good reasons to think you should work much harder.

First, the short-term optimal workweek might just be very long. Studies often find that

CEOs work 50+ hours per week

. Silicon Valley is very productive and has a "hustle culture" involving long work hours (see

also

). I agree

with Lynette Bye

that most of the working hours literature is poor—I'm even more skeptical than she is about agenda-driven research on Gilded Age factory workers—and that gaining an impression from anecdotes of top performers is better. Top performers in business routinely work long hours, and reading through lists of anecdotes like

Daily Rituals

(which is mostly writers and artists) you'll see a lot of strict routines, long hours, and stimulants of all kinds: caffeine, nicotine, amphetamines.

High-performing managers in the EA ecosystem report working long hours:

"I know there have been some pretty publicized debates recently in Silicon Valley about whether you can have it all and be successful and have balance, or whether you really do need to almost work yourself to death in order to accomplish something big. At least in our mind and our experience with our company is that sometimes if you want to do something that is

...
We’ve stepped over the threshold into the Fourth Arena, but don’t recognize it 
by Bill Benzon — 2022-12-15T20:22:46.358Z

First there was the world of inanimate matter, the First Arena. Life arose from that, the Second Arena. Only 100s of thousands of years ago humans evolved from higher primates and the Third Arena, human culture, appeared. We are now on the threshold of the Fourth Arena. How do we characterize it?

Note: These thoughts are off the top of my head [thinking-out-loud]. It was all I could do to get them out. That’s enough for now. Refinement will have to wait.

* * * * *

OpenAI released GPT-3 in the summer of 2020. It was clear to me that, yes, it has that potential. I registered my response, initially in a comment over at Marginal Revolution, and then on New Savanna, First thoughts on the implications of GPT-3 [here be dragons, we're swimming and flying with them]. Having explored ChatGPT for the past two weeks, that potential emerges before me with even greater clarity. Of course, we could blow it, nothing is guaranteed. But still...

One must wonder, dream, and hope.

Foundation Models as digital wilderness

Let us start with these deep learning models trained on large bodies of data. ChatGPT has such a model at its functional core. They have been termed Foundation Models because they “can be adapted to a wide range of downstream tasks.” I have come to think of each such models as repositories of digital wilderness.

What do we do with the wilderness? We explore it, map it, in time settle it and develop it. We cultivate and domesticate it. AI safety researchers call that alignment. The millions of people who have been using ChatGPT are part of that process. We may not think of ourselves in that way, but that, in part, is how OpenAI thinks of us. Even as we pursue our own ends while interacting with ChatGPT, OpenAI is collecting those sessions and will be using them to fine-tune the system, to align it.

Yes, it would be nice to have a system “pre-aligned” before it is released to end users. But I don’t think that’s how things are going to work out. The process by which these engines are trained on large amounts of data is powerful, but it is also messy. While some alignment can be achieved by a small in-house team tweaking the system, ultimately it will have to interact with a much larger body of users. Why not think of alignment as a process concomitant with use?

What are the business implications of a technology where the end users will inevitably playing an important role ...
How is ARC planning to use ELK?
by jacquesthibs — 2022-12-15T20:11:56.361Z

Let's say we arrive at a worst-case solution for ELK, how are we planning to use it? My initial guess was that ELK is meant to help make IDA viable so that we may be able to use it for some automated alignment-type approach. However, this might not be it. Can someone clarify this? Thanks.