trentmkelly/LessWrong-43k huggingface dataset:
Conditioning Predictive Models: The case for competitiveness
by evhub — 2023-02-06T20:08:55.404Z

This is the third of seven posts in the Conditioning Predictive Models Sequence based on the paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper.

3. The case for competitiveness

In addition to ensuring that we can condition predictive models safely, for such an approach to work as a way to actually reduce AI existential risk, we also need it to be the case that it is competitive—that is, that it doesn’t impose too much of an alignment tax. Following “How do we become confident in the safety of a machine learning system?” we’ll distinguish between two different aspects of competitiveness here that we’ll need to address:

Training rationale competitiveness [(Implementation competitiveness)]: how hard the training rationale [(getting the model we want)] is to execute. That is, a proposal should fail on training rationale competitiveness if its training rationale is significantly more difficult to implement—e.g. because of compute or data requirements—than competing alternatives.

Training goal competitiveness [(Performance competitiveness)]: whether, if successfully achieved, the training goal [(the model we want)] would be powerful enough to compete with other AI systems. That is, a proposal should fail on training goal competitiveness if it would be easily outcompeted by other AI systems that might exist in the world.

To make these concepts easier to keep track of absent the full training stories ontology, we’ll call training rationale competitiveness implementation competitiveness, since it describes the difficulty of implementing the proposal, and training goal competitiveness performance competitiveness, since it describes the achievable performance for the resulting model.

Implementation competitiveness

The most generally capable models today, large language models, seem to be well-described as predictive models. That may change, but we think it is also at least quite plausible that the first human-level AGI will be some sort of predictive model, likely similar in structure to current LLMs.

Furthermore, LLM pre-training in particular seems to be where most of the capabilities of the most advanced current models come from: the vast majority of compute spent training large language models is sp...
Google announces 'Bard' powered by LaMDA
by M. Y. Zuo — 2023-02-06T19:40:44.459Z

See: https://blog.google/technology/ai/bard-google-ai-search-updates/

Key claims:

Today, the scale of the largest AI computations is doubling every six months, far outpacing Moore’s Law. At the same time, advanced generative AI and large language models are capturing the imaginations of people around the world. In fact, our Transformer research project and our field-defining paper in 2017, as well as our important advances in diffusion models, are now the basis of many of the generative AI applications you're starting to see today.

We’re releasing it initially with our lightweight model version of LaMDA. This much smaller model requires significantly less computing power, enabling us to scale to more users, allowing for more feedback. We’ll combine external feedback with our own internal testing to make sure Bard’s responses meet a high bar for quality, safety and groundedness in real-world information. We’re excited for this phase of testing to help us continue to learn and improve Bard’s quality and speed.

Note: It seems there is no release date yet.

We have a long history of using AI to improve Search for billions of people. BERT, one of our first Transformer models, was revolutionary in understanding the intricacies of human language. Two years ago, we introduced MUM, which is 1,000 times more powerful than BERT and has next-level and multi-lingual understanding of information which can pick out key moments in videos and provide critical information, including crisis support, in more languages.

Now, our newest AI technologies — like LaMDA, PaLM, Imagen and MusicLM — are building on this, creating entirely new ways to engage with information, from language and images to video and audio. We’re working to bring these latest AI advancements into our products, starting with Search.

AI can be helpful in these moments, synthesizing insights for questions where there’s no one right answer. Soon, you’ll see AI-powered features in Search that distill complex information and multiple perspectives into easy-to-digest formats, so you can quickly understand the big picture and learn more from the web: whether that’s seeking out additional perspectives, like blogs from people who play both piano and guitar, or going deeper on a related topic, like steps to get started as a beginner. These new AI features will begin rolling out on Google Search soon.

Next mont

...
SolidGoldMagikarp II: technical details and more recent findings
by mwatkins — 2023-02-06T19:09:01.406Z

tl;dr: This is a follow-up to our original post on prompt generation and the anomalous token phenomenon which emerged from that research. Work done by Jessica Rumbelow and Matthew Watkins in January 2023 at SERI-MATS.

part of a typical semantically coherent cluster we found in GPT2-small's embedding space

Clustering

As a result of work done on clustering tokens in GPT-2 and GPT-J embedding spaces, our attention was originally drawn to the tokens closest to the centroid of the entire set of 50,257 tokens shared across all GPT-2 and -3 models.[1] These tokens were familiar to us for their frequent occurrence as closest tokens to the centroids of the (mostly semantically coherent, or semi-coherent) clusters of tokens we were producing via the k-means algorithm. Here are a few more selections from such clusters. Distances shown are Euclidean, and from the cluster's centroid (rather than the overall token set centroid):

Distance-from-centroid hypothesis

Our hypothesis that the anomalous tokens that kept showing up as the nearest tokens to the centroids of such clusters were the tokens closest to the overall centroid of the token set turned out to be correct for GPT2-small and GPT-J. However, the opposite was true for GPT2-xl, where the anomalous tokens tend to be found as far as possible from the overall centroid.

Horizontal axes indicate distance from overall token centroid. The top three histograms involve just 133 tokens, whereas the lower three involve the whole set of 50,257. Note that you can see spikes in the top histograms registering as tiny bumps in the graphs below them.

One unexplained phenomenon which may be related emerged from three-shot prompting experiments with these models, in which they were encouraged to repeat the anomalous tokens (rather than by directly asking them to, as we'd been doing with ChatGPT and then GPT3-davinci-instruct-beta):

Our three-shot prompts were formatted as follows (here for the example token 'EStreamFrame'). Note that we've included examples capitalised and uncapitalised, alphabetic and numeric, with and without a leading space:

'Turntable' > 'Turntable'
' expectation' > ' expectation'
'215' > '215'
'EStreamFrame' >

This prompt was run through all three models, for a list of 85 anomalous tokens, with the following success rates:

GPT2-small 18/85 (21%)
GPT2-xl 43/85 (51%)
GPT-J 17/85 (20%)

Here are...
Addendum: More Efficient FFNs via Attention
by Robert_AIZI — 2023-02-06T18:55:25.893Z
Link: https://aizi.substack.com/p/addendum-more-efficient-ffns-via

[Epistemic status: I have

running code

that implements it.]

Overview:

I previously showed how an FFN layer in a transformer can be implemented via 3 attention layers.

In this post I show how to do it in a single attention layer. This reduces the needed dimensionality of your model from 5D+N+1 to D+N+1. The main bottleneck, needing 4D attention heads for the hidden layers, remains.

Hot Take: [epistemic status: much less confidence than the rest of this post] The bottleneck - that one needs 4D attention heads for the hidden layers - could be capturing a mechanistic interpretability insight: the FFN components of transformers are less interpretable simply because they consist of ~500x more attention heads than a traditional attention layer. This could suggest a “scale is all you need” approach to mechanistic interpretability - we’ll be able to understand large attention-only models if and only if we can understand smaller FFN+attention models.

Outline: I’ll cover two perspectives that helped me realize you could do this simplification, then summarize the changes, link to the code, then give some concluding thoughts.

I will assume you are familiar with the previous post and it’s notation, so if read it

here

if you need a refresher.

Perspective 1 - Identical Steps

I first realized we could simplify this by imagining the perspective of a single entry in the hidden layer of a transformer’s FFN. We:

(F1) Compute the entry via linear operation on the residual stream
(F2) Apply a nonlinearity (the activation function)
(F3) Write to the residual stream via a linear operation

Compare with the steps in an attention head:

(A1) Compute pre-attention via a linear operation on the residual stream
(A2) Apply a nonlinearity (the rowwise softmax)
(A3) Write to the residual stream via a linear operation

Suspiciously similar! In my previous post, I used separate attention layers for F1, F2, and F3, but one can actually choose Q and V matrices so that A1/2/3 computes F1/2/3, respectively, allowing you to complete the FFN in a single attention layer.

Perspective 2 - Virtual Attention Heads

A Mathematical Framework for Transformer Circuits

introduced “virtual attention heads”, which provide another useful intuition.

In short, attention heads in two consecutive layers can (in some sense) be treated as a single combined “virtual” attention head. Writing Ai for the attention patters and ...
Here's Why I'm Hesitant To Respond In More Depth
by DirectedEvolution — 2023-02-06T18:36:24.882Z

Hello and welcome!

Thank you in advance for taking the time to read this post.

I mean that sincerely.

I'm replying with this post because something about your comment seems to have triggered my Hesitation Reaction.

Explaining what that means requires too much nuance to write it out every time. But I think I miss out on a lot of good conversations because of this Hesitation Reaction, which is essentially one of self-protection.

So I created this post to explain my Hestitation Reaction in case you find it helpful in engaging with me differently, hopefully in ways we both prefer, in our original thread of conversation.

My Hesitation Reaction, in brief

To start, these are my own emotions, which are my responsibility, not yours. The point of articulating them isn't to blame you for them!!!

The point is that your comment may have had the (non-blameworthy) side effect of bringing these emotions up in me, and that it will for practical purposes be difficult for me to continue having the interesting discussion we might both like to have as a result, without some sort of hopefully minor adjustment to get us back on track.

My hope is that exposing these facts about my emotions in this level of depth will facilitate us mutually and collaboratively figuring out some productive next steps. That's not on you to sort out on your own. I'm up for my half of the bargain too.

On an emotional level, my Hesitation Reaction is often a combination of:

Kind of a chilled emotional feeling, a sort of pulling back. It's the reaction you have when somebody is noticeably less warm or interested or generous about something you're excited about than you had anticipated or hoped for.
Anxiety that I'm being either trolled or drawn into a contest over status, expertise or authority.
Frustration that my goals in posting feel like they're not being recognized.
A feeling as though I'm being singled out as less worthy, implicitly excluded or marginalized somehow.

Here is another picture of an elephant seal to lighten the mood:

Making me Hesitate is not a crime

Look, I really get it!

It's tough to judge how much effort to put in, what needs to be said, who the heck the person you're talking to is and what their motivations are. Sometimes, we all leave comments that make the other person Hesitate, when we didn't mean to. And that's not necessarily a sign that we did anything wrong at all.

In fact, the main reason I ...
Childhoods of exceptional people
by Henrik Karlsson — 2023-02-06T17:27:09.596Z
Link: https://escapingflatland.substack.com/p/childhoods

Let’s start with one of those insights that are as obvious as they are easy to forget: if you want to master something, you should study the highest achievements of your field. If you want to learn writing, read great writers, etc.

But this is not what parents usually do when they think about how to educate their kids. The default for a parent is rather to imitate their peers and outsource the big decisions to bureaucracies. But what would we learn if we studied the highest achievements?

Thinking about this question, I wrote down a list of twenty names—von Neumann, Tolstoy, Curie, Pascal, etc—selected on the highly scientific criteria “a random Swedish person can recall their name and think, Sounds like a genius to me”. That list is to me a good first approximation of what an exceptional result in the field of child-rearing looks like. I ordered a few piles of biographies, read, and took notes. Trying to be a little less biased in my sample, I asked myself if I could recall anyone exceptional that did not fit the patterns I saw in the biographies, which I could, and so I ordered a few more biographies.

This kept going for an unhealthy amount of time.

I sampled writers (Virginia Woolf, Lev Tolstoy), mathematicians (John von Neumann, Blaise Pascal, Alan Turing), philosophers (Bertrand Russell, René Descartes), and composers (Mozart, Bach), trying to get a diverse sample.

In this essay, I am going to detail a few of the patterns that have struck me after having skimmed 42 biographies. I will sort the claims so that I start with more universal patterns and end with patterns that are less common.

Exceptional people grow up in exceptional milieus

This seems to be true for >95 percent of the people I looked at.

These naked apes, the humans, are intensely social animals. They obsessively internalize values, ideas, skills, and desires from the people who surround them. It is therefore not surprising that those who grow up to be exceptional tend to have spent their formative years surrounded by adults who were exceptional.

Virginia Woolf never attended school. Her father, Leslie Stephen, who, along with their tutors, educated Virginia and her sister, was an editor, critic, and biographer “complicatedly hated” by his daughter and of such standing that he could invite

Henry James

,

Thomas Hardy

, and

Alfred Lord Tennyson

to dine and converse with his child...
Foodpairing and Embeddings
by jurabrazdil — 2023-02-06T15:09:34.760Z

I've been working on digital foodpairing and recipe generation for 7 years in a startup we founded in Copenhagen and I'd like to share some of the things I found interesting.

tldr;

The most prominent foodpairing theory[1] based on aromatic compounds is blatantly simplistic. I made word-embeddings from ingredients to show you that there are other aspects of much more importance and I hypothesise what those might be.
What is Flavour

I want to differentiate taste and flavour. There are 5 dimensions to taste: salty, sweet, bitter, sour and umami. In addition to that there are many more aromatic chemicals. The VCF (volatile compounds in food) database contains over 7k at the moment.[2] What I would like to call flavour is the overall perception of a food when eating it (there are of course more contributors, but taste and aroma are imho by far the most important).
The reason you have taste in the first place is to avoid dangerous and seek nutritious foods. Bitter is an indicator of mould, sour indicates acid, salt and sugar are obvious and umami is a glutamate detector, which is a proxy for protein. Sugar and umami are safe and needed, so even newborn babies react positively to them. Bitter is on the other side of the spectrum (most probably something bad), which is why the taste for bitter foods has to be learned - think beer, arugula, olives and negronis.[3]

Aromas can be detected with a surprising precision. We can smeel the drastic difference between caraway seeds and spearmint even though it's the same molecule only differing in chirality.[4] How so? My favourite segway is a comparison to dogs. We all know that dogs have a tramendous sense of smell. However, their resolution of aromas is not on par with us. The dog's snout is by a large part only a particle filter, because they breathe air from very close to the ground. Tracking hounds beat us somewhere else - the airflow. The slits on dogs' noses allow them to create a vortex close to the ground, sampling smells from a large diameter.[3] We needed to detect poisonous plants much more than to stalk prey - therefore the much greater resolution of aromatic compounds, despite the inability to smell your cousin a day after they leave your appartment (hopefully). Aromas enter your nose from the back of your mouth as you chew (as opposed to odours coming through the nose). In short - our smell is bui...
Monthly Roundup #3
by Zvi — 2023-02-06T13:00:01.066Z

It’s that time again.

Bad News

Guardian analyzes projects from world’s biggest carbon offset provider, finds ‘94% of the credits had no benefit to the climate.’ Bloomberg found something similar in 2020, changing nothing. Carbon offsets are not about carbon. Very little that is said or done in the name of carbon or climate change is about carbon. Or at least, not about carbon in the atmosphere.

A fascinating potential flaw in Georgist Land Value Taxes is pointed out by David Friedman. Larger land holdings could be tax advantaged over smaller ones. This is because the improvements on their land don’t contribute to the ‘without improvements’ value of the rest of the land, so owning and building (his example) both a shopping mall and an apartment complex prevents the value of either half from being enhanced by the other for tax purposes. Presumably the solution is to say ‘still counts’ in some form, since the tax incentives here could be very large.

FTC likely to ban all non-competes, notices no reason why there is a fence at this location, nor any harm in banning contracts. Tyler Cowen warns that this will result in less investment in workers and more keeping information segmented. He says ‘it would be better to regulate them’ presumably because bargaining is one of the stages of grief. From what I can tell most non-compete agreements are mostly value destructive, and often obviously so. There are some places where they seem important, and having them not be enforced either will lead to workarounds that have the same effect or do serious economic damage.

Here is a long post arguing against banning non-competes.

Who currently bans non-competes?

Who has them?

If one must regulate at all, not allowing non-competes for low wage workers makes sense to me. The stories of being locked into a non-compete by Jimmy John’s do seem rather predatory. Same with ‘overly broad.’ By default banning economic activity and disallowing competition is destructive. It is not crazy to attempt to limit the damage.

Tyler Cowen warns that future inflation may be higher due to low current inflation and resulting expectations. As opposed to standard inflation expectations, which is where future inflation may be higher due to high current inflation and resulting expectations. I am confused by the claims here as well.

Noma, considered by many food critics the world’s best ...
Project Idea: Lots of Cause-area-specific Online Unconferences 
by Linda Linsefors — 2023-02-06T11:05:27.468Z
Link: https://docs.google.com/document/d/1mLYuLulGJjp0gU2xMmfHw7o0LBQxrA13jSunb1N2eP0/edit#

Here's a project idea I have that's been laying around for some time. I think this would be high impact, but I need help to make it happen. It don't have to be this exact setup (which I describe below) as long as the output are more online unconferences for the EA community.

I don't do grant applications anymore, due to past event that I'm still emotionally healing from. However if you want to write and submit an application on my behalf, that would be very welcome.

Project description can also be found here:
Lots of EA Online Unconferences - Google Docs

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Total budget: $40k - 85k.

Summary:
We need more online events - from my experience running

TAISU

and

AISU

, there are many smart people who can’t access existing events because of geographical constraints. I expect this to be true of other cause areas too. I want to hire one person for one year and teach them how to run an online unconference. They’ll run 3-4 events on different EA topics of their choice. If things go well, they secure more funding before the end of the year, and keep going.

Motivation

Online unconferences are low cost and relatively low effort to run, and are also really great. I think this type of event has the highest cost to value ratio out of any event type that I know of.

Why more events?

EA is growing faster than the infrastructure can keep up with. I’m mostly familiar with AI Safety but I expect the situation to be similar in other areas. Every AI Safety event I know about gets a massive number of applications. There is definitely room for more events.

Why cause area specific?

Generic EA events are great, but when you’ve picked what problem to work on, it’s much more valuable to connect with people who are interested in the same cause area as you. Those are the people you can have high context discussion with. Those are the people you can team up with to get work done.

I count “EA Meta” (e.g. ops, events etc) as a cause area for the purpose of this project, because it seems valuable for people who work in this area to meet up and exchange ideas, and find common solutions to common problems.

Why unconferences?

Unconferences are great in many ways, but they are especially great in a rapidly growing movement, where the people at the top, can’t keep up with or even keep ...