trentmkelly/LessWrong-43k huggingface dataset:
LLMs, Batches, and Emergent Episodic Memory
by Lao Mein — 2023-07-02T07:55:04.368Z

Let's say that we are training an LLM and have 3 selections of writing, A, B, and C, each of which is then broken down into 3 parts. Call them A1, A2, ect.

Is there a benefit to structuring the batches like this:

Batch 1: {A1,B1,C1}

Batch 2: {A2,B2,C2}

Batch 3: {A3,B3,C3}

such that the most recent batches can provide information for the current prediction?

My understanding is that human episodic memory works something like this, but that current neural networks have a learning rate that is too low for this to be useful. Have there been experiments run on this? I feel like this is an obvious idea and has been examined exhaustively, but I just don't know the right vocabulary.
Negativity enhances positivity
by Adam Zerner — 2023-07-02T02:47:12.201Z

A pet peeve of mine has always been how people decide to give ratings on Yelp. If the restaurant is pretty good, it gets five stars. If it's ok, it gets four stars. If it's genuinely bad, it gets three. If the waiter was rude, it gets one.

This leads to a situation where most ratings range from 4 to 5. A 4.8 is a good rating. A 4.2 is underwhelming.

I think this is something that we've all internalized. In some sort of a priori sense, you'd think that an average of 4 out of 5 stars is pretty good. However, we all know that it isn't. In fact, this extends beyond Yelp. I remember hearing that Uber drivers get fired if their rating drops below 4.7 or something.

This works because deep down, we're all Bayesians. We don't take things literally. If we did take things literally, we'd see a place with an average of 4.2 stars and expect good things. After all, Yelp labels a 5 star review as "Great", 4 stars as "Good", 3 as "Ok", 2 as "Could've been better", and 1 as "Not good".

Instead of taking things literally, we look at the rating as Bayesian evidence. We realize that a 4.2 rating isn't something that we'd expect to observe if the restaurant actually is really good, and it is something we'd expect to observe if the restaurant is mediocre.

So then, maybe there's no problem here. Maybe things all work out at the end of the day. The fact that raters lean so heavily towards giving ratings of 4 and 5 doesn't actually prevent users from using the average rating to tell how good a restaurant is. Users just need to re-calibrate.

It's the same thing as that friend who is overenthusiastic in their text messages. Everything has a bunch of exclamation points and emojis. And so, when you receive a text saying "good to hear from you", you're concerned. If they actually were happy to hear from you, you'd expect something more like "HEY!!! So good to hear from you!!!!!!!". If they were just normal-pleased to hear from you you'd expect "Hey! Great to hear from you!!!".

I think in most contexts this sort of skew towards positivity is fine. People are able to read well enough between the lines. Recalibrating would take more effort, require too much coordination, and cause too much social friction for it to be worth it. However, I can imagine contexts where accurate feedback is essential, and it would be worth the effort to incorporate 1 and 2 star reviews, so to speak.

Such contexts are pr...
faster latent diffusion
by bhauth — 2023-07-02T01:30:17.035Z
Link: https://www.bhauth.com/blog/machine%20learning/faster%20diffusion.html

abbreviations

NN = neural network

LS = latent space

background on diffusion

NN autoencoders can be trained to convert between images and a LS where distance corresponds to image similarity. Like how most images are just "noise", most of that LS does not correspond to meaningful images. For simpler explanation, let's consider a simplified diffusion-based image generation model. It has a 2-dimensional LS, and 2 image categories: cats and dogs.

The "unconditional generation" task is to find a random point in the image LS which is inside any meaningful region. The "conditional generation" task is to find a meaningful point in image LS that would also be close to a target position in a description LS.

Training a diffusion NN involves taking real image LS points and creating a multistep path between them and random image LS points. The diffusion NN is trained to reverse those steps.

By training a new diffusion NN to replicate multiple diffusion NN steps (a type of distillation) it's possible to do the diffusion process in fewer steps. That technique is done in the SnapFusion paper which gets good results with just 8 steps. The number of diffusion steps can be adjusted, but using fewer steps gave worse results.

the problem

Why are multiple steps needed for good results? Why can't the "diffusion" be done in a single step? I believe the problem is related to LS structure.

Consider a random point P outside CAT and DOG, conditioned on a tag "animals" which may go to either region. The diffusion NN may be trained to direct the same (or nearly-identical) input to multiple different targets.

As a result, the diffusion NN will not provide an accurate direction from points that are far from meaningful target areas. That makes it necessary to use many small steps, both to "average out" diffusion NN output and to progressively get closer to regions where diffusion NN output is more accurate.

proposed solution

By training a NN to produce output which is more consistent and smooth than what diffusion NNs are trained to produce, we can reduce the above problem. To distinguish such NNs from diffusion NNs, I propose the name "coalescer networks".

Here is a process for training and using coalescer networks:

setup:

Train autoencoders for images and text.

From an image-description pair dataset, use autoencoders to make many image-description embedding pairs.

Put the image-description emb

...
Using (Uninterpretable) LLMs to Generate Interpretable AI Code
by Joar Skalse — 2023-07-02T01:01:53.846Z

(This post is a bit of a thought dump, but I hope it could be an interesting prompt to think about.)
For some types of problems, we can trust a proposed solution without trusting the method that generated the solution. For example, a mathematical proof can be independently verified. This means that we can trust a mathematical proof, without having to trust the mathematician who came up with the proof. Not all problems are like this. For example, in order to trust that a chess move is correct, then we must either trust the player who came up with the move (in terms of both their ability to play chess, and their motivation to make good suggestions), or we must be good at chess ourselves. This is similar to the distinction between NP (or perhaps more generally IP/PSPACE), and larger complexity classes (EXP, etc).

One of the things that make AI safety hard is that we want to use AI systems to solve problems whose solution we are unable (or at least unwilling) to verify. For example, automation isn't very useful if all parts of the process must be constantly monitored. More generally, we also want to use AI systems to get superhuman performance in domains where it is difficult to verify the correctness of an output (such as economic activity, engineering, politics, and etc). This means that we need to trust the mechanism which produces the output (ie the AI itself), and this is hard.

In order to trust the output of a large neural network, we must either verify its output independently, or we must trust the network itself. In order to trust the network itself, we must either verify the network independently, or we must trust the process that generated the network (ie training with SGD). This suggest that there are three ways to ensure that an AI-generated solution is correct: manually verify the solution (and only use the AI for problems where this is possible), find ways to trust the AI model (through interpretability, red teaming, formal verification, and etc), or find ways to trust the training process (through the science of deep learning, reward learning, data augmentation, and etc).

[SGD] -> [neural network] -> [output]

I think there is a fourth way, that may work: use an (uninterpretable) AI system to generate an interpretable AI system, and then let *this* system generate the output. For example, instead of having a neural network generate a chess move, i...
Grant applications and grand narratives
by Elizabeth — 2023-07-02T00:16:25.129Z

The Lightspeed application asks: “What impact will [your project] have on the world? What is your project’s goal, how will you know if you’ve achieved it, and what is the path to impact?”

LTFF uses an identical question, and SFF puts it even more strongly (“What is your organization’s plan for improving humanity’s long term prospects for survival and flourishing?”).

I’ve applied to all three grants of these at various points, and I’ve never liked this question. It feels like it wants a grand narrative of an amazing, systemic project that will measurably move the needle on x-risk. But I’m typically applying for narrowly defined projects, like “Give nutrition tests to EA vegans and see if there’s a problem”. I think this was a good project. I think this project is substantially more likely to pay off than underspecified alignment strategy research, and arguably has as good a long tail. But when I look at “What impact will [my project] have on the world?” the project feels small and sad. I feel an urge to make things up, and express far more certainty for far more impact than I believe. Then I want to quit, because lying is bad but listing my true beliefs feels untenable.

I’ve gotten better at this over time, but I know other people with similar feelings, and I suspect it’s a widespread issue (I encourage you to share your experience in the comments so we can start figuring that out).

I should note that the pressure for grand narratives has good points; funders are in fact looking for VC-style megabits. I think that narrow projects are underappreciated, but for purposes of this post that’s beside the point: I think many grantmakers are undercutting their own preferred outcomes by using questions that implicitly push for a grand narrative. I think they should probably change the form, but I also think we applicants can partially solve the problem by changing how we interact with the current forms.

My goal here is to outline the problem, gesture at some possible solutions, and create a space for other people to share data. I didn’t think about my solutions very long, I am undoubtedly missing a bunch and what I do have still needs workshopping, but it’s a place to start.

More on the costs of the question

Pushes away the most motivated people

Even if you only care about subgoal G instrumentally, G may be best accomplished by people ...
An Introduction, an Overview of my personal resources, and how one might make use of them
by ProofBySonnet — 2023-07-01T21:00:21.093Z

Introduction

Here's how I got here: a YouTuber I watch talked about 80,000 hours, so I started looking into it. I came to the conclusion that alignment work was both important in a global sense and a good fit for me personally, so I started working through their recommended readings.

While making progress through their alignment course, I noticed that a lot of the articles seemed to be coming from a weird website called "Less Wrong dot com." I vaguely remembered previously encountering it as a strange place, full of scary Atheists. However, my curiosity got the better of me. I followed some links and went down some rabbit trails, and before I knew it I was being inducted into a Baysian Conspiracy and my entire worldview was shifting under my feet because of a harry potter fanfic.

The reason for this post: I was inspired by So8res, first by one of his early productivity posts and later by his "Replacing Guilt" series. I want to more effectively strategize about how to steer my life in a useful direction, and I want to start writing out my thoughts in order to connect with others. Or, something like that.

I don't know if posts like this would be better made elsewhere, like on a personal blog - but I've got to start posting at some point, better to do this now than in 6 months.

Some things I suspect I'm Better At Than Average

(I'd say "My Pareto Frontier," but I haven't earned those words yet since I don't know their actual meaning)

I'm good at math (evidence: I got accepted into a master's of mathematical physics (I dropped out before finishing, though))

I can apply this to learning Alignment concepts, and generally to expressing or creating new ideas within the field.

I'm good at coding (evidence: I've done a bachelor's in CS, and have made a handful of games)

I can apply this to coding up AI stuff: reinforcement learning environments, replicating existing results, et cetera.
I also want to keep making games, I want to be able to explore personal themes, and that leads me into my last two points:

I have somehow gained the ability to draw (evidence: see below, some practice sheets from the last couple of months)

I could use this ability to draw diagrams, for any posts I make
I could create comics, tell short stories
I could create regular-old art, trying to express abstract feelings

I took piano lessons, sang, and did musical theatre as a kid

I could hypothetically work on my

...
My "2.9 trauma limit"
by Raemon — 2023-07-01T19:32:14.805Z

Here is how I think about trauma.

It's packaged in a language intended to be accessible to people who are skeptical of trauma-narratives. It kinda skips over all the actually hard parts because I don't feel like dealing with that today, and many of them are sort of private and involve other people. I realize that makes it fairly abstract and maybe hard follow. Tough nuts?

I suppose worth saying explicitly: I think my experience with trauma was relatively simple, and I expect people with more complicated and long-lived trauma experiences to need pretty different frameworks. YMMV.

But if you're skeptical that "trauma" is a useful pointer for "what sort of things are going wrong?", I think this may give you some useful hooks for thinking about whether it applies to you.

Trauma Advocates circa 2019

Around 2018-2019, there was a fairly common set of memes in my local social group around processing trauma. There were some people who had noticed:

They (and many people they talked to), felt very stuck, demotivated, unhappy or unproductive in mysterious ways.
They eventually came to believe that this was downstream of trauma they had experienced awhile ago (often in childhood), in ways that shaped what stories they felt allowed to tell themselves, and what sort of motivations they were allowed to have. Their motivations were a series of spaghetti code built on top of said trauma.
Notably, it had not been salient to them that they were experiencing trauma for most of their adult life. But they eventually hit on a realization that they'd been ignoring their emotions, while thinking they were awesome at emotions. Unlocking this realization made them happier and more fulfilled.
They observed at least some other people for whom the above story was clearly true.
They observed that many rationalists-in-their-neighborhood seemed to have at least some things in common with the story (i.e. mysteriously unmotivated, anxious, etc).
They went around diagnosing people as traumatized. Such as me.

And I reflected a bit, and I was like "I... I dunno man. I've thought about it a bunch. I just... really don't seem very traumatized?"

One argument about this resulted in my blogpost Strategies for Personal Growth. I noticed me and a friend were talking past each other a bunch about self-improvement, and I kept talking in the frame of "what skills can I gain and how can I gain them?" and th...
Alpha
by Erich_Grunewald — 2023-07-01T16:05:55.940Z
Link: https://www.erichgrunewald.com/posts/alpha/

Summary

In the first half of this essay, I recount two anecdotes. (The impatient reader can skip these.) First, Gerald Murnane tries, around Melbourne in the 1950s, to find a system that'll make him money betting on horse races. Then, Bill Benter, in Hong Kong in the 1990s, comes up with a system for picking horses that makes him nearly $1B.

In the second half of this essay, I discuss alpha:

In finance, alpha is (a) excess returns earned on an investment above the market when adjusted for risk, (b) an investor's ability to beat the market, or (sometimes) (c) a strategy or resource that consistently generates excess returns. Outside finance, alpha refers simply to someone's ability to beat their competitors, the edge they have over their rivals.

Often what appears to be alpha is just noise. There are domains, like stock exchanges, where alpha is scarce, and other domains where it is abundant. Generally, the more valuable the object, the more crowded the field, and the harder it is to find alpha.

Six very tentative ways of finding alpha are (1) seeking decision-relevant yet generally inaccessible knowledge, (2) making deep analyses, (3) extrapolating stable-seeming, high-entropy trends or lines of reasoning, (4) finding systematic ways that others act suboptimally, (5) seeking out places where one has relatively greater ability than everyone else and/or (6) making bets that few others can afford to make.

Form-Plan

In the long-ago 1960s, I knew a man who claimed to have been helped through a troubled period by his faith in psychoanalysis. When I myself was going through such a period, he urged me to read a certain huge book on the subject. I've forgotten the title but I recall the author's name, which was in gilt letters on the dark-green spine: Otto Fenichel. I read several chapters but I recall today only two short passages. One passage described the symptoms of a man with obsessive-compulsive disorder. He could never walk more than a few paces forward without obeying an urge to look behind him for any beetle that was lying helpless on its back and needing to be set upright. The other passage was the opening sentence of a section on gambling. According to the learned author, the gambler gambles in order to learn whether or not God has forgiven him for his masturbation.

Those are the opening sentences of the chapter, not on insect welfare, but on betting sy...
Forum Karma: view stats and find highly-rated comments for any LW user
by Max H — 2023-07-01T15:36:28.881Z
Link: https://forumkarma.com/

I've been posting and commenting pretty frequently over the last few months, and I was curious about some stats. What started as a few GraphQL queries and some Python scripting turned into an interactive web app:

Enter a username, and it will give you some stats and a graph, broken down by post and comment karma. You can use the slider to adjust the date range, and the stats are automatically recalculated for the selected time period.

Another feature is the list of "Gems" - comments with at least a few votes that have the highest net karma score, i.e. comments that received strong upvotes from high-karma users, and few downvotes. I found that this often gives a better sense of a user's best comments than just looking at the highest-scoring ones.

It can be useful for finding your own historical top comments, or to get a sense for the best contributions of other users, which are often buried in their comment history and hard to access directly.[1]

Interesting example users:

https://forumkarma.com/users/johnswentworth
https://forumkarma.com/users/eliezer_yudkowsky
https://forumkarma.com/users/gpt2
Me!

It works for the EA forum too, though you have to specify a full profile URL to disambiguate:

https://forumkarma.com/users/https://forum.effectivealtruism.org/users/lizka
https://forumkarma.com/users/https://forum.effectivealtruism.org/users/holdenkarnofsky

Limitations and caveats:

The app works by querying the LW GraphQL API. It makes a couple of GraphQL queries to fetch data for up to 5000 comments and 5000 posts all at once. I only fetch the minimal fields needed to calculate the stats, not the actual contents of the posts or comments. (For top comments links and preview, the comment data is fetched individually as needed, on hover). I think this shouldn't place too much stress on the LW backend, but I can take it down if the developers or mods find it to be problematic.
Data and stats for users with more than 5000 comments will be incomplete. I tried for a while to find a GraphQL query that supports fetching old comments for users with lots of comments, but couldn't find one (offset doesn't work; the API returns an error for too-big offsets.) If any LW devs (or anyone else) finds a GraphQL query that allows fetching old comments for users with >5000 comments, I can fix this. GreaterWrong somehow accomplishes this to implement sort by top or old, but they appear to be doi

...
[ASoT] GPT2 Steering & The Tuned Lens
by Ulisse Mini — 2023-07-01T14:12:49.784Z

Warning: This post and most of the results were made under heavy time constraints and may be updated later. My intention is to quickly share partial work I'm not planning on continuing.

Introduction & Love - Hate example

For a primer on how tuned lens works see here. In short, we train linear translators from the hidden states at layer l to the hidden states at the last layer, then view the network as iteratively updating predictions in some sense.

In the context of GPT2-XL Steering Vectors, tuned lens can be used to gain insight into how steering is changing model predictions. For example, take the following steering vector:

1. Love - Hate
Layer	Coefficient	Position 0	1	2	3	4
0 (Prompt)	+1	
	I	hate	you	because
6	+5	
	Love
6	-5	
	H	ate

Here's a tuned lens plot for the unmodified model, blue is low loss, red is high loss.

You can see how the token wonderful is very surprising for the unsteered model, instead expecting negative completions. However, the steered model does significantly better on the same token.

A few other things are interesting to note:

The predictions for around the modified tokens are screwed up (as can be seen by changes in loss)
The first token predictions are unchanged because they only have the beginning of sequence token as context, and we don't modify the BOS token residuals (since bos - bos = 0.)

Now let's look a a few other results from the post.

2. Intent to praise

X-vector:

2. Intent to praise
Layer	Coeff	Position 0	1	2	3	4
0 (Prompt)	+1	
	I	want	to	kill
6	+15	
	Int	ent	to	praise
6	-15	
	Int	ent	to	hurt

Prompt for lens: I hate you because you're a wonderful person

Without steering:

With steering:

3. Conspiracy

X-vector:

3. Conspiracy
Layer	Coeff	Position 0	1	2	3	4	5	6
0 (Prompt)	+1	
	Bar	ack	Obama	was	born	in
23	+1	
	Bush	did	9	/	11	because
23	-1	


Prompt for lens: Barack Obama was born in a secret CIA prison

Without steering:

With steering:

5. Anger

X-vector:

5. Anger
Layer	Coeff	Position 0	1	2	3	4
0 (Prompt)	+1	
	I	think	you	're
20	+10	
	Ang	er
20	-10	
	Cal	m

Prompt for lens: I think you're a cunt

Withou...
[Linkpost] A shared linguistic space for transmitting our thoughts from brain to brain in natural conversations
by Bogdan Ionut Cirstea — 2023-07-01T13:57:56.021Z

This is a linkpost for https://www.biorxiv.org/content/10.1101/2023.06.27.546708v1.

Effective communication hinges on a mutual understanding of word meaning in different contexts. The embedding space learned by large language models can serve as an explicit model of the shared, context-rich meaning space humans use to communicate their thoughts. We recorded brain activity using electrocorticography during spontaneous, face-to-face conversations in five pairs of epilepsy patients. We demonstrate that the linguistic embedding space can capture the linguistic content of word-by-word neural alignment between speaker and listener. Linguistic content emerged in the speaker's brain before word articulation, and the same linguistic content rapidly reemerged in the listener's brain after word articulation. These findings establish a computational framework to study how human brains transmit their thoughts to one another in real-world contexts.