trentmkelly/LessWrong-43k huggingface dataset:
GPT-2's positional embedding matrix is a helix
by AdamYedidia — 2023-07-21T04:16:26.286Z

In the context of transformer models, the "positional embedding matrix" is the thing that encodes the meaning of positions within a prompt. For example, given the prompt:

Hello my name is Adam

the prompt would generally be broken down into tokens as follows:

['
', 'Hello', ' my', ' name', ' is', ' Adam']

(For whatever reason prompts to GPT-2 generally have an 
 token prepended to them before being fed through, to match how the model was trained.)

For this prompt, the mapping of tokens to positions would be as follows:

'
': 0

'Hello': 1

' my': 2

' name': 3

' is': 4

' Adam': 5

The positional embedding maps the positions of those tokens (0, 1, 2, 3, 4, and 5) to the meanings of those positions in vectorspace. More concretely, the positional embedding matrix maps each of those five numbers to a 768-dimensional vector of floating-point numbers, and that 768-dimensional vector gets added to a different vector that represents the semantic meaning of the token. But the first vector comes directly from the positional embedding matrix, and it is the only way the transformer has of identifying where in the prompt a given token was. So we should expect that each row of the positional embedding matrix is unique. Otherwise, two different positions would be mapped to the same vector, and the transformer would have no way of knowing which of those two positions a given token was in!

There are 1024 rows in the positional embedding matrix; this is because there are 1024 possible positions in the prompt, and each possible position gets its own row.

We should also expect that the different 768-dimensional vectors live in a low-rank linear subspace, which is just a fancy way of saying a line or a plane or something like that. After all, if you were a human engineer designing a transformer from scratch, you might devote just one of the 768 entries in each vector to encode the position—for example, you might use the first entry of each vector for this purpose, and make it be a 0 if the token was in position 0, 1 if it was in position 1, 2 if it was in position 2, and so on, and then you'd have the other 767 entries to use to encode the semantic meaning of each token, without interfering with your encoding of the token's position. Then, in vector-space, the whole positional embedding matrix would lie on a single line—the line of p...
Problems with predictive history classes
by dkl9 — 2023-07-20T23:28:20.417Z

Here I collect and (partly) address problems noted by commenters on "Predictive history classes", in no particular order. I do not expect to refute all of them. Some of them may actually cripple the proposal/show it to be impractical.

I figure this is more honest than leaving just the one-sided original proposal for those who don't read comments.

Thank you, commenters (Raemon, Dweomite, cousin_it, Inari, stonefly), for noticing these problems.

This already exists (Pastcasting)

Great! But that doesn't try to teach methods, nor is it in actual schools.

No one knows how to do this (model history well)

So we figure it out. Maybe have the students figure it out as they study, but that might be expecting too much of them.

Or hold the students to the standards of the professionals, rather than the standards of perfection. Give partial credit; curve the tests. If all that humanity knows how to do is 30% accuracy,[1] a student's 25% is regarded as good.

Also, don't be so sure of the limits of knowledge.

But maybe even the professionals would be so bad that the students' noise-in-guessing easily crosses from 0% to professional%, thus making the subject unassessable.

Situation seen now isn't situation seen then

That is, the summary of a historical situation, as used for teaching, may show the information very differently from what one would get as it happened (which is what you're supposed to practise for).

This can probably be mitigated by careful choice of content, e.g. using only news-from-that-time. Most news is horribly biased, but that might not matter, sith it closely reflects what you could readily access when studying the present and future.

Inari's objections

tl;dr (I think): history classes are for establishing a cultural basis, not thinking (at least before university); current studies have a method (not prediction-based) that needs to still be taught; there are other things you need to teach and can't make this a whole class.

I don't want to dismiss this — some of these may completely cripple the proposal — but I don't understand them yet.

They can be a source of more indoctrination

A predictive history teacher could pick a contrived subset of history to show particular patterns that support a message to indoctrinate.

Maybe we mitigate this be requiring that the examples be sufficiently spread out (at least X from all these places Y and time periods Z) — but who design...
Announcement: AI Narrations Available for All New LessWrong Posts
by Solenoid_Entity — 2023-07-20T22:17:33.454Z

TYPE III AUDIO is running an experiment with the LessWrong team to provide automatic AI narrations on all new posts. All new LessWrong posts will be available as AI narrations (for the next few weeks).

You might have noticed the same feature recently on the EA Forum, where it is now an ongoing feature. Users there have provided excellent feedback and suggestions so far, and your feedback on this pilot will allow further improvements.

How to Access

On Post Pages

Click the speaker icon to listen to the AI narration:

Podcast Feeds

The existing "LessWrong Curated” podcast feed will host Curated posts and those with over 125 karma.
The new "LessWrong (All Audio)" feed features all posts with over 30 karma. (Spotify), (Apple Podcasts) (RSS/Other)

Perrin Walker

(AKA Solenoid Entity) of TYPE III AUDIO will continue narrating most curated posts for now.

Send us your feedback.

Please send us your feedback! This is an experiment, and the software is improved and updated daily based on user feedback.

You could share what you find most useful, what's annoying, bugged or difficult to understand, how this compares to human narration, and what additional features you'd like to see.

For comments on a specific narration, use the feedback button on the audio player or visit t3a.is.
For general feedback or suggestions, please comment on this post or email us at lesswrong@type3.audio.
Writers interested in having their work narrated or requesting a particular narration, please contact team@type3.audio.

Is this just text-to-speech on posts?

It's an improvement on that.

We spoke with the Nonlinear Library team about their listeners' most-requested upgrades, and we hope our AI narrations will be clearer and more engaging than unimproved TTS. Some specific improvements:

Audio notes to indicate headings, lists, images, etc.
Image alt-text is narrated.
Specialist terminology, acronyms and idioms are handled gracefully. Footnotes too.
LaTeX math notation handled gracefully in all cases and narrated in some simple cases.
We skip reading out long URLs, academic citations, and other things that you probably don't want to listen to.
Episode descriptions include a link to the original post. According to Nonlinear, this is their most common feature request!
More podcast feed options.

We'd like to thank Kat Woods and the team at Nonlinear Library for their work, and for giving us helpful advice on this project....
AI #21: The Cup Overfloweth
by Zvi — 2023-07-20T21:30:11.064Z

By the Matt Levine Vacation Rule, I took several days to go to Seattle and there was a truly epic amount of news. We had x.AI, Llama 2, upgrades to ChatGPT, a profile of Anthropic, a ton of very interesting papers on a variety of topics, several podcasts that demand listening, fully AI-generated South Park episodes and so much more. I could not fully keep up. Oh, and now we have Barbieheimer.

Thus, I have decided to spin out or push to next week coverage of four stories:

The release of Llama 2.

The plans of x.AI.

The profile in Vox of Anthropic.

Whether GPT-4 is getting worse, as was claimed.

These might get their own posts or they might get pushed to next week, depending on what I find on each. Same with my coverage of Oppenheimer since I haven’t seen it yet, and my bonus thoughts on Mission Impossible: Dead Reckoning (spoiler-free review of MI:DR for now: Good fun if you like such movies, some interesting perspective of on how people would handle such a situation, a lot of clear struggling between the writer who knows how any of this works and everyone else involved in the film who didn’t care and frequently but not always got their way.)

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Claude 2, sentiment analysis.

Language Models Don’t Offer Mundane Utility. Bard and economic comparisons.

Reasoning Out Loud. Two Anthropic papers analyze the impact.

Fun With Image and Sound Generation. Can the AI take it to 11?

Deepfaketown and Botpocalypse Soon. Gonna have myself an autogenerated time.

They Took Our Jobs. Actors are on strike. They have good reason to be.

Get Involved (aka They Offered Us Jobs). Several opportunities to help.

Introducing. Prosthetics that used to cost $50k might now cost $50?

In Other AI News. Google testing something harmless called Genesis.

Quiet Speculations. To protect the future one must first envision it.

The Superalignment Taskforce. Jan Leike responds in excellent fashion.

Superforecasters Feeling Super. Extinction tournament doesn’t go anywhere.

The Quest for Sane Regulation. China, FTC an the SEC are all talking the talk.

The Week in Audio. An avalanche of great stuff. I’ll get to you soon, everyone.

Rhetorical Innovation. Christopher Nolan officially joins the fun. Bingo, sir.

No One Would Be So Stupid As To.

...
All AGI Safety questions welcome (especially basic ones) [July 2023]
by smallsilo — 2023-07-20T20:20:45.675Z
Link: https://forum.effectivealtruism.org/posts/vGfJnwq6X7hwhG3wy/all-agi-safety-questions-welcome-especially-basic-ones-july

tl;dr: Ask questions about AGI Safety as comments on this post, including ones you might otherwise worry seem dumb!

Asking beginner-level questions can be intimidating, but everyone starts out not knowing anything. If we want more people in the world who understand AGI safety, we need a place where it's accepted and encouraged to ask about the basics.

We're putting up monthly FAQ posts as a safe space for people to ask all the possibly-dumb questions that may have been bothering them about the whole AGI Safety discussion, but which until now they didn't feel able to ask.

It's okay to ask uninformed questions, and not worry about having done a careful search before asking.

AISafety.info - Interactive FAQ

Additionally, this will serve as a way to spread the project Rob Miles' team[1] has been working on: Stampy and his professional-looking face aisafety.info. This will provide a single point of access into AI Safety, in the form of a comprehensive interactive FAQ with lots of links to the ecosystem. We'll be using questions and answers from this thread for Stampy (under these copyright rules), so please only post if you're okay with that!

Stampy - Here to help everyone learn about stamp maximization AGI Safety!

You can help by adding questions (type your question and click "I'm asking something else") or by editing questions and answers. We welcome feedback and questions on the UI/UX, policies, etc. around Stampy, as well as pull requests to his codebase and volunteer developers to help with the conversational agent and front end that we're building.

We've got more to do before he's ready for prime time, but we think Stampy can become an excellent resource for everyone: from skeptical newcomers, through people who want to learn more, right up to people who are convinced and want to know how they can best help with their skillsets.

Guidelines for Questioners:

No previous knowledge of AGI safety is required. If you want to watch a few of the Rob Miles videos, read the WaitButWhy posts, or the The Most Important Century summary that's great, but you can ask a question if you haven't.
Similarly, you do not need to try to find the answer yourself before asking a question (but if you want to test Stampy's in-browser tensorflow semantic search that might get you an answer quicker! - let us know how it goes).
Also feel free to ask questions that you're pretty sure you k

...
Growth of Publicly Available Genetic Sequencing Data
by jefftk — 2023-07-20T19:50:01.004Z

The largest source of publicly available genetic sequencing data is
the

Sequence
Read Archive (SRA), a joint project of the US (

NCBI),
Europe (

EBI),
and Japan (

DDBJ).
Most relevant funding agencies and journals require sequencing
data to be deposited in the SRA. I was curious how quickly it has
been growing, so I ran some queries.

The metadata for the SRA is available in
the cloud and we can access it through BigQuery. I ran:

SELECT EXTRACT(YEAR from releasedate),
EXTRACT(MONTH from releasedate),
SUM(mbases),
SUM(mbytes)
FROM `nih-sra-datastore.sra.metadata`
GROUP BY EXTRACT(YEAR from releasedate),
EXTRACT(MONTH from releasedate)

This gave me how much new data there was each month, in terms of both
genetic bases and (compressed) bytes on disk.

Note the logarithmic y-axis.

This is reminiscent of another chart, the cost to sequence 1M bases:

(This is a pretty amazing chart, with the huge drop around 2008 coming
from NGS
Sequencing)

We could combine these, to get a rough estimate for how much
money is being spent to sequence the data going into the SRA, but to
do this we need to know how long a delay there is between sequencing
and releasing: if it cost $400/Mb in 2007-10, $100/Mb in 2008-01, and
$15/MB in 2008-04, then which cost should we use for interpreting data
released in 2008-06? Here's a plot showing models 0-, 6-, 12-, and
24-month delays:

It looks like maybe ~9m is the initial delay, and with costs changing
more slowly in recent years it doesn't matter much for more recent
data.

Looking just at the last five years, after it has leveled out some, it
looks like a steady ~1.2e16 bases annually:

Comment via: facebook, mastodon
Progress links and tweets, 2023-07-20: “A goddess enthroned on a car”
by jasoncrawford — 2023-07-20T18:28:23.649Z
Link: https://rootsofprogress.org/links-and-tweets-2023-07-20

Opportunities

The Roots of Progress Blog-Building Intensive endorsed by @sapinker, @vpostrel, @johanknorberg, @CharlesCMann, @antonhowes, @s8mb, @kevin2kelly, @Marian_L_Tupy, @atrembath, @jposhaughnessy, @eric_is_weird, @CJHandmer, @srajagopalan, @s_r_constantin, @danielgolliher, @krishnanrohit, and @krisgulati
Progress studies meetup in Miami on Saturday, July 22
Progress studies reading group (via @jayendra_ram)
Hollywood for Intellectuals: Writing Big Ideas for the Big (and Small) Screen
University of Austin’s summer high school program (via @mbateman, who will be teaching there)
Sid Sijbrandij wants a postdoc to help him learn about incentive structures
Loyal (longevity startup) starting to look for a board member

Announcements

Elon Musk announces x.ai (via @pitdesi)
Claude 2 improves performance in coding, math and reasoning (via @AnthropicAI)
Major new AI alignment effort from OpenAI (via @janleike)
Fundraiser for an asylum-seeker who left Russia rather than fight Ukraine (organized by @Marian_L_Tupy)

Audio & video

Very good Kevin Esvelt talk on securing DNA synthesis against misuse
Panel discussion on fundamental research featuring me and others (via @PrimeMoversLab, recorded last October but just published)
Nick Touran (@whatisnuclear) has been scanning old film of nuclear reactors: here’s one about a reactor that powered a military radar station and one about a prototype nuclear cargo ship

Other links

A “protein printer” could “change biology as much as printers changed the news.” Related, “DNA printers are on the cusp of a big upgrade”
“The power to maintain is the power to improve.” Stewart Brand on auto maintenance
The prospects of molecular manufacturing (via @annasofialesiv)
Cleaning wool in classical times (via @NadyaWilliams81)
Brian Potter tells the story of titanium
Moralizing about consumption vs. economic reality (via @atrembath)
The Engineering and Technology History Wiki (via @Ben_Reinhardt)

Queries

Who have been the best research and development leaders in industry history?
What is the best writing about positive AI futures?
Which companies have incredibly thorough, detailed books written about them?
Foundational texts for a course on economic growth? (And some actual examples)
What happened to public trust in government in 1964?

Quotes

“A goddess enthroned on a car”: automobiles represented empowerment for women
A Ford factory as “a daring expression of the human

...
Boundary Placement Rebellion
by tailcalled — 2023-07-20T17:40:00.190Z

The idea for this post all started because I was confused about the concept of "narcissism". I learned about "narcissism" from reading differential psychology, where they often measure it by asking people whether they agree with statements such as:

I have a natural talent for influencing people
Modesty doesn't become me
I think I am a special person
I really like to be the center of attention

In practice, empirically this correlates with being an assertive, confident person, which didn't match the discourse about narcissism, which typically seemed to more be about domestic abuse or people's ideologies, and it also doesn't AFAIK match the way "narcissism" gets used clinically, since clinical narcissists don't score higher than average on psychometric Narcissism scales used in personality psychology.

Eventually, something clicked about what people were saying about narcissism. They were talking about a dynamic that occurs when someone feels morally entitled to violate a boundary. Such a person can exhibit signs of narcissism because they become focused on enforcing their own views on the boundary, and put a lot of effort into gaining power to continue violating it.

Still, since I'm not sure whether what I'm talking about is actually "narcissism", I'm going to use a new, more descriptive term to refer to my concept: Boundary Placement Rebellion.

Boundary Placement Rebellion comes up a lot in my experience. It's a key issue in AI safety, as well as in civil rights, psychology research, hierarchies, families and other areas of society. For good and for bad, I think Boundary Placement Rebellion sees far more symmetry between the sides than "narcissism" does.

An Example: AI Safety vs Capabilities

Common sense - at least among many right-wing techies - is that you have the right to work on whatever software projects you want. If people don't like your software, they can just not buy it. There may be exceptions when it comes to software whose purpose is to be used as a weapon, such as ransomware.[1] If new issues are discovered, then maybe we can update this norm, but we should also be careful to not strangle the tech industry with paperwork.

This is a boundary; you get sole decisive control over what you work on. AI capabilities research (as well as everything else in tech) makes a lot of use of this boundary, as it means that they can keep trying new things to push the s...
Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity
by zhanpeng_zhou — 2023-07-20T17:38:13.476Z
Link: https://openreview.net/pdf?id=vORUHrVEnH

Hi! I am excited to share our new work Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity [NeurIPS 2023 Accepted] with the LessWrong Community. In our new work, we introduce a stronger notion of linear connectivity, Layerwise Linear Feature Connectivity (LLFC), which says that the feature maps of every layer in different trained networks are also linearly connected.

Background

For years, despite the successes of modern deep neural networks, theoretical understanding of them still lags behind. While the loss functions used in deep learning are often regarded as complex black-box functions in high dimensions, it is believed that these functions, particularly the parts encountered in practical training trajectories, contain intricate benign structures that play a role in facilitating the effectiveness of gradient-based training.

One intriguing phenomenon discovered in recent work is Mode Connectivity [1, 2]: Different optima found by independent runs of gradient-based optimization are connected by a simple path in the parameter space, on which the loss or accuracy is nearly constant. More recently, an even stronger form of mode connectivity called Linear Mode Connectivity (LMC) (as shown in Definition 1) was discovered [3]. It said the networks that are jointly trained for a short number of epochs before going to independent training are linearly connected, referred as spawning method. With the empirical understanding, two networks (modes) that are linearly connected can be viewed as entering into the same basin in the loss landscape.

Surprisingly, [4] first demonstrate that two completely independently trained ResNet models (trained on CIFAR10) could be connected after accounting for permutation invariance. In particular, [4, 5] found that one can permute the weights in different layers while not changing the function computed by the network. Based on such permutation invariance, [4, 5] could align the neurons of two independently trained networks such that the two networks could be linearly connected, referred as permutation method. Therefore, [4, 5] conjectured that most SGD-solutions (modes) are all in one large basin in the loss landscape.

Layerwise Linear Feature Connectivity

The study of LMC is highly motivated due to its ability to unveil nontrivial structural properties of loss landscapes and training dynamics. On the othe...