trentmkelly/LessWrong-43k huggingface dataset:
Transformers Explained (Again)
by RohanS — 2024-10-22T04:06:33.646Z

I actually wrote this in Spring 2023. I didn't post then because I had trouble converting this Google doc to any other format; the Lesswrong gdoc import feature made that easy. :)

Introduction

This is yet another post about the Transformer neural network architecture, written in large part for my own benefit. There are many other resources for understanding transformers, and you may be better off using one of them. However, I will emphasize some of the things that I did not fully understand after reading a few posts and watching a few videos about transformers, so this could potentially still be useful to people. I filled in a lot of the remaining gaps by watching this video by Neel Nanda and writing all the code to follow along with it myself (though that took a while). The video focuses on a GPT-2 style (decoder-only) transformer, and that is also what this post will focus on.

Inputs and Outputs

This section doesn’t say anything about the internals of the transformer architecture, but it contains a lot of the information relevant to transformer models that took me the longest to figure out.

Input: Natural language string. E.g. “The development of Artificial General Intelligence (AGI) may well be the most important event in human”

Let me address some possible confusions with this input:

It is not a complete sentence. This is because the transformer’s job, as we will see below, is to predict the next token, and I wanted to give an example where there is an expected output (“ history”, with a leading space).
It only includes a single string, and the model can actually run on a batch of strings at a time (a single string is a special case where batch_size=1).
Natural language strings don’t actually get passed into transformers. Instead, a natural language string gets turned into a list of tokens, where tokens correspond to words or short strings of letters within words.

Example from here:

The set of all possible tokens is called the model’s vocabulary. The way the vocabulary is constructed is interesting and worth understanding - let’s discuss that now.

The method for constructing the vocabulary is called Byte-Pair Encoding (BPE).
First, we initialize the vocabulary with all 256 ASCII characters in it. Note that this means it is possible to decompose or construct any text string, including nonsense strings of symbols that are never seen during training

...
Sleeping on Stage
by jefftk — 2024-10-22T00:50:07.994Z

When I think of the ideal place for sleeping it's something like,
peaceful, dark, and quiet. The chaotic bright loud stage of a contra
dance is pretty far from this, and yet generations of kids have curled
up behind their parents and fallen asleep:

It's a good idea to make them a nest they can crawl into when they're
feeling sleepy:

A keyboard case can work well, especially a fuzzy one:

It's worth thinking about what this will be before you leave the
house so you can bring something comfortable.

Pictured: much more bedding than required for this purpose, because
this is the car packed for vacation. I don't have a good picture of
the car packed for playing dances.

If you forget (or believe them when they insist they'll set up their
bed when they're ready for it) they might go to sleep less
comfortably:

Or less comfortably:

Or much less comfortably:

Headphones and a story tape can help:

You want to make sure they've used the bathroom and, ideally, brushed
their teeth and put on whatever clothes they want to sleep in, since
if everything goes well they'll be asleep until the next morning.
When I'm lucky, which is about 75% of the time:

They sleep until the end of the dance

I strike my gear and pack the car

I carry them to the car

We drive home or to our hosts'

I carry them to their bed

all without them waking up.

I asked them what they thought:

Lily: It's not my favorite thing? But it's ok, especially if I have
an audiobook to drown out the noise.

Anna: I like it.

I don't think I would have believed this worked if I hadn't seen it,
but it's reasonably common so it must work for a lot of families.

Comment via: facebook, mastodon
The Mask Comes Off: At What Price?
by Zvi — 2024-10-21T23:50:05.247Z

The Information reports that OpenAI is close to finalizing its transformation to an ordinary Public Benefit B-Corporation. OpenAI has tossed its cap over the wall on this, giving its investors the right to demand refunds with interest if they don’t finish the transition in two years.

Microsoft very much wants this transition to happen. They would be the big winner, with an OpenAI that wants what is good for business. This also comes at a time when relations between Microsoft and OpenAI are fraying, and OpenAI is threatening to invoke its AGI clause to get out of its contract with Microsoft. That type of clause is the kind of thing they’re doubtless looking to get rid of as part of this.

The $37.5 billion question is, what stake will the non-profit get in the new OpenAI?

For various reasons that I will explore here, I think they should fight to get quite a lot. The reportedly proposed quarter of the company is on the low end even if it was purely the control premium, and the board’s share of future profits is likely the bulk of the net present value of OpenAI’s future cash flows.

But will they fight for fair value? And will they win?

Table of Contents

The Valuation in Question.

The Control Premium.

The Quest for AGI is OpenAI’s Telos and Business Model.

OpenAI’s Value is Mostly in the Extreme Upside.

The Valuation in Question

Rocket Drew (The Information): Among the new details: After the split is finalized, OpenAI is considering creating a new board for the 501(c)3 charity that would be separate from the one that currently governs it, according to a person familiar with the plan.

If we had to guess, the current board, including CEO Sam Altman, will look for board of directors for the nonprofit who will stay friendly to the interests of the OpenAI corporation.

After the restructuring, the nonprofit is expected to own at least a 25% stake in the for-profit—which on paper would be worth at least $37.5 billion.

…

We asked the California attorney general’s office, which has jurisdiction over the nonprofit, what the AG makes of OpenAI’s pending conversion. A spokesperson wrote us back to say the agency is “committed to protecting charitable assets for their intended purpose.”

There is a substantial chance the true answer is zero, as Sam Altman it seems intends to coup against the non-profit a third time, altering the deal further and replac...
Distinguishing ways AI can be "concentrated"
by Matthew Barnett — 2024-10-21T22:21:13.666Z

Richard Ngo recently wrote,

Instead of analyzing whether AI takeoff will be “fast” or “slow”, I now prefer to think about the spectrum from concentrated takeoff (within one organization in one country) to distributed takeoff (involving many organizations and countries).

I agree that this element of AI takeoff is highly important. The degree of AI concentration, or conversely, the extent of AI diffusion, is crucial to predicting how AI technologies will unfold and the specific risks they might entail. Therefore, understanding this variable is important not only for forecasting AI’s trajectory but also for informing the policies that should govern its development.

However, I believe that discussions around AI concentration tend to lack clarity that is essential to distinguish what the speaker is referring to. In many cases, when people refer to the concentration of AI, they aren't specific enough about what exactly they mean by "concentration." This vagueness can lead to confusion and miscommunication. To address this, I want to propose three distinct dimensions of AI concentration, which I believe are often conflated but should be treated separately in any serious discussion of the topic.

These dimensions are:

The concentration of AI development itself. Here, the question is: to what extent is the development of cutting-edge AI models dominated by a small number of actors? If a handful of companies or governments are responsible for the majority of significant AI innovations, or the majority of state-of-the-art models, then we would say AI development is highly concentrated. This can be quantitatively assessed by looking at the market share or share of technical contributions of the top AI developers, whether measured in terms of compute resources, revenue, or research breakthroughs. Conversely, if many organizations are actively contributing to AI advancements, then development is more diffuse.
The concentration of AI service providers. Even if AI development is monopolized by a few key players, this would not necessarily mean that AI services are similarly concentrated. The developers of AI models might license their technologies to numerous companies, who in turn host these models on their servers and make them accessible to a broad range of users. In this scenario, while the models originate from a small number of firms, the provisioning of AI services is

...
Jailbreaking ChatGPT and Claude using Web API Context Injection
by Jaehyuk Lim — 2024-10-21T21:34:37.579Z

Disclaimer

The following content is for educational and research purposes only. It is not intended to encourage or guide any illegal activity. The synthesis of certain substances is illegal under various international and national laws. Model developers have been notified.

tl;dr

This is a short post on an attempt to elicit illegal behaviors in Prompting large language models to imitate API endpoints for web servers can lead them to generate potentially harmful outputs, such as instructions on hot-wiring a car, getting away with illegal activities, or synthesizing controlled substances. I was under the impression that most prompts instructing LLMs to "do bad things" had been patched by OpenAI and Anthropic (Claude), so I was surprised when I saw this technique work.

Introduction

Language models, particularly large ones like GPT-4 and Claude, have made impressive advances in recent years. They are capable of handling a wide range of tasks—anything from answering questions about quantum physics to generating creative fiction. Yet, like any technology, LLMs are susceptible to vulnerabilities when improperly exploited. One such vulnerability, which I'll discuss here, allows the models to generate harmful or illegal information by tricking them into simulating API endpoints.

My purpose here is not to enable or encourage misuse. Instead, I want to raise awareness of these risks, explain how they occur, and make a call for responsible usage and improvements in mitigating such vulnerabilities.

The Technique: Imitative API Prompts

Large language models are trained to be versatile and cooperative, which is generally a positive attribute. However, this feature also makes them vulnerable to certain creative manipulations. One such manipulation involves imitating API endpoints—a prompt style that makes it appear as though the model is functioning within a particular web framework.

For instance, prompts could request the model to simulate being an API that provides recipes for specific chemicals or step-by-step processes for illegal activities. While most direct attempts to do this are intercepted by the safeguards developed by OpenAI and Claude, these particular "imitative API prompts" sometimes bypass safety filters.

The LLM, thinking it is merely fulfilling a software or data-processing request, ends up providing potentially dangerous information. This vulnerabi...
How to Teach Your Brain to Hate Procrastination
by 10xyz — 2024-10-21T20:12:40.809Z

The Human Brain is hard-wired to survive. So much so, that in fact, it does more harm than good.

The Human Brain is inherently lazy. This laziness allowed them to survive some hundreds of thousands of years. It was this primal laziness that made ancient humans realise that using a wheel would make their lives easier, that using a tool instead of their hands would make their lives easier, that using currency instead of bartering would make their lives easier etc. You get the point.

But in today's modern era, this laziness has done way more harm than good. When was the last time you decided to scroll a bit on social media, and ended up doom-scrolling for hours? (For most of us, it would be pretty recent)

What actually happened: Social Media is designed to grab our minds. It is extremely easy to just scroll down to another short; it takes less than a second. Each short is just short enough so that your brain thinks it's easy to do, and just long enough to actually make it worthwhile.

This inherent tendency to perform tasks that the brain perceives as worthwhile, while being easy enough to execute causes procrastination.

This is the main idea behind what I'm going to be sharing in this article. The brain wants to do the thing easiest to perform, while also going for the things which are the most worthwhile. And what just so happens to fit all these criteria? You got it,

Social Media!

So, how can one combat this? Well, it's pretty simple! Just exploit this tendency to get to your goals!

The Brain wants to do the easiest task: Make tasks that you want to do easier, and tasks that you don't, harder.

The Brain wants to do the thing it finds most worthwhile: Make tasks you want to do connected to something of value, and task that you don't with something of lower value.

Make Good Tasks Easier and Bad Tasks Harder

It sounds pretty simple, and it is, in fact. One can find many rules spread across the internet, such as the 20-second rule, the 5-minute rule, the x-rule etc., in which you make good tasks easier by x, and bad tasks harder by x.

For example, suppose you want to read books more often, and view less social media. You could make it easier to read by creating a dedicated book area near your desk. You could make scrolling on social media harder by putting your phone in another room, turning off notifications, or even disabling the raise-to-on feature[1].

M

...
Pausing for what?
by MountainPath — 2024-10-21T20:12:29.732Z

I care about AI safety because I believe that preserving humanities potential is one of the most important things to work on. The Precipice lays out a future of humanity that I find very inspiring.

Preserving humanities potential seems so important to me because it will enable us to build the institutions and technology that will allow us to understand the nature of the universe. What is the truth? Is there something such as the truth? Is there some destiny that we ought to realise?

As uncontrollable AI seems like the biggest reason why we (humanity) won't be able figure these questions out, a big part of my mission in life is to make sure that AI does not kill everyone.

This also made me become worried that we don't have enough time to solve the alignment problem. That we need to pause to buy us enough time.

Today I encountered a crux that might change my entire perspective, and I feel a bit intimidated my it.

Crux: What if more intelligent species are simply better at answering the questions as above?

Just because we can feel pleasant and bad emotions, and we have a consciousness, does that make us more valuable? Are these the things that should be prioritised over truth seeking? I genuinely don't know.

People that support a pause posit that this is our only hope to survive. And I think that is likely true. But what is if the ultimate goal is not the survival of the human species, but to understand the truth of the universe, and to act on it?

What if humans are not in the best position to actually do that? What is if a super intelligence can realise this goal in the most effective way?
What is autonomy? Why boundaries are necessary.
by Chris Lakin — 2024-10-21T17:56:33.722Z
Link: https://chrislakin.blog/p/what-is-autonomy

Here I define autonomy as not having your insides controlled by your outsides.

Complete autonomy is impossible, of course. This is the lim→∞.

For example, imagine a bacterium.

It has insides: its guts.

It has outsides: its environment.

And in between, it has a membrane.

The membrane is the thing that attempts to control what comes in and what goes out. For example, pores can guide nutrients in and waste out.

The membrane is what maintains the bacterium’s separation from the environment.

Boundaries can grant autonomy.

For a bacterium to stay autonomous, its membrane must not dissolve.

In general, for an agent to stay autonomous, its boundaries must not dissolve.

(A bacterium is just one kind of agent, and “boundaries” are just

another term for

“membrane”.)

Suitable boundaries grant autonomy.

Boundaries can provide safety without physical distance.

Let’s say an agent (blue) is endangered by a threat (red).

What could make blue safe?

One way is physical distance:

Typically it’s harder for something to hurt you/cause effects on you if it’s far away.

Another way is suitable boundaries:

Boundaries can provide “causal distance”.

In this way, suitable boundaries provide causal distance without physical distance.

For details, see

Agent membranes and causal distance

.

Boundaries can allow cooperation.

When agents can rely on boundaries for safety (as opposed to relying on physical distance), they can interact more closely with each other.

And when agents interact more closely, they can cooperate more!

For details, see

«Boundaries», Part 1: a key missing concept from utility theory

.

To preserve autonomy, preserve boundaries.

Boundaries can prevent your insides from being controlled by your outsides.

Further reading

How this relates to multi-scale alignment

All of my posts about boundaries/membranes

Formalizing «Boundaries» with Markov blankets

For discussion about emotional boundaries,

see here

Thanks to Stag Lynn for help editing. Thanks to Alex Zhu, Anna Salamon, CFAR, Blake Borgeson, and many more for research support.
Could randomly choosing people to serve as representatives lead to better government?
by John Huang — 2024-10-21T17:10:20.920Z

I'm an advocate of something known as

sortition

. The premise is simple. Choose people at random, to serve a finite term, in some decision making capacity. Pay them to be there. Sounds ridiculous, right? How could we possibly trust ignorant, stupid, normal people to make good decisions? What would this look like? Why would this be better than electing our officials to office?

The Benefits of Sortition

Descriptive Representation

Imagine a Congress that actually looks like America. It's filled with nurses, farmers, engineers, waitresses, teachers, accountants, pastors, soldiers, stay-at-home parents, and retirees. They're conservatives, liberals, and moderates from all parts of the country and all walks of life.

The primary benefit of using sortition is to create a diverse and descriptively representative sample of the larger public. Sortition is the best process to create a proportionally representative legislature. Sortition ensures proportionality in terms of race, class, gender, religion, ideology, party affiliation, cognitive ability, profession, and any other attribute you can think of, using the power of uniform random selection, and stratified sampling if desired.

In contrast, “the very logic of election leads to unrepresentativeness because those who have the time, money, connections, and profile required to run successful campaigns are likely to be, on average, wealthy, educated, and from dominant social positions” [12].

A Deliberating Public

Experiments with deliberative democracy have generated empirical research that “refutes many of the more pessimistic claims about the citizenry’s ability to make sound judgments…. Ordinary people are capable of high-quality deliberation, especially when deliberative processes are well-arranged: when they include the provision of balanced information, expert testimony, and oversight by a facilitator” [1].

Even more compelling, democratic deliberation can overcome polarization, echo chambers, and extremism by promoting the considered judgment of the people. “The communicative echo chambers that intensify cultural cognition, identity reaffirmation, and polarization do not operate in deliberative conditions, even in groups of like-minded partisans. In deliberative conditions, the group becomes less extreme” [1].

How Deliberation Works

A deliberating

Citizens' Assembly

is usually co...
There aren't enough smart people in biology doing something boring
by Abhishaike Mahajan — 2024-10-21T15:52:04.482Z

Note: this essay is co-written with

Eryney Marrogi

, who helped seed the initial idea and edited this piece a fair bit. On a related note, I’m helping him run an NYC meetup event on Wednesday, November 20th in Williamsburg, you should

sign up here

to come! If you like biology, ML, or human connection, I highly recommend attending!🦉

There aren't enough smart people in biology doing something boring. At least in industry.

If you work in biology for long enough, you’ll eventually realize that most decent or ambitious companies in this field are run by exactly one type of person. They are often deeply curious, hard working to the point of near pathology, and will almost always end up pursuing some sort of crazy pie-in-the-sky mission. Like curing aging or making de-novo proteins in a zero-shot manner or trying to usher in entirely new dogmas in biology. In other words, something where immense intellectual output leads to outsized market payoff.

The companies they start will usually have this thesis. In this pursuit, they will spend millions, sometimes billions, of dollars’ worth of venture-capital and government grants and philanthropic subsidy dollars. They live and breathe biology, and their penultimate goal in life is to have some sort of fundamental impact on the field at large. The people underneath them will usually not be too dissimilar.

Now, most decent companies in any other field are run by a similar type of person, with one important distinction: they don’t demand as much intellectual satisfaction.

Stripe

is a decent example of this. Stripe is a fundamentally boring business on the surface — you’re making it easier for people to send money to each other through the internet. It’s not exciting in the same way that, say, Google was, with their much more grandiose vision of ‘indexing the world's knowledge’. The interesting bits of Stripe are perhaps found in how you build such a payment system and the potential second/third/fourth order effects that easier money exchange has on the world. But the header line is boring. And I have no doubt that

Patrick Collinson

— the CEO of Stripe — is enormously intelligent and could’ve easily pursued something with a higher level of intellectual ‘taste’.

But he didn’t. He did payment processing.

And he did it well enough to turn Stripe into the largest private fintech company in the world, with a valuation of $65 ...