trentmkelly/LessWrong-43k huggingface dataset:
Mapping the semantic void: Strange goings-on in GPT embedding spaces
by mwatkins — 2023-12-14T13:10:22.691Z

TL;DR: GPT-J token embeddings inhabit a zone in their 4096-dimensional embedding space formed by the intersection of two hyperspherical shells. This is described, and then the remaining expanse of the embedding space is explored by using simple prompts to elicit definitions for non-token custom embedding vectors (so-called "nokens"). The embedding space is found to naturally stratify into hyperspherical shells around the mean token embedding (centroid), with noken definitions depending on distance-from-centroid and at various distance ranges involving a relatively small number of seemingly arbitrary topics (holes, small flat yellowish-white things, people who aren't Jews or members of the British royal family, ...) in a way which suggests a crude, and rather bizarre, ontology. Evidence that this phenomenon extends to GPT-3 embedding space is presented. No explanation for it is provided, instead suggestions are invited.

[Mapping the semantic void II: Above, below and between token embeddings]
[Mapping the semantic void III: Exploring neighbourhoods]

Work supported by the Long Term Future Fund.

GPT-J token embeddings

First, let's get familiar with GPT-J tokens and their embedding space.

GPT-J uses the same set of 50257 tokens as GPT-2 and GPT-3.[1]

A typical selection of GPT-2/3/J tokens and their indices. Note the mixture of full words, sub-word chunks, abbreviations, numerical and typographic strings, as well as the presence/absence of leading spaces and upper/lower case combinations.

These tokens are embedded in a 4096-dimensional space, so the whole picture is captured in a shape-[50257, 4096] tensor. (GPT-3 uses a 12888-dimensional embedding space, so its embeddings tensor has shape [50257, 12888].)

Each token's embedding vector was randomly initialised in the 4096-d space, and over the course of the model's training, the 50257 embedding vectors were incrementally displaced until they arrived at their final positions, as recorded in the embeddings tensor.

So where are they?

It turns out that (with a few curious outliers) they occupy a fuzzy hyperspherical shell centred at the origin with mean radius of about 2, as we can see from this histogram of their Euclidean norms:

As you can see, they lie in a very tight band, even the relatively few outliers lying not that far out.

Despite this, the mean embedding or "centroid", which we can think of as a kind of centre-o...
Categorical Organization in Memory: ChatGPT Organizes the 665 Topic Tags from My New Savanna Blog
by Bill Benzon — 2023-12-14T13:02:33.073Z

Cross posted from New Savanna.

I've just posted a new working paper. Title above, links, abstract, TOC, and two opening sections below:

Academia.edu: https://www.academia.edu/111354740/Categorical_Organization_in_Memory_ChatGPT_Organizes_the_665_Topic_Tags_from_My_New_Savanna_Blog
SSRN: https://ssrn.com/abstract=4663978
ReserchGate: https://www.researchgate.net/publication/376481426_Categorical_Organization_in_Memory_ChatGPT_Organizes_the_665_Topic_Tags_from_My_New_Savanna_Blog

Abstract: I gave ChatGPT three lists of topics for which it had to propose categories into which the topics could be sorted. Two lists were relatively short, 56 and 53 topics; I asked ChatGPT to propose six organizing categories for each. One list was much longer, 655 topics; I asked ChatGPT to propose 12 for categories for it. In all cases the proposed categories were reasonable. ChatGPT explained each proposed category either with a pair of sentences (the short lists), or with characterizing phrases (the long list). These characterizations were reasonable. In a further task, when asked to place topics under the proposed categories, ChatGPT placed many topics under the first two categories and very few under the last two. Though quite different in detail, this task has a rough formal similarity to generating a coherent story that is organized on three levels: 1) the whole story, 2) story segments, 3) sentences in story segments.

Contents

Organizing lists of categories into a coherent structure 1
The categories ChatGPT proposed 3
Formal similarity with story generation 5
What happens when ChatGPT lists tags under each category? 8
How would I have approached these tasks myself? 10
Propose categories to sort a short “top-level” list of 56 topics 10
Propose categories to sort a short arbitrary sub-list of 53 topics 14
Propose categories to sort the full list of 665 topics 16
Sort a short list and place the topics under the appropriate category 21

Organizing lists of categories into a coherent structure

Some months ago I decided to see how ChatGPT would react to some associative clusters I had made. Associative cluster? Simple, a list of words I created by free association on some particular topic, for example:

atoms, periodic table, bonds, compounds, elements, molecules, reaction, oxidation, acids and bases, reagents, alchemy, changing liquids from one color to another, distilling, condenser...
Moral Mountains
by Adam Zerner — 2023-12-14T10:40:06.179Z

Suppose that someone asks you to help them move in to their new apartment. Should you help them?

It depends, right?

Depends on what?

My answer: moral weights.

Suppose it's your brother who is the one who asked you. And suppose you really love your brother. You guys are really close. You care about him just as much as you care about yourself. You assign a "moral weight" of 1-to-1. Suppose that helping him move will bring him 10 utilons. If so, you should help him move as long it costs you less than 10 utilons.[1]

Now let's suppose that it's your cousin instead of your brother. You like your cousin, but not as much as your brother. You care about yourself four times as much as you care about your cousin. Let's call this a "moral weight" of 4-to-1, or just "4". In this situation, you should help him move as long as it costs you less than 2.5 utilons.

Suppose it's a medium-good friend of yours. You care about yourself 10x as much as you care about this friend. Here, the breakeven point would be when it costs you 1 utilon to help your friend move.[2]

What about if it's a coworker who is cool but also isn't your favorite person in the world. You assign a moral weight to them of 100. The breakeven point is 0.1 utilons.

Framing things this way makes me think back to the following diagram from 10 Types of Odd Friendships You're Probably Part Of:

In that blog post, Tim uses the diagrams to represent how close you are to various people. I think it can also be used to represent how much moral weight you assign to various people.[3] For example, maybe Tier 1 includes people for whom you'd assign moral weights up to 5, Tier 2 up to 30, and Tier 3 up to 1,000.

Different people have different "moral mountains". People like Near-and-Dear Nick care a lot about the people who are close to them, but below that things get a little sparse.

For others like Individualistic Ian, there isn't anyone who they care about nearly as much as they care about themself, but there are still a decent amount of people in Tier 2 and Tier 3.

Then you've got your Selfish Shelby's who pretty much just think about themself.

If I were able to draw like Tim Urban I would include more examples, but unfortunately I don't have that skill. But for fun, let's just briefly think about some other examples, and ask some other questions.

What would Peter Singer's mountain look like? Is everyone in Tier

...
Update on Chinese IQ-related gene panels
by Lao Mein — 2023-12-14T10:12:21.212Z

It turns out that Chinese 23-and-me-esque gene panel already include intelligence markers! For example, 23mofang uses Interleukin 3 as a proxy for brain volume, citing this paper. It's... only significant in women. Not a good sign. But the cited study notes a Danish gene-correlation study that included brain scans, from which they obtained brain volume. Apparently, the correlation is true across races.

In any case, I've been reaching out to past coworkers, and they agree with my assessment that a polygenic database for the purpose of embryo selection would be easy and mostly cheap to do. Being Chinese, we of course have no issues with including factors like eye color, height, intelligence, ect. However, I have several questions before I proceed further.

What is the overall demand? How many customers would be interested? What specific traits are parents most interested in? Funding is the single biggest obstacle I face, and I will be applying for AstralCodexTen funding. If enough interest is displayed or if I get charitable funding, my gut feeling is that I can offer analysis for <$100 per genome, maybe even <$10.
Would Westerners accept Chinese IQ/educational attainment metrics? The easiest metric to use would be GaoKao scores, but would that be legible to Westerners? Would ratings of attended college be acceptable? What about the applicability of the results across different races?
Publications/computer code. Would I need to publish a paper to be considered legitimate?
Reputational concerns. What Western institutions should I be expected to be blacklisted from because of this?

I have previously published genome-wide association studies regarding cancer outcomes, so I can actually do the analysis and write-up myself. I just need the raw data, which I should be able to obtain by paying a sequencing company with a gene bank to call back their past clients and ask for GaoKao scores.

I would really appreciate it if anyone can direct me towards a source of European ancestry genomic data with labeled donor information (education, height, ect).

Update: For some reason, I didn't consider using previous research like the n=1.1 million educational attainment study. It seems... I can just do everything by myself? I'll sleep on it, but it is certainly interesting.
Red Line Ashmont Train is Now Approaching
by jefftk — 2023-12-14T02:50:11.382Z

The MBTA has now fully fixed the sign issue I was

complaining about!
It's great:

For years I grumbled inside whenever I was approaching a station and
saw the digital signs switch to "Attention passengers, the next red
line train". This could signal any of:

25%: My train is coming, I can catch it if I hustle.

25%: I'm missing my train.

50%: A train I don't care about is doing something.

If I waited the sign would switch to, for example, "to Ashmont is now
approaching", but time when I'm trying to decide whether to hurry to
catch a train is unusually precious!

The issue was that they were displaying a transcription
of the audio announcement instead of tailoring the textual
notification to the medium. Several months ago they fixed the arriving version, and
recently they fixed the "approaching" version as well. Yay!

Comment via: facebook
Various AI doom pathways (and how likely they are)
by Logan Zoellner — 2023-12-14T00:54:09.424Z
Link: https://midwitalignment.substack.com/p/various-ai-doom-pathways-and-how

The current discourse on AI extinction appears highly polarized. Consider for example, this survey.

If disagreements about AI causing extinction were caused by small disagreements about various probabilities, we would expect answers to follow a normal distribution. A strong bi-modal distribution like this suggests that people are using fundamentally different methods to estimate P(Doom), rather than disagreeing on minor details.

One possible source of disagreement is the path whereby AI causes human extinction. In my opinion, there are a few major pathways that people generally consider when thinking about P(doom).

Here are various AI doom pathways (and how likely they are):

Foom

The theory: Foom is the theory that once AI passes a “critical threshold”, it will rapidly self-improve quickly increasing in intelligence and power.

What a typical story looks like: One day, an AI researcher working in his basement invents a new algorithm for self-improving AI. The AI rewrites itself and begins self-propagating on the internet. Shortly thereafter (minutes or at most days), the AI invents nanotechnology which it uses to disassemble the earth into a Dyson Swarm murdering all humans in the process.

Major proponents: Eliezer Yudkowsky

How likely it is to happen: Extremely unlikely. All evidence currently suggests that we are in a slow takeoff and the binding constraint on AI is available compute, not the existence of a special self-improving algorithm.

Terminator

The theory: AI might simply ignore directions from its human creators and decide to take over the world and kill us all.

What a typical story looks like: The Skynet funding bill is passed.The system goes on-line on August 4, 1997. Human decisions are removed from strategic defense. Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.M. Eastern time, August 29. In a panic, they try to pull the plug.

Major proponents: Every sci-fi movie about robots ever.

How likely it is to happen: Very unlikely. That’s not how computers work.

Alignment Failure

The theory: AI is built in a way that appears superficially good but results in a bad outcome due to subtly misstated goals.

What a typical story looks like: The experts at Google DeepMind were confident that AlphaTwo was perfect. After many tests, they determined that it was scientifically programmed to maximized human happine...
Are There Examples of Overhang for Other Technologies?
by Jeffrey Heninger — 2023-12-13T21:48:08.954Z
Link: https://blog.aiimpacts.org/p/are-there-examples-of-overhang-for

TL;DR: No.

What Do I Mean By ‘Overhang’?

Hardware Overhang for AI

One major concern about pausing AI development, from a purely safety perspective, is the possibility of hardware overhang.[1] Here is the concern as I understand it:

Suppose a policy were put in place tomorrow that banned all progress in AI capabilities anywhere in the world for the next five years.[2] Afterwards, the ban would be completely lifted.

Hardware would continue to progress during this AI pause. Immediately after the pause ended, it would be possible to train new AI systems using significantly more compute than was previously possible, taking advantage of the improved hardware. There would be a period of extremely rapid growth, or perhaps a discontinuity,[3] until the capabilities returned to their previous trend. Figure 1 shows a sketch of what we might expect progress to look like.

Figure 1: What AI progress might look like if there were a temporary pause in capabilities progress. The ‘overhang’ is the difference between what AI capabilities currently are as a result of the pause and what AI capabilities could be if the pause had never been enacted, or were completely lifted.

It might be worse for safety to have a pause followed by extremely rapid growth in capabilities than to have steady growth in capabilities over the entire time frame. AI safety researchers would have less time to work with cutting edge models. During the pause, society would have less time to become accustomed to a given level of capabilities before new capabilities appeared, and society might continue to lag behind for some time afterwards.

If we knew that there would be catch-up growth after a pause, it might be better to not pause AI capabilities research now and instead hope that AI remains compute constrained so progress is as smooth as possible.

We do not know if there would be extremely rapid growth after a pause. To better understand how likely hardware overhang would be, I tried to find examples of hardware-overhang-like-things for other technologies.

Overhang for Other Technologies

Many technologies have an extremely important input - like GPUs/TPUs for AI, or engines for vehicles, or steel for large structures. Progress for these technologies can either come from improvements in the design of the technology itself or it can come from progress in the input which makes it easier to improve ...
Is being sexy for your homies?
by Valentine — 2023-12-13T20:37:02.043Z

Epistemic status: Speculation. An unholy union of evo psych, introspection, random stuff I happen to observe & hear about, and thinking. Done on a highly charged topic. Caveat emptor!

Most of my life, whenever I'd felt sexually unwanted, I'd start planning to get fit.

Specifically to shape my body so it looks hot. Like the muscly guys I'd see in action films.

This choice is a little odd. In close to every context I've listened to, I hear women say that some muscle tone on a guy is nice and abs are a plus, but big muscles are gross — and all of that is utterly overwhelmed by other factors anyway.

It also didn't match up with whom I'd see women actually dating.

But all of that just… didn't affect my desire?

There's a related bit of dating advice for guys. "Bro, do you even lift?" Depending on the context, the dudebros giving this advice might mention how you shouldn't listen to women about what they say they want. Because (a) the women don't really know and (b) they have reason to hide the truth anyway.

But… I mean… there's an experience here that's common enough to be a meme:

The more I connect the puzzle pieces, the weirder this looks at first.

For instance, my impression is that there is a kind of male physicality that does tend to be attractive for women. But it's mostly not about body shape (other than height). It's about functionality. Actual strength matters more than muscle size for instance. Coordination and physical competence are often turn-ons. Building stuff that's hard to build, and doing it with physical grace? Yeah.

So you'd think the ideal form of physical training for a guy to attract a woman might be things like mobility training plus some kind of skill practice like dance or woodworking.

But guys mostly be like "Nah."

(I mean, some go for it. I loved dance and martial arts, and I really tried to get into parkour, and these days I play with qigong & acrobatics. But the reason for these activities wasn't (& isn't) to attract a mate. It was (and is) mostly because I find them fun.)

But gosh, getting big does seem to attract men!

I mean, literally this seems to happen in gay contexts I think? But even setting aside sexual attraction, there's something about getting other men's "Looking big, king" that somehow… matters.

And if you take the feminist thing about male gaze seriously, that'd suggest that the physique of action heroes and comic bo...
How bad is chlorinated water?
by bhauth — 2023-12-13T18:00:12.640Z
Link: https://www.bhauth.com/blog/biology/chlorine.html

chlorine disinfection

Today, most water distributed to people for drinking is chlorinated. Bleach is also widely used for disinfection of surfaces.

The exact ways chlorine kills microbes aren't fully understood, but the main reasons are believed to be:

oxidation of thiols of enzymes

ring chlorination of amino acids

direct DNA damage

Obviously, high levels of chlorine are bad for humans. Chlorine gas was used as a chemical weapon in WW1, and drinking bleach can kill people. As for longer exposure to lower levels, studies have found associations between lung damage and use of indoor swimming pools, but the extent to which harmful effects of chlorine have thresholds from saturation of enzymes is still unclear.

Dietary studies are notoriously hard to get good results from, and studying chlorinated water has similar issues. Studies have concluded that, eg, over a few weeks, chlorinated water doesn't affect lipid metabolism. But is that what you'd expect to see? If there were effects, what would they be?

effects of ingested chlorine

Engineers try to minimize levels of some compounds in water that can react with chlorine to produce toxic substances, such as chloramines and chloroform. But...there are organic compounds in the stomach. What about reactions of chlorine after it's consumed?

Stomachs are acidic. That means amines are mostly protonated and unlikely to react, but other chlorination reactions are catalyzed. My understanding is that the main types of chlorine reaction in stomachs are:

oxidation of thiols (this doesn't concern me much)

phenol chlorination (eg 3-chlorotyrosine production)

tryptophan oxidation

double bond oxidation to halohydrins

Chlorotyrosine production happening is intuitive, and it's been validated by some rat studies. But the topic of reactions of chlorine in stomachs hasn't been studied very much in general.

What happens to chlorotyrosine and halohydrins afterwards?

In cells, aliphatic chlorinated compounds tend to have chlorine replaced with a ketone group by enzymes. For example, dichloromethane becomes formyl chloride which decomposes to carbon monoxide and HCl, which are less toxic than products from other chloromethanes, making it the least toxic of them. Obviously it's also possible for halocarbons to react spontaneously with amines before an enzyme gets to them; that's less likely with chlorine than bromine, but any amount is still...
Suggestions for net positive LLM research
by Cole Wyeth — 2023-12-13T17:29:11.666Z

I am starting a PhD in computer science, focusing on agent foundations so far, which is great. I intend to continue devoting at least half my time to agent foundations.

However, for several reasons, it seems to be important for me to do some applied work, particularly with LLMs:

I believe I'm otherwise pretty well positioned to get an impactful job at Google DeepMind, but apparently some impressive machine learning engineering creds are necessary to get on even the safety team currently.
My PhD supervisor is pushing for me to work on LLMs, though he seems to be pretty flexible about the details.
As much fun as math is, I also value developing my practical skills (in fact, half the fun of learning things is becoming more powerful in the real world).
LLM experts seem likely to be in high demand right now, though I am not sure how long that will last.

Now, I've spent the last couple of years mostly studying AIXI and its foundations. I'm pretty comfortable with standard deep learning algorithms and libraries and I have some industry experience with machine learning engineering, but I am not an expert on NLP, LLMs, or prosiac alignment. Therefore, I am looking for suggestions from the community about LLM related research projects that would satisfy as many as possible of the following criteria:

Is not directly focused on improving frontier model capabilities (for ethical reasons; though my timelines seem to be longer than the average lesswronger, I'm not able to accept the risk that I am wrong).
Produces mundane utility. I find it much more fulfilling to work on things that I can see becoming useful to people, and I also want a measure of my success which is as concrete as possible.
Contributes to prosiac alignment. It would be particularly nice if the experimental/engineering work involved is likely to inform my ideas for mathematical alignment research.
Machine learning engineering/research creds.

Any suggestions are appreciated. I may also link this question to a Manifold market in the future (probably "conditional on working full time at DeepMind within 18 months of graduation, which areas of research did my PhD thesis include") or something along those lines. Thanks!