trentmkelly/LessWrong-43k huggingface dataset:
A Bi-Modal Brain Model
by Johannes C. Mayer — 2024-05-22T20:10:08.919Z

When I am programming, writing, reading, browsing the internet, watching a movie, or playing a game, my brain is in a different mode of operation compared to when I am sitting in an empty room with nothing to do.

In the empty room, my brain will continuously generate fragments of language and other thoughts. In general, thoughts are either shaped like a sensory input channel, e.g. visual, auditory, touch, etc. or are conceptual thoughts. When doing any of the activities in the first list my brain won't generate these thoughts.

This leads to the common failure mode of being so absorbed in an activity that you don't even notice anymore what you are doing. Reflective thoughts like "Is what I am doing right now a good thing to do?" seem to generate much too infrequently by default. And even when such thoughts are generated it is much too easy to ignore them. It's common to get sucked back into the non-thought generation mode of operation within seconds.

Walks

This model provides an explanation for why I find walks so useful. You literally force yourself (though it does not feel like it) to inhabit the reflective state of mind. Most engrossing activities require a physical device like a computer, book, or notebook, and I usually don't have such devices at hand during a walk.

TAPs

I now want to try the following strategy (not sure how well it works yet). Imagine I am programming something. Usually, it is easy to notice when you have correctly implemented a function. E.g. you might run some tests and now they all pass. This is an easy-to-recognize event, which usually also presents a good point to reflect, as now you are in between tasks. So we can use this to set up a TAP.

For different activities, similar TAPs can be created. E.g. each time you add a new heading when writing an article.

Completing a function is a very generic trigger. I expect that most of the time when this trigger fires you will conclude "Yes actually just implementing the next function is best." I still think it is a good trigger to train, simply because it is so simple. But there are better ones.

It very often happens that I am confused, and notice that I am confused but don't take appropriate action. E.g. I know that trying to explain th...
Offering service as a sensayer for simulationist-adjacent beliefs.
by mako yass — 2024-05-22T18:52:05.576Z

A Sensayer is[1] a specialist in the private discussion of religion[2] in small groups, or one on one, a bit like a councilor with theological acumen. Many of us, believe it or not, harbor private beliefs. Sometimes those beliefs get heavy, and they can often benefit a lot from a second pair of eyes.

Most religions, in our world, do not need sensayers, as they can meet in groups and discuss their beliefs openly and there are usually healthy inter-faith dialogs, simulationism is different. It's rougher. It has no churches. It's not immediately obvious that it instructs any actions, given that it claims non-intervention, so its adherents are naturalists, and most think of themselves as atheists. It has no relationship with other faiths and when it confesses publicly it only receives chides from within and without ("clearly you're just trying to fill a void.") Some simulationist concepts, I've found, are easily misused, cynical, troubling or depersonalizing, the believer would not wish them upon anyone else. Nowhere in simulationist thought is there an obligation to proselytize (in some cases it demands the opposite), so I don't expect these things to change. So, a simulationist sometimes needs a sensayer. Myself included.

I've kind of had this role before. I approached a troubled friend and came to understand their ideas. Many of the ideas we found were beautiful, and a bit healing. There was never a reason to publish any of it and I mostly haven't. So it goes. I continue to stare into the abyss. And I often need someone to talk to, about that. At least twice I've been visited with thoughts I couldn't carry alone and also couldn't share with anyone around me, I was extremely grateful to have distant philosophically inclined intimates who I could summon into my cave and ask them to help untangling the thoughts.
Not everyone has that.

So I'm offering an open ear. You can DM me about this kind of thing. I can hold it all. I've seen the worst of it. And it stops with me, no exceptions.
I'll alter this paragraph if I ever feel that I can't do this any more. So if you've found it, I'm still here.

^

The term comes from Terra Ignota, a science fiction series of four books, set in a mostly okay future where, among other things, religion has become taboo. The author is a historian, and a lot of it is about geopolitical posturing, but she is also a historian who watch

...
Do Not Mess With Scarlett Johansson
by Zvi — 2024-05-22T15:10:03.215Z

I repeat. Do not mess with Scarlett Johansson.

You would think her movies, and her suit against Disney, would make this obvious.

Apparently not so.

Andrej Karpathy (co-founder OpenAI, departed earlier), May 14: The killer app of LLMs is Scarlett Johansson. You all thought it was math or something.

You see, there was this voice they created for GPT-4o, called ‘Sky.’

People noticed it sounded suspiciously like Scarlett Johansson, who voiced the AI in the movie Her, which Sam Altman says is his favorite movie of all time, which he says inspired OpenAI ‘more than a little bit,’ and then he tweeted “Her” on its own right before the GPT-4o presentation, and which was the comparison point for many people reviewing the GPT-4o debut?

Quite the Coincidence

I mean, surely that couldn’t have been intentional.

Oh, no.

Kylie Robison: I asked Mira Mutari about Scarlett Johansson-type voice in today’s demo of GPT-4o. She clarified it’s not designed to mimic her, and said someone in the audience asked this exact same question!

Kylie Robison in Verge (May 13): Title: ChatGPT will be able to talk to you like Scarlett Johansson in Her.

OpenAI reports on how it created and selected its five selected GPT-4o voices.

OpenAI: We support the creative community and worked closely with the voice acting industry to ensure we took the right steps to cast ChatGPT’s voices. Each actor receives compensation above top-of-market rates, and this will continue for as long as their voices are used in our products.

We believe that AI voices should not deliberately mimic a celebrity’s distinctive voice—Sky’s voice is not an imitation of Scarlett Johansson but belongs to a different professional actress using her own natural speaking voice. To protect their privacy, we cannot share the names of our voice talents.

…

Looking ahead, you can expect even more options as we plan to introduce additional voices in ChatGPT to better match the diverse interests and preferences of users.

Jessica Taylor: My “Sky’s voice is not an imitation of Scarlett Johansson” T-shirt has people asking a lot of questions already answered by my shirt.

OpenAI: We’ve heard questions about how we chose the voices in ChatGPT, especially Sky. We are working to pause the use of Sky while we address them.

Variety: Altman said in an interview last year that “Her” is his favorite movie.

Variety: OpenAI Suspen

...
How Multiverse Theory dissolves Quantum inexplicability 
by mrdlm — 2024-05-22T14:55:28.592Z

This is a link post for https://viderationale.blog/2024/05/04/quantum-path-to-the-multiverse/
Much of the arguments here comes from David Deutsch's two books The Beginning of Infinity and The Fabric of Reality. I try to explain parts of it in more detail and hopefully they make sense.
Central claim is that the problems posed by the Copenhagen Interpretation, crisply laid out in the Schrodinger's Cat thought experiment, disappear if you view it through the lens of David Deutsch's modification of Everettian Quantum Mechanics. It explains wave-particle duality, interference phenomenon, and probabilistic outcomes of measurement, all completely physically.
Procedural Executive Function, Part 3
by DaystarEld — 2024-05-22T11:58:53.031Z

The Off Road project has since been folded into Rethink Wellbeing, but I’ve continued working to better understand and treat Executive Dysfunction. You can read more about the project’s origins here.

If you haven't, I suggest reading the start of my overview and exploration of Executive Function. Parts 1 and 2 of Procedural Executive Function can be found here and here.

TL;DR - Organization is the system of habits and external aids that minimizes effort for accomplishing work.

Working Memory is your capacity for holding information in your attention at once.

Flexible Thinking is your ability to creatively problem solve and avoid getting stuck.

Each helps you maintain progress on your goals, and can help maintain flow by reducing difficulty in solving problems, which is one of the main bottlenecks for productive work.

As we begin the final(?) installment of the series, I hope it’s clear now how the many parts of Executive Function work together to take someone from an initial notion about what they “want to do,” to actually doing it.

We can generally put the problems our executive function faces into two buckets: things that prevent us from starting something, and things that cause us to stop doing something.

The first post, covering Planning, Prioritizing and Task Initiation, addressed things in the first bucket. The second post’s contents, Self Monitoring, Impulse Control, and Emotional Control, address things that can be found in both buckets. And this post’s topics, Organization, Working Memory, and Flexible Thinking, largely have to do with things in the second bucket.

But this frame falls apart, a bit, if we zoom in to each step of the path. Do we ever “write an essay?” Or do we write sentences that make up an essay? Each sentence is of course itself made up of words, and each word is made up of letters, but when people say “I have writer’s block,” they don’t normally mean they’re blocked at the level of “I don’t know how to spell a word.” Sometimes they mean “I don’t know the best word to put here.” But often they mean something like “I don’t know what the next sentence should be,” as a subpart of “I don’t know what the next idea to start exploring is,” or “I don’t know how to best explain this idea.”

Again, things that cause us to “stop” doing something are often the same sorts that prevent us from starting the next bit.

As mentioned in Part 1, the t...
Cicadas, Anthropic, and the bilateral alignment problem
by kromem — 2024-05-22T11:09:56.469Z

There have been a number of responses to today's Anthropic interpretability research, and while I think there were a number of salient points, there may be a degree of specialization blindness going on in contextualizing the work in the broader picture of alignment goals.

Alignment as a problem domain is not unilateral.

Most discussions I see on here are about alignment are focused on answering the question of roughly "how can we align future AGI to not be Skynet?" It's a great question. Perhaps more importantly, it's an interesting question.
It involves cross-discipline thinking intersecting an emerging research front channeling Jesse Ventura in Predator: "I ain't got time to peer review." Preprint after preprint move forward our understanding and while the rest of academia struggles under the burden of improper influences on peer review and a replication crisis, this field is one where peer reviews effectively are just replication.
So yes, today's research from Anthropic shouldn't be too surprising for anyone who has been paying the least bit of attention to emerging research in the area. Personally, I expected much of what was shown today by the time I finished reading Li et al. Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task (2023), and was even more sure of it after @Neel Nanda replicated the work with additional insight (with even more replications to follow). Of course a modern LLM with exponentially more parameters fed an exponentially larger broad data set was going to be modeling nuanced abstractions.
As @Seth Herd said in their post on the work:

Presumably, the existence of such features will surprise nobody who's used and thought about large language models. It is difficult to imagine how they would do what they do without using representations of subtle and abstract concepts.

But let's take a step back, and consider: some cicadas emerge every 17 years.

That's a pretty long time. It's also the average amount of time that it's historically taken the average practicing doctor to have incorporated emerging clinical trial research.
It's very easy when in tune with a specialized area of expertise to lose touch with how people outside the area (even within the same general domain) might understand it. It's like the classic xkcd:

"Average Familiarity"

I'm not even talking about the average user of ChatGPT. I've seen tenured C...
Announcing Human-aligned AI Summer School
by Jan_Kulveit — 2024-05-22T08:55:10.839Z
Link: https://humanaligned.ai/

The fourth Human-aligned AI Summer School will be held in Prague from 17th to 20th July 2024. We will meet for four intensive days of talks, workshops, and discussions covering latest trends in AI alignment research and broader framings of AI alignment research.

Apply now

, applications are evaluated on a rolling basis.

The intended audience of the school are people interested in learning more about the AI alignment topics, PhD students, researchers working in ML/AI outside academia, and talented students.

Format of the school

The school is focused on teaching and exploring approaches and frameworks, less on presentation of the latest research results. The content of the school is mostly technical – it is assumed the attendees understand current ML approaches and some of the underlying theoretical frameworks.

This year, the school will cover these main topics:

Overview of the alignment problem and current approaches.
Alignment of large language models: RLHF, DPO and beyond. Methods used to align current large language models and their shortcomings.
Evaluating and measuring AI systems: How to understand and oversee current AI systems on the behavioral level.
Interpretability and the science of deep learning: What's going on inside of the models?
AI alignment theory: While 'prosaic' approaches to alignment focus on current systems, theory aims for deeper understanding and better generalizability.
Alignment in the context of complex systems and multi-agent settings: What should the AI be aligned to? In most realistic settings, we can expect there are multiple stakeholders and many interacting AI systems; any solutions to alignment problem need to solve multi-agent settings.

The school consists of lectures and topical series, focused smaller-group workshops and discussions, expert panels, and opportunities for networking, project brainstorming and informal discussions.
Detailed program of the school will be announced shortly before the event. See below for a program outline and e.g.

the program of the previous school

for an illustration of the program content and structure.

Confirmed speakers

Stephen Casper – Algorithmic Alignment Group, MIT.
Stanislav Fort – Google DeepMind.
Jesse Hoogland – Timaeus.
Jan Kulveit – Alignment of Complex Systems, Charles University.
Mary Phuong – Google DeepMind.
Deger Turan – AI Objectives Institute and Metaculus.
Vikrant Varma – Google D

...
"Which chains-of-thought was that faster than?"
by Emrik — 2024-05-22T08:21:00.269Z

Here's some good advice from Eliezer:

TAP: "How could I have thought that faster?"

WHEN[1] you complete a chain-of-thought
THEN ask yourself, "how could I have thought that faster?"

I really like this heuristic, and it's already paid its rent several times over for me. Most recently today, so I'll share the (slightly edited) cognitive trace of it as an example:

Example: To find the inverse of something, trace the chain forward a few times first

I was in the context of having just asked myself "what's the set of functions which have this function as its derivative?"
This is of course its integral, but I didn't want to use cached abstractions, and instead sought to get a generalized view of the landscape from first-principles.
For about ~10 seconds, I tried to hold the function f in my mind while trying to directly generate the integral landscape from it.
This seemed awfwly inefficient, so I changed tack: I already know some specific functions whose derivatives equal f, so I held those as the proximal thing in my mind while retracing the cognitive steps involved in their derivation.
After making those steps more salient in the forward direction (integral→derivative), it was easier to retrace the path in the opposite direction.
And once the derivative→integral trace was salient for a few examples, it was easier to generalize from the examples to produce the landscape of all the integrals.
There are multiple takeaways here, but one is:

"If you struggle to generalize something, find a way to generate specific examples first, then generalize from the examples."

TAP: "Which chains-of-thought was that faster than?"

Imo, more important than asking "how could I have thought that faster?" is the inverse heuristic:

WHEN you complete a good chain-of-thought
THEN ask yourself, "which chains-of-thought was that faster than?"

Although, ideally, I wouldn't scope the trigger to every time you complete a thought, since that overburdens the general cue. Instead, maybe limit it to those times when you have an especially clear trace of it AND you have a hunch that something about it was unusually good.

WHEN you complete a good chain of thought

AND you have its trace in short-term memory
AND you hunch that something about it was unusually effective

THEN ask yourself, "which chains-of-thought was that faster than?"

Example: Sketching out my thoughts with pen-and-paper

Yesterday I

...
Each Llama3-8b text uses a different "random" subspace of the activation space
by tailcalled — 2024-05-22T07:31:32.764Z

This is kind of a null result (or WIP research) I got with a few days of fiddling, so don't get too excited. Also, because it's a null result, it's always conceivable that there's just some slight change in the approach which could suddenly flip it to get a real result. More on that in the "Discussion" section. I would expect my findings to generalize somewhat beyond this approach on Llama-3, but it's unclear how much, as I haven't tested it more broadly yet, so I'm going to limit my claims to this approach on this model.

GPT residual stream

Llama3-8b is a generative pretrained transformer released by Meta. In order to make it easier to train, it uses residual connections, which basically means that the core of the network consists of layers of the form:

output = input + f(input)

This is called the "residual stream" of the network.

A transformer is a sequence-to-sequence model, and so each of these variables is really a sequence of vectors, one for each token in the processing. I just concatenate all these vectors into a single matrix M:RLT×H, where L is the number of layers (32, I think), T is the number of tokens (depends on the prompt and the amount of tokens you request to generate), and H is the number of hidden dimensions (4096).

Singular vector decomposition

For any matrix M:RA×B, we have M=USV where U and V are rotation matrices and S is a diagonal matrix. The diagonal of S is called the singular values of M, and by convention one lists them in descending order[1].

I use it as a matrix approximation method by taking the diagonal matrix C(n) which contains n ones along the first part of the diagonal, and zeroes along the rest, as then USC(n)V yields an optimal n-rank approximation to M.

Since the U and V just consists of unit vectors, the only thing that moderates the magnitude of the matrix is S, so one can learn a lot about how good approximations of different ranks are just by plotting S:

The diagonal of S plotted for the M matrix assembled from the vectors for the text below.

Here, the [prompt] and generation I'm plotting the results for is:

[I believe the meaning of life is] to be happy. It is a simple concept, but it is very difficult to achieve. The only way to achieve it is to follow your heart. It is t

...
ARIA's Safeguarded AI grant program is accepting applications for Technical Area 1.1 until May 28th
by Brendon_Wong — 2024-05-22T06:54:55.206Z
Link: https://www.aria.org.uk/wp-content/uploads/2024/04/ARIA-Safeguarded-AI-TA1.1-Theory-Call-for-proposals.pdf

Note: I am completely unaffiliated with ARIA. I figured I'd post this since applications are closing soon and I didn't see anyone post about this.

My Takeaways:

ARIA is funding the development of Safeguarded AI which is an update to and specific implementation of davidad's Open Agency Architecture.
This grant round is for Technical Area (TA) 1.1, the first phase of developing Safeguarded AI.

TA1.1 involves developing mathematical and CS "syntax" for "language(s) that can be used by teams of humans (and, later, AI systems) to define 'world models', probabilistic specifications, neural network controllers, and proof certificates" for the ultimate purpose of "defin[ing] safety" to AI systems to ensure that they "are not taking actions that would lead to catastrophic outcomes."

Grant details:

This round is expected to distribute up to £3.5M across 10–16 applicants. The round opened on April 11th, 2024 and will close on May 28th, 2024 (at "12:00 BST").
Funding is primarily intended for UK entities (e.g. a person, company, nonprofit, or university), but non-UK entities can qualify if they can boost the impact of programs in the UK.
This grant round is focused on supporting open source conceptual/research work, but will also fund people to communicate concepts in TA1.1 to broader audiences.

The proposal mentions funding learning materials like blog posts, tutorials, and educational videos. This might be an interesting opportunity for people interested in Safeguarded AI to receive funding to learn about and share davidad's ideas.

The next grant round, for TA1.2, is expected to open later this year.