trentmkelly/LessWrong-43k huggingface dataset:
AISC Project: Benchmarks for Stable Reflectivity
by jacquesthibs — 2023-11-13T14:51:19.318Z

Apply to work on this project with me at AI Safety Camp 2024 before 1st December 2023.

Summary

Future prosaic AIs will likely shape their own development or that of successor AIs. We're trying to make sure they don't go insane.

There are two main ways AIs can get better: by improving their training algorithms or by improving their training data.

We consider both scenarios and tentatively believe data-based improvement is riskier than architecture-based improvement. Current models mostly derive their behaviour from their training data, not training algorithms (meaning their architectures, hyperparameters, loss functions, optimizers or the like).

For the Supervising AIs Improving AIs agenda, we focus on ensuring stable alignment when AIs self-train or train new AIs and study how AIs may drift through iterative training. We aim to develop methods to ensure automated science processes remain safe and controllable. This form of AI improvement focuses more on data-driven improvements than architectural or scale-driven ones.

Agenda:

https://www.lesswrong.com/posts/7e5tyFnpzGCdfT4mR/research-agenda-supervising-ais-improving-ais

Twitter thread explaining the agenda:

https://twitter.com/jacquesthibs/status/1652389982005338112?s=46&t=YyfxSdhuFYbTafD4D1cE9A

The non-summary

We imagine a future where AIs self-augment by continuously seeking out more and better training data, and either creating successor AIs or training themselves on that data. Often, these data will come from the AIs running experiments in the real world (doing science), deliberately seeking data that would cover a specific gap in its current capabilities, analogous to how human scientists seek data from domains where our current understanding is limited. With AI, this could involve AgentGPT-like systems that spin up many instances of themselves to run experiments in parallel, potentially leading to quick improvements if we are in an agency overhang.

We want to find methods of ensuring such 'automated science' processes remain safe and controllable, even after many rounds of self-directed data collection and training. In particular, we consider problems such as:

Preventing self-training from amplifying undesirable behaviours
Preventing semantic drift in concept representations during self-training
Ensuring cross-modality actions (such as a generated image for a text-to-image model or robot movemen

...
Research Adenda: Modelling Trajectories of Language Models
by NickyP — 2023-11-13T14:33:56.407Z

Apply to work on this project with me at AI Safety Camp 2024 before 1st December 2023.

What are the possible outcomes?

Summary

Rather than asking “What next token will the Language Model Predict?” or “What next action will an RL agent take?”, I think it is important to be able to model the longer-term behaviour of models, rather than just the immediate next token or action. I think there likely exist parameter- and compute-efficient ways to summarise what kinds of longer-term trajectories/outputs a model might output given an input and its activations. The aim of this project would be to conceptually develop ideas, and to practically build a method to summarise the possible completion trajectories of a language model, and understand how this might or might not generalise to various edge cases.

The non-summary

Motivation for the Project

There is a large amount of effort going into understanding and interpreting the weights and activations of Language Models. Much of this is focused on directly looking at the logits for next token predictions. While I think one can learn a lot about models this way, I think there are potentially more ways one can understand behaviour of models. Another method for investigation is to sample some possible outputs, but this is expensive and time-consuming, and it can be difficult to inspect all of the outputs.

I think that it should be possible to get sufficient information from understanding model behaviour and outputs in a much more condensed and compute-efficient way, such that one could iterate through different interpretability strategies, and understand more high-level insights about the models. In addition, I think that being able to get a broader understanding of what kinds of outputs a model might likely output would make it easier to monitor models, and prevent potential trajectories we might not like.

More explicitly:

“Machine Unlearning”/ablation experiments is one of the key ways of trying to have better metrics for interpretability tools. For example, if we identify a circuit that we think is doing some task A, it would be great we can remove that circuit and quickly recognise that this only affects task A, and does not affect other tasks.
Can we build a better dashboard for monitoring possible outputs/”actions” a model might take, before it makes these outputs/actions? Could we monitor when a model is starting to

...
Bostrom Goes Unheard
by Zvi — 2023-11-13T14:11:07.586Z

[Editor’s Note: This post is split off from AI #38 and only on LessWrong because I want to avoid overloading my general readers with this sort of thing at this time, and also I think it is potentially important we have a link available. I plan to link to it from there with a short summary.]

Nick Bostrom was interviewed on a wide variety of questions on UnHerd, primarily on existential risk and AI, I found it thoughtful throughout. In it, he spent the first 80% of the time talking about existential risk. Then in the last 20% he expressed the concern that it was unlikely but possible we would overshoot our concerns about AI and never build AGI at all, which would be a tragedy.

How did those who would dismiss AI risk and build AGI as fast as possible react?

About how you would expect. This is from a Marginal Revolution links post.

Tyler Cowen: Nick Bostrom no longer the Antichrist.

The next link in that post was to the GPT-infused version of Rohit Krishnan's book about AI, entitled Creating God (should I read it?).

What exactly changed? Tyler links to an extended tweet from Jordan Chase-Young, mostly a transcript from the video, with a short introduction.

Jordan Chase-Young: FINALLY: AI x-risker Nick Bostrom regrets focusing on AI risk, now worries that our fearful herd mentality will drive us to crush AI and destroy our future potential. (from an UnHerd podcast today).

In other words, Nick Bostrom previously focused on the fact that AI might kill everyone, thought that was bad actually, and attempted to prevent it. But now the claim is that Bostrom regrets this - he repented.

The context is that Peter Thiel, who warns that those warning about existential risk have gone crazy, has previously on multiple occasions referred seemingly without irony to Nick Bostrom as the Antichrist. So perhaps now Peter and others who agree will revise their views? And indeed, there was much ‘one of us’ talk.

Frequently those who warn of existential risk from AI are told they are saying something religious, are part of a cult, or are pattern matching to the Christian apocalypse, usually as justification for dismissing our concerns without argument.

The recent exception on the other side that proves the rule was Byrne Hobart, author of the excellent blog The Diff, who unlike most concerned about existential risk is explicitly religious and gave a talk about this at a religious conference. The...
The Science Algorithm AISC Project
by Johannes C. Mayer — 2023-11-13T12:52:43.150Z
Link: https://docs.google.com/document/d/1GoXaYtUyanRrkBcjAknafqdKDCyTzzgxpckYMBtgPkQ/edit#heading=h.9lmc73wscx1r

I put up a project for AISC. The easiest way to evaluate your fit is to look at the applicant questions I created. If the kind of reasoning required to answer these questions, is something you are excited about doing, then you are probably a good fit.

For more details, you can look at the project proposal. This is the introductory section thereof:

Modern deep learning is about having a simple program (SGD) search over a space of possible programs (the weights of a neural network) and select one that performs well according to a loss function.

Even though the search program is simple, the programs it finds are neither simple nor understandable. This makes it determine if a program that is spit out by the search procedure will be misaligned. It also makes it difficult to robustly bias our search procedure to select programs with specific properties such as non-deception, corrigibility, honesty, wanting what we want, etc.

My goal is to build an AI system that enables a pivotal act by figuring out the algorithms of intelligence directly, without running algorithmic search procedures that yield uninterpretable results. You could say I want to make myself play the part that SGD plays in the modern paradigm. The ideal outcome is to figure out how to write down the entire pivotal system as a non-self-modifying program explicitly, similar to how I can write down the algorithm for quicksort.

The idea is to create an algorithm that is analyzable and amenable. At every step during the design process, I want to push the system towards being understandable, aligned, and capable. Any alignment-related problems should be fixed as they come up, by deeply reaching inside the system and making the necessary changes instead of applying a superficial patch.

The goal is not to solve alignment in full generality but to build a highly restricted system that enables a pivotal act. Note that I am using Eliezer's definition of pivotal act which means something quite specific. People tend to get confused by this term. I wrote this article in an attempt to clear up some of the confusion.

The next three sections will become increasingly more concrete. The "Background" section presents background concepts to this agenda. "Directions" outlines some directions I would like to explore during the AISC. "The Plan" outlines a concrete plan, mainly for the beginning of the AISC.
You can just spontaneously call people you haven't met in years
by lc — 2023-11-13T05:21:05.726Z

Here's a recent conversation I had with a friend:

Me: "I wish I had more friends. You guys are great, but I only get to hang out with you like once or twice a week. It's painful being holed up in my house the entire rest of the time."

Friend: "You know ${X}. You could talk to him."

Me: "I haven't talked to ${X} since 2019."

Friend: "Why does that matter? Just call him."

Me: "What do you mean 'just call him'? I can't do that."

Friend: "Yes you can"

Me:

Later: I call ${X}, we talk for an hour and a half, and we meet up that week.

This required zero pretext. I just dialed the phone number and then said something like "Hey ${X}, how you doing? Wanted to talk to you, it's been a while." It turns out this is a perfectly valid reason to phone someone, and most people are happy to learn that you have remembered or thought about them at all.

Further, I realized upon reflection that the degrees of the people I know seem related to their inclination to do things like this.
Zvi's Manifold Markets House Rules
by Zvi — 2023-11-13T00:28:02.147Z

This document was updated August 11, 2024. Broad intent has not changed.

All markets created by Zvi Mowshowitz shall be graded according to the rules described herein, including the zeroth rule.

The version of this on LessWrong shall be the canonical version, even if other versions are later posted on other websites.

Rule 0: If the description of a particular market contradicts these rules, the market’s description wins, the way a card in Magic: The Gathering can break the rules. This document only establishes the baseline rules, which can be modified.

Effort put into the market need not exceed that which is appropriate to the stakes wagered and the interestingness level remaining in the question. I will do my best to be fair, but I will mostly not answer clarifying questions if the answers are more trouble than they're worth. I'm definitely not interested in hypothetical nitpicks or complaints about objectivity unless the market gets economically meaningful action.
If you are going to endlessly nitpick, or complain about the first rule, then please don't trade on my markets. It's not worth the trouble for either of us.
If a market becomes too much of a headache I will hand resolution to the mods.
Obvious errors will be corrected. If for example a date is clearly a typo, I will fix.
If the question description or resolution mechanism does not match the clear intent or spirit of the question, or does not match its title, in an unintentional way, or is ambiguous, I will fix that as soon as it is pointed out. If the title is the part in error I will fix the title. If you bet while there is ambiguity or a contradiction here, and no one including you has raised the point, then this is at your own risk.
If the question was fully ambiguous in a scenario, I will choose resolution for that scenario based on what I feel upholds the spirit of the question and what traders could have reasonably expected, if such option is available.
When resolving potentially ambiguous or disputable situations, I will still strive whenever possible to get to either YES or NO, if I can find a way to do that and that is appropriate to the spirit of the question.
Ambiguous markets that have no other way to resolve, because the outcome is not known or situation is truly screwed up, will by default resolve to the manipulation-excluded market price, if I judge that to be a reasonable assessment of

...
What’s your best utilitarian model for risking your best kidneys? 
by Ilio — 2023-11-12T23:01:58.103Z

You are the great commander of many robotic soldiers. Each soldier has two robotic kidneys. In war time they frequently need another one. If they don’t, they break (100%). If they do, the donor might break (1%).

If all soldiers were equally good at war and equally willing to give one kidney, there would be little to discuss. But war is not that simple.

In your army 1/10 are 2 times better than the median, 1/100 are 4 times better than the median, 1/1000 are 16 times better than the median, 1/10000 are 256 times better than the median, etc.

If « better » would only meant « better at war », there would be little to discuss. But systematic winning is not that simple.

In your army better also means « more likely to give a kidney » and « more likely to set an example for the others to follow ». Which means that, conditional on this robotic soldier wants to give a robotic kidney, it’s also more likely to be critical to war effort and more likely to set an example that the less systematic winners will follow. Oh well.

At this point, my brain want to retreat to heuristics like « Let’s assume Scott already compute that [1]», but that sounds sloppy. What’s your utilitarian model?

https://www.astralcodexten.com/p/my-left-kidney ↩︎
Helpful examples to get a sense of modern automated manipulation
by trevor — 2023-11-12T20:49:57.422Z

Why this is valuable

In the face of

unclear AGI timelines

, thinking about SOTA human behavior manipulation technology is not intrinsically valuable (in fact,

all

along

I’ve

repeatedly

asserted

that it has a serious threat of distracting people from AGI, which will

kill everyone

, and should not be researched as a new X-risk).

However, it is probably instrumentally valuable for

understanding AI governance, geopolitics, and race dynamics

, and because

the AI safety community is disproportionately at risk of being targeted

due to

entering an arena

with

heavy hitters

.

One of the main ways that people

transmit knowledge and intuitive understanding

is through

intuitive examples

e.g. use cases. Examples of manipulation strategies are critical to develop an intuitive understanding of what's probably already out there; however, although it's easy to predict whether

governments

are working on autonomous manipulation in general, which is more than sufficient to indicate that the AI safety community should take precautions to minimizing the attack surface; it's still much harder to trace the specific outlines of particular autonomous manipulation strategy that has likely already been discovered and exploited.

The

clown attack

example was low-hanging fruit for me to discover and write about; its use for autonomous manipulation was both unusually powerful, obvious, and inevitable. Lots of people found that example extremely helpful for getting oriented towards the situation with human thought control. Other examples of automated manipulation are harder to be confident were 1) easy to discover and 2)

existing institutions

were incentivized to deploy and test; but they will still be helpful for demonstrating what kinds of things are probably out there.

This model is falsifiable; I predict at 95% that similar types of automated manipulation strategies as these were deployed by US, Russia, or Chinese companies or agencies to steer people’s thinking on Ukraine War and/or Covid-related topics. I lose a large amount of bayes points if that didn’t happen e.g.

governments weren’t sufficiently aware

of these capabilities due to

uniform incompetence

, or the

engineering problems

are too difficult e.g. due to

spaghetti towers

or

data poisoning/data security persistently thwarting th

...
Two children's stories
by Optimization Process — 2023-11-12T20:29:45.808Z

(I wanted to write some rationalist-flavored bedtime stories for my friends' four-year-old. I don't really have the knack, but ChatGPT did an imperfect-but-respectable job, given a fair bit of thematic guidance. Here are the first two I've churned out, intended to be read together, for reasons which... will become apparent.)

1. The Whispering Trees of Green Meadows

In the quaint town of Green Meadows, where the sun smiled warmly upon colorful cottages and gardens bloomed with laughter, there was a little fact everyone cherished. The townsfolk believed that their trees were not just tall and leafy guardians, but whisperers of secrets when the moon took its throne in the night sky. These ancient trees, with their gnarled branches and rustling leaves, were said to murmur truths and tales about the future to those who listened closely.

Five-year-old Lily, with her wide eyes full of wonder and a head full of curly hair, had grown up hearing these enchanting stories. Every night, as the stars began to twinkle, her parents would tell her about the time the old oak tree by the river predicted a bountiful harvest, or when the willow near the school whispered about the early arrival of spring. Lily loved these stories, her heart dancing with the magic of the whispering trees of Green Meadows.

One crisp, golden autumn morning, as Lily skipped along a carpet of fallen leaves, she noticed a moving truck outside a house near the edge of town. A new neighbor was moving in. His name was Mr. Simon, a kind man with a gentle smile and a love for books and stargazing. Unlike the other adults in Green Meadows, Mr. Simon didn't seem to know about the whispering trees. When Lily excitedly told him about the trees' nightly secrets, he listened intently with a curious twinkle in his eye.

"Well, Lily," Mr. Simon said thoughtfully, "I think the trees might have a different story to tell. You see, when the wind dances through their branches, it makes the leaves rustle and sway. That's what creates those whispering sounds. It's not magic, but the music of nature."

Lily's eyes widened in surprise. Never before had she heard such an idea!

The next day at school, under a sky painted with playful clouds, Lily found herself amidst her circle of friends during recess. Eagerly, she recounted Mr. Simon's thoughts about the whispering trees. "He said that it's actually the wind making the leaves ...
The Fundamental Theorem for measurable factor spaces
by Matthias G. Mayer — 2023-11-12T19:25:25.583Z

I present the fundamental theorem for all finitely factored measurable spaces.
The fundamental theorem is that
two events are orthogonal if and only if they are independent in all product probability distributions.
It tells us that the definition of orthogonality really captures the essence of
structural independence by the following arguments:

Whenever things are structurally independent, they should be probabilistically independent, regardless of the specific chosen distribution.

Orthogonality should be the strongest notion that entails the previous point.

This theorem was previously proved in Finite Factored Sets for the finite case.
The general case is interesting, since we can't use the finite structure.
All the possible arguments are limited to the axioms
of a measurable space. In particular, infinite things
are sort of limits of finite things,
so we can expect, through this result, that there
should be nice approximation theorems for
orthogonality. Something like, if I get more and more data
about the world, then I can refine my view of which things
are structurally independent.

To understand the technical result, it is necessary to understand the definition of the history in this setting.
All the maths is in this document.
I will try to describe a bit of the intuition used to derive the theorem.

The core idea is to express mathematically that the history tells us,
when the conditional probability of an event depends on which factor.

I show that the history can be expressed mathematically exactly as this
and show that this representation can be used to deduce that
structural independence (independence for all factored distributions)
implies orthogonality.

My definition of history still uses a probability distribution
to define the disintegration of the index function,
i.e. we need that πJ⫫πJc for all factorized probability distributions.
It turns out that it suffices to show the condition for one such distribution.

Furthermore, in Lemma 9, we can write a more explicit form that conditional probabilities need to take,
to satisfy this criterion.
I am positive that this can be leveraged to deduce a criterion that does not
reference probabilities at all.

It is noteworthy, that we don't even need to assume polish spaces,
the arguments work for any measure space modulo nullsets.

The easiest way to extend this to infinitely factored spaces is to simply
only allow...