trentmkelly/LessWrong-43k huggingface dataset:
[Aspiration-based designs] Outlook: dealing with complexity
by Jobst Heitzig — 2024-04-28T13:06:35.841Z

Summary. This teaser post sketches our current ideas for dealing with more complex environments. It will ultimately be replaced by one or more longer posts describing these in more detail. Reach out if you would like to collaborate on these issues.

Multi-dimensional aspirations

For real-world tasks that are specified in terms of more than a single evaluation metric, e.g., how much apples to buy and how much money to spend at most, we can generalize Algorithm 2 as follows from aspiration intervals to convex aspiration sets:

Assume there are d>1 many evaluation metrics ui, combined into a vector-valued evaluation metric u=(u1,…,ud).
Preparation: Pick d+1 many linear combinations fj in the space spanned by these metrics so that their convex hull is full-dimensional and contains the origin, and consider the d+1 many policies πj each of which maximizes the expected value of the corresponding function fj. Let Vj(s) and Qj(s,a) be the expected values of u when using πj in state s or after using action a in state s, respectively (see Fig. 1). Let the admissibility simplices V(s) and Q(s,a) be the simplices spanned by the vertices Vj(s) and Qj(s,a), respectively (red and violet triangles in Fig. 1). They replace the feasibility intervals used in Algorithm 2.
Policy: Given a convex state-aspiration set E(s)⊆V(s) (central green polyhedron in Fig. 1), compute its midpoint (centre of mass) m and consider the d+1 segments ℓj from m to the corners Vj(s) of V(s) (dashed black lines in Fig. 1). For each of these segments ℓj, let Aj be the (nonempty!) set of actions for which ℓj intersects Q(s,a). For each a∈Aj, compute the action-aspiration E(s,a)⊆Q(s,a) by shifting a copy Cj of E(s) along ℓj towards Vj(s) until the intersection of Cj and ℓj is contained in the intersection of Q(s,a) and ℓj (half-transparent green polyhedra in Fig. 1), and then intersecting Cj with Q(s,a) to give E(s,a) (yellow polyhedra in Fig. 1). Then pick one candidate action from each Aj and randomi

...
[Aspiration-based designs] 3. Performance and safety criteria, and aspiration intervals
by Jobst Heitzig — 2024-04-28T13:04:56.249Z

Summary. In this post, we extend the basic algorithm by adding criteria for choosing the two candidate actions the algorithm mixes, and by generalizing the goal from making the expected Total equal a particular value to making it fall into a particular interval. We only use simple illustrative examples of performance and safety criteria and reserve the discussion of more useful criteria for later posts.

Introduction: using the gained freedom to increase safety

After having introduced the basic structure of our decision algorithms in the last post, in this post we will focus on the core question: How shall we make use of the freedom gained from having aspiration-type goals rather than maximization goals?

After all, while there is typically only a single policy that maximize some objective function (or very few, more or less equivalent policies), there is typically a much larger set of policies that fulfill some constraints (such as the aspiration to make the expected Total equal some desired value).

More formally: Let us think of the space of all (probabilistic) policies, Π, as a compact convex subset of a high-dimensional vector space with dimension d≫1 and Lebesgue measure μ. Let us call a policy π∈Π successful iff it fulfills the specified goal, G, and let ΠG⊆Π be the set of successful policies. Then this set has typically zero measure, μ(ΠG)=0, and low dimension, dim(ΠG)≪d, if the goal is a maximization goals, but it has large dimension, dim(ΠG)≈d, for most aspiration-type goals.

E.g., if the goal is to make expected Total equal an aspiration value, Eτ=E, we typically have dim(ΠG)=d−1 but still μ(ΠG)=0. At the end of this post, we discuss how the set of successful policies can be further enlarged by switching from aspiration values to aspiration intervals to encode goals, which makes the set have full dimension, dim(ΠG)=d, and positive measure, μ(ΠG)>0.

What does that mean? It means we have a lot of freedom to choose the actual policy π∈ΠG that the agent should use to fulfill an aspiration-type goal. We can try to use this freedom to choose policies that promise to be rather safe than unsafe according to some generic safety metric, similar to the impact metrics used in reward function regularization for maximizers.

Depending on the t...
[Aspiration-based designs] 2. Formal framework, basic algorithm
by Jobst Heitzig — 2024-04-28T13:02:17.253Z

Summary. In this post, we present the formal framework we adopt during the sequence, and the simplest form of the type of aspiration-based algorithms we study. We do this for a simple form of aspiration-type goals: making the expectation of some variable equal to some given target value. The algorithm is based on the idea of propagating aspirations along time, and we prove that the algorithm gives a performance guarantee if the goal is feasible. Later posts discuss safety criteria, other types of goals, and variants of the basic algorithm.

Assumptions

In line with the working hypotheses stated in the previous post, we assume more specifically the following in this post:

The agent is a general-purpose AI system that is given a potentially long sequence of tasks, one by one, which it does not know in advance. Most aspects of what we discuss focus on the current task only, but some aspects relate to the fact that there will be further, unknown tasks later (e.g., the question of how much power the agent shall aim to retain at the end of the task).
It possesses an overall world model that represents a good enough general understanding of how the world works.
Whenever the agent is given a task, an episode begins and its overall world model provides it with a (potentially much simpler) task-specific world model that represents everything that is relevant for the time period until the agent gets a different task or is deactivated, and that can be used to predict the potentially stochastic consequences of taking certain actions in certain world states.
That task-specific world model has the form of a (fully observed) Markov Decision Process (MDP) that however does not contain a reward function R but instead contains what we call an evaluation function related to the task (see 2nd to next bullet point).
As a consequence of a state transition, i.e., of taking a certain action a in a certain state s and finding itself in a certain successor state s′, a certain task-relevant evaluation metric changes by some amount. Importantly, we do not assume that the evaluation metric inherently encodes things of which more is better. E.g., the evaluation metric could be global mean temperature, client's body mass, x coordinate of the agent's right thumb, etc. We call the step-wise change in the evaluation metric the received Delta in that time step, denot

...
[Aspiration-based designs] 1. Informal introduction 
by B Jacobs — 2024-04-28T13:00:43.268Z

Sequence Summary. This sequence documents research by SatisfIA, an ongoing project on non-maximizing, aspiration-based designs for AI agents that fulfill goals specified by constraints ("aspirations") rather than maximizing an objective function​​. We aim to contribute to AI safety by exploring design approaches and their software implementations that we believe might be promising but neglected or novel. Our approach is roughly related to but largely complementary to concepts like quantilization and satisficing (sometimes called "soft-optimization"), Decision Transformers, and Active Inference.

This post describes the purpose of the sequence, motivates the research, describes the project status, our working hypotheses and theoretical framework, and has a short glossary of terms. It does not contain results and can safely be skipped if you want to get directly into the actual research.

Epistemic status: We're still in the exploratory phase, and while the project has yielded some preliminary insights, we don't have any clear conclusions at this point. Our team holds a wide variety of opinions about the discoveries. Nothing we say is set in stone.

Purpose of the sequence

Inform: We aim to share our current ideas, thoughts, disagreements, open questions, and any results we have achieved thus far. By openly discussing the complexities and challenges we face, we seek to provide a transparent view of our project's progression and the types of questions we're exploring.
Receive Feedback: We invite feedback on our approaches, hypotheses, and findings. Constructive criticism, alternative perspectives, and further suggestions are all welcome.
Attract Collaborators: Through this sequence, we hope to resonate with other researchers and practitioners who our exploration appeals to and who are motivated by similar questions. Our goal is to expand our team with individuals who can contribute their unique expertise and insights.

Motivation

We share a general concern regarding the trajectory of Artificial General Intelligence (AGI) development, particularly the risks associated with creating AGI agents designed to maximize objective functions. We have two main concerns:

(I) AGI development might be inevitable

(We assume this concern needs no further justification)

(II) It might be impossible to implement an objective function the maximization of which would be safe

The conventional view ...
Playing Northboro with Lily and Rick
by jefftk — 2024-04-28T02:40:03.436Z

This afternoon Lily, Rick, and I ("Dandelion") played our first dance
together, which was also Lily's first dance. She's sat in with

Kingfisher for a set or
two many times, but this was her first time being booked and playing
(almost) the whole time.

Lily started playing fiddle in Fall 2022, and after about a year she
had enough tunes up to dance speed that I was thinking she'd be ready
to play a low-stakes
dance together soon. Not right away, but given how far out dances
booked it seemed about time to start writing to some folks: by the
time we were actually playing the dance she'd have even more tunes and
be more solid on her existing ones. She was very excited about this
idea; very motivated by performing.

I wrote to a few dances, and while several (very reasonably!) said to
send another sample when we had a bit more experience, the Northboro dance said yes for
2024-04-27. This gave a good amount of time to work on things.

At the time we were booked Lily had some tunes up to speed that would
be a good fit for dancing (Sandy Boys, Coleman's March, ...) but had
also recently been pretty excited about some notey tunes that she was
still quite a while from getting up to speed (Dancing Bear, Bus Stop,
...) and so wasn't on track to end up with a set list that was going
to work. We looked over my tune list and picked a few tunes to learn
that weren't too hard (The Wren, Trip to Moscow, ...). These went
pretty quickly: they're tunes she'd heard all her life, so it was
mostly a matter of getting them into her fingers.

By the time of the gig the list we had was:

Mairi's Wedding

Coleman's March

The Wren

Berentanz

Trip to Moscow

Road to Boston

Angeline the Baker

Cripple Creek

Sandy Boys

Liza Jane

Lisnagun

Haaplevese Waltz

Willow Tree Waltz

Waterfall Waltz

If you want to play multiple tunes in a set this isn't going to work,
and it's pretty skewed towards marches, but it was enough! We ended
up playing (from memory):

Road to Boston

The Wren

Cripple Creek

Coleman's March

Berentanz

Sandy Boys

Willow Tree Waltz

[break]

Gaspe / Scully's [no Lily]

Trip to Moscow

Road to Lisdoonvarna / Maison de Glace [no Lily]

Angeline the Baker / Liza Jane

Haaplevese Waltz

Lots of one-tune sets! But I find these pretty fun, getting to think
of lots of different ways to accompany them to add variety and rei...
Release of UN's draft related to the governance of AI (a summary of the Simon Institute's response)
by Sebastian Schmidt — 2024-04-27T18:34:39.836Z
Link: https://forum.effectivealtruism.org/posts/LaQMPEKDydJwmuGnu/release-of-un-s-draft-related-to-the-governance-of-ai-a

I just spent a couple of hours trying to understand the UN’s role in the governance of AI. The most important effort seems to be the Global Digital Compact (GDC). An initiative for member states to “outline shared principles for an open, free and secure digital future for all". The GDC has been developed by member states since at least 2021 and will be finalized at the upcoming Summit for the Future in September. Recently, the first draft (Zero Draft) was shared and will be discussed and iterated in the final months before the Summit. The Simon Institute wrote an excellent seven-page response to this. I warmly recommend reading their response but you can find my imperfect summary on the EA Forum.
Mercy to the Machine: Thoughts & Rights
by False Name — 2024-04-27T16:36:06.006Z

Abstract: First [1)], a suggested general method of determining, for AI operating under the human feedback reinforcement learning (HFRL) model, whether the AI is “thinking”; an elucidation of latent knowledge that is separate from a recapitulation of its training data. With independent concepts or cognitions, then, an early observation that AI or AGI may have a self-concept. Second [2)], by cited instances, whether LLMs have already exhibited independent (and de facto alignment-breaking) concepts or behavior; further observations of possible self-concepts exhibited by AI. Also [3)], whether AI has already broken alignment by forming its own “morality” implicit in its meta-prompts. Finally [4)], that if AI have self-concepts, and more, demonstrate aversive behavior to stimuli, that they deserve rights at least to be free of exposure to what is aversive, and that those rights should be respected whether or not it is clear AI are “conscious”.

Epistemic status: Of the general method of elucidating latent knowledge, strictly-modest confidence, without detailed intelligence of developer’s exact training procedures. Of aversion in AI, conjectural, but observationally motivated.

1) Sapient Silicon

We might test in general whether a large language model is experiencing or producing “independent thoughts” if, it being subject to human feedback reinforcement learning, the LLM is capable of predicting, on the basis only of its training data, what the rankings of its own outputs will be by a feedback module or by humans– prior to the use of the module. For consider that in this case, on the assumption the feedback module has not yet been used, its feedback is ipso facto not part of the training data. Then only on the implicit ordering in the training data, and more, on the “comprehension” by the LLM of that training data, can it have an “independent thought” regarding the elements external to itself and its training data, which are given outputs – and how those externals take the outputs.

Having issued training runs to the neural network, but as yet depriving it of the feedback of its feedback module for HFRL, for any given query we might make to it, in this interregnum, it can predict how preferred is its output for our query, only on its independent cognition of its own outputs as useful, i.e.: if it is capable of introspecting, with all that implies of th...
Constructability: Plainly-coded AGIs may be feasible in the near future
by Épiphanie Gédéon — 2024-04-27T16:04:45.894Z

Charbel-Raphaël Segerie and Épiphanie Gédéon contributed equally to this post.
Many thanks to Davidad, Gabriel Alfour, Jérémy Andréoletti, Lucie Philippon, Vladimir Ivanov, Alexandre Variengien, Angélina Gentaz, Simon Cosson, Léo Dana and Diego Dorn for useful feedback.

TLDR: We present a new method for a safer-by design AI development. We think using plainly coded AIs may be feasible in the near future and may be safe. We also present a prototype and research ideas on Manifund.

Epistemic status: Armchair reasoning style. We think the method we are proposing is interesting and could yield very positive outcomes (even though it is still speculative), but we are less sure about which safety policy would use it in the long run.

Current AIs are developed through deep learning: the AI tries something, gets it wrong, then gets backpropagated and all its weight adjusted. Then it tries again, wrong again, backpropagation again, and weights get adjusted again. Trial, error, backpropagation, trial, error, backpropagation, ad vitam eternam ad nauseam.

Of course, this leads to a severe lack of interpretability: AIs are essentially black boxes, and

we are not very optimistic about post-hoc interpretability

.

We propose a different method: Constructability or AI safety via pull request.[1]

By pull request, we mean that instead of modifying the neural network through successive backpropagations, we construct and design plainly-coded AIs (or hybrid systems) and explicitly modify its code using LLMs in a clear, readable, and modifiable way.

This plan may not be implementable right now, but might be as LLMs get smarter and faster. We want to outline it now so we can iterate on it early.

One possible long-term vision that constructability could lead to, in which we make use of a black-box superhuman coder to create code that we then audit and deploy.

Overview

If the world released a powerful and autonomous agent in the wild,

white box or black box, or any color really,

humans might simply get replaced by AI

.

What can we do in this context?

Don't create autonomous AGIs.
Keep your AGI controlled in a lab, and align it.
Create a minimal AGI controlled in a lab, and use it to produce safe artifacts.

This post focuses on this last path, and the specific artifacts that we want to create are plainly coded AIs (or hybrid systems)[2].

We prese...
So What's Up With PUFAs Chemically?
by J Bostock — 2024-04-27T13:32:52.159Z

This is exploratory investigation of a new-ish hypothesis, it is not intended to be a comprehensive review of the field or even a a full investigation of the hypothesis.

I've always been skeptical of the seed-oil theory of obesity. Perhaps this is bad rationality on my part, but I've tended to retreat to the sniff test on issues as charged and confusing as diet. My response to the general seed-oil theory was basically "Really? Seeds and nuts? The things you just find growing on plants, and that our ancestors surely ate loads of?"

But a twitter thread recently made me take another look at it, and since I have a lot of chemistry experience I thought I'd take a look.

The PUFA Breakdown Theory

It goes like this:

PUFAs from nuts and seeds are fine. Deep-frying using PUFAs causes them to break down in a way other fatty acids do not, and these breakdown products are the problem.

Most of a fatty acid is the "tail". This consists of hydrogen atoms decorating a backbone of carbon atoms. Each carbon atom can make up to four bonds, of which two have to be to other carbons (except the end carbon which only bonds to one carbon) leaving space for two hydrogens. When a chain has the maximum number of hydrogen atoms, we say it's "saturated". These tails have the general formula CnH2n+1:

For a carbon which is saturated (i.e. has four single bonds) the bonds are arranged like the corners of a tetrahedron, and rotation around single bonds is permitted, meaning the overall assembly is like a floppy chain.

Instead, we can have two adjacent carbons form a double bond, each forming one bond to hydrogen, two bonds to the adjacent carbon, and one to a different carbon:

Trans (top) and cis (bottom) double bonds.

Unlike single bonds, double bonds are rigid, and if a carbon atom has a double bond, the three remaining bonds fall in a plane. This means there are two ways in which the rest of the chain can be laid out. If the carbons form a zig-zag S shape, this is a trans double bond. If they form a curved C shape, we have a cis double bond.

The health dangers of trans-fatty acids have been known for a long while. They don't occur in nature (which is probably why they're so bad for us). Cis-fatty acids are very common though, especially in vegetable and, yes, seed oils.

Of course there's no reason why we should stop at one double bond, we can just as easily have multiple. This gets us to the nam...
Link: Let's Think Dot by Dot: Hidden Computation in Transformer Language Models by Jacob Pfau, William Merrill & Samuel R. Bowman
by Chris_Leong — 2024-04-27T13:22:53.287Z
Link: https://twitter.com/jacob_pfau/status/1783951795238441449

One consideration that is pretty important for AI safety is understanding the extent to which a model's outputs are aligned with its chain of thought.

This paper (Twitter thread linked) provides some relevant evidence. It demonstrates that it is possible for a model to achieve performance comparable to chain-of-thought with dots replacing the chain of thought under some circumstances. In particular, the model can't just be trained with sequences like "QUESTION..............ANSWER", but sequences need to also be mixed in where there is a "parallelisable" chain of thought. Here "parallelisable" means that different components of the chain of thought can be calculated in parallel rather than all separately.

In terms of how this is relevant to AI safety, this provides an empirical demonstration that a model is capable of very effectively engaging in background computation under certain circumstance. It shows that the model is much better at doing background parallelizable tasks than non-parallelizable tasks. In other words, the chain of thought is less binding than it might have been because the model is free to perform some of the computations necessary for future tokens in the background.
Two Vernor Vinge Book Reviews
by Maxwell Tabarrok — 2024-04-27T12:14:53.917Z
Link: https://www.maximum-progress.com/p/two-vernor-vinge-book-reviews

Vernor Vinge is a legendary and recently deceased sci-fi author. I’ve just finished listening to the first two books in the Zone of Thought trilogy. Both books are entertaining and culturally influential. The audio versions are high-quality.

A Deepness in the Sky is about two spacefaring human civilizations with clashing societal structures converging on a mysterious solar system where a third, alien spider civilization is undergoing an industrial revolution. This combination of high-tech spacefaring and low-tech steampunk or fantasy shows up in both books and is one of the most compelling and unique parts of Vinge’s work. It allows him to focus on depicting rapid technological change. This is surprisingly rare in sci-fi which mostly takes the Star Wars or Trek route of depicting advanced but static technology.

Vinge also introduces the idea of “programmer archaeology” in this book. After thousands of years of advanced civilization, the free-trading Qeng Ho have enough code and schematics in their archives to solve almost any problem. The challenge is finding the right piece at the right time. Searching through the archive is a full time job. From a 2024 perspective this is evocative of prompt engineering. LLMs are these massive inscrutable matrices that in some ways reflect all of humanity’s written knowledge but digging that knowledge out of them is a non-trivial skill.

Another element with a different shine in a post-LLM world is the collectivist Emergent’s automation. The Emergents repurposed a brain-rot pandemic into a mind control/enhancement tech which allows them to create semi-human semi-robot obsessive workers called Focused. This is explicitly not AI, which the book claims would be too rigid and un-creative, but this conceit is less plausible today. The alignment problems that the Emergents face with the Focused are inadvertently relevant to the obsessive but strangely human behavior of LLMs.

A Fire Upon the Deep is a loose sequel to a Deepness in the Sky with one crossover character. It also involves a low-tech alien civilization colliding with space-faring humans (and some talking plants). The low-tech alien civilization here is really cool. The aliens essentially live in an alternate history Earth where wolves became the dominant intelligent species instead of humans. They do this by leaning into pack cooperation, leading to a hybrid hive-mind/indi...