trentmkelly/LessWrong-43k huggingface dataset:
Elements of Computational Philosophy, Vol. I: Truth
by Paul Bricman — 2023-07-01T11:44:22.154Z
Link: https://compphil.github.io/truth/

We're excited to share the first volume of Elements of Computational Philosophy, an interdisciplinary and collaborative project series focused on operationalizing fundamental philosophical notions in ways that are natively compatible with the current paradigm in AI.

The first volume paints a broad-strokes picture of operationalizing truth and truth-seeking. Beyond this high-level focus, its 100+ pages can be framed in several different ways, which is why we placed multiple topic-based summaries at the beginning of the document. The note to the reader and the table of contents should further help scope and navigate the document.

Have a pleasant read, and feel free to use this linkpost to comment on the document as you go. Questions, criticism, and suggestions are all welcome.

PS: There will soon be a presentation about the overarching project series as part of the alignment speaker series hosted by EleutherAI. Expect more information soon on the #announcements channel of their Discord server. In general, keep an eye on this space.
Micro Habits that Improve One’s Day
by silentbob — 2023-07-01T10:53:57.280Z

This is my fifth attempt at writing this post. I’m starting to think that I’ve already spent way too much time on this topic, which I’m convinced is valuable, but maybe not so valuable as to spend 20 hours perpetually rewriting a post about it. So obviously my solution is to rewrite it again, but this time in bullet points.

Here’s a tl;dr: There are some habits people can pick up that are very cheap, and may have positive effects, but these effects are too small to reliably notice consciously. Hence these habits are often neglected. In this post I argue to take some of these habits more seriously, and if they’re low-cost enough for you to implement, stick to them even absent of any feeling of them being useful.

One way to look at habits is to look at two axes: usefulness and effort
Both can be positive or negative

“Good habits” typically are beneficial, but it takes some effort to install them
“Bad habits” are the opposite in both directions, they are detrimental in some important way, but it is more effortful to get rid of the habit than to stick to it
There are of course also things that are either both beneficial and effortless (like breathing), or detrimental and effortful (like banging your head against a wall), but we typically don’t think much about these two quadrants because there’s no reason to override our natural inclinations

There’s a particular area in the “good but effortful” space that I call “useful micro habits”:

Interventions that are beneficial, but also take very little effort to maintain
A common problem is that for some of them their usefulness lies below the threshold of perception: it’s hard to tell if they really do anything, because benefits are small and/or indirect
So people may try such habits for a while, and then often drop them again for the apparent lack of benefits

Some examples of such useful micro habits (note that both usefulness and effort differ between people, so not all these examples may fit the definition above for you):

Paying attention to better hydration
Keeping CO2 levels low in your home/office
Dressing well (even at home)
Supplementing creatine
Planning one’s day in advance
Exposure to sunlight early in the day
Taking regular short breaks
Breathing exercises & meditation
Leaving the house daily, even when it’s not necessary
Journaling
Hugs
(Partially) cold showers

Many of these things may provide, say, a mar

...
Ateliers: But what is an Atelier?
by Stephen Fowler — 2023-07-01T05:57:19.510Z

TL;DR: Ateliers is an "Agent Foundations" research approach that reasons about the system that produces an intelligent agent and its surrounding environment. After introducing the concept, we use Ateliers to discuss the concept of "Goal-like Correlations", deception, corrigibility and many others.

Ateliers

Picture a small room floating freely somewhere in space. The atelier contains innumerable shelves of every resource imaginable, from unrefined ores to 300nm wafers. It even contains physical drives with petabytes of appropriate training data. We could view this object as a model of the Earth as it sits today, containing everything you could possibly need to build an AGI.

Now, meet the atelier's only employee, a robot. An odd fellow, this robot's movement and actions are dictated by a large reel of punched tape feeding its head. This automaton is the Builder and the long tape that feeds into its head is The Instructions. Note that the Builder is not an AGI; just diligently follows what's on the tape.

We imagine that this Atelier is itself floating inside of an Enclosure, a finite enclosed region of space. If the Atelier is the Earth, we could set the Enclosure to be a sphere around the Solar System, or even the Galaxy.

We are most interested in the cases where the Atelier contains the Resources and appropriate Instructions to allow a Builder to construct and deploy AGI, but this does not have to be the case. (We can define an Atelier that consists of a Builder doing nothing, or constructing car parts)

In reasoning about ateliers it will be helpful to also introduce Evaluations, measurements of some property of the system. We will be particularly interested in evaluations that are made by viewing light some distance away from the Atelier, along the edge of the Enclosure.

Think of this type of Evaluation as “questions you could ask Eliezer with a telescope floating in a space capsule nearby” and may include things like “what color is the atelier” or “is there a Dyson Sphere around a nearby sun”. Much of what we will be discussing below will concern correlations between small sections of data in the tape and observations made some distance away.

While we could imagine an Atelier consisting of a Builder doing nothing, we will focus our attention on scenarios in which the Builder constructs an intelligent optimizer.

Motivation

We want a theory of alignme...
Predicting: Quick Start
by duck_master — 2023-07-01T03:43:50.379Z

Have you ever wanted to predict the future? Did you ever take a peek at Metaculus or Manifold or some other site, suddenly feel super intimidated about the effort you'd need to put in, and never try again? If so, fear not, for this post is for you![1]

This post is intended to be a how-to guide. I will add pictures because apparently people really like reading posts with pictures. However, since there are exactly zero good explainers on how to be a good predictor as far as I know, I'm hoping that more experienced predictors will find value in this post too.

Thanks to Emily Thomas for proofreading this post!

Epistemic status/content warnings: Basically correct in the big picture, but I might have missed or misstated some details. May be too wordy or technical in parts.

1. What is probabilistic predicting, anyways?

Throughout history, people have often been interested in predicting the future. In ancient times, people would ask the local prophets or soothsayers. (This is called "prophecy", and it's mostly extinct.) In more recent decades, they have tried other, more systematic approaches, such as extrapolating trend lines on a graph into infinity. (This was called "futurism", and it still exists to some extent these days.) Needless to say, the track record of such efforts is pretty bad.

The key insight of probabilistic prediction is simple: Instead of saying that things will definitely happen, just soften it to saying that they will only probably happen. And quantify the probability!

1.1 What is probability?

Probability is a weird and mystical concept that does not sit well with some people. For our purposes, a probability is a number associated with an event that describes what fraction of the times it could happen that it actually does happen. For example, if an event has a probability of 0.8, then given 10 different chances, the event should happen about 8 times. To make our lives easier, we predictors usually use percentages to write probabilities (for example, "70%" instead of "0.7"). Note that the probability of any event is always between 0% and 100%.

You should be aware of discrete distributions, which have a fixed list of outcomes whose probabilities are nonnegative and sum to 100%. You should also be aware of continuous distributions, where the true value can range continuously over some interval. Probabilists describe using a probability density function (rou...
Horizontal and Vertical Integration
by Jeffrey Heninger — 2023-07-01T01:15:26.987Z

TLDR: Slowing AI is easiest if AI companies are horizontally integrated, but not vertically integrated.

Some people have expressed a preference for there being only one leading AI company,[1] or for leading AI companies to become better at coordinating with one another.[2] If there is one company which has a clear lead and that company is safety conscious, then it could move cautiously while developing increasingly powerful AI systems. Having multiple leading AI companies could lead to a race, and these companies might be less cautious in order to get ahead of their competitors.

It is not clear whether the incentives actually make developing powerful AI a race.[3] Even if they are not, if people believe that there is a race, they might accelerate AI development. If there were fewer or no competitors, companies would be less likely to believe that there is an AI arms race.

This is an argument in favor of horizontal integration. It would be better for AI companies to merge or cooperate with their competitors.

It is not an argument in favor of vertical integration. It would be easier to slow the development of potentially dangerous AI if no one company controlled all or most of the supply chain for developing AI.

Consider a hypothetical horizontally integrated, but not vertically integrated, AI sector:[4] There is one company which produces GPUs. A different company writes and maintains operating systems. A third company collects training data. A fourth company trains base models. A fifth company does fine tuning.

This supply chain has five bottlenecks. Any one of them could decide that AI systems are getting scary and decide to slow things down. Since each level is horizontally integrated, actors at other levels cannot turn to their competitors to replace them. It does not seem like any one of the five companies could completely stop further development, because there are multiple ways to make progress. Some of them are better positioned to slow development than others, but any one of them could make further developments harder for other levels of the supply chain.

If the AI sector were completely integrated, it would remove the potential race dynamics, but it also makes everything depend on a single actor. If that actor is not cautious, then other private actors would have more limited ways to respond. In a not vertically integr...
Inflection AI announces $1.3 billion of funding led by current investors, Microsoft, and NVIDIA
by SandXbox — 2023-06-30T21:32:09.943Z
Link: https://inflection.ai/inflection-ai-announces-1-3-billion-of-funding

Palo Alto, CA, June 29th, 2023 – Inflection AI today announced that the company has raised $1.3 billion in a fresh round of funding led by Microsoft, Reid Hoffman, Bill Gates, Eric Schmidt, and new investor NVIDIA. The new funding brings the total raised by the company to $1.525 billion.

Along with its partners CoreWeave and NVIDIA, Inflection AI is building the largest AI cluster in the world comprising 22,000 NVIDIA H100 Tensor Core GPUs. In just over a year, Inflection AI has developed one of the most sophisticated large language models in the market to enable people to interact with Pi, your Personal AI (pi.ai), in the most simple, natural way and receive fast, relevant and helpful information and advice.

The funding will support the company’s continued work to build and design Inflection’s first product, launched in May. A new class of AI, Pi is designed to be a kind and supportive companion offering text and voice conversations, friendly advice, and concise information in a natural, flowing style. [...]

Largest AI cluster in the world

The deployment of 22,000 NVIDIA H100 GPUs in one cluster is truly unprecedented, and will support training and deployment of a new generation of large-scale AI models. Combined, the cluster develops a staggering 22 exaFLOPS in the 16-bit precision mode, and even more if lower precision is utilized. We estimate that if we entered our cluster in the recent TOP500 list of supercomputers, it would be the 2nd and close to the top entry, despite being optimized for AI – rather than scientific – applications. The rollout of the cluster is actively under way, and we have already been able to confirm its performance in the recent MLPerf benchmark.
Introduction
by Robert Kralisch — 2023-06-30T20:45:10.229Z

This series of posts presents our work on the frameworks of “Positive Attractors” and “Inherently Interpretable Architectures” during the AI Safety Camp 2023.
(Note: Sections written by the different team members were marked with their initials in square brackets, since the investigated research strands were more distinct than a fully unified presentation could capture.)
We believe that the AI Alignment community is still largely lacking in clear frameworks or methodologies for progress on the Alignment Problem. While there are some frameworks of concrete progress towards a solution, most of the well-regarded work is still about becoming less confused - either regarding current AIs (mechanistic interpretability) or the concepts involved with reasoning about synthetic intelligence in the first place (agent foundations).

It was from this context that we set out to clarify the suitability of two additional broad frameworks of progress, trying to understand how they would inform and shape research, and contrast with already existing agendas.
The research on this project was predominantly conceptual, relying on discussion and first principle reasoning aided by literature review. Team members were encouraged to come to independent views about the frameworks during the initial explorations, to have meaningful discussion and convergence during the later stages of the project.

To briefly introduce the two frameworks:
Positive Attractors are about the idea that cognitive systems or their training processes can be imbued with features that make them resistant against broad kinds of undesirable change, features that one can think of as positive attractors that pull the system into a safe region of algorithmic space. The intention is that such positive attractors can make systems resilient against failure modes that we haven’t fully understood yet and potentially enable some research approaches that would otherwise be discarded early.

Inherently Interpretable Architectures speak to an approach that is complementary to mechanistic interpretability research. Rather than relying on the hope that mechanistic interpretability research will provide adequate understanding about modern systems in time for us to base clear solutions and arguments on its findings, it seems sensible to also pursue the path of designing and studying systems that are inherently very interp...
Inherently Interpretable Architectures
by Robert Kralisch — 2023-06-30T20:43:23.176Z

Definition

[JP] The framework of inherently interpretable architectures (or architectures for interpretability) involves two overarching parts: firstly, the general research direction of creating architectures that lend themselves a-priori to human interpretability; and second, the crystallisation of this into specific suggestions for architectural features that allow this.

This would be in contrast to projects such as mechanistic interpretability that aim to understand extremely complex models after they have been formed by opaque architectures (perhaps better expressed by the term mechanistic explainability). This would also be different to interpretability approaches being used for large language models (LLMs) that aim to nudge the agent towards honesty (Bai et al., 2022) or to explain its reasoning steps (The Visible Thoughts Project, 2023). These depend on the ability of the agent to translate its inner workings, which makes the approach more vulnerable to deception.

Core features of inherently interpretable architectures

[JP] These are suggestions for features of inherently interpretable architectures. They are intended as an aspiration and a guideline, rather than a prescription. Indeed, there may be complex, practical obstacles that make it infeasible to perfectly put them into practice.

Research Methodology

[RK] An important aspect of the inherently interpretable architectures framework is not captured in the features of the architecture itself, but in the recommended methodology for research progress: Start out with a small and definitely interpretable architecture, scale it up or expand it a bit, and now try to make predictions about what it will learn and how quickly it will do so, given a certain environment/training data set + objective + training time, then test it. This sort of test of the researchers’ predictive theory is an imperative part of the agenda, as it forces the theoretical work to be a step ahead of the empirical work and puts an upper limit on the increment of confusion. While the standards of what constitutes adequate prediction success are negotiable, this constitutes a relatively clear signal of when one needs to get back to the drawing board of theory, and allows us to demonstrate a level of understanding that is difficult to concretise or form in the current setting (e.g. making a prediction about when a particular cap...
Positive Attractors
by Robert Kralisch — 2023-06-30T20:43:19.416Z

[DL] Positive attractors are perhaps best understood in contrast to negative attractors. Negative attractors are scenarios that lead to failure modes. Some types of negative attractors include deceptive alignment, Sharp Left Turns, and instrumental convergence; these all necessitate one or more bad outcomes for humanity, and conditioning on current alignment research, they do so robustly. In other words, we’ve identified scenarios that converge to failure, and we don’t have any immediate solutions to them.

This “identify failure → attempt to resolve failure” loop is a standard procedure in alignment research. However, we propose an alternative route: instead of starting from failure and aligning from there, is it possible to force scenarios that naturally lead away from failure modes? Do positive attractors exist?

[RK] When we train an AI, the training involves the gradual change of an algorithm towards a target, e.g. minimising a loss function. There is an intractable space of possible algorithms, within which the actual algorithm moves according to the learning mechanisms (of e.g. gradient descent). Further aspects of this algorithm can be selected for, based on the precise architecture or training set for the network.
Danger can occur due to multiple reasons. It could be the case that there are multiple plausible destinations (or local minima) for our algorithm transformation. Some of these destinations (or attractor states) involve problematic qualities beyond the simple minimization of the loss function – it may even be the case that all “great” solutions in terms of the loss function involve some form of perverse instantiation. In addition, there may be safe and unsafe regions for the algorithm to traverse as it changes towards one of these attractor basins. For example, once an AI has “discovered a hack” during training, this will have an influence in steering its further algorithmic development, e.g. if the hack is costly to unlearn.

The intuition for positive attractors is that, if we have the goal of avoiding “dangerous waters”, we should not only try to understand and avoid dangerous regions of the search space, but also actively steer to remain in safe waters, perhaps prevent certain classes of shortcuts, and aim at attractor basins that we understand to have better safety properties.
Rather than relying on our ability to come up w...
Agency from a causal perspective
by tom4everitt — 2023-06-30T17:37:58.376Z

Post 3 of Towards Causal Foundations of Safe AGI, preceded by Post 1: Introduction and Post 2: Causality.

By Matt MacDermott, James Fox, Rhys Ward, Jonathan Richens, and Tom Everitt representing the Causal Incentives Working Group. Thanks also to Ryan Carey, Toby Shevlane, and Aliya Ahmad.

The purpose of this post is twofold: to lay the foundation for subsequent posts by exploring what agency means from a causal perspective, and to sketch a research program for a deeper understanding of agency.

The Importance of Understanding Agency

Agency is a complex concept that has been studied from multiple perspectives, including

social science

,

philosophy

, and

AI research

. Broadly it refers to a system able to act autonomously. For the purposes of this blog post, we interpret agency as goal-directedness, i.e. acting as if trying to direct the world in some particular direction.

There are strong incentives to create more agentic AI systems. Such systems could potentially do many tasks humans are currently needed for, such as independently researching topics, or even run their own companies. However, making systems more agentic comes with an additional set of potential

dangers

and

harms

, as goal-directed AI systems could become capable adversaries if their goals are misaligned with human interest.

A better understanding of agency may let us:

Understand dangers and harms from powerful machine learning systems.
Evaluate whether a particular ML model is dangerously agentic.
Design systems that are not agentic, such as

AGI scientists

or

oracles

, or which are agentic in a safe way.
Lay a foundation for progress on other AGI safety topics, such as interpretability, incentives, and generalisation.
Preserve human agency, e.g. through a better understanding of the conditions under which agency is enhanced or diminished.

Degrees of freedom

(Goal-directed) agents come in all shapes and sizes – from bacteria to humans, from football teams to governments, and from RL policies to

LLM simulacra

– but they share some fundamental features.

First, an agent needs the freedom to choose between a set of options.[1] We don’t need to assume that this decision is free from causal influence, or that we can’t make any prediction about it in advance – but there does need to be a sense in which it could either go one way or another. Dennett c...
Little attention seems to be on discouraging hardware progress
by RussellThor — 2023-06-30T10:14:59.862Z

There is a lot of attention being given to AI software progress, potential regulation etc., however articles such as this celebrating potential progress towards a step change in AI ability seem to get little attention and a free pass in the press as being uncontroversial.

As someone who believes a slow takeoff is likely (though not with enough certainty to be comfortable), I would like to see more emphasis placed on the potential disruption from hardware advancements. I contend that with our current production capacity of roughly 50 million GPUs per year, and given their current specifications, it is unlikely that they could pose an existential risk, regardless of the level of software progress. If this is the case, then slowing or halting the development of more advanced chip factories could be a straightforward way to mitigate such a threat.

From this perspective, potential progress in new hardware could be regarded more akin to advancements in biological, chemical, or sophisticated weaponry technologies, rather than just another feel-good tech story. Even commenting on such articles could have a measurable impact if done consistently and effectively.