trentmkelly/LessWrong-43k huggingface dataset:
The limited upside of interpretability
by Peter S. Park — 2022-11-15T18:46:15.547Z

TL;DR: A strategy aiming to elicit latent knowledge (or to make any hopefully robust, hopefully generalizable prediction) from interpreting an AGI’s fine-grained internal data may be unlikely to succeed, given that the complex system of an AGI’s agent-environment interaction dynamics will plausibly turn out to be computationally irreducible. In general, the most efficient way to predict the behavior of a complex agent in an environment is to run it in that exact environment. Mechanistic interpretability is unlikely to provide a reliable safety plan that magically improves on the default strategy of empiricism. Coarse-grained models of the complex system have a realistic chance of making robust predictions out-of-distribution, although such predictions would then necessarily be limited in scope.

The paradigm of deep learning, potentially with a small extra step, seems likely sufficient for creating AGI. This is because AI capabilities seem likely to continue increasing, given empirical scaling laws, the decreasing cost of compute, and AI corporations’ unabating desire to build increasingly powerful AI. The most recent findings from neuroscience also suggest that across different species, scale is the causal factor behind most general capabilities.

Biological systems like those of evolutionary biology and neuroscience are plausibly the best reference class for large-scale deep-learning models we have. To quote arguably the world’s leading expert in interpreting neural nets, Chris Olah:

“The elegance of ML is the elegance of biology, not the elegance of math or physics.

Simple gradient descent creates mind-boggling structure and behavior, just as evolution creates the awe inspiring complexity of nature.”

For a more detailed case, see (1) Olah’s excellent writeup "Analogies between biology and deep learning" as well as (2) the review paper by Saxe et al. about the analogy between deep learning and systems neuroscience.

Computational irreducibility

Predicting the dynamics of biological systems is often computationally intractable. To quote "The limits to prediction in ecological systems" by Beckage et al.:

“An ecological system changes through time, updating its state continuously, and the process of system evolution can be thought of as computation…Our use of the term ‘system evolution' is much broader in meaning than biological

...
Why bet Kelly?
by AlexMennen — 2022-11-15T18:12:05.981Z

Suppose you know what outcomes are better than what other outcomes, but not by how much. Given two possible actions you could take, A and B, if you know what probability distributions over outcomes each of them results in, this doesn't necessarily help you pick which is better. Being able to compare outcomes isn't enough for you to be able to compare probability distributions over outcomes.

But what if I told you that the worst outcome that could possibly occur if you pick A is better than the best outcome that could possibly occur if you pick B? Now the choice is very easy. There are no trade-offs, and you don't need to do a risk analysis. A is just better.

Now what if it's not quite true that every possible outcome of A is better than every possible outcome of B, but it's almost true, in the sense that there's some threshold such that the probability that A is better than that threshold is almost 1, and the probability that B is worse than that threshold is almost 0? For instance, let's say the 0.01%ile outcome if you pick A is better than the 99.99%ile outcome if you pick B. Then it's not quite as clear-cut. Maybe most of the time A is only slightly better than B, but the tiny possibility of A being worse than usual involves A being enormously worse than usual, or the tiny possibility of B being better than usual involves B being enormously better than usual. But this is still a pretty big hint that A is better in expectation. A is better unless your expected utility calculations are radically altered by negligible-probability tail events.

When to act like your utility is linear

Let's suppose from now on that the outcomes you're comparing are how much money you have. Of course, how rich you are is far from a perfect metric for measuring the extent to which you're achieving your goals, but that's okay; money is still a thing that people generally want more of, and we can assume that everything important that's not downstream of wealth is being held constant.

Suppose you're given a long sequence of opportunities to make bets. On each step, you can choose among some set of options for what bet to place, and there will be a maximum stakes you'll be able to bet at each time, because whoever or whatever ...
Entropy Scaling And Intrinsic Memory 
by Alexander Gietelink Oldenziel — 2022-11-15T18:11:42.219Z

This is the second post in a series that explores a circle of ideas around entropy and information processing in complex systems. We hope it will form an accessible introduction to James Crutchfield's Computational Mechanics framework.

In the previous post we saw that

Data is, in general, a mixture of randomly generated components and deterministic hidden structure
Kolmogorov Complexity and Shannon Entropy are powerful measures of structure. However, we also encountered some limitations: Kolmogorov complexity overestimates deterministic structure while the Shannon Entropy underestimates deterministic structure.
We need a framework that allows us to find and characterize both structured and random parts of our data.

In this post we will show that we can characterize both structure and randomness in a system by looking at how entropy (irreducible uncertainty) changes over increasing scales. This will lead to ways in which we can understand transformer function in language tasks.

Structure is found in how entropy scales

We want something conceptually in between the randomness of Shannon Entropy and the determinism of Kolmogorov complexity. The way to bridge the gap is by realizing that systems that have deterministic structure at large scales look random from the perspective of entropy at short scales, but look less and less random as we increase the scale. In other words, if we can't see far or remember a lot, then things seem random, and as we see farther and remember more, structure is revealed. And what's more is that the way randomness turns into structure at longer scales tells us many computational properties of the a system. Let's see how this works.

Imagine we come across some mystery data:

...00110100100100110110110100110100100100110110110100100110110100...

Let's assume that we don't know how it was generated.

Notice that the frequency of 0 and 1's is equal in our string.

So maybe it's just a random string generated by a fair coin? After all, if we compare to the distribution of 0s and 1s of a string generated by a fair coin we get the same results.

That's a good initial guess but let's increase our scale and look at the length 2 blocks as well. By length 2 blocks we mean here the distribution over 00, 01, 10, and 11 in our data.

Notice that the 01 block occurs more often than the 00 block, so the statistics of our data are inconsistent with...
Will nanotech/biotech be what leads to AI doom?
by tailcalled — 2022-11-15T17:38:18.699Z

I don't know much about nanotech/biotech, but what little I know suggests that this will be the earliest failure point where AI can cause doom for humans. I thought that because of this, I should start learning more about nanotech/biotech, and I thought that asking LessWrong for direction might be a place to start.

My heuristic for why nanotech/biotech is critical, and for why I am lumping them together:

AI doom due to misalignment is arguably about the "attack surface"; that is, if there is a high-dimensional way to have big effects on humans, then various standard arguments about the importance of alignment will apply. This seems to be the case for nanotech/biotech, in the sense that there are many kinds of germs, nanobots, chemicals, and so on that we could imagine could be made if we had the correct technological development, and these would exist in ~the same environment as people (e.g. they could get released into the atmosphere). Human civilization essentially makes the assumption that people can safely breathe the atmosphere, but that assumption could be broken by nanotech/biotech.
Nanotech/biotech is not the only thing with an exponentially high-dimensional attack surface; there's also social networks, computer networks, probably more than that. However, nanotech/biotech seems to have the "advantage" of being small-scale; it can equillibriate in milliseconds-to-minutes, and can exist in cubic millimeters to cubic meters, which makes it much more feasible to model and collect data on than grand societal things. This suggests that you would not need all that advanced of an AI to amplify nanotech/biotech. It doesn't even need to be a general intelligence, it just needs to come up with more powerful ways of doing nanotech/biotech. So AI-powered nanotech/biotech seems likely to arrive years if not decades before AGI. (Similar to how people see GPT-3 as the precursor to Prosaic AGI, think of AlphaFold 2 as the precursor to AI-powered biotech.)
Incidentally, this also makes it much harder to align. A common counterargument to AI x-risk is "wouldn't a superintelligence understand that we don't want it to do bad stuff?", after which a common counterreply is "yeah, but we need some way of specifying that it should care about what we want". However, an AI that doesn't understand large-scale things such as humans and our wants might still understand small-scale stu

...
Value Formation: An Overarching Model
by Thane Ruthenis — 2022-11-15T17:16:19.522Z

0. Introduction

When we look inwards, upon the godshatter, how do we make sense of it? How do we sort out all the disparate urges, emotions, and preferences, and compress them into legible principles and philosophies? What mechanisms ensure our robustness to ontological crises? How do powerful agents found by a greedy selection process arrive at their morals? What is the algorithm for value reflection?

This post seeks to answer these questions, or at least provide a decent high-level starting point. It describes a simple toy model that embeds an agent in a causal graph, and follows its moral development from a bundle of heuristics to a superintelligent mesa-optimizer.

The main goal of this write-up is to serve as a gears-level model — to provide us with a detailed step-by-step understanding of why and how agents converge towards the values they do. This should hopefully allow us to spot novel pressure points — opportunities for interventions that would allow us to acquire a great deal of control over the final outcome of this process. From another angle, it should equip us with the tools to understand how different changes to the training process or model architecture would impact value reflection, and therefore, what kinds of architectures are more or less desirable.

Let's get to it.

1. The Setup

As the starting point, I'll be using a model broadly similar to the one from my earlier post.

Let's assume that we have some environment E represented as a causal graph. Some nodes in it represent the agent, the agent's observations, and actions.

Every turn t (which might be a new training episode or the next time-step in a RL setup), the agent (blue node) reads off information from the (green) observation nodes O, sets the values for the (red) action nodes A, all nodes' values update in response to that change, then the agent receives reward based on the (purple) reward nodes R. The reward is computed as some function U:Rt→R, where Rt represents the reward nodes' current values.

The agent is being optimized by some optimization/selection process — the SGD, evolution, human brain reward circuitry, whatever. What matters is that this process is non-intelligent and greedy: it only ever makes marginal improvements to the agent's architecture, with an eye for making it perform marginally better the next turn.

As per the prev...
Internal communication framework
by rosehadshar — 2022-11-15T12:41:23.316Z

Usually when we think about humans and agency, we focus on the layer where one human is one agent.

We can also shift focus onto other layers and treat those entities as agents instead - for example, corporations or nations.

Or, we can take different parts within a human mind as agenty -- which is a lens we explore in this post.

In this post, we first introduce a general, pretty minimalist framework for thinking about the relationship between different parts of the mind (Internal Communication Framework, or ICF). Based on this framework, we build a basic technique that can be used to leverage ICF, and which we believe can be helpful for things like:

Resolving inner conflicts
Building internal alignment
Increasing introspective access

We have shared ICF with a small number of people (~60-80) over a few years, and are writing this post partly to create a written reference for ICF, and partly to make the framework accessible to more people.

The framework

Some background context:

ICF is rooted in thinking about hierarchical agency (i.e. how agency, and strategic dynamics, play out composite agents and their parts).
It is mostly an attempt to generalise from a class of therapy schools and introspection techniques working with parts of the mind, and to make the framework cleaner from a theoretical perspective. (More on the relationship between ICF and some particular techniques from that class here
ICF was developed in 2018 by Jan Kulveit with help from Nora Amman, in the context of the Epistea Summer Experiment and then CFAR instructorship training.

ICF makes two core assumptions:

The human mind is made of parts
Interactions between parts, and between parts and the whole, work best when the interactions are cooperative and kind

There is a possibly interesting discussion about the epistemic status of these assumptions, which is not the point of this post, so we invite you to interpret them in whatever spirit you like - as a model which is somewhat true, a model which is wrong but useful, a metaphor for processing your experience…

The human mind is made of parts

These parts have different goals and models of the world, and don't always agree. They are still all part of your mind.

If we make this assumption that the mind is made of parts, it becomes less obvious what we mean by terms like ‘you’ or ‘the self’. One metaphor for thinking about this is to view the whole set of parts as som...
Better Mastodon Aliases
by jefftk — 2022-11-15T12:10:01.866Z

When I

joined Mastodon I was

@jefftk@schelling.pt. A few days ago that server had an
incident and I moved to

mastodon.mit.edu, becoming

@jefftk@mastodon.mit.edu. This got me thinking about a
wart in the current Mastodon design: while you can normally move
followers with you if you switch instances, this requires the
cooperation of the instance you're leaving. In this case that
instance couldn't cooperate because its domain name had expired, but I
could also imagine someone having a falling out with their local
admins and getting kicked off in a way that precluded transfer.

Ideally this would be like email: I send and receive email as
jeff@jefftk.com. The actual server I use is
gmail.com, but the people I interact with don't need to
know that. If I ever want to move to a different email server I don't
need Gmail's permission and I don't need to tell my contacts; I can do
it entirely on my end. Similarly jefftk.com currently
points to a Contabo VPS, but if I had an issue with them I could move
anywhere just by adjusting my domain settings.

Current Mastodon is partway there: I
can give out @jeff@jefftk.com, which uses a webfinger file
(thanks, Chris)
to redirect @[anything]@jefftk.com to
@jefftk@mastodon.mit.edu. But then the original address
is forgotten, and it only records that they're following my
mastodon.mit.edu account.

One way for this to work would be that when following someone you
store both their current account name and a canonical account name.
Almost everything would use the former, but once a day in your
instance would check whether the canonical account still pointed to
the same place and adjust if necessary.

(For more discussion along these lines see Issue 3796.)

Comment via: facebook, mastodon
The economy as an analogy for advanced AI systems
by rosehadshar — 2022-11-15T11:16:03.750Z

Eric Drexler’s Comprehensive AI Services (CAIS), particularly as set out in his 2019 report Reframing Superintelligence, is a complex model with many different assumptions and implications. It’s hard to grasp the whole thing at once, and existing summaries are brief and partial.[1]

One way of trying to understand CAIS is to seek generative intuitions for the whole model. These intuitions will be imprecise, but they can also make it easier to see why one might end up thinking that something like CAIS made sense.

In this post, we offer one such generative intuition for CAIS: using the economy rather than rational agents as an analogy for thinking about advanced AI systems.

Note that:

We are not making the historical claim that thinking about economies was in fact the main generator of Drexler’s thinking on CAIS.[2]

There are other generative intuitions for CAIS, and other bodies of theory which the CAIS model is rooted in.[3]

The basic analogy

An economy is an abstraction for the sum total of ‘the production, distribution and trade, as well as consumption of goods and services’. Prescriptively, we want the economy to serve human needs and preferences - and it does this at least to some extent.

Prescriptively, we also want advanced AI systems to serve human needs and preferences.

In worlds where we get advanced AI systems right, they would therefore be serving a similar function to the economy: serving human needs and preferences.

Whether we get AI right or not, it seems likely that advanced AI systems will become heavily integrated with the economy, such that it might become hard to distinguish them.

It therefore seems reasonable to imagine advanced AI systems in analogy with the economy, and to use what we know about economic dynamics to reason about dynamics which might shape those systems.

In the modern economy, specialised services are provided by a range of entities, mostly companies and governments. We don’t see one giant global monopoly providing all services.

Thinking analogically about advanced AI systems, the CAIS model expects an array of specialised AI services working on decomposed tasks, rather than a single generally superintelligent agent (a global monopoly in the base metaphor).

This can be further unpacked. The reason that the human economy isn’t structured as a global monopoly is that specialisation is efficient. It’s often cheaper for an organisation to ...
We need better prediction markets
by eigen — 2022-11-15T04:54:20.524Z

I have 0 exposure to cryptocurrencies and I don't work in that industry, but the recent news about FTX plummeting to the ground have left me thinking quite a bit.

It's extremely surprising how a person can go from one day being considered a "genius" or a shrewd business man to being a world wide piece of shit, conman, idiot in some aspects and overall hated by everyone. With this hindsight, now, we can tell that everything that Bankman-Fried said was a lie... he never did tell the truth.

Plus the arrogance that comes from the deep insecurity of having a bad hand where the only choice is to double down every time until you get caught (he was giving product advice to Stripe CEO while alt-tabbing on an interview; now, who is this guy really...) I feel like this could happen again, and to anyone. And, looking back it's obvious, this guy was offering 3 billion dollars to buy a piece of Twitter to a surprised Musk saying, "Does Sam have 3b liquid?"

As soonest as that message did come out it should have been aggregated, it should have moved the needle strongly towards the insolvency of this company and hopefully save thousands of users from having their funds locked. The same goes for any centralized exchange functioning right now, this lowers the probability that they will be able to sustain a bank-run (by different amounts.)

We have to find ways in which the skepticism of people are accounted for and people that were right are rewarded, in the case of FTX this would have been a service to the tune of a billion dollar value.

This will happen again, how can we save the next thousands of users and the next billion of dollars?
Preventing, reversing, and addressing data leakage: some thoughts
by VipulNaik — 2022-11-15T02:09:22.482Z

In the last few months, I've been thinking about the problem of
accidental leakage of data and how to prevent it, reverse it, and
address its aftermath. This post includes various thoughts, some of
them including tool-specific guidance, and some of them more general.

Examples of data leakage are as follows:

Sending a password or sensitive credential to somebody you shouldn't
send it to, or posting it in a forum with wide, even public, access.

Sending factual information accidentally to somebody you didn't
intend to share it with (or at least, didn't intend to share it with
yet) or putting it in a forum with wide, even public, access.

The prevention steps are fairly similar for both cases, but the steps
for addressing are somewhat different. Specifically, for the case of
passwords or sensitive credentials, changing it is part of the best
practice response. For factual information, on the other hand, it is
often infeasible to change the facts since they're the facts!

This post is largely focused on leakage in the digital realm. Some
similar ideas apply in the physical realm as well.

This post also doesn't cover large-scale data leakage, nor does it
cover complementary best practices such as password security, other
forms of authentication security (such as IP-based limits and
two-factor authentication) and encryption. I might write additional
posts about some of those topics, but those topics are in general more
widely covered, hence my desire to write what I'm writing first.

I end the post with meta comments including more on its potential
relevance to LessWrong as well as the distinction between general
ideas and details specific to particular tools and services.

Prevention strategies: general philosophy

The accident triangle philosophy and the conjunctive nature of accidents

The accident
triangle idea is
that for every accident with major injury, there are several accidents
with minor injury and even more accidents with no injury. A similar
idea applies here: for every case of leaked sensitive data, there are
several cases where something went wrong, but sensitive data didn't
end up being leaked. Since sensitive data leakage is a
low-probability-but-high-cost occurrence, we don't get a lot of data
points for it.

One underlying insight here is that a major failure such as accidental
data leakage is often conjunctive: for instance, an accidental data
leakage...
Winners of the AI Safety Nudge Competition
by Marc Carauleanu — 2022-11-15T01:06:27.846Z

The AI Safety Nudge Competition aims to encourage people to do things related to AI Safety today instead of procrastinating it into the future by allowing them to enter into a draw if they complete the goal that they set for themselves.

Participants defined a specific goal for themselves, examples include:

• Finish reading Superintelligence

• Finish writing up a relevant blog post

• Organize a local dinner for people interested in AI Safety

If they completed the goal they set out for themselves, they were entered into the draw:

• ten prizes of $100

• two prizes of $100 specifically for Australia and New Zealand

The 10 winners of the prize draw:

Jim Vine

Lili Emery

Varsha Raghavan

Brayden McLean

Ashwin Mahadevan

Aishwarya Gurung

Philippe Rivet

Peter Chatain

Jacek Karwowski

Matthew Chiu

The 2 winners of the prize draw specifically for AU/NZ applicants:

Hannah Betts

Lachlyn Keiller

Congratulations to the winners and to everybody that committed to finishing their AI Safety projects this October!