trentmkelly/LessWrong-43k huggingface dataset:
When can a mimic surprise you? Why generative models handle seemingly ill-posed problems
by David Johnston — 2022-11-05T13:19:37.384Z

Thanks to Chris Leong and Nora Belrose for their feedback. This is meant to be part of an entry to the Future Fund AI Worldview Competition, but a later post is intended to address the competition questions head on.

In this post, I explore mimics. Mimics are what you get when you join a simulator with a generator. Examples are language models that learn to predict text sequences (the simulator), and generate samples of text sequences from their predictions (the generator). A number of AI safety researchers have mentioned that mimics seem to be safer than "traditional" AI architectures like reinforcement learners, with the proposed reason for this often being that mimics are less "agentic" or "goal-driven" than traditional architecture. moire's Simulators is a particularly thorough overview that makes a similar point.

In this post, I argue that a key feature of mimics is unrelated to their "agentiness": someone who can forecast a mimic's training data can also forecast a mimic's behaviour. I call this phenomenon synchronisation. Synchronisation is possible even when the operator can only forecast some crude features of the training sequence.

Certain methods for fine-tuning mimics allow mimics to be optimised for certain tasks while staying synchronised with the operator. This enables mimics to be controlled in a manner that maintains synchronisation and consequently remain easy to predict.

However, some kinds of objectives do not facilitate synchronised control of mimics. If an operator fine-tunes a mimic to control some feature of the world over which it wouldn't normally have complete control, then the operator should generally expect the mimic's output to diverge from forecasts based on the training data. In practice, the consequences of this divergence is reminiscent of failures due to Goodhart's law.

The extremely brief summary of this post is:

Idealised mimics do what you expect them to when you're trying to control features of their output
Idealised mimics can surprise you when you're trying to control features of the world

Safety relevance

Suppose you've been reading books all your life, and you have a pretty good estimate of how likely a book is to actually be good (by your lights) given it gets a 4.8 star rating on Amazon - and, being a good Bayesian, you represent this with a conditional probability F(Actuallygood|4.8stars). One of the key claims...
My summary of “Pragmatic AI Safety”
by Eleni Angelou — 2022-11-05T12:54:53.532Z

Crossposted from the EA Forum: https://forum.effectivealtruism.org/posts/XxWsAw7DefKipzRLc/my-summary-of-pragmatic-ai-safety

This post is my summary of “Pragmatic AI Safety” on complex systems and capabilities externalities.

To examine AI safety as a complex system means to understand that various aspects of the problem are too interconnected to effectively be broken down into smaller parts and too organized from the perspective of statistics. As a result, traditional methods of reductionism and statistical analysis respectively are inadequate. The insights from complex systems study help reframe the problem of how to make AI safer with consideration for the multiple dimensions of the problem. The first insight is that if we aim at solving alignment, we should have a broader and more inclusive definition of impact. This means that we should be mindful of the value of contributing factors that aren’t, strictly speaking, “direct impact”, i.e., researchers working on the technical/mathematical/engineering safety solutions. The accurate description of systemic factors makes the value of their effect clearer even when such an effect doesn’t point to a specific measurable outcome, e.g., trying a new set of experiments. For AI safety, forecasting and rationality have an evident positive effect that is difficult to measure as “increasing the intelligence of the system” (where the system is the safety community). Analyzing the AI x-risk at the societal level is also fruitful. Bettering people’s epistemics should generally make them better Bayesian thinkers with the obvious benefits that follow. Moreover, how people think about “tail risks” i.e., rare risks is currently a problem in dealing with AI x-risk.

It’s crucial to contextualize the AGI x-risk by developing a safety culture. Safety won’t become the community norm immediately; it’s necessary to have a good understanding of what safety entails as well as develop the infrastructure for AI safety research. The criticisms against AI safety which come both from AI ethics (bias, equality, etc.) and the broader discourse, call for strengthening the safety community and increasing its reliability.

Who gets to choose AI safety research directions is another key contributing factor. So far, it hasn’t been easy to attract top researchers based solely on pecuniary incentives: these people are inter...
Spectrum of Independence
by jefftk — 2022-11-05T02:40:03.822Z

When I compare our parenting decisions to peers and neighbors' we tend
to be near the far end of thinking our kids are ready to do things

on their own. I don't see other
first graders (6y) unaccompanied at the park or

crossing the street, and it was
only this year that our third grader's (8y) friends started ringing
the doorbell to ask if she could go out and play, while she's been
going over and ringing theirs for a couple years. Last year when we
gave permission for our second grader (7y) and kindergartner (5y) to
walk home together the school said yes, but somewhat reluctantly and
they didn't remember this situation with a kindergartner before. [1]

It's reasonably common for people to point at the past for examples of
this level of independence being normal. Reading older children's
books the kids wander everywhere, my dad walked to school and back
alone in kindergarten, and articles comparing child walksheds by
generation are evergreen
viral content. The phenomenon of declining children's
independence over time is well documented. But what I hadn't
internalized until seeing it recently was that even in the US there is
still much more variation than I see in my daily life.

A few weeks ago I brought the
kids along to a dance I was playing, and we stayed with a couple
who lived adjacent to a farm. They're on good terms with the farmers,
and one afternoon they brought us over to look around. While I knew it
was common for farm kids to have more independence and responsibility,
it was different seeing it in person.

The farmers have several kids, and we ran into their 9yo moving
soybeans for storage. They drove past us in a large tractor pulling a
short train of wagons, and lined it up carefully with a hopper. They
had another tractor set up with an unshrouded power take-off
for an auger to get the beans up into the bin. After transferring the
beans in the first trailer, they moved the train up to unload the
second. All of this was completely on their own, and they moved like
someone who had done this many times.

I'm not saying I'd let my kids operate serious machinery unsupervised
at the same age—this is dangerous work even for adults—but
it illustrates how wide a range there is in what different people
consider appropriate and normal levels of independence and makes me
feel less like I have extreme views.

[1] The school is ab...
[paper link] Interpreting systems as solving POMDPs: a step towards a formal understanding of agency
by the gears to ascension — 2022-11-05T01:06:39.743Z
Link: https://www.semanticscholar.org/paper/Interpreting-systems-as-solving-POMDPs%3A-a-step-a-of-Biehl-Virgo/a7c5bdc1c06e497ac0ed7b813e8aaeef52ea9a7b

I found this paper while browsing on semantic scholar - it is so far the only paper known by semanticscholar to cite Discovering Agents, and seems to me like a promising refinement. It's going to take me some time to read it properly, and I'd love to get others' takes on it as well.

Abstract:

Under what circumstances can a system be said to have beliefs and goals, and how do such agency-related features relate to its physical state? Recent work has proposed a notion of interpretation map , a function that maps the state of a system to a probability distribution representing its beliefs about an external world. Such a map is not com-pletely arbitrary, as the beliefs it attributes to the system must evolve over time in a manner that is consistent with Bayes’ theorem, and conse-quently the dynamics of a system constrain its possible interpretations. Here we build on this approach, proposing a notion of interpretation not just in terms of beliefs but in terms of goals and actions. To do this we make use of the existing theory of partially observable Markov processes (POMDPs): we say that a system can be interpreted as a solution to a POMDP if it not only admits an interpretation map describing its beliefs about the hidden state of a POMDP but also takes actions that are optimal according to its belief state. An agent is then a system together with an interpretation of this system as a POMDP solution. Although POMDPs are not the only possible formulation of what it means to have a goal, this nevertheless represents a step towards a more general formal deﬁnition of what it means for a system to be an agent.
Metaculus is seeking Software Engineers
by dschwarz — 2022-11-05T00:42:24.909Z
Link: https://apply.workable.com/metaculus/

(See Metaculus's previous career-tagged posts here, here, and here.) tl;dr Metaculus is now hiring Frontend Engineers and Full Stack Engineers.

Since 2015,

Metaculus

has been a core part of the epistemic infrastructure at the heart of the LW/OB/EA world. I'm posting here because I myself was just hired at Metaculus in part due to one of the above linked posts advertising to LW.

I expect many of you have been following the forecasting space for a long time. We're at an inflection point now, especially at Metaculus as evident from our recent funding and partnerships, but also at a variety of other institutions.

If you're interested in getting involved in forecasting, now's the time. And especially if you're looking for a remote software job at a very fun, mission-driven, and ambitious company.

In particular, we're hiring people to build great experiences for forecasters. In my view this is the greatest unsolved problem in the space: making forecasting natural & engaging.

The link at the top has all of our roles. As of this week we're hiring Frontend Engineers (Senior Frontend Engineer) and Full Stack Engineers (Senior Full Stack Engineer). And if you're more interested in AI, we're also still hiring people directly to study AI risk from a forecasting lens. We're hoping to hire a good number of people very quickly.
Should we “go against nature”?
by jasoncrawford — 2022-11-04T22:14:44.172Z
Link: https://rootsofprogress.org/should-we-go-against-nature

Should we “go against nature”? Or live in “harmony” with it? There are two senses of “nature.” Teasing them apart clarifies this issue.

“Nature” can mean immutable natural law. We defy this at our peril. If we dump raw sewage where we get our drinking water, we will suffer epidemics. If we expose ourselves to radiation, we will die from cancer. If we fail to irrigate our fields, we will go hungry at the first drought. (These are Kipling’s “gods of the copybook headings.”)

But another sense of “nature” is: whatever exists and whatever happens apart from the agency of humanity. It is the chance arrangement of molecules and their motions, before or separate from the conscious, directed, purposefulness of human beings. A river, pursuing its natural course, whether or not it is navigable, whether or not it causes dangerous flooding. A field, with whatever natural level of fertility it happens to have, and whatever plants happen to be growing in it, whether or not they are edible. Wild animals, whether or not they are good companions, whether or not they attack us, whether or not they destroy our crops or our homes, whether or not they carry disease.

This second sense of “nature” is amoral and indifferent to the needs of life. All living organisms survive through an active process of exploiting the resources of their environment. The only difference between humans and other life forms is that we do it using conceptual intelligence. I’ve been admonished that “we are a part of nature.” Of course, we are—but science, technology, and industry are a part of our nature.

To champion “nature” in this sense is not, strictly speaking, to be for anything. It is not in favor of animals or plants, who face a brutal struggle for survival in nature. It is not in favor of rocks or rivers, which are inanimate and have no needs or desires. It is only against. It is against humanity and human agency—choice and purpose—because it is for whatever humans have not done, have not touched, have not interfered with. If pursued with clarity and consistency, it is nihilistic—as in the case of biologist David Graber, who wrote: “I know social scientists who remind me that people are part of nature, but it isn’t true. Somewhere along the line … we quit the contract and became a cancer. We have become a plague upon ourselves and upon the Earth.”

The advocates of “harmony with nature” constantly conf...
How much should we care about non-human animals?
by bokov — 2022-11-04T21:36:57.836Z
Link: https://www.lesswrong.com/posts/hRqop6A6ho2dyhpSC/open-letter-against-reckless-nuclear-escalation-and-use?commentId=AEJwJfvWF2MSpFiYP

This is inspired by a long and passionate post from Bernd Clemens Huber. It was off-topic in its original context so I will respond to it here.

Briefly, Bernd makes a convincing case that most animals living in the wild spend most of their time experiencing fear, hunger, and suffering. He then goes on to say that this imposes an ethical obligation on humans to do everything possible to prevent seeding life on other planets to avoid spreading more suffering. Bernd, please respond and correct me if I'm not accurately summarizing your position.

I see two implicit axioms here that I would like to question explicitly.

What if I don't buy the axiom that suffering is worse than non-existence? It would take a lot of fear and pain for me to consider my life not worth living. I would live as a brain in a jar rather than die. Probably most people place a lower value than I do on continuing to experience life no matter what, but that implies that the value of existence is subjective and you cannot choose for other individuals whether or not their life is worth living. Let alone entire species.
Perhaps the hope that one's descendants will someday escape scarcity and predation like humans have makes one's current suffering "worth it"

What if don't buy the axiom that it's my ethical duty to prevent the suffering of all other beings? What if I'm comfortable with the idea of people in my limited monkey-sphere being the ones whom I'm truly concerned about, and that concern radiating out some distance to strangers because what happens to them could come back to haunt myself and those close to me. The more removed someone is from me, the fewer resources I should expend per unit of their suffering.
Some of LessWrong's famous dilemmas can be seen instead as a reductio ad absurdium argument for my distance-penalized concern model:

They eat their young?!!! Let aliens be alien, as long as it's not our young.
Someone is trying to get me to do something by helping/harming a large number of "copies" of me in some simulation? I let them enjoy their creepy video game, and this instance of me will continue living my life exactly as I have been in this instance of reality.
I have to give up on the dream of my descendants exploring the galaxy and possible also be on the hook to solve the very difficult problem of providing a happy life for the billions of lifeforms alre

...
For ELK truth is mostly a distraction
by c.trout — 2022-11-04T21:14:52.279Z

Epistemic Status: Pretty confident in the central conclusions, and very confident in the supporting claims from meta-logic. Any low confidence conclusions are presented as such. NB: I give an intentionally revisionary reading of what ELK is (or should be) about. Accordingly, I assume familiarity with the ELK report. Summary here.

Executive Summary

Eliciting Latent Knowledge (ELK) collapses into either the automation of science or the automation of mechanistic interpretability. I promote the latter.

Abstract

After reframing ELK from the perspective of a logician, I highlight the problem of cheap model-theoretic truth: by default reporters will simply learn (or search for) interpretations of the predictor’s net that make the teacher’s answers “true” in the model-theoretic sense, whether or not they are True (correspond with reality)! This will be a problem, even if we manage to avoid human simulators and are guaranteed an honest translator.

The problem boils down to finding a way to force the base optimizer (e.g. gradient descent) to pay attention to the structure of the predictor’s net, instead of simply treating it like putty. I argue that trying to get the base optimizer to care about the True state of affairs in the vault is not a solution to this problem, but instead the expression of a completely different problem – something like automating science. Arguably, this is not the problem we should be focused on, especially if we’re just trying to solve intent alignment. Instead I tentatively propose the following solution: train the reporter on mechanistic interpretability experts, in the hope that it internalizes and generalizes their techniques. I expand this proposal by suggesting we interpret in parallel with training, availing ourselves of the history of a predictor’s net in order to identify and track the birth of each term in its ontology. The over-arching hope here is that if we manage to fully interpret the predictor at an earlier stage in its development, we can then maintain that transparency as it develops into something much more complex.

To finish off, I zoom out, outlining three different levels of ELK-like projects, each building on the last, each more difficult than the last.

Terminology

I need to address some conflicts in namespace. I use the noun "model" in three different senses: as standing for logical models, ML models, and final...
Toy Models and Tegum Products
by Adam Jermyn — 2022-11-04T18:51:41.688Z

(Thanks to Adam Scherlis, Kshitij Sachan, Buck Shlegeris, Chris Olah, and Nicholas Schiefer for conversations that informed this post. Thanks to Aryan Bhatt for catching an error in the loss minimization.)

Anthropic recently published a paper on toy models of superposition [Elhage+22]. One of the most striking results is that, when features are sparse, feature embeddings can become divided into disjoint subspaces with just a few vectors per subspace. This type of decomposition is known as a tegum product.

This post aims to give some intuition for why tegum products are a natural feature of the minima of certain loss functions.

Setup

Task

Suppose we’ve got d embedding dimensions and n>d features. We want to embed features into dimensions in a way that minimizes overlap between their embedding vectors. In the simplest case this could be because we’re building an autoencoder and want to compress the features into a low-dimensional space.

One approach is to encode all of the features in all of the dimensions. With this approach there is some interference between every pair of features (i.e. no pair is embedded in a fully orthogonal way), but we have a lot of degrees of freedom that we can use to minimize this interference.

Another approach is to split the d dimensions into k orthogonal subspaces of d/k dimensions. This has the advantage of making most pairs of vectors exactly orthogonal, but at the cost that some vectors are packed more closely together. In the limit where k=1 this reduces to the first approach.

Our aim is to figure out the k that minimizes the loss on this task.

Loss

Suppose our loss has the following properties:

L=∑i≠jℓ(cosθij). That is, the loss decomposes into a sum of terms involving the cosine similarities of feature vectors, and all features are equally important.
ℓ(0)=0. The loss vanishes for orthogonal vectors.
dℓ/dcosθ>0. The loss is greater the more the vectors overlap.

Using these properties, we find that the loss is roughly

L≈n2ℓ(ϵ)2k

where ϵ is the typical cosine similarity between vectors in a subspace.

Loss-Minimizing Subspaces

Edit: The formula for ϵ below is a simplification in the limit of small ϵ. This simplification turns out to matter, and affects the subsequent loss minimization. I've struck through the affected sections below, explain...
Follow up to medical miracle
by Elizabeth — 2022-11-04T18:00:01.858Z

The response to my medical miracle post has been really gratifying. Before I published I was quite afraid to talk about my emotional response to the problem, and worried that people would strong arm in the comments. The former didn’t happen and the latter did but was overwhelmed by the number of people writing to share their stories, or how the post helped them, or just to tell me I was a good writer. Some of my friends hadn’t heard about the magic pills or realized what a big deal it was, so I got some very nice messages about how happy they were for me.

However, it also became clear I missed a few things in the original post.

Conditions to make luck-based medicine work

In trying to convey the concept of luck-based medicine at all, I lost sight of traits I have that made my slot machine pulls relatively safe. Here is a non-exhaustive list of traits I’ve since recognized are prerequisites for luck based medicine:

I can reliably identify which things carry noticeable risks and need to be assessed more carefully. I feel like I’m YOLOing supplements, but that’s because it’s a free action to me to avoid combining respiratory depressants, and I know to monitor CYP3A4 enzyme effects. A comment on LessWrong that casually suggested throwing activated charcoal into the toolkit reminded me that not everyone does this as a free action, and the failure modes of not doing so are very bad (activated charcoal is typically given to treat poison consumption. Evidence about its efficacy is surprisingly equivocal, but to the extent it works, it’s not capable of distinguishing poison, nutrients, and medications).

This suggests to me that an easy lever might be a guide to obvious failure modes of supplements and medications, to lower the barrier to supplement roulette. I am not likely to have the time to do a thorough job of this myself, but if you would like to collaborate please e-mail me (elizabeth@acesounderglass.com).

A functioning liver. A lot of substances that would otherwise be terribly dangerous are rendered harmless by the human liver. It is a marvel. But if your liver is impaired by alcohol abuse or medical issues, this stops being true. And even a healthy liver will get overwhelmed if you pile the load high enough, so you need to incorporate liver capacity into your plans.

A sufficiently friendly epistemic environment. If it becomes common and know

...