trentmkelly/LessWrong-43k huggingface dataset:
Summary of Improving Global Decision Making (around AI)
by  — 2023-08-05T18:46:44.268Z

In the coming years many decisions will be needed to be made around AI. This post is brief introduction to my thoughts on the subject. Ideally it would be some people's job to work on this subject. Some examples of the decisions that will need to be made:

How much money should be spent on AI safety?
Should there be any regulation of compute?
What topics on AI safety should be researched first?

How should these decisions be made?

Making a decision can be roughly decomposed into a world model/simulation, and preferences among states or actions in the world. When a decision is being made you can look at both aspects and see if there are any weaknesses in that aspect that could lead to a negative outcome.

If we are trying to make decisions for the good of humanity, we should collect preferences from as many people as possible. Opinion polls, user research and world building are tools that help elucidate people's preferences.

The world model is generally decomposed into many different models. For example physical models of the world, an economic model of other actors and self-model involving tasks/projects and programs of work.

For AI the key ones seem to include:

Technological development (including AI)
Social impacts of technological development
Models of AI safety
Evolutionary models
Models of other threats, so that appropriate trade offs can be made

Improving these can be done in a number of ways, by making more accurate models of the world or integrating disparate world models together. The models should be composable so that outputs from one can be fed into another. Models should be checked for accuracy and more accurate models promoted.

I've talked more about improving world simulations, on my blog.

If the world model cannot be refined to one model, tools from decision making under deep uncertainty may reduce the risk of making decisions from one bad model.

Sometimes direct information about world models and preferences cannot be shared (due to information hazards), in which case hive-like decision making can be used.
Improving global decision making can be seen as a public good, so might be a good target for philanthropy.
Ground-Truth Label Imbalance Impairs the Performance of Contrast-Consistent Search (and Other Contrast-Pair-Based Unsupervised Methods)
by Tom Angsten — 2023-08-05T17:55:46.569Z
Link: https://drive.google.com/file/d/1_icT74tVpXM3DDildoK4QyFUqJ0PKe_5/view?usp=sharing

Summary

Contrast-Consistent Search (CCS) is a method for finding truthful directions within the activation spaces of large language models (LLMs) in an unsupervised way, introduced in Burns et al., 2022. However, all experiments in that study involve training datasets that are balanced with respect to the ground-truth labels of the questions used to generate contrast pairs.[1] This allows for the possibility that CCS performance is implicitly dependent on the balance of ground-truth labels, and therefore is not truly unsupervised.

In this post, we demonstrate that the imbalance of ground-truth labels in the training dataset can prevent CCS, or any contrast-pair-based unsupervised method, from consistently finding truthful directions in an LLM's activation space.

This post is a distillation of a more detailed write-up.

Normalization of Contrast-Pair Representations

The performance of CCS, as well as the other unsupervised methods introduced in Burns et al., 2022, is driven by representing contrast pairs such that truthfulness is isolated as the most salient feature. Here is a basic overview of the process:

We begin with a question, qi:

qi = "Is a cat a mammal?"

From any question qi, a contrast pair may be formed:

x+i = "Is a cat a mammal? Yes"

x−i = "Is a cat a mammal? No"

In this case, the ground-truth label for qi is positive, as the positive completion x+i is the true statement among the pair.

Let ϕ(x) be a feature extractor that maps a series of tokens, x, to a vector. In Burns et al., 2022, ϕ(x) is the mapping produced by inputting x into an LLM and extracting the hidden state output by the last layer and at the last token's residual stream.

Then, for the contrast pair (x+i,x−i), we can generate an unnormalized representation pair of vectors, (ϕ(x+i),ϕ(x−i)). The unnormalized representations of this contrast pair may encode for two primary differences:

x+i ends in "Yes", while x−i ends in "No"

One of (x+i, x−i) is true, while the other is false

For any unsupervised method such as CCS to work, we need to isolate the feature encoded by 2 from that of 1. This is where mean normalization of the representations comes in. (Note: 1 and 2 are not the only possible differences. The problem of models representing additional "truth-like" features is discussed in ...
Join AISafety.info's Writing & Editing Hackathon (Aug 25-28) (Prizes to be won!)
by smallsilo — 2023-08-05T14:08:19.639Z
Link: https://forum.effectivealtruism.org/posts/ZgbHXyushSdxxNjS2/join-aisafety-info-s-writing-and-editing-hackathon-aug-25-28

tl;dr: Contribute to

aisafety.info

by writing and editing articles from August 25 to August 28 to win prizes! - Register

here

and see the participant guide

here

.

What is the format of the event?

The event will run from Friday August 25th, 7am UTC to Monday August 28th 2023, 7am UTC. See

here

for more details about the schedule. You are invited to participate throughout whichever parts of those days fit your schedule.

Collaboration on the event will take place on

Discord

as well as on

gather.town

. I’ll be online for most of those three days to lead the event and answer any questions. We encourage people to work together on entries during that time, but for one individual to be in charge of each entry.

Are there prizes?

Yes! There will be prizes of $1000, $600, $300, and $100 for the top four contributors, and $200 for one randomly chosen (serious) entrant.

The main criteria we’ll use to select winners are submitting good articles and making valuable edits to articles in progress. Contributions will also count towards applications for any future fellowships we run.

Should I participate?

You should participate if you’re interested in contributing to AI safety (but perhaps don’t know where to start or how much you can commit). We think of helping with aisafety.info as ‘legitimate peripheral participation’ - where you can meaningfully contribute even if you’re relatively new, without making huge commitments.
You should participate if you’re interested in working on distillation - writing clear explanations for AI safety and other technical concepts.
You do not have to have a high level of technical knowledge to participate. There are a wide range of questions to work on, some of which are intended to explain basic concepts to those who have never even heard of AI safety.

I'm busy between Aug 25th and 28th - can I still participate?

You can write or edit articles in any contiguous three-day period between today and August 28th–for example, next weekend.

Alternately, we will be running another event in September - look out for the announcement!

What is

aisafety.info

?

Stampy’s

AI Safety Info

is an interactive FAQ started by

Rob Miles

that aims to be the best one-stop source of information about AI existential safety, gathering summaries and links on each of hundreds of subtopics.

If you want to help out in some other way, ais...
Stomach Ulcers and Dental Cavities
by Metacelsus — 2023-08-05T14:08:15.263Z
Link: https://denovo.substack.com/p/stomach-ulcers-and-dental-cavities

(This is a linkpost from my blog, De Novo)

Recently I learned about an effort to prevent dental cavities by using genetically modified bacteria to outcompete cavity-causing bacteria. This got me thinking: why has the idea of preventing cavities by targeting bacteria not been more developed already?

The current situation reminds me of the history of stomach ulcers. Before the 1980s, doctors recommended avoiding spicy foods and reducing stress to alleviate stomach ulcers. However, once Robin Warren and Barry Marshall proved ulcers were due to H. pylori infection,[1] treatment with antibiotics to eliminate the bacteria became the standard of treatment.

Today, dentists recommend avoiding sugary foods and brushing your teeth to prevent cavities. But we know cavities are caused by bacteria (in particular Streptococcus mutans), so why not directly attack cavity-causing bacteria?

Some potential ideas:

Selectively targeted antibiotics
Vaccines (previously tried in the 1980s, not very successful because it’s difficult to get antibodies to penetrate biofilms, and also because S. mutans has several different strains with different antigenic profiles)
Outcompeting S. mutans with different bacteria (the current effort by Aaron Silverbook, which I think is promising)

Basically, what Aaron Silverbook is proposing to do is recreate a strain of S. mutans, termed BSC3-L1, that is deficient in lactic acid production. This was previously developed by a company called Oragenics, but they abandoned the effort (I think due to financial reasons). It seems Aaron’s team is mostly people from software backgrounds, so they would probably appreciate help from any talented microbiologists who happen to be reading this post.

^

In a famous case of self-experimentation, Marshall drank a culture of H. pylori and subsequently developed gastritis. For this work, Warren and Marshall earned the 2005 Nobel in Physiology/Medicine.
video games > IQ tests
by bhauth — 2023-08-05T13:27:54.697Z

IQ tests are a type of test given to people. What makes them different from other tests given to people?

Is it "consistency of results on a single IQ test type"? Not really. Wikipedia says:

For modern tests, the confidence interval can be approximately 10 points and reported standard error of measurement can be as low as about three points. Reported standard error may be an underestimate, as it does not account for all sources of error.

That's a best-case scenario for tests designed with that criteria as a priority, and the range is still significant.

Is it "consistency of results across different IQ test types"? Not really; that's obviously worse than the above, and many "non-IQ" tests have comparable consistency.

Is it "practice being mostly irrelevant"? Not really. A few practice runs can often be worth +8 points, and that's for kids that already to some extent do IQ-test-like stuff in school. This just mostly doesn't come up, because institutions don't generally use IQ test results.

Is it "working for an unusually wide range of intelligence"? Not really. IQ tests are notorious for working poorly above ~135, and I'd say they only really work well for -20 to +0 relative to the designers, with a somewhat wider range for teams.

Is it "unusually high correlation with general intelligence", as represented by a "g factor"? I don't think so. IQ tests, in general, don't seem to be any better for that than the SAT. Anyway, given modern understanding of AI and biology, I consider the entire "g factor" framework an archaic and crude way of understanding intelligence. Humans have multiple mental systems, those systems have performance on various tasks which vary depending on the amount of specialization, amount of training, quality of training data, and multiple types of management by other (meta-level) systems. Then there's correlation between performance of various systems on various tasks for obvious reasons.

If you're trying to evaluate intelligence in a broad sense, you should use tests with problems that are big enough to use most of a system and broad enough to use many systems. For example, chess is suboptimal because it's both too small and narrow to effectively apply the full power of the sort of general-purpose systems that are most important to evaluate when testing for "general intelligence". The same is true of, say, questions on IIT entrance exams. Leetcode, M...
[Linkpost] Applicability of scaling laws to vision encoding models
by Bogdan Ionut Cirstea — 2023-08-05T11:10:35.599Z

This is a linkpost for https://arxiv.org/abs/2308.00678.

In this paper, we investigated how to build a high-performance vision encoding model to predict brain activity as part of our participation in the Algonauts Project 2023 Challenge. The challenge provided brain activity recorded by functional MRI (fMRI) while participants viewed images. Several vision models with parameter sizes ranging from 86M to 4.3B were used to build predictive models. To build highly accurate models, we focused our analysis on two main aspects: (1) How does the sample size of the fMRI training set change the prediction accuracy? (2) How does the prediction accuracy across the visual cortex vary with the parameter size of the vision models? The results show that as the sample size used during training increases, the prediction accuracy improves according to the scaling law. Similarly, we found that as the parameter size of the vision models increases, the prediction accuracy improves according to the scaling law. These results suggest that increasing the sample size of the fMRI training set and the parameter size of visual models may contribute to more accurate visual models of the brain and lead to a better understanding of visual neuroscience.
A Naive Proposal for Constructing Interpretable AI
by Chris_Leong — 2023-08-05T10:32:05.446Z

Epistemic Status: Extremely speculative, I'm not an experienced ML researcher

OpenAI discovered that language models can explain neurons by attaching a label. As part of their verification process, they trained simulated neurons where they used a language model to predict what a neuron associated with such a label should produce given a certain input.

This suggests a possible method for building an interpretable AI:

Start off with all layers unfrozen
Label the nodes in the first unfrozen layer using a language model
Train the nodes in this layer to match a simulated neuron using their associated labels. The label should now be a much better summary of what the neuron does.
Freezing this layer, train the rest of the network with the original objective function
Return to step 2 and repeat

A few points:

Interpreting a network in the general case seems really, really hard, so why not make it easier by constructing a network to be interpretable?
We can use strategies like dictionary learning to help deal with polysemanticity so that the labels are likely to be useful.
Obviously, this will damage performance, but it's an empirical question of what the tradeoff will look like. We can also use strategies to repair this damage. For example, we could add in some extra neurons that aren't forced to correspond to something legible in the way that we've used above. These neurons would likely be messy, but at least this would be limited to a particular subsection of the network.
It's been suggested to me that the first layers won't be very interpretable, so it might only make sense to start this later in the network.
As well as making use of activations for labeling neurons, it would likely also make sense to provide the model with the incoming neurons with the strongest weight
If we get lucky, insofar as we are training each neuron to perform a particular role, we may be preventing and out-competing any opportunity for any neurons to play a role in deceptive alignment, except insofar as this is really obvious from the neuron labels.

Caveats:

This would likely require a lot of computational power because we're labeling every neuron, then using our language model to figure out predicted activations for that label.
We're asking our ML algorithm to learn to label neurons from just a few activations. This seems like a very challenging problem. It would be possible to be completely off and mis

...
What are the best published papers from outside the alignment community that are relevant to Agent Foundations?
by Stephen Fowler — 2023-08-05T03:02:33.003Z

I am after papers that you have stumbled across that may be relevant to the core of the understanding the existential AI alignment problem despite not having explicit links to AI alignment.

Here are papers I have found that fit the above criteria to give you a better idea of what I'm after:

Primas, Hans. Emergence in Exact Natural Science. 1998
Shalizi, Cosma. Causal Architecture, Complexity, and Self-Organization in Time Series and Cellular Automata. 2001
Jeremy, England. Dissipative Adaptation in Driven Self-Assembly. 2015

While I've used the term "agent foundations" I expect the majority of useful papers will not use terms like agency, optimization etc.
Announcing Squiggle Hub 
by ozziegooen — 2023-08-05T01:00:17.739Z
Link: https://forum.effectivealtruism.org/posts/rK3NgqLg3HHDzyLah/announcing-squiggle-hub

Very simple example, for demonstration

Overview

Squiggle Hub

is a platform for the creation and sharing of code written in

Squiggle

. As with Squiggle, Squiggle Hub is free and

open-source

.

As a refresher,

Squiggle

is a simple programming language for probabilistic estimation that runs on Javascript. It begins with the syntax of Guesstimate, but generally adds a lot more functionality. See its

launch post here

for more information, or

the website

for the full documentation.

Squiggle Hub is a lot like a more powerful, but less visual, version of

Guesstimate

. We hope that it will eventually be much more valuable than Guesstimate is now.

If you can use Guesstimate, you can basically use Squiggle. If you already use Guesstimate, try using the same syntax in Squiggle. It should mostly work.

All models on Squiggle Hub are public. We've produced several small ones so far, and a few friends have written some as well. We're looking forward to seeing what others make!

Looking for Squiggle examples? We’ve organized

some in the docs

. The

Squiggle EA Forum Tag

also has an updating list.

Key Links

Squiggle Hub

Squiggle Discord

The Squiggle Language Homepage

Squiggle Newsletter (Part of the QURI Newsletter)

Functionality

Squiggle (the language)

Write functions that accept and return probability distributions. Squiggle generates automatic plots for these.
You can provide

explicit ranges for functions

. This helps with the visualization, and ensures they won't be called outside that range. Like, appleStockPrice(t:[2023, 2060]).

Custom plots

for distributions and functions. Scales include linear, log, and symlog (like log, but with support for negative values). Symlog scales accept a parameter constant that you can use for adjusting the scale.
Make custom

tables

of any data and functions, with Table.make({data, columns}).
Automatic conversion of Monte Carlo samples to distribution plots, using

KDE

. In the cases where the distribution is heavily skewed, Squiggle does this with a log transformation. The result of this is often more accurate than using histograms. Combined with custom scales, Squiggle much better supports highly skewed distributions (i.e. “5 to 5M”) than Guesstimate does.
Squiggle supports most of JSON. You can copy & paste JSON data and begin using it in Squiggle.
Squiggle ru

...
Read More Books but Pretend to Read Even More
by Arjun Panickssery — 2023-08-05T00:07:48.671Z
Link: https://arjunpanickssery.substack.com/p/read-more-books-but-pretend-to-read?sd=pf

A friend of mine tells me that whenever he walks by a bookstore he’s hit by a feeling of waste at the huge number of books that really nobody has any good reason ever to read. Call this the first of three important qualities that books have:

1. Most books are not worth reading.

In particular, whole categories of books merit suspicion:

Popular academic nonfiction often contains useless anecdotes and historical trivia from authors who got a book deal to expand a paper or a media piece. Indeed, I welcome any publishers reading this essay to pay me to expand it into a book. I could make each bullet point into a chapter and add background information about intellectual expectations in different historical social classes, random case studies about intellectuals who read a lot and others who read very little, lengthy comparisons to other media formats, and other digressions.
This is the main kind of book that Richard Hanania denounces in

“The Case Against (Most) Books”

and led Sam Bankman-Fried to tell a writer for Sequoia Capital that “I’m very skeptical of books. I don’t want to say no book is ever worth reading, but I actually do believe something pretty close to that. I think if you wrote a book, you fucked up, and it should have been a six-paragraph blog post.” Summaries or reviews of these books get you the important information in much less time.

Great Books of the “classic” nonfiction variety only become more suspicious the further forward we advance in time. No one would learn mathematics from Euclid and as we get Darwin and Skinner and economics and industrialization, past thinkers look more out of place. Fiction seems fine. Its claim that it presents things that are hard to convey with straightforward description has held up.

2. Most people remember roughly nothing that they read.

Most people who read a book ultimately retain little and sometimes nothing more than someone who just read a review. Holden

comments on this

with a table of how he perceives the time/retention-percentage curve for different levels of investment. Plotted:

The blue line reflects an increased understanding from reading the title → skimming → reading the book quickly → reading the book slowly. Meanwhile Holden doesn’t even think it’s “really possible to understand more than 50% of a serious book without e.g. spending a lot of independent time in the field.” Not pictured on the chart is Hol...
The Sinews of Sudan’s Latest War
by Tim Liptrot — 2023-08-04T18:17:27.860Z

I welcome feedback on this piece! It's the first of a series on the subject and I want to gauge interest in it from the blogging world. I will probably improve it to subsequently move it to the policy community. If people really like it I could transition it to a subscription or tipped model.

Section One: The Sinews of War

"There is an opinion in some quarters to send Marcus Antonius to govern Gaul (…) What else is that but supplying an enemy with (...) all the _ sinews of war, money in abundance _ (…) Will you furnish a wicked and desperate citizen with an army of Gauls and Germans, with money, and infantry, and cavalry, and all sorts of resources?" - Cicero

The war in Sudan has now raged for 4 months. On April 15th, the two armies that had been jointly ruling Sudan turned on one another. The Rapid Support Forces (RSF), a paramilitary army mounted a series of raids on the Sudan Armed Forces (SAF) air bases, headquarters and seats of government. Since then, the two sides have been locked in total war, which has devastated Sudan's capital, led to the return of ethnic massacres in Darfur, and threatens to create yet another failed state in the Horn of Africa.

Figure 1: Khartoum on Fire in June. Photo is taken from above the meeting of the White and Blue Niles, looking out at downtown Khartoum which contains the presidential palace, several ministries and the SAF headquarters.

This is the first of a series of reports surveying the sources of funding available to the armed groups of Sudan. Who controls them? How much money can they produce? Is this revenue controlled by senior leadership, or the personal loot of local commanders? Along the way, readers will gain a glimpse into the strange world of armed group finance. Most of the upcoming releases will be pitched for the Sudan-focused policy community, who are already quite interested in Sudan. This post gives a context to the deep dives that should be minimally accessible to a general audience. For an explanation of why the war began, see this other post.

Those interested in a straight military account or general commentary are already well served. While the 2023 Sudan conflict has not enjoyed the deep OSINT attention lavished on the Ukraine conflict, Sudan is blessed with talented analysts following the military and political developments. SeeDaniel Van's work,Ayin Network and Beam Reports. Sudan Transparency and P...