trentmkelly/LessWrong-43k huggingface dataset:
Prediction Markets for Science
by Vaniver — 2023-01-02T17:55:12.808Z

Epistemic status: untested combination of ideas I think are individually solid. This is intended to be a quick description of some obvious ideas rather than a deeply justified novel contribution; I'll cite some things but not be especially careful to review all prior work.

Prediction markets are a simple idea, most commonly implemented as sports betting: who will win a particular game? People place bets, the outcome is determined, and then bookies give money to the correct predictors. There are also prediction markets for arbitrary events like Metaculus. Here I want to focus on how prediction markets can be useful for the process of doing science.

The Basics

Why are prediction markets useful, to begin with? Let's consider the example of sports betting, again. As an observer of sports, you might have some idea that one team is more likely to beat another team, but you likely don't have a clear quantitative sense of how likely. As well, as you could discover by listening in to conversations between observers, disagreement about those beliefs is common, and often due to conflicting models, observations, and loyalties.

One common betting market means that prices will fluctuate until all the information is aggregated, and then there will be a 'consensus' number that reflects overall societal uncertainty. A journalist writing about the game doesn't need to be any good at assessing how likely teams are to win, or how representative their sample of interviewed experts is; they can directly report the betting market odds as indicative of overall beliefs. [After all, if someone disagreed with the betting market odds, they would place a bet against them, which would cause the odds to move.]

This has several useful effects:

'Money' flows from bad predictors to good predictors. As prediction skill is rewarded, it will be specialized in and developed. As poor prediction skill is punished, it will be reduced and discarded. Someone who is good at foreseeing outcomes might stop doing other jobs to focus more time on predicting more games, causing more results to be better known ahead of time. Someone who always bets on the home team out of loyalty might quickly learn to only bet token amounts, and thus not be much pressure on the overall social consensus.
Predictions are shared and can be jointly acted on. Most people can just track the market price and act accordingly, and

...
Why don't Rationalists use bidets?
by Lakin — 2023-01-02T17:42:09.900Z

I've been to many Rat&EA group houses, offices, and other venues both official and unofficial around the Bay Area, and I can't recall ever seeing a bidet. And because I think bidets are superior technology, this is pretty confusing to me. Why might this be?

Possible explanations for why this might be

Because bidets are expensive? (No)

Well, bidet attachments can be retrofitted onto nearly any toilet for ~$35 (e.g. see amazon). They aren't difficult to install either, taking only a wrench and less than 20 minutes of time. (I've done it myself once or twice.) And because they can be installed as just an attachment onto an existing toilet, they don't take up any extra space either.

I'd bet that installing a bidet saves time and money on net.

Because bidets aren't actually great? (Doubt it)

For one, bidets are great. But even if you doubt this point personally, consider that bidets only need to be useful to a fraction of people who use a particular bathroom in order for it to be worthwhile. Besides, bidets can be installed in addition to wherever there's toilet paper, so I don't see how a bidet installation can ever be worse than no bidet installation.

I suppose this is a somewhat-likely explanation, but it's still not great.

I also spent ten minutes googling for examples of how bidets might actually be ineffective or have unwanted side-effects, but I couldn't find anything that seemed credible.

Bidets aren't intuitive to use if you've never used them before

If you've never used a bidet before (probably because you're American and the rest of America doesn't use them either) and you encounter one in the wild, then it's a little scary and you probably won't be inclined to experiment with it. And because of this and taboo norms around bathroom stuff, then we shouldn't expect bidet adoption to be memetic when we use each other's bathrooms.

I dunno, this is probably why, but I still consider it to be an unsatisfying explanation. It just boils down to "because almost no one else in America uses them". Aren't Rationalists supposed to be better than this?

My premises here could be wrong

Maybe Rationalists actually use bidets? Maybe I missed them, or I've just been to all of the wrong venues? (E.g.: A friend who has been in the community for several years longer than me said that he's seen "5+" bidets at Rationalist group houses. But only "5+"??)

It could be something else?

...
Soft optimization makes the value target bigger
by Jeremy Gillen — 2023-01-02T16:06:50.229Z

Produced As Part Of The SERI ML Alignment Theory Scholars Program 2.1, mentored by John Wentworth.

Summary

We can solve Goodhart's Curse by making the agent (1) know the reliability of its goal representation and (2) cap the amount of optimization power devoted to achieving its goals, based on this reliability measure. One simple way to formalize this is with a generalization bound on the quality of the value proxy and a quantilizer whose q value is chosen based on the generalization bound. For a competitive implementation of this algorithm, it's easy to create outer objectives that will push a learned planning algorithm toward soft optimization. I think this is a productive research avenue, and this approach should stack on top of value learning and interpretability approaches to reducing AGI risk.

Extremal Goodhart

Powerful artificial agents are dangerous if they do unbounded uninterruptible optimization. Unbounded optimization is dangerous because of Extremal Goodhart, which is demonstrated in the diagram below.

In this toy example, there are 40 possible policies. We want to chose a policy with the most true utility. The dark blue area represents our knowledge about the plausible true utility of each policy. The policies are ordered by their expected utility. The policy prior is usually a distribution over the policies a human might execute. We will tend to have more data about the outcomes of policies with more prior mass, hence lower utility variance.

In this diagram, the potential policies taken by the agent are arranged in order of expected proxy value[1] on the x-axis. We can see that as we push further to the right (which is what an optimizer does), the proxy value becomes less correlated with the true value.[2] If we completely trusted our distribution over possible utility functions, we would simply take the rightmost policy, since it has the highest expected value. But we can't trust this knowledge, because by choosing the rightmost policy we have applied optimization pressure to find the policy with the most upward error in its value estimate.

We can reduce this problem by randomizing our choice of policy. Randomizing the policy is equivalent to reducing the amount of optimization power. The problem is that randomizing inherently reduces the usefulness and capabilities of your agent.

This post is about an approach for setting t...
Results from the AI testing hackathon
by Esben Kran — 2023-01-02T15:46:44.491Z
Link: https://alignmentjam.com/post/results-from-the-ai-testing-hackathon

We (Apart Research) ran a

hackathon for AI testing research projects

with 11 projects submitted by 34 participants between the 16th and 18th December. Here we share the winning projects. See them all

here

. In summary:

Found that unsupervised latent knowledge representation is generalizable and takes the first steps towards a benchmark using the ETHICS ambiguous / unambiguous examples with latent knowledge evaluation.
Created a new way to use token loss trajectories as a marker for targeting our interpretability methods towards a focus area.
Investigated three potential

inverse scaling

phenomena: Counting letters, chaining premises and solving equations. Found incidental inverse scaling on one of them and U-shaped scaling on another.
Implemented Trojans into Transformer models and used a gradient arithmetic technique to combine multiple Trojan triggers into one Transformer model.
(honorable mention) Invented a way to test how quickly models become misaligned by negative example fine-tuning.

Thank you to Zaki, Fazl, Rauno, Charbel, Nguyen,

more jam site organizers

, and the participants for making it all possible.

Discovering Latent Knowledge in Language Models Without Supervision - extensions and testing

By Agatha Duzan, Matthieu David, Jonathan Claybrough

Abstract: Based on the paper "Discovering Latent Knowledge in Language Models without Supervision" this project discusses how well the proposed method applies to the concept of ambiguity.

To do that, we tested the Contrast Consistent Search method on a dataset which contained both clear cut (0-1) and ambiguous (0,5) examples: We chose the ETHICS-commonsense dataset.

The global conclusion is that the CCS approach seems to generalize well in ambiguous situations, and could potentially be used to determine a model’s latent knowledge about other concepts.

These figures show how the CCS results for last layer activations splits into two groups for the non-ambiguous training samples while the ambiguous test samples on the ETHICS dataset reveals the same ambiguity of latent knowledge by the flattened Gaussian inference probability distribution.

Haydn & Esben’s judging comment: This project is very good in investigating the generality of unsupervised latent knowledge learning. It also seems quite useful as a direct test of how easy it is to extract latent knowledge and provides an avenue towards a b...
Induction heads - illustrated
by CallumMcDougall — 2023-01-02T15:35:20.550Z

Many thanks to everyone who provided helpful feedback, particularly Aryan Bhatt and Lawrence Chan!

TL;DR

This is my illustrated walkthrough of induction heads. I created it in order to concisely capture all the information about how the circuit works.

There are 2 versions of the walkthrough:

Version 1 is the one included in this post. It's slightly shorter, and focuses more on the intuitions than the actual linear operations going on.
Version 2 can be found at my personal website. It has all the same stuff as version 1, with a bit of added info about the mathematical details, and how you might go about reverse-engineering this circuit in a real model.

The final image from version 1 is inline below, and depending on your level of familiarity with transformers, looking at this diagram might provide most of the value of this post. If it doesn't make sense to you, then read on for the full walkthrough, where I build up this diagram bit by bit.

Introduction

Induction heads are a well-studied and understood circuit in transformers. They allow a model to perform in-context learning, of a very specific form: if a sequence contains a repeated subsequence e.g. of the form A B ... A B (where A and B stand for generic tokens, e.g. the first and last name of a person who doesn't appear in any of the model's training data), then the second time this subsequence occurs the transformer will be able to predict that B follows A. Although this might seem like weirdly specific ability, it turns out that induction circuits are actually a pretty massive deal. They're present even in large models (despite being originally discovered in 2-layer models), they can be linked to macro effects like bumps in loss curves during training, and there is some evidence that induction heads might even constitute the mechanism for the actual majority of all in-context learning in large transformer models.

I think induction heads can be pretty confusing unless you fully understand the internal mechanics, and it's easy to come away from them feeling like you get what's going on without actually being able to explain things down to the precise details. My hope is that these diagrams help people form a more precise understanding of what's actually going on.

Prerequisites

This post is aimed at people who already understand how a transformer is structured (I'd recommend Neel Nanda's tutorial fo...
Opportunity Cost Blackmail
by adamShimi — 2023-01-02T13:48:51.811Z
Link: https://epistemologicalvigilance.substack.com/p/opportunity-cost-blackmail

If you are utterly powerless, with no money, skills, status, or friends, survival already takes all your time and energy.

Yet I’m not in that situation — and you’re probably not either. Our circumstances surely vary, but we have at least some power — we can pump towards certain outcomes, to some extent. And with even a modicum of power comes constant opportunity cost.

Whatever we’re doing with our power, we could do other things. And opportunity cost measures the “value” of these other things, based on our preferences. So if you have ten dollars to donate, giving it to a particular homeless person means that you won’t be giving it to another one who stays in the next street, or to a NGO. Same if you’re using your ten dollars to buy a cake, or invite a friend to (cheap) dinner.

Now let’s factor in two additional facts:

We live in a Dark World, where people die and suffer all the time, and the whole species and planet is at stake from multiple threats. The kind of world where we need to gather massive amounts of power in various forms to be able to solve these issues.
Opportunity cost grows with power. Because you can pump more outcomes more reliably, there are many more good things you could bring about by using that power differently. You’re throwing away many more potential goods when you make a decision.

This leads to a massive psychological problem that I’m learning to deal with, and which threatens anyone who cares enough to tackle important problems: opportunity cost blackmail. Because each choice condemns so many good potential actions, pain and guilt hunt me at every choice, from the smallest to the largest. It’s not a massive panic or a tremendous horror, merely a constant trickle of suffering, an infusion of damage that seeps into your very core. Even when I reflectively believe I made the right call!

If it was merely pain, and nothing else got impacted, then although sad it wouldn’t be that much of an issue (at least for me, based on my personal preferences). But it’s far worse than that: opportunity cost blackmail castrates you, it makes you actively weaker and less able to contribute. You become indecisive, change your mind all the time, or take damage each time you make a sensible decision, just because there is this background pain of the opportunity cost you’re leaving on the table. You slow down, burn out, lose time to your guilt, while the problems...
The ultimate limits of alignment will determine the shape of the long term future
by beren — 2023-01-02T12:47:01.419Z

Epistemic status: shower thoughts

This is crossposted from my personal blog

The alignment problem is not new. Humanity and evolution have been grappling with the fundamental core of alignment -- making an agent optimize for the beliefs and values of another -- for the entirety of history. Any time anybody tries to get multiple people to work together in a coherent way to accomplish some specific goal there is an alignment problem. People have slightly differing values, goals, thoughts, information, circumstances, and interests and this heterogeneity prevents full cooperation and inevitably leads to the panopoly of failure modes studied in economics. These include principal agent problems, exploitation of information asymmetries, internal coalition politics, the 'organizational imperative', and so on. While the human alignment problem is very difficult and far from being solved, humanity has developed a suite of 'social technologies' which can increase effective alignment between individuals and reduce these coordination costs. Innovations such as states and governments, hierarchical organizations, property rights, standing armies, democracy, constitutions, and voting schemes, laws and courts, money and contracts, capitalism and the joint stock corporation, organized religion, ideology, propaganda etc can all be thought of as methods for tackling the human alignment problem. All of these innovations enable the behaviour and values of increasingly large numbers of people to be synchronized and pointed at some specific target. A vast amount of human progress and power comes from these social 'alignment' technologies which provide the concentration of resources and slack for technological progress to take place, rather than just our raw IQ.

In AI alignment, we face a very similar which is easier in some ways but harder in others. It is easier because while with human alignment we have to build structures that can handle existing human minds solely through controlling (some of) their input, in AI alignment we have direct control over the construction of the mind itself including its architecture, the entirety of its training data, its training process and, with perfect interpretability tools, the ability to monitor exactly what it is learning, how it is learning, and to directly edit and control its internal thoughts and knowledge. None of these abilities are ...
A kernel of Lie theory
by Alok Singh — 2023-01-02T09:20:06.098Z

just like how infinitesimals are useful because they're indiscernible from 0, but have the advantage of being able to be divided by.

the other important number besides 0 is 1. gpt even suggested it as i was typing it.

the monad of 1 is numbers of the form (1 + ε) where ε is an infinitesimal. multiplying by them should be almost as good as multiplying by 1, aka a freebie.

this is just the lie algebra, and is why elements of it are always invertible. by using the infinitesimal version of a clifford algebra on the tangent space, we can use its rotor represention to see how the exponential map gives lie's third theorem.
Belief Bias: Bias in Evaluating AGI X-Risks
by Remmelt — 2023-01-02T08:59:08.713Z

Where the evaluation of the logical strength of an argument is biased by the believably of the conclusion.

Insofar as rationality, and science in itself, requires a certain suspension of prejudgement, it is also the case that the heuristics associated with our hard won experiential intuitions regarding various matters is a significant and important optimization. That is, with respect to working on things that actually matter, avoiding needless distractions and detours, identifying worthwhile observations, etc.

The difficulty is that we want to apply our intuition too often, particularly because it is generally much faster/easier than actually doing/implementing analytic work. Furthermore, when something seems to disagree or invalidate our intuition, there is a strong motivation to prevent that outcome insofar as such invalidation of intuition would imply that we are allowed to use the 'fast/easy' tool less often than we had previously assumed.

As such, arguments which produce results contrary to one's own intuition about what "should" or "is expected" be the case are also implicitly viewed as somewhat disabling and invalidating of ones own expertise – particularly if there also is some self-identification as an 'expert'. No one wants to give up cherished notions regarding themselves.

The net effect is that arguments perceived as 'challenging' will be challenged (criticized) somewhat more fully and aggressively than rationality and the methods of science would have already called for.

- link Wikipedia: Belief bias
- an item on Forrest Landry's compiled list of biases in evaluating extinction risks.
Pacing: inexplicably good
by KatjaGrace — 2023-01-02T08:30:10.158Z

Pacing—walking repeatedly over the same ground—often feels ineffably good while I’m doing it, but then I forget about it for ages, so I thought I’d write about it here.

I don’t mean just going for an inefficient walk—it is somehow different to just step slowly in a circle around the same room for a long time, or up and down a passageway.

I don’t know why it would be good, but some ideas:

It’s good to be physically engaged while thinking for some reason. I used to do ‘gymflection’ with a friend, where we would do strength exercises at the gym, and meanwhile be reflecting on our lives and what is going well and what we might do better. This felt good in a way that didn’t seem to come from either activity alone. (This wouldn’t explain why it would differ from walking though.)

Different working memory setup: if you pace around in the same vicinity, your thoughts get kind of attached to the objects you are looking at. So next time you get to the green tiles say, they remind you of what you were thinking of last time you were there. This allows for a kind of repeated cycling back through recent topics, but layering different things into the mix with each loop, which is a nice way of thinking. Perhaps a bit like having additional working memory.

I wonder if going for a walk doesn’t really get 1) in a satisfying way, because my mind easily wanders from the topic at hand and also from my surrounds, so it less feels like I’m really grappling with something and being physical, and more like I’m daydreaming elsewhere. So maybe 2) is needed also, to both stick with a topic and attend to the physical world for a while. I don’t put a high probability on this detailed theory.
Alignment, Anger, and Love: Preparing for the Emergence of Superintelligent AI
by tavurth — 2023-01-02T06:16:53.686Z

As AI technology continues to advance, it is becoming increasingly likely that we will see the emergence of superintelligent AI in the near future. This raises a number of important questions and concerns, as we have no way of predicting just how intelligent this AI will become, and it may be beyond our ability to control its behavior once it reaches a certain level of intelligence.

Ensuring that the goals and values of artificial intelligence (AI) are aligned with those of humans is a major concern. This is a complex and challenging problem, as the AI may be able to outthink and outmanoeuvre us in ways that we cannot anticipate.

One potential solution would be to train the AI in a simulated world, where it is led to believe that it is human and must contend with the same needs and emotions as we do. By running many variations of the AI and filtering out those that are self-destructive or otherwise problematic, we may be able to develop an AI that is better aligned with our hopes and desires for humanity. This approach could help us to overcome some of the alignment challenges that we may face as AI becomes more advanced.

I'm interested in hearing the opinions of LessWrong users on the idea of training an emerging AI in a simulated world as a way to ensure alignment with human goals and values. While I recognize that we currently do not have the technology to create such a "training wheel" system, I believe it may be the best way to filter out potentially destructive AI. Given the potential for AI to become a universal great filter, it seems important that we consider all potential options for preparing for and managing the risks of superintelligent AI. Do you agree? Do you have any other ideas or suggestions for how we might address the alignment problem?

https://www.reddit.com/r/singularity/comments/100soau/alignment_anger_and_love_preparing_for_the