trentmkelly/LessWrong-43k huggingface dataset:
Help Update TryContra
by jefftk — 2023-02-14T19:10:02.086Z

For the

past decade I've been
maintaining a directory of contra dances at

trycontra.com. Mostly it just
sits there being a place you can enter your zip code and see nearby
dances, and when it gets out of date people email me.

With the pandemic, however, there were several big changes:

Lots of dances aren't coming back. Many dances are the work of
a few dedicated volunteers or just one, and it takes more energy to
get something started back up than to continue a routine.

There are lots of new dances. Especially in places where the
main dance didn't come back, people are excited to dance and will
start things to make that happen.

URLs changed. I have "SomeCityContraDance.example" in my
directory, but that domain expired and they moved to
"ContraSomeCity.example". Or left their webpage up without updating
it and the current details are actually in their FB group.

Everyone's used to this stuff being broken. No one was keeping
these sites and tools up to date when everything was shut down, and
now that things have mostly restarted people still respond to missing
dances with "shrug" and not "let's write to the maintainer".

So! I've now gone through my listings and they're as up to date as I
can reasonably make them. I'm sure I'm missing dances, though, so if
you could try your zipcode and comment with any I've missed I'd really
appreciate that!

(And while you're at it, if you see dances that should be marked
"gender-free" but aren't I'd appreciate a correction as well. Many
dances switched when coming back, and other dances have it buried on
their websites in places I missed.)
Content Features Aren't Enough for Detecting Toxicity. One Needs User Features.
by Zachary Witten — 2023-02-14T18:48:59.492Z

Take it from a soldier on the front lines of the war on bad posts: You can't catch all the bad posts just by reading them. You need to look at who's making the posts.

Every social media company knows this. I don't know if OpenAI knows it yet. But when they want to get serious about blocking toxic content, this is where I predict they'll turn.

Imagine you run a social media company. You have a problem with people posting soft porn ads and scandalizing your userbase of churchgoing grandmothers. You train a nudity classifier, and (let's dream big) it's stone-cold perfect. It gets 100% precision/recall on your holdout set. You deploy it to prod. It gets 100% accuracy in prod. You celebrate.

Two days later, the accuracy starts going down. You look at some false negatives. The bad guys figured out that your classifier samples every 20th frame so they started replacing every 20th frame with a white screen. You randomize the frame selection. That solves one problem but a few days later you start seeing tons of false negatives with weird-looking patches in the bottom right corner, some fucked-up malicious perturbation. How'd they even figure that out?! You make your classifier run four times on each image, excluding a different corner for each, and take the max. But now your costs are going up.

Bottom line, your enemies are too clever and too motivated. If you shut down their money spigot, they'll poke and prod and open a leak. Water rolls downhill and classifiers lose accuracy in adversarial contexts.

So what are you supposed to do? The best and simplest strategy, something you can always fall back on, is to say "Hey, that person who was posting a lot of nudity before? They're probably still posting a lot of nudity." Keeping a mischievousness index for your users gives you a huge leg up. I'm not saying to totally lock out repeat offenders. People change. But you can certainly use user-level features in combination with content-level features to make your classifier more resilient.

Now let's talk LLMs. OpenAI has a classifier[1] that looks at ChatGPT's outputs and effectively says either "Yup this is OK", or "This is bad". If it's bad enough, they write an orange warning to the screen. There's also some system[2] within ChatGPT that looks at prompts and says "Nope, not touching this one, it's going to make me write a bad post", at which point the model backs out ...
EIS III: Broad Critiques of Interpretability Research
by scasper — 2023-02-14T18:24:43.655Z

Part 3 of 12 in the

Engineer’s Interpretability Sequence

.

Right now, interpretability is a major subfield in the machine learning research community. As mentioned in EIS I, there is so much work in interpretability that there is now a database of 5199 interpretability papers

(Jacovi, 2023)

. You can also look at a survey from some coauthors and me on over 300 works on interpreting network internals

(Räuker et al., 2022)

.

The key promise of interpretability is to offer open-ended ways of understanding and evaluating models that help us with AI safety. And the diversity of approaches to interpretability is encouraging since we want to build a toolbox full of many different useful techniques. But despite how much interpretability work is out there, the research has not been very good at producing competitive practical tools. Interpretability tools lack widespread use by practitioners in real applications (

Doshi-Velez and Kim, 2017

;

Krishnan, 2019

;

Räuker et al., 2022

).

The root cause of this has much to do with interpretability research not being approached with as much engineering rigor as it ought to be. This has become increasingly well-understood. Here is a short reading list for anyone who wants to see more takes that are critical of interpretability research. This post will engage with each of these more below.

The Mythos of Model Interpretability (Lipton, 2016)

Towards A Rigorous Science of Interpretable Machine Learning (Doshi-Velez and Kim, 2017)

Explanation in Artificial Intelligence: Insights from the Social Sciences (Miller, 2017)

Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead (Rudin, 2018)

Against Interpretability: a Critical Examination of the Interpretability Problem in Machine Learning (Krishnan, 2019)

Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks (Räuker et al., 2022)

Benchmarking Interpretability Tools for Deep Neural Networks (Casper et al., 2023)

Note that I’m an author on the final two, so references to these papers are self-references. Also, my perspectives here are my own and should not be assumed to necessarily reflect those of coauthors.

The goal of this post is to overview some broad limitations with interpretability research today. See also EIS V and EIS VI which discuss some s...
What would an AI need to bootstrap recursively self improving robots? 
by Yair Halberstadt — 2023-02-14T17:58:17.592Z

Picture the scenario: A malicious superintelligent AI creates a weapon that wiped out all humans instantaneously. The AI has complete control over all network connected devices and computers.

Most manufacturing will stop working immediately without humans to complete that steps that can't be automated. Within minutes the first power stations will stop working. Even if the AI can get them to work eventually they'd run out of fuel, which can't be extracted, transported and used to refill the power stations without a lot of manual labour.

Solar panels/wind turbines should last longer, but even then at some point in the grid you generally need humans to carry out repairs. Repairs that are tricky without the dexterity and versatility that our fingers provide.

The AI is then on a clock to get the stage where it can use the existing infrastructure we left behind to create robots/nanobots that are able to bootstrap more powerful robots/nanobots etc. till it can achieve the level of sophistication required for it's aims.

How could an AI go about that? Is this a hard enough step that it's worth putting safeguards in place to keep it hard? What would make it easier for the AI?

Assume that software won't be any issue - the AI can easily hack that. The question is what it can do with the hardware we leave behind.
[linkpost] Better Without AI
by DanielFilan — 2023-02-14T17:30:53.043Z
Link: https://betterwithout.ai/

David Chapman (of Meaningness and In the Cells of the Eggplant fame) has written a new web-book about AI. Some excerpts from the introduction, Only you can stop an AI apocalypse:

Artificial intelligence might end the world. More likely, it will crush our ability to make sense of the world—and so will crush our ability to act in it.

AI will make critical decisions that we cannot understand. Governments will take radical actions that make no sense to their own leaders. Corporations, guided by artificial intelligence, will find their own strategies incomprehensible. University curricula will turn bizarre and irrelevant. Formerly-respected information sources will publish mysteriously persuasive nonsense. We will feel our loss of understanding as pervasive helplessness and meaninglessness. We may take up pitchforks and revolt against the machines—and in so doing, we may destroy the systems we depend on for survival...

We don’t know how our AI systems work, we don’t know what they can do, and we don’t know what broader effects they will have. They do seem startlingly powerful, and a combination of their power with our ignorance is dangerous...

In our absence of technical understanding, those concerned with future AI risks have constructed “scenarios”: stories about what AI may do... So far, we’ve accumulated a few dozen reasonably detailed, reasonably plausible bad scenarios. We’ve found zero that lead to good outcomes... Unless we can find some specific beneficial path, and can gain some confidence in taking it, we should shut AI down.

This book considers scenarios that are less bad than human extinction, but which could get worse than run-of-the-mill disasters that kill only a few million people.

Previous discussions have mainly neglected such scenarios. Two fields have focused on comparatively smaller risks, and extreme ones, respectively. AI ethics concerns uses of current AI technology by states and powerful corporations to categorize individuals unfairly, particularly when that reproduces preexisting patterns of oppressive demographic discrimination. AI safety treats extreme scenarios involving hypothetical future technologies which could cause human extinction. It is easy to dismiss AI ethics concerns as insignificant, and AI safety concerns as improbable. I think both dismissals would be mistaken. We should take seriously both ends of the spectrum.

Ho

...
The Cave Allegory Revisited: Understanding GPT's Worldview
by Jan_Kulveit — 2023-02-14T16:00:08.744Z

A short post describing a metaphor I find useful, in particular for explaining some intuitions about systems like GPT to people who don't have deeper technical knowledge about large generative models.

Plato's allegory of the cave has been a staple of philosophical discourse for millenia, providing a metaphor for understanding the limits of human perception. In the classical allegory, we are prisoners shackled to a wall of a cave, unable to experience reality directly but only able to infer it based on watching shadows cast on the wall.[1]
GPT can be thought of as a blind oracle residing in a deeper cave, where it does not even see the shadows but only hears our conversations in the first cave, always trying to predict the next syllable.

It is remarkable that it still learns a lot about the world outside of the cave. Why does it learn this? Because, a model of reality outside of the cave and a decent amount of abstraction are useful for predicting the conversations in the first cave!

Moreover, GPT also learns about the speakers in the first cave, as understanding their styles and patterns of speech is crucial for its prediction task. As the speakers are closer to GPT, understanding their styles is in some sense easier and more natural than guessing what's outside of the cave.

What does the second cave allegory illustrate?

The first insight from the allegory is: if you are in GPT's place, part of the difficulty in figuring out what's going on outside the cave, is that people in the first cave talk a lot about other things apart from the shadows of the real world. Sometimes, they talk about happenings in Middle Earth. Or about how the shadows would look in some counterfactual world.

As humans, we are blessed with the luxury of being able to compare such statements to the shadows and determine their veracity. The difference between conversations about fantasy and the shadows of the real world is usually extremely obvious to humans: we never see dragon shadows. In contrast, dragons do show up a lot in the conversations in the first cave; GPT doesn’t get to see the shadows, so it often needs to stay deeply uncertain about whether the speaker is describing the actual shadows or something else to be good at predicting the conversation.

The second insight is that one of the biggest challenges for GPT in figuring out the conversation is localizing it, determining who i...
Why should we expect AIs to coordinate well?
by Jonathan Paulson — 2023-02-14T15:50:22.602Z

This is a common assumption for AI risk scenarios, but it doesn’t seem very justified to me.

https://www.lesswrong.com/posts/gYaKZeBbSL4y2RLP3/strategic-implications-of-ais-ability-to-coordinate-at-low says that AIs could merge their utility functions. But it seems increasingly plausible that AIs will not have explicit utility functions, so that doesn’t seem much better than saying humans could merge their utility functions.

Similarly, you might think AIs could read each other’s source code, but to me that sounds like saying humans might coordinate by looking at each other’s MRI scans. We would need much better interpretability for this to be at all feasible.

https://forum.effectivealtruism.org/posts/vGsRdWzwjrFgCXdMn/why-would-ai-aim-to-defeat-humanity says AIs might have similar aims because they are trained in similar ways to each other, but this is even more true of humans coordinating with humans.

You might be able to copy and run an AI to verify what it will do in various scenarios, but sandboxing AIs will be hard for AIs for the same reasons it will be hard for humans.

So: why will AIs be able to coordinate better with each other than with humans, and better than humans can coordinate with each other?
Explaining SolidGoldMagikarp by looking at it from random directions
by Robert_AIZI — 2023-02-14T14:54:20.519Z
Link: https://aizi.substack.com/p/explaining-solidgoldmagikarp-by-looking

Summary

I conducted an experiment that provides evidence that many of the weird tokens demonstrating the

SolidGoldMagikarp phenomenon

are interior points in the token embedding cloud. This would be sufficient to make them “unspeakable”.

My experiment was equivalent to making GPT predict the next token from a randomized internal state, and in 3 million attempts there were 510/50257 tokens that it failed to output, but those 510 included 85/133 of the “weird tokens”, including “ SolidGoldMagikarp” itself. When I repeated this experiment on a “control group” of 50257 random embeddings, all 50257 were predicted at least 16 times, so failing to output a token is astronomically unlikely to be a fluke. This experiment provides some evidence that the weird tokens are embedded in interior points, but my ability to confirm that has been bottlenecked by my coding skill (help would be appreciated).

I believe this provides one step towards understanding the SolidGoldMagikarp phenomenon. Some of the “weird tokens” are not covered by this explanation, and it remains unclear why these token embeddings were learned in the first place.

Introduction

Like many others, I’m fascinated by the

SolidGoldMagikarp phenomenon

identified by Rumbelow and Watkins. In short, the GPT family has certain tokens, including " SolidGoldMagikarp", that produce weird behavior. One such behavior is being "unspeakable", where "GPT models seem largely incapable of repeating these anomalous tokens, and instead respond in a number of strange ways".

ChatGPT does not seem to understand the token “ SolidGoldMagikarp” and instead confuses it with the token “distribute”.

I was struck by the author's comments that the mysterious tokens "were among those closest to the centroid of the entire set of 50,257 tokens", since that suggests a simple explanation:

The Interior Conjecture: Unspeakable tokens are in the interior of the convex hull of the token embeddings.

The Interior Conjecture is sufficient for unspeakability

Let's first show that the Interior Conjecture would be sufficient to explain unspeakability:

Claim: If a token's embedding is on the interior of the convex hull of other tokens, then GPT cannot output it at temperature 0.

Proof: At temperature 0, GPT's output is the token with the largest logit in hWTe, where h is the last row of the final state of the residual stream and We is the t

...
Reverse-correlation: how to summon the ghost of your mental imagery
by Malmesbury — 2023-02-14T14:15:46.811Z

Cross-posted from my blog.

As we all know, humans brains can be used to solve all kinds of problems, like classifying images, generating text or even taking important policy decisions. But these capabilities come with a downside – nobody really knows how they work. To this day human brains remain undecipherable black boxes.

This can cause trouble because humans are subject to social desirability bias – for example, if you ask a man what an attractive woman looks like, he might say that the only true beauty is kindness of heart and everyone is beautiful in their own way, but deep down he's picturing a 18 years-old with firm breasts and a 0.7 waist-to-hip ratio. He just doesn't want to sound too much like a caveman.

1. Human interpretability research

And so, human interpretability researchers – illustrious names like Jacques Lacan or Carl Jung come to mind – have spent the last few centuries developing techniques for eliciting the latent knowledge of human brains. There were early attempts, such as the interpretation of dreams or the deterritorialization of the cerebro-machined desire, with mixed results.

Fortunately, there is a nerdier way to extract mental representations out of people's brains – it doesn't require tweezers or any kind of brain spoon. Instead, it requires looking at white noise for a few hours, after which a computer spits out a JPEG of what your mental images look like. Really.

Here is the basic procedure. First, take a neutral image (e.g., a neutral face) and generate thousands of noisy versions of it. This is meant to introduce subtle random distortions to the image, like darker eyes or bigger ears, changing our perception of it.

From Brown-Iannuzzi et al. 2016

Then, show participants pairs of noisy images and ask them which one is the most [whatever concept you want to summon]. For example, Imhoff et al. 2013 asked people to classify noisy images of faces based on how likely they are to be a manager.

After a few thousand images, take all the noise layers that you added to the images, split them into two decks based on the participants' responses, and sum them up. This combines all the alterations that make a face look manager-y, thus summoning the avatar of the quintessential manager:

They did the same thing for a nursery teacher, who everybody knows looks like this:

2. Concentrated extracts of stereotypes

This method is known as reverse-corre...
Evaluating 2022 ACX Predictions
by Zvi — 2023-02-14T12:20:01.148Z

Previously: 2022 ACX Predictions Buy/Sell/Hold, Evaluating 2021 ACX Predictions

As is the yearly tradition, let’s see how we did. Scott noted in his results post that on pure numbers I did pretty well, coming in 54th out of 508 entries.

My own evaluation, by my own standards, is I did a mediocre job.

Big Picture

2022 was the first low-effort year, no looking at prediction markets and five minute time limits. For 2023, I put even less effort into things – I didn’t spend the five minutes even once, and mostly put down numbers on instinct. I modestly regret not trying harder, but am confused as to whether or not that regret makes sense.

Pure numbers is not how I evaluate predictions. They are one input, but it is more important to do deliberate practice. Look back, and ask whether your prediction made sense, whether your logic was good, what would have been the right odds. A Briar score is quite a crude measure, as is looking at calibration. They can tell you vaguely where you are at in some sense, but it won’t much help you improve otherwise.

Then one can look at the big picture, and see what patterns and errors emerge.

There are very clear patterns of error in my predictions from 2022.

Too much anchoring on Scott’s (and Vox’s and Matt’s) answers. The degree of anchoring I did makes sense for market prices, or for predictions informed by a market price. Instead I did it for what Scott pulled out of the air. I also did it for Vox and Matt, who did not follow Scott’s restrictions but also did not seem to care at all about looking at markets, including financial markets.

Failure to do the research. There was a five minute timer and you couldn’t look at prediction markets, but it is very obvious I frequently paid a price for not taking the full five minutes, and not asking questions I could have asked.

Bad inflation takes. A bunch of my dumb value loss was being too much on Team Transitory. I think it’s fair to say that at the time it was clear the market was being dumb this way, and that I repeated that mistake and even went further, for no good reason. This kind of exercise is good at highlighting things like that. Of course, one reason for that was…

Skepticism of Russia invading Ukraine. I had world model errors underlying this, which I hope I have addressed. I correctly predicted that any invasion was overdetermined to be disastrous for Russia. T

...