trentmkelly/LessWrong-43k huggingface dataset:
Interpreting Neural Networks through the Polytope Lens
by Sid Black — 2022-09-23T17:58:30.639Z

Sid Black*, Lee Sharkey*, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, Connor Leahy

*equal contribution

Research from

Conjecture

.

This post benefited from feedback from many staff at Conjecture including Adam Shimi, Nicholas Kees Dupuis, Dan Clothiaux, Kyle McDonell. Additionally, the post also benefited from inputs from Jessica Cooper, Eliezer Yudkowsky, Neel Nanda, Andrei Alexandru, Ethan Perez, Jan Hendrik Kirchner, Chris Olah, Nelson Elhage, David Lindner, Evan R Murphy, Tom McGrath, Martin Wattenberg, Johannes Treutlein, Spencer Becker-Kahn, Leo Gao, John Wentworth, and Paul Christiano and from discussions with many other colleagues working on interpretability.

Summary

Mechanistic interpretability aims to explain what a neural network has learned at a nuts-and-bolts level. What are the fundamental primitives of neural network representations? What basic objects should we use to describe the operation of neural networks mechanistically? Previous mechanistic descriptions have used individual neurons or their linear combinations to understand the representations a network has learned. But there are clues that neurons and their linear combinations are not the correct fundamental units of description - directions cannot describe how neural networks use nonlinearities to structure their representations. Moreover, many instances of individual neurons and their combinations are polysemantic (i.e. they have multiple unrelated meanings). Polysemanticity makes interpreting the network in terms of neurons or directions challenging since we can no longer assign a specific feature to a neural unit. In order to find a basic unit of description that doesn’t suffer from these problems, we zoom in beyond just directions to study the way that piecewise linear activation functions (such as ReLU) partition the activation space into numerous discrete polytopes. We call this perspective the ‘polytope lens’. Although this view introduces new challenges, we think they are surmountable and that more careful consideration of the impact of nonlinearities is necessary in order to build better high-level abstractions for a mechanistic understanding of neural networks. The polytope lens makes concrete predictions about the behavior of neural networks, which we evaluate through e...
The heterogeneity of human value types: Implications for AI alignment
by geoffreymiller — 2022-09-23T17:03:02.420Z

Note: This essay was originally posted on EA Forum here on Sept 16. I’d welcome comments from LessWrong readers and AI Alignment Forum experts. I have posted some related essays on EA Forum about the importance for AI alignment of considering corporal/body values, religious values, and the diversity of values across individuals. (I might revise those and post them here soon, if people seem interested.)

Overview

I argue that AI ‘alignment with human values’ needs to take more seriously the fact that there are many different types of human values that work in different ways, that have been studied for many decades by diverse behavioral and social sciences, and that need to be explicitly considered when we’re trying to develop alignment strategies that could actually handle the full diversity and heterogeneity of human values.

Introduction

I worry that a lot of AI alignment research seems to rely on a dangerously simplistic view of human values, and that this will undermine our ability to safely align AI systems with human values.

The simplistic view seems to arise from taking expected utility theory too seriously as a model of human values and preferences. It’s true that we can often describe human decisions, post hoc, at a rather abstract and generic level, using the language of utility theory, Bayesian rationality, and statistical decision theory. This rather abstract and generic way of modeling human values has often been useful in the fields of economics, game theory, rational choice, consequentialist moral philosophy, and reinforcement learning theory.

However, within standard utility theory, there’s no fundamental difference between a consumer’s food preference for a certain flavor of jelly bean and a Muslim’s sacred taboo against eating pork. Utility theory doesn’t distinguish very well between someone who’s a vegan for health reasons, someone who’s a vegan for ethical reasons, and someone who has food allergies to animal proteins. Utility theory doesn’t distinguish very well between someone who’s polyamorous based on libertarian principles, someone who’s polyamorous so they can conform to their Burning Man peer group, and someone who just happened to inherit genes for a high degree of ‘sociosexuality’. Utility theory can’t even distinguish very well between deontologists, consequentialists, virtue ethicists, and religious fundamentalists.

With all due r...
How to use DMT without going insane: On navigating epistemic uncertainty in the DMT memeplex
by cube_flipper — 2022-09-23T16:32:03.201Z
Link: https://smoothbrains.net/posts/2022-09-22-how-to-use-dmt-without-going-insane.html

DMT has a certain popular reputation – that of something discombobulating, inscrutable, not necessarily for the faint of heart.

While I do believe DMT is capable of providing a comprehensible and even wholesome experience, it definitely deserves its reputation as an extremely powerful psychedelic. While it may only last twenty minutes or so, you might encounter situations which challenge foundational aspects of your belief system.

The risk involved is clear: the unprepared mind may start believing things which are bad for its ongoing well-being. Of course, the inverse is also true: DMT may help dislodge bad priors, break you out of unproductive behavioural patterns, or introduce you to new, more compelling models of reality.

This is why DMT has the mystical reputation it does. So, how can we prepare ourselves for this potential onslaught of ontological mayhem? It's wise to spend some time planning how to orient ourselves at the meta-level.

Navigating epistemic uncertainty

In order to navigate this landscape of epistemic ambiguity, you must be prepared to flick between multiple belief systems and ideological frames of reference, without building too much attachment to any of them and judging each by the fruit it bears. If you can do this already, great.

Beliefs can be weighted not only by their truthiness, but also by their potential outcomes. You might wind up convinced that UFOs are real and President Eisenhower sold us all out to the Greys – and I cannot dismiss such a theory outright – but consider how much attention you might want to feed such a belief. What are you going to do with this information? Will you alienate your friends?

Some belief systems may contain infinitising aspects which may motivate one to make sudden and drastic lifestyle changes. Some people seem to have a natural immunity against these; some don't. In many situations, agnosticism can be a healthy default stance. If you can find a way to avoid making a belief load-bearing; i.e., your behavioural outcomes will be similar or identical regardless of whether you decide it may be true or not, even better.

And hey, this is not just good DMT advice, it's good life advice.

Epistemic hygiene practices

Sarah Perry, in her 2017 ribbonfarm.com post, The Limits of Epistemic Hygiene, contrasts humanity's healthy response to germ theory with our comparatively ill-developed response to meme theory:

Since the

...
Shahar Avin On How To Regulate Advanced AI Systems
by Michaël Trazzi — 2022-09-23T15:46:46.155Z
Link: https://theinsideview.ai/shahar

Shahar Avin is a senior researcher at the Center for the Study of Existential Risk in Cambridge. In his past life, he was a Google Engineer, though right now he spends most of his time thinking about how to prevent the risks that occur if companies like Google end up deploying powerful AI systems, by organizing AI Governance role-playing workshops.

In this episode, we talk about a broad variety of topics, including how we could apply what Shahar learned running AI Governance workshops to governing transformative AI, AI Strategy, AI Governance, Trustworthy AI Development and end up answering some twitter questions.

Below are some highlighted quotes from our conversation (available on Youtube, Spotify, Google Podcast, Apple Podcast). For the full context for each of these quotes, you can find the accompanying transcript.

We Are Only Seeing The Tip Of The Iceberg

The Most Cutting Edge AI Research Is Probably Private

“I don’t know how much of the most cutting edge research today is public. I would not be confident that it is. It is very easy to look at all of the stuff that is public and see a lot of it, and infer from the fact that you’re seeing a lot of public research that all research must, therefore be public. I don’t think that is a correct inference to make.”

AI companies may not be showing all of their cards

“My guess would be that they're not always showing all of the cards. It's not always a calculated decision, but there is a calculated decision to be made of, if I have a result, do I publish or not? And then what goes into the calculation is if there is a benefit from publishing. It increases your brand, it attracts more talent, it shows that you are at the cutting edge, it allows others to build on your result and then you get to benefit from building on top of their results. And you have the cost of, as long as you keep for yourself, no one else knows it, and you can keep on doing the research.”

Aligning Complex Systems Is Hard

Narrow AI Do Not Guarantee Alignment

“One failure mode is that there is an overall emergent direction that is bad for us. And another is there is no emergent direction, but the systems in fact are conflicting with each other, undermining each other. So one system is optimizing for one proxy. It generates it externality that is not fully captured by its designers that gets picked up by another system that has a bad proxy for it, a

...
Interlude: But Who Optimizes The Optimizer?
by Paul Bricman — 2022-09-23T15:30:06.638Z

This post is part of my hypothesis subspace sequence, a living collection of proposals I'm exploring at Refine. Preceded by representational tethers.

Intro

Consider diffusion models. DALL-E is able to take in an image as input and imperceptibly edit it so that it becomes slightly more meaningful (i.e. closer to the intended content). In this, DALL-E is an optimizer of images. It can steadily nudge images across image space towards what people generally deem meaningful images. But an image diffusion model need not nudge images around in this specific way. Consider an untrained version of DALL-E, whose parameters have just been initialized a moment ago. It still implicitly exerts currents across image space, it's just that those currents don't systematically lead towards the manifold of meaningful images. We deem an image diffusion model good if the field it implements nudges images in a way we deem appropriate. Conversely, a bad one simply specifies a wildly different field.

Naturally, we'd like good models. However, we lack an exhaustive closed-form description of the desired field for pushing towards meaningful images. The best we managed to do is come up with a way of roughly telling which of two models is better. We then use (1) this imperfect relative signal, and (2) a sprinkle of glorified hill-climbing as means of obtaining a good model by charting our way through model space. We might fill those in with e.g. (1) incremental denoising performance on ImageNet, and (2) SGD. Those two ingredients help us define a trainer for the diffusion model, a way from getting from bad models towards good ones. We deem a trainer good if the field it implements nudges models in a way we deem appropriate (i.e. from bad to good). Conversely, a bad one simply specifies a wildly different field across model space.

Naturally, we'd like good trainers. However, we lack an exhaustive closed-form description of the desired field for reliably pushing towards truly good models. You've guessed it, we did manage to come up with a way of roughly telling which of two trainers is better. We similarly use this rough relative signal and our favorite hill-climbing algorithm as means of obtaining a good trainer by charting our way across trainer space. We might fill those in with e.g. (1) convergence speed, and (2) grid search across hyperparameters. Those two help us define a hyperparameter ...
Why do so many things break in a 2 element set?
by Alok Singh — 2022-09-23T06:30:16.747Z

The math jargon for a set where you can implement a 4 function calculator
(\(+,-,*,/\)) is a field. Fields can have a finite number of elements, and
such fields always have a prime power number of elements, aka \(p^k\) where \(p\) is prime and \(k\) is a natural number.

Many results are not true if the underlying field has exactly 2 elements. The
reason why:

\(x = -x\) \(x + x = -x + x\) \(2x = 0\)

The final formula is taken to mean that \(x\) is zero. But there’s another interpretation.

What if 2 = 0?

In a field with 2 elements, this is true. Moreover, \(x = -x\) is a tautology AKA always true AKA worthless. \(-1 \mod 2 = 1\) and \(0 \mod 2 = 0\).

This is only when the underlying field has 2 elements. A 2 element set is exceptional because negation doesn’t actually do anything.
Intelligence as a Platform
by Robert Kennedy — 2022-09-23T05:51:48.600Z

In this post I review the platform/product distinction and note how GPT can be modeled as a platform, which further products will be built upon. I argue that prosaic alignment is best viewed as hypnotism capabilities training instantiating products on that platform. I explain why this slightly pushes out my X-Risk timing while increasing my X-Risk factor.

Products vs Platforms

In Stevey's Google Platforms Rant (2011), Steve Yegge lays out the differences between products and platforms.

The other big realization [Jeff Bezos] had was that he can't always build the right thing. I think Larry Tesler [senior UX from Apple] might have struck some kind of chord in Bezos when he said his mom couldn't use the goddamn website. It's not even super clear whose mom he was talking about, and doesn't really matter, because nobody's mom can use the goddamn website. In fact I myself find the website disturbingly daunting, and I worked there for over half a decade. I've just learned to kinda defocus my eyes and concentrate on the million or so pixels near the center of the page above the fold.

I'm not really sure how Bezos came to this realization -- the insight that he can't build one product and have it be right for everyone. But it doesn't matter, because he gets it. There's actually a formal name for this phenomenon. It's called Accessibility, and it's the most important thing in the computing world.

[...]

Google+ is a knee-jerk reaction, a study in short-term thinking, predicated on the incorrect notion that Facebook is successful because they built a great product. But that's not why they are successful. Facebook is successful because they built an entire constellation of products by allowing other people to do the work. So Facebook is different for everyone. Some people spend all their time on Mafia Wars. Some spend all their time on Farmville. There are hundreds or maybe thousands of different high-quality time sinks available, so there's something there for everyone.

Our Google+ team took a look at the aftermarket and said: "Gosh, it looks like we need some games. Let's go contract someone to, um, write some games for us." Do you begin to see how incredibly wrong that thinking is now? The problem is that we are trying to predict what people want and deliver it for them.

[...]

Ironically enough, Wave was a great platform, may they rest in peace. But making something a pl

...
Public-facing Censorship Is Safety Theater, Causing Reputational Damage 
by Yitz — 2022-09-23T05:08:14.149Z

It's so common it's a stereotype.

A large corporation releases a cutting-edge AI model, and puts out a press release talking about how their new, [larger/smaller]-than-ever model provides unprecedented freedom for [underprivileged artists/small business owners/outside researchers] to do whatever it is their AI does. You go to their website, start playing with the model, and before long—

Results containing potentially sensitive content have been omitted. Further requests of this type may result in account suspension, etc., etc., etc....

—or something along those lines. The prompt you gave was pretty innocuous, but in retrospect you can sort of see how maybe the output might have resulted in something horrifically offensive, like a curse word, or even (heaven forbid) an image that has a known person's face in it. You've been protected from such horrors, and this is reassuring. Of course, your next prompt for whatever reason elicits [insert offensive stereotype/surprisingly gory or uncanny imagery/dangerously incorrect claim presented with high confidence/etc. here], which is slightly less reassuring.

Checking the details of the press release, you see a small section of the F.A.Q. with the disclaimer that some outputs may be biased due to [yadda yadda yadda you know the drill]. You breathe a sigh of relief, secure in the comforting knowledge that [faceless company] cares about AI safety, human rights, and reducing biases. Their model isn't perfect, but they're clearly working on it!

The above scenario is how [large corporations] seem to expect consumers to react to their selective censorship. In reality I strongly suspect that the main concern is not so much protecting the consumer as it is protecting themselves from liability. After all, by releasing a model which is clearly capable of doing [harmful capability], and by giving sufficient detail to the public that their model can be replicated, [harmful capability] has effectively been released, if perhaps delayed by a few months at most. However, whoever makes the open-source replication will not be [large corporation], absolving the company of perceived moral (and legal) culpability in whatever follows. If the concern were actually that [harmful capability] would lead to real danger, then the moral thing to do would be not to release the model at all.

There are a few serious problems with this. The most obvious (an...
A game of mattering
by KatjaGrace — 2022-09-23T02:30:15.714Z

When I have an overwhelming number of things to do, and insufficient native urge to do them, I often arrange them into a kind of game for myself. The nature and appeal of this game has been relatively stable for about a year, after many years of evolution, so this seems like a reasonable time to share it. I also play it when I just want to structure my day and am in the mood for it. I currently play something like two or three times a week.

The game

The basic idea is to lay out the tasks in time a bit like obstacles in a platformer or steps in Dance Dance Revolution, then race through the obstacle course grabbing them under consistently high-but-doable time pressure.

Here’s how to play:

Draw a grid with as many rows as there are remaining hours in your hoped for productive day, and ~3 columns. Each box stands for a particular ~20 minute period (I sometimes play with 15m or 30m periods.)

Lay out the gameboard: break the stuff you want to do into appropriate units, henceforth ‘items’. An item should fit comfortably in the length of a box, and it should be easy enough to verify completion. (This can be achieved through house rules such as ‘do x a tiny bit = do it until I have a sense that an appropriate tiny bit has been done’ as long as you are happy applying them). Space items out a decent amount so that the whole course is clearly feasible. Include everything you want to do in the day, including nice or relaxing things, or break activities. Drinks, snacks, tiny bouts of exercise, looking at news sites for 5 minutes, etc. Design the track thoughtfully, with hard bouts followed by relief before the next hard bout.

To play, start in the first box, then move through the boxes according to the time of day. The goal in playing is to collect as many items as you can, as you are forced along the track by the passage of time. You can collect an item by doing the task in or before you get to the box it is in. If it isn’t done by the end of the box, it gets left behind. However if you clear any box entirely, you get to move one item anywhere on the gameboard. So you can rescue something from the past, or rearrange the future to make it more feasible, or if everything is perfect, you can add an entirely new item somewhere.

I used to play this with tiny post-it stickers, which I would gather in a large moving pile, acting as a counter:

Now I just draw the who...
Making Prunes
by jefftk — 2022-09-23T02:13:23.659Z

Since

organizing a group
flour buy in April 2020 I've been occasionally putting in orders
at a

restaurant supply store.
Generally this happens because we're running out of chocolate: I've
failed at maintaining my

taste for bad chocolate,
but if we buy in bulk we can get some

very
good vegan chocolate for $4.20/lb ($0.26/oz). A lot of what they
sell is quite expensive, and everything else is in pretty large
quantities, but lots of interesting options! This time I decided to
get half a bushel of prune plums.

This is really quite a lot of plums: about 22lb. After eating all we
wanted I was worried they might go bad soon, so the kids and I decided
to try making prunes. We cut the plums in half, took out the pits,
and put them on a rack on a baking sheet. I put in a pizza stone to
keep the oven temperature steady, and set the oven to "warm":

While the oven claims this is 170F, they pretty consistently read at
140F, which is a much better temperature for this anyway.

After a day or so they tasted about right, so I took them out of the
oven. You can see that there are quite a few missing from sampling
them as they cooked:

The flavor got much more intense, beyond what you would expect from
simple concentration.

The prunes had lost some juice to dripping down into the pan. I
scraped it up and it's good fruit leather:

I liked these enough that I decided to do another batch of plumns,
this time for a "peach" cake. I
didn't halve them or remove the pits this time, because they don't
need to hold together, and I didn't use a rack:

After about a day I took them out and removed the pits:

I made the peach cake, which is essentially a half inch of cake on top of a
half inch of cooked fruit, and brought it to Tuesday family dinner. While the
kids weren't fans it was popular with the adults. Forgot to take a
picture, sorry!

Now I'm cooking the rest of the plums down to keep in the fridge as
something a bit jam-like (but not jam, that's obsolete).