trentmkelly/LessWrong-43k huggingface dataset:
Slack matters more than any outcome
by Valentine — 2022-12-31T20:11:02.287Z

About a month ago Ben Pace invited me to expand on a point I'd made in a comment.

The gist is this:

Addictions can cause people to accomplish things they wouldn't accomplish otherwise.
But if the accomplishment were worthwhile, why would the addiction be helpful? Why wouldn't the clarity that it's worthwhile be enough?
I postulate that the reason is a kind of metaphorical heaviness in culture. A particular structural tendency to eat slack.
So I think it'd be net better to let some worthwhile things go unaccomplished in favor of lifting the metaphoric burden. Creating more slack.
And I'd even say that this is the main plausible pathway I see for creating a great future for humanity. I don't think we can get there by focusing on making worthwhile things happen.

I felt inspired to write up an answer. Then I spent a month working on it. I clarified my thinking a lot but slid into a slog of writing. Kind of a perfectionist thing.

So I'm scrapping all that. I'm going to write a worse version, since the option is (a) a quickly hacked together version or (b) nothing at all.

Addictions

My main point isn't really about addictions, but I need to clarify something about this topic anyway. They're also a great example cluster.

When I say "addiction", I'm not gesturing at a vague intuition. I mean a very particular structure:

There's some unwelcome experience that repeatedly arises.
There's a behavior pattern that can temporarily distract the person in question from the unwelcome experience.
But the behavior pattern doesn't address the cause of the unwelcome experience arising in the first place.

So when someone engages in the distraction, it provides temporary relief, but the unwelcome experience arises again — and now the distraction is a little more tempting. A little more habit-forming. When that becomes automatic, it can feel like you're trapped inside it, like you're powerless against some behavior momentum.

Which is to say, this structure eats slack.

Some rapid-fire examples:

Caffeine dependency becomes an addiction when you autopilot react to the withdrawal symptoms by reaching for another cup of coffee.
Alcoholism as an addiction is often (usually? always?) about avoiding emotional experiences. Since the causes of the emotions don't go away, sobriety can result in the unwelcome experience arising, which the alcoholic knows how to numb away.
I have a long-standing habit of feel

...
To Be Particular About Morality
by AGO — 2022-12-31T19:58:12.259Z

I'm beginning this post with the assumption that there is no "moral reality," namely that there is no objective truth about what we should or should not do.[1]

Instead, I will (I hope non-controversially) consider morality to be some set of emotional intuitions possessed by any given individual. These moral intuitions, like any other human quality, were formed through a combination of evolutionary benefit and cultural context. And, as a notable characteristic, these intuitions do not form a particularly self-consistent system. A common example used here is the failure to scale: the resources we would expend to do some moral good X three times is not necessarily thrice the amount we would expend to do X once.

Let us consider a prototypical rationalist, B, who dislikes this inconsistency and instead wants to create a set of general moral principles[2] which acts as a consistent formalization of his moral instincts. These principles may be rules of action, a metric for determining the goodness of ends, or really any systematization which deals in abstract moral terms. Getting started, he thinks to himself, "While I do not emotionally care 1000 times more that X happens one billion times than that X happens one million times, I ought to act as though I do so that my behavior will be logically consistent."

Of course, since our moral instincts are inconsistent in many ways, anyone attempting to create a consistent system of formal moral principles must make some compromises. Suppose B likes pure utilitarianism as his formal moral system, but he instinctively despises the idea of a utility monster. B decides he has two options. First, he could stick with pure utilitarianism and concede that he is obligated by his moral system to satiate the utility monster. Or, second, he could abandon pure utilitarianism and augment his moral system so that he is no longer required to satiate the monster.

B eventually arrives on a consistent system of general principles that he feels produces very similar answers to his moral intuition in most cases. B resolves to abide by this system, even when it produces an answer that clashes with his moral intuition. Over time, he may even be able to develop a strong enough emotional attachment to his moral system that he really feels that the utility monster deserves to be satiated, even when the idea was revolting to him ...
200 COP in MI: Interpreting Algorithmic Problems
by Neel Nanda — 2022-12-31T19:55:39.085Z

This is the fourth post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability.

Start here

, then read in any order. If you want to learn the basics before you think about open problems, check out

my post on getting started

. Look up jargon in

my Mechanistic Interpretability Explainer

Motivation

Motivating paper: A Mechanistic Interpretability Analysis of Grokking

When models are trained on synthetic, algorithmic tasks, they often learn to do some clean, interpretable computation inside. Choosing a suitable task and trying to reverse engineer a model can be a rich area of interesting circuits to interpret! In some sense, this is interpretability on easy mode - the model is normally trained on a single task (unlike language models, which need to learn everything about language!), we know the exact ground truth about the data and optimal solution, and the models are tiny. So why care?

I consider my work on grokking to be an interesting case study of this work going well. Grokking (shown below) is a mysterious phenomena where, when small models are trained on algorithmic tasks (eg modular addition or modular division), they initially memorise the training data. But when they keep being trained on that data for a really long time, the model suddenly(ish) figures out how to generalise!

In my work, I simplified their setup even further, by training a 1 Layer transformer (with no LayerNorm or biases) to do modular addition and reverse engineered the weights to understand what was going on. And it turned out to be doing a funky trig-based algorithm (shown below), where the numbers are converted to frequencies with a memorised Discrete Fourier Transform, added using trig identities, and converted back to the answer! Using this, we looked inside the model and identified that despite seeming to have plateaued, in the period between memorising and "grokking", the model is actually slowly forming the circuit that does generalise. But so long as the model still has the memorising circuit, this adds too much noise to have good test loss. Grokking occurs when the generalising circuit is so strong that the model decides to "clean-up" the memorising circuit, and "uncovers" the mature generalising circuit beneath, and suddenly gets good test performance.

OK, so I just took this as an excuse to explain my paper to you. Why should you ...
The Feeling of Idea Scarcity
by johnswentworth — 2022-12-31T17:34:04.306Z

Here’s a story you may recognize. There's a bright up-and-coming young person - let's call her Alice. Alice has a cool idea. It seems like maybe an important idea, a big idea, an idea which might matter. A new and valuable idea. It’s the first time Alice has come up with a high-potential idea herself, something which she’s never heard in a class or read in a book or what have you.

So Alice goes all-in pursuing this idea. She spends months fleshing it out. Maybe she writes a paper, or starts a blog, or gets a research grant, or starts a company, or whatever, in order to pursue the high-potential idea, bring it to the world.

And sometimes it just works!

… but more often, the high-potential idea doesn’t actually work out. Maybe it turns out to be basically-the-same as something which has already been tried. Maybe it runs into some major barrier, some not-easily-patchable flaw in the idea. Maybe the problem it solves just wasn’t that important in the first place.

From Alice’ point of view, the possibility that her one high-potential idea wasn’t that great after all is painful. The idea probably feels to Alice like the single biggest intellectual achievement of her life. To lose that, to find out that her single greatest intellectual achievement amounts to little or nothing… that hurts to even think about. So most likely, Alice will reflexively look for an out. She’ll look for some excuse to ignore the similar ideas which have already been tried, some reason to think her idea is different. She’ll look for reasons to believe that maybe the major barrier isn’t that much of an issue, or that we Just Don’t Know whether it’s actually an issue and therefore maybe the idea could work after all. She’ll look for reasons why the problem really is important. Maybe she’ll grudgingly acknowledge some shortcomings of the idea, but she’ll give up as little ground as possible at each step, update as slowly as she can.

And this is where a bunch of the

standard advice from the sequences

comes in.

Once you’ve chosen the idea, the only way to improve it is to change the idea or choose a new idea; finding arguments that it’s a good idea will not actually make it better

.

You won’t make real progress until you give up

,

say “oops” and move on, so do that quickly rather than slowly

.

What is true is already so; owning up to it will not make it worse

.

… but that adv...
Curse of knowledge and Naive realism: Bias in Evaluating AGI X-Risks
by Remmelt — 2022-12-31T13:33:14.300Z
Link: https://www.lesswrong.com/posts/h3qMKzCrzJvjvSWbR/reactive-devaluation-bias-in-evaluating-agi-x-risks

Curse of knowledge

When better-informed people find it extremely difficult to think about problems from the perspective of lesser-informed people.

Unfortunately, particularly among prominent academic people, with high prestige, along with a natural inclination of topic specialists to view and interact with the world (other people) in fairly unique ways, leads to some isolation inherently, significantly strengthening the tendency to form 'in' and 'out' groups.

This factor combines with the above bias so as to to strengthen its occurrence significantly.

- link Wikipedia: Curse of knowledge
- an item on Forrest Landry's compiled list of biases in evaluating extinction risks.

Naive realism

The belief:

that we see reality as it really is objectively and without bias; that the facts are plain for all to see;
that rational people will agree with us;
and that those who do not are either uninformed, lazy, irrational, or biased.

Strongly associated with the 'Curse of knowledge' bias.

- link Wikipedia: Naive realism
- an item on Forrest Landry's compiled list of biases in evaluating extinction risks.
What career advice do you give to software engineers?
by Antb — 2022-12-31T12:01:16.920Z

I am a defendant of the idea that we have already achieved rudimentary AGIs with modern LLMs (as much of a hot take this is), and even though the path to superintelligence is going to be difficult and will probably require a few more technical breakthroughs to make more effective use of available data, I don't think this will take us longer than a decade, or 15 years at most.

When I discuss this idea with some of my CS friends and co-workers, about how AI will inevitably replace most software engineering jobs (picture supercharged Github Copilot that can make entire websites and back-end services on command) most of them ask me the obvious follow-up question: So what can I do about it? Yeah, I can help with AI safety/alignment progress but my job is going to disappear no matter what I do, and probably sooner than many other more 'physically' demanding ones.

I am always left stumped by this − I simply don't know what to tell them, specially to undergraduates that are still full of hope and totally didn't sign up for this dumpster fire. Should I tell them to just continue doing their thing and see what happens? Let fate take its course and hope for the best? This sounds all too happy-go-lucky for my taste.

I'd like to hear what you guys think about this matter, what do you answer when asked such questions?
Are Mixture-of-Experts Transformers More Interpretable Than Dense Transformers? 
by simeon_c — 2022-12-31T11:34:18.185Z

Intuitively, I would expect Mixture-of-Experts (MoE) models (e.g. https://arxiv.org/abs/2101.03961) to be a lot more interpretable than dense transformers:

The complexity of an interconnected system increases way faster than linearly with the number of connected units. It is probably at least quadratic. Thus, studying a system with n units is a priori way harder than studying 5 systems with n/5 units. In practice MoE transformers seem to require at least an order of magnitude more parameters than dense transformers for similar capabilities but I still expect the sum of complexity of each expert to be much lower than the complexity of one single dense transformer.
MoE forces specialization and thus gives a strong prior on what a set of neurons is doing. Having a prior is probably very helpful to move faster in doing mechanistic interpretability.

So my question is: Do you think MoEs are more interpretable than dense transformers, and is there some evidence of it or the opposite (e.g. papers or past LW posts)?

I think this question matters because it doesn't seem implausible to me that MoE models could be at par with dense models in terms of capabilities. And thus it could be an avenue worth pursuing or promoting if we had strong evidence that they were a lot more interpretable. You can see more tentative thoughts on this here (https://twitter.com/Simeon_Cps/status/1609139209914257408?s=20)
In which cases can ChatGPT be used as an aid for thesis or scientific paper writing?
by Bob Guran — 2022-12-31T10:50:45.141Z

Language models like ChatGPT have gained popularity in recent years for their ability to generate human-like text. In this context, it is natural to wonder if a tool like ChatGPT could potentially be used as an aid for writing a thesis or scientific paper.

As a non-native English speaker, I experimented with using ChatGPT to write introductions and to edit/rewrite my sentences to make them sound more natural in English. Although the text generated by ChatGPT was well-written, it did not include any real references.

For editing, personally, I found the text produced by ChatGPT to be a superior alternative to my own writing, although I may be overly critical of my own writing style.

Doing this experiments some concern came to my mind and I am sure that I am not alone in this

How might ChatGPT be used as an aid for writing a thesis or scientific paper, if at all?
What ethical considerations should be taken into account when using ChatGPT as an aid for writing a thesis or scientific paper?
In what ways might the use of ChatGPT as an aid for academic writing be considered plagiarism, and how can these issues be avoided?
Two Issues with Playing Chicken with the Universe
by Chris_Leong — 2022-12-31T06:47:52.988Z

If you are unfamiliar with the 5-and-10 problem, please refer to the Action Counterfactuals section of this post on Embedded Agency. I regret that I am unable to recommend a resource for learning about the concept of "Playing Chicken with the Universe". If any reader has recommendations for such a resource, I welcome their suggestions in the comments section below.

Consider a variant of the 5-and-10 problem where the agent visualises an image of the number they are going to select immediately before making their decision, and never entertains such an image at any other time. Further, imagine that we have access to the agent's brain scans, we can use it to demonstrate that the agent will select the number 5. It is highly likely that we would similarly be able to prove that the agent would imagine 5, without first proving that it will choose 5. We should also have enough information about the agent to show that conditional on the agent choosing 10, it will first have imagined 10 so it will imagine both 5 and 10. This contradiction is a spurious counterfactual and it would allow us to prove that in the condition where we choose the option 10 would give us whatever utility we want to prove. Playing chicken with the universe doesn't prevent this as the agent never proves it will or will not take a particular option, but instead proves facts about correlates.

It is possible to demonstrate a similar issue by utilitising perfect predictors instead of making the agent imagine its choice. Imagine we have an agent which chooses 5 and we have access to the agent's brain scans, plus technical details of how the predictor works. In at least some scenarios, we should be able to use our knowledge of the brain scans + how the predictor works to prove that the predictor will predict the agent choosing 5, without first proving anything about what it will choose. Conditional on choosing 10, we could show it would be predicted to take 10, which would again give us a contradiction as we would then be expecting the predictor to predict both 5 and 10. Again, playing chicken with the universe doesn't seem to offer a resolution to this issue.

Playing chicken with the universe is a hack. It attempts to solve the paradox of imagining an agent that takes 5 and then conditioning on it taking 10 by sweeping the issue under the rug. However, even though this patch works in some cases, it does...
Extreme risk neutrality isn't always wrong
by Grant Demaree — 2022-12-31T04:05:18.556Z

Risk neutrality is the idea that a 50% chance of $2 is equally valuable as a 100% chance of $1. This is a highly unusual preference, but it sometimes makes sense:

Exampleville is infested with an always-fatal parasite. All 1,024 residents have the worm.

For each gold coin you pay, a pharmacist will deworm one randomly selected resident.

You only have one gold coin, but the pharmacist offers you a bet. Flip a weighted coin, which comes up heads 51% of the time. If heads, you double your money. If tails, you lose it all.

You can play as many times as you like.

It makes sense to be risk neutral here. If you keep playing until you've saved everyone or go broke, each resident gets a 0.51^10 = 1 in 840 chance of survival. If you don't play, it's 1 in 1,024.

For a few of the most scalable interventions, I think EA is like this. OpenPhil seems to expect a very low rate of diminishing returns for GiveWell's top charities. I wouldn't recommend a double-or-nothing on GiveWell's annual budget at a 51% win rate, but I would at 60%.[1]

Most interventions aren't like this. The second $100M of AI safety research buys much less than the first $100M. Diminishing returns seem really steep.

SBF and Caroline Ellison had much more aggressive ideas. More below on that.

Intro to the Kelly Criterion

Suppose you're offered a 60% double-or-nothing bet. How much should you wager? In 1956, John Kelly, a researcher at Bell Labs, proposed a general solution. Instead of maximizing Expected Value, he maximized the geometric rate of return. Over a very large number of rounds, this is given by

(1+l)P(1−l)1−P

Where l is the proportion of your bankroll wagered each round, and P>0.5 is the probability of winning each double-or-nothing bet.

Maximizing with respect to l gives

l=2P−1

Therefore, if you have a 60% chance of winning, bet 20% of your bankroll on each round. At a 51% chance of winning, bet 2%.

This betting method, the Kelly Criterion, has been highly successful in real life. The MIT Blackjack Team bet this way, among many others.

Implicit assumption in the Kelly Criterion

The Kelly Criterion sneaks in the assumption that you want to maximize the geometric rate of return.

This is normally a good assumption. Maximizing the geometric rate of return makes your bankroll approach infinity as quickly as possible!

But there are plenty of cases there the g...
Verbal parity: What is it and how to measure it? + an edited version of "Against John Searle, Gary Marcus, the Chinese Room thought experiment and its world"
by philosophybear — 2022-12-31T03:46:59.867Z

I've written previously about an idea I call verbal parity, viz:

A machine has verbal parity when, if given input in written form, it can give an appropriate output in written form as a response, as well as a human, for any input.

Now this definition has many problems. For example, there is a huge variation among humans in how well they can respond to inputs. Is this as well as any human? For any prompt? That would be infeasible, even for a vastly superhuman AI ("question, what's in Tom Cruise's pocket right now?" A question only Tom Cruise can answer, presumably). So it's not really a precise concept.

It is, however, somewhat more precise than "General intelligence". It pretty clearly is a kind of generalization of the intuition that infused Turing when he described the imitation game.

I have been interested in this idea for a long time. You can see relevant work here:

Here: https://www.lesswrong.com/posts/GHeEdubBHjcqeoxjP/recent-advances-in-natural-language-processing-some-woolly

And here: https://www.lesswrong.com/posts/mypCA3AzopBhnYB6P/language-models-are-nearly-agis-but-we-don-t-notice-it

And in the appended essay, at the end of this document which I recently made major edits too after some useful feedback [thanks to TAG's suggestions- they were invaluable]

I initially introduced the idea of verbal parity here on my Substack, but I won't repost it in this forum unlike my other essays because I don't think it contains enough useful content. beyond the basic idea I've already introduced.

I'm looking for help

Having written about the topic of verbal parity before, I have a sense that we should try to estimate when it will happen for a number of reasons:

It matters a great deal for AI timelines
When it happens, it will change people's perceptions of AI. The ability to talk to something really shapes your idea of it.
It will be enormously economically, socially and politically disruptive in good and bad ways.
It's more concrete, and thus easier to predict than "AGI".

Now, as I noted above, there are problems with the definition I gave- it's far from precise. Let me try and state it more precisely.

Verbal parity: A machine has verbal parity if and only if, for any task presented in English, it can do better at that task than 95% of English speakers.

This rules out hyper-specific tasks, and also unreasonable expectations like being able to write Nobel prize-winni...
Should AI systems have to identify themselves?
by Darren McKee — 2022-12-31T02:57:11.429Z

A short post to hear what people think of the policy proposal to have any interaction with an AI (eg chatbot) include an overt notification that one is indeed interacting with an AI.
Clearly, one wouldn't want a frequent pop up in video games but in the world of providing services to consumers, such a thing could help reduce confusion among clients/users.
Alternatively, perhaps it isn't that helpful or might be treated like the cookie notices we get and just click accept.
What do you think?