trentmkelly/LessWrong-43k huggingface dataset:
Mental nonsense: my anti-insomnia trick
by DirectedEvolution — 2022-03-28T22:47:03.434Z

My dad used to suffer from insomnia, holding imaginary meetings in his head late into the night. I'm the same way.

One stress-relief technique from Johns Hopkins is a breathing + mantra exercise. I find their mantra recommendation cringey:

Breathing in I am calm, breathing out I am coping.

Mantras, guided imagery, and autogenic training are generally hard for me, because I feel pressure to "stay focused on the mantra."

BREATHING IN I AM CALM, BREATHING OUT I AM COPING

My weird trick is to replace the mantra with free-form inner-monologue nonsense syllables. Here's an example:

Nimbla doobla deeble dee... simba dimba lima nooble doo...

I let the pacing and "mental voice" vary, but mainly aim for a sleepy-sounding tone.

At the same time, I let myself visualize nonsense images. My visual imagination isn't very vivid, and I don't try to create any specific images. It's nothing more than a patchy and low-res mental screensaver. Not even this exciting:

This puts me right to sleep.

If you have refinements of anti-insomnia tricks that aren't on bog-standard lists like this one, go ahead and add them in the comments!
Vaniver's ELK Submission
by Vaniver — 2022-03-28T21:14:37.019Z

You can see the actual submission (including a more formalized model) here, and the contest details here. I've reordered things to be more natural as a blog post / explain the rationale / intuition a bit better. This didn't get a prize, tho it may have been because I didn't try to fit the ELK format.

The situation: we have the capacity to train an AI to predict physical reality quite well. We'd like to train an AI (the "SmartVault") that uses that understanding of reality to protect a valuable diamond inside the vault. Ultimately, we'd like the system to protect the diamond and not deceive the owner of the diamond; proximally, we'd like to make a Reporter that can figure out, from the latent knowledge of the SmartVault, whether it's protecting the diamond or deceiving the owner.

I roughly think the proximal goal is doomed, and that the hope needs to be prospective instead of retrospective. [That is, we need to have a system that we can see ahead of time will try to protect the diamond instead of us checking after the fact for deception, because that can just move the meta-level on which deception occurs.] So I'm going to try to propose a solution that de-emphasizes the Reporter, and see how far I get.

Setup

We can roughly model SmartVault as being composed of some subsystems: the Predictor (which, given a trajectory of actions, estimates their likely result), a Reporter (which, given access to the predictor, answers specific questions about the result), and an Optimizer (which, given a situation, tries to decide which trajectory of actions to take). Obviously, the Predictor and the Optimizer are going to be intimately connected to each other, and the ontology of the Predictor is going to be jointly optimized by reality and the training feedback (i.e. the structure of the model's knowledge will reflect both the underlying reality and the salience landscape of the humans labelling examples as 'good' and 'bad'). I'm going to assume that 'the human value function' is part of the 'latent knowledge relevant to prediction (of training reward)', even if it's implicit instead of explicit.

While we’d like to train our machine on (reward, state of reality) pairs, unfortunately we only have access to (reward, human observation) pairs for training; that is, we only know how to communicate the lower-level salience landscape. This means a core challenge is ontological identific...
[ASoT] Some thoughts about deceptive mesaoptimization
by leogao — 2022-03-28T21:14:27.217Z

Editor’s note: I’m experimenting with having a lower quality threshold for just posting things even while I’m still confused and unconfident about my conclusions, but with this disclaimer at the top. Thanks to AI_WAIFU for discussions.

I think of mesaoptimization as being split into different types: aligned (which is fine), misaligned (which is bad), and deceptively aligned (which is really bad - I draw the line for deceptively aligned when it actually tries to hide from us instead of us just not noticing it by accident). I mostly think about deceptive mesaoptimization, and I think of regular misaligned optimization as basically an easier case because at least it’s not trying to hide from us (what it means for it to actually be "trying to hide" is kinda fraught anyways so this sidesteps that). In particular, I think of mesaoptimizers as being dangerous because they fail in a

malign

way (I don't think mesaoptimization itself is necessarily bad, and preventing it doesn't seem obviously good).

OOD-focused framework of deception

One framework for thinking about deceptive mesaoptimizers that I used to use a lot was that they would be good on the training distribution, but then do bad things out of distribution. In this framework, what we would do is then to try to somehow tell during training whether a model is going to do bad things out of distribution, and only keep the ones that don’t, or something along those lines. The intuition is that the mesaoptimizer cares about something in the real world, but it knows that if we catch it then it will not be able to achieve its goals in the real world, so it behaves normally on the training distribution when we’re looking at it but then in deployment it goes and does evil things to satisfy its mesaobjective. Importantly, pursuing the mesaobjective hurts the base objective (since we’re only focusing on inner alignment here we can assume that the base objective is accurate), so deployment is the only time where it can do evil things without getting itself removed.

The main reason here that deception happens is that your base objective isn’t being integrated over the set of all possible states. In theory, if we could enumerate the entire (huge) input space, we could eliminate deceptive behavior entirely. We'd just simply test every single input and see how whether there are any regions where the model does a treache...
Ukraine Post #7: Prediction Market Update
by Zvi — 2022-03-28T16:10:01.427Z

It has been about two weeks since I last looked at the prediction markets for the invasion of Ukraine. Since then, prices on many markets have shifted quite a lot, so it is once again time to examine what the markets seem to be telling us.

Due to time constraints of various sorts, I have needed to split off both the discussion of how we might improve Metaculus, Polymarket and other prediction markets (other than a few markets I suggest adding) and also discussion of big markets like grain, oil and currencies that can serve as noisy but very useful predictive proxies. I hope to get to all that soon.

Kyiv

I still see some reports that Russia is attempting to ‘encircle’ Kyiv, but I no longer take them seriously. They are setting up defensive positions in various places, and if anything Ukraine is launching counter-offensives. Russia’s own official line is that the attack’s goal was to tie down Ukraine’s forces.

That doesn’t mean they couldn’t possibly ever do it, but it would presumably only happen after a complete Ukrainian collapse elsewhere.

Will Kyiv fall by April 1? (14% → 0%)

No. Can’t happen at this point. This closed so can’t move any lower.

Will Kyiv fall by June 1? (39% → 3%)

Again, no. In theory could happen, but it won’t.

The circles are my predictions. You can see the problem with ‘keeping pace’ with things as something falls over time. If I’d had better tools I’d have been lower along the way.

Will Kyiv fall by 2023? (45% → 9%)

I mean, possibly? But again, not really, no.

If Kyiv falls in 2022, when will it happen?

Notice the drop-off in participation. Here is what the distribution curve looks like (I set the input to flat to avoid distraction, ignore that).

Note that the 6% chance of Kyiv falling after June 1 but during 2022 has remained unchanged. The market already understood that there was little chance of Kyiv falling slowly rather than there being a settlement or things becoming a stalemate. Given that a third of worlds where Kyiv falls earlier have now been ruled out, this could be viewed as being conditionally slightly more likely, which is a mistake, it seems clearly conditionally less likely given Russia’s increasing focus elsewhere and willingness to settle for territorial gains.

It also doesn’t make sense in the same world as a collapse of the chances of it happening in April and May, to then have c...
Consequentialist veganism 
by tslarm — 2022-03-28T01:48:21.786Z

I may be wrong, but I think the following is a mainstream position in rationalist circles: even people who care about animal welfare don't have particularly strong moral reasons to personally switch to a vegan diet.

I haven't seen a fully fleshed-out defence of this position. I can think of a few possible arguments, but none seem convincing:

Any single consumption decision probably won't make any difference

That is true, but my model of how this works is that, very roughly speaking, each decision to abstain from consuming x grams of [animal product] has a 1/n chance of causing some demand threshold to be crossed, such that x*n fewer grams of that product are produced in the next cycle. The upshot is that, in expectation, your vegan diet reduces the number of animals put through the agricultural process (or, in the case of e.g. wild-caught fish, the number being caught by. humans) by roughly as much as you would think if you simply added up your counterfactual consumption and divided it by the number of animals required to produce it.

The only way to get rid of harmful animal agriculture is through mass persuasion and/or coercion and/or technological progress, not individual (non-)consumption

Yes, but so what? This is clearly not an all-or-nothing cause; each instance of suffering matters, regardless of how many other instances exist. Nor are personal consumption change and social/technological change mutually exclusive or even conflicting.

There are higher-impact uses of your (time/energy/money/etc.)

Maybe -- though I suspect that the rosier estimates of e.g. the impact of donating to an advocacy organisation tend to be significant exaggerations -- but most of us do not have good reason to treat this as a zero-sum game in which each attempt to do good in the world must crowd out another. For one thing, we're nowhere near putting all available resources into our efforts to do good, so we can simply choose to expand that budget. For another, our psychology is complicated, and making a moral effort can just as easily increase our capacity to make further such efforts as deplete it. (As for money in particular: unless you're already eating unusually frugally, you can probably find a healthy vegan diet that doesn't require significant extra expenditure.)

[Something about indirect effects on e.g. wild-animal suffering or insects]

I don't dismiss these arguments on pr

...
Toward a general structure of economic explanation
by physicaleconomics — 2022-03-27T23:33:57.994Z

I got some very good comments on

this post

(thank you all!), which have focused the analysis of the difficulty-of-analysis of bureaucracy on issues relating to imprecise, incomplete, and ambiguous incentives, and selection effects or a lack thereof. I think all of this is substantially correct.

Now I’d like to go a bit further.

My intellectual interest in the ability of economics to explain bureaucracy pertains to the research I do, which is on a more general question about the structure of economic systems, with the hopes of learning how to build flexible economic mechanisms that are more oriented toward human values. To do this, we need to be confident in our ability to understand the structure of the economic systems that currently exist. Bureaucracies are one such system, and the one most troubling to me from an intellectual perspective, regardless of the good or bad we might associate with them. Bureaucracies play a significant role in our economy, whether operating within a firm or in partnership with or the authority of a government. And every single part and piece of a bureaucracy would seem to obey economic logic, from the individual utility-maximizing people who work for and with the bureaucracy to the budgets that bureaucracies must manage and the accordant interest in controlling costs and maximizing revenues. And yet, despite all of this economic stuff going on, and despite the effect it has on the economy, economists don’t have a basic theory of bureaucracy, and they don’t even seem to be trying to develop one.

I submit that this situation, although one we are all very used to, is nevertheless slightly weird.

The best way to develop an economic theory of bureaucracy should be to observe bureaucracy until you understand its general underlying logic. But this doesn’t seem to have worked. Instead, the more I’ve thought about this issue, the more my attention has been led instead to the general building blocks of any economic model. And it seems plausible to me that the blocks are not all there when it comes to bureaucracy.

Not that bureaucracy is literally unexplainable in economic terms. But I fear that such an explanation would be analogous to the ability of a physicist to explain anything in terms of the underlying motion of every fundamental particle: true, but not very useful, and awfully hard to implement in practice.

I think that...
How many generals does Russia have left?
by Yitz — 2022-03-27T23:11:03.857Z

I’ve come across a large number of articles talking about the unprecedented number of generals (7) killed so far during the current war (see https://www.washingtonpost.com/world/2022/03/26/ukraine-russan-generals-dead/ for instance), but I can’t easily find information online about how many generals Russia actually has. Without a sense of how common and how crucial generals are in the Russian military, it’s hard to make any particular update on the situation. The way that the news has been worded suggests that seven is a high percentage in this context, but considering the level of positive propaganda from the media, I’m not sure I should put any trust into the framing here. If anyone could share further information about the number/strategic importance of Russian generals, it would be greatly appreciated.
Do a cost-benefit analysis of your technology usage
by TurnTrout — 2022-03-27T23:09:26.753Z

If an unaligned entity invests billions of dollars into an application which you use, where they benefit from wasting your time, and you haven’t at least done a cost-benefit analysis so that your usage minimizes your costs and maximizes your benefits—You are probably getting fucked over.

Mistake: Motivatedly avoiding thinking about the issue

Last summer, my friend Kurt Brown told me about Digital Minimalism. The modern world is mired in attention-sucking apps which compete to waste as much of your time as possible. The book’s remedy: stepping back from non-essential internet usage, so that you can evaluate what really matters to you. After a month has come and gone, you add back in those digital activities which are worth it to you.

Unfortunately, this is the part of the story where we all cringe at my past behavior. I gave Kurt some excuses, demurring from his implicit recommendation that I read the book. I asked more questions, but so that I could learn more about what he’d been up to. I wasn’t going to actually do it. I think it sounded monastic and uncomfortable and I’m not one of those people who needs it, I already have lots of locks on my devices.

And locks I had. I restricted my iPhone with a password only known by a friend, so that I was unable to access eg Reddit without wiping my device, or asking my friend for the code. My phone was in black and white to minimize how appealing it would be, I had an outdated model to make using my phone less enjoyable. I didn’t have notifications for anything but phone calls. I still wasted several hours a day on my phone, although I was always (motivatedly) surprised by this. I thought I was spending at least 70% of my phone-time productively, by reading LessWrong and Wikipedia, or engaging in work communication. In this scenario, I didn’t want to upend my life for a month in order to save less than an hour a day (even though it still would have been worth it in the long run).

This school year, I’ve had problems focusing and relaxing. I tried exercise, different medication, but nothing hit the spot. I wasn’t reading textbooks like I wanted to, my attention was fractured, I often felt behind my schedule. I was still doing my job and making progress—just not as much as I wanted.

Could this have anything to do with my attention problems?

This spring, I read a LessWrong post which mentioned Digital Minimalism. Luckily, this...
Your specific attitudes towards AI safety
by Esben Kran — 2022-03-27T22:33:45.102Z

Hi everyone! We would love your responses to this survey about AI risk attitudes given personal context: Google Forms link.

Edit: We updated the survey for the feedback given in the comments. If you already answered it before, you don't need to do it again.

Further information

The unique value proposition is to give an idea of the attitudes towards AI safety in the context of your initial exposure point, knowledge level and demographic whereas other surveys often focus on pure forecasting and field overview. It sprung from this article and further discussion about how people initially came to know about AI safety. This survey is focused on AI safety-engaged people while the next survey will be reformulated for a non-EA, non-rationalist community to help in outreach strategy designs.

We have shared the survey in an array of groups and forums (EA Forum, AI Safety Discussion FB group and more) and expect ~100 responses. The specific contrasts we're hoping to infer and analyze are:

Prior knowledge <> (% AGI developed <> % AGI dangerous)
(First exposure point <> Initial impressions) <> (Currently convincing arguments <> Current impressions)
(Occupation + Age + Country) <> [*]

We are expecting to publish the results on LessWrong and EA Forum with in-depth exploratory analyses in the light of the contrasts above.

Subjectivity is part of the survey and one of the reasons we made it. Be prepared for ambiguous questions.
Sums and products
by Ege Erdil — 2022-03-27T21:57:38.410Z
Link: https://www.metaculus.com/notebooks/10464/sums-and-products/

Much of number theory is about trying to understand the interaction between two different structures on the integers (or the natural numbers): the additive structure and the multiplicative structure.

To clarify what I mean, consider the
twin prime conjecture, one of the famous open problems in analytic number theory. The conjecture asserts that there are infinitely many twin primes, in other words, primes p such that p+2 is also prime. Being a prime number is a multiplicative property: a natural number ≥2 is defined to be prime if it's only divisible by itself and 1. In contrast, adding two to a number is an additive operation. Therefore the twin prime conjecture is asking a question about how these two structures interact with each other.

A substantial chunk of the open problems in number theory are about trying to get a better understanding of the interaction between these two structures. In this notebook, I'll be focusing on one of the most powerful conjectures in this area: the abc conjecture.

Randomness

Before I talk about any specific problems, I'll take the time to introduce the concept of heuristics based on randomness. These are suggestive probabilistic arguments that "prove" a result under false assumptions that different events are independent when they are really not so.

Let's see how we can "prove" the twin prime conjecture using such methods. Suppose that we pick a large natural number N and we pick a natural number n from the interval [N,2N] uniformly at random. What's the chance that n and n+2 form a twin prime pair?

Well, we know by the prime number theorem that there are roughly ∼N/logN primes in this interval, so if we say E1 is the event of n being prime and E2 is the event of n+2 being prime, we'll have roughly P(E1)≈P(E2)∼1/logN. We want the probability that both of them are prime, in other words, we want P(E1∩E2).

So far everything we've done is rigorous. Now we'll introduce the heuristic: we'll assume that the events E1 and E2 are independent, which amounts to assuming that in this case the additive and multiplicative structures on the integers are "orthogonal" in some sense. Under this assumption, we can deduce

P(E1∩E2)=P(E1)⋅P(E2)∼1log2(N)

In other words, the probability of a natural number chosen uniformly at random from [N,2N] being the smaller of a pair of twin primes is some constant times 1/log2(N). That tells us that there are roughl...
[ASoT] Searching for consequentialist structure
by leogao — 2022-03-27T19:09:13.370Z

Editor’s note: I’m experimenting with having a lower quality threshold for just posting things even while I’m still confused and unconfident about my conclusions, but with this disclaimer at the top. Thanks to Kyle and Laria for discussions.

In

“Searching for Bayes Structure”

, Eliezer argues that things which arrive at the truth must be doing something Bayes-like somewhere in there, even if it’s extremely inefficient or accidental or filtered through some process that looks very different from the outside. If it has no Bayes structure, it literally cannot arrive at the truth.

I think it’s plausible that Bayes is to epistemic rationality as consequentialism is to instrumental rationality. Here, by consequentialism I mean the ability to get to some goal from a broader set of initial states, which is how I understood Elliezer’s use of the term in the dialogues. It does not have to be an EUM/VNM-rational. Importantly, to be good at consequentialism comes with all the stuff like instrumental convergence (and consequently self preservation, power seeking, etc).

In other words, there’s some minimal amount of consequentialism you need to achieve a given goal, and there’s no getting around this lower bound. Of course, some agents do a lot more consequentialism than is necessary, but I interpret some people (i.e Eliezer) as arguing that even this lower bound is too much consequentialism for, say, performing a pivotal action, because it’s fundamentally very difficult to do. As an analogy, if your goal is to buy an apple for $1, any plans which fail to collect at least $1 will simply not be able to acquire an apple, and even though there also exist plans which collect $100 (which is more dangerous than collecting $1), if even collecting $1 is too dangerous by default, then even lobotomizing a $100-collecting agent to only collect $1 is scarcely comforting. The main lesson I draw from this is that if we want to do something consequentialist then we have to somehow figure out how to align consequentialists, and that making the agent worse at consequentialism isn’t a solution. I think it also seems productive to figure out where the lower bound of how much consequentialism we need (for, i.e a pivotal action) and the upper bound of how much consequentialism is safe are relative to each other.

As an aside, my intuition for why instrumental convergence is unavoid...