trentmkelly/LessWrong-43k huggingface dataset:
Goal-directedness: imperfect reasoning, limited knowledge and inaccurate beliefs
by Morgan_Rogers — 2022-03-19T17:28:04.695Z

This is the second post in my Effective-Altruism-funded project aiming to deconfuse goal-directedness. Comments are welcomed. All opinions expressed are my own, and do not reflect the attitudes of any member of the body sponsoring me.

In my first post I started thinking about goal-directedness in terms of explanations, and considered some abstract criteria and mathematical tools for judging and comparing explanations. My intention is to consider a class of explanations representing goals that an agent might be pursuing, and to directly compare these with other classes of explanations of the agent's behaviour; goal-directed behaviour will be behaviour which is better explained by pursuit of goals than by other possible explanatory models.

However, in the process, I wasn't really able to escape the problem I mentioned in the comments of my preliminary post: that goal-directedness seems like it should be mostly independent of competence. Any explanation in the form of a goal, interpreted as a reward function to be optimized, will have low predictive accuracy of an agent's behaviour if that agent happens not to be good at reaching its goal.

I gave the example of a wasp getting caught in a sugar water trap. Explaining this behaviour in terms of the wasp wanting to collect sugar should produce reasonable results, but the explanation that the wasp wants to end up in the trap ranks better (for reasonably general choices of the measures of accuracy, explanatory power and complexity)! Since we have good reason to believe that wasps do not actually desire to be trapped, this problem demands a solution.

Questions I would like to solicit feedback on:

Are there ingredients obviously missing from my explanation break-down in this post?
Can you come up with any examples of behaviour that you consider goal-directed which cannot be easily explained with the structure presented in this post?

NB. I have highlighted terms which I expect to reuse in future in the sense intended in this post in bold.

Constrained rationality[1]

The most straightforward remedy to separating competence from goal-directedness, which I will be exploring in this post, is to extend our (goal-based) explanations to include constraints on the agent. This will enable us to retain an intuitive connection between goal-directedness and optimization, since we could conceivably generate predicted behaviour from a given "...
Thoughts on the SPIES Forecasting Method?
by T431 — 2022-03-19T15:22:43.277Z

Here, I discuss the SPIES forecasting method, and ask for the community's thoughts on it.

Not too long ago, I came across the SPIES (Subjective Probability Interval Estimates) method for judgmental forecasting. The method was developed by Uriel Haran, and seems first to have been published as part of his 2011 dissertation Subjective Probability Interval Estimates: A Simple and Effective Way to Reduce Overprecision in Judgment. Haran writes

Overprecision in judgment is the most robust type of overconfidence, and the one least
susceptible to debiasing. It refers to people’s excessive certainty in the accuracy of their
estimates, predictions or beliefs. Research on overprecision finds that confidence intervals, estimated ranges that judges are confident will include the correct answer, tend to include the correct answer significantly less often than what their assigned confidence level would suggest. For example, 90% confidence intervals typically include the correct answer about 50% of the time (Klayman, Soll, González-Vallejo, & Barlas, 1999).

He eventually makes the claim that SPIES reduces the overprecision of confidence interval forecasts, and evidences this claim with the results of several forecasting experiments he conducted. Participants made forecasts using the following methods: provide a confidence interval that contains the target value 90% of the time; provide a 5% lower bound and 95% upper bound that they believe the target value would not be below and above, respectively; use the SPIES method, which consists of decomposing a numerical range into several intervals, then having participants assign a likelihood of 0-100 for each interval, then normalizing these likelihoods into a probability uniformly distributed over the values in the interval, and then finding the shortest subinterval of the numerical range that constitutes 90% of the cumulative probability.

For example, if we want to forecast the monthly rainfall for NYC in March, we can begin by looking at the following intervals 40-65mm, 66-90mm, 91-115mm, 116-140mm, and >140mm (40mm to 140+mm could have been partitioned into 5, 10, etc... intervals, I just chose 5 for this example). I do not know much about rainfall in NYC, but might assign these intervals the following likelihoods: 35/100, 55/100, 85/100, 25/100, and 5/100, respectively. My probabilities for these intervals woul...
Retrospective: Context and Communication
by The Guild of the Rose — 2022-03-19T14:45:06.297Z
Link: https://guildoftherose.org/articles/retrospective-context-and-communication?utm_campaign=lw

Context and Communication was the third Guild of the Rose course. The course served as an introduction to communication and spanned the four weeks from 12/01/2021 to 01/01/2022. It followed our established flipped-classroom model and was taught by Olivia, one of our Guild members.

For more information, see the linked article.
Notes on Social Responsibility
by David Gross — 2022-03-19T14:44:19.572Z

This post examines the virtue of social responsibility. It is meant mostly as an exploration of what other people have learned about this virtue, though I’ve been selective about what I found interesting or credible, according to my own inclinations.

I wrote this not as an expert, but as someone who wants to learn more and to become better at it. I hope it will help people who want to know more about this virtue and how to nurture it.

I will mostly shy away from writing about specific contentious things that some people consider to be socially responsible. I don’t intend to stake out a position in the “social justice” debates, for example. I plan instead to consider this question: whatever it is that socially responsible people do, how do you become the sort of person who characteristically does that?

(I may not be able to totally avoid the temptation to harangue a bit, however.)

What is this virtue?

“That which isn’t good for the hive isn’t good for the bee.” ―Marcus Aurelius[1]

“Man is by nature a political animal.”[2]
“The polis is a community of equals for the sake of a life which is potentially the best.”[3] ―Aristotle

“In no other realm does human excellence approach so closely the paths of the gods as it does in the founding of new and in the preservation of already founded communities.” ―Cicero[4]

People participate in groups of all sorts: internet discussion fora, churches, fandoms, nations, audiences, mobs, corporations, neighborhoods, book clubs, posses, gangs, unions, classes. When things go well, we get a lot of value from this. Some groups are better than others at contributing to the thriving of those in them; some can even be harmful. Any particular group may have aspects that work better or worse in this way. Groups also may be more or less sustainable: there are potential costs to be borne in starting a group, in the disintegration of a group, and in keeping a group going.

All of which is to say that the flourishing of each of us as individuals is influenced by the health of the groups in which we participate.[5] The health of these groups in turn is in large part a function of the decisions made by the people in them.

The person with the virtue of social responsibility recognizes this, and works earnestly to sustain, defend, and improve the groups that person participates in (or perhaps in extreme cases, to sabotage, disband, or replace tho...
What to do after inventing AGI?
by elephantcrew — 2022-03-18T22:30:29.622Z

Imagine a research team makes a breakthrough when experimenting with neural networks. They train a model with only 1 billion parameters, and it displays general intelligence that exceeds GPT-3s. They conclude that a 100 billion parameter model, possible to train for a few million dollars, very likely would surpass human intelligence.

The research team would likely decide they can’t publish their breakthrough, since that would lead to an arms race between different nations and organizations to create an AGI. Further, the government in the country they reside in, would likely force them to create an AGI as quickly as possible, disregarding their safety concerns.

So, what should they do in this situation?

From my search, I haven’t been able to find any resources on what to do or organizations to contact in such a situation.

So my first question is, does anyone know of resources targeting inventors, or potential inventors, of AGI? (asking for a friend)

Second question is, if such resources do not exist, should we create one?

Third question, if we should prepare something, what should be prepare?
Cold Takes reader survey - let me know what you want more and less of!
by HoldenKarnofsky — 2022-03-18T19:20:24.261Z

I'm interested in readers' takes on what I should be doing more and less of on this blog. I will generally write what I feel like writing, even if my audience has asked for less of it, but knowing what my audience wants will have some effect on what I feel like writing.

If you're up for weighing in on this, I appreciate it! Please use this Google form. It also has some basic questions about my readers that I'm interested in the answers to (all questions are optional).

Estimated time: 10 minutes.

Take the survey!
What Should I Do Today?
by Cui — 2022-03-18T16:36:04.472Z

Skip to the last 3 paragraphs if too long.

What should I do today? This infamous question. I find the start of the day sets the precedent for the remainder—consciously or unconsciously. When you’re not severely unemployed, at least you have somewhere to be most days or at least some kind of tasks to do. But I’ve been in a kind of state of perpetual free-time for a while. You know it’s kind of my dream, and many others’, to have to real financial obligations, no boss demanding from you, a long life ahead of you. But somehow I find myself lost, confused, anxious. Maybe operating “as your own boss” and pursuing meaningful personal projects takes extreme discipline and resolve. Probably clear thinking as well.

So I’m still here wondering. Lunch is in an hour. Certainly I could find something to kill an hour with: wait out my time until I can go to my greatest hedonisitc craving. I feel most carnal when I eat. “Killing an hour” waiting to be a hog would set perhaps one of the worst precedents possible and possibly start a depressive episode.

So should I write down a list of things to do? This writing has been a fun exercise, but is starting to feel like procrastination. Perhaps I am missing a key insight. What I really want to do is work on the thing that will fill me up with enthusiasm to do other things. (THIS DOES NOT MEAN READING THROUGH ENDLESS BLOG POSTS, MAYBE 1 OR 2 USEFUL ONES, BUT IT RATHER INVOLVES ACTUALLY DOING SOMETHING, PRODUCING SOMETHING, LEARNING SOMETHING). I really should try to solve the most important problem right?

So, I think the most important problem is what in the world should I do in the next year, in the next 5 years, and with my life? Narrowing down, what should I do professionally? My current situation: I will have a Chemistry degree from a top 20 (though lesser known) university. I don’t know or much like Chemistry though. Biology is ok. Healthcare is ok. What I am REALLY interested in is neuroscience/mindfulness/meditation (very important tool to lessen suffering), neurotechnology more broadly, startups, mental health, maybe AI/ML (though it’s over my head right now). These are my current passions (though my passions seem to shift every couple months).

So what are my current options? Find research in mindfulness. Find people at the cutting edge of mindfulness. Find startups in mindfulness. Qualia Res...
Only needs a flying saucer
by Mary Chernyshenko — 2022-03-18T15:11:38.934Z

This is a personal entry. I do not write it to convey much information, it's more of a point of view thing. I was asked for an inside view, but I guess that's not exactly it.

My mother, my son and I are now in the Czech Republic. This we owe to the goodwill and kindness of very many people, starting with a random guy on the way to the train station in Kyiv. To really describe it, I would need to put "incredible" before every second word, so, like. Let it just be "incredible people".

And now I am in a position to extend the invitation to another friend or family member. I have to choose between N., a mother with her two children and another N., also a mother with two children. Or between one of the mothers-and-kids sets and O., my pregnant sister. Or between O. and the bedridden son of A., the man to whom I owe not jumping out the window. (Not literally. Maybe even not nearly this. Don't know for sure. Only know for sure that A. is not going to leave Kyiv if his son is there. And there is this interesting detail that I am not in direct contact with his son's caretaker. So if anything happens to the man who talked me out of doing stupid things with my life, I will have no way of knowing if anything happens to his son. And I know that, and A. knows that. And A. asks for nothing. Well, he has asked if I have a flying saucer. I don't.)

And if anyone of them manages to come to Czech - it might take a week or more just to come here to Czech - my debt to the master of the house where we are staying grows, well, incredibly much more... My debt to the master of the house who wants himself a Ukrainian wife who knows how to cook and manage the household...

I will have to somehow find a job to help the household because the subsidy for hosting Ukrainians is capped at three Ukrainians. I don't know how I will do it. There is enough work about the house that some nights I don't read any news after dinner; I go to sleep.

My mother is unwell. She doesn't speak Czech. My ten-year-old son is going to start school here. He doesn't speak Czech. I don't speak Czech, and most people I have met here do not speak English, German, Ukrainian or Russian. Frankly, I have my hands full without a job.

And the people back home are hiding in their cellars from bombs falling overhead and hoping,
hoping,
hoping
somebody would tell them they can come and not be alone.
Learned Blankness and Expectations of Solubility
by brook — 2022-03-18T12:42:49.723Z

Epistemic status: babble

I've observed a difference between the smart and the very smart. It seems like a refinement of learned blankness which I've found useful for spotting and (where appropriate) avoiding it.

Most people have large swathes of learned blankness. When I point this out to smart people, one common answer I get is something like "That kind of problem isn't solvable". Believing a solution is impossible, they (quite reasonably) decide not to waste their time looking.

The very smart tend to expect things to be solvable. This doesn't seem to me like an explicit belief so much as a high-level generator or an alief, but it is there. This is the reason people will look at you funny when you say things like "I attached a string to my bedside lamp switch so I can turn it off while in bed" or "Our boiler was making funny noises, and its pressure gauge is outside the range in the manual, so I opened the valve briefly".

An added benefit of the latter approach is that you (hopefully) develop a sense for which types of problems really aren't solvable, leading you to (hopefully) more often tackle truly solvable problems. This should, in turn, improve your solubility-alief to be well-attuned to the world, rather than assuming things-in-general are unsolvable.

Unfortunately, there's a pretty obvious confounder of this observation. The smart-but-not-very-smart could just be right. We don't have to presume that solubility-alief causes you to be smarter; it also seems pretty plausible to me that this alief could come about in the already-very-smart because they really do solve problems more often.

Indirect evidence against this confounder: many of the problems solved because of solubility-alief don't seem especially... well, hard. Like clogged drains, it doesn't seem like an IQ of 150 is required to 1. spot how annoying it is to have to get out of bed to turn your lamp off, and 2. generate the idea of a piece of string. The hard part seems to be somewhere in the valley between 1 and 2, where you have to actually believe that minor annoyances aren't an inherent and unfixable facet of life.

I'm not sure what other observations I'd expect to differ between these two models, short of success or failure of attempts to train/learn solubility-alief. Let me know if you can think of any.

To finish, some exercises for you, if you're an exercises...
On expected utility, part 2: Why it can be OK to predictably lose
by Joe Carlsmith — 2022-03-18T08:38:52.045Z

(Cross-posted from Hands and Cities)

Previously in sequence: Skyscrapers and madmen

This is the second essay in a four-part series on expected utility maximization (EUM). This part focuses on why it can make sense, in cases like “save one life for certain, or 1000 with 1% chance,” to choose the risky option, and hence to “predictably lose.” The answer is unsurprising: sometimes the upside is worth it. I offer three arguments to clarify this with respect to life-saving in particular. Ultimately, though, while EUM imposes consistency constraints on our choices, it does not provide guidance about what’s “worth it” or not — and in some cases, the world doesn’t either. Ultimately, you have to decide.

I. Sometimes it’s worth it

When I first started looking into expected utility maximization (EUM), the question of “why, exactly, should I be OK with predictably losing in one-shot cases?” was especially important to me. In particular, it felt to me like the fact that you’ll “predictably lose” was the key problem with fanaticism (though some fanaticism cases, like Pascal’s Mugging, introduce further elements, like an adversarial dynamic, and/or a very “made-up” tiny probability as opposed to e.g. a draw from a ludicrously large urn). But it also applies (to a far milder degree) to choices that EUM fans endorse, like passing up (A) saving one life for certain, in favor of (B) a 1% chance of saving 1000 lives.

Now, I’m not going to tackle the topic of fanaticism here. But I think the EUM fans are right about B over A – indeed, importantly so. What’s more, I think it’s useful to get a more direct, visceral flavor for why they’re right – one that assists in navigating the emotional challenge of passing up “safer,” more moderate prizes for the sake of riskier, larger ones. This essay draws on moves from the theorems I’ll discuss in parts 3 and 4 to try to bring out such a flavor with respect to life-saving in particular.

I’ll flag up front, though, that here (and throughout this series) I’m going to be focusing on highly stylized and oversimplified cases. Attempts to apply EUM-ish reasoning in the real world are rarely so straightforward (see e.g. here for some discussion), and I think that hesitations about such attempts sometimes track important considerations that naive estimates leave out (for example, considerations related to how much one expects the relevant probabi...
Nuclear Deterrence 101 (and why the US can't even hint at intervening in Ukraine)
by Darmani — 2022-03-18T07:25:25.473Z

This is a linkpost for https://acoup.blog/2022/03/11/collections-nuclear-deterrence-101/ . I found it a very good read for explaining the strategy behind the decisions and signaling in this war. I was inspired to post it as a supplement to https://www.lesswrong.com/posts/WX7tpnBCHWrmJcDym/why-a-no-fly-zone-would-be-the-biggest-gift-to-putin-and-why , as this piece explains why it's in Zelensky's interest to continue to call for a no-fly zone that could turn into a hot war.

It also answered a question I've had since a child, visiting old friends of my mother from her childhood growing up on military bases, wondering what the US was doing in so many countries. The answer? Their job is quite literally to die to provoke a US response.

Some excerpts below:

One such method that Beaufre discusses is what he calls the ‘piecemeal maneuver,’ but is often in English referred to as ‘salami tactics’ – including in this absolutely hilarious bit from Yes, Prime Minister, which is also a surprisingly good explanation of the method. The idea is that to make gains while avoiding escalation, a state can break up the gains they would make into a series of smaller actions, each with its own exterior maneuver ‘cover,’ so that it doesn’t rise to the level of triggering nuclear escalation. Putting together several such maneuvers could allow a state to make those gains which had they all been attempted at once, certainly would have triggered such an escalation. Beaufre’s example, unsurprisingly, was Hitler’s piecemeal gains before his last ‘bite’ into Poland triggered WWII.

Beaufre notes that for piecemeal maneuvers to be effective, they have to be presented as fait accompli – accomplished so quickly that anything but nuclear retaliation would arrive too late to do any good and of course nuclear retaliation would be pointless: who is going to destroy the world to save a country that was already lost? Thus Beaufre suggests that the piecemeal maneuver is best accomplished as a series of coups de main accomplished with fast moving armored, mechanized and airborne forces seizing control of the target country or region before anyone really knows what is happening. The attacking power can then present the maneuver as fait accompli and thus the new status quo that everyone has to accommodate; if successful, they have not only made gains but also moved everyone’s red lin

...
Our time in history as evidence for simulation theory?
by Garrett Garzonie — 2022-03-18T03:35:46.225Z

Probably AI development and alignment is something that civilizations want to simulate. Seems like a given civilization will simulate this much more than any other time in history. We live during the development of AI, is this evidence that we live in a simulation?

I’m confused by the anthropic principle-y aspects of this (a neolithic person wouldn’t see their time of existence as evidence against simulation theory). Has this argument been made before? If so where should I read about it?