trentmkelly/LessWrong-43k huggingface dataset:
Are there non-AI projects focused on defeating Moloch globally?
by  — 2020-09-14T02:13:11.252Z

Meditations on Moloch lays out a rather pessimistic view of the future, and then offers a super-intelligent AI "gardener" as the solution. A lot of the rationalist community is focused on AI, which makes sense in that light (and of course because of the existential risk of unaligned AI), but I don't know of any projects focused on non-AI solutions to countering or defeating Moloch. Some projects exist to counter specific local coordination problems, but apparently none to counter the global gardening problem in the original post? Am I missing such a project? Is there a reason that AI is the only plausible solution? Is this low-hanging fruit waiting to be picked?

edited to add some clarifications:

By defeating Moloch "globally" I mean in the sense of the global race to the bottom - preventing humanity from "reaching the sea" in the metaphor from the original Meditations on Moloch (which itself is borrowed from the Apocrypha Discordia). This doesn't mean solving all local coordination problems forever, just preventing us from reaching the absolute worst case that Bostrom conjures of our own destruction, the "Disneyland with no children".
Yes, I've read Inadequate Equilibria.
Decision Theory is multifaceted
by Michele Campolo — 2020-09-13T22:30:21.169Z

Related: Conceptual Problems with UDT and Policy Selection, Formalising decision theory is hard

Target

Anyone who is interested in decision theory. The post is pretty general and not really technical; some familiarity with counterfactual mugging can be useful, but overall the required background knowledge is not much.

Outline

The post develops the claim that identifying the correct solution to some decision problems might be intricate, if not impossible, when certain details about the specific scenario are not given. First I show that, in counterfactual mugging, some important elements in the problem description and in a possible formalisation are actually underspecified. Next I describe issues related to the concept of perfect prediction and briefly discuss whether they apply to other decision scenarios involving predictors. Then I present some advantages and disadvantages of the formalisation of agents as computer programs. A summary with bullet points concludes.

Missing parts of a “correct” solution

I focus on the version of the problem with cards and two humans since, to me, it feels more grounded in reality—a game that could actually be played—but what I say applies also to the version with a coin toss and Omega.

What makes the problem interesting is the conflict between these two intuitions:

Before Player A looks at the card, the best strategy seems to never show the card, because it is the strategy that makes Player A lose the least in expectation, given the uncertainty about the value of the card (50/50 high or low)
After Player A sees a low card, showing it seems a really good idea, because that action gives Player A a loss of 0, which is the best possible result considering that the game is played only once and never again. Thus, the incentive to not reveal the card seems to disappear after Player A knows that the card is low.

[In the other version, the conflict is between paying before the coin toss and refusing to pay after knowing the coin landed tails.]

One attempt at formalising the problem is to represent it as a tree (a formalisation similar to the following one is considered here). The root is a 50/50 chance node representing the possible values of the card. Then Player A chooses between showing and not showing the card; each action leads to a leaf with a value which indicates the loss for Player A. The peculiarity of count...
A Brief Chat on World Government
by  — 2020-09-13T18:33:47.009Z

[This is the transcript of a chat conversation I had with another member of my local rationalist meet-up, on the topics of Moloch, world government, and colonization. Lightly edited for clarity, spelling, etc. and shared with their permission. Cross-posted from Grand, Unified, Empty.]

Me: Here are some thoughts on Moloch. Moloch basically guarantees that anybody who can figure out how to successfully convert other values into economic value will out-compete the rest. So in the end, we are the paperclip maximizers, except our paperclips are dollar bills.

Scott proposes that to defeat Moloch we install a gardener, specifically a super-intelligent AI. But if you don’t think that’s going to happen, a world government seems like the next best thing. However if we escape earth before that happens, speed of light limitations will forever fragment us into competing factions impossible to garden. Therefore we should forbid any attempts to colonize Mars or other planets until we have world government and the technology to effectively manage such colonies under that government.

Them: The superorganisms in his parable only function because of… external competitive pressures. If cells didn’t need to band together to survive, they wouldn’t. If governments don’t have to fend off foreign governments they will accumulate corruption and dysfunctions.

Sort of related, I’m not persuaded by the conclusion to his parable. Won’t superintelligent AIs be subject to the same natural selective pressures as any other entity? What happens when our benevolent gardener encounters the expanding sphere of computronium from five galaxies over?

Me: Cells were surviving just fine without banding together. It was just that cells which banded together reproduced and consumed resources more effectively than those which didn’t. Similarly, I think a well constructed world government could survive just fine without competitive pressure. We haven’t necessarily found the form of that government yet, but liberal democracy seems like a decent first step.

Regarding competitive pressure on AI, he deals with that off hand by assuming that accelerating self-improvement gives an unbreakable first mover advantage. I don’t think that’s actually true, but then I’m much less bullish on super-intelligent AI in general.

Them: It wo...
Egan's Theorem?
by johnswentworth — 2020-09-13T17:47:01.970Z

When physicists were figuring out quantum mechanics, one of the major constraints was that it had to reproduce classical mechanics in all of the situations where we already knew that classical mechanics works well - i.e. most of the macroscopic world. Likewise for special and general relativity - they had to reproduce Galilean relativity and Newtonian gravity, respectively, in the parameter ranges where those were known to work. Statistical mechanics had to reproduce the fluid theory of heat; Maxwell's equations had to agree with more specific equations governing static electricity, currents, magnetic fields and light under various conditions.
Even if the entire universe undergoes some kind of phase change tomorrow and the macroscopic physical laws change entirely, it would still be true that the old laws did work before the phase change. Any new theory and any new theory would still have to be consistent with the old laws working, where and when they actually did work.
This is Egan's Law: it all adds up to normality. When new theory/data comes along, the old theories are still just as true as they always were. New models must reproduce the old in all the places where the old models worked; otherwise the new models are incorrect, at least in the places where the old models work and the new models disagree with them.
It really seems like this should be not just a Law, but a Theorem.

I imagine Egan's Theorem would go something like this. We find a certain type of pattern in some data. The pattern is highly unlikely to arise by chance, or allows significant compression of the data, or something along those lines. Then the theorem would say that, in any model of the data, either:

The model has some property (corresponding to the pattern), or
The model is "wrong" or "incomplete" in some sense - e.g. we can construct a strictly better model, or show that the model consistently fails to predict the pattern, or something like that.

The meat of such a theorem would be finding classes of patterns which imply model-properties less trivial than just "the model must predict the pattern" - i.e. patterns which imply properties we actually care about. Structural properties like e.g. (approximate) conditional independencies seem particularly relevant, as well as properties involving abstractions/embedded submodels (in which case the theorem should tell how to find the abstraction/e...
Gems from the Wiki: Acausal Trade
by habryka — 2020-09-13T00:23:32.421Z
Link: https://www.lesswrong.com/tag/acausal-trade

During the LessWrong 1.0 Wiki Import we (the LessWrong team) discovered a number of great articles that most of the LessWrong team hadn't read before. Since we expect many others to also not have have read these, we are creating a series of the best posts from the Wiki to help give those hidden gems some more time to shine.

Most of the work for this post was done by Joshua Fox who I've added as a coauthor to this post, wiki edits were also made by all of the following: Lukeprog, Gwern, Vladimir Nesov, Sauski, Deku-shrub, Caspar42, Joe Collman and Jja. Thank you all for your contributions!

In acausal trade, two agents each benefit by predicting what the other wants and doing it, even though they might have no way of communicating or affecting each other, nor even any direct evidence that the other exists.

Background: Superrationality and the one-shot Prisoner's Dilemma

This concept emerged out of the much-debated question of how to achieve cooperation on a one-shot Prisoner's Dilemma, where, by design, the two players are not allowed to communicate. On the one hand, a player who is considering the causal consequences of a decision ("Causal Decision Theory") finds that defection always produces a better result. On the other hand, if the other player symmetrically reasons this way, the result is a Defect/Defect equilibrium, which is bad for both agents. If they could somehow converge on Cooperate, they would each individually do better. The question is what variation on decision theory would allow this beneficial equilibrium.

Douglas Hofstadter (see references) coined the term "superrationality" to express this state of convergence. He illustrated it with a game in which twenty players, who do not know each other's identities, each get an offer. If exactly one player asks for the prize of a billion dollars, they get it, but if none or multiple players ask, no one gets it. Players cannot communicate, but each might reason that the others are reasoning similarly. The "correct" decision--the decision which maximizes expected utility for each player, if all players symmetrically make the same decision--is to randomize a one-in-20 chance of asking for the prize.

Gary Drescher (see references) developed the concept further, introducing an ethical system called "acausal subjunctive morality." Drescher's approach relies on the agents being identical or at least similar, so t...
Progress: Fluke or trend?
by jasoncrawford — 2020-09-13T00:21:36.025Z
Link: https://rootsofprogress.org/progress-fluke-or-trend

A foundational conviction of The Roots of Progress is that progress is a trend with definite, substantive causes, and that it can continue far, far into the future. Progress is not automatic or inevitable: it can slow, stop, even reverse. But the history of progress over the last 200+ years convinces me that much more is possible.

Not everyone agrees, however. To learn more about how people think about this, I

posed a question on Twitter

:

Do you think the last 200+ years of technological/industrial progress were…

… a trend with substantive causes, that we can expect to continue?

… a fluke, a stroke of luck, not to be repeated?

And why?

After discussing it with people all day, most of the “fluke” arguments were:

Argument from failure of imagination: “I can’t see or imagine any big breakthroughs, therefore I don’t expect any.”
Materialism: Progress is primarily driven by material resources (such as fossil fuels); therefore it will slow when those inevitably run out.

Failure of imagination is not a compelling argument to me, for both logical and historical reasons. The logical reason should be obvious. The historical reason is that the big breakthroughs of the past were not easy to imagine or predict before they happened. In a different context, Eliezer Yudkowsky points out that even the creators of inventions such as the airplane or the nuclear reactor felt that their breakthroughs were fifty years out, or even impossible, shortly before they happened. Now is no different. (This point seems exceedingly difficult to get through to people; no matter how much you point out the logical fallacy, or the historical precedent, they continue to repeat the same points. I don’t know if this is because the logical fallacy itself is unclear, or if it’s just a form of mood affiliation, or what.)

There’s a variation of this argument which goes: The universe is finite, so there’s a finite number of breakthroughs to make, so they have to run out eventually. But even granting this, why assume we have found even 1% of the big breakthroughs so far? Or 0.01%? If there are many more to be had, then progress can continue for a long time.

As for materialism, I disagree with the premise. I don’t think progress is primarily driven by material resources. When we think of the Industrial Revolution, we often think of steam engines, iron foundries, and locomotives, all run on coal. But there were equ...
Notes on good judgement and how to develop it (80,000 Hours)
by habryka — 2020-09-12T17:51:27.174Z
Link: https://80000hours.org/2020/09/good-judgement/

This post by 80,000 hours struck me as more than usually relevant to my interests in developing the art of rationality. It doesn't really say anything new, but it does provide a decent summary of a frame that I think is an important subset of epistemic rationality, in the form of "good judgement".

More practically, I think of someone with good judgement as someone able to:

Focus on the right questions
When answering those questions, synthesise many forms of weak evidence using good heuristics, and weigh the evidence appropriately
Be resistant to common cognitive biases by having good habits of thinking
Come to well-calibrated conclusions

Owen Cotton-Barratt wrote out his understanding of good judgement, breaking it into ‘understanding’ and ‘heuristics’. His notion is a bit broader than mine.

Here are some closely related concepts:

Keith Stanovich’s work on ‘rationality’, which seems to be something like someone’s ability to avoid cognitive biases, and is ~0.7 correlated with intelligence (so, closely related but not exactly the same)
The cluster of traits (listed later) that make someone a good ‘superforecaster’ in Philip Tetlock’s work (Tetlock also claims that intelligence is only modestly correlated with being a superforecaster)

Here are some other concepts in the area, but that seem more different:

Intelligence: I think of this as more like ‘processing speed’ – your ability to make connections, have insights, and solve well-defined problems. Intelligence is an aid in good judgement – since it lets you make more connections – but the two seem to come apart. We all know people who are incredibly bright but seem to often make dumb decisions. This could be because they’re overconfident or biased, despite being smart.
Strategic thinking: Good strategic thinking involves being able to identify top priorities, and develop a good plan for working towards those priorities, and improving the plan over time. Good judgement is a great aid to strategy, but a good strategy can also make judgement less necessary (e.g. by creating a good back-up plan, you can minimise the risks of your judgement being wrong).
Expertise: Knowledge of the topic is useful all else equal, but Tetlock’s work (covered more below) shows that many experts don’t have particularly accurate judgement.
Decision making: Good decision making depends on all of the above: strategy, intelligence, and judgement.

...
The universality of computation and mind design space
by alanf — 2020-09-12T14:58:46.759Z

A Turing machine is a universal computer: it can compute anything that any other computer can compute. A human being can specify a Turing machine and the data it's acting on and carry out the steps that the machine would execute. Human beings have also constructed computers with the same repertoire as a Turing machine, such as the computer on which I am writing this question. There are articles on Less Wrong about mind design space, such as this one:

https://www.lesswrong.com/posts/tnWRXkcDi5Tw9rzXw/the-design-space-of-minds-in-general

in which the author writes:

The main reason you could find yourself thinking that you know what a fully generic mind will (won't) do, is if you put yourself in that mind's shoes - imagine what you would do in that mind's place - and get back a generally wrong, anthropomorphic answer.

But a person thinking about what an AI would do needn't imagine what he would do in that other mind's place. He can simulate that mind with a universal computer.

So what is the Less Wrong position on whether we could understand AIs and how is that claim compatible with the universality of computation?
How Much Computational Power Does It Take to Match the Human Brain?
by habryka — 2020-09-12T06:38:29.693Z
Link: https://www.openphilanthropy.org/brain-computation-report

Joe Carlsmith with a really detailed report on computational upper bounds and lower bounds on simulating a human brain:

Open Philanthropy is interested in when AI systems will be able to perform various tasks that humans can perform (“AI timelines”). To inform our thinking, I investigated what evidence the human brain provides about the computational power sufficient to match its capabilities. This is the full report on what I learned. A medium-depth summary is available here. The executive summary below gives a shorter overview.

[...]

Let’s grant that in principle, sufficiently powerful computers can perform any cognitive task that the human brain can. How powerful is sufficiently powerful? I investigated what we can learn from the brain about this. I consulted with more than 30 experts, and considered four methods of generating estimates, focusing on floating point operations per second (FLOP/s) as a metric of computational power.

These methods were:

Estimate the FLOP/s required to model the brain’s mechanisms at a level of detail adequate to replicate task-performance (the “mechanistic method”).1
Identify a portion of the brain whose function we can already approximate with artificial systems, and then scale up to a FLOP/s estimate for the whole brain (the “functional method”).
Use the brain’s energy budget, together with physical limits set by Landauer’s principle, to upper-bound required FLOP/s (the “limit method”).
Use the communication bandwidth in the brain as evidence about its computational capacity (the “communication method”). I discuss this method only briefly.

None of these methods are direct guides to the minimum possible FLOP/s budget, as the most efficient ways of performing tasks need not resemble the brain’s ways, or those of current artificial systems. But if sound, these methods would provide evidence that certain budgets are, at least, big enough (if you had the right software, which may be very hard to create – see discussion in section 1.3).2

Here are some of the numbers these methods produce, plotted alongside the FLOP/s capacity of some current computers.

Figure 1: The report’s main estimates. See the conclusion for a list that describes them in more detail, and summarizes my evaluation of each.

These numbers should be held lightly. They are back-of-the-envelope calculations, offered alongside initial discussion of complications and objec

...
Rationality and playfulness
by DirectedEvolution — 2020-09-12T05:14:29.624Z

Can rationality help us be playful? Can we be playful when we're solitary?

Play is usually interactive. It's about connecting with other people, or even with a pet animal. When people do "playful" things by themselves, it's usually for relaxation or practice.

The presence of a second person changes everything. You can react to each other, surprise or influence each other, create structure together. Group decisions are easier to commit to after a choice is made.

Many activities can be fun, engaging, and interesting, without being obviously "playful." A chess game can involve more mental concentration and stillness than is required of most people at their jobs, yet be a delightful hobby activity for the participants. We even say we "played" a game of chess. So why doesn't chess feel playful?

Partly, it's because play is usually physical. Even if we're just having a playful conversation, our body language and voices can bring a physical element to the exchange. Chess, like writing, reading, and many other fun-but-not-playful activities doesn't typically use our big muscles or our social muscles.

What about exercise? That's not conventionally playful either, even though it uses our muscles. Even athletics, like a game of tennis, can feel fun-but-not-playful, unless the participants are joking around and being social while they play.

It really does seem to be the social element that's key for a sense of play. If we watch a talk show, the participants often have very playful interactions, even though they're mostly just sitting in chairs talking.

Even in a social setting where both participants desire a playful conversation, though, it's often very difficult to achieve. It's so easy for even good friends to feel awkward, formal, and serious in each others' company. Coming up with a playful text message takes work for many people. Especially at first. If a conversation chain gets going that has taken on a playful tone, it might stay that way. Positive energy, a combination of kindness and rudeness, and not taking things literally all can be fertile ground for a playful conversation.

If you're leading a solitary life, though, is it possible to be playful? What about in these lonely times?

Can you think playful thoughts? After all, our inner world can often feel like we have multiple perspectives, multiple voices within us. Is it possible for them to have a playful interactio...