trentmkelly/LessWrong-43k huggingface dataset:
Fundamental Uncertainty: Chapter 2 - Why do words have meaning?
by Gordon Seidoh Worley — 2022-04-18T20:54:24.539Z

N.B. This is a chapter in a book about truth and knowledge. It is the first draft. I have since revised it and replaced this old version with a new major revision. You should read that one instead! You can also find the most up-to-date info/version on the book's website.

Words have meanings right? When I say the word "cup" you know I'm talking about a cup: a thing to drink out of. But why does the word "cup"—quoted so I can refer to the word apart from its meaning—mean cup—unquoted so I can point directly to the thing referenced by the word? It might seem like a silly question, but in fact it's the gateway to understanding fundamental uncertainty.

Think back to when you were a kid. I'll do the same. If someone had asked me why "cup" meant cup, I would have said something like "because that's what it is". The question wouldn't have registered as interesting, just a boring statement of fact turned into a strange question. Maybe I would have thought someone was playing a joke on me, asking a silly nonsense question. And it would have seemed a totally reasonable response at the time, since I'd never encountered a situation where "cup" meant anything other than cup.

But as I got older I started to realize there was a little space between the word "cup" and actual cups. For example, I have memories of getting into arguments with adults if they handed me a mug and called it a "cup". "No way", I'd say, "this is a mug, not a cup!". "Same difference", the response would come, but that seemed wrong to me. "Mugs" were mugs and "cups" were cups. Yet over time I began to see that "cup" was perhaps not as specific a word as I thought it was. In a certain sense, a mug is a type of cup, and it's totally reasonable that if you ask for a "cup" someone might give you a mug because it serves what they believe to be the purpose of asking for a "cup"—to be able to drink a liquid! So I learned there was more to words than just naming things; the form and function mattered.

At the same time I was learning this, I also learned to play games making up fake names for things. My friends and I could develop a secret language where "bloob" meant "cup" and "blig" meant "water" and "blarg" meant "drink" and we'd collapse into giggles making up silly sentences like "I'm blarging a bloob of blig". So it wasn't just that words had meanings and sometimes there was more to those meanings than direct...
Ensembling the greedy doctor problem
by Ryan Kidd — 2022-04-18T19:16:00.916Z

The greedy doctor problem is defined as follows: suppose you want a possible illness to be diagnosed correctly, but your doctor is incentivized to diagnose you sick because they can then charge you more for treatment. How do you get an accurate diagnosis and avoid paying more than you need?

A naive solution is to get a second opinion. However, a second doctor is also incentivized to diagnose you ill! What if you leverage adversarial dynamics and pay one doctor for treatment only if both doctors agree you are ill and the other only if both doctors agree you are hale? Unfortunately, this instantiates a form of the "battle of the sexes" game and the doctors randomise their diagnoses according to a mixed-strategy Nash equilibrium. Neither of these approaches reliably gets you diagnosed correctly.

Assuming the doctors cannot coordinate through signalling, I think ensembling might solve the problem in the absence of unforseen acausal coordination. My (totally impractical) strategy is as follows:

I assemble a host of patients (ideally independent and identically distributed in symptoms) and submit them to both doctors for diagnosis.

Each doctor is told that they will only be paid if they diagnose all patients identically to the other doctor, with the exception that neither will be paid if they diagnose all patients ill or hale (this assumes that at least one patient is not ill).

If the doctors each try to diagnose only a few patients as hale, it is unlikely that they will choose the same subset of patients. The doctors are thus incentivized to coordinate on a Schelling point, ideally "telling the truth".

There are three obvious Schelling points: diagnosing all patients ill; diagnosing all patients hale; and diagnosing all patients truthfully (assuming that the doctors agree). Likely, at least one patient will be genuinely ill and at least one will be genuinely hale. The doctors can get more money if they diagnose some patients as ill, which biases them away from the all-hale Schelling point. The doctors will not get paid if they diagnose all patients as ill, which biases them away from the all-ill Schelling point. Thus, the only remaining obvious Schelling point is all-truthful.

It is possible that the doctors could acausally coordinate on an alternative Schelling point. For example, there might be mutually exploitable patterns in the data or precommited doctor stra...
Is "Control" of a Superintelligence Possible?
by Mahdi Complex — 2022-04-18T16:03:49.126Z

Epistemic status: confused.

I currently see two conceptualization of an aligned post-AGI/foom world:

1. Surrender to the Will of the Superintelligence

Any ‘control’ in a world with a Superintelligence will have to be illusory. The creation of an AGI will be the last truly agentic thing humans do. A Superintelligence would be so far superior to any human or group of humans, and able to manipulate humans so well, that any “choice” humanity faces will be predetermined. If an AI understands you better than yourself, can predict you better than yourself, and understands the world and human psychology well enough that it can bring about whatever outcome it wants, then any sense of ‘control’ – any description of the universe putting humans in the driver’s seat – will be false.

This doesn’t mean that alignment is necessarily impossible. The humans creating the Superintelligence could still instil it with a goal of leaving the subjective human experience of free-will intact. An aligned Superintelligence would still put humans into situations where the brain’s algorithm of deliberate decision making is needed, even if the choice itself as well as the outcome are ‘fake’ in some sense. The human experience of control should rank high in an aligned Superintelligence's utility function. But only a faint illusory glimmer of human choice would remain, while the open-ended, agentic power over the Universe would have left humanity with the creation of the first Superintelligence. That’s why it’s so crucial to get the value-loading problem right on the first try.

2. Riding the Techno-Leviathan

The alternative view seems to be something like: it will be possible to retain human agency over a post-AGI world thanks to boxing, interpretability, oracle AI, or some other selective impairment scheme that would leave human agency unspoiled by AGI meddling. The goal appears to be to either limit an AI’s optimization power enough, or insulate the algorithms of human decision making (brain, institutions of group decision making...) well enough, such that humanity remains sovereign, or “in the loop,” in a meaningful sense. This, while also having the AI’s goals point towards the outcomes of human decision making.

This view implies that the integrity of human decision making can be maintained even in the face of an AGI’s optimization power.

I currently associate 1. more with MIRI/Superintellig...
Are deference games a thing?
by Daniel Kokotajlo — 2022-04-18T08:57:47.742Z

Everyone starts off with a voting card. People can give away their cards. At the end, people cast votes using the cards they have, so people with lots of cards get lots of votes. (If you don't like what your card is being used for, you always have the right to take it back.)

Maybe this could be used to implement something more like direct democracy? Have lots of votes every day instead of just one every four years. Most people don't vote on most issues, instead they give their card to their wisest friend who gives it to their wisest friend who has hundreds of cards and spends some free time each day thinking about what to vote on. Other people (e.g. Oprah, Elon Musk) would have millions of cards and would have staffs of advisors telling them how to vote. They'd basically be like congresspeople.

Surely there's a literature on this idea. What terms should I search for?

Also: Is it a good idea? How would it be worse than the current system?
Don't be afraid of the thousand-year-old vampire
by Ulisse Mini — 2022-04-18T01:22:39.140Z

Background

I first started reading Jaynes Probability Theory around 2 years ago, I did not last long, I wasn't able to follow the derivations in Chapter 2, so I gave up, assuming it would only get harder from there what was the point in continuing?

This was a big mistake! For several reasons

Trying to follow every detail is dumb, you're there to learn not to pass a trivia test on solving the associativity equation. You're here to learn probability not how to solve functional equations

It doesn't get harder, it gets easier! You should always skim a textbook before reading it (and especially before giving up)

As I'm finally rectifying my mistake I figured I'd write an explanation for what Chapter 2 is actually about

Think "Alien" Not Robot

The reasoning in Chapter 2 shows that any rules of plausible-inference must match our own theory of probability after a suitable change of units.
The convoluted functional-equation argument is needed to construct a function p:R→[0,1] which translates the alien's plausibility A|B into our probability p(A|B).

This is very different to how p is usually defined, for Jaynes p is a translation between the plausibilities of the alien and our "nicer" probabilities that obey the sum and product rules.
Contrast this with standard probability where p is a function from subsets of the sample space to real numbers obeying certain axioms.

Any alien civilization's concept of "probability the dice lands five" must be a monotonic function of our probability 1/6.
That's amazing when you think about it, It shows that probability is discovered not invented. [1]

Furthermore, if you want the sum and product rule to take their natural forms you have to pick our units, this is analogous to how degrees Kelvin are defined to make the laws of thermodynamics look natural. [2]

Near the end of Chapter 2 (after we've shown uniqueness) Jaynes switches to a more traditional function P:Prop→[0,1] where Prop is some logical proposition like "the dice lands five".

Why Jaynes doesn't mention the intuitive explanation until after dragging you through a hard-to-follow argument involving tons of functional equations and Calculus I have no idea, I guess Thousand-year old vampires are bad at pedagogy? [3]

Some study tips

Hopefully I've convinced you Jaynes isn't that scary, and motivated you to start reading. A few quick tips

Check out these lectures for higher level (

...
Code Generation as an AI risk setting
by Not Relevant — 2022-04-17T22:27:38.097Z

Historically, it has been difficult to persuade people of the likelihood of AI risk because the examples tend to sound “far-fetched” to audiences not bought in on the premise. One particular problem with many traditional framings for AI takeover is that most people struggle to imagine how e.g. “a robot programmed to bake maximum pies” figures out how to code, locates its own source-code, copies itself elsewhere via an internet connection and then ends the world.

There’s a major logical leap there: “pie-baking” and “coding” are things done by different categories of agent in our society, and so it’s fundamentally odd for people to imagine an agent capable of both. This oddness makes it feel like we must be far away from any system that could be that general, and thus pushes safety concerns to a philosophical exercise.

I want to make the case that the motivating example we should really be using is automatic code generation. Here’s a long list of reasons why:

It’s obvious to people why and how a system good at generating code could generate code to copy itself, if it were given an open-ended task. It’s a basic system-reliability precaution that human engineers would also take.

For non-experts, they are already afraid of unrestrained hackers and of large tech companies building software products that damage society - this being done by an unaccountable AI fits into an emotional narrative.

For software people (whom we most need to convince) the problem of unexpected behaviors from code is extremely intuitive - as is the fact that it is always the case that code bases are too complex for any human to be certain of what they’ll do before they’re run.

Code generation does seem to be getting dramatically better, and the memetic/media environment is ripe for people to decide how to feel about these capabilities.

Nearly all conceivable scalable prosaic alignment solutions will require some degree of “program verification” - making sure that code isn’t being run with an accidentally terrible utility function, or to verify the outputs of other AIs via code-checking Tool AIs. So we want substantial overlap between the AI safety and AI codegen communities.

The “alignment problem” already exists in nearly all large software engineering projects: it’s very difficult to specify what you want a program to do ahead of time, and so we mostly just run codebases and see what happe

...
Stupid Reasons to Have More Kids
by SebastianG  — 2022-04-17T20:05:03.931Z

A list with counter-commentary:

“In the past kids assisted in farm work, and child labor was legal; the incentive then was to have many kids. But today kids take forever to be economically productive and are much more expensive. Thus many people have fewer kids. Circumvent the incentive structure by training your kids useful at younger ages.” -Various authors
I don't think this is a factual representation of the dynamics. In the past people had lots of kids for two reasons. One, sex caused children. Two, any group which failed to have enough children survive into adulthood would die out quickly and visibly. Making children useful at younger ages might give us some semi-useful 8 year-olds, who are helpful with the little ones. However, in a larger family one parent is still going to be a 4/5 time domestic engineer, even with help from the 8 year old and not contributing to GDP. I don't see how this pronatalism 3+ kids stuff works without pulling one parent out mostly out of the GDP equation. TANSTAAFL.

“Technological innovation is more likely when the next generation is bigger. As the gains from technology slows to a halt, so too will the stock market cease to grow. In a world that is not growing, many financial instruments will collapse.” - https://pronatalist.org/
AI is probably going to allow GDP to keep rising. Eventually, we won’t need people at all, and the stock market will be fine. Don’t worry about it.

"National greatness, and staying ahead of China for world influence requires that we have the biggest economy. To do that, we need more people." -Matt Yglesias, One Billion Americans.
Yeah, the guy who has chosen to have one child is going to inspire me to make the sacrifices involved in having four. It might be good for America, but the ‘ask’ here looks like it is that I sacrifice my utility for Matt’s one kid, and thus is not cooperate-cooperate. I’ll jump when you jump.

"You might not like the sacrifices of having children now, but look out over the course of your life. The cost of children is finite and the pleasure of grandchildren is, while not infinite, very high. Most people don't want to limit their number of grandchildren. On the margin, you should increase your chances of having more grandchildren by having more kids!" -Bryan Caplan, Selfish Reasons to Have More Kids
Bryan Caplan, at least, has three four kids. Looking ou

...
Org announcement: [AC]RC
by Vivek Hebbar — 2022-04-17T17:24:07.597Z

It seems that certain thermodynamic phenomena are highly relevant to alignment. From Paul Christiano's rebuttal to a recent seminal paper:

In fact, it appears that thousands of words may have been written on this topic, from which many lesser researchers have bounced off.

Seeing that Paul Christiano and John Swentworth have spent countless hours investigating this important topic, it's importance is quite evident. In order to free up their time for other strategies like ELK, I am launching a new EA-funded initiative:

Our initial staff consists of myself and 10 Mechanical Turk workers, who shall be tasked with data gathering. For seed funding, we are requesting a modest sum of $50M. Given that Paul's time is worth $20 trillion per hour, this seems like a reasonable tradeoff.

Our initial research agenda consists of investigating the space of designs with a complex number of hoses. Further, we have reason to believe that regularizing by the complexity of the stupidest argument for a given design leads to good inductive bias properties. Our work may even have direct implications for other alignment agendas, as it shares structure with many important problems.

Furthermore, building infrastructure for practical engineering projects is of great value, even if air conditioners aren't quite mechanistically identical to AGI.

You can support our work by loudly advertising your loyalty to the 1-hose or 2-hose camp.
Fuck Your Miracle Year
by rogersbacon — 2022-04-17T16:10:23.822Z
Link: https://rogersbacon.substack.com/p/fuck-your-miracle-year?s=w

Dwarkesh Patel’s essay “

The Mystery of the Miracle Year

” just went COVID-19-level viral (woah, too soon). Fucking Jeff Bezos read the article at tweeted at him (and

Marc Andreessen

and

Paul Graham

).

I’ve appeared on

Dwarkesh’s podcast

and he has given me feedback on some of my writing before. I like Dwarkesh and think he’s super smart and deserves every bit of attention and praise that is coming his way. But I do have a bone to pick, not with him or the essay per se, but with this kind of essay and the growing obsession with what I call “innovation porn”.

In the essay, Dwarkesh attempts to explain the

annus mirabilis

phenomenon, the extraordinary career-defining year of productivity found in the lives of various scientists, inventors, and creatives.

Einstein had his annus mirabilis in 1905. While he was still a patent clerk, he wrote four papers that revolutionized our understanding of the photoelectric effect, Brownian motion, and special relativity.

He briefly discusses a few possible explanations—the right problem at the right time (i.e. luck), youth and freedom (“Perhaps there’s a brief window in a person’s life where he has the intelligence, curiosity, and freedom of youth but also the skills and knowledge of age”) and the obligations that success brings (as a reason why these remarkable periods of creativity tend to fizzle out). The article ends with a forceful conclusion:

Given how many of the great scientific discoveries have come about during miracle years, we should do everything we can to help smart 20-somethings have an annus mirabilis. We should free them up from rote menial work, prevent them from being too exposed to the current paradigm, and give them the freedom to explore far fetched ideas without arbitrary deadlines or time consuming obligations. It’s a bit depressing that I have just described the opposite of the modern PhD program.

My first criticism, one that is often discussed in these kinds of debates, is the whole “

ideas are getting harder to find

” thing. As Dwarkesh points out, miracle years tend to happen in your 20s (Einstein was 26 during his). Newton, another miracle-year-haver (1665, age 22), famously remarked that if he has seen further it’s because he has stood on the shoulders of giants. Well, the giants are a lot taller nowadays. The most recent example of a scientific annus mirabilis that Dwarkesh discusse...
What is causality to an evidential decision theorist?
by paulfchristiano — 2022-04-17T16:00:01.535Z

(Subsumed by: Timeless Decision Theory, EDT=CDT)

People sometimes object to evidential decision theory by saying: “It seems like the distinction between correlation and causation is really important to making good decisions in practice. So how can a theory like EDT, with no role for causality, possibly be right?”

Long-time readers probably know my answer, but I want to articulate it in a little bit more detail. This is essentially identical to the treatment of causality in Eliezer Yudkowsky’s manuscript Timeless Decision Theory, but much shorter and probably less clear.

Causality and conditional independence

If a system is well-described by a causal diagram, then it satisfies a complex set of statistical relationships. For example:

In the causal graph A ⟶ B ⟶ C, the variables A and C are independent given B.
In the graph A ⟶ B ⟵C, the variables A and C are independent, but are dependent given B.

To an evidential decision theorist, these kinds of statistical relationships are the whole story about causality, or at least about its relevance to decisions. We could still ask why such relationships exist, but the answer wouldn’t matter to what we should do.

EDT = CDT

Now suppose that I’m making a decision X, trying to optimize Y.

And suppose further that there is a complicated causal diagram containing X and Y, such that my beliefs satisfy all of the statistical relationships implied by that causal diagram.

Note that this diagram will necessarily contain me and all of the computation that goes into my decision, and so it will be (much) too large for me to reason about explicitly.

Then I claim that an evidential decision theorist will endorse the recommendations of CDT (using that causal diagram):

EDT recommends maximizing the conditional expectation of Y, conditioned on all the inputs to X. Write Z for all of these inputs.

It might be challenging to condition on all of Z, given limits on our introspective ability, but we’d recommend doing it if possible. (At least for the rationalist’s interpretation EDT, which evaluates expected utility conditioned on a fact of the form “I decided X given inputs Z.”)
So if we can describe a heuristic that gives us the same answer as conditioning on all of Z, then an EDT agent will want to use it.
I’ll argue that CDT is such a heuristic.

In a causal diagram, there is an easy graphical condition (d-connectedness) to see

...