trentmkelly/LessWrong-43k huggingface dataset:
Intertheoretic utility comparison: examples
by Stuart_Armstrong — 2019-07-17T12:39:45.147Z

A previous post introduced the theory of intertheoretic utility comparison. This post will give examples of how to do that comparison, by normalising individual utility functions.

The methods

All methods presented here obey the axioms of Relevant data, Continuity, Individual normalisation, and Symmetry. Later, we'll see which ones follow Utility reflection, Cloning indifference, Weak irrelevance, and Strong irrelevance.

Max, min, mean

The maximum of a utility function u is maxs∈Su(s), while the minimum is mins∈Su(s). The mean of u ∑s∈Su(s)/||S||.

The max-min normalisation of [u] is the u∈[u] such that the maximum of u is 1 and the minimum is 0.

The max-mean normalisation of [u] is the u∈[u] such that the maximum of u is 1 and the mean is 0.

The max-mean normalisation has an interesting feature: it's precisely the amount of utility that an agent completely ignorant of its own utility, would pay to discover that utility (as a otherwise the agent would employ a random, 'mean', strategy).

For completeness, there is also:

The mean-min normalisation of [u] is the u∈[u] such that the mean of u is 1 and the minimum is 0.

Controlling the spread

The last two methods find ways of controlling the spread of possible utilities. For any utility u, define the mean difference: ∑s,s′∈S|u(s)−u(s′)|. And define the variance: ∑s∈S(u(s)−μ)2, where μ is the mean defined previously.

These lead naturally to:

The mean difference normalisation of [u] is the u∈[u] such that u has a mean difference of 1.

The variance normalisation of [u] is the u∈[u] such that u has a variance of 1.

Properties

The different normalisation methods obey the following axioms:

Property
Max-min
Max-mean
Mean-min
Mean difference
Variance

Utility reflection
YES
NO
NO
YES
YES

Cloning indifference
YES
NO
NO
NO
NO

Weak Irrelevance
YES
YES
YES
NO
YES

Strong Irrelevance
YES
YES
YES
NO
NO

As can be seen, max-min normalisation, despite its crudeness, is the only one that obeys all the properties. If we have a measure on S, then ignoring the cloning axiom becomes more reasonable. Strong irrelevance can in fact be seen as an anti-variance; it's because of its second order aspect that it fails this.
RAISE AI Safety prerequisites map entirely in one post
by RAISE — 2019-07-17T08:57:18.226Z

After reflecting on recent criticism, all RAISE projects have been discontinued, and a postmortem is in our plans. (Update: the postmortem is here) One of those projects was the AI Safety prerequisites online course originally announced here. We're sharing the curriculum here in almost plaintext format so that people can easily find and access it. There is also a GDoc here. (Note: two other products of RAISE are series of lessons on IRL and IDA. They don't have such neatly formed curriculums, and they are still accessible as online lessons.)

It was supposed to be a guide for helping people who want to get into AI safety research. It contains only foundations of math topics (Logic and proof, ZF(C) Set theory, Computability theory to be precise), which are more useful for agent foundations stuff and not useful for machine learning stuff. It was planned to be extended to cover more topics, but that never happened.

How to use this

The main path contains 20 levels. It is the recommended path through the curriculum. Its levels are actually short sequences of levels from the three other paths.

To see what textbooks are required to study a path, see its beginning. Computability theory and set theory paths require two paid textbooks.

13 out of 20 levels of the main path are covered by our online course, which is free (but still requires paid textbooks). To use the online course, register here. You might prefer to use it instead of the text below because it provides more sense of progress, contains solutions to the exercises, has some writing mistakes fixed, maybe feels like a less tedious thing, and provides some additional exercises which we don't think are important.

Credits

The curriculum was made by Erik Istre and Trent Fowler. People who created the online course are: Philip Blagoveschensky, Davide Zagami, Toon Alfrink, Hoagy Cunningham, Danilo Naiff, Lewis Hammond. Also these people contributed: Jacob Spence, Roland Pihlakas. Also, thanks to Grasple for providing their services for free.

Main path

Level 1. Basic logic

The big ideas:

Sentential Logic

Truth Tables

Predicate Logic

Methods of Mathematical Proof

To move to the next level you need to be able to:

Translate informal arguments into formal logic.

Evaluate an argument as either valid or invalid.

Explain how to prove an implication/conditional, a conjunction, a disjunction, and a negation and know what thi

...
Doublecrux is for Building Products
by Raemon — 2019-07-17T06:50:26.409Z

Previously:

What product are you building?

2 years ago, CFAR's doublecrux technique seemed "probably good" to me, but I hadn't really stress tested it. And it was particularly hard to learn in isolation without a "real" disagreement to work on.

Meanwhile, some people seemed skeptical about it, and I wasn't sure what to say to them other than "I dunno man this just seems obviously good? Of *course* you want to treat disagreements like an opportunity to find truth together, share models, and look for empirical tests you can run?"

But for people who didn't share that "of course", that wasn't very helpful.

For the past two years I've worked on a team where big disagreements come up pretty frequently, and where doublecrux has been more demonstrably helpful. I have a clearer sense of where and when the technique is important.

Intractable Disagreements

Some intractable disagreements are fine

If you disagree with someone on the internet, or a random coworker or something, often the disagreement doesn't matter. You and your colleague will go about their lives, one way or another. If you and your friends are fighting over "Who would win, Batman or Superman?", coming to a clear resolution just isn't the point.

It might also be that you and your colleague are doing some sort of coalition-politics fight over the overton window, and most of the debate might be for the purpose of influencing the public. Or, you might be arguing about the Blue Tribe vs Red tribe as a way of signaling group affiliation, and earnestly understanding people isn't the point.

This makes me sad, but I think it's understandable and sometimes it's even actually important.

Such conversations are don't need to be doublecrux shaped, unless both participants want them to be.

Some disagreements are not fine

When you're building a product together, it actually matters that you figure out how to resolve intractable disagreements.

I mean "product" here pretty broadly – anything that somebody is actually going to use. It could be a literal app or widget, or an event, or a set of community norms, or a philosophical idea. You might literally sell it or just use it yourself. But I think there is something helpful about the "what if we were coworkers, how would we resolve this?" frame.

The important thing is "there is a group of people collaborating on it" and "there is a stakeholder who cares about it getting built."

If you're...
Robust Artificial Intelligence and Robust Human Organizations
by Gordon Seidoh Worley — 2019-07-17T02:27:38.721Z
Link: https://arxiv.org/abs/1811.10840v1

This academic note was previously linked but undiscussed in AN#35, and I finally got around to reading it and really liked it so wanted to highlight it with a post.

The abstract:

Every AI system is deployed by a human organization. In high risk applications, the combined human plus AI system must function as a high-reliability organization in order to avoid catastrophic errors. This short note reviews the properties of high-reliability organizations and draws implications for the development of AI technology and the safe application of that technology.

The gist is that AI, including narrow AI, can be a dangerously powerful technology similar to nuclear power and weapons, and the organizations that best control these systems have certain features that categorize them as High Reliability Organizations (HROs). The features of an HRO are (quoted from Wikipedia, quoted from this book by the originators of the HRO concept):

Preoccupation with failure: HROs treat anomalies as symptoms of a problem with the system. The latent organizational weaknesses that contribute to small errors can also contribute to larger problems, so errors are reported promptly so problems can be found and fixed.

Reluctance to simplify interpretations: HROs take deliberate steps to comprehensively understand the work environment as well as a specific situation. They are cognizant that the operating environment is very complex, so they look across system boundaries to determine the path of problems (where they started, where they may end up) and value a diversity of experience and opinions.

Sensitivity to operations: HROs are continuously sensitive to unexpected changed conditions. They monitor the systems’ safety and security barriers and controls to ensure they remain in place and operate as intended. Situational awareness is extremely important to HROs.

Commitment to resilience: HROs develop the capability to detect, contain, and recover from errors. Errors will happen, but HROs are not paralyzed by them.

Deference to expertise: HROs follow typical communication hierarchy during routine operations, but defer to the person with the expertise to solve the problem during upset conditions. During a crisis, decisions are made at the front line and authority migrates to the person who can solve the problem, regardless of their hierarchical rank.

The note argues for finding ways to incorporate AI in an H...
What is your Personal Knowledge Management system?
by sayan — 2019-07-16T20:20:05.558Z

Personal knowledge management (PKM) is a collection of processes that a person uses to gather, classify, store, search, retrieve and share knowledge in their daily activities (Grundspenkis 2007) and the way in which these processes support work activities (Wright 2005). It is a response to the idea that knowledge workers need to be responsible for their own growth and learning (Smedley 2009). It is a bottom-up approach to knowledge management (KM) (Pollard 2008).

Wikipedia

I am curious about how LessWrongers manage personal information, including but not limited to - research reading/ research output, side-projects/ hobbies, blogging, managing watchlist/ to-read list, social information, incremental-reading, making flashcards, threads-from-unfinished-conversations, braindumps, and maybe even general GTD-stuff.

Information about any of the following will be greatly appreciated.

General framework/ philosophy for PKM.
Can be very concrete and well-defined like Building a Second Brain or a very a general rule-of-thumb like 'I bookmark everything/ I write down all thoughts in org-mode.'

Toolchain.
What and How. Everything counts - wikis, org-mode, physical notebook, etc.

Workflow.
Both local (how you manage fuzzy and perpetually arriving information on a day-to-day basis) and global (how you PKM for bigger and concrete projects over longer time).

Comparison of multiple frameworks, if used.

Tips and tricks/ info about building PKM habits.
Wolf's Dice
by johnswentworth — 2019-07-16T19:50:03.106Z

Around the mid-19th century, Swiss astronomer

Rudolf Wolf

rolled a pair of dice 20000 times, recording each outcome. Here is his data:

I’ve taken the data from

Jaynes

, who uses it as an example for maximum-entropy methods. We’ll use it for several examples, including some of Jaynes’ models.

The most interesting fact about Wolf’s dice data is that some faces come up significantly more often than others. As a quick back-of-the-envelope check, we can see this by calculating the expected number of times a face should come up on one die (20000∗16≈3333.3), and the standard deviation of this number (√20000∗(16)∗(1−16)≈52.7). A quick glance at the data shows that the column- and row-totals differ from their expected value by roughly 2 to 6 standard deviations, an error much larger than we’d expect based on random noise.

That, however, is an ad-hoc test. The point of this sequence is to test our hypotheses from first principles, specifically the principles of probability theory. If we want to know which of two or more models is correct, then we calculate the probability of each model given the data.

White Die: Biased or Not?

Let’s start with just Wolf’s white die. Our two models are:

Model 1: all faces equally probable
Model 2: each face i has its own probability pi (we’ll use a uniform prior on pi for now, to keep things simple)

Our data is the bottom row in the table above, and our goal is to calculate P[modeli|data] for each model.

The first step is Bayes’ rule:

P[modeli|data]=1ZP[data|modeli]P[modeli]

Here’s what those pieces each mean:

P[modeli] is our prior probability for each model - for simplicity, let’s say the two are equally likely a priori.
Z is the normalizer, chosen so that the posterior probabilities sum to one: P[model1|data]+P[model2|data]=1 (implicitly, we assume one of the two models is “correct”).
P[data|modeli] is computed using modeli

… so the actual work is in calculating P[data|modeli].

Model 1: Unbiased

For the first model, P[data|model1] is a standard probability problem: given n=20000 unbiased die rolls, what’s the probability of n1=3246 1’s, n2=3449 2’s, etc? This is a

multinomial distribution

; the answer is

P[n1…n6|model1]=n!n1!…n6!pn11…pn66=20000!3246!...3932!(16)20000≈6.3∗10−70

Note the symmetry factor with the factorials: we're computing the probability of the observed counts, not the probability of a particular string of outcomes, so we h...
Very Short Introduction to Bayesian Model Comparison 
by johnswentworth — 2019-07-16T19:48:40.400Z

At least within Bayesian probability, there is a single unique unambiguously-correct answer to "how should we penalize for model complexity?": calculate the probability of each model, given the data. This is Hard to compute in general, which is why there's a whole slew of of other numbers which approximate it in various ways.

Here's how it works. Want to know whether model 1 or model 2 is more consistent with the data? Then compute P[model1|data] and P[model2|data]. Using Bayes' rule:

P[modeli|data]=1ZP[data|modeli]P[modeli]

where Z is the normalizer. If we're just comparing two models, then we can get rid of that annoying Z by computing odds for the two models:

P[model1|data]P[model2|data]=P[data|model1]P[data|model2]P[model1]P[model2]

In English: posterior relative odds of the two models is equal to prior odds times the ratio of likelihoods. That likelihood ratio P[data|model1]P[data|model2] is the Bayes factor: it directly describes the update in the relative odds of the two models, due to the data. Calculating the Bayes factor - i.e. P[data|modeli] for each model - is the main challenge of Bayesian model comparison.

Example

20 coin flips yield 16 heads and 4 tails. Is the coin biased?

Here we have two models:

Model 1: coin unbiased
Model 2: coin has some unknown probability θ of coming up heads (we'll use a uniform prior on θ for simplicity)

The second model has one free parameter (the bias) which we can use to fit the data, but it’s more complex and prone to over-fitting. When we integrate over that free parameter, it will fit the data poorly over most of the parameter space - thus the "penalty" associated with free parameters in general.

In this example, the integral is exactly tractable (it's a dirichlet-multinomial model), and we get:

P[data|model1]=(204)(12)16(12)4=0.0046
P[data|model2]=∫θ(204)(θ)16(1−θ)4dθ=0.048

So the Bayes factor is (.048)/(.0046) ~ 10, in favor of a biased coin. In practice, I'd say unbiased coins are at least 10x more likely than biased coins in day-to-day life a priori, so we might still think the coin is unbiased. But if we were genuinely unsure to start with, then this would be pretty decent evidence in favor.
Economic Thinking
by limerott — 2019-07-16T17:33:54.648Z
Link: https://limerott.com/economic-thinking/

Economics is often portrayed as a dismal science. It seems to be about a small group of experts talking mostly about monetary policy. I want to argue that this is not true and that there is an elegant, natural and simple interpretation that admits far-reaching applications.

Finite thinking

First, we introduce finite thinking. Finite thinking is a kind of thinking that takes into account that resources are finite, or more generally, that entities in the real world have to obey the constraints of the real world.

When you are thinking about how to spend the $10 in your wallet, you carefully weigh different options against each other. You are performing finite thinking. But if you know that you are carrying around plenty of money, you no longer think of it as a scarce resource and, thus, no longer think finitely.

This applies to time management as well. If you have plenty of time, you cease to think of time as finite and tend to slack off. But if there are only two hours left until the deadline, you start to think finitely and try to use the remaining time as wisely as possible.

A more subtle example is this: When a thief robs a bank and escapes, the police could throw their hands in the air and say that he must have already left the country. Or, they acknowledge that the speed of his vehicle is not infinite and that he must be somewhere within a certain radius around the bank.

Economical thinking is nothing other than finite thinking: Anyone and anything that interacts with reality is subject to certain constraints. To think economically means to factor in these constraints.

Infinite thinking

It is instructive to look at its counterpart, which is infinite thinking. Here, it is implicitly assumed that a resource is available in an infinite amount. At first, this seems abstruse and not how people think at all because it clearly defies the basic truths of reality. But in fact, this mode of thinking is commonplace and pervades all aspects of life. It can also be very useful.

When buying groceries, most people do not take a close look at the price nor do they sum up the total amount in their heads. They pick the products they planned to buy and then pay the cashier the amount that is asked of them. Even though they are not aware of it, they implicitly assume that they have an infinite amount of money. However, if the resulting total is exorbitantly high, finite thinking

...
Nutrition is Satisficing
by aaq — 2019-07-16T16:13:52.741Z

Reason for writing: I didn't see a clear, unambiguous explanation of this concept when I did a search on LW.

Epistemic status: Confident, but this is a normative claim, not a descriptive one. Skip if you hate normies.

Eat plenty of vegetables. Drink plenty of water. Get enough protein.

For the average person in a developed nation, mixing these 3 heuristics into their current nutritional profile is likely to improve their health. I don't know if anyone has actually penned a distance function p:Nutrition→R that lets us rank how good or bad someone's nutrition *is*, but I'm pretty sure that if you aren't following these 3 heuristics, and then you start, p(you2) will be greater than p(you1), even if you're inconsistent.

Past general heuristics, we run into a problem when we try to compare specific foods to find the best one in a given class. For example: is rotisserie chicken or canned tuna better for your meal plan? That's a question of moderate difficulty to answer; we would need information such as

The nutritional profiles of the chicken and the tuna,
Whether you prefer the taste of one or the other, and
How much you're willing to spend and the relative prices of both in your area. (Fun fact: A 1200 calorie $7 rotisserie chicken from a supermarket is actually less expensive, calorie-per-calorie, than a 120 calorie $1 can of tuna. Otherwise it would be >$10.)

But there are dozens of common kinds of common protein sources out there, and they all can all be prepared in so many different ways. When you factor in meal prep and individual variation and all the different kinds of recipes you can make, finding the "best" food for any given metric honestly starts to look intimidating.

The answer? Don't look for the best; look for good enough. Nutrition is satisficing. Take some time to learn some general heuristics about nutrition, don't be afraid to explore randomly a bit at the supermarket, but also don't stress out about your own nutrition once you've got a few solid principles in place.
Intellectual Dark Matter
by Samo Burja — 2019-07-16T15:20:00.648Z

Knowledge that we can show exists, but cannot directly access, rests at the foundations of society and technology.

Messier 106

From my post published on the Long Now Foundation's blog. This is an excerpt from the draft of

my upcoming book

on great founder theory.

Missing mass, missing knowledge

Many galaxies would fly apart if they had as much mass as estimates based on their visible signature suggest. Although some have posited alternative theories of gravitation to explain this discrepancy, most physicists now hypothesize the existence of mass-bearing particles that are not detectable through emitted radiation such as visible light. We call these particles dark matter, and it is estimated to compose about 85% of all matter in the observable universe.

In analyzing the functional institutions of our society, we are not able to see for ourselves most of the knowledge that created them. Knowledge of this sort includes trade secrets, tacit technical knowledge, private social networks, private intelligence-gathering operations, management and persuasive skill, cooperation and collusion among founders and their allies, and founders’ long-term plans for their institutions.¹

This knowledge has profound effects on the social landscape. We must understand it if we hope to understand society. We therefore must examine intellectual dark matter: knowledge we cannot see publicly, but whose existence we can infer because our institutions would fly apart if the knowledge we see were all there was.³ Such intellectual dark matter rests at the foundations of our society, dwarfing in scope and importance the accessible, shareable, visible knowledge on which we normally focus.

There are many forms of intellectual dark matter, but the three principal ones are lost, proprietary, and tacit knowledge.

Read the rest here.

Read more from Samo Burja

here.