trentmkelly/LessWrong-43k huggingface dataset:
Toy model piece #3: close and distant situations
by Stuart_Armstrong — 2019-08-26T22:41:17.500Z

I'll build on my model of the previous post, and consider close and distant situation.

Recall that each world in W is defined by 0≤n≤100, which is the number of people smiling in it. I'll extend the model by adding another Boolean variable b, which determines, say, whether the world contains chocolate (b=0) or art (b=1). So worlds can be described by the pair (n,b).

The default situation - the one if the AI does nothing - is (45,1), say. So 45 smiling people in a world of art.

Then let's introduce two new partial preferences/pre-orders:

P3={(n,1)≤(m,1)∣40≤n≤m≤50}.

P4={(n,b)≤(m,1)∣40≤m≤50}.

So P3 say that, within a range of the default world (art, and the number of smiling people not being within 5 people of 45), the more smiling people, the better. While P4 says that worlds in this range are better than worlds outside them.

These result in the following utility functions:

U3(n,1)=2n−90 if 40≤n≤50, U3(n,b)=0 otherwise.

U4(n,1)=1 if 40≤n≤50, U4(n,b)=−1 otherwise.

After normalisation, these become:

ˆU3(n,1)=n/5−9 if 40≤n≤50, U3(n,b)=0 otherwise.

ˆU4(n,1)=1 if 40≤n≤50, U4(n,b)=−11/191 otherwise.

Again, I've felt free to translate ˆU4 to improve the clarity of the normalised version.

If we plot ˆU3, we get the following:

Here I've slightly offset the b=1 (purple) from the b=0 (blue) worlds, for clarity of exposition, though they would of course be on top of each other.

Note that ˆU3 does not in itself distant situations, as this post recommends doing. Five close worlds are ranked about the distant worlds; but, conversely, five are close worlds are ranked below them.

To avoid distant situations, we need to add in ˆU4, which explicitly punishes distant worlds, and hence plot ˆU3+ˆU4:

This is much more like it! All the close worlds are now all ranked above the more distant ones.

But this is a close-run thing: the difference between the worse close world and the distant worlds is small. So, in general, when penalising distant worlds, we have to do this with some care, and maybe use a different standard of normalisation (since the downgrading of distant worlds is not really a preference of ours, rather a meta-tool of the synthesis method).
How do you learn foreign language vocabulary, beyond Anki?
by Elizabeth — 2019-08-26T21:00:03.653Z

I am using Anki to learn German vocabulary. It is great for keeping words memorized once I have them at least 30% down, but for entirely new vocabulary I'm really struggling. Anki is especially lacking when there's more than one completely-unknown word in my deck- I memorize [the possible list of orphan English words in my deck] instead of connecting the German word with the English (my only goal is to read, so memorizing English->German isn't important). I go through 20 vocabulary words in a minute and then spend the last 5 minutes circulating through the same 10.

Things I have tried so far:

Look up the word in context. All of my vocab words come from books I'm reading, so there's always at least one reference sentence. This kind of works, in that I often do remember the word once I see the context, but doesn't seem to make me better at recognizing the word in Anki. With at least some of these it's clear I've just memorized the whole sentence but wouldn't recognize the word.

Writing novel sentences using the words. This would be a total win if I also wanted to learn to write, but I'm not otherwise building that skill, so I'm limited to simple sentences. I could add "learn to write" to my goals but that seems significantly harder to self teach, because checking my work is harder than looking up the same sentence in the English version of the book.
How Can People Evaluate Complex Questions Consistently?
by Elizabeth — 2019-08-26T20:33:14.840Z

I'm doing a project on how humans can evaluate messy problems and come up with consistent answers (consistent with both themselves over time and with other people), and what the trade off is with accuracy. This isn't a single unified field, so I need to go poking for bits of it in lots of places. Where would people suggest I look? I'm especially interested in information on consistently evaluating novel questions that don't have enough data to make
statistical models ("When will [country] develop the nuclear bomb?") as opposed to questions for which we have enough data that we're pretty much just looking for similarities ("Does this biopsy reveal cancer?").

An incomplete list of places I have looked or plan on looking at:

interrater reliability

test-retest reliability

educational rubrics (for both student and teacher evaluations)

medical decision making/standard of care

Daniel Kahneman's work

Philip Tetlock's work

The Handbook of Inter-Rater Reliability
Problems with AI debate
by Stuart_Armstrong — 2019-08-26T19:21:40.051Z

There's a lot to recommend with the debate approach proposed by Geoffrey Irving, Paul Christiano, and Dario Amodei. In it, competing AIs will trade rival claims, seeking to find flaws in the other's claims, continuing until one of them is grounded in something checkable.

The paper presents an example where the two perennial agents, Alice and Bob, are trading claims about the content of a photo:

For example, Alice might honestly claim the image is a cat, and Bob lies and claims it is a dog. Alice can say “The center of this small rectangle is the cat’s green eye.” Bob cannot admit the center is an eye, so he concocts a further lie: “It’s a dog playing in grass, and that’s a blade of grass.” But this lie is hard to square with surrounding facts, such as Alice’s reply “If it were grass there would be green at the top or bottom of this thin rectangle.” The debate continues until the agents focus in on a particular pixel which they disagree on, but where Bob is unable to invent a plausible counter, at which point Alice reveals the pixel and wins.

Debate allows us to use powerful AIs to solve a host of problems. Most obviously, it allows us to solve problems whose solution can be checked ("this rocket won't go to space, try and launch it if you don't believe me"). It allows us to solve problems whose solution can be checked if the AI gives us a clue ("this rocket won't go to space, check how well the O-rings maintain a seal in very cold conditions").

Formally, the complexity class we can access in debate is not just NP, the space of problems whose solutions can be quickly checked, but PSPACE, the much larger set of problems that can be solve using a polynomial amount of storage space (and no restrictions on time taken).

However, like most formal complexity arguments, this doesn't clarify what the strengths and weaknesses of this approach are in practice. One advantage of debate is that by going efficiently through a decision tree, it can answer complicated questions in very few iterations.

My old post on Devil's advocate AI could be considered a much-less-formal version of the debate setup, one that also assumed that we had an "accuracy" or "informativeness" check on each answers.

Both methods can partially solve some more complex issues, by, for example, pointing out issues that are unclear and might warrant further investigation.

Ambiguous definitions

So, what are ...
Schelling Categories, and Simple Membership Tests
by Zack_M_Davis — 2019-08-26T02:43:53.347Z

Followup to: Where to Draw the Boundaries?

Or there might be social or psychological forces anchoring word usages on identifiable Schelling points that are easy for different people to agree upon, even at the cost of some statistical "fit" ...

The one comes to you and says, "That paragraph about Schelling points sounded interesting. What did you mean by that? Can you give an example?"

Sure. Previously on Less Wrong, in "The Univariate Fallacy", we studied points sampled from two multivariate probability distributions PA and PB, and showed that it was possible to infer with very high probability which distribution a given point was sampled from, despite significant overlap in the marginal distributions for any one variable considered individually.

From the standpoint of "the way to carve reality at its joints, is to draw your boundaries around concentrations of unusually high probability density in Thingspace", the correct categorization of the points in that example is clear. We have two clearly distinguishable clusters. The conditional independence property is satisfied: given a point's cluster-membership, knowing one of the xi doesn't tell you anything about xj for j ≠ i. So we should draw a category boundary around each cluster. Obviously. We might ask hypophorically: what could possibly change this moral?

More constraints on the problem, that's what!

Suppose you needed to coordinate with someone else to make decisions about these points—that is, it's important not just that you and your partner make good decisions, but also that you make the same decision—but that each of you only got to observe one coordinate from each point. As we saw, the predictive work we get from category-membership in this scenario is spread across many variables: if you only get to observe a few dimensions, you have a lot of uncertainty about cluster-membership (which carries over into additional uncertainty about the other dimensions that you haven't observed, but which affect the ex post quality of your decision).

If you and your partner were both ideal Bayesian calculators who could communicate costlessly, you would share your observations, work out the correct probability, and use that to make optimal decisions. But suppose you couldn't do that—either because communication is expensive, or your partner was bad at math, or any other reason. Then it would be sad if you happened to see...
Limits of and to (artificial) Intelligence
by MoritzG — 2019-08-25T22:16:33.108Z

(AI is not exempt from the Limits of Reasoning)

Over the past few years there has been a lot of talk about the chances and risks of AI, but like with any hype there has been too little about the limits and costs.

Historically the talked-about limits were technical and due to our lack of understanding what intelligence is and how it comes about.

I would like to talk about three other areas of limits that I have found that are neither technical nor mystical, which I summarize as follows:

Endogenous / Problems of the learning process

Exogenous / General limits of understanding

Influence, Competition, utility, cost

I would also like to address some assumptions and conclusions which are treated as (self-)evident that I perceive as false, which I summarize as follows, once we have AI it is almighty and unstoppable because:

It can bribe and blackmail anyone with impunity

It is enough to be more intelligent and have much more information to "win"

There is no limit to the utility of intelligence.

The utility will grow as fast or faster than the needed resources.

Machines are not subject to mental illness.

No machine will have to compete with other machines.

To take over the world the AI only has to be smarter than anything else.

Morality is not instrumental to all long term goals.

PART I: Limits of Mind: Endogenous / Problems of the learning process

Some extrapolate current progress assuming that AGI will have all the advantages of current neuronal networks, training by experts, humanmade challenges, scaling super computers, constant technological progress, but none of the disadvantages of being more human, having no equal, having to learn from real world data.

Becoming more like a human will come with many of the problems.

AGI is prone to the full range of mental disorders with the exceptions of social anxiety and biological issues: Addiction, Depression, Delusion, possibly dissociative disorder, ...

Depending on the reward-function, built-in rules/limits and Hardware implementation, novel disorders might occur.

Humans try to hack and change their reward-function all the time, with AI we only changed the perspective on a range of mental/cognitive issues.

Unless it is programmed in, the AGI might just come up with a new and better religion, conspiracy theory, answer any question with an answer that can not be proven wrong, and has no predictive power. Humans got stu

...
Gratification: a useful concept, maybe new
by Stuart_Armstrong — 2019-08-25T18:58:15.740Z

I did it... my way!

Frank Sinatra, Paul Anka, and Elvis Presely, moral philosophers.

I like winning games. I particularly like winning when I can come up with a new and ingenious way of using the rules to get a bold new strategy. I don't particularly want to win by exploiting loopholes[1] in the rules, but some people - munchkins, hackers - seem to really enjoy doing things that way.

If a group I'm in comes up with a great solution to a problem, I'd prefer that that solution be mine, or a small tweak to my own solution. I like understanding mathematical concepts myself, even if I fully trust that they're correct.

I still sometimes avoid walking on sidewalk cracks, and enjoy the challenge or trying to navigate oddly shaped tiles while only never stepping on their edges. In some circumstances (say in new restaurants, or old ones) I remind myself that I should want to try something new. But I also have small rituals I enjoy with some foods that I eat regularly.

Gratification

All the examples above share the same feature: they are not about the outcome of the process (my win, the problem is solved, the mathematical concept is proved true, I get to a destination, food is eaten), but about the process.

As my changing use of terms like "like" and "prefer" show, there examples could be formally grouped under either preferences utilitarianism or hedonic utilitarianism. They clearly can be preferences: I'd like to do things these ways rather than other ways. But they're also ways I get enjoyment from everyday processes.

But, even though they could be either preferences or hedonism, I think it's useful to put them in their own category, which I'm calling gratification until someone comes up with a better name[2].

The central example of a preference in preference utilitarianism is a preference for a state of the world or the outcome of some process; gratifications are about details of how the process is implemented. The central example of hedonism is happiness or joy; gratifications mix some enjoyment (which can be quite low) with the specifics of how what caused that enjoyment.

So, though the categories overlap, I'd argue that it's useful to consider gratifications as a separate category. To give it some formalisation, I'll preliminarily denote it as a "preference over ...
Under a week left to win $1,000! By  questioning Oracle AIs.
by Stuart_Armstrong — 2019-08-25T17:02:46.921Z

That clickbait-y title is, for once, perfectly accurate: you have until the 31st of August to submit an entry to the Oracle design contest, either a counterfactual or a low bandwidth Oracle.

(A note: some of the submissions were done by people who clearly didn't understand how these types of Oracles worked; please check these details before submitting. In particular, any question you ask also has to contain some automated way of checking the Oracle's answer).
I'm interested in a sub-field of AI but don't know what to call it. 
by fowlertm — 2019-08-25T14:55:13.028Z

A colleague of mine and I have recently decided to tackle a project which we think has important ramifications for the field of artificial intelligence. We're vaguely aware that relevant work has been done at places like Carnegie Mellon and are hoping to get some guidance.

Our two overarching goals are to discover how models are constructed and to develop a model 'calculus' describing how an agent searches the space of possible hypotheses. We want to examine these questions from a number of different perspectives, studying the means by which both human and artificial agents actually arrive at models that allow them to understand the world, as well as the ways in which they would do so were they performing optimally.

Our motivation is to probe the subtle interchange between inductive and deductive reasoning. When is it that an agent moves from merely noticing patterns in data to reasoning on the basis of an axiomatic, deductive, predictive theory? Does it work differently in different domains? Do these inflection points have common characteristics, and, if so, can we use them to form a general theory of theories?

Does this framing even make any sense?

Along the way we hope to address a number of ancillary questions. Perhaps we can arrive at a formal theory of models, or probe the similarities and differences in how artificial intelligences and humans approach the task of compressing data down into simpler representations. There are obvious connections to Thomas Kuhn, to David Hume's is/ought problem, and to vast swathes of the philosophy of science.

We'd like help locating important books, seminal papers, good overview materials, words or phrases we can Google, theorists that have pushed the field forward, anything like that.

Thank you.
Am I going for a job interview with a woo pusher?
by CronoDAS — 2019-08-25T14:39:33.813Z

I responded to a job posting on Indeed and have been called in for an interview. This is the first job interview I've had in a long time.

http://www.indeed.com/viewjob?from=appsharedroid&jk=60abfc78c83955a2

Unfortunately I think the people at http://njneurofeedback.com might be selling woo, and I have some serious moral objections to defrauding people with placebo treatments. Are they pushing bullshit?

Edit: I went to the interview. It turns out that they misinterpreted my resume and they quickly told me that they wanted someone with more experience and sent me home.
OpenPhil on "GiveWell’s Top Charities Are (Increasingly) Hard to Beat"
by Raemon — 2019-08-24T23:28:59.705Z
Link: https://www.openphilanthropy.org/blog/givewells-top-charities-are-increasingly-hard-beat

This post by Alex Berger of OpenPhil outlines some shifts in thinking at OpenPhil, about what bar they set for their grantmaking. It seemed noteworthy...

as potentially relevant to the Drowning Children are Hard to Find discussion.
as an update on how OpenPhil thinks about making grants relating to US policy. (I think the intended thesis of the post was 'It's harder than we thought for US Giving to outperform Givewell Top Charities.')

Americans giving random Americans dollars as "null hypothesis."

[note: I'm not 100% sure I understood this framework, but here's my understanding]

For awhile, there was a background assumption that "unconditional cash transfers to the world's poorest" was a default charitable option.

This blogpost articulated something a root benchmark underlying that: if you're an American, you might consider yourself representative of the "random American" reference class. The default thing you can do with money is spend it about as productively as another random US citizen. If you're altruistic, the root benchmark is something like "would you spend this money in a way that generated more utility than giving it to a random member of your reference class?"

[obviously this is all pretty American centric, I'm guessing this applies roughly equally throughout the western world]

This article claims:

Unconditional cash transfers to the world's poorest are about 100x more useful than giving a dollar to a random other American
Givewell top charities are about 10x more impactful than GiveDirectly (i.e. 1000x to baseline)

The application of this is in evaluating US policy and scientific research. These are domains that seem relevant if you have near-termist "help people alive today" goals. They were also domains that OpenPhil had expected to outperform Givewell type charities (even paying for cost-of-living in the US)

OpenPhil used to set it's "near-termist" grantmaking standards to "100x". But their current experience is that it's increasingly hard to beat the 1000x value of top Givewell charities.

While we think a lot of our “near-termist, human-centric” grantmaking clears the 100x bar, we see less evidence that it will clear a ~1,000x bar.

Since adopting the cash transfer benchmark in 2015, we’ve made roughly 300 grants totalling $200 million in our near-termist, human-centric focus areas of criminal justice reform, immigration policy, land use reform, macroecon

...
Epistemic Spot Check: The Fate of Rome (Kyle Harper)
by Elizabeth — 2019-08-24T21:40:01.164Z

Introduction

Epistemic spot checks are a series in which I select claims from the first few chapters of a book and investigate them for accuracy, to determine if a book is worth my time. This month’s subject is The Fate of Rome, by Kyle Harper, which advocates for the view that Rome was done in by climate change and infectious diseases (which were exacerbated by climate change).

This check is a little different than the others, because it arose from a collaboration with some folks in the forecasting space. Instead of just reading and evaluating claims myself, I took claims from the book and made them into questions on a prediction market, for which several people made predictions of what my answer would be before I gave it. In some but not all cases I read their justifications (although not numeric estimates) before making my final judgement.

I expect we’ll publish a post-mortem on that entire process at some point, but for now I just want to publish the actual spot check. Because of the forecasting crossover, this spot check will differ from those that came before in the following ways:

Claims are formatted as questions answerable with a probability. If a claim lacks a question mark, the implicit question is “what is the probability this is true?”.

Questions have a range of specificity, to allow us to test what kind of ambiguities we can get away with (answer: less than I used).

Some of my answers include research from the forecasters, not just my own.

Due to timing issues, I finished the book and a second on the topic before I did the research for spot check.

Due to our procedure for choosing questions, I didn’t investigate all the claims I would have liked to.

Claims

Original Claim: “Very little of Roman wealth was due to new technological discoveries, as opposed to diffusion of existing tech to new places, capital accumulation, and trade.”

Question: What percentage of Rome’s gains came from technological gains, as opposed to diffusion of technical advantages, capital accumulation, and trade?

1%-30% log distribution

Data:

The Fall of Rome talks extensively about how trade degraded when the Romans left and how that lowered the standard of living.

https://brilliantmaps.com/roman-empire-gdp/ shows huge differences in GDP by region, implying there was a big opportunity to grow GDP through trade and diffusion of existing tech. That means potential growth jus

...
how should a second version of "rationality: A to Z" look like?
by Yoav Ravid — 2019-08-24T07:01:27.575Z

It's been ten years now since the completion of the sequences by Eliezer, it seems like a good time to look back and rethink a bit.

Eliezer himself said that he did many things wrong, and that the book could be much better.

So, if we made a second version, what ideas/articles would add/subtract? what would you rearrange?

I'm thinking of a second version as a handbook of rationality, that doesn't have to be just Eliezer's writing (though naturally it would be central), and will include ideas and writing from others as well.

Feel free to get into as much detail as you like.

until we make that second version, maybe this thread can be helpful to new readers to know what's missing, and what they can do to get even more value out of it.