trentmkelly/LessWrong-43k huggingface dataset:
The Parable of Predict-O-Matic
by abramdemski — 2019-10-15T00:49:20.167Z

I've been thinking more about partial agency. I want to expand on some issues brought up in the comments to my previous post, and on other complications which I've been thinking about. But for now, a more informal parable. (Mainly because this is easier to write than my more technical thoughts.)

This relates to oracle AI and to inner optimizers, but my focus is a little different.

1

Suppose you are designing a new invention, a predict-o-matic. It is a wonderous machine which will predict everything for us: weather, politics, the newest advances in quantum physics, you name it. The machine isn't infallible, but it will integrate data across a wide range of domains, automatically keeping itself up-to-date with all areas of science and current events. You fully expect that once your product goes live, it will become a household utility, replacing services like Google. (Google only lets you search the known!)

Things are going well. You've got investors. You have an office and a staff. These days, it hardly even feels like a start-up any more; progress is going well.

One day, an intern raises a concern.

"If everyone is going to be using Predict-O-Matic, we can't think of it as a passive observer. Its answers will shape events. If it says stocks will rise, they'll rise. If it says stocks will fall, then fall they will. Many people will vote based on its predictions."

"Yes," you say, "but Predict-O-Matic is an impartial observer nonetheless. It will answer people's questions as best it can, and they react however they will."

"But --" the intern objects -- "Predict-O-Matic will see those possible reactions. It knows it could give several different valid predictions, and different predictions result in different futures. It has to decide which one to give somehow."

You tap on your desk in thought for a few seconds. "That's true. But we can still keep it objective. It could pick randomly."

"Randomly? But some of these will be huge issues! Companies -- no, nations -- will one day rise or fall based on the word of Predict-O-Matic. When Predict-O-Matic is making a prediction, it is choosing a future for us. We can't leave that to a coin flip! We have to select the prediction which results in the best overall future. Forget being an impassive observer! We need to teach Predict-O-Matic human values!"

You think about this. The thought of Predict-O-Matic deliberately steering the futur...
Strong stances
by KatjaGrace — 2019-10-15T00:40:01.286Z

I. The question of confidence

Should one hold strong opinions? Some say yes. Some say that while it’s hard to tell, it tentatively seems pretty bad (probably).

A quick review of purported or plausible pros:

Strong opinions lend themselves to revision:

Nothing will surprise you into updating your opinion if you thought that anything could happen. A perfect Bayesian might be able to deal with myriad subtle updates to vast uncertainties, but a human is more likely to notice a red cupcake if they have claimed that cupcakes are never red. (Arguably—some would say having opinions makes you less able to notice any threat to them. My guess is that this depends on topic and personality.)

‘Not having a strong opinion’ is often vaguer than having a flat probability distribution, in practice. That is, the uncertain person’s position is not, ‘there is a 51% chance that policy X is better than policy -X’, it is more like ‘I have no idea’. Which again doesn’t lend itself to attending to detailed evidence.

Uncertainty breeds inaction, and it is harder to run into more evidence if you are waiting on the fence, than if you are out there making practical bets on one side or the other.

(In a bitterly unfair twist of fate) being overconfident appears to help with things like running startups, or maybe all kinds of things.

If you run a startup, common wisdom advises going around it saying things like, ‘Here is the dream! We are going to make it happen! It is going to change the world!’ instead of things like, ‘Here is a plausible dream! We are going to try to make it happen! In the unlikely case that we succeed at something recognizably similar to what we first had in mind, it isn’t inconceivable that it will change the world!’ Probably some of the value here is just a zero sum contest to misinform people into misinvesting in your dream instead of something more promising. But some is probably real value—suppose Bob works full time at your startup either way. I expect he finds it easier to dedicate himself to the work and has a better time if you are more confident. It’s nice to follow leaders who stand for something, which tends to go with having at least some strong opinions. Even alone, it seems easier to work hard on a thing if you think it is likely to succeed. If being unrealistically optimistic just generates extra effort to be put toward your project’s success, rather t

...
Impact measurement and value-neutrality verification
by evhub — 2019-10-15T00:06:51.879Z

Recently, I've been reading and enjoying Alex Turner's Reframing Impact sequence, but I realized that I have some rather idiosyncratic views regarding impact measures that I haven't really written up much yet. This post is my attempt at trying to communicate those views, as well as a response to some of the ideas in Alex's sequence.

What can you do with an impact measure?

In the "Technical Appendix" to his first Reframing Impact post, Alex argues that an impact measure might be "the first proposed safeguard which maybe actually stops a powerful agent with an imperfect objective from ruining things—without assuming anything about the objective."

Personally, I am quite skeptical of this use case for impact measures. As it is phrased—and especially including the link to Robust Delegation—Alex seems to be implying that an impact measure could be used to solve inner alignment issues arising from a model with a mesa-objective that is misaligned relative to the loss function used to train it. However, the standard way in which one uses an impact measure is by including it in said loss function, which doesn't do very much if the problem you're trying to solve is your model not being aligned with that loss.[1]

That being said, using an impact measure as part of your loss could be helpful for outer alignment. In my opinion, however, it seems like that requires your impact measure to capture basically everything you might care about (if you want it to actually solve outer alignment), in which case I don't really see what the impact measure is buying you anymore. I think this is especially true for me because I generally see amplification as being the right solution to outer alignment, which I don't think really benefits at all from adding an impact measure.[2]

Alternatively, if you had a way of mechanistically verifying that a model behaves according to some impact measure, then I would say that you could use something like that to help with inner alignment. However, this is quite different from the standard procedure of including an impact measure as part of your loss. Instead of training your agent to behave according to your impact measure, you would instead have to train it to convince some overseer that it is internally implementing some algorithm which satisfies some minimal impact criterion. It's possible that this is what Alex actually has in mind in terms of how

...
Schematic Thinking: heuristic generalization using Korzybski's method
by romeostevensit — 2019-10-14T19:29:14.672Z

Epistemic status: exploration of some of the intuitions involved in discussions behind this post at MSFP. Could be considered part of an unstructured sequence titled 'towards fewer type errors in metaphilosophy'

Summary: When moving up or down the ladder of abstraction you're forced to make choices about restriction or relaxation of domain of relevance. Don't just reason about the valid range/scope, reason about your reasoning about the valid range/scope and ask what sort of operations you'd need to be doing to make a different choice seem valid. This resolves many types of confusions and is a crux identifier for 2-place confusions.

Alfred Korzybski directs us to develop the faculty to be conscious of the act of abstracting. This means that that one has meta cognitive awareness when one does things like engage in the substitution effect, analogical reasoning, shifting the coarse-grainedness of an argument, use of the 'to be' verb form, shifting from one Marr Level to another in mid sentence etc. One of the most important skills that winds up developed as a result of such training is much more immediate awareness of what Korzybski calls the multiordinality of words, which one will be familiar with if you have read A Human's Guide to Words (esp. 24 and 36) or are otherwise familiar with the Wittgensteinian shift in analytic philosophy (related: the Indeterminacy of Translation). In short, many words are underdetermined in their referents along more than one dimension, leading to communication problems both between people and internally (for an intuitive example, one can imagine people talking past each other in a discussion of causation when they are discussing different senses of Cause without realizing it).

I want to outline what one might call second order multiordinal words or maybe schematic thinking. With multiordinal words, one is aware of all the values that a word could be referring to. With schematic thinking one is also aware of all the words that could have occupied the space that word occupies. Kind of like seeing everything as an already filled out madlibs and reconstructing the unfilled out version.

This may sound needlessly abstract but you're already familiar with a famous example. One of Charlie Munger's most famous heuristics is inversion. With inversion we can check various ways we might be con...
[AN #68]: The attainable utility theory of impact
by Rohin Shah — 2019-10-14T17:00:01.424Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Stuart Russell at CHAI has published a

book

about AI safety. Expect a bonus newsletter this week summarizing the book and some of the research papers that underlie it!

Audio version

here

(may not be up yet).

Highlights

Reframing Impact - Part 1

(Alex Turner) (summarized by Rohin): This sequence has exercises that will be spoiled by this summary, so take a moment to consider whether you want to read the sequence directly.

This first part of the sequence focuses on identifying what we mean by impact, presumably to help design an impact measure in the future. The punch line: an event is impactful to an agent if it changes the agent's ability to get what it wants. This is Attainable Utility (AU) theory. To quote the sequence: "How could something possibly be a big deal to us if it doesn't change our ability to get what we want? How could something not matter to us if it does change our ability to get what we want?"

Some implications and other ideas:

- Impact is relative to an agent: a new church is more impactful if you are a Christian than if not.

- Some impact is objective: getting money is impactful to almost any agent that knows what money is.

- Impact is relative to expectations: A burglar robbing your home is impactful to you (you weren't expecting it) but not very impactful to the burglar (who had planned it out). However, if the burglar was unsure if the burglary would be successful, than success/failure would be impactful to them.

While this may seem obvious,

past work

(

AN #10

) has talked about impact as being caused by changes in state. While of course any impact does involve a change in state, this is the wrong level of abstraction to reason about impact: fundamentally, impact is related to what we care about.

Rohin's opinion: To quote myself from a discussion with Alex, "you're looking at the optimal Q-function for the optimal utility function and saying 'this is a good measure of what we care about' and of course I agree with that". (Although this is a bit inaccurate -- it's not the optimal Q-function, but the Q-function relative to what we expect a...
Whistle-based Synthesis
by jefftk — 2019-10-14T12:10:01.968Z

I'm reasonably happy with my

Bass Whistle,
where I can whistle and have it come out as a decent sounding bass.
I've been using it when playing mandolin in a duo, and it fits well
there. When playing piano or with a piano player, however, there's
already bass so something that falls in a different place in the
overall sound would be better. That could be melody,

though I can't whistle fast enough
for everything, but probably something simpler: harmonies, riffs,
horn lines.

When I take my current software, optimized for bass, and tell it to
synthesize notes a few octaves up, it sounds terrible:

Raw whistled input:

(mp3)

Bass version (needs headphones or good speakers):
(mp3)

Treble version:
(mp3)

I'm using simple additive synthesis with the first four harmonics,
which means adding together four sine waves. I think what's going on
is that higher notes need more complexity to sound good? Playing
around with distortion and fading the harmonics at different rates it
sounds a bit more interesting:

Adding distortion:
(mp3)

Adding fade:
(mp3)

Adding both:
(mp3)

I'm still not very happy with it, though. It sounds artificial and
silly. There are good synthesizers, the product of decades of work on
turning "play this note at this time" into good sounding audio, so
perhaps I could use my pitch detection to drive a standard
synthesizer?

I made some stand-alone open
source software that pipes the pitch detection through to MIDI.
This was kind of tricky: MIDI doesn't have a way to say "play this
frequency". Instead you just have "play this note" and "bend the note
by this much". How to interpret pitch bend is up to the synthesizer,
but generally the range is ±2 half steps. So we need some
math:

in: wavelength
in: sample_rate
in: current_note

# Convert from "this wave is 23.2 samples long"
# to "the frequency is 1896.6 HZ".
frequency = sample_rate / wavelength

# MIDI is equal tempered, with each octave divided
# into twelve logarithmically equal pieces. Take
# A440 as a reference point, so represent our
# 1896.6 HZ as "25.29 half steps above A440":
distance_from_a440 = 12 * log2(frequency / 440)

# A440 is A4, or midi note 69, so this is 94.29.
fractional_note = 69 + distance_from_a440

# MIDI uses a note + bend encoding. Stay on the
# same note if possible to avoid spurious attacks.
if (current_note and
current_note

...
Maybe Lying Doesn't Exist
by Zack_M_Davis — 2019-10-14T07:04:10.032Z

In "Against Lie Inflation", the immortal Scott Alexander argues that the word "lie" should be reserved for knowingly-made false statements, and not used in an expanded sense that includes unconscious motivated reasoning. Alexander argues that the expanded sense draws the category boundaries of "lying" too widely in a way that would make the word less useful. The hypothesis that predicts everything predicts nothing: in order for "Kevin lied" to mean something, some possible states-of-affairs need to be identified as not lying, so that the statement "Kevin lied" can correspond to redistributing conserved probability mass away from "not lying" states-of-affairs onto "lying" states-of-affairs.

All of this is entirely correct. But Jessica Taylor (whose post "The AI Timelines Scam" inspired "Against Lie Inflation") wasn't arguing that everything is lying; she was just using a more permissive conception of lying than the one Alexander prefers, such that Alexander didn't think that Taylor's definition could stably and consistently identify non-lies.

Concerning Alexander's arguments against the expanded definition, I find I have one strong objection (that appeal-to-consequences is an invalid form of reasoning for optimal-categorization questions for essentially the same reason as it is for questions of simple fact), and one more speculative objection (that our intuitive "folk theory" of lying may actually be empirically mistaken). Let me explain.

(A small clarification: for myself, I notice that I also tend to frown on the expanded sense of "lying". But the reasons for frowning matter! People who superficially agree on a conclusion but for different reasons, are not really on the same page!)

Appeals to Consequences Are Invalid

There is no method of reasoning more common, and yet none more blamable, than, in philosophical disputes, to endeavor the refutation of any hypothesis, by a pretense of its dangerous consequences[.]

—David Hume

Alexander contrasts the imagined consequences of the expanded definition of "lying" becoming more widely accepted, to a world that uses the restricted definition:

[E]veryone is much angrier. In the restricted-definition world, a few people write posts suggesting that there may be biases affecting the situation. In the expanded-definition world, those same people write posts accusing the other side of being liars perpetrating a fraud. I am

...
Regarding Archimedes (a philosophy of math anecdote)
by KyriakosCH — 2019-10-13T21:25:27.271Z

Regarding tales told of Archimedes, other than those about his enthusiasm when he managed to conceive of the law of flotation, his contempt for the Roman soldier who demanded he would cease his calculations on those circles he had drawn on sand and the epigram on his tomb where his discovery of the relation between the volumes of a sphere, a cone and a cylinder was accounted for, there is one other worth of mentioning:

He once met, in a beach near Syracuse, a young boy and a slightly older and much taller youth. It was a warm day and those two were wearing white himatia without chitons underneath, as was usual for poorer people. Both left the right shoulder bare, with the fabric rising diagonally to the left shoulder where a pin held it in place, yet Archimedes noticed that in the way they had positioned themselves right next to each other the upper part of the child’s clothes was reaching exactly to the height of the lower part of the youth’s next to him. This formed an elegant, straight line, starting at the lower edge of the first cloth and ending to the upper edge of the other, and from a distance one would get the impression that it was all one continuous material instead of two distinct himatia.

Despite his surprise, he immediately became aware that while he could notice the effect due to standing opposite those two, their position naturally rendered it impossible for them to witness and enjoy this harmonious sight.

That they remained there, immobile, because they were indeed aware of the effect and meant for anyone coming near to observe it, was readily obvious; yet Archimedes couldn’t refrain from walking closer and asking just why they went into so much trouble so as to produce something they themselves weren’t allowed to observe, given that if they’d even attempt to steal a glance at it the delicate balance instantly would be ruined and therefore nothing would remain to be seen...

The youth kept silent, but the child, being more impulsive, replied that Archimedes was wrong. It wasn’t at all true that they stood there in this manner out of an intention to present to others the form he saw. To be precise, they were fully oblivious that anyone would be seeing such a thing when coming from afar. They were standing so close to each other out of mere habit. Lastly, they weren’t moving at all beca...
MA Price Accuracy Law
by jefftk — 2019-10-13T12:00:01.815Z

Massachusetts has an interesting law for grocery stores to make sure
price scanners are configured correctly: if your item rings up for
more than the price on the shelf you get one for free (or $10 off it's
it's more than $10). Specifically:

if there is a discrepancy between the advertised price, the sticker
price, the scanner price or the display price and the checkout price
on any grocery item, a food store or a food department shall charge a
consumer the lowest price. If the checkout price or scanner price is
not the lowest price or does not reflect any qualifying discount, the
seller:

shall not charge the consumer for 1 unit of the grocery
item, if the lowest price is $10 or less;

shall charge the consumer the lowest price less $10 for 1 unit of
the grocery item, if the lowest price is more than $10

— MGL
I.XV.94.184.C

The grocery store is required to put a sign at each register
describing the law, which means that when you notice this you can
point to the sign. Which is way better than trying to show the
cashier the relevant text of the law on your phone would be.

I have fun trying to remember the price I see for each item as I put
it into my cart so if it rings up at a different price I can point
that out. The law has an exception for cases where the price is a
"gross error" (off by half) but in most cases discrepancies are small:
ringing up at $4.99 when it said $4.50 on the shelf. Because you get
the item for free if they've overcharged you, however, what matters is
just that they put a misleadingly low price on the shelf.

I've noticed stores rarely have a good system in place for fixing
these problems. When I catch one they generally check and give me the
item for free, but that doesn't usually translate into fixing the
price on the shelf. Which means that when I come in next time, it's
often still wrong.

This seems like something that a group of shoppers could use together.
Whenever anyone noticed a mispricing they could post to a mailing list
("the store brand blueberries are marked $3.99 but ring up as $4.29"),
and then everyone on the list could go get some free blueberries.
This would probably get stores to be faster about updating their
prices.

Even if the stores got very fast at fixing things, though, it could
still be rough for them. Say one person goes through and notices
they've been overcharged for s...
What's going on with "provability"?
by Sunny from QAD — 2019-10-13T03:59:08.748Z

Every so often I hear seemingly mathematical statements involving the concept of being provable. For example:

I've seen Gödel's Incompleteness Theorem stated as "if a mathematical system is powerful enough to express arithmetic, then either it contains a contradiction or there are true statements that it cannot prove."
On the AI alignment forum, one of the pinned sequences describes Löb's Theorem as "If Peano Arithmetic can prove that a proof of P would imply the truth of P, then it can also prove P is true".

I find each of these statements baffling for a different reason:

Gödel: What could it mean for a statement to be "true but not provable"? Is this just because there are some statements such that neither P nor not-P can be proven, yet one of them must be true? If so, I would (stubbornly) contest that perhaps P and not-P really are both non-true.
Löb: How can a system of arithmetic prove anything? Much less prove things about proofs?

And I also have one more general confusion. What systems of reasoning could these kinds of theorems be set in? For example, I've heard that there are proofs that PA is consistent. Let's say one of those proofs is set in Proof System X. Now how do we know that Proof System X is consistent? Perhaps it can be proven consistent by using Proof System Y? Do we just end up making an infinite chain of appeals up along a tower of proof systems? Or do we eventually drive ourselves into the ground by reaching system that nobody could deny is valid? If so, why don't we just stop and PA or ZFC?

Oh, speaking of ZFC. There seems to be a debate about whether we should accept the Axiom of Choice. Isn't it...obviously true? I don't really understand this topic well enough to have any particular question about the AC debate, but my confusion definitely extends to that region of concept space.

So here's my question: Where can I learn about "provability" and/or what clarifying insights could you share about it?