trentmkelly/LessWrong-43k huggingface dataset:
Meetup Notes: Ole Peters on ergodicity
by Orborde — 2019-11-03T23:31:01.889Z

Ole Peters

claims that the standard expected utility toolbox for evaluating wagers is a flawed basis for rational decisionmaking. In particular, it commonly fails to take into account that an investor/bettor taking a series of repeated bets is not an

ergodic

process.

Optimization Process, internety, myself, and a couple others spent about 5 hours across a couple of Seattle meetups investigating what Peters was saying.

Background

Why do we care?

Proximally, because

Nassim Taleb is bananas about ergodicity

.

More interestingly, expected utility maximization is widely accepted as the basis for rational decisionmaking. Finding flaws (or at least pathologies) in this foundation is therefore quite high leverage.

A specific example: many people's retirement investment strategies might be said to be taking the "ensemble average" as their optimization target - i.e. their portfolios are built on the assumption that, every year, an individual investor should make the choice that, when averaged across (e.g.) 100,000 investors making that choice for that year, will maximize the mean wealth (or mean utility) of investors in the group at the end of that year. It's

claimed

that this means that individual retirement plans can't work because many individuals will, in actuality, eventually be impoverished by market swings, and that social insurance schemes (e.g. Social Security) where the current rich are transferring wealth to the current poor avoid this pitfall.

Claims about shortcomings in expected utility maximization are also interesting because I've felt vaguely confused for a long time about why expected value/utility is the right way to evaluate decisions; it seems like I might be more strongly interested in something like "the 99th percentile outcome for the overall utility generated over my lifetime". Any work that promises to pick at the corners of EU maximization is worth looking at.

What does existing non-Peters theory say?

The

Von Neumann-Morgenstern theorem

says, loosely, that all rational actors are maximizing some utility function in expectation. It's almost certainly not the case that Ole Peters has produced a counterexample, but (again) identifying apparently pathological behavior implied by the VNM math would be quite useful.

Economics research as a whole tends to take it as given that individual actors are trying to maximi...
Picture Frames, Window Frames and Frameworks
by Raemon — 2019-11-03T22:09:58.181Z

Some commenters on Noticing Frames were confused about what I actually meant by "frame." I defined it briefly as "different ways of seeing, thinking and communicating", but this was a bit vague.

I'm definitely using the word "frame" as a broad metaphor, rather than a concrete specific phenomenon. I'm not at a point where I'm sure I have a more concrete phenemonon to explicitly describe. I recognize this as an important red flag – specificity is good, and I'll try to get more concrete in future posts. For now, though, I think the broad metaphor is useful, and want to explain it slightly better.

After reflecting on the feedback, I realized it was actually three different metaphors:

Picture Frames (ways of communicating)
Window Frames (ways of seeing)
Frameworks (ways of thinking)

Upon reflection, I endorse using all three metaphors fuzzily rolled into one.

The past few years, I kept thinking I had figured out why resolving disagreements was hard, and kept being surprised by new ways to subtly miss each other. At first I thought the main reasons to disagree were "different beliefs", "different values", and "confusion over beliefs/values." This isn't wrong, but it wasn't nuanced enough for me to notice all the disconnects.

I've come to believe there's an important general skill, which goes something like:

Trigger: Notice when a conversation is going nowhere, especially over the timescale of months/years.

Action: Look for frame differences outside of the ones you know to make sense of, and figure out how to make sense of them.

That's a hell of a vague action, I admit. It's composed of smaller, more specific actions (which are probably easier to learn). I was worried that if I named the most specific actions I could point at, people would focus on that and lose sight of the more general problem. I'm not even certain that the "different ways of seeing, thinking, and communicating" schema is complete – it's meant to be illustrative rather than comprehensive.

Someday, ideally, there'll be a "how to" doublecrux sequence that teaches concrete skills and exercises that are each immediately useful. (I think Eli might be working on something like this). But this sequence is more about "Why" than "How", much of what I want to talk about is the overall mindset, and how various skills fit together.

I'm using "frame" to mean "the broadest possible way for two people can miss eac...
Skill and leverage
by KatjaGrace — 2019-11-03T21:10:02.416Z

Sometimes I hear people say ‘how can make a big difference to the world, when I can’t make a big difference to that pile of dishes in my sock drawer?’ How can I improve the sustainability of world energy usage when I can’t improve the sustainability of my own Minecraft usage? The basic thought is that if you can’t do ‘easy’ things that humans are meant to be able to do, on the scale of your own life, you probably lack general stuff-doing ability, and are not at the level where you can do something a million times more important.

I think this is a generally wrong model, for two reasons. One is that the difficulty of actions is not that clearly well ordered—if you have a hard time keeping your room tidy, this just doesn’t say that much about whether you can write well or design rockets or play the piano.

The second reason is that the difficulty of actions doesn’t generally scale with their consequences. I think this is more unintuitive.

Some examples:

Applying for funding for a promising new anti-cancer drug is probably about as hard as applying for funding for an investigation into medieval references to toilet paper (and success is probably easier), but the former is much more valuable.

Having a good relationship with your ex Bob might be about as hard and take about the same skills as having a good relationship with your more recent ex Trevor, but if you have children with Trevor, the upside of that effort may be a lot higher.

If you have a hard time making a speech at your brother’s birthday, you will probably also have a hard time making a speech to the UN. But, supposing it is fifty thousand times more important, it isn’t going to be fifty thousand times harder. It’s not even clear that it is going to be harder at all—it probably depends on the topic and your relationship with your family and the UN.

Writing a good book about x-risk is not obviously much harder than writing a good book about the role of leprechauns through the ages, but is vastly more consequential in expectation.

My basic model is that you can have skills that let you do particular physical transformations (an empty file into a book, some ingredients into a cake), and there are different places you can do those tricks, and some of the places are just much higher leveraged than others. Yet the difficulty is mostly related to the skill or trick. If you are trying to start a fire, holdin...
But exactly how complex and fragile?
by KatjaGrace — 2019-11-03T18:20:01.268Z

This is a post about my own confusions. It seems likely that other people have discussed these issues at length somewhere, and that I am not up with current thoughts on them, because I don’t keep good track of even everything great that everyone writes. I welcome anyone kindly directing me to the most relevant things, or if such things are sufficiently well thought through that people can at this point just correct me in a small number of sentences, I’d appreciate that even more.

~

The traditional argument for AI alignment being hard is that human value is ‘complex’ and ‘fragile’. That is, it is hard to write down what kind of future we want, and if we get it even a little bit wrong, most futures that fit our description will be worthless.

The illustrations I have seen of this involve a person trying to write a description of value conceptual analysis style, and failing to put in things like ‘boredom’ or ‘consciousness’, and so getting a universe that is highly repetitive, or unconscious.

I’m not yet convinced that this is world-destroyingly hard.

Firstly, it seems like you could do better than imagined in these hypotheticals:

These thoughts are from a while ago. If instead you used ML to learn what ‘human flourishing’ looked like in a bunch of scenarios, I expect you would get something much closer than if you try to specify it manually. Compare manually specifying what a face looks like, then generating examples from your description to using modern ML to learn it and generate them.

Even in the manually describing it case, if you had like a hundred people spend a hundred years writing a very detailed description of what went wrong, instead of a writer spending an hour imagining ways that a more ignorant person may mess up if they spent no time on it, I could imagine it actually being pretty close. I don’t have a good sense of how far away it is.

I agree that neither of these would likely get you to exactly human values.

But secondly, I’m not sure about the fragility argument: that if there is basically any distance between your description and what is truly good, you will lose everything.

This seems to be a) based on a few examples of discrepancies between written-down values and real values where the written down values entirely exclude something, and b) assuming that there is a fast takeoff so that the relevant AI has its values forever, and takes over th...
[Math] Proofs vs. documentation vs. "it's trivial"
by aaq — 2019-11-03T16:50:17.428Z

In my abstract algebra and real analysis classes, my TAs say that I frequently come up with "technically right" solutions. They don't mean in the Futurama sense; "technically right" here is a little looser - I did some trick that wasn't really expected, or a bit outside of the nature of the class, in order to arrive at my proof.

This is really cool! I readily admit my ego is a bit boosted by contrasting my less obvious approach to the more studied approach of my classmates. It's good to embrace what's unique about yourself.

The problem is that the solutions I generate on a homework problem, where I have all the time in the world to putt about and generate little sub-proofs and lemmas, don't work in an exam situation. I don't write that fast. Even if I did write that fast, I don't think that fast. The things I claim aren't quite so trivial that I feel comfortable using them without proof - I'd lose points if I did.

Calling something "trivial" in math is more of a judgment call than I think people realize. The universal-accessibility option for triviality might be to hyperlink to someone else's proof of a claim, that is in roughly the same flavor as your own; that way, a reader who feels confident they could reconstruct the proof can just read over it, without hurting the flow of the text, and someone who feels less confident could slow down. Since a lot of good mathematics texts are designed to be reread to jog one's own memory from time to time, this seems like an especially good balance - after reading the proof through the first or second time for the nontrivial triviality, it probably genuinely does start to look trivial to you. You've absorbed whatever was missing that made it look nontrivial.

And that's interesting. I think that "trivial" statements are more like heuristic-checks than nontrivial ones. Do you have the requisite background knowledge to grind this proof out without breaking a sweat? Do you realize you have that knowledge - is it a "known known" for you? Maybe it's an unknown known, that happens often. But it also happens often that you genuinely have mismatched heuristics for the problems in question. They probably aren't wrong -- they're just not a good fit for right now.

Mismatched heuristic-checks are the same phe...
[Question] When Do Unlikely Events Should Be Questioned?
by NSegall — 2019-11-03T15:21:25.930Z

Today someone shared a picture on Facebook showing four d20 dies (d20 is a 20-sided die), supposedly all landed on 20. He was saying how cool it was that he and his friends were playing a tabletop role-playing game and they all got a 20 on their spot check at the same time.

My first reaction was "Huh, neat!"

My second reaction was "p = (1/20)^4 = 160,000. In Israel's small role-playing community this seems just very unlikely. This picture is probably a fake."

1st me: "Well, this just happened to be exactly 4 dies. If that happened to be 5 dies, 6 dies, etc ... we would still consider this as an exceptional event with low probability. We have to sum over all the probabilities for all plausible die numbers"

2nd me: "Sure, but every extra die reduces the probability by a factor of 1/20, so it seems likely that we can save ourselves the trouble of summing and just assume that 1/160,000 gives us a fair estimate."

1st me: "But what about if there were only 3 dies? 2 dies? What if they had all shown 1's instead of 20? What if they had shown 17, 18, 19, 20? What if we had encountered that picture on an international role-playing group, much larger than the Israeli one? Where do we draw the line? We need to find a way to estimate the probability of observing a picture on social media of something unlikely that is drawn from a huge set of unlikely possibilities"

At that point me no. 2 usually frowns and forgets about the matter until it emerges again, leaving it unresolved.

So I am now seeking the community's wisdom; let the elders speak. How do you estimate the likelihood of occurrences such as this? Can this problem be easily resolved somehow?
Where should I ask this particular kind of question?
by Long try — 2019-11-03T11:27:24.478Z

I am pondering a hypothetical scenario that I think is fascinating but quite unrealistic and involves knowledge across a wide variety of fields, of which IMO physics gets the better part.

I'm considering some sites that I know. Reddit has a sub called r/AskScienceDiscussion but this sub is not very warm to this type of query. Quora has degraded so much and doesn't even have the option to expand the subject over a length of mere 150 chars or so, which is utterly ridiculous. I'm not sure about Stackexchange - should I post in their physics site? LessWrong boasts that people can ask anything and it aims to be better than the former 2, but judging from what I read, questions almost exclusively focus on rationality, and the reach (the amount of folks who can read it, or how popular the platform is within a population) is small. There's another option of posting in fora such as physicsforum, but I don't have an account there so my knowledge is limited about how people will respond.

Thus, my question is a bit meta: what do you think is the best place to post the question? If there's something better than what I listed, then please kindly enlighten me. Also, I'd like to ask you guys to not be biased toward our own platform we're standing on. Thanks!
“embedded self-justification,” or something like that
by nostalgebraist — 2019-11-03T03:20:01.848Z

preamble

Sometimes I wonder what the MIRI-type crowd thinks about some issue related to their interests. So I go to alignmentforum.org, and quickly get in over my head, lost in a labyrinth of issues I only half understand.

I can never tell whether they’ve never thought about the things I’m thinking about, or whether they sped past them years ago. They do seem very smart, that’s for sure.

But if they have terms for what I’m thinking of, I lack the ability to find those terms among the twists of their mirrored hallways. So I go to tumblr.com, and just start typing.

parable (1/3)

You’re an “agent” trying to take good actions over time in a physical environment under resource constraints. You know, the usual.

You currently spend a lot of resources doing a particular computation involved in your decision procedure. Your best known algorithm for it is O(N^n) for some n.

You’ve worked on the design of decision algorithms before, and you think this could perhaps be improved. But to find it, you’d have to shift resources some away from running the algorithm for a time, putting them into decision algorithm design instead.

You do this. Almost immediately, you discover an O(N^(n-1)) algorithm. Given the large N you face, this will dramatically improve all your future decisions.

Clearly (…“clearly”?), the choice to invest more in algorithm design was a good one.

Could you have anticipated this beforehand? Could you have acted on that knowledge?

parable (2/3)

Oh, you’re so very clever! By now you’ve realized you need, above and beyond your regular decision procedure to guide your actions in the outside world, a “meta-decision-procedure” to guide your own decision-procedure-improvement efforts.

Your meta-decision-procedure does require its own resource overhead, but in exchange it tells you when and where to spend resources on R&D. All your algorithms are faster now. Your decisions are better, their guiding approximations less lossy.

All this, from a meta-decision-procedure that’s only a first draft. You frown over the resource overhead it charges, and wonder whether it could be improved.

You try shifting some resources away from “regular decision procedure design” into “meta-decision-procedure-design.” Almost immediately, you come up with a faster and better procedure.

Could you have anticipated this beforehand? Could you have acted on that knowledge?

parable (3/3)

Oh, y...
Open & Welcome Thread - November 2019
by habryka — 2019-11-02T20:06:54.030Z

If it’s worth saying, but not worth its own post, here's a place to put it.

You can also make a shortform post.

And, if you are new to LessWrong, here's the place to introduce yourself.

Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ.

The Open Thread sequence is here.
Economics and Evolutionary Psychology
by Chris_Leong — 2019-11-02T16:36:34.026Z
Link: http://www.daviddfriedman.com/Academic/econ_and_evol_psych/economics_and_evol_psych.html?fbclid=IwAR2EokR7Z1lUPCDKTrJOFCAlZ-M-vPqbjhau1fxdaMYptrcgsPS-ZXaJTDo

This article provides game theoretic reasons for acting in ways that may seem naive from the standard economic point of view.

Key quotes from the article:

"As a member of a hunter-gatherer band, you engage in a variety of transactions with your fellows, trading goods and services—food, sex, support in intra-group conflict, and the like... In this world all markets are thin—it is, after all, a small band—so the typical transaction is a bilaterial monopoly bargain.

Assume an environment sufficiently stable so that, for some transactions, there are “usual prices.” Those prices must be within the bargaining ranges of both buyer and seller, since otherwise the transactions would not occur. The environment is not, however, perfectly stable. Sometimes the circumstances of one party or another shift his bargaining range—the range of terms for which the transaction is in his interest.

You are a buyer whose current circumstances make the good much more valuable to you than usual, widening the bargaining range. If you could somehow commit yourself not to pay more than the usual price, you, rather than the seller, would get the increased benefit from this transaction. One way to do so is to be emotionally programmed to resent any increase above the usual price—resent it enough so that the humiliation of being “cheated” will outweigh the gain from the transaction.

As in any bilateral monopoly game, the argument works both ways. If the seller could somehow commit himself not to accept less than your reservation price, he would be the one to pocket the gains from the trade. There is, however, an important difference between your situations. You know the usual price and, assuming the special circumstances affect only you, know that it is probably within the seller’s bargaining range. So your commitment strategy is unlikely to commit you to a price outside the bargaining range—which would make the transaction impossible. The seller does not know your reservation price, so if he commits himself to his guess at what you are willing to pay he may choose a price at which the transaction can not occur.

... What about the situation where the seller’s costs are unusually high, making him unwilling to sell at the usual price? If the result is to eliminate the bargaining range, no transaction will or s...
Speaking up publicly is heroic
by jefftk — 2019-11-02T12:00:01.882Z

Content warning: discussion of abuse.

I was recently in a discussion of safety, "me too", and responding to
harmful behavior where someone wrote, paraphrased:

Discussing things like this, publicly and in writing, is dangerous.
Posting negative things about people could get you sued for libel, or
even physically assaulted if the person takes it poorly enough. Why
would you put yourself at risk, just to get something off your chest?

While they're right that speaking up is risky, the idea that people
who make public posts are doing it lightly doesn't match what I've
seen. Most people who post know the risk, but decide to go ahead for
important reasons including "no one else should have to go through
this" and "if people know bad behavior will come back to bite them
they'll behave better."

In November 2017 someone in one of my communities posted publicly
about someone who had abused them, and then in September 2018 two
people in another of my communities posted about a different
person. [1] While these two abusers were different in a series of
ways, they had a lot in common: they were relatively popular, they
used age, inexperience, and intoxicants to take advantage of people,
they had abused a series of people, and there were whispered warnings
but not everyone heard or believed them.

In both cases, the public posts completely changed the situation by
pulling all of this into common knowledge. The abusers were rightly
ejected and communities started working on figuring out how they had
let things get to this point.

We need to build a world where this is not needed, and where we have
good social systems for handling harmful behavior. I don't know what
this should look like yet, however, and all the proposals I've seen
have good and bad parts. In my corner of the world I've been working
on this with BIDA's safety
committee, and I'm glad that communities are generally taking this
more seriously. We also need to handle this on a personal level,
supporting friends who've been hurt and holding others accountable
when they hurt people. But while stopping abusers should not require
this level of personal risk, as long as it does I have enormous
respect for people who decide to take that risk and speak up.

[1] I'm not linking the two here, even though both are world-readable,
because they've served their purposes and I'm not tryi...