trentmkelly/LessWrong-43k huggingface dataset:
The Credit Assignment Problem
by abramdemski — 2019-11-08T02:50:30.412Z

This post is eventually about partial agency. However, it's been a somewhat tricky point for me to convey; I take the long route. Epistemic status: slightly crazy.

I've occasionally said "Everything boils down to credit assignment problems."

What I really mean is that credit assignment pops up in a wide range of scenarios, and improvements to credit assignment algorithms have broad implications. For example:

Politics.

When politics focuses on (re-)electing candidates based on their track records, it's about credit assignment. The practice is sometimes derogatorily called "finger pointing", but the basic computation makes sense: figure out good and bad qualities via previous performance, and vote accordingly.
When politics instead focuses on policy, it is still (to a degree) about credit assignment. Was raising the minimum wage responsible for reduced employment? Was it responsible for improved life outcomes? Etc.

Economics.

Money acts as a kind of distributed credit-assignment algorithm, and questions of how to handle money, such as how to compensate employees, often involve credit assignment.
In particular, mechanism design (a subfield of economics and game theory) can often be thought of as a credit-assignment problem.

Law.

Both criminal law and civil law involve concepts of fault and compensation/retribution -- these at least resemble elements of a credit assignment process.

Sociology.

The distributed computation which determines social norms involves a heavy element of credit assignment: identifying failure states and success states, determining which actions are responsible for those states and who is responsible, assigning blame and praise.

Biology.

Evolution can be thought of as a (relatively dumb) credit assignment algorithm.

Ethics.

Justice, fairness, contractualism, issues in utilitarianism.

Epistemology.

Bayesian updates are a credit assignment algorithm, intended to make high-quality hypotheses rise to the top.
Beyond the basics of Bayesianism, building good theories realistically involves identifying which concepts are responsible for successes and failures. This is credit assignment.

Another big area which I'll claim is "basically credit assignment" is artificial intelligence.

In the 1970s, John Holland kicked off the investigation of learning classifier systems. John Holland had recently invented the Genetic Algorithms paradigm, which applies an evolutionary parad...
LW Team Updates - November 2019 (Subscriptions & More)
by Ruby — 2019-11-08T02:39:29.498Z

This is the once-monthly updates post for LessWrong team activities and announcements.

Summary

This month we launched 1) a complete subscriptions overhaul, 2) bookmarks, 3) pingbacks (experimental opt-in only) [full announcement of those features here]. Soon we'll release our new editor, LessWrong Docs into beta. Lastly, we made a post explaining why we spent Q3 optimizing for karma and how that went.

Recent Features

Subscriptions Overhaul

At long last we have released our subscriptions overhaul. You can now subscribe to precisely what you want to see (users, posts, comments, events) and get notifications on-site and/or by email batched at a frequency of your choosing.

Subscription options for a post

See the announcement post for complete documentation.

Bookmarks

It is now possible to bookmark posts and access these on their own page and/or as a list at the top of your homepage. Use the triple-dot drop down menu to bookmark any post.

Appearance of the Bookmarks page

Use the gear icon in the Recommendations section of the homepage to configure. Full documentation here.

Pingbacks [Experimental Opt-In]

When a post has been linked to by another post on LessWrong, the pingback feature will list any referencing posts at the bottom of the post page. "Pingback" is equivalent to "cited by".

When pingbacks are enabled, they are displayed at the bottom of post pages. Currently not all historical URL formats are supported and so many pingback lists are incomplete. This will be fixed before full-release.

Pingbacks displayed at the bottom of a post

To view pingbacks, you must be opted into experimental features. This can be done in your account settings.

Upcoming Features

LessWrong Docs

We've been promising the new editor for a few months now and it's getting closer and closer to beta release. This very announcement was written in it.

Look of the new editor, LessWrong Docs

We hope the new editor will provide an overall improved experience from Draft-JS, but more importantly, it introduces many of the features people are used to from Google Docs such as collaborative editing and inline commenting.

Expect to hear about this once we've ironed out a few more bugs.

Tagging

We're working on a new tagging system for posts. The idea is simple yet requires careful design to ensure the relevance of tags and display them in useful ways.

I'm hoping the tagging system will provide a new way to acc...
To Be Decided #2
by Ian David Moss — 2019-11-08T02:35:42.084Z

TBD is a quarterly-ish newsletter about deploying knowledge for impact, learning at scale, and making more thoughtful choices for ourselves and our organizations. This is the second issue, which was originally published in June 2019. Enjoy! --Ian

An Introduction to Decision Modeling

Decision-making is life. Over time, our decisions carve an identity for ourselves and our organizations, and it is our decisions, more than anything else, that determine how we are remembered after we’re gone. Despite their importance, though, we barely pay attention to most of the decisions we make. Biology has programmed in us a powerful instinct to make decisions using our intuitions rather than our conscious selves whenever possible. There are good reasons for this; if we had to think about every little decision we made, we’d never get anything done. But complex decisions require us to compare the likelihood and desirability of many possible futures on multiple, disparate, and often conflicting criteria, something our intuitions just aren’t naturally equipped to do.

Thankfully, there is a better way. The secret to resolving complex, risky dilemmas with justified ease and confidence is to

model your decisions explicitly

. At its best, modeling our decisions can help us make the very human exercise of decision-making not only more likely to lead to the outcomes we want, but more instinctively satisfying as well.

(

Keep reading

)

What I've Been Reading

Most Funders Admit Their Own Evaluations Are Not Useful
I really wish that headline was an exaggeration, but it's not much of one. In 2015, the Center for Evaluation Innovation and the Center for Effective Philanthropy surveyed evaluation and program executives at 127 US and Canadian foundations with $10 million or more in annual giving. The resulting report, "Benchmarking Foundation Evaluation Practices," contains startling revelations about how little evaluation reports are used. Most remarkably, more than three-quarters of respondents reported that they have a hard time commissioning evaluations that yield meaningful insights for the field, grantees, or even their own colleagues!
(

Twitter thread

)

Our Cognitive Biases Can Tell Us a Lot About the Meaning of Life You probably know Daniel Kahneman's classic volume Thinking, Fast and Slow as a comprehensive catalogue...
[Team Update] Why we spent Q3 optimizing for karma
by Ruby — 2019-11-07T23:39:55.274Z

In Q3 of 2019, the LessWrong team picked the growth of a single metric as our only goal. For the duration of this quarter, the overwhelming consideration in our decision-making what would most increase the target metric.

Why target a simple, imperfect metric?

The LessWrong team pursues a mixture of overlapping long-term goals, e.g. building a place where people train and apply rationality, building a community and culture with good epistemics, and building technologies which drive intellectual progress on important problems.

It’s challenging to track progress on these goals. They’re broad, difficult to measure, we don’t have complete agreement on them, and they change very slowly providing an overall poor feedback loop. If there’s a robust measure of “intellectual progress” or “rationality skills learnt” which doesn’t break down when being optimized for, we haven’t figure them out yet. We’re generally left pursuing hard to detect things and relying on our models to know that we’re making progress.

Though this will probably continue to be the overall picture for us, we decided that for three months it would be a good exercise for us to attempt maximizing a metric. Given that it’s only three months, it seem relatively safe [1] to maximize a simple metric which doesn’t perfectly capture everything we care about. And doing so might have the following benefits:

It would test our ability to get concrete, visible results on purpose.
It would teach us to operate with a stronger empirical feedback loop.
It would test how easily we can drive raw growth [2], i.e. see what rate of growth we get for our effort.
The need to hit a clear target would introduce a degree or urgency and pressure we are typically lacking. This pressure might prompt some novel creativity.
Targeting a metric is what YC advises their startups and it seems worth following that school of thought and philosophy for a time (see excerpts below).

So we decided to pick a metric and optimize for it throughout Q3.

[1] We had approval for this plan from our BDFL/admin, Vaniver. For extra safety, we shared out plans with trusted-user Zvi and told him we'd undo anything on the site he thought was problematic. We ran the plan by others too, but stopped short of making a general announcement lest this confound the exercise.

[2] Historically the team has been hesitant to pursue growth strategies out of fear that we could gr...
New evidence on popular perception of "AI" risk
by eg — 2019-11-07T23:36:08.183Z

The Mozilla Foundation recently conducted a survey about AI perception. I participated after receiving an email invitation; I don't know if they recruited elsewhere. They just released a writeup of the results today, which can be found here.

Mozilla seems to take a broad view of "AI", with a lot of weight given to current and near-term issues and less to potential fate-of-humanity issues. This is, of course, a different perspective from many in the LW crowd.

The popular perception is nonetheless potentially relevant to AI safety because it may represent the perception of some AI capabilities researchers. It may also be relevant to predicting political intervention in AI research.

Highlights:

51k sample size, 67k including partial submissions (see pdf linked from writeup)
overall, respondents were optimistic (i.e. not especially worried about risks)
most optimistic demographics were 19-24 years old, male, South Americans
only 10% self-reported being "well-educated" about AI, only 4% unfamiliar with term "AI"
respondents generally interested in learning more about AI

From link:

"24% of respondents said AI will make our lives better. 41% of respondents think AI will make our lives both better and worse. Only 10% of respondents think AI will only make our lives worse"

" Men (27%) are almost twice as optimistic as women (14%) that AI will make our lives better. Nearly half of South Americans (46%) are optimistic that AI is going to make their world better, making them the most positive region in the world. And young people 19 - 24 years old (35%) were the most likely age group to say AI will make the world better. Only 5% of this age group said they thought AI would make the world worse. "

"Respondents aged 19 - 44 were most likely to be “very interested” in learning more about AI (52%).
Respondents 65 and over were most likely to say they are somewhat interested in learning more about AI (64%)
South Americans (61%) and Africans (63%) said they were most interested in learning more about AI."
Self policing for self doubt
by KatjaGrace — 2019-11-07T23:10:01.683Z

Sometimes it seems consequentially correct to do things that would also be good for you, if you were selfish. For instance, to save your money instead of giving it away this year, or to get yourself a really nice house that you expect will pay off pragmatically while also being delightful to live in.

Some people are hesitant to do such things, and prefer for instance to keep a habit of donating every year, or err toward sparse accommodation more than seems optimal on the object level. I think because if their behavior is indistinguishable from selfishness, it is hard for them to be sure themselves that they aren’t drifting into selfishness. Not that selfishness would be bad if the optimal behavior was in fact the selfish one, but the worry is that if a selfishness-identical conclusion will bring them great personal gains, then they will tend toward concluding it even if they should not have.

This all makes sense, but there is something about it that I don’t like. It seems good to be able be coherent and curious and strategic and to believe in yourself and what you are doing in ways that I think this is at odds with. For instance, under this kind of arrangement you don’t get to have a solid position on ‘is this house worth having?’. You have your object level reasoning, and then not even a meta-level reason to adjust it, but a meta-level reason to distrust your whole thinking process, which leaves you in the vague epistemic state of not allowed to have certain conclusions on the house at all, or allowed to have them but not act on them. And having views but not acting on them is a weird state, because you are knowingly doing what is worse for the broader world, out of misalignment with yourself. And all this is to fend off the possibility that your motives are actually bad, or will become bad. I kind of want to say, ‘if your motives are bad, maybe you should just go and do something bad instead of rigging up some complicated process to thwart yourself’, but presumably there is some complicated relationship between bad and good parts of you that are trying to negotiate some kind of arrangement here. And maybe that is the way it must be, for you to do good. But it sounds suffocating and enfeebling.

On my preferred way of living, you do notice if you seem too excited about living in a nice house. But if you think you might have ‘the wrong values’ you address tha...
Kansas City Dojo Meetup: 11-5-19
by Bae's Theorem — 2019-11-07T21:32:06.985Z

Kansas City Dojo Meetup 11-5-19
(names besides my own have been changed for privacy)

PART I: 6pm-6:30

There are three attendees total, including myself, and one is a newcomer and a stranger
to the Rationalist community. He found us on Meetup.com. I begin by explaining the Dojo proceedings to him: we each bring a problem/project we are working on, and the others at the table proceed to ask questions and then provide advice on that problem/project, with the backdrop of Rationality (logic, science, a Bayesian framework, etc).
I begin with myself as an example. I pull out my “Character Sheet” (commonly known in the Rationalist Community as a “Bug List”; a list of problems in our lives. I have reframed it in a more positive light; a list of features that the Best Version of Alex has). I pick on feature off this list: my desire to be good at synthesizing information. The regular attendee, one of our founding members (hereafter known as ‘Life Engineer’), begins by asking me to be more specific.
“What do you mean by synthesis?” he asks. I say “the ability to take multiple pieces of information, and use them to make deductions about reality.”

He asks a few more probing questions to help me better articulate what I’m talking
about. He mentions “SMART Goals” (Specific, Measurable, Attainable, Relevant, Time-bound”) as a good template to follow as inspiration. In an attempt to be more specific, I use two examples of different problems at my job: one problem involves social tension between myself and a coworker. Another problem involves repairing a corrupt operating system. Two drastically different situations, but they share a few things in common:

You need to examine the problem.

You need to be able to generate a hypothesis about a possible solution.

You need to be able to test your hypothesis.

We taboo “synthesis” and use “problem solving” instead, to see if it serves to articulate what I’m trying to accomplish. Life Engineer did a year of professional problem solving for his job. His advice is to learn how to take better measurements. Of course, knowing what to measure requires some domain-specific knowledge, supporting my prior belief that Rationalists need to read books from many domains in order to know what to measure in any given situation. The most actionable thing I get from this is that I need to renew my efforts into tackling my reading list.

This reminds

...
Literature on memetics?
by moridinamael — 2019-11-07T19:18:05.748Z

In certain circles, it's common to talk about memes as entities, in a metaphorical and handwaving way. But it seems to me that thinking rigorously and precisely about memes in the framework of "memes::genes, memeplexes::infectious agents, minds::hosts" actually has a lot of explanatory and predictive power. This kind of thing ought to be a tool in the rationalist toolkit. In the space of politics, for example, it can be very illuminating to strip away the content of an idea and analyze it purely as a replicator, or as a component of a larger replicator conferring a specific fitness advantage.

A cursory online search for existing literature on memetics reveals that the field is about as confused as you would expect. But perhaps I am going about my search in the wrong way. Is there another field that encompasses the kind of thing I am gesturing at here?
Randomness vs. Ignorance
by Rafael Harth — 2019-11-07T18:51:55.706Z

A distinction I don't see made often enough is between what I call randomness and ignorance. Roughly, every expression of uncertainty is either about "where in the universe am I?" or "what is the universe like?" (or both). The former is the domain of randomness, the latter of ignorance.

Suppose you roll a die. You know that you're in a situation where you've just rolled a die, and that, in roughly 1/6th of the situations where one has just rolled a die, the die will come up a three. Thus, your uncertainty about the die roll is random.

Suppose you're wondering whether or not an omnipotent and immortal being exists. Whatever the answer is, it is the same every time someone is asking this question. Thus there is no randomness involved, but you are ignorant of what the answer is (though you might have a hunch).

Often, your uncertainty will have components of both. Suppose someone hands you a die, you roll it five times, and every time you roll a three. You can probably guess that the die is biased. But why? One way to answer this question is that, in most of the situations where one is handed a die and rolls it five times and gets the same number five times, the die is biased. You are ignorant of exactly how often this is the case, though. It could be the case every 9 out of 10 times this happens, or perhaps every 8 out of 10 times. Now suppose you knew that it was 9 out of 10 times. Then it would still be random whether you are in one of the 9 cases, or in the tenth.

Next up: Reference Classes for Randomness
Units of Action
by ryan_b — 2019-11-07T17:47:13.141Z

Unit of action is a term I have been using internally to be more specific when thinking about groups of people. This post is for fleshing out and clarifying my thinking for myself, and seeing if it would be useful to anyone else. Also it feels like there really ought to be a term for this already, and I might be able to find more information if someone knows it.

Definition

A unit of action is a group that takes actions, as a group.

I take the word unit from the military, and also from unit of analysis, reflecting my belief that this is the correct level of analysis for big-picture problems. Action is in the colloquial sense of "did something on purpose."

To be more concrete, the kinds of things I am pointing at are like families, corporations, and government agencies. The kinds of things I am pointing away from are like race, class, gender, and religion.

Causal vs correlational

One way to see the dividing line is between groups that cause things to happen, and groups to whom things happen. Families take vacations; corporations launch products; government agencies sue people for breaking regulations. The examples I pointed away from are demographic - they don't do anything as a group, but races may be segregated, genders have different bathrooms, and classes get tax breaks.

Agent heuristic

Another heuristic is whether the group can plausibly be modeled as a single agent. Following game-theoretic intuition further, the unit of action can be broken down into other units of action the same way agents can be broken apart into multiple agents. We might model one firm as one agent when looking at the behavior of firms in competition, but when looking at the behavior of one firm model the different departments within the firm as different agents. Sometimes, as in the case of the military, these components are very explicit.

Perpetual Coordination

Units of action are stable, successful-at-some-rate coordinators. I have a vague intuition that we can think of them as coordination strategies which propagate in the same way organisms have reproduction strategies, but the analogy is hardly precise; reproduction propagates genetic information, and units of action mostly propagate relationships.

Miscellaneous thoughts

Hierarchy is usually sufficient, but not necessary, to indicate a unit of action.
There is no taxonomy; it is defined by actions, not org type.
If a given group stops being

...