trentmkelly/LessWrong-43k huggingface dataset:
Are limited-horizon agents a good heuristic for the off-switch problem?
by  — 2021-12-05T19:27:59.894Z

(This is my first post, sorry if this is covered elsewhere.)

Implicit in the problem of a superhuman AI's reward being misspecified and turning everything into paperclips is the fact that the agent is optimizing over e.g. "number of paperclips" without any particular time-bound, area-of-effect bound, or probability confidence bound. For example, imagine if a MuZero+++++ agent were given the reward function "maximize the expected amount of money in this bank account until 1 day from now, then maximize the probability of shutting yourself off", where e.g. "1 day from now" was set by a consensus of satellites and/or deep-space probes. The agent could do a lot of bad things via its misspecified reward, but at least the blast-radius seems like it'd probably be limited. This agent might cause a lot of damage, but it probably wouldn't end the world in pursuit of these objectives.

This is not a strategy for executing a Yudkowskian "pivotal act", which by necessity would need to have longer-lasting effects. So it wouldn't stop some other group from creating unfriendly AGI. But as a general AI safety heuristic to try and spread across AGI practitioners, "do this as much as possible for a fixed time and then shutoff" seems like a promising social intervention. We also know how to implement it today.

As far as specific risks from the pursuit of convergent instrumental goals: "capability gain" and "make sure you don't get shutoff" seem much less likely to be convergent instrumental goals when you are planning over a short fixed horizon. "Create hidden copies of yourself" is still possibly a deployable strategy, but there's no reason for those copies to pursue a reward beyond the time-bound described, so I'd hold out hope for us to find a patch. "Deception" is again possible in the short term, but given this reward function there's no clear reason to deceive beyond a fixed horizon.

More broadly, this is a result of my thinking about AI safety social heuristics/memes that could be spreadable/enforceable by centralized power structures (e.g. governments, companies, militaries). If others have thoughts about similar heuristics, I'd be very interested to hear them.

I'm assuming I'm not the first person to bring this up, so I'm wondering whether someone can point me to existing discussion on this sort of fixed-window reward. If it is novel in any sense, feedback extreme...
Explicit model visualizations
by Dorian Stern vukotic — 2021-12-05T18:37:25.737Z

What is this about?

I believe explicit model visualizations explain systems better, and if we had the tools to effectively work with them, they could improve the quality of debates and increase rationality in institutions.

In

Lies, Damn Lies and Fabricated Options

, the basic idea is that mental models of reality that people hold are often wrong. Not only that, it's hard to estimate just how wrong they are.

In

Prioritization Research for Wisdom

, the basic idea is that good things happen with wisdom, and we should find ways to be wiser. Wisdom is “the quality of having experience, knowledge, and good judgement”, which basically means “high quality mental models and the skill to use them to achieve the right goals”.

I believe that going beyond simple text explanations will help the creation and support of high quality mental models. We have the tools like rationality and the scientific method for their creation, but little infrastructure beyond linear text to effectively express, share and discuss them. I believe we can build that infrastructure in the form of software tools.

Such tools can contribute in 3 areas:

Useful discussions
Rationality in institutions
AI (maybe?)

Origin of (most) disagreements

Most (but not all) disagreements in politics and business are not due to core values misalignment, but rather an effect of mismatching mental models of how the world works.

The statement “prosperity is good” is not controversial, but saying “the best way to cause prosperity is capitalism” or “the best way to cause prosperity is communism” is extremely controversial, even though this is one of the most important question of the 20th century.

This is an infrastructure problem

(Even though

conflict theory

and misaligned incentives exist)

When we compare the two options, like communism and capitalism, we run a simulation using our mental model, and make predictions about how they will contribute to our core values like prosperity.

Simulations like : “Since people are incentivized to work harder under capitalism, they will be more prosperous” or “Since the greedy elites cannot hoard wealth, median prosperity will be higher under communism”.

Both of these internally make perfect sense, and both are possible. The real question in this example should be: “how does incentivizing hard work in a capitalist system compare to disincentivizing hoarding in a communist sy...
What would be the pros and cons of a government-backed, property-based cryptocurrency?
by Asgård — 2021-12-05T17:57:26.171Z

I'll begin with a theoretical demonstrative example.
The Bureau of Land Management (sorry for immediately Americanizing the question, to all the European readers out there, but this idea might not work well in Europe anyways, for reasons that will soon become clear.) manages ca. 250 million acres (1 million km2, and 2.5 acres to a hectare), a landmass approximately the size of France and Germany combined. The Bureau offers land leases regularly, for a multitude of different reasons. I vaguely remember them offering land for sale at times (which probably means it's never happened), but for this thought experiment, let's imagine that land sales are on the table.
Land prices in America can vary between $1k and $100k an acre (napkin math), depending on where you're looking. Most BLM land is "what homesteaders didn't want", which would probably put it in the lower bound of that spectrum, but let's say that most of it could sell for $5,000 an acre, and that BLM would "set aside" y million acres for this program.
BLM would then create x amount of this coin per acre, and sell an amount of it to investors based off of the amount of land they would be putting up for sale in the coming year/amount of land up for lease. The BLM would promise to accept the coin as legal tender for land sales and leases, putting a minimum USD value to the coin. Most land sales and leases that I know of are auction based, and they usually have a minimum bid amount, so there would be a link between the value of the coin and USD. If they made 10 coins per acre, and set that as a value, then an example auction would have a minimum bid of: 5 acres for sale, min. bid of $25,000 or 50 "coins". This gives the coin a minimum value, backed by a government agency.
Now say we jump ahead a few years, and cryptocurrency hype has driven up the price (government accepted crypto, fiat-backed to some degree), BLM is able to sell these coins for more than it costs them, there's more interest in BLM land sales, also driving up their revenue due to increased bidding on the auctions.
So, from BLM's perspective, why would/wouldn't this be a valuable path to take?

Also a relevant question, what are the theoretical pros and cons of this cryptocurrency over other existent cryptocurrencies?

(and is there a short word for cryptocurrency, it feels ridiculous writing it out every time)
As I write this all out, I se...
Interpreting Yudkowsky on Deep vs Shallow Knowledge
by adamShimi — 2021-12-05T17:32:26.532Z

Here is an exploration of what Eliezer Yudkowsky means when he writes about deep vs shallow patterns (although I’ll be using "knowledge" instead of "pattern" for reasons explained in the next section). Not about any specific pattern Yudkowsky is discussing, mind you, about what deep and shallow patterns are at all. In doing so, I don’t make any criticism of his ideas and instead focus on quoting him (seriously, this post is like 70% quotes) and interpreting him by finding the best explanation I can of his words (that still fit them, obviously). Still, there’s a risk that my interpretation misses some of his points and ideas— I’m building a lower-bound on his argument’s power that is as high as I can get, not an upper-bound. Also, I might just be completely wrong, in which case defer to Yudkowsky if he points out that I’m completely missing the point.

Thanks to Eliezer Yudkowsky, Steve Byrnes, John Wentworth, Connor Leahy, Richard Ngo, Kyle, Laria, Alex Turner, Daniel Kokotajlo and Logan Smith for helpful comments on a draft.

Back to the FOOM: Yudkowsky’s explanation

In

recent discussions

, Yudkowsky often talks about deep patterns and deep thinking. What he made clear in a comment on this draft is that he has been using the term “deep patterns” in two different ways:

What I’ll call deep knowledge, which is a form of human knowledge/theory as well as the related epistemic strategies. This is what I explore below.
What I’ll call deep cognition, which is the sort of deep patterns that Yudkowsky points out AGI would have. There’s a link and an analogy with the deep knowledge, but I don’t get it enough to write something convincing to me and Yudkowsky, so I’ll mostly avoid that topic in this post.

Focusing on deep knowledge then, Yudkowsky recently seems to ascribe his interlocutors’ failure to grasp his point to their inability to grasp different instances of deep knowledge.

(All quotes from Yudkowsky if not mentioned otherwise)

(From the

first discussion

with Richard Ngo)

In particular, just as I have a model of the Other Person's Beliefs in which they think alignment is easy because they don't know about difficulties I see as very deep and fundamental and hard to avoid, I also have a model in which people think "why not just build an AI which does X but not Y?" because they don't realize what X and Y have in common, which is something that draws deeply on having deep mo

...
Scott Alexander's "Ivermectin: Much More Than You Wanted To Know"
by Raemon — 2021-12-05T05:43:00.000Z
Link: https://astralcodexten.substack.com/p/ivermectin-much-more-than-you-wanted

This is a linkpost, created for the 2021 Review.

I know I’m two months late here. Everyone’s already made up their mind and moved on to other things.

But here’s my pitch: this is one of the most carefully-pored-over scientific issues of our time. Dozens of teams published studies saying ivermectin definitely worked. Then most scientists concluded it didn’t. What a great opportunity to exercise our study-analyzing muscles! To learn stuff about how science works which we can then apply to less well-traveled terrain! Sure, you read the articles saying that experts had concluded the studies were wrong. But did you really develop a gears-level understanding of what was going on? That’s what we have a chance to get here!
What have your romantic experiences with non-EAs/non-Rationalists been like?
by Randomized, Controlled — 2021-12-05T04:41:29.230Z

Epistemic status: trawlin' for anecdotes/commiserations; satiating-morbid-curiosity

I'm also interested in non-romantic relationships, but I'm particularly interested in hearing about long term relationships between folks within EA and/or LW and someone[s] outside it.

By outside of EA/Rat I don't just mean "not active in the communities". I mean: doesn't have the philosophical machinery of rationality [ethics/meta-ethics/epistemology/cog sci/etc] and doesn't seem inclined to pick it up.

Things that I'm particularly curious about:

how long these relationships lasted

how you feel about the relationship: in what ways was/is it successful or unsuccessful?

if your partner doesn't have a "rationalist frame", how you deal with that

any other random thoughts this elicits
Behavior Cloning is Miscalibrated
by leogao — 2021-12-05T01:36:01.802Z

Behavior cloning (BC) is, put simply, when you have a bunch of human expert demonstrations and you train your policy to maximize likelihood over the human expert demonstrations. It’s the simplest possible approach under the broader umbrella of Imitation Learning, which also includes more complicated things like Inverse Reinforcement Learning or Generative Adversarial Imitation Learning. Despite its simplicity, it’s a fairly strong baseline. In fact, prompting GPT-3 to act agent-y is essentially also BC, just rather than cloning on a specific task, you're cloning against all of the task demonstration-like data in the training set--but fundamentally, it's a scaled up version of the exact same thing. The problem with BC that leads to miscalibration is that the human demonstrator may know more or less than the model, which would result in the model systematically being over/underconfident for its own knowledge and abilities.

For instance, suppose the human demonstrator is more knowledgeable than the model at common sense: then, the human will ask questions about common sense much less frequently than the model should. However, with BC, the model will ask those questions at the exact same rate as the human, and then because now it has strictly less information than the human, it will have to marginalize over the possible values of the unobserved variables using its prior to be able to imitate the human’s actions. Factoring out the model’s prior over unobserved information, this is equivalent[1] to taking a guess at the remaining relevant info conditioned on all the other info it has (!!!), and then act as confidently as if it had actually observed that info, since that's how a human would act (since the human really observed that information outside of the episode, but our model has no way of knowing that). This is, needless to say, a really bad thing for safety; we want our models to ask us or otherwise seek out information whenever they don't know something, not randomly hallucinate facts.

In theory we could fix this by providing enough information in the context such that the human doesn’t know anything the model doesn’t also know. However, this is very impractical in the real world, because of how much information is implicit in many interactions. The worst thing about this, though, is that it fails silently. Even if we try our best to supply the model with all...
Privacy and Manipulation
by Raemon — 2021-12-05T00:39:18.484Z

Previously:

“Can you keep that confidential? How do you know?”

Parameters of Privacy

Norm Innovation and Theory of Mind

My parents taught me the norm of keeping my promises.

My vague societal culture taught me a norm of automatically treat certain types of information as private.

My vague rationalist culture taught me norms that include:

noticing when I'm confused
noticing when societal norms were inadequate to handle my situation
being very honest, and having strong theoretical underpinnings for handling situations where I felt it was inadvisable to be honest.

Eliezer's post about meta-honesty was one of the most influential posts I've read in the past few years, and among the posts that inspired the coordination frontier. I was impressed that Eliezer looked at ethical edgecases, and wasn't content to make a reasonable judgment call and declare himself done.

He went on to think through the ramifications of various policies, devise a potential new norm/protocol, examine reasons that protocol might work or not work. He noted considerations like [paraphrased] "It matters that the norm be simple enough that people can reliably understand and use it." Or, quoted directly: "This norm is too subtle for Twitter. It might be too subtle for us, too."

From this post, I derived a (not-quite-spelled-out) norm of "when you try to navigate the edge cases of your norms, try thinking through the underlying principles. But don't try to be too clever, and consider the ways your edge-case-handling may fail to scale."

With this in mind, I want to walk through one of the ethical dilemmas I faced that I reflected on, when writing Norm Innovation and Theory of Mind. This is more of an object-level post, primarily a followup to my Privacy Practices sequence. But it seemed like a useful illustrative example for the Coordination Frontier concept.

Privacy norms can be wielded as an obfuscating weapon

Sometimes, privacy is wielded as a tool to enable manipulation.

I’ve run into a couple people who exploited my good faith / willingness to keep things confidential, as part of an overall manipulative pattern. Unfortunately, I don't feel comfortable going too far into the details here (please don't speculate in the comments), which makes it a bit harder to sanity check.

"Manipulation" is a tricky to define. It's a blurry line between "communicating normally" and "communicating in a way th...
Covid Christmas
by jefftk — 2021-12-04T22:00:06.147Z

Every year, my extended family gets together for Christmas. Several
days of lots of people in the same place, eating, playing games,
making music, and generally spending time together. We skipped last
year, because of covid, and are thinking about what makes sense to do
this year.

If rates had stayed at about the same level they had been at since
September, I think we would have been happy to get together more or
less as normal. With everyone 5+ fully vaccinated, and most adults
boosted, the risk seemed relatively low. But rates
have gone up substantially here, especially in the last few days:

We know this isn't just a post-Thanksgiving data issue, because we
also see it in the sewage:

This is changing quickly, and in the 2-3 weeks between now and when
we'd be getting together there's a lot of time for rates to go up
further or not.

With this much uncertainty, I think it makes the most sense to plan by
preparing options. This is a bit similar to the thinking a month
ago on making dance
events safer, but it's a smaller group of people with much higher
trust.

I think air purifiers make a lot of sense here, and box-fan cubes give you a lot of
filtration cheaply and simply:

(I'm considering putting a second cube on top to get five more filters
per fan, which would also make it quieter.)

Each 12-pack
is $80 shipped, so the cube is $27 plus a box fan ($20, or you may
have one) and some tape. Per microcovid, one in each room cuts
risk by ~3/4. Opening windows is also a good idea if you have a
heating system that can keep up (my dad's house does; ours does not).

Since this is the same group of people getting together over a few
days, spending a lot of time together each visit, I think rapid tests
also make sense. They're down to $7/each,
and while they do have some false negatives, they probably cut
risk by ~3/4
if you do them at the beginning of each visit.

If people wore decent (surgical+) masks the whole time, microcovid has
this as cutting risk by ~7/8. If people took them off to snack, and
are snacking ~10% of the time, then it's ~4/5 instead. [1] I think
large group dinners with people packed very close together, are not a
great idea. (Our family usually sits close enough at these meals that
chairs are too wide we need to set some boards across chairs to make
benches.) If we decided not to wear masks most of t...
A Generalization of ROC AUC for Binary Classifiers
by Adam Scherlis — 2021-12-04T21:47:35.988Z
Link: https://adam.scherlis.com/2021/12/04/a-generalization-of-roc-auc-for-binary-classifiers/

Suppose you have a binary classifier. It looks at things and tries to guess whether they’re Dogs or Not Dogs.

More precisely, the classifier outputs a numeric score, which is higher for things it thinks are more likely to be Dogs.

There are a bunch of ways to assess how good the classifier is. Many of them, like false-positive rate and false-negative rate, start by forcing your classifier to output discrete predictions instead of scores:

Fix some threshold. Anything higher is a “predicted Dog”, anything lower is a “predicted Not Dog”.
See how often the classifier correctly predicts that Dogs are Dogs, and how often it correctly predicts that Not Dogs are Not Dogs.
Calculate some function of those numbers.

A lot of metrics — like F1 score — also assume a population with a particular ratio of Dogs to Not Dogs, which can be problematic in some applications.

The AUC metric doesn’t require a fixed threshold. Instead, it works as follows:

Select a random Dog and a random Not Dog.
Compare the score for the Dog to the score for the Not Dog.
Repeat steps 1-2 many times. AUC is the fraction of times the Dog scored higher.

Or rather, that’s one way to define it. The other way is to draw the ROC curve, which plots the relationship between true-positive rate (sensitivity) and false-positive rate (1-specificity) as the classification threshold is varied. AUC is the Area Under this Curve. That means it’s also the average sensitivity (averaged across every possible specificity), and the average specificity (averaged across sensitivities). If this is confusing, google [ROC AUC] for lots of explanations with more detail and nice pictures.

AUC is nice because of the threshold-independence, and because it’s invariant under strictly-monotonic rescaling of the classifier score. It also tells you about (an average of) the classifier’s performance in different threshold regimes.

Sometimes, though, you care more about some regimes than others. For example, maybe you’re okay with misclassifying 25% of Not Dogs as Dogs, but if you classify even 1% of Dogs as Not Dogs then it’s a total disaster. Equivalently, suppose you care more about low thresholds for Dogness score, or the high-sensitivity / low-specificity corner of the ROC curve.

As I recently figured out, you can generalize AUC to this case! Let’s call it N-AUC.

There are two ways to define N-AUC, just as with AUC. First way:

Select N random Do

...
Agents as P₂B Chain Reactions
by Daniel Kokotajlo — 2021-12-04T21:35:06.403Z

tl;dr: Sometimes planners successfully

P

2

B

, kicking off a self-sustaining chain reaction / feedback loop of better and better plans (made possible by better and better world-models, more and more resources, etc.) Whereas fire takes a concentration of heat as input and produces a greater concentration of heat as output, agents take a concentration of convergent instrumental resources (e.g. data, money, power) as input and produce a greater concentration as output.

Previously we described the Convergent-Instrumental-Goal-to-rule-them-all,

P2B: Plan to P2B Better

. The most common way to P2B is to plan to plan again, in the immediate future, but with some new relevant piece of data. For example, suppose I am walking somewhere. I look up from my phone to glance at the door. Why? Because without data about the location of the doorknob, I can’t guide my hands to open the door. This sort of thing doesn’t just happen on short timescales, though; humans (and all realistic agents, I’d wager) are

hierarchical planners

and medium and long-term plans usually involve acquiring new relevant data as well.

This image just talks about collecting good new data, but there are other important convergent instrumental goals too. Such as staying alive, propagating your goals to other agents, and acquiring money. One could modify the diagram to include these other feedback loops as well — more data and money and power being recruited by sensor-learner-planners to acquire more data and money and power. It’s not that all of these metrics will go up with every cycle around the loop; it’s that some goals/resources are “instrumentally convergent” in that they tend to show up frequently in most real-world P2B loops.

Agents, I say, are P2B chain reactions / P2B feedback loops. They are what happens when planners successfully P2B. Whereas fire is a chain reaction that inputs heat + fuel + oxygen and outputs more heat (and often more oxygen and fuel too, as it expands to envelop more such) agents are chain reactions that input sensor-learner-planners with some amount of data, knowledge, power, money, etc. and output more, better sensor-learner-planners with greater amounts of data, knowledge, power, money, etc.

(I don’t I think this definition fully captures our intuitive concept of agency. Rather, P2B chain reactions seem like a big deal, an important concept worth talking about, and close enoug...