trentmkelly/LessWrong-43k huggingface dataset:
Wisdom of the crowds?
by amarai — 2021-09-29T19:14:37.644Z

According to an aggregate of forecasts on metaculus, the probability of bitcoin going above 100k within the next 5 years is 60%.
Under which conditions is it rational to have my probability for some event match the (i) median of a large group of estimates (e.g. from metaculus)
or (ii) the average of their estimates? Or, (iii) is it merely rational to set (i) or (ii) as my prior?
If yes, to any of these possibilities, why? What is the justification? Could someone point me towards the literature on this question?
[Book review] Gödel, Escher, Bach: an in-depth explainer
by Sam Marks — 2021-09-29T19:03:20.234Z

[Thanks to Jacob Falkovich, Eric Neyman, and my LessWrong reviewer, among others, for their feedback on earlier drafts.]

Initially, Gödel, Escher, Bach comes across as a perplexingly well-regarded conspiracy theory text. But reading on, you come to see the magic: all of the conspiracies are actually true. Gödel numbering actually is just like RNA translation, and recursive transition networks really are similar to renormalization of elementary particles. Who knew? GEB author Douglas Hofstadter did, and he wrote a 700-page exploration of the ideas behind Gödel’s incompleteness theorem so that you could too.

GEB has two parts. Part I is an exposition of many interesting and deeply related ideas: formal systems like math and physics acquire meaning by modeling the world; recursion gives these systems power but also enables self-reference; and self-reference ultimately poses a serious problem for these systems. These ideas build to the statement and proof of Godel’s Incompleteness Theorem. Part II, roughly speaking, claims that the ideas of part I have something to do with artificial intelligence and the nature of consciousness.

This “book review” is really an in-depth explainer of the key ideas in GEB part I. That said, I’ll also briefly touch on part II at the end.

Before I start, let me tell you some things that won’t be in this review because you really can’t get them from anywhere but GEB itself.

First, this review will feature very few of Hofstadter’s actual words. The reason is simple: there’s way too many of them. In a previous draft of this review, I tried quoting out of GEB for a few simple things, but it would always turn out like “Hofstadter thinks humans are sometimes different than machines: [300 word quote that somehow essentially involves an analogy about how you think your wife wants you to turn off the TV, but she wants you to start a government-overthrowing revolution] (page 37).”

Second, this review will leave out the fascinating interconnections Hofstadter draws throughout the text. Gödel numbering really is just like RNA translation, I promise, but if you want to know why you’ll have to check out GEB from your local library, sorry.

And third, GEB is really idiosyncratic in a way no one can imitate. The book’s chapters are separated by entertaining Carrollian dialogues which illustrate key ideas that reappear later in the text, imitating the way...
Easily Top 20%
by Jacob Falkovich — 2021-09-29T18:24:06.281Z

Cross-posted from Putanumonit.

I’ve written a lot about

approaching dating

as a

cooperative game

: you and your potential partners against the

assholes

, the

algorithm

, the

politics of skewed ratios

, and the

sex-negative society

. If you’re a straight guy, you have to be rooting for the women you meet to win.

And yet,

most of the comments

I get treat the idea of cooperation as anathema, a fantasy available only to the GigaChads who monopolize the world’s tiny handful of generous women and are simultaneously deluded by them. They talk about “the research”, which mostly ends up being

that one worthless paper

on dark triad attractiveness. And when I give advice based not on p-hacked studies but on what worked in my own life,

they inform me

that I simply cannot fathom what it’s like for the “bottom 80%” of men doomed to eternal lonely suffering:

The issue with Jacob is that, percentage-wise, he’s easily in top 20%. White/Jewish, quite handsome, in his 30s, living in NYC, working in finance, with top 1% IQ, previous military experience, and a small celebrity status. Despite of all of this, the best he could secure is a “poly marriage”.

I never had the patience to argue with these commenters and I’m going to start blocking them for sheer tediousness. Those celibate men who declare themselves beyond redemption deserve

their safe spaces

, but Putanumonit will not be one. That’s not who I’m writing for.

I’m writing my blog, in large part, for younger me. For Jacob from a couple of weeks ago who hadn’t read some particular book yet, or for teenage Jacob confused about the basics of how people’s minds work. When I write about dating it’s for the Jacob of not-that-long-ago who found dating frustrating and difficult, someone whom only his grandma would assuredly anoint as “easily top 20%”. I wish that younger version of me would have had my current posts to read, sans the humorless losers in the comments posting their dreary screeds.

I want to tell you a bit more about that young man and his early experiences with women. These stories aren’t supposed to prove any particular point, mostly I just find them weird and amusing. But if you recognize yourself in them, perhaps you’ll know you came to the right blog.

White/Jewish

*clap*

I turn my head right and see the back of Neta’s head. We are sitting on chairs back-to-back, turning our heads on a whim at the sound of each clap. If we turn...
Sex-Positivity is Countercultural
by Jacob Falkovich — 2021-09-29T18:22:35.007Z

Cross-posted from Putanumonit.

A month ago I wrote an

off-the-cuff Twitter thread

about how most modern cultures that say they are sex positive really aren’t. It generated a huge response, probably putting into words what many people felt for a long time but hadn’t seen spelled out. I turned that thread into an

article on the sex negativity of mainstream culture in Quillette

. Go read that first, then come back here for some follow-up thoughts.

Fuckability

I’ve

written before

about people’s pursuit of fuckability, euphemized for Quillette as “sex-worthiness”, as opposed to actually fucking. I mostly focused on how it manifests in women, the prestigious degrees and impressive careers and yoga prowess and political activism that fail to conjure the relationships they believe they deserve. Since then I’ve heard from several women on the male version of this, like my friend whose ex-husband spent so much time working late and working out that he was always too exhausted to have sex.

It’s not that professional, intellectual, and physical achievements aren’t attractive for both men and women — they are! The problem happens when someone’s identity as a fuckable/dateable individual takes center stage. Then, any romantic rejection or even renegotiation becomes a threat to one’s whole laboriously-built self-image.

This is really exacerbated by the invasion of society and social media into the intimate space. Society is always judgmental, even when pretending to be supportive. And it can only judge on visible features and identity — fuckability — not on how someone makes you feel in private. Is he progressive enough? Is she thin enough? Will they post cringe and tag you in it? If you’re thinking about how a partner’s presence will affect your social media clout, you’re probably not really connecting.

And of course, if people are having bad sex then they’d naturally focus more on fuckability and less on fucking.

Bad So Often

Why is casual sex bad so often

?, asked a woman famous for her sexuality. It’s bad because it’s casual, came hundreds of responses, people don’t try.

It doesn’t have to be this way. People could put effort into making their partner ecstatic, and take pride in it. People could tell their partners ahead of time what they’re hoping would happen in bed and what they don’t want to happen — this conversation in itself can be arousing and intimacy-building. Is it rea...
A brief review of the reasons multi-objective RL could be important in AI Safety Research
by Ben Smith — 2021-09-29T17:09:56.728Z

By Ben Smith, Roland Pihlakas, and Robert Klassert

Thanks to Linda Linsefors, Alex Turner, Richard Ngo, Peter Vamplew, JJ Hepburn, Tan Zhi-Xuan, Remmelt Ellen, Kaj Sotala, Koen Holtman, and Søren Elverlin for their time and kind remarks in reviewing this essay. Thanks to the organisers of the AI Safety Camp for incubating this project from its inception and for connecting our team.

For the last 9 months, we have been investigating the case for a multi-objective approach to reinforcement learning in AI Safety. Based on our work so far, we’re moderately convinced that multi-objective reinforcement learning should be explored as a useful way to help us understand ways in which we can achieve safe superintelligence. We’re writing this post to explain why, to inform readers of the work we and our colleagues are doing in this area, and invite critical feedback about our approach and about multi-objective RL in general.

We were first attracted to the multi-objective space because human values are inherently multi-objective--in any number of frames: deontological, utilitarian, and virtue ethics; egotistical vs. moral objectives; maximizing life values including hedonistic pleasure, eudaemonic meaning, or the enjoyment of power and status. AGI systems aiming to solve for human values are likely to be multi-objective themselves, if not by explicit design, then multi-objective systems would emerge from learning about human preferences.

As a first pass at technical research in this area, we took a commonly-used example, the “BreakableBottles” problem, and showed that for low-impact AI, an agent could more quickly solve this toy problem if it uses a conservative but flexible trade-off between alignment and performance values, compared to using a thresholded alignment system to maximize a certain amount of alignment and only then maximizing on performance. Such tradeoffs will be critical for understanding the conflicts between more abstract human objectives a human-preference-maximizing AGI would encounter.

To send feedback you can (a) contribute to discussion by commenting on this forum post; (b)

send feedback anonymously

; or (c) directly send feedback to Ben (

benjsmith@gmail.com

), Roland (

roland@simplify.ee

), or Robert (

robertklassert@pm.me

).

What is multi-objective reinforcement learning?

In reinforcement learning, an agent learns which actions lead to reward, and...
Unsolved ML Safety Problems
by jsteinhardt — 2021-09-29T16:00:19.466Z

Cross-posted from the BAIR Blog and from my blog, Bounded Regret.

Along with researchers from Google Brain and OpenAI, we are releasing a paper on Unsolved Problems in ML Safety. Due to emerging safety challenges in ML, such as those introduced by recent large-scale models, we provide a new roadmap for ML Safety and refine the technical problems that the field needs to address. As a preview of the paper, in this post we consider a subset of the paper's directions, namely withstanding hazards ("Robustness"), identifying hazards ("Monitoring"), and steering ML systems ("Alignment").

Overview. In this post, we consider problems in Robustness, Monitoring, and Alignment. In the full paper, we also consider the problem of External Safety.

Robustness

Robustness research aims to build systems that are less vulnerable to extreme hazards and to adversarial threats. Two problems in robustness are robustness to long tails and robustness to adversarial examples.

Long Tails

Examples of long tail events. First row, left: an ambulance in front of a green light. First row, middle: birds on the road. First row, right: a reflection of a pedestrian. Bottom row, left: a group of people cosplaying. Bottom row, middle: a foggy road. Bottom row, right: a person partly occluded by a board on their back. (Source)

To operate in open-world high-stakes environments, ML systems will need to endure unusual events and tail risks. However, current ML systems are brittle in the face of real-world complexity, and they become unreliable when faced with novel situations like those above. To reduce the chance that ML systems choose a wrong course of action in environments dominated by rare events, models need to be unusually robust.

Adversarial Examples

Adversarial perturbations. An example of an input image altered by an adversarial perturbation. After the adversarial perturbation, the neural network makes a high confidence mistake. (Source)

Adversaries can easily manipulate vulnerabilities in ML systems and cause them to make mistakes. As shown above, carefully crafted small perturbations are enough to break ML systems. In the paper, we focus on this problem but also suggest that researchers consider more realistic settings, like when attackers can create perceptible images or when attack specifications are not known beforehand.

Monitoring

Monitoring research aims to create t...
Brain-inspired AGI and the "lifetime anchor"
by Steven Byrnes — 2021-09-29T13:09:44.141Z

Last year Ajeya Cotra published a

draft report on AI timelines

. (See also: summary and commentary by Holden Karnofsky,

podcast interview with Ajeya

.)

I commented at the time (

1

,

2

,

3

) in the form of skepticism about the usefulness of the "Genome Anchor" section of the report. Later I fleshed out those thoughts in my post Against Evolution as an Analogy for how Humans Will Create AGI, see especially the "genome=code" analogy table near the top.

In this post I want to talk about a different section of the report: the "Lifetime Anchor".

1. Assumptions for this post

Here are some assumptions. I don’t exactly believe them—let alone with 100% confidence—but for the purpose of this post let’s say I do. I’m not going to present any evidence for or against them here. Think of it as the

Jeff Hawkins perspective

or something.

ASSUMPTION 1:

There’s a “secret sauce” of human intelligence, and it looks like a learning algorithm (and associated inference algorithm).

ASSUMPTION 2:

It’s a fundamentally different learning algorithm from deep neural networks. I don’t just mean a different neural network architecture, regularizer, etc. I mean really different, like “involving probabilistic program inference algorithms” or whatever.

ASSUMPTION 3:

The algorithm is human-legible, but nobody knows how it works yet.

ASSUMPTION 4:

We'll eventually figure out this “secret sauce” and get Transformative AI (TAI). [Note added for clarification: To simplify the discussion, I'm assuming that when this is all happening, we don't

already

have TAI independently via some unrelated R&D path.]

If you think these assumptions are all absolutely 100% wrong, well, I guess you might not find this post very interesting.

To be clear, Ajeya pretty much explicitly rejected these assumptions when writing

her report

(cf. discussion of “algorithmic breakthroughs”

here

), so there's no surprise that I wind up disagreeing with what she wrote. Maybe I shouldn't even be using the word "disagree" in this post. Oh well; her report is still a good starting point / foil for present purposes.

2. Thesis and outline

I will argue that under those assumptions, once we understand that “secret sauce”, it’s plausible that we will then be <10 years away from optimized, tested, well-understood, widely-used, industrial-scale systems for training these models all the way to TAI.

I’ll also argue that training these models from scratch w...
Bayeswatch 10: Spyware
by lsusr — 2021-09-29T07:01:25.529Z

Alice worked for Facebook.

"Are you familiar with advertising fraud?" she said.

"Remind me," said her boss Bob.

"People write plugins that simulate clicks. This has gotten worse and worse over time," said Alice.

"I am aware. Are you implying there is something we can do about it?" said Bob.

"We have this new Bayesian model that creates a completely simulated Facebook trained on the clicktracker data. It's robust against adversarial data, which means it should perform better than a naïve clicktracker," said Alice.

"What about privacy concerns?" said Bob.

"The model doesn't output a real copy of Facebook. It outputs a fake Facebook with fake people. Ethical advertisers don't care about individuals. All they care is that the data is statistically accurate. Statistically, our simulation behaves the same as real Facebook (which is why the advertising numbers come out the same) but nobody in the fake Facebook corresponds to a real person on real Facebook. This lets us sell 100% of our data to our advertisers without compromising the privacy of any real people. Cool, huh?" said Alice.

"How hard will it be to plug the new system into our advertising pipeline?" said her boss.

"Everything's modular. It all uses the same data format. Give me a week to prepare and the actual transition could be done in an hour. It'll increase revenue by at least 5%," said Alice.

"Let's do this," said Bob, "We can always revert the change if things go horribly wrong."

Eve worked for the NSA.

"The Pareto Principle is nuts. Here we are, having built an entire surveillance state, and yet 80% of our data comes from hacking into Facebook's ad recommendation system," said Eve, "It's emasculating."

"Get back to work," said her boss.

Charlie worked for the CIA.

"You can't fake an entire online life history. If an agent goes undercover and they don't have 20 years of family photos on Facebook it's obvious to a terrorist organization that they're using a fabricated identity. The old identity fabrication techniques don't work anymore," said Charlie.

"What do you propose we do instead?" said his boss.

"Steal real identities," said Charlie, "Impersonate real people."

"I'll call the NSA," said his boss, "We'll use the identities of people they collect from their mass surveillance system."

"Your model says this guy named Dzhokhar Dachiev is alive, but actually he's dead," said Bob.

"What are you talking about?" said...
[Book Review] "Houdini on Magic" by Harry Houdini
by lsusr — 2021-09-29T02:37:29.954Z

Houdini on Magic is a collection of the writings of Harry Houdini. It mostly covers three things.

Houdini's greatest tricks. (Mostly escapes.)

Houdini's work discrediting spiritualists. Spiritualists are people who pretend to talk to the dead. These aren't well-meaning but over-zealous Tarot card readers. Spiritualists knowingly used magic tricks to make it seem like ghosts were moving tables and making noises.

Some magic tricks you can do yourself.

The do-it-yourself magic trick section is mediocre. If you want to read an introductory book on magic tricks the book to read is The Amateur Magician's Handbook by Henry Hay. YouTube is even better.

Was Houdini's work discrediting spiritualists important? Maybe? I'm not sure if they were actually important or if combating spiritualists was one of two[1] times in history magicians ever mattered.

I know what you're really reading this book review for.

Lockpicking

Houdini explains lockpicks and skeleton keys. His lockpicks look nothing like modern lockpicks (he uses no rakes) but I bet that's just because lock and lockpicking technology has advanced. Houdini's skeleton keys are obsolete. We have different tools to accomplish the same task.

Houdini claims crooks "resort to the quicker and easier method of forcing the doors with heavy chisels." I believe this. Lockpicking is slow and frustrating. The regulars I've seen at lockpicking meetup are all nerds. An obvious burglar showed up exactly once and it a funny experience. (We don't gatekeep!) We were happy to show him how to pick locks but he lacked the patience for it. Even security professionals avoid lockpicking when they can because lockpicking is so slow

Houdini doesn't have anything to say about the importance of using spring steel to make your lockpicks. Looking at his lockpicks, I think I can deduce why. Those picks were big and chunky, presumably because he used them to pick crude locks. Spring steel wasn't necessary for most (perhaps all) of his tools. Primitive manufacturing made their lockpicks worse but it also made their locks worse.

[T]he majority of locks used in England were of such a nature that at the present date they would be called "jokes."

According to the lockpicker who taught me, European locks have improved. However, the majority of home locks in the United States are easy to pick. That said, I would be astonished if average locks today are ...
What Heuristics Do You Use to Think About Alignment Topics?
by Logan Riggs — 2021-09-29T02:31:16.034Z

Or, what kind of hammers do find yourself whacking alignment topics ("nails") with? Alignment topics include problems (e.g. instrumental convergence, mesa-optimizers, goodhart's law), proposed solutions (e.g. quantilizers, debate, IRL), and whatever else get's brought up in this forum.

For example, I imagine Steven Byrnes would see an alignment problem and think about what algorithm or structure in the brain might solve it (please correct me if I'm wrong!).

I imagine Rohin Shah would see a proposed solution and ask how it intervenes on a threat model (also correct me if I'm wrong!).

Or, in general, asking for specific examples of the topic.

Related: including specific examples where you've used your heuristic would be appreciated. The same for full, gears-level model of when your heuristic is useful.
A review of Steven Pinker's new book on rationality
by Matthew Barnett — 2021-09-29T01:29:58.151Z

Steven Pinker's new book on rationality came out today. I figured someone on LessWrong would write a review for it, so I might as well be the one to do it.

Unlike Pinker's prior books, such as The Blank Slate and The Better Angels of Our Nature, this book lacks a straightforward empirical thesis. Instead, he mirrors the sequences by building a science of rationality and then tries to convince the reader that rationality is important, both personally and socially.

Unfortunately, long-time readers of LessWrong are unlikely to learn much from Pinker's new book; his content is too similar to the content in the sequences. An upside is that Pinker's treatment is more concise, and his style more closely resembles mainstream thought. Consequently, I'm tempted to recommend this book to people who might otherwise be turned away by Rationality: From A to Z.

He starts by asking a simple question: how come it seems like everyone is so irrational? Pointing to religion, conspiracy theorists, ghost-believers, anti-vaxxers, alternative medicine adherents, and postmodernists, Pinker makes a good case that there's a lot of irrationality in the world. On the other hand, he continues, shouldn't humans have evolved to be more rational? How could such persistent, widespread irrationality be so common in humans, if our survival impinges on our ability to reason?

Pinker provides a simple answer: humans are very rational animals, just not in every domain. In those domains on which our survival depended, such as finding and eating food, humans are much less clueless than you might have been lead to believe. Pinker provides the example of the San people of the Kalahari Desert in southern Africa, who, despite their mythological beliefs, are stunningly successful at hunting prey. He cites Louis Liebenberg, who documented how the San people use Bayesian reasoning to hunt, applying it to footprints and animal droppings in order to build an accurate picture of their natural world: a dry desert on which they have subsisted for over a hundred thousand years.

It's not hard to see this dual phenomenon of rationality and irrationality reflected in the modern day: many young Earth creationists believe that the moon's craters were literally planted by God to give the appearance of old age, but these same people rarely apply the same standards of reason to matters in their ordinary life.

Yet, as Pinker o...