trentmkelly/LessWrong-43k huggingface dataset:
How should my timelines influence my career choice?
by Tom Lieberum — 2021-08-03T10:14:33.722Z

TL;DR: Recent progress in AI has tentatively shortened my expected timelines of human-level machine intelligence to something like 50% in the next 15 years. Conditional on that being a sensible timeline (feel free to disagree in the comments), how should that influence my career choice?
Context:
I am currently a master student in Artificial Intelligence at the University of Amsterdam with one more year to go (mainly doing my master's thesis). So far, my go-to plan was to apply for safety-relevant PhD positions, probably either in NN generalization or RL, and then try to become a research scientist in a safety-oriented org. Given the shorter timelines, I am now considering becoming an engineer instead since that seems to require much less upskilling time, compared to doing a PhD for 4-5 years. I think the answer to my question hinges upon

my personal fit for engineering vs. research
the marginal value of an engineer vs. researcher in the years directly preceding HLMI
the marginal value of an engineer now (i.e. a year from now) vs. a researcher in 5-6 years.

The reason I split these is that maybe the value changes significantly once HLMI is clearly on the horizon or already there in a number of relevant domains.

I feel like I enjoy research more than pure engineering, but it's not like I don't enjoy engineering at all. Engineering seems more competitive in terms of coding skills, which I might lack compared to the most skilled other applicants. However, that is something I could practice pretty straightforwardly.
How have other people thought about this question, and how would you judge the questions about marginal value of the two roles?
What was my mistake evaluating risk in this situation?
by Andrew Vlahos — 2021-08-03T02:54:35.887Z

At the start of the pandemic, I heard that Covid was likely to be a problem. I didn't pay much attention, because I remembered the Swine Flu "pandemic", and Covid was getting less attention than Swine Flu did at the time. I also knew that the media has a strong incentive to cause hype over things that aren't really dangerous trends. I am in an age range very unlikely to die from disease, and claims of a lung plague causing lasting brain problems sounded absurd.

I figured that if a disease about as deadly as the regular flu caused so much panic, and Covid caused less and mainly harmed people already at respiratory risk, I didn't need to worry about it.

I eventually updated my beliefs, but I don't see how I could have avoided making my mistake. How could I have done better?

Now that a bunch of answers are in, I will say what I think the mistakes were.

First of all, a clarification. I heard about it sometime in early February (I don't remember specifically since I wasn't paying much attention), and I re-examined the data and updated my belief in whether it was a problem at the beginning of March, when I noticed that even people who normally don't worry about things like this started considering a lockdown. Remember: I knew it was spreading, I thought that it wouldn't be much worse than regular diseases.

The mistake wasn't "I should have listened to the trustworthy sources instead of the untrustworthy ones". I didn't know much about which sources were the most reliable, and what I did know wouldn't have helped. For example, I know that where people put their own money is much more reliable than what they tell other people to do (why prediction markets are so much more accurate than other things). However, the US stock market reached an all time high on February 12, indicating that people whose whole job is paying attention to this stuff weren't worried at all.

The mistakes I did do were these:

I had been too overconfident in civilization's ability to withstand plague. Disease has been pretty much eradicated, and even if a new one appears there are over-the-counter cough medicine and fever reducers and things.

Approximately how I thought.

My mistake was not re-evaluating this belief based on things that almost became big. I should have updated based on Ebola and SARS but didn't.

2. Personal experience said it was even less dangerous than the flu. My elderly great-aunt test...
Is iterated amplification really more powerful than imitation?
by Chantiel — 2021-08-02T23:20:29.369Z

Here I propose the idea that creating AIs that perform human mimicry would result in similar capabilities and results to an AI made with iterated amplification. However, it may provide are greater degree of flexibility than using any hard-coded iterated amplification, which might make it preferable. I don't know if this has been brought up before, but I would be interested what people think or links to previous discussion.

The basic idea of iterated amplification is to create a slow, powerful reasoning system, then train a faster system that approximates this powerful power one. Then repeat this process indefinitely. The main way of coming up with the slow procedure I've seen proposed is to use HCH, which involves emulating a large number of humans interacting with each other to come to come up with an ideal output.

But suppose you instead had a bunch of human imitators without any fixed use of iterated amplification.

AI is valuable, so human researchers do AI research. Since human imitators do the same thing as humans, they could also create more powerful AI systems.

One way for them to do this is with explicit iterated amplification. If the humans mimics see this as the most effective way to increase their capabilities, then human mimics could just perform iterated amplification on their own by reading about it, rather than requiring it to be hard-coded.

However, iterated amplification is not necessarily the most efficient way to create a more powerful AI, and a human mimics would have the flexibility to choose other techniques. Current AI researchers don't usually try to increase AI capabilities by iterated amplifications, but instead by coming up with new algorithms. So perhaps for the human mimics, increasing capabilities using some method other than iterated amplification, like what actual AI researchers do, would be more effective.

For example, suppose the human-mimicking AIs see they could use a more powerful pathfinding algorithm than the messy one implicitly implemented in the learned model of a human mimic. Then they could just use their human-level intelligence to program a custom, efficient path-finding algorithm and then modify themselves to make use of it.

There are of course some safety concerns about making incorrect modifications. However, there are similar safety concerns when making an incorrect fast approximations to a slow, powerful reasoning ...
The Myth of the Myth of the Lone Genius
by rogersbacon — 2021-08-02T20:55:33.412Z

“Our species is the only creative species, and it has only one creative instrument, the individual mind and spirit of man. Nothing was ever created by two men. There are no good collaborations, whether in music, in art, in poetry, in mathematics, in philosophy. Once the miracle of creation has taken place, the group can build and extend it, but the group never invents anything. The preciousness lies in the lonely mind of a man.”

- John Steinbeck

“The Great Man theory of history may not be truly believable and Great Men not real but invented, but it may be true we need to believe the Great Man theory of history and would have to invent them if they were not real.”

- Gwern

The Myth of the Lone Genius is a bullshit cliche and we would do well to stop parroting it to young people like it is some deep insight into the nature of innovation. It typically goes something like this - the view that breakthroughs come from Eureka moments made by geniuses toiling away in solitude is inaccurate; in reality, most revolutionary ideas, inventions, innovation etc. come from lots of hard work, luck, and collaboration with others.

Here is a good description of the myth from

The Ape that Understood the Universe: How the Mind and Culture Evolve

by psychologist Steve Stewart-Williams.

”We routinely describe our species’ cultural achievements to lone-wolf geniuses – super-bright freaks of nature who invented science and technology for the rest of us. ... It's a myth because most ideas and most technologies come about not through Eureka moments of solitary geniuses but through the hard slog of large armies of individuals, each making—at best—a tiny step or two forward”

The problem here is that the myth of the lone genius is itself a myth. History (ancient and recent) is full of geniuses who came up with a revolutionary idea largely on their own - that’s why the archetype even exists in the first place (Aristotle, Newton, Darwin, Einstein to name the most obvious examples). The author of the above quote would seem to grant that at least some ideas and technologies come from eureka moments of solitary geniuses. Others would seem to go further - the author of an article entitled “

The Myth of the Genius Solitary Scientist is Dangerous

” holds up Einstein, Maxwell, and Newton as examples of this archetype, but then exposes the falsehood of these examples by saying:

“Newton looke

...
How Many Affordable Units?
by jefftk — 2021-08-02T16:20:02.327Z

With

last week's post on
affordable housing I was wondering: if we did build out all the
Somerville lots zoned for four stories (MR4) to be seven stories of
affordable housing, how much housing would that be? I count 254 lots
zoned MR4 with a total area of 1.8M sqft. Built out at seven stories
and an average of 80% coverage that's 10M sqft of housing. Figure 10%
for stairs, elevators, and hallways, and it's really 9M for units.
Let's say you build:

bedrooms
portion of construction
unit size

2br
25%
850 sqft

3br
25%
1000 sqft

4br
25%
1100 sqft

5br
25%
1200 sqft

Then building all of this would give you:

bedrooms
number of units

2br
2230

3br
2230

4br
2230

5br
2230

These properties currently comprise just under 1k units, so this is
7.9k new units of affordable housing and 1k units converted to
affordable housing. This would bring Somerville from ~35k units to
~43k.

Comment via: facebook
What made the UK COVID-19 case count drop?
by ChristianKl — 2021-08-02T13:01:30.048Z

According to Google COVID-19 data, something made COVID-19 cases drop after July 21, 2021. To me that event is unexpected given that the measures where in the process of being lifted and I wouldn't expect that to coincide with citizens engaging in more risk avoiding behavior.

If there's a factor bringing cases down we don't know, understanding what the factor is could be very valuable. Does anybody have a good explanation for why the cases dropped?
Do Bayesians like Bayesian model Averaging?
by  — 2021-08-02T12:24:17.611Z

Pages 4-5 of the International Society for Bayesian Analysis Newsletter Vol. 5 No. 2 contains a satirical interview with Thomas Bayes. In a part of the interview, they appear to criticise the idea of model Averaging

MC: So what did you like most about Merlise Clyde's paper, Tom?
Bayes: Well, I thought it was really cleaver how she pretended that the important question was how to compute model averages. So nobody noticed that the real question is whether it makes any sense in the
first place.

What’s going on here? I thought Bayesians liked model averaging because it allows us to marginalise over the unknown model:

p(y|x,D)=∑ip(y|x,D,Mi) p(Mi|D)

where Mi represents the i-th model and D represents the data.
Referential Information
by Jack R — 2021-08-01T21:07:11.371Z

Summary: pointing out a concept and giving examples.

When you learn about a thing, I claim that there are generally two kinds of information you get:

Direct information: the direct, object-level information you get
Referential information: the information you get about other things in a similar reference class to the object-level thing

I think often the referential information value is substantial, and I tentatively suspect that people don’t account for it enough in their decisions. This post is just meant to point out referential information and the value of it.

Examples

Looking into the terms and conditions of a certain credit card, and how you go about setting it up

Direct info = information about this specific credit card
Referential info = information about how credit cards in general probably work

Specializing in chemistry

Direct info = chemistry knowledge
Referential info = what other STEM fields are probably like, how hard it is to become an expert in a field, how to go about becoming an expert in a field, how research within a field is conducted, how progress is generally made

Reading a paper from a field you know little about

Direct info = the specific stuff you read about
Referential info = some idea of what the frontier of the field looks like, the kinds of problems the field tackles, how the field tackles them

Talking in-depth about the details of a complicated but mostly unimportant social interaction with the other person involved

Direct info = what happened in that social interaction
Referential info = how other people work, how complicated social interactions can be, how useful digging into details about social interactions can be

Visiting another country or learning about a new culture

Direct info = learning about that country or culture
Referential info = learning how different a country/culture can be from your own

Learning a new language

Direct info = knowledge of how to read/write/speak the language
Referential info = what it is like and how difficult it might be to learn a new language
Who decides what should be worked on?
by Aglaya Ivanovna — 2021-08-01T17:26:48.349Z

I work at a tech startup, and part of my job is to talk to job candidates and answer their questions about the company. One of the questions I often get asked is, "How do prioritization decisions get made? Who decides what should be worked on and what shouldn't be? How are the goals of the company determined?"

On the one hand, this question is eminently reasonable - people's work has to be assigned somehow or other, and it's natural to be curious how that happens.

However, I always get a bit frustrated when trying to answer it, because I know that it is really not the right question to be asking. It's based on a flawed model of how and why companies succeed, the model hammered into their worldview by big institutions and the MBAs that write about them.

This model tells them to expect grand councils of grand leaders, looking into the future and charting a course through the competitive landscape, having civilized debates or furious shouting matches, eventually coalescing on a top-secret plan detailing a strategy which will determine the future of the company.

I know for sure that this model is far from the truth at my company:

The plans are made at the lower levels of the company, and basically rubber-stamped by senior leadership
We pretty much don't worry about what we're going to be doing a year from now (though we do consider the impact of our actions on the future)
Competitors are not the main focus
The resulting document is fairly boring, reflecting things we all already knew
The success or failure of the company is not impacted much at all by strategy documents

Furthermore, I believe (though I'm less sure about this) that these ideas generalize beyond just my company and represent a superior way of operating than the one most people intuitively think of.

To elaborate on each:

The plans are made at the lower levels of the company, and basically rubber-stamped by senior leadership

So, then, what is the point of senior leadership?

Joel Spolsky explains this better than I can:

When you’re trying to get a team all working in the same direction, we’ve seen that Command and Control management (military-style) and Econ 101 management (pay for performance) both fail pretty badly in high tech, knowledge- oriented teams. That leaves a technique that I’m going to have to call The Identity Method. The goal here is to manage by making people identify w

...
Forecasting Newsletter: July 2021
by NunoSempere — 2021-08-01T17:00:07.550Z

Highlights

Biatob is a new site to embed betting odds into one’s writing
Kalshi, a CFTC-regulated prediction market, launches in the US
Malta in trouble over betting and gambling fraud

Index

Prediction Markets & Forecasting Platforms
In The News
Blog Posts
Long Content

Sign up here or browse past newsletters here.

Prediction Markets & Forecasting Platforms

Metaculus

SimonM (a) kindly curated the top comments from Metaculus this past July. They are (a):

SimonM (a), alexrjl (a), alexrjl & discussion (a), alexrjl (a), and alexrjl (a) make good points on the the recent COVID-19 wave in the UK.
Uncle Jeff (a) and JonathanShi (a) don't think commercial animal farming is going anywhere.
NunoSempere (a) and elifland_ought (a) make the case for a higher prediction of a civil war in the US.
Charles (a) looks at the chances of a US-Russia war in the next 30 years
ege_erdil (a) looks at the base rates on inadvertent nuclear detonations
PeterWildeford (a) links to FiveThirtyEight's new Olympic medal tracker (a)
elifland_ought (a) has some suggestions for better metrics for predictors.
KnowName (a) explains why questions should close earlier

Round 2 of the Keep Virginia Safe Tournament (a) will begin in early August. It'll focus on Delta and other variants of concern, access to and rollout of the vaccine, and the safe reopening of schools in the fall.

Charles Dillon—a Rethink Priorities volunteer—created a Metaculus series on Open Philanthropy's donation volumes (a). Charles also wrote an examination of Metaculus' resolved AI predictions and their implications for AI timelines (a), which tentatively finds that the Metaculus community expected slightly more progress than actually occurred.

Polymarket

Polymarket had several prominent cryptocurrency prediction markets. Will Cardano support smart contracts on Mainnet by October 1st, 2021? (a) called Cardano developers out on missed deadlines (a) (secondary source (a)).

Will EIP-1559 be implemented on the Ethereum mainnet before August 5, 2021? (a) saw Polymarket pros beat Ethereum enthusiasts by more accurately calculating block times. Lance, an expert predictor market player, covers the topic here (a).

Polymarket also started their first tournament, the first round of which is currently ongoing. 32 participants each received $100, and face-off in a sudden-death tournament (a). Participants' profits can be followed on PolymarketWhale...
Destroying Insecurity and Boosting Confidence Through Your Interests and Values
by Olitness — 2021-08-01T14:34:49.025Z
Link: https://olitness.wordpress.com/2021/08/01/destroying-insecurity-and-boosting-confidence-through-your-interests-and-values/

I have hypothesis that diversifying your interests, values and desires makes you much less insecure and anxious while boosting your confidence.

Personal Example and How Idea Came to Being

In recent months I had huge increase in my confidence in all areas of my life. But I haven't got any better at them. But my social, business, intellectual and athletic confidence is booming like never before. Anxiety and overthinking is gone, too. I care less, I am more happy and I seem to accomplish everything with so much ease. At first I thought I am just getting older so I care less and am getting naturally more secure and confident. But I wasn't happy with that as full explanation. While I am sure there might be multiple factors at play, I think diversifying my interests, values and desires did the trick.

When I was starting my gym rat journey and saw all buff guys being more jacked and stronger than me, I felt insecure. When I got into business and at some business conference I saw everyone making more zeroes per month than I did per year, it made me feel insecure. When I started cooking and I saw and tasted dishes of other cooks, it made me feel insecure about my skills.

But then came tipping point. At some point, I became slightly above average in all of those things. Never the best, but good enough than whenever I came to my gym I was richest entrepreneur, best cook, best at BJJ, knew most about quantum physics etc. Even though I wasn't strongest and best looking in my gym, I was most confident because gym wasn't my single interest from which I derived my values and self-worth. I was looking at myself as a whole of all other skills and experiences. I stopped being too focused on one single domain. And this went for all other areas. I meet with multimillionaires but I am more confident in conversation than they are, because I don't perceive myself as being worse at business than them, but as being more fit, better at BJJ, better at physics, better cook etc. etc.

What also probably accounted for such increase of confidence could be added recognition of diverse range of my new (mediocre) skills. It is funny to be worst at my BJJ class, but then being recognised as best grappler among my buddies who never did it. Or being that guy who knows lot about physics and business at my BJJ gym, while being barely among average in both fields. Being average, if you are average ...
Torture vs Specks: Sadist version
by andrew sauer — 2021-07-31T23:33:42.224Z

Suppose that instead of the classic version of torture vs specks where the choice is between specks in the eyes of 3^^^3 people or one person tortured for 50 years, there are no specks but rather there are 3^^^3 people who just want the one guy to be tortured. (No particular reason, this just happens to be part of their utility function, which is not up for grabs) The preference of each is mild but somewhat stronger than the preference to not get a speck in one's eye. Is torture the right decision?

I am especially interested in hearing from people who answer differently in this situation than in the original situation.