trentmkelly/LessWrong-43k huggingface dataset:
AI Fire Alarm Scenarios
by PeterMcCluskey — 2021-12-28T02:20:34.637Z
Link: http://www.bayesianinvestor.com/blog/index.php/2021/12/23/ai-fire-alarm-scenarios/

I've been pondering whether we'll get any further
warnings
about when AI(s) will exceed human levels at general-purpose tasks, and
that doing so would entail enough risk that AI researchers ought to take
some precautions. I feel pretty uncertain about this.

I haven't even been able to make useful progress at clarifying what I
mean by that threshold of general intelligence.

As a weak substitute, I've brainstormed a bunch of scenarios describing
not-obviously-wrong ways in which people might notice, or fail to
notice, that AI is transforming the world.

I've given probabilities for each scenario, which I've pulled out of
my ass and don't plan to defend.

These probabilities add up to more than 100% - they're not mutually
exclusive. I'll estimate an additional 15% chance that we get a fire
alarm some other way, and a 25% chance that we get neither an alarm nor
an interesting story about what prevented the alarm.

What do I mean by "a fire alarm"? Roughly, something that causes 90%
of AI researchers to sound more worried than Brian
Christian
currently sounds.

Please don't infer too much from this post about the importance of fire
alarms. Katja has written an appropriately nuanced evaluation of fire
alarms - most people should read just the
conclusions.
Also, I'm not predicting whether an AI fire alarm would reduce the
risks associated with AI.

I'll start with some relatively mundane scenarios, and build up to
stranger ones.

Expert Opinion

New AI researchers and/or research teams come along every couple of
years with impressive enough new accomplishments that they get
recognized by most AI researchers as one of the top three or four most
promising sources of progress toward AI. The world gradually notices
that each of these new leaders express at least as much concern about
the risks of AI as prior researchers, and that they increasingly agree
that AI will pass human levels sometime in the 2030s.

By the late 2020s, there's a clear enough consensus among the
recognized AI experts that no major news organization is willing to deny
that AI represents an important source of near-term risk.

Probability: 15%.

Venture Capital

Venture capitalists have above average competence at noticing new
technologies, and they're relatively alert to low probability outcomes.

I find it easy to imagine that a couple of years before we get AGI,
we'll see VCs making multi-billion dollar investments i...
Don't Look Up (Film Review)
by Nicholas / Heather Kross — 2021-12-27T20:36:02.527Z
Link: https://www.thinkingmuchbetter.com/main/dont-look-up-film-review/

On my uni group's Discord, we have a running joke: "If the world ended tomorrow, it'd be out of the news cycle in a week". Adam McKay's Don't Look Up takes this joke completely literally, and the results are actually pretty funny.

The really quick version

The guy who directed The Big Short and Vice has just released a big-budget, star-studded Netflix movie about extinction risk, and it's a good movie, probably. Go watch it. Unless you hated The Big Short / Adam McKay's filmmaking style, then just substitute by reading this post.

Jennifer Lawrence plays an astronomer who discovers an asteroid comet headed for Earth. Everyone will die in about 6 months, unless somebody nukes the comet to make it go away.

Hilarity ensues because all the people in charge are clueless, narcissistic, frivolous, or all three. Literally "the world's ending, quick gotta take a selfie", the movie.

Much of this post will focus on ideas about extinction risk and how DLU introduces them. If you're concerned about spoilers, stop reading here, go watch the movie, and come back. DLU is kind of a "gateway drug" for understanding some of the key problems and concepts in real-life extinction-stopping.

Spoilers for Don't Look Up ahead!

Ideas DLU uses, explained

Extinction Risks (and handling them poorly)

That big idea of the movie, that of course humanity could bungle extinction-level threats, is relatable to those of us interested in preventing those threats. The movie especially comments on the U.S. government's shitty response to COVID-19.

Many steps in the COVID response are paralleled in the movie[1]:

Early coverage of COVID-19 was dismissive of the danger. Once the danger became obvious, news outlets changed their mind... and never apologized or reflected on their lack of concern. The same thing happen's with DLU's in-universe talk-show; the comet impact is initially framed as a quirky astronomy special-interest story.

Something objective and easy to agree on (virus bad, comet impact bad) turns into a polarized political tribal marker. COVID split society into pro-lockdown and anti-lockdown, mostly based on what party you vote for. The title Don't Look Up references the conflict between those who see the comet's danger ("Just Look Up") and denialists ("Don't Look Up").

Despite recent dangers, our response still didn't improve thanks to COVID. Big investments aren't going into pandemic pr

...
Ben Pace's Controversial Picks for the 2020 Review
by Ben Pace — 2021-12-27T18:25:30.417Z

This year, the LessWrong books had about 187,000 words in them. This was the top 59 posts last year in the review.

If we count up the posts in the early vote this year, then we get the top 43 posts. Basically, it’s everything up-to-and-including “Forecasting Thread: AI Timelines”. The spreadsheet where I did the math is here.

Now, we may do something fairly different with the results of the review this year. But for now I'm going to run with this as a "passed review" and "didn't pass review" watermark. Then in this post I'm going to make my case for 15 underrated posts in the review. (I encourage others to try this frame out for prioritizing which posts to review.)

Note that I'm about to defend my picks that were controversial within the LW crowd, which is a fun and weird optimization criteria. I'm not going to talk about the super defensible posts or the posts everyone here loved, but the posts many people don't share my impressions of, in the hope that people change their votes. Here goes.

Covid

First are my three Covid picks.

Covid-19: My Current Model was where I got most of my practical Covid updates. It so obvious now, but risk follows a power law (i.e. I should focus on reducing my riskiest 1 or 2 activities), surfaces are mostly harmless (this was when I stopped washing my packages), outdoor activity is relatively harmless (me and my housemates stopped avoiding people on the street around this time), and more. I give this +4.

A Significant Portion of COVID-19 Transmission is Presymptomatic also argued for something that is blindingly obvious now, but a real surprise to me at the time. Covid has an incubation period of up to 2 weeks at the extreme, where you can have no symptoms but still give it to people. This totally changed my threat model, where I didn't need to know if someone was symptomatic, but instead I had to calculate how much risk they took in the last 7-14 days. The author got this point out fast (March 14th). I give this +4.

Crisis and opportunity during coronavirus seemed cute to me at the time, and now I feel like an idiot for not realizing it more. My point here is "this post was really right in retrospect and I should've listened to it at the time". This post, combined with John's "Making Vaccine", have led me to believe I was in a position to create large amounts of vaccine during the pandemic, at least narrowly for my community, and (more...
A good rational fiction about IT-inspired magic system?
by qbolec — 2021-12-27T14:21:54.014Z

I'd love to read fantasy novel in which the system of magic would be inspired by the IT world:

Spells are nearly free, don't require any sacrifices, nor conservation of energy - that's not the issue. The issue is: to be able to super-precisely, down to the atom, describe what exactly the given spell is supposed to do. Sorcerers are needed, who can specify what exactly is meant by "turn a city to ashes". If you prefer, you can try to omit this step, but next thing you know is that your touch turned your whole family into gold statues, or the enemy's city turned into a plasma-filled hole sucking atmosphere with you, and/or paperclips. Spells are by necessity very complicated, multi-layer and modular, therefore mages spend years in libraries and conferences. Most importantly though, they need to cooperate to create anything substantial. Wizards, despite their absurdly high power, do not represent big threat to the rulers, because on their own, a mage can only use whatever they've learned and understood and it would be difficult for a single person to surprise an organized army of the ruler with something new, at least not consistently over and over again. Subordination of the wizards comes chiefly from bribing them, where implicitly the big part of the offer is that the employer is capable of motivating the other mages to work on the same enterprise - there is no natural, convergent goal which wizards would naturally gravitate towards, except perhaps a longing slowly developed over the years to describe something huge. Sitting in their libraries, watching battlefields and factories only telepathically through eyes of their gnomes, they lose more and more the grip on reality and what used to be means to a goal, became the end goal in itself: to precisely describe what's difficult to describe - and the question of it being good or bad, or even useful at all - doesn't matter that much anymore. Obviously, in a population this big, there are also mages, who dream of creating something, who have ambitious vision - the truth is, though, that if you train yourself for years in "HOW?" - not "WHAT?" - your ideas rarely turn out to be good at any other axis than being a spectacular use of your talent of description, and tend to flop, even if they temporarily gain interest of a group of wizards. Nonetheless, it happens - extremely rarely - that a mage, either thro...
Why did computer science get so galaxy-brained?
by Nicholas / Heather Kross — 2021-12-27T08:50:13.579Z

Not a joke question.

Was it early military funding? Then why didn't nuclear power become The Big Smart Field/Industry? Where's the Google of nuclear fusion? Why aren't a double-digits % of LW users nuclear physicists?

Was it Von Neumann and other smart scientists' early involvement? Then again, why didn't nuclear (or some obscure area of math besides CS) get big?

Was it universal applicability? Perhaps. (Then again, they used to think we'd have cars with fusion reactors in them... and why didn't steam power turn into a superweapon program?)

Cold War paranoia/global-impact/apocalyptic mindset? Rise of world population alerting people to scale? Availability of the Commodore 64 to children?

My main guess is "universalizability", but I'd also like to know about historical factors and/or other things about the structure/uniqueness-of-the-field, that may have caused this.
Conversation as path traversal
by Adam Zerner — 2021-12-27T08:41:31.187Z

Suppose Alice and Bob have a conversation. Alice begins by saing "Hi Bob, how are you?" And Bob responds by saying "I'm good. I saw the new Matrix movie yesterday. Have you seen the originals?"

Here is Alice initiating the conversation.

Here are all of the possible responses Bob could have gave. All of the different directions he could have taken things.

Here is Bob deciding to choose one of those paths to walk down.

By choosing to talk about the new Matrix movie, Bob chooses a path to go down.

Actually, the same thing happened with Alice when she started the conversation off by saying "Hi Bob, how are you?" There were many possible paths she could have walked down. She chose to walk down the "Hi Bob, how are you?" path.

With this model in mind, I have a few thoughts.

Interrupting

I think interrupting can be explained really well with this model. One type of interrupting is when the listener thinks they see where things are headed, and wants to fast forward to that point.

For example, below Alice responds to Bob with some long-winded reply. Bob feels like he gets it pretty early on in the reply. He sees that Alice is trying to take them from A to B. But she's taking a really long time getting them there. Bob sees a faster path to B, so he interrupts to say "Hey, it looks like you're trying to take us to B. I see this other really quick route to B. Let's jump down it real quick. Is this where you wanted to take me? Was there anything I skipped that you wanted to address?"

In this diagram, when Bob interrupted, he took them pretty close to where Alice was trying to take them, but he wasn't quite on point, so once he finishes, Alice can course correct and take them where she initially intended.

I am a fan of this type of interruption. Well, sort of. 1) I think it makes sense to do, but something about interrupting feels intrinsically icky to me, and I err pretty strongly against it. 2) I think a better norm is a norm against long winded replies in the first place. I like the idea of the speaker knowing that they can speak freely without being interrupted, but also having the responsibility to keep a rough eye on how much time they are using up and to pause periodically to see if the other person has something they want to say. That would give the other person the chance to say something like, "Yeah actually. I think I see where you're headed. Mind if I take a quick stab...
On Stateless Societies
by Martin Sustrik — 2021-12-27T07:28:42.315Z

Acemoglu & Robinson in their book The Narrow Corridor use Tiv people as an example of a stateless society.

The statelessness among Tiv was achieved by penalizing or eliminating those who grew successful or powerful enough to become an eventual condensation point for creation of a state. The traditional mechanism for doing so was accusing such people of witchcraft.

This, on one hand, left Tiv without the state and the associated oppression, but, on the other hand, kept them dirt-poor.

That sounds all right on the theoretical level, but what made me really visualize the process was a story about how Shaka Zulu, the builder of the Zulu Kingdom, broke the power of stateless elements in the traditional Zulu society.

When a hammerhead crane flew over his kraal and a porcupine wandered into it, when a crow perched on a fence and began to utter human words he summoned a team of witch doctors led by a woman called Nobela:

The Zulus were lined up and Nobela and here associates began "smelling out" the witches who had brought on the evil omens. They picked on prosperous people. One had grown rich through frugality. Another had put cattle manure on his lands as fertilizer, producing a bountiful harvest much greater than his neighbors'. Yet another was a fine stock breeder who had picked the best bulls and taken great care of his stock and as a result had seen a prodigious expansion of his herds.

Shaka took offense, accused the witch doctors of false accusations and demanded that two of them must die in compensation. The witch doctors panicked and asked Shaka for protection. He agreed, conditional on that they "won't cheat any more". That way he broke the power of the institution of witch doctors.

Another interesting data point about stateless societies is this paper by Lowes, Nunn, Robinson and Weigel (yes, the same Robinson as in The Narrow Corridor book).

In turns out that in Congo there is a set of tribes which are ethnically and culturally pretty homogeneous, except that while all of them are historically stateless, one tribe managed to establish a state. The researchers made the people from this area play different economic games and find out that people with a state are less cooperative than stateless people.

This is a single study and may eventually prove to be non-reproducible, but do have a look at the paper: The methodology is pretty rigorous and the experiment...
Reinforcement Learning Study Group
by Kay Kozaronek — 2021-12-26T23:11:34.985Z

Hey everyone,

my name is Kay. I'm new to the forum and I came here with a specific goal in mind:

I'm putting together a crew for a reinforcement learning study group.

Main objectives:

Mathematical Foundations: We will work through key passages in Sutton & Barto's Book to get a good foundation of RL
Research Papers: We will follow the "Spinning up in Deep Reinforcement Learning" by OpenAI resource to select key research papers in RL to read.
Code: The above resource can also be used to explore main RL algorithms and ideas which we'll aim to program from scratch in Pytorch/Tensorflow
Get Practical Skills: In the long term, the goal of this group is to prepare ourselves for work in AI Alignment and adjacent fields as engineers, researchers etc. (Work at DeepMind, OpenAI, MIRI, FHI, etc.)

Important Self-Selection:

It is important for me to be surrounded by ambitious and self-motivated people who are reliable, friendly, and helpful. I expect members of the study group to invest a good amount of time to study and even more to try to teach one another. Members who only plan on participating passively will be asked to become more active or leave the group.

Think of the study group as a sports team. The goal is to put the best people together on the field. Those who aren't ready to play will have to sit on the sideline.

Other helpful prerequisites are being able to program in Python 3, some knowledge of Probability and Statistics, as well as Linear Algebra and Calculus. In case this seems a little intimidating, don't worry, err on the side of joining the group. You'll pretty soon discover whether you're a good fit or not. Remember, we're all beginners who are united by the goal of becoming better.

Becoming our Coach:

In the above spirit, I'm also looking for a more advanced coach, someone who knows the terrain of reinforcement learning relatively well and who's willing to guide the teams' efforts towards a more fruitful, albeit challenging learning experience.

Anyone who's familiar with the main concepts in Sutton & Barto's Book would fit this role.

Schedule:

The study group will start as soon as possible in January. Once there is at least one other participant aside from me, we will find a suitable timetable and meet at least once a week to do pair-programming, code reviews, work through mathematical fo...
An interview with Danica Remy on protecting the Earth from asteroids.
by fowlertm — 2021-12-26T21:40:12.226Z

Danica Remy is the President of the B612 Foundation, an organization dedicated to protecting Earth from asteroid impacts, and we recently interviewed her on the Futurati Podcast.

She makes the provocative claim that asteroid impacts are the one major existential risk that we have a pretty good idea how to solve. Evidently, it won't be nearly as much of a technological or political lift as I'd originally thought.

Because this discussion contained a detailed discussion on one promising way of mitigating existential risk I thought it would be of interest here.

Show notes

Danica describes herself as a 'techno-optimist', meaning a person who believes very strongly that technological developments are going to have a positive impact on the future. Trent identifies himself as being techno-volatile (meaning he thinks that the impact of technology could be good or bad depending upon how it's used), and Danica largely agrees, clarifying that she's '80% an optimist'.
One thing that really drew Danica to the field of asteroid detection and mitigation is that it's one of the only existential risks we actually know how to solve.
However much ground might be left to cover in building asteroid-detecting probes, or coming up with ways of quickly and economically diverting asteroids on a collision course with Earth, we have a pretty good idea of what a solution would look like and the basic technology exists today.
What's more, mitigating the existential risk of asteroid impacts doesn't require that many people and so is a relatively small coordination problem.
Early in the life of the B612 Institute the focus was more on deflection, i.e. using something like an #enhanced gravity tractor or kinetic impactor to divert an asteroid off its current trajectory and onto another one. Over time this focus shifted to detection--the most plausible big step forward will likely be in asteroid mapping.
To that end, the Vera Rubin Observatory is being built and should go live in a couple of years. Danica likens it to 'Google Maps' for space, and as the project is planned to be open source, it should be straightforward for programmers, data analysts, and similar quantitative folks to contribute to charting the heavens.
Thomas jokingly asks whether or not there's an 'FBI most-wanted' list for asteroids. It turns out that such a thing already exists. The list is compiled from dat

...
San Francisco shares COVID data only when it's too late
by knite — 2021-12-26T20:12:59.001Z

In the early months of the pandemic, San Francisco posted case and death counts on a daily basis. It was sometimes messy with significant corrections, but the city did the best they could (or at least better than most) to keep the public informed.

Around half a year ago, they got worse. Data is now published with a 5 day lag. Presumably, the politicians and decision makers get timely data. Everyone else has to wait.

This is not okay. This is very not okay. In the post vaccination but pre-Omicron days, I was annoyed philosophically. But now. Now?

https://sf.gov/data/covid-19-cases-and-deaths

At this point, we're fucked and it's definitely over. But this is still not okay, and now I am furious. Anyone who wants to know the true state of the world cannot, unless they are On The Inside.

So, where is the pebble that starts an avalanche? Who do I need to Tweet or email or call to pressure the city into changing this?
The Debtor's Revolt
by Vaniver — 2021-12-26T19:32:32.980Z
Link: http://benjaminrosshoffman.com/the-debtors-revolt/

Ben Hoffman describes the transition from a old world order to a debtor-aristocracy world order.

Consider a standard microeconomics model, where profits are good and debts represent some loss of that profit in interest; individual actions are taken based on whether they will be return more than they cost (and the return is above the best publicly available return). People lend gold to governments to get gold in return, and that gold is valuable across the world regardless of government policy.

Consider also a debt-based model, where there are more winner-takes-all dynamics. There might be some new (economic) land opened up, and a competition to see who gets it; if money can be turned into increased ability to win that competition, then more access to debt means more chance of win. (Think Uber vs. Lyft.)

With the debt-based model, there's also more coalitional dynamics at play, as debt ties together borrower and lender; with the rest of their stake at risk, lenders are more likely to throw 'good money after bad', and with access to capital as a kingmaker, lenders may be more interested in propping up those that share their interests.

Of course, the main event is the transition from the gold-based economy with weaker states, to the cash-based economy with stronger states, and the main psychological event is the transition from an expectation of 'being in the clear' to an expectation of being heavily indebted. People used to talk about taking unsavory jobs to 'pay the mortgage', and now many talk about that to 'pay their student loan debt'.