trentmkelly/LessWrong-43k huggingface dataset:
What the future will look like
by avantika.mehra — 2021-11-15T05:14:15.362Z

The alarm rings — gentle, cascading sounds, increasing gradually in volume, to softly rouse you from your REM cycle. The sound radiates from micro-speakers distributed around your room; the 360° soundscape makes it sound like the walls are singing, beckoning you to take on a new day.

You wake up rejuvenated, and peel off the neuro-pellets from the side of your head, that quietly stimulated delta-waves in your brain through the stages of deepest sleep, accelerating cognitive regeneration while you explored a different world — one that hasn’t been touched by humans yet.

“Aliza, play today’s Robinhood Snacks podcast,” you tell your voice-enabled smart-home device, one that you built yourself in minutes from a DIY-kit you found online, because you know better than to install big-tech listening devices in your own home.

The podcast continues to play as you enter the bathroom — the audio syncs across the house, switching on and off as you enter each room to conserve energy. Speaking of energy, you wonder how much you saved yesterday. You click into an app, and a giant dashboard tells you you hit this month’s goals — “Huzzah! Check your texts for a special reward.” Amazing, they’ve sent you $10 in cryptocurrency for being a good energy citizen.

Your phone knows what to do with rewards — you’ve trained it well to manage personal finances. 10% goes to your favourite social cause — AI Safety Research. You volunteer with a local nonprofit that is working to keep humans and machines safe from each other by aligning their goals, and you’re their top contributor. 50% is distributed across your investment portfolio with predetermined weighting, and you use the remaining 40% to invest in metaverse assets — last time you bought a tiny bit of land in Decentraland-Iceland, and it’s looking beautiful, so you buy a few more square-feet. In September, the developers ship property-owners special VR-headsets that click into your neuro-pellets, creating the most dazzling multisensory Northern Lights experience, transporting you to the 2020s when the lights were scintillant and unclouded by pollution and space-dust.

The podcast ends, and you add a small tip for the creators before going to make breakfast — people who don’t tip creators are so rude, you think. The fridge displays the recommended breakfast items — based on your caloric history, the weather, and your previous meals — but you...
Attempted Gears Analysis of AGI Intervention Discussion With Eliezer
by Zvi — 2021-11-15T03:50:01.141Z

Recently, a discussion of potential AGI interventions and potential futures was posted to LessWrong. The picture Eliezer presented was broadly consistent with my existing model of Eliezer’s model of reality, and most of it was also consistent with my own model of reality.

Those two models overlap a lot, but they are different and my model of Eliezer strongly yells at anyone who thinks they shouldn’t be different that Eliezer wrote a technically not infinite but rather very large number of words explaining that you need to think for real and evaluate such things for yourself. On that, our models definitely agree.

It seemed like a useful exercise to reread the transcript of Eliezer’s discussion, and explicitly write out the world model it seems to represent, so that’s what I’m going to do here.

Here are some components of Eliezer’s model, directly extracted from the conversation, rewritten to be third person. It’s mostly in conversation order but a few things got put in logical order.

Before publishing, I consulted with Rob Bensinger, who helped refine several statements to be closer to what Eliezer actually endorses. I explicitly note the changes where they involve new info, so it’s clear what is coming from the conversation and what is coming from elsewhere. In other places it caused me to clean up my wording, which isn’t noted. It’s worth pointing out that the corrections often pointed in the ‘less doom’ direction, both in explicit claims and in tone/implication, so chances are this comes off as generally implying more doom than is appropriate.

Nate rather than Eliezer, but offered as a preface with p~0.85: AGI is probably coming within 50 years. Rob notes that Eliezer may or may not agree with this timeline, and that it shortens if you condition on ‘unaligned’ and lengthens conditional on ‘aligned.’
By default this AGI will come from something similar to some of today’s ML paradigms. Think enormous inscrutable floating-point vectors.
AGI that isn’t aligned ends the world.
AGI that isn’t aligned carefully and on purpose isn’t aligned, period.
It may be possible to align an AGI carefully and have it not end the world.
Right now we don’t know how to do it at all, but in theory we might learn.
The default situation is an AGI system arises that can be made more powerful by adding more compute, and there’s an extended period where it’s not aligned yet and if

...
An Emergency Fund for Effective Altruists (second version)
by bice — 2021-11-14T18:28:53.275Z

This is a rewrite of an old post of mine, incorporating ideas from Alicorn's earlier post on the same topic.

One of the things stopping people from donating a substantial part of their income to charity is the risk of a future financial downfall, which can be anything from unemployment to illness. Even in a rich country with a solid social safety net, the idea of having no financial buffer at all seems daunting. However, if effective altruists precommitted to helping other effective altruists in need, they could all donate more money, as the risk would be spread.

This sounds a lot like insurance, and effective altruists could indeed insure themselves against a myriad of risks. Unfortunately, insurers will reject claims whenever possible, many risks cannot be easily insured, and insurance requires monthly payment, which requires a stable income. In other words, insurance is no replacement for a personal financial buffer. Insurance is useful to mitigate specific high-cost risks, but that's it.

(Additionally, insurance introduces the overhead of an insurance company. When I asked for a ballpark estimate at the insurer I interned at, I was told that at most one-third of all premium money is put back into payment of legitimate claims.)

I propose an intermediary charity that acts as a shared emergency fund. If a donor spends $100 through this charity, this intermediary charity directly forwards $80 of incoming donations to charities of the donor's choice. The remaining $20 is put in an emergency fund. In situations where the donor would otherwise make use of their financial buffer, they can take up to $95 from the emergency fund.

Crucially, donors cannot recoup as much as they originally spent, so there is little opportunity for fraud. As such, recoupments would not need to be subjected to the same thorough vetting as insurance claims.

The example above works under the assumption that, on average, donors need 20% of their donations back. It might make sense to create different funds for donors with different risk profiles: if a subset of donors think there's a 50% chance they'd need donated money back, they could donate to a fund which puts half of their donation in the emergency fund.

Even so, I would not recommend relying entirely on the emergency fund as a replacement for a personal financial buffer, because it can be drained in times of national and global crisis. Ho...
Televised sports exist to gamble with testosterone levels using prediction skill
by Lucent — 2021-11-14T18:24:02.756Z

Testosterone levels go up and down based on success and failure in conflicts. It indicates the likelihood you’ll win the next conflict and determines your propensity to engage in further conflict. Win a lot, your levels go up, and you’re more likely to stand your ground or even initiate a conflict. Lose a lot, and that indicator will downregulate your behavior to prevent you from risking conflicts that have, historically, not gone well for you. It’s like a built-in, unconscious hint that regulates your desire to roll a die based on how well you’ve rolled in the past.

I feel this most viscerally when battling with phone support and getting a victory or concession a strict reading of the policy would not have entitled me to. The high of the win motivates me to make other calls I’ve been putting off, and I’m sure that extra motivation comes from this chemical being upregulated after success, indicating further success is possible.

What if there were a way to gamble testosterone without engaging in any conflict at all—if you could place some on the line in the hopes of getting a higher level through prediction rather than conflict? Watching sports allows such bets to be placed. First, because evolution is not stupid, you must associate yourself with one of the battling factions. You can’t make their victories yours without accepting their defeats. If you are to receive higher testosterone when your side wins, you must equally risk lower testosterone when they lose.

Sports affecting testosterone level is a well-known fact. I’m proposing that this is not an effect of recreational sports viewing, but the cause for it—to deliberately wager testosterone on one’s predictive abilities in substitution for actual combat. Further, wagers can be both safe or risky, with corresponding greater bets, like rooting for an underdog.
Improving on the Karma System
by Raelifin — 2021-11-14T18:01:30.049Z

TL;DR: I think it would improve LW to switch from the current karma system to one that distinguishes [social reward] from [quality of writing/thought] by adding a 5-star scale to posts and comments. I also explore other options.

Karma

Back in 2008, LessWrong was created by forking the codebase for Reddit. This was pretty successful, and served the community for a few years, but ultimately it became clear that Reddit's design wasn't a great fit for our community. In 2017 Habryka and others at (what is now) Lightcone rebuilt the site from scratch, improving not only the code and aesthetic, but also the basic design. The sequences were prioritized, short-form-posts and question-posts were invented, and the front-page was made into a curated collection of (ideally) high-quality writing.

But the primary mechanism that Reddit uses to promote content was kept: Karma.

Karma is a fairly easy system to understand: A post (or a comment, which I'll be lumping in with top-level posts for brevity) has a score that starts around zero. Users can upvote a post to increase its score, or downvote it to decrease the score. Posts with a high score are promoted to the attention of other users, both by algorithms that decide the order of posts (often taking into account other factors, such as the time it was posted), and by auto-minimizing low-scoring posts or simply by the social-proof of big-number-means-good drawing the eye.

The simplicity/familiarity of karma is one of its major selling points. As Raemon points out, it takes work to evaluate a new system, and the weirder that system is, the more suspicious you should be that it's a bad scheme. Voting is familiar enough that new users (especially in 2021!) will have no difficulty in understanding how to engage with the content on the site.

Giving positive/negative karma is also very low-effort, which is good in many circumstances! Low-effort engagement is a way to draw passive readers into participating in the community and feeling some investment in what they're reading. I may not be able/interested to write a five-page rebuttal to someone's sloppy argument, but I can sure as heck downvote it. Similarly, the opportunity to simply hit a button to approve of quality content can reduce a lot of noise of people patting each other on the back.

It feels good to get karma, despite the fake-internet-points thing. This encourages people to po...
[Linkpost] Paul Graham 101
by Gunnar_Zarncke — 2021-11-14T16:52:02.415Z

There’s probably no one who knows more about startups than Paul Graham. Having helped thousands of startups through Y Combinator, the startup accelerator he co-founded, there’s a thing or two to learn from his essays. And Graham’s wisdom isn’t limited to startups either; his essays, read by millions, touch on education, intelligence, writing, society, the human mind, and much more.

Paul Graham 101 by Jaakko

An alternative on the subject of startups is probably Jessica Livingston's Founders at Work. I don't have a third source so it doesn't qualify for The Best Textbooks on Every Subject.
My current uncertainties regarding AI, alignment, and the end of the world
by dominicq — 2021-11-14T14:08:50.956Z

As I read the interview with Eliezer Yudkowsky on AI alignment problems, I had a couple of thoughts of my own. These are poorly researched, and maybe poorly formulated. I intend to think more about them, but I thought this might be a good place to post them for feedback. I'm basically using this post as a large interactive bookmark for "hey, these are the things you thought about, think about them some more" with the added benefit of other people commenting.

I feel like there's a difference between "modeling" and "statistical recognition", in the sense that current (and near-future) AI systems currently don't necessarily model the world around them. I don't yet know if this actually is a difference or if I'm inventing a dichotomy that doesn't exist. Even if this is true, it's still unclear to me how or if it's better that current AI systems are statistical recognizers instead of world-modelers. You'd think that to destroy a world, you first need to have a model of it, but that may not be the case.

There may be a sense in which generating text and maneuvering the real world are very different. There may be a sense in which successfully imitating human speech without a "model" or agency is possible.

There may be strongly binding constraints on an agent's success in the world which do not depend on raw intelligence. Meaning, even if an agent has extremely high intelligence but lacks some other quality, its effective output in changing the world around it may not be as frightening as we might currently imagine. Imagine an extremely evil and extremely intelligent person who can effectively work one minute per week due to e.g. having no energy.

There may also be such strongly (or even more strongly) binding constraints that prevent even a superintelligent agent from achieving their goals, but which aren't "defects" in the agent itself, but in some constant in the universe. One such example is the speed of light. However intelligent you are, that's a physical constraint that you just can't surpass.

There may also be a sense in which AI systems would not self-improve further than required for what we want from them. Meaning, we may fulfill our needs (for which we design and produce AI systems) with a class of AI agents that stop receiving any sort of negative feedback at a certain level of proficiency or ability. I do not yet understand how training AI models works

...
Education on My Homeworld
by lsusr — 2021-11-14T10:16:33.408Z

I appreciate Eliezer Yudkowsky going public with where he came from. It has given me the courage to speak about the planet I'm from.

Sometimes people ask me how I know so much about such a wide range of topics. It flatters my ego to say the answer is self-study, but while that is technically true, the concept of "self-study" is out-of-place on my homeworld. The closest word we have to the concept literally translates to "deliberate play".

We have several forms of compulsory education. We call those places "gulags", "prisons" and "brainwashing camps". There is no concept of benevolent compulsory education. It would be considered a contradiction of terms, like compassionate torture or scaring a child into laughing. We do have education, but child rearing operates very differently from here on Earth.

Let us begin with the start of one's life. There are fewer caesarean deliveries. There is more breastfeeding. Women have a constitutionally-protected right to carry infants into offices wherever it would not place an undue hardship on her employer. Technically this was granted under the Equal Rights Amendment, but the custom had already been established for so long the amendment was just a formality.

Childcare, the raising of young children, is more hands-off, with a strong emphasis on the outdoors. Injury rates of children are commensurately higher. Pets too. My own dog fell off a cliff when I was a teenager. (The dog turned out alright in the end, but we evaced him just in case.) I could operate a band saw before I could multiply two-digit numbers in my head. Fewer children survive to adulthood than on Earth (I almost died in a motorcycle accident before I went to college) but those who do are more self-reliant. In order to become a man, boys in my hometown were expected to travel across the country with one day's pay and a backpack full of supplies, working for money or scavenging food along the way. You are allowed a bicycle, a motorcycle or a Greyhound pass. I choose the Greyhound pass. The hardcore kids choose "none" and go moose hunting in the tundra.

Adults lie to children all the time. I don't mean that adults conceal taboo truths from children. I mean they just make stuff up for the fun of it. It's like having a new Santa Claus, Tooth Fairy and Easter Bunny every day except the deceptions get more sophisticated the older you get and there's no sharp line betwe...
What would we do if alignment were futile?
by Grant Demaree — 2021-11-14T08:09:29.318Z

This piece, which predates ChatGPT, is no longer endorsed by its author.

Eliezer's recent discussion on AGI alignment is not optimistic.

I consider the present gameboard to look incredibly grim... We can hope there's a miracle that violates some aspect of my background model, and we can try to prepare for that unknown miracle

For this post, instead of debating Eliezer's model, I want to pretend it's true. Let's imagine we've all seen satisfactory evidence for the following:

AGI is likely to be developed soon*
Alignment is a Hard Problem. Current research is nowhere close to solving it, and this is unlikely to change by the time AGI is developed
Therefore, when AGI is first developed, it will only be possible to build misaligned AGI. We are heading for catastrophe

How we might respond

I don't think this is an unsolvable problem. In this scenario, there are two ways to avoid catastrophe: massively increase the pace of alignment research, and delay the deployment of AGI.

Massively increase the pace of alignment research via 20x more money

I wouldn't rely solely on this option. Lots of brilliant and well-funded people are already trying really hard! But I bet we can make up some time here. Let me pull some numbers out of my arse:

$100M per year is spent per year on alignment research worldwide (this is a guess, I don't know the actual number)
Our rate of research progress is proportional to the square root of our spending. That is, to double progress, you need to spend 4x as much**

Suppose we spent $2B a year. This would let us accomplish in 5 years what would otherwise have taken 22 years.

$2B a year isn't realistic today, but it's realistic in this scenario, where we've seen persuasive evidence Eliezer's model is true. If AI safety is the critical path for humanity's survival, I bet a skilled fundraiser can make it happen

Of course, skillfully administering the funds is its own issue...

Slow down AGI development

The problem, as I understand it:

Lots of groups, like DeepMind, OpenAI, Huawei, and the People's Liberation Army, are trying to build powerful AI systems
No one is very far ahead. For a number of reasons, it's likely to stay that way

We all have access to roughly the same computing power, within an OOM
We're all seeing the same events unfold in the real world, leading us to similar insights
Knowledge tends to proliferate among researchers. This is in part a natural te

...
A pharmaceutical stock pricing mystery
by DirectedEvolution — 2021-11-14T01:19:54.930Z

Epistemic status: case study based on a few hours' research. I own shares in a health sciences fund, but don't actively trade or own individual stocks in either of these companies. This is part of a larger project to understand the pharmaceutical industry, and isn't meant as investment advice.

Merck and Pfizer are two of the biggest pharmaceutical companies in the world. Each has been working on an antiviral pill for COVID-19. Merck's is called molnupiravir, and Pfizer's is Paxlovid; you can remember them by alliteration.

Blue is Pfizer, yellow is Merck. Look how beautifully the rise in Pfizer's stock price mirrors Merck's fall.

At 6:45 AM on Nov. 5th, Pfizer announced preliminary results of the phase III Paxlovid trial, showing that it has 89% efficacy. Their stock price jumped from $43.85 to $48.61. With 5.725 billion shares outstanding, that represents a $27 billion jump in their market cap.

Interestingly, that same day, Merck's stock price droped from $90.54 to $81.61, representing a $23 billion loss in market cap. Why?

Merck had already released preliminary results for molupiravir back on Oct. 1. Molnupiravir has an efficacy of 50%, while Paxlovid has 89% efficacy. So the market isn't just updating on the fact that Pfizer produced such a highly effective drug, but on the fact that that molnupiravir is second-rate.

Let's go back to Oct. 1. What did Merck and Pfizer's stock do that day?

Merck's jumped from $75.11 to $81.40, representing an increase in $16 billion in market cap. But this time, Pfizer's stock declined a mere 8 cents.

That's the mystery.

Let's assume that the market price for both Merck and Pfizer reflected the market's expectation for the absolute effectiveness of molnupiravir and Paxlovid and

demand

for the drug; and also thereby their relative effectiveness and

market share

.

If Merck's stock increase on Oct. 1 was due molnupiravir exceeding expectations for its effectiveness, then this should also have made the market increase its confidence that molnupiravir would turn out to be more effective than Paxlovid. This, therefore, should have translated into prediction that Merck would attain greater market share than previously predicted. And that should have translated into a decreased stock price for Pfizer. This is not what we see.

One explanation is that some factor unrelated to molnupiravir or Paxlovid cancelled out the expected negative effect of ...