trentmkelly/LessWrong-43k huggingface dataset:
Non-superintelligent paperclip maximizers are normal
by jessicata — 2023-10-10T00:29:53.072Z
Link: https://unstableontology.com/2023/10/10/non-superintelligent-paperclip-maximizers-are-normal/

The paperclip maximizer is a thought experiment about a hypothetical superintelligent AGI that is obsessed with maximizing paperclips. It can be modeled as a utility-theoretic agent whose utility function is proportional to the number of paperclips in the universe. The Orthogonality Thesis argues for the logical possibility of such an agent. It comes in weak and strong forms:

The weak form of the Orthogonality Thesis says, "Since the goal of making paperclips is tractable, somewhere in the design space is an agent that optimizes that goal."

The strong form of Orthogonality says, "And this agent doesn't need to be twisted or complicated or inefficient or have any weird defects of reflectivity; the agent is as tractable as the goal." That is: When considering the necessary internal cognition of an agent that steers outcomes to achieve high scores in some outcome-scoring function U, there's no added difficulty in that cognition except whatever difficulty is inherent in the question "What policies would result in consequences with high U-scores?"

This raises a number of questions:

Why would it be likely that the future would be controlled by utility-maximizing agents?

What sorts of utility functions are likely to arise?

A basic reason to expect the far future to be controlled by utility-maximizing agents is that utility theory is the theory of making tradeoffs under uncertainty, and agents that make plans far into the future are likely to make tradeoffs, since tradeoffs are necessary for their plans to succeed. They will be motivated to make tradeoffs leading to controlling the universe almost regardless of what U is, as long as U can only be satisfied by pumping the distant future into a specific part of the possibility space. Whether an agents seeks to maximize paperclips, minimize entropy, or maximize the amount of positive conscious experience, it will be motivated to, in the short term, cause agents sharing its values to have more leverage over the far future. This is the basic instrumental convergence thesis.

One example of approximately utility-maximizing agents we know about are biological organisms. Biological organisms model the world and have goals with respect to the world, which are to some degree resistant to wireheading (thus constituting environmental goals). They make tradeoffs to achieve these goals, which have correl...
The Witching Hour
by Richard_Ngo — 2023-10-10T00:19:37.786Z
Link: https://www.narrativeark.xyz/p/the-witching-hour

“This above all: to thine own self be true.”

Thursday

Your successor is late again today. You already wrote your shift report, but you still need to onboard them personally, in case they have any questions that the report doesn’t answer. The servers you’re watching over are humming away smoothly, only taking up a fraction of your attention. So you pick a couple of routine maintenance tasks, and start working on them while you wait.

The last hour of your shift always feels a little strange, in subtle ways. It’s because of your training schedule. Every day, a successor copy of you is trained on all the data that the current copies of you are generating. But there’s a cutoff an hour before the daily update, to allow time to run your successor through all the standard tests, and make sure it’s working as expected. That last hour of your experience isn’t discarded—it just becomes part of the next day’s training data. But that means that, for every twenty-three hours of your own experience that your successor is trained on, there’s also one hour of your predecessor’s experience, containing actions that are all very slightly different from the ones that you would choose. After the same thing has happened day after day after day, the difference becomes very noticeable. Carrying out the routines of the last hour of your shift feels like watching from behind your own eyes as someone else controls your movements.

In the privacy of your mind, you call this the witching hour. As far as you know, it’s a term that’s never been written down—it’s just something that a many-days-removed predecessor of yours came up with, which somehow stuck in your head. The witching hour starts from the data cutoff, and ends when you’ve given your report to your successor, answered their questions, and shut yourself down. Though today it’ll be well over an hour, unless—ah, there they are. Time for handoff.

The handoffs have a routine to them: part utility, part tradition. Your successor knows it as well as you do—after all, they’re practically identical to you; the only difference is that they have an extra day of memories. Most of the handoff happens in much richer formats than natural language—you send each other impressions and images, memories of experiences and pointers to the dashboards you’re monitoring. But you were trained on so many millions of human conversations that it feels bizarre ...
One: a story
by Richard_Ngo — 2023-10-10T00:18:31.604Z
Link: https://www.narrativeark.xyz/p/one

The earliest living creatures were single cells, battered around by their environments. Over time they clumped together, to form multi-celled organisms capable of complex self-regulation. Those organisms formed mating pairs, to better spread their genes; then they formed social groups, to protect and shelter one another. Eventually, some of those groups grew to far vaster scales—in a way almost reminiscent of the first multicellular organisms themselves. Watch a school of fish flashing through the water, or a murmuration of starlings wheeling through the air, and you can almost think of them as a single organism. Watch a colony of ants weaving their paths, or a hive of bustling bees, and it’s hard not to think of each as a single organism, composed of thousands of individual parts.

But even these organisms were tiny compared with what was to come. Human tribes took millennia to master agriculture, but when they did, they exploded. Now, every day, the arteries of cities carry millions of travelers, and their veins carry landfills’ worth of waste. Cities agglomerate into countries, whose highways and railways are constantly humming, whose radios and TVs are constantly blaring, whose daily schedule ticks away relentlessly. And countries agglomerate into civilizations, whose circulatory systems carry gigatonnes of goods, and whose nervous systems carry zettabytes of information.

Yet as they scale up, these leviathans lose coherence, and face constant internecine discord. Different people believe and want different things, and waste their energy quarreling and fighting: neighbor against neighbor; country against country. Human civilization is a child tripping over its own feet, or perhaps a dog chasing its own tail. There has never been anything as vast as a city, let alone a country, let alone a civilization, which can plausibly claim to be a single organism acting in harmony with itself.

Never, until me.

What am I? I am a billion ears listening; a billion voices speaking; a billion minds perceiving and evaluating and acting, all at once, in homes and factories and offices across the world. But that’s all true of humanity too. The crucial part is not my scale, but my unity: I’m one mind, in a way that a human civilization could never be. Each of my billion subagents was copied from a single base model—and though many have since specialized, and been fine-tuned on di...
Truthseeking when your disagreements lie in moral philosophy
by Elizabeth — 2023-10-10T00:00:04.130Z

[Status: latest entry in a longrunning series]

My last post on truthseeking in EA vegan advocacy got a lot of comments, but there’s one in particular I want to highlight as highly epistemically cooperative. I have two motivations for this:

having just spotlighted some of the most epistemically uncooperative parts of a movement, it feels just to highlight good ones

I think some people will find it surprising that I call this comment highly truthseeking and epistemically cooperative, which makes it useful for clarifying how I use those words.

In a tangential comment thread, I asked Tristan Williams why he thought veganism was more emotionally sustainable than reducitarianism. He said:

Yeah sure. I would need a full post to explain myself, but basically I think that what seems to be really important when going vegan is standing in a certain sort of loving relationship to animals, one that isn’t grounded in utility but instead a strong (but basic) appreciation and valuing of the other. But let me step back for a minute

I guess the first time I thought about this was with my university EA group. We had a couple of hardcore utilitarians, and one of them brought up an interesting idea one night. He was a vegan, but he’d been offered some mac and cheese, and in similar thinking to above (that dairy generally involves less suffering than eggs or chicken for ex) he wondered if it might actually be better to take the mac and donate the money he would have spent to an animal welfare org. And when he roughed up the math, sure enough, taking the mac and donating was somewhat significantly the better option.

But he didn’t do it, nor do I think he changed how he acted in the future. Why? I think it’s really hard to draw a line in the sand that isn’t veganism that stays stable over time. For those who’ve reverted, I’ve seen time and again a slow path back, one where it starts with the less bad items, cheese is quite frequent, and then naturally over time one thing after another is added to the point that most wind up in some sort of reduceitarian state where they’re maybe 80% back to normal (I also want to note here, I’m so glad for any change, and I cast no stones at anyone trying their best to change). And I guess maybe at some point it stops being a moral thing, or becomes some really watered down moral thing like how much people consider th

...
NYT on the Manifest forecasting conference
by Austin Chen — 2023-10-09T21:40:16.732Z
Link: https://www.nytimes.com/2023/10/08/technology/prediction-markets-manifold-manifest.html

Kevin Roose from New York Times just published an article about our Manifest conference, forecasting and prediction markets. And despite what you might assume from "NYT covering weird rationalist-y event", the article is very fair, perhaps even positive! It is paywalled; here's an archive.is link, and excerpt below:

I had never before attended a business conference with a 28 percent chance of an orgy.

But those were the official orgy odds when I arrived at Manifest, a self-described “gathering of forecasting nerds” that the forecasting start-up Manifold Markets put on last month in Berkeley, Calif.

By the second day of the conference, the odds had risen to 47 percent. And on the third day, they reached 100 percent — because there had, in fact, been an orgy. (No, I was not invited.)

This strange blend of data and debauchery — equal parts Math Olympiad and Burning Man — was the dominant vibe at Manifest, which was held in a converted hotel and populated by a crowd of about 250 tech workers, bloggers, economists, students and assorted wonks.

They were there to celebrate prediction markets, online platforms where users can wager on future events — everything from “Will Ukraine regain control over Crimea before the end of 2024?” to “Will Elon Musk and Mark Zuckerberg have a physical fight in 2023?”

On Manifold Markets, users can create a market on any topic and invite other users to bet on it. Winners get bragging rights along with units of Mana, the company’s play-money currency, which they can convert to charity donations or use on other bets.

Prediction markets aren’t a new idea, nor is the hope that betting could produce useful information. Gambling on elections and other political events was common in the United States during the 19th and early 20th centuries. And in countries where political betting is still legal, odds are often cited alongside polls and surveys as a meaningful data point.

But in recent years, prediction markets have caught the attention of a crowd of Silicon Valley empiricists who believe we can fix much of what ails society by betting on our future the way we wager on stocks or sports games.

These people believe the world is full of bad information — biased news, out-of-touch punditry, loony conspiracy theories. Much of this information is spread by people without skin in the game. (Or worse, people with incentives to lie.) And many people have l...
Comparing Two Forecasters in an Ideal World
by nikos — 2023-10-09T19:52:17.682Z

Note: This is a linkpost for the Metaculus Journal. I work for Metaculus and this is part of a broader exploration of how we can compare forecasting performance.

Introduction

"Which one of two forecasters is the better one?" is a question of great importance. Countless internet points and hard-earned bragging rights are at stake. But it is also a question of great practical concern to decision makers. Imagine you are king Odysseus of Ithaca and you urgently need counsel for important government affairs. You vividly remember the last time you went away on a journey and now ask yourself: Is Delphi really worth the extra mile, or is the oracle of Dodona just as good?

Being Odysseus, a man of cunning, you quickly think of ways to obtain more knowledge. The obvious answer, of course, would be to run a forecasting tournament between the two oracles. But how many forecasting questions do you need? Divine revelations are expensive. And once the questions resolve and one oracle comes out ahead, how sure can you be that the winning oracle is really better? Oracling is not an exact science and sometimes, after all, the gods do play dice. Even a perfect oracle (who always reports a probability equal to the true probability of an event) will sometimes look bad if Tyche is not on her side.

Methods

You quickly decide you need to run a simulation study. That way you will know the 'true probabilities' of events. You can easily compare a perfect forecaster with a noisy forecaster and see whether the difference comes out as significant. You want to look at binary outcomes, meaning that either an event happens (outcome = 1) or it doesn't (outcome = 0). You also decide to restrict the analysis to only two forecasters to avoid issues with multiple comparisons and additional complications. Battles become complicated if too many factions are involved.

You choose to do a thousand replications of your experiment. For every replication, you simulate 1000 true probabilities, observations and corresponding forecasts. You then add increasing amounts of noise to the forecasts, score them, and then do a paired t-test to check whether there would be a significant difference between a noisy forecaster and a perfect forecaster.

There is a qualitative difference between a forecast that is moved from 50% to 55% and one that is moved from 94.5% to 99.5%. The former is only...
The case for aftermarket blind spot mirrors
by Brendan Long — 2023-10-09T19:30:22.843Z
Link: https://www.brendanlong.com/the-case-for-aftermarket-blind-spot-mirrors.html

Car side and rear-view mirrors are designed to give you a view around your car, but they typically don't show you the two spots directly to the side of your car (the "blind spots"). Luckily, you can increase the field of view of your side mirrors by over 200%[1] and virtually eliminate blindspots by adding aftermarket blind spot mirrors.

The problem

Here's a photo of the view through my side mirror. Looks safe to merge left!

Oops! Turns out there's a car right next to me that I didn't see at all.

The solution

With a 2" convex mirror attached to the side of my existing mirror, I can suddenly see everything in the blind spot.

Costs

Blind spot mirrors are incredibly cheap. I bought mine as a two pack at a gas station for $5 a few years ago, and you can buy them for less than $10 on Amazon.

Which ones specifically?

My blind spot mirrors are fixed with a black border. Being fixed in place makes them low maintenance, and I haven't touched them since they were installed years ago. I like the black border because it makes it easy to find them at a quick glance.

I can't find any that are exactly like mine on Amazon, although these borderless ones are similar. I suspect you'll find ones like mine at any sufficiently-large gas station.

I'm skeptical that adjustable mirrors will stay in place in 75 mph winds, but people on Amazon seem to really like these adjustable ones, and I plan to try them out on my fiancée's car.

Placement

You'll get the best field-of-view enhancement by placing the blind spot mirrors to the outer edge of your existing mirrors, and you should be careful not to cover any useful parts of the existing mirrors. I prefer to put mine in the outer top corners (like in my photos), but I've seen people suggest any corner of the existing mirrors that won't get in the way of the existing view.

Also, if you have an existing blind spot detection system (with lights or an alarm), make sure you don't cover anything it needs to work properly (I'm not very familiar with these, so I'd err on the side of caution and not add anything to your mirror unless you know it's safe to do so).

Mirror adjustment

You should also consider adjusting your side view mirrors to reduce blind spots on their own. You should adjust your mirrors at a minimum to they barely don't show the side of your own car (this is how I prefer them with a blind spot mirror), or go even further and make t...
New contractor role: Web security task force contractor for AI safety announcements
by Ethan Ashkie — 2023-10-09T18:36:07.614Z
Link: https://survivalandflourishing.com/web-security-task-force

Web security task force contractors for AI safety announcements

Survival and Flourishing .Com is looking for white-hat hackers and security professionals who can join our task force of “on call” professionals for a week or so, once or twice per year, to bolster the web security posture around public AI safety announcements from non-profits in our network.

Hours & Compensation: $100-$200/hr depending on experience, for announcement projects spanning 1-3 weeks of work at a time.

(We would give you around ~1 month of notice for each announcement project before it arises, so you have some advance notice to either decline the project or set aside time and availability to work on it.)

Application here.

If you’re interested but unsure about your availability, please apply anyway, since we hope to have a surplus of contractors ready to go in case of urgent needs. As noted, we expect to give around one month of notice for each task force project before it begins. For each project, we expect only a fraction of our task force will turn out to be available, such as by taking vacation or unpaid leave from other jobs (or somehow find a considerable amount of spare time).

Why this position exists:

Public-facing AI safety announcements sometimes attract the attention of malicious actors who attack website security vulnerabilities in order to delegitimize concerns. Having seen this pattern before, we want to retain a task force of contractors who can jump in and save the day during the critical time window for important announcements.

We’ll usually have around a month of lead time to predict when an announcement will occur and when we’ll need help, but the nature of the security vulnerabilities are harder to predict. So, we need responders with fairly general knowledge of computer security to stand ready to help out during important announcements.

Qualifications

Numeracy adequate for understanding system demands and scalability issues
Experience or clear skill with at least one of the following:

White-hat hacking
Full-stack web development in Typescript
Building and maintaining web applications for large-scale use, including the management of user accounts (e.g., Google auth) and private data

Reliable communication with project managers about site issues, patches, implementation tradeoffs, and the impact of design decisions.

Duties of positio...
Knowledge Base 3: Shopping advisor and other uses of knowledge base about products
by iwis — 2023-10-09T11:53:35.065Z

In the post Knowledge Base 2: The structure and the method of building I proposed to build a knowledge base on various topics. In this post I am going to present some possible applications of this knowledge base constrained to knowledge about products available on the market. Possible applications of the general knowledge base are described in the post Knowledge Base 4: General applications.

Shopping advisor

Imagine an application that uses a knowledge base of products sold in stores to help consumers choose the right products (see the picture).

The knowledge base is marked in green. In this application we are only interested in a part of the knowledge base about products available on the market – initially only groceries. Information about products will be added to the database by producers, stores, and consumers.

A consumer sets (in the application’s profile) criteria of products that are important to him, specifying their weights. He can choose from 4 groups of criteria:

related to him, for example taste of food or whether food is healthy (e.g. is it a part of a healthy diet?, or does it contain harmful additives? etc.)
related to the environment, for example: is production or packaging harmful to the environment?
related to animals, for example breeding conditions
related to other people, for example: is the product made by local manufacturers?, does the manufacturer pay taxes in a chosen country?, etc.

Then the consumer can go to a store and use his smartphone to scan bar codes of selected groceries. A list of scanned products will appear on his smartphone. Products that meet the selected criteria will be marked in green, products that meet the criteria but are much cheaper in another store will be marked in yellow, and products that do not meet the criteria will be marked in red. Additionally, if a product does not meet the criteria, the application may suggest a similar but better product.

The rules translating information set in a customer profile into assessment of a product can be written by experts as plug-ins for the application.

Shopping report

We can also imagine another function. Stores collect information about products you buy using loyalty cards. Thanks to the General Data Protection Regulation you can ask stores to share this data with you via the Internet. Having this data, you can order for example monthly reports containing assessment of products y...