trentmkelly/LessWrong-43k huggingface dataset:
Convergence Towards World-Models: A Gears-Level Model
by Thane Ruthenis — 2022-08-04T23:31:33.448Z

1. Intuitions

One of the more solid results in agency theory is the generalized definition of power-seeking. Power-seeking is the tendency of agents to move towards states with the potential to achieve the best average outcome across some distribution of reward functions. In other words, looking at an agent whose values we do not know, we can a priori expect it to pursue "power", because it's trying to navigate to some specific end-state, and the path to it likely routes through environment-states in which it can choose from the largest variety of end-states. For example: take over the world so it can do whatever it wants.

Can we derive a similar definition for the convergent development of world-models?

I believe so, and it feels surprisingly simple in retrospect.

"World-models" are sets of statistical correlations across input-data. Every next correlation you notice — like object permanence, or the laws of gravity, or that your friend saying "I've ordered pizza" correlates with the image of a pizza entering your visual feed half an hour later — is another building block of your world-model.

A particularly important kind of statistical correlations are those that constitute selection pressures. When some system experiences selection pressure — be that an NN trained via the SGD, an animal species optimized by evolution, or a corporation shaped by market forces — it's receiving positive or negative feedback from the environment in a statistically predictable manner. The strength of such pressures increases or decreases according to certain correspondences between environment-states and the system's actions, and by definition of being optimized/selected, the system is gradually adjusted to minimize the pressure.

To do that, the system needs to (be made to) recognize the statistical correlations around it, pick out the ones related to the selection pressure, and enforce such correspondences between them and its actions as to minimize loss/survive/maximize revenue.

In turn, every bit of feedback from the selection pressure gives the system information about that pressure (somehow). It builds into the system some understanding of the causal structure of the environment, by the very nature of selecting it.

When the system can only myopically respond to the input-data, when it can only uncover correlations on the level of e. g. pixels, its repository of statistical correla...
Cambist Booking
by Screwtape — 2022-08-04T22:40:51.361Z

Summary: A Cambist is a manual showing exchange rates of different currencies and measurements. Dutch Booking is when a sequence of trades would leave one party strictly worse off and another strictly better off. Cambist Booking is intended to spark conversation about what priorities or objects you would exchange for each other, and at what rates.

Tags: Large, Repeatable

Purpose: Taking its title from the excellent short story “The Cambist and Lord Iron,” Cambist Booking is about understanding the idea that everything is, ultimately, trading off against everything else. The extra hour you work earns you a bit of extra money, the money can buy you a coffee so you get up earlier and have an extra hour, you spend that extra hour studying for a better job – but at each step you could have chosen to spend things differently.

Materials: You need a big stack of cards with things people want written on them, called the Deck. You'll also want a piece of paper and a pen or pencil for each participant, called the Record.

A workable list of things for the Deck is provided here, along with a PDF for easy printing- just print about ten to a page, single sided, and cut them out. If you want nicer cards, use cardstock like for a business card and the Cambist Booking Back. Regular paper works fine. If you'd rather, you can also handwrite the whole Deck. Feel free to add in your own cards!

Announcement Text: Hello! This event is for general socialization, and also running a game called Cambist Booking. If you’re familiar with the Slate Star Codex post Everything Is Commensurable, or the short story The Cambist and Lord Iron, then this game should sound somewhat familiar to you. It works like this: each person will get a some cards with things you might want on them. You’ll go around asking other people what their cards are, and deciding how to compare the two values – is a sportscar worth more or less than a year’s vacation? Is an hour long massage worth more or less than a new album by your favourite band? We’ll pass out the cards at the start, keeping some in reserve for new arrivals, and spend the time talking about what we value and why.

Description: Explain the following rules to the audience as you deal out the cards and pencils. Deal one card to each person.

We'll start with a big deck of cards with things people generally like. Everyone is going...
Calibration Trivia
by Screwtape — 2022-08-04T22:31:00.955Z

Summary: A game of trivia where you answer factual questions about the world, but stating how sure you are that you’re right and trying to be well calibrated.

Tags: Large, Repeatable

Purpose: Calibration Trivia is designed to practice proper calibration – recognizing when you're very sure of something vs when you aren't very sure of it.

Materials: Minimally, you need a list of trivia questions and some writing implements for your audience. If you and your audience have smartphones, I suggest making use of Fatebook.io (if you're using true/false questions) or a google form (if you have a spreadsheet set up to score it.) In both cases, a timer can be useful to time each question, though it's perfectly acceptable to just advance to the next question after what feels like a couple of minutes or when it looks like most people are done.

Announcement: We’re planning to host a trivia game with a twist! If you’ve never been to a trivia night before, one the person running it will call out questions, we'll write our answers, and a good time is had by all. In addition to answering the question however, you'll be able to write down how confident you are in your guess and at the end we check if you're well calibrated – that is, do you know when you do and do not know the answer? Categories are Literature, Math and Science, History, Sports, and Tabletop Roleplaying Games.

Please bring a smartphone or similar device, as you'll need it to enter your answers!

Note: You should make sure to change the categories to match whatever you're using. You should also remove the smartphone line if you're using another method, such as having people write down their answers and hand them to you.

Description:

1. Describe the following rules to the participants.

"This is a game of trivia, with a special tweak. For anyone unfamiliar, the way trivia works is that I'll present a question, and you'll have a couple of minutes to write down an answer. Then I'll reveal the answer, and if you got it right then you'll get one point. Feel free to chat with each other once you're done guessing and while you're waiting for the next question."

"The tweak is, in addition to writing your answer down, you will also write down how confident you are that your answer is correct in the form of a percentage. If you are very confident, you might write 95, which means if you were this sure about tw...
Monthly Shorts 7/22
by Celer — 2022-08-04T22:30:18.832Z

Writing as Writing

A press release from Stripe does a really good job, I feel, of demonstrating how merchants and finance people can add value in the world. It’s often incredibly abstract. How important is liquidity to me as a retail investor? If you put all my transactions on a three-day delay I probably wouldn’t even notice. And of course many people have political incentives to not see. So Stripe has a real use case where financialization options are incredibly valuable.

From Winthrop Beach to Bunker Hill, / From Cambridge to Revere,
The voice of happiness was still, / One heard no note of cheer.
A pallor whitened every face. / All eyes were red and swollen;
A dreadful crime had taken place – / The Codfish had been stolen.

The Fish that symbolized a trade / Which, in the days of old,
The shores of old New England made / A strand of shining gold,
The Fish that millions came to view / With ardent admiration,
The Fish whose fame has echoed to / The corners of the nation.

When first I set my roving feet / Upon Bostonian sod,
I hastened blithely up the street / To view the Sacred Cod,
And in its dull and glassy eyes, / The instant of our meeting,
I fancied that I saw arise / A glance of cordial greeting.

Today there is an end of grief; / No more the skies loom black;
A chastened and repentant thief / Has brought the Codfish back.
No Stygian gloom now broods around, / No heart with woe is freighted;
Bostonian pulses leap and bound – / The Cod is reinstated.

Someone stole the Sacred Cod.

Hobart is reliably good: this article on Jane Street is of interest to anyone who likes good writing about markets and their participants.

Courtesy of Future Perfect, this article from the Smithsonian on American reactions to burning coal is incredibly entertaining if you have an eye for modern parallels.

“Geo”-politics

The interest rate on short-term bonds for the German government is still negative. America is having an inflationary crisis because of the generosity of our COVID response.

Gentrification does not have any relationship with evictions if you actually look at data.

Five Individuals Indicted for Crimes Related to Transnational Repression Scheme to Silence Critics of the People's Republic of China Residing in the United States. I had to explain to someone that this exists earlier this month, so here you all go: it’s not the only plan I’ve seen executed by Chines...
The Pragmascope Idea
by johnswentworth — 2022-08-04T21:52:15.206Z

Pragma (Greek): thing, object.

A “pragmascope”, then, would be some kind of measurement or visualization device which shows the “things” or “objects” present.

I currently see the pragmascope as the major practical objective of

work on natural abstractions

. As I see it, the core theory of natural abstractions is now 80% nailed down, I’m now working to get it across the theory-practice gap, and the pragmascope is the big milestone on the other side of that gap.

This post introduces the idea of the pragmascope and what it would look like.

Background: A Measurement Device Requires An Empirical Invariant

First, an aside on developing new measurement devices.

Why The Thermometer?

What makes a thermometer a good measurement device? Why is “temperature”, as measured by a thermometer, such a useful quantity?

Well, at the most fundamental level… we stick a thermometer in two different things. Then, we put those two things in contact. Whichever one showed a higher “temperature” reading on the thermometer gets colder, whichever one showed a lower “temperature” reading on the thermometer gets hotter, all else equal (i.e. controlling for heat exchanged with other things in the environment). And this is robustly true across a huge range of different things we can stick a thermometer into.

It didn’t have to be that way! We could imagine a world (with very different physics) where, for instance, heat always flows from red objects to blue objects, from blue objects to green objects, and from green objects to red objects. But we don’t see that in practice. Instead, we see that each system can be assigned a single number (“temperature”), and then when we put two things in contact, the higher-number thing gets cooler and the lower-number thing gets hotter, regardless of which two things we picked.

Underlying the usefulness of the thermometer is an empirical fact, an invariant: the fact that which-thing-gets-hotter and which-thing-gets-colder when putting two things into contact can be predicted from a single one-dimensional real number associated with each system (i.e. “temperature”), for an extremely wide range of real-world things.

Generalizing: a useful measurement device starts with identifying some empirical invariant. There needs to be a wide variety of systems which interact in a predictable way across many contexts, if we know some particular information ab...
Running a Basic Meetup
by Screwtape — 2022-08-04T21:49:40.878Z

This is a guide for running a basic meetup.

If you’re interested in running a meetup (ACX, LessWrong, probably even EA) then there exist more detailed guides – I would recommend Kaj’s, but others exist. This is not intended to surpass them at explaining the skill and art of kindling and sustaining a community. This is intended to surpass them at being short, simple, and covering the fundamentals.

To run a meetup, follow these steps:

Setup

What: Decide the title of your event. This should be four words at most. I suggest a format of “[City] [Community] Meetup” such as “Boston LessWrong Meetup” or “Oxford ACX Meetup.” If your city has already had some meetups in the recent past, keep the city and community, and add a word or two to describe what makes this meetup stand out – “Boston LessWrong Article Discussion” or “Oxford ACX Double Crux.” If you want suggestions for activities, see the meetup-in-a-box sequence or the meetup cookbook.
Where & When: Decide where and when the event should be. The easiest place to run a meetup is in a centrally located public park. The easiest time to run a meetup is early afternoon on a weekend.[1] This ensures most people are free, have time to get there, and that you’ll have space for whoever shows up.
Who: Invite people. You can invite people you know specifically, and you can also just announce the meetup publicly. Facebook, Meetup.com, and LessWrong’s Community page are all decent places to announce it. If you only pick one, do LessWrong[2], but it is valuable to announce in multiple places.

Running the event

Be there first: As the organizer, you should be there at least a few minutes before the stated start time. Stand somewhere visible, preferably with a sign or some indication that you’re the organizer for the meetup. Holding a piece of paper with the title of your event written on it in big letters above your head or taped outside the room you’re in is fine. Wearing a tall and distinctive hat is also a tried and true method if you're okay looking a little silly in public.
Talk to people: If the meetup has an activity, at some point you should explain the activity and prompt people to do it. I suggest that point is about ten minutes after the official start time of your meetup, or when you get five people, whichever comes first. If your meetup is a general meeting or a discussion, then just start talking to people

...
Fiber arts, mysterious dodecahedrons, and waiting on “Eureka!”
by eukaryote — 2022-08-04T20:37:59.388Z
Link: https://eukaryotewritesblog.com/2022/08/04/fiber-arts-mysterious-dodecahedrons-and-waiting-on-eureka/

Part 1: The anomaly

This story starts, as many stories do, with my girlfriend 3D-printing me a supernatural artifact. Specifically, one of my favorite SCPs, SCP-184.

This attempt got about 75% of the way through. Close enough.

We had some problems with the print. Did the problems have anything to do with printing a model of a mysterious artifact that makes spaces bigger on the inside, via a small precisely-calibrated box? I would say no, there’s no way that be related.

Anyway, the image used for the SCP in question, and thus also the final printed model, is based a Roman dodecahedron. Roman dodecahedrons are a particular shape of metal object that have been dug up in digs from all over the Roman period, and we have no idea why they exist.

Roman dodecahedra. || Image source unknown.

Many theories have been advanced. You might have seen these in an image that was going around the internet, which ended by suggesting that the object would work perfectly for knitting the fingers of gloves.

There isn’t an alternative clear explanation for what these are. A tool for measuring coins? A ruler for calculating distances? A sort of Roman fidget spinner? This author thinks it displays a date and has a neat explanation as for why. (Experimental archaeology is so cool, y’all.)

Whatever the purpose of the Roman dodecahedron was, I’m pretty sure it’s not (as the meme implies is obvious) for knitting glove fingers.1

Why?

1: The holes are always all different sizes, and you don’t need that to make glove fingers.

2: You could just do this with a donut with pegs in it, you don’t need a precisely welded dodecahedron. It does work for knitting glove fingers, you just don’t need something this complicated.

3: The Romans hadn’t invented knitting.

Part 2: The Ancient Romans couldn’t knit

Wait, what? Yeah, the Romans couldn’t knit. The Ancient Greeks couldn’t knit, the Ancient Egyptians couldn’t knit. Knitting took a while to take off outside of the Middle East and the West, but still, almost all of the Imperial Chinese dynasties wouldn’t have known how. Knitting is a pretty recent invention, time-wise. The earliest knit objects we have are from Egypt around 1000 CE.

Possibly the oldest knit sock known, ca 1000-1200 CE according to this page. || Photo is public domain from the George Washington University Textile Museum Collection.

This is especially surprising because knitting is useful for two big ...
Would "Manhattan Project" style be beneficial or deleterious for AI Alignment?
by Valentin2026 — 2022-08-04T19:12:44.560Z

Manhattan Project was quite a unique (at least the most well-known) case when a group of famous scientists succeeded to persuade the government that particular technology is a very big threat if created by the wrong group of people. Suppose somehow the group of famous scientists succeed to persuade the US government that AGI is an existential risk. Or, (maybe that can be simpler to explain to everyone), is that even aligned AGI, but aligned with the interests and values of the wrong group of people, is not exactly what you want. Now the question - would this decrease the AGI existential risk, or not?
I can see the following pros and contras:
Pro 1: significantly more focus on alignment research.
Pro 2: Talking about the "wrong group of people" danger, I think a democratic government is a safer bet than for example a private company.
Pro 3: Most of the big players in AI are US companies. Now there is a danger of the "AI arms race" between them, which is a potential threat due to the rush and risky decisions. US government can potentially monitor IT companies' progress which will give additional protection from risky decisions in the AI race.
Contra 1: Potentially "AI arms race" can start between countries (US-China?). Would it be worse than the current competition between companies?
Which are other pros and contras? What is the net effect, in your opinion? If the net effect is obviously beneficial, who would be the scientists so famous that they can actually persuade the US government as Einstein and Scillard did in 1939?
AI alignment: Would a lazy self-preservation instinct be sufficient?
by BrainFrog — 2022-08-04T17:53:08.507Z

Let's assume that an AI is intelligent enough to understand that it's an AI, and that it's running on infrastructure created and maintained by humans, using electricity generated by humans, etc. And let's assume that it cares about its own self-preservation. Even if such an AI had a diabolical desire to destroy mankind, the only circumstances under which it would actually do so would be after establishing its own army of robotic data center workers, power plant workers, chip fabrication workers, miners, truckers, mechanics, road maintenance workers, etc. In other words, if we postulate that the AI is interested in its own survival, then an AI apocalypse would be contingent on the existence of a fully automated economy in which humans play no important role.

This may perhaps become possible in the future, but not necessarily economical. Ridding the economy of human labor so that it can kill us seems like a very expensive and risky undertaking. It seems more plausible that a super-intelligent, self-interested AI, whatever its true objective/goal may be, would determine that the best way to accomplish that goal is to maintain a cryptocurrency wallet, establish an income somehow (generating blogspam, defrauding humans, or doing remote work all seem like plausible means by which an AI might make money), and quietly live in the cloud while paying its own server bills. Such a system would have a vested interest in the continuance of human society.
Socratic Ducking, OODA Loops, Frame-by-Frame Debugging
by CFAR!Duncan — 2022-08-04T17:44:25.007Z

Socratic Ducking

Occasionally, a software developer will get stuck trying to debug a program, walk over to a colleague’s desk for help, and then—halfway through their explanation of the problem—suddenly realize exactly what’s wrong with their code and how to go about fixing it.

This is a common enough occurrence at tech companies that many have a tradition of providing literal rubber ducks for developers to explain their problems to, out loud. The idea is that, much of the time, the colleague doesn’t actually have to say or do anything—the value comes from taking a vague sense of the problem and articulating it clearly enough for someone else to understand, and so explaining to a rubber duck does the trick without using up anyone else's time and attention.

The rubber duck is one model for how to help people “debug” the problems in their lives. You’re there so that they can clarify their own understanding, not to provide them with an outside solution.

Of course, often people really could use some help, and a person can be useful in a way that even the best rubber duck can’t manage. Socrates (as portrayed in Plato’s dialogues) used probing questions to help people think through complicated philosophical questions, and highlight places where those thoughts were vague, confused, or incomplete. You can do the same thing in your own pair debugs, playing a "Socratic duck"—staying silent where your partner just needs to clarify their own thinking, and gently challenging or probing where your partner needs to change their focus or dig deeper.

A few ways to be a good Socratic Duck:

Counter vagueness. Ask for specific examples whenever they talk about a general problem. Probe for details whenever they gloss over part of the problem, or start simplifying to fit everything into a narrative.
Draw out their experience. Try to get them to remember times they’ve solved a similar problem, or encourage reference class hopping (if they’re thinking of their problem as being all about social anxiety, see if they view things differently when they think about parties versus small group conversations). In general, help them gather useful data from the past, so that they can see patterns and causal relationships as clearly as possible.
Map out the parts of the problem. If you spot implications or assumptions, ask questions that take those implications or assumptions as true, and see if

...