trentmkelly/LessWrong-43k huggingface dataset:
World, mind, and learnability: A note on the metaphysical structure of the cosmos [& LLMs] 
by Bill Benzon — 2023-09-05T12:19:37.791Z

Cross-posted from New Savanna.

There is no a priori reason to believe that world has to be learnable. But if it were not, then we wouldn’t exist, nor would (most?) animals. The existing world, thus, is learnable. The human sensorium and motor system are necessarily adapted to that learnable structure, whatever it is.

I am, at least provisionally, calling that learnable structure the metaphysical structure of the world. Moreover, since humans did not arise de novo that metaphysical structure must necessarily extend through the animal kingdom and, who knows, plants as well.

“How”, you might ask, “does this metaphysical structure of the world differ from the world’s physical structure?” I will say, again provisionally, for I am just now making this up, that it is a matter of intension rather than extension. Extensionally the physical and the metaphysical are one and the same. But intensionally, they are different. We think about them in different terms. We ask different things of them. They have different conceptual affordances. The physical world is meaningless; it is simply there. It is in the metaphysical world that we seek meaning. [See my post, There is a fold in the fabric of reality. (Traditional) literary criticism is written on one side of it. I went around the bend years ago.]

A little dialog

Does this make sense, philosophically? How would I know?
I get it, you’re just making this up.
Right.
Hmmmm… How does this relate to that object-oriented ontology stuff you were so interested in a couple of years ago?
Interesting question. Why don’t you think about it and get back to me.
I mean, that metaphysical structure you’re talking about, it seems almost like a complex multidimensional tissue binding the world together. It has a whiff of a Latourian actor-network about it.
Hmmm… Set that aside for awhile. I want to go somewhere else.
Still on GPT-3, eh?
You got it.[1]

A little diagram: World, Text, and Mind

Text reflects this learnable, this metaphysical, structure, albeit at some remove:

Learning engines are learning the structure inherent in the text. But that learnable structure is not explicit in the language model created by the learning engine.

There are two things in play: 1) the fact that the text is learnable, and 2) that it is learnable by a statistical process. How are these two related?

If we already had an explicit ‘old school’ propositional model in compu...
Text Posts from the Kids Group: 2023 I
by jefftk — 2023-09-05T02:00:04.118Z

We have a

Facebook group
for kid stuff, because if we post a mixture of kid things and
other stuff FB's algorithm gets very confused about who to show our
posts to. While my annual

pictures posts mostly
cover the visual side, the text posts are only on FB and I don't like
that. So: here's the first ~half of 2023.

(Some of these were from me; some were from Julia. Ones saying "me"
could mean either of us.)

2023-01-02

Anna: I thought a blue heron was a bird with blue hair that was in?

2023-01-22

Lily: I've figured out that if you tell grown-ups something is
healthy, they're more likely to get it.

2023-01-22

Lily: [Confined to her room with covid] Could you refill my water
cup?

Me: Sure! [Gets cup]

[Fills cup. Starts doing something else.]

Lily: [Over walkie-talkie] I'm having trouble remembering where I put
my water cup, have you seen it?

Me: [trying not to laugh] Sorry, I forgot to bring it back up!

Lily: Your voice sounds funny, are you ok?

Me: I was trying not to laugh. Had you actually forgotten or were you
being polite?

Lily: Mostly being polite; did I do something funny?

Me: Yes, I mean no, I mean I didn't that approach was something you
knew how to do yet.

Lily: Thanks, I guess?

(Worrying when your 8yo is better at social stuff than you are.)

2023-01-23

Anna: dad, I'm really cold.

Me: how about a sweater?

Anna: I can't find any of my sweaters.

Me: have your looked in your drawer?

Anna: I don't want to go *upstairs*!

2023-02-05

Anna: Nora, should Lily... not be allowed to play in the fort?

Nora: ???

Anna: Is that true?

Nora: Yeah!

Anna: See Lily, you have to get out!

Lily: But Nora says yes to everything!

2023-02-05

Me: I'm worried you're going to jump on me in a way that hurts.

Anna: No, I'm only going to jump on the blanket

Me: Yes, but I'm under the blanket!

2023-02-04

Anna: I don't like it when someone wins and I'm not the person who
wins

2023-02-09

Things Nora is really into right now:

Balls, or other round things that could plausibly be considered
balls (M&Ms, the globe)

Shutting the dishwasher door

Animals that roar, especially lions, but also bears, tigers, and
other animals that she thinks might roar (monkeys, wombats,
cows). There's a house near us with concrete lion statues out
front, and she likes to go roar at them.

2023-01-08

Anna: In the ...
Action theory is not policy theory is not agent theory
by Cole Wyeth — 2023-09-05T01:38:27.175Z
Link: https://colewyeth.com/rationality.html

There are several proposed alternative theories of rationality. Some of them (causal decision theory (CDT), evidential decision theory (EDT), functional decision theory (FDT)) are referred to as decision theories, and there is endless debate over which decision theory is the best. Alternatively, there is the long prophesied, still-not-actually-discovered theory of embedded agency. Interestingly, everyone seems to pretty much agree that once this theory exists, it will be the actual really best theory of rationality, but regardless continues to argue about the best decision theory in the meantime.

I think that these theories of rationality really form a kind of ladder of increasing simplification. "Embedded agency" happens to be the lowest (say, deepest) rung on the ladder, making it the hardest to analyze. Causal decision theory is the top rung, making a lot of simplifications, and EDT is in some sense in between. This means that arguing over whether CDT or EDT is "better" is analogous to arguing about whether EDT or embedded agency is better - there are a lot of complications depending on what you want from your theory.

I will call the top rung simply "Action Theory"[1] because I believe that a theory of the value of specific actions (reasonably equivalent to CDT) is the furthest simplification.

Action Theory

Decision theory, with its origins dating back to arguably the 1700s, was invented by a host of mathematicians and economists, most notably von Neumann and Morgenstern (in the 1900s), to help humans win at gambling and make other choices. Because it's hard to mathematically formalize the problem "how should I human?" (or for that matter, "how can I build the most useful A.I.", though no one was thinking about that in the 1700s) this theory ignores the internal details of the decision maker, walls them off from the rest of the universe, and asks what decision it would be best for them to make. For that reason, the classical decision theory is an action theory. This theory has been incredibly influential throughout economics, finance, A.I., and casino design. I wouldn't even know how to get started quantifying its effects on government policy but it's certainly the basis for a mathematical justification of capitalism, an institution which has had some impact. All of reinforcement learning (RL) is based on action theory. The framework is sufficiently gene...
The purpose of the (Mosaic) law
by mruwnik — 2023-09-04T23:38:41.606Z

Epistemic status: an attempt to shoehorn 3000 years of thought into a different framework that works surprisingly well. A lot of obvious points to most people here

Approaches to Ethics

There are three basic approaches to ethics, each focusing on a different aspect:

Name	Focus	Example
Virtue Ethics	Integrity	Greek heros
Deontology	Rules	Kant
Consequentialism	Outcomes	“The ends justify the means”

Most schools of ethics are a mixture of the three, but you can find some examples of purish versions, e.g. Stoicism is a pretty pure form of Virtue Ethics, Kant’s categorical imperative an example of Deontology, and Utilitarism an example of Consequentialism.

Each of the approaches is useful and valid, but not always in the same contexts. Especially when taken to absurd limits. The obvious problem with virtue ethics is that it’s not very objective. Deontology has interesting failure modes with too rigorous adherence to rules, e.g. the common thought experiment with the Gestapo knocking at your door asking if you have hidden Jews (though check this for an interesting counterpoint), while utilitarism leads to the Repugnant Conclusion (with Scott Alexander’s commentary here).

Which is best?

In general, it seems like Consequentialism is the obviously correct one. After all, the whole point of ethics is coming up with a way of behaving that will result in good outcomes. The problem with that is that humans are generally not omniscient. Which makes it hard to work out what the consequences of any given action will be. It’s all very well that you want to end suffering, but if by feeding that hungry child you support the next Hitler, then there is a case to be made that you maybe shouldn’t. Robbing the rich to feed the poor is an interesting example. Stealing 1% of a billionare’s money won’t really impact the rich person’s quality of life, but that money could drastically improve the lives of many poor people. That being said, a world where it’s normalised to forcibly and unpredictably redistribute wealth seems like a worse one than the current one. Or at least a lot more chaotic, and so requiring a lot more investment in preventive measures. It feels like the result would not be a more equal society, rather a less trusting one, with large sums being put into stopping would-be Robbin Hoods from stealing things. Not to mention slippery-slopes, where the redistribution is from anyone better off ...
Against the Open Source / Closed Source Dichotomy: Regulated Source as a Model for Responsible AI Development
by alex.herwix — 2023-09-04T20:25:56.662Z
Link: https://forum.effectivealtruism.org/posts/GBAzW4pZ5JgJqGMJg/against-the-open-source-closed-source-dichotomy-regulated

Context

This is a short post to illustrate and start testing an idea that I have been playing around with in the last couple of days after listening to a recent 80,000 hours podcast with Mustafa Suleyman. In the podcast Suleyman as well as Rob Wiblin expressed concerns about Open Source AI development as a potential risk for our societies while implying that closed source development would be the only reasonable alternative. They have not delved deeper into the topic to examine their own assumptions about what makes reasonable alternatives in this context, or to look for possible alternatives beyond the "standard" open source/closed source dichotomy. With this post, I want to encourage our community to join me in the effort to reflect our own discourse and assumptions around responsible AI development to not fall into the trap of naively reifying existing categories, and develop new visions and models that are better able to address the upcoming challenges which we will be facing. As a first step, I explore the notion of Regulated Source as a model for responsible AI development.

Open Source vs. Closed Source AI Development

Currently, there are mainly two competing modes for AI development, namely, Open Source and Closed Source (see Table for comparison):

Open Source “is source code that is made freely available for possible modification and redistribution. Products include permission to use the source code, design documents, or content of the product. The open-source model is a decentralized software development model that encourages open collaboration. A main principle of open-source software development is peer production, with products such as source code, blueprints, and documentation freely available to the public” (

Wikipedia

).
Closed Source “is software that, according to the free and open-source software community, grants its creator, publisher, or other rightsholder or rightsholder partner a legal monopoly by modern copyright and intellectual property law to exclude the recipient from freely sharing the software or modifying it, and—in some cases, as is the case with some patent-encumbered and EULA-bound software—from making use of the software on their own, thereby restricting their freedoms.” (

Wikipedia

)

Table 1. Comparison Table for Open Source vs. Closed Source inspired by ChatGPT 3.5.

Criteria	Open Source	Closed Source
Acc

...
Notes on nukes, IR, and AI from "Arsenals of Folly" (and other books)
by tlevin — 2023-09-04T19:02:58.283Z

Richard Rhodes’s The Making of the Atomic Bomb has gotten

lots of attention in AI circles lately

, and it is a great read. I get why the people developing AI find it especially interesting, since a lot of it is about doing science and engineering and thinking about the consequences, but from my perspective as someone working on AI governance, the most powerful stuff was in the final few chapters as the scientists and policymakers begin to grapple with the wild implications of this new weapon for global politics.

Rhodes’s (chronologically) first follow-up, Dark Sun: The Making of the Atomic Bomb, is even more densely useful for thinking about emerging technology governance, and I probably recommend it even more strongly than TMOTAB for governance-focused readers.

However, I didn’t really start taking notes during my audiobook-listening until I started my third Rhodes tome, Arsenals of Folly: The Making of the Nuclear Arms Race. AOF is probably less applicable to AI governance than its predecessors, since it mostly focuses on a time when nuclear weapons had been around for several decades rather than when they were a new and transformative technology, but it still had a bunch of interesting details, and I figured I’d spare some of you the trouble of finding them by posting my notes to the forum. (Unfortunately, since I audiobooked, I don’t have page numbers.)

I’ve also included a list of other cool finds from my nuclear/Cold War reading from the last few months in an appendix.

Gell-Mann caveat

Before I say the rest of these facts, I should note that I had some “Gell-Mann Skepticism”[1] at some of Rhodes’s analysis. Mostly, in his increasingly strong rhetoric about the titular folly of nuclear weapons, he makes pretty questionable counterarguments to the usual cases for nuclear weapons being advantageous. He cites this paper by Jacek Kugler, who argues that nuclear-armed states didn’t seem to be able to impose their policy goals against non-nuclear armed states in a sample of Cold War conflicts like the Berlin Airlift, Vietnam War, invasion of Hungary, etc. It might initially seem surprising that, as Kugler claims, the nuclear-armed states lost these conflicts about as often as they won. But this ignores the enormous selection bias of what conflicting interests become actual disputes in the first place. It seems likel

...
Hertford, Sourbut (rationality lessons from University Challenge)
by Oliver Sourbut — 2023-09-04T18:44:24.359Z
Link: https://www.oliversourbut.net/p/hertford-sourbut

Amongst the huge range of excitements offered by joining the University of Oxford was the unexpected opportunity to join this lovely bunch

Hertford College University Challenge team 2023, with our mascot Simpkin the cat

You can tune in on 2023-09-04 at 20:30 UK time on BBC 2 or watch or catch up online if you want to see us in action.

As a relative quiz-noob[1], joining an elite quizzing team (hold your applause) was an eye-opening experience in a few ways. I'm not allowed to talk about how things went on the show (on pain of getting told off by the NDA police), but actually (as with all forms of performance and competition), the vast majority of the time was spent in prep and practice, which is where most of the insights came in anyway.

I'm going to talk a bit about University Challenge, and also gesture at how the experience as a competitive quizzer relates to broader theory and practice in decision-making under uncertainty. If you just want to see some fun quiz questions and my take at answering them, you can skip the middle Real-time calibrated decision-making section, or just skip reading this entirely and watch the show.

The format and some example questions

For readers unfamiliar with University Challenge, it's a competitive quiz, where each match consists of two teams head-to-head. Importantly for this discussion, a key part of the format is buzzer rounds ('Starter for 10'): that means you don't just have to know the answer, you have to know the answer and buzz before your opponent if they also know, otherwise you get nothing. But buzz too soon with a wrong answer and you lose points[2].

Here are some example questions. Maybe you know some of the answers! If you want to, imagine hearing the question word by word - when do you have a good guess or some ideas? At what point are you confident of the answer? Would you risk buzzing early and losing points if you're wrong - and on what basis?

I'll go through these examples later, and give the answers (my realtime guesses and the actual ground truth).

What single-digit number links: the element boron, the fourth root of 625, and the planet Jupiter’s position from the Sun?

Resembling a cornet but having a slightly larger bell, which instrument is a standard in British brass bands, its name being the German for ‘wing horn’?

Rayleigh-Taylor, Kelvin-Helmholtz and Rayleigh-Bénard are all types of what general physical

...
a rant on politician-engineer coalitional conflict
by bhauth — 2023-09-04T17:15:25.765Z

Sometimes, a group in some organization has a highly technical and highly effective leader. Kelly Johnson (Skunk Works) and Hyman Rickover (US Navy nuclear propulsion) are famous examples. A naive economist might expect such people to be well-liked by management above them, because their skills are good for the organization and complementary to those of non-technical managers. That's not what we generally see in reality.

In my experience, and in the stories I've heard, such technical leaders are especially disliked by upper management, far more than a highly effective non-technical MBA would be. I've even been told that unique competence being noticed by upper managment is a negative for career prospects in that situation.

Why would that be the case? The only explanation that makes sense to me is that effective technical managers are considered a threat by management above them - but why would they be more of a threat than a MBA who talks the business talk?

There are some cultural differences between engineers and non-technical managers, but I don't think that's an explanation. One reason is, technical leaders can find allies even higher up that support them. For example, Rickover had allies in Congress, and that's the only reason he wasn't pushed out...until he got pushed out by John Lehman, a Ph.D. in American foreign policy who's worked as an investment banker. Leslie Groves was almost pushed out in 1927, but Major General Edgar Jadwin interceded and noted that Groves's superiors were at fault for the problems blamed on him - that was a guy 5 ranks above Groves in the Army.

My current view is that politician-type managers and engineer-type managers naturally form opposing coalitions. They each favor people of the same type, and try to push local organization norms in different directions.

In America, today, politican-type managers have won conclusively almost everywhere. I've actually seen some of a conflict between such coalitions play out once, and I'd say it's an even match when the groups are equal in size and nobody else is involved. One group backstabs and fights over social dominance like high school girls, but that's balanced out by the other group spending half their time arguing about who's smarter and the other half arguing about emacs vs vim; the underlying dynamic is the same, but one group has the pretense of authority being tied to intelligence,...
How ForumMagnum builds communities of inquiry
by Jim Fisher — 2023-09-04T16:52:11.037Z

The website you're currently using is powered by ForumMagnum. But what really is ForumMagnum? What is it for, and why was it designed this way? In this post, I cast ForumMagnum as a medium for building communities of inquiry. I show how ForumMagnum is designed to build norms like rationality and long-form. Lastly, I suggest how the ForumMagnum developers could use the body of CoI research to guide their future product design.

What is ForumMagnum for?

ForumMagnum describes itself as "the codebase powering LessWrong and the Effective Altruism Forum."[1] That's the what ... but what's the why? Here's why I believe ForumMagnum exists:

ForumMagnum is a medium for building online communities of inquiry. It's a web forum that embeds norms of rationality and long-form comms.

That is: ForumMagnum is not defined by its users, or its features, or its codebase! ForumMagnum is a medium designed to carry a message, and its message is a set of social norms. Let's see what those norms are, and how ForumMagnum's features are designed to build those social norms.

What are Communities of Inquiry?

In the 6884th most cited paper of all time, Garrison et al. set out to "investigate the features of the written language used in computer conferences [e.g., forums] that seem to promote the achievement of critical thinking."[2] They built on the concept of a Community of Inquiry (CoI), which Wikipedia nicely defines:

The community of inquiry (CoI) is a concept first introduced by early pragmatist philosophers C. S. Peirce and John Dewey, concerning the nature of knowledge formation and the process of scientific inquiry. The community of inquiry is broadly defined as any group of individuals involved in a process of empirical or conceptual inquiry into problematic situations. This concept was novel in its emphasis on the social quality and contingency of knowledge formation in the sciences ...[3]

The LessWrong community, the EA community, the Alignment community, the Progress community — what unites them? It's not just that each community uses ForumMagnum! It's that each community is a CoI. Let's see how each community describes itself.

LessWrong is online forum/community that was founded with the purpose of perfecting the art of human rationality. ... LessWrong is a good place for [those] who want to work collaboratively with others to figure out what's true.[4]

Effective al

...