trentmkelly/LessWrong-43k huggingface dataset:
The impossibility of rationally analyzing partisan news
by RationalDino — 2023-11-16T16:19:50.485Z

This is a response to Thinking About Filtered Evidence Is (Very!) Hard that I thought deserved to be its own post.

The post said that it lacked a practical/interesting failure-case for Bayes' Law. Here is a case that is both practical and interesting.

Suppose that we have a variety of sources, divided into two camps. Each camp wishes to persuade us to their point of view. Each source within each camp has a different, unknown to us, amount of reliability. Critically, if we are persuaded by either camp, we will find most of the sources in that camp believable. And most in the other camp to be not believable.

We further know that each camp does contain reliable sources who are accurately reporting on filtered versions of the same underlying reality. We just don't know how common that is on either side.

This is hopefully recognizable as a description of trying to reconstruct the world from partisan news sources of varying quality. That makes it practical. I'll make it interesting by asserting some key facts about it.

Our set of beliefs in this scenario form a Bayesian network. (Straight from the definition.)

Correctly updating a Bayesian network after a single observation is an NP-complete problem. See "The Computational Complexity of Probabilistic Inference Using Bayesian Belief Networks".

Approximately updating a Bayesian network after a single observation to determine beliefs to any accuracy below 50% is ALSO NP complete. That is, one observation can be the difference between being confident that A is true to being confident that B is true. And it is computationally intractable to find which observation should do it. See On the hardness of approximate reasoning.

This particular network, where we flip from one set of reinforcing beliefs to the other set of reinforcing beliefs, is of the kind that demonstrates this exact type of numerical intractability.

Additionally sources may make statements of mathematical fact. Perhaps encoded as, say, a statement about economics. Thereby making updates computationally impossible as well as numerically intractable.

The result? It is both numerically intractable, and occasionally computationally impossible, to maintain rational opinions about what's true when your information comes filtered through partisan news networks.

Any struggles we have with cognitive biases come on top of that basic impossibility result.
We are Peacecraft.ai!
by MadHatter — 2023-11-16T14:15:53.478Z

I would like to announce my new alignment organization. We have no funding as of yet, but we have a lot of exciting plans and schemes.

Our alignment philosophy is simple: we cannot align AI's to human values until we know approximately what human values actually are, and we cannot know that until we solve the human alignment problem. Thus we will both operate as an AI alignment certification organization, and as a peacebuilding organization to solve human-on-human conflicts.

Our main novel idea is the Coordination Market. This is an equivalent to a prediction market for issues that are matters of opinion rather than matters of fact. In the rest of this post, we outline how a coordination market works, and provide a worked example, allowing more housing to be built in the Bay area.

A coordination market has futures that resolve upon a specified event in the real world, just like a prediction market. So, for the Bay area housing market, the futures will resolve when a proposed development is approved, built, and open for tenants to move into.

The futures are associated with different drafts of proposals. People betting in a coordination market are trying to put their money on the proposal that will end up winning. Since 90% of getting a winning political coalition together is signaling that you already have most of the coalition you need, these bets also function as a sort of "thumb on the scale": people with lots of money can influence the outcome, a little bit, but if their thumb is too heavy on the scale, then they'll lose whatever money they're using to weight the scale, and they won't have a thumb to put on the scale next time it comes up.

In parallel to the coordination market, we add a liquid democracy component to gauge public opinion in the relevant part of the world. For instance, to build a building in Berkeley one presumably needs a coalition of voters sufficient to replace whoever is responsible for appointing the zoning committee. So, a developer trying to build a development would need to set up a coalition of likely voters sufficient to intimidate any zoning board members who stood in the way of the project. At the same time, the developer could only assemble this coalition by genuinely seeking and obtaining meaningful consent from local stakeholders. So the liquid democracy component does a nice job of balancing local democratic ideals with t...
A dialectical view of the history of AI, Part 1: We’re only in the antithesis phase. [A synthesis is in the future.] 
by Bill Benzon — 2023-11-16T12:34:35.460Z

Cross-posted from New Savanna.

The idea that history proceeds by way of dialectical change is due primarily to Hegel and Marx. While I read bit of both early in my career, I haven’t been deeply influenced by either of them. Nonetheless I find the notion of dialectical change working out through history to be a useful way of thinking about the history of AI. Because it implies that that history is more than just one thing of another.

This dialectical process is generally schematized as a movement from a thesis, to an antithesis, and finally, to a synthesis on a “higher level,” whatever that is. The technical term is Aufhebung. Wikipedia:

In Hegel, the term Aufhebung has the apparently contradictory implications of both preserving and changing, and eventually advancement (the German verb aufheben means "to cancel", "to keep" and "to pick up"). The tension between these senses suits what Hegel is trying to talk about. In sublation, a term or concept is both preserved and changed through its dialectical interplay with another term or concept. Sublation is the motor by which the dialectic functions.

So, why do I think the history of AI is best conceived in this way? The first era, THESIS, running from the 1950s up through and into the 1980s, was based on top-down deductive symbolic methods. The second era, ANTITHESIS, which began its ascent in the 1990s and now reigns, is based on bottom-up statistical methods. These are conceptually and computationally quite different, opposite, if you will. As for the third era, SYNTHESIS, well, we don’t even know if there will be a third era. Perhaps the second, the current, era will take us all the way, whatever that means. Color me skeptical. I believe there will be a third era, and that it will involve a synthesis of conceptual ideas computational techniques from the previous eras.

Note, though, that I will be concentrating on efforts to model language. In the first place, that’s what I know best. More importantly, however, it is the work on language that is currently evoking the most fevered speculations about the future of AI.

Let’s take a look. Find a comfortable chair, adjust the lighting, pour yourself a drink, sit back, relax, and read. This is going to take a while.

Symbolic AI: Thesis

The pursuit of artificial intelligence started back in the 1950s it began with certain ideas and certain computational capabilities. The latt...
How much fraud is there in academia?
by ChristianKl — 2023-11-16T11:50:41.544Z

Quinn wrote a while ago "I heard a pretty haunting take about how long it took to discover steroids in bike races. Apparently, there was a while where a "few bad apples" narrative remained popular even when an ostensibly "one of the good ones" guy was outperforming guys discovered to be using steroids."

I have been thinking about that notion after researching BPC 157 where it seems that the literature around it is completely fraudulent.

How do you think about the issue of how much of the literature is fraudulent?
Learning coefficient estimation: the details
by Zach Furman — 2023-11-16T03:19:09.013Z
Link: https://colab.research.google.com/github/zfurman56/intro-lc-estimation/blob/main/Intro_to_LC_estimation.ipynb

What this is for

The learning coefficient (LC), or RLCT, is a quantity from singular learning theory that can help to quantify the "complexity" of deep learning models, among other things.

This guide is primarily intended to help people interested in improving learning coefficient estimation get up to speed with how it works, behind the scenes. If you're just trying to use the LC for your own project, you can just use the library without knowing all the details, though this guide might still be helpful. It's highly recommended you read this post before reading this one, if you haven't already.

We're primarily covering the WBIC paper (Watanabe 2010), the foundation for current LC estimation techniques, but the presentation here is original, aiming for better intuition, and differs substantially from the paper. We'll also briefly cover Lau et al. 2023.

Despite all the lengthy talk, what you end up doing in practice is really simple, and the code is designed to highlight that. After some relatively quick setup, the actual LC calculation can be comfortably done in one or two lines of code.

What this isn't for

A good overview of SLT, or motivation behind studying the LC or loss landscape volume in the first place. We're narrowly focused on LC estimation here.
Sampling details. These are very important! But they're not really unique to singular learning theory, and there are plenty of good resources and tutorials on MCMC elsewhere.
Derivations of formulas, beyond the high-level reasoning.

TLDR

What is the learning coefficient? (Review from last time)

The learning coefficient (LC), also called the RLCT, measures basin broadness.
This isn't new, but typically "basin broadness" is operationalized as "basin flatness" - that is, via the determinant of the Hessian. When the model is singular (eigenvalues of the Hessian are zero), this is a bad idea.
The LC operationalizes "basin broadness" as the (low-loss asymptotic) volume scaling exponent. This ends up being the right thing to measure, as justified by singular learning theory.

How do we measure it?

It turns out that measuring high-dimensional volume directly is hard. We don't do this.
Instead we use MCMC to do what's known in statistics as "method of moments" estimation. We contrive a distribution with the LC as a population parameter, sample from that distribution and calculate one of its moments, and solve for th

...
Extrapolating from Five Words
by Gordon Seidoh Worley — 2023-11-15T23:21:30.865Z

If you only get about five words to convey an idea, what will someone extrapolate from those five words? Rather than guess, you can use LLMs to experimentally discover what people are likely think those five words mean. You can use this to iterate on what five words you want to say in order to best convey your intended meaning.

I got this idea because I tried asking Claude to summarize an article at a link. Claude doesn't follow links, so it instead hallucinated a summary from the title, which was included in the URL path. Here's an example of it doing this with one of my LessWrong posts:

It hallucinates some wrong details and leaves out lots of details that are actually in the post, but it's not totally off the mark here. If my ~Five Words were "the problem of the criterion matters", this would be a reasonable extrapolation of why I would say that.

Rather than using a link, I can also ask Claude to come up what it thinks I would have put in a post with a particular title:

Strangely it does worse here in some ways and better in others. Unlike when it hallucinated the summary of the link, this time it came up with things I would absolutely not say or want someone to come away with, like the idea that we could resolve the problem of the criterion enough to have objective criteria for knowledge.

But maybe prompting it about LessWrong was the issue, since LessWrong puts off a lot of positivists vibes, Eliezer's claims to the contrary not withstanding. So I tried a different prompt:

This is fine? It's not great. It sounds like a summary of the kind of essay a bored philosophy undergrad would write for their epistemology class.

Let me try asking it some version of "what do my ~Five Words mean?":

This is pretty good, and basically what I would expect someone to take away from me saying "the problem of the criterion matters". Let's see what happens if I tweak the language:

Neat! It's picked up on a lot of nuance implied by saying "important" rather than "matters". This would be useful for trying out different variations on a phrase to see what those small variations change about the implied meaning. I could see this being useful for tasks like word smithing company values and missions and other short phrases where each word has to carry a lot of meaning.

Now let's see if it can do the task in reverse!

Honestly, "uncertainty undermines knowledge" might be better than anything I...
In Defense of Parselmouths
by Screwtape — 2023-11-15T23:02:19.344Z

Prerequisites: The Quaker and the Parselmouth.

I.

First, a quick summary.

In the prerequisite post, Benjamin Hoffman describes three kinds of people. These people are hypothetical extremes: they're the social and epistemic equivalents of perfect spheres interacting in frictionless vacuums. There are Quakers, who always tell the truth and keep their word when they say they'll do something. There are Actors, who always say what seems good to say at the moment and who don't reliably keep their word even if they swear and oath. Lastly, there are Parselmouths, who can lie freely to Actors but speak only the truth to other Parselmouths and (by implication) speak only truth to Quakers.

I approve of this distinction. It is abstracted and the real world is never this clear, but in my experience it does get at something useful to understand. I think truthtelling is a powerful institutional advantage, and wish more people were Quakers in this taxonomy[1]. Benjamin points out that Parselmouths are somewhat odd, in that habitually telling lies likely erodes the instinct or maybe even ability to tell the truth; it may not be possible for real people to stay consistently Parselmouths without slowly becoming Actors.

Speaking truth is hard. It's hard work to figure out what the true state of the world is. It's hard to quickly and accurately state what you think is true; the English language makes "I believe there's a ninety percent chance of rain tomorrow" a much longer sentence than "it's going to rain tomorrow." There's a lot of extra emotional sharp elbows you wind up throwing when someone asks you how you liked the (burned and unseasoned) casserole they brought to the potluck. Quakers of the world, I salute you. Actors of the world, I get it.

My first claim is that it's reasonable to be a Parselmouth.

II.

Storytime! The following story details events that happened about two decades ago, when I was several feet shorter than I am now. Some details have been substantiated by other people who were around at the time, but many likely have morphed over the years.

When I was a kid, I had to get a bunch of shots. My mom took me into the office, and I goofed around in waiting area for a little bit before a nurse waved me past the front desk and Mom and I went in. The nurse sat me down in the doctor's office on a big plastic chair and rubbed my shoulder with something cold while aski...
Life on the Grid (Part 1)
by rogersbacon — 2023-11-15T22:37:30.621Z
Link: https://www.secretorum.life/p/life-on-the-grid-part-1

The physical layout of the environment in which someone is raised impacts their cognitive abilities as an adult.

A recent study

based on video game data from 397,162 people across 38 countries found that “people who grew up outside cities are better at navigation.” More specifically, “growing up in cities with low Street Network Entropy (e.g. Chicago) led to better results at video game levels with a regular layout, while growing up outside cities or in cities with higher Street Network Entropy (e.g. Prague) led to better results at more entropic video game levels.”

In plain english: if you grow up in a grid-like environment, then you will be worse at navigating less grid-like environments.

Association between Environment and Video Game (Sea Hero Quest) Wayfinding Performance stratified by age, gender, and education. The SHQ Wayfinding Performance is computed from the trajectory length and has been averaged within 5-years windows.

Two examples of cities with low (Chicago) and high (Prague) SNE. Right: Distribution of the street bearings across 36 bins of 10 degrees.

This finding is perhaps not wholly surprising, and at first blush might not seem all that significant. The vast majority of us hardly ever need to truly wayfind IRL anymore. Technology has made our navigational skills all but obsolete. So what does it matter whether certain environments are better or worse at preserving them?

It matters a great deal. To understand why, we will need to take a small detour.

In his appearance on

Making Sense with Sam Harris

, complexity scientist David Krakauer makes a distinction between complementary cognitive artifacts—technologies that make us more intelligent after using them—and competitive cognitive artifacts (if you can’t guess what these do then maybe you’ve been using them too much). The canonical example of a competitive artifact is a calculator: repeated usage leaves you worse at mental arithmetic than you were before. Contrast this with an abacus, which can have quite the opposite effect: expert users can eventually develop such a high-fidelity mental model that they no longer even need to use the physical abacus, and are able to maintain their enhanced arithmetic skills without it.

Typus arithmeticae tratta da Margarita Philosophica (1503)

The brain is a complex system, full of interconnected representational systems. This is why an abacus doesn’t j...
Glomarization FAQ
by Zane — 2023-11-15T20:20:49.488Z

The main reason I'm posting this here is because I want it to be publicly recorded that I wrote this post in 2023. That way, I can show people in the future that I didn't just make this up on the spot; it's been around since long before whatever it is they're asking me about this time. I don't have a blog of my own, so this is where I'm putting it. Still, some of you might find it interesting, or want to use it yourself.

If I've sent you this post, you've probably just asked me a question, to which I responded something along the lines of "I refuse to confirm or deny that." Maybe you asked if I wrote a particular story, or if I had sex with someone, or if I robbed a bank. When you heard my refusal you likely became more suspicious, assuming it means that I really did write/bang/rob the story/person/bank.

This is the explanation of why that's not necessarily the case.

Why are you refusing to answer my question?

Suppose - and as generally holds throughout this post, I'm not saying this is true or false, just asking you to consider the hypothetical - I had never committed a crime in my life, any crime, with the singular exception of... let's say, public urination. With a bag over my head, so people didn't know for sure that it was me. The police are nevertheless able to narrow the suspects down to a pretty small list, with me on it, and they ask me if I did it.

Some people would just lie to the police. The crucial element here, though, is that I don't want to lie about this. This entire policy is about avoiding lies, and about being able to maintain my privacy while staying completely honest.[1] If I'm refusing to lie, but I also don't want them to know the truth, I can't answer the question at all. So I say, "I refuse to confirm or deny that."

But then suppose the police ask me if I killed someone - which I didn't do. And they ask if I robbed a bank, and if I sell drugs, and all sorts of other questions about crimes that I've never done before. I always tell the truth: "No, I didn't do that." And of course, now the police know that the answer to the public urination question was yes, because that's the only one where I refused to answer the question.

The only policy which doesn't tell them my secrets, other than outright lying, is to refuse to confirm or deny whether I committed any of the crimes - even the ones that I really didn't commit.

All ...
Testbed evals: evaluating AI safety even when it can’t be directly measured

by joshc — 2023-11-15T19:00:41.908Z

A few collaborators and I recently released the paper, “Generalization Analogies (GENIES): A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains. (tweet thread). In this post, I’ll explain how the GENIES benchmark relates to a broader methodology for predicting whether AI systems are safe even when it is impossible to directly evaluate their behavior.

Summary: when AI safety is hard to measure, check whether AI alignment techniques can be used to solve easier-to-grade, analogous problems. For example, to determine whether developers can control how honesty generalizes to superhuman domains, check whether they can control generalization across other distribution shifts like ‘instructions 5th graders can evaluate’ to ‘instructions that PhDs can evaluate.’ Or to test if developers can catch deception, check whether they can identify deliberately planted ‘trojan’ behaviors. Even when the safety of a particular AI system is hard to measure, the effectiveness of AI safety researchers and their tools is often much easier to measure – just like how it’s easier to measure rocket components in Aerospace testbeds like wind tunnels and pressure chambers than to measure them by launching a rocket. These ‘testbed’ evals will likely be an important pillar of any AI regulatory framework but have so far received little attention.

Background: why is AI safety ‘hard to measure’

There are two basic reasons why it's hard to tell whether AI systems follow developer instructions.

AI behavior could look good but not actually be good. For example, it's hard to tell if superhuman AI systems obey instructions like “develop successor AIs that you are confident are safe.” Humans can look at AI plans and try their best to determine whether they are reasonable, but it's hard to know if AI systems are gaming human evaluations — or worse – if they have hidden intentions and are trying to pull a fast one on us.

AI behavior cannot be observed in some environments without incurring risk. Many safety failures of frontier LLMs have been discovered after deployment, which will become obviously unacceptable after AI systems exceed some threshold of capability. Instead, developers must thoroughly evaluate safety in test environments where models are

unable

to take

truly dangerous

actions. It will be especially challenging to evaluate...