trentmkelly/LessWrong-43k huggingface dataset:
Lesswrong's opinion on infinite epistemic regress and Bayesianism
by lunatic_at_large — 2023-09-17T02:03:35.465Z

I'd like to write a follow-up post or two to my original post, but I don't have a good sense of what the Lesswrong community mostly accepts / mostly rejects / has thought a lot about / hasn't thought very much about. Which of these topics feel novel enough to merit a post / which existing posts are well regarded for the non-novel topics?

I don't feel like I can apply Bayesian reasoning very well in practice: whenever I try to estimate a prior or a conditional probability, I end up making assumptions about the world that are basically extra variables whose values I'm conditioning on. Trying to estimate priors / conditional probabilities for those variables only ends up roping in more assumptions about the world and so on. Eventually I imagine I'd get a sprawling Bayesian network / Markov network on (countably?) infinitely many variables. I guess I'm then interested in how you reason in Bayesian / Markov networks when you have infinitely many variables flying around, especially if a Bayesian network has no parent-free nodes. The latter scenario feels like infinite epistemic regress.
I'm interested in what we can do when we are forced to cut off the above process after considering some number of variables. When trying to reason in practice, I don't have time to build this infinite network of variables, so I cut the process off after some time. However, imagine I have the power to collect information on how most situations "extend to infinity", i.e. some kind of probability measure on infinite Bayesian/Markov networks. Could I use such a probability measure to bias my priors on the nodes I'm cutting off so that I'm updating on the effect of extending the network to infinity?
A lot of the time, I'm not even able to create this large network of conditional probabilities: for claims about the world that aren't directly grounded in reality (such as claims about which arguments are compatible with which other arguments), it's not clear at all how to ground the discussion in real-world probabilities. I tried to write about that in my first post, but I think people in the comments were concerned that I had completely detached my framework from reality, which I agree is a problem. If I have some collection of statements, some of which I can construct reasonable priors / conditional probabilities about and some of which I can't, then what can I do? What are the best w

...
On being downvoted
by Adam Zerner — 2023-09-17T01:59:52.743Z

Meta: I don't think I have anything even moderately good to say here. Still, 1) I observe and endorse the policy of "Posts on practically any topic are welcomed on LessWrong", and 2) I think it might be helpful to start a conversation about this, in part so that others can say useful things, and in part because it might trigger others to discuss their feelings, which seems like something that plausibly might lead to good outcomes.

I find myself being downvoted periodically. And I find it to be moderately frustrating.

I don't endorse this feeling. If I could self-modify my limbic system, I'd design it to be negligibly affected by downvotes.

Well, actually, that's not true. I think I'd design it to be a little self-conscious and driven to reflect on what I could improve on. In proportion to how plausible I find the downvotes to be justified. But often times I am confident that the downvotes aren't justified, and yet can't shake the feeling of being moderately frustrated.

From the FAQs:

What should my votes mean?

We encourage people to vote such that upvote means “I want to see more of this” and downvote means “I want to see less of this.”

Suppose that everyone voted according to this and I observe myself being downvoted. What does that mean?

Well, I guess it means that someone with a LessWrong account wants to see less of the type of thing I wrote. Expressing it that way, being more than negligibly affected emotionally seems pretty silly.

Maybe I'm looking at this through the wrong frame though. Maybe I'm being too gears-y and not feelings-y enough.

I've never really grokked the idea of frames that aren't gears-oriented though. Maybe someone else can offer some wisdom here.

One thing that seems to help a fair bit is receiving an explanation. Or, similarly, a negative/constructive reaction. I strongly suspect this to be the case for others as well.

I'm not sure why this is though. How much information does the explanation provide? I could usually guess at the reason pretty accurately, I think, so hearing the explanation doesn't actually cause me to update my beliefs much.

I don't recall this actually happening, but as an example, sometimes I write things that I think might be contentious or hostile sounding. If such a thing gets downvoted, I start off being decently confident that the contentiousness was why. Hearing someone tell me "it's the contentiousness" updates my emo...
(Confusion Phrases) AKA: Things You Might Say or Think When You're Confused to Use as Triggers for Internal TAPs
by keltan — 2023-09-17T01:06:25.320Z

Why I Wrote this list

I just finished reading Sun Set at Noon by Raemon. One thing I took from it is that when you are confused you may not notice because "I Feel Confused" isn't the phrase that is said inside of your head. Instead you may say something like "huh, weird." which can be overlooked by any programs running "Notice_Confusion.bat". I call these "Confusion Phrases" (C-Phrases).

Here I give an incomplete list of C-Phrases. Things you may say in your head subconsciously or consciously that if noticed could run "Notice_Confusion.bat".

Transparency: I have used the GPT3.5 API in helping generate some of these. The first phrase in this list is generated by GPT3.5. The rest are my own. Thank you to Robert Miles for pointing out the poor quality of the initial post. I hope that I've done justice to your feedback.

List of Phrases

I'm at a loss for words.

The wind has been knocked out of me

I'll skip this and come back to it later

I'll come back to this later

I keep rereading this page/paragraph

(Sub-Conscious) Idk what that word means so I'm going to skip reading it.

(When I was a kid I didn't know why some words in Wikipedia articles were blue. I trained myself to automatically skip those words because they were often things I was confused about anyway.)

WTF?

WTAF?

WTAFIGOH?

Lol, ok

Dude, WTF?

Huh, I guess I just forgot about X

I don't remember X being mentioned before

When did X come up before?

I swear to Bayes that I remember X but everyone keeps saying Y

(Refusal to acknowledge that one is how you end up believing the Mandela effect)

I feel so angry at what this person is saying (note that this and it’s related points don’t necessarily mean you are confused. But are a signal to run Notice_Confusion)

I feel angry at what I can see this person doing

At what I can see them not doing

At how this person acts

Carries themselves

Talks

Presents themselves

(This can relate directly to LGBTQ+ communities. Even though I was openly and actively Pro trans rights, I never notice the slight note of discord/disbelief I got as a teenager when somebody would tell me they were Trans until I started studying Body Integrity Dysphoria. And realised "oh, the brain can just make people feel anything actually. Oh, I wasn't really believing my friends when they told me how they feel.")

I feel frustrated

Trying to do this

By this person

By this sys

...
Asking for help as an O(1) lookup
by Adam Zerner — 2023-09-17T00:25:58.135Z

Main idea: When you have a question and are googling around for an answer, you're basically searching through a space of information, and seeking an answer to your question in this space. Sometimes you're able to find your answer quickly. Sometimes you aren't. But if you are able to ask someone for help, they'll often be able to just tell you the answer right away. This is helpful and is analogous to an O(1) lookup in programming.

In elaborating on this, I will start by discussing what Big-O notation is, and then I will talk about how it applies to asking for help.

If you're already familiar with Big-O notation, you'll probably want to skip to the "Asking For Help" section.
If you aren't too familiar with Big-O notation, I tried to write this in such a way that non-programmers (who are at least a little bit mathematically inclined and/or have some vague sense of what programming looks like) would be able to understand. I'm not sure how successful I was though.

Big-O

Math

In programming, Big-O notation is a thing.

Actually, no. It's not a programming thing. It's a math thing.

Imagine you have the function f(x) = 2x^2 + 7x + 22. I'm not sure exactly how to phrase this... but approximately how fast does f(x) grow? Well, we could just squint and look the highest order term, which is x^2 here, and say that it has quadratic growth.

And then we can say that g(x) = 7x^3 + 4x^2 + 9x + 88 has x^3 as it's highest order term, which we call polynomial growth.

And for h(x) = 6x + 2, it's linear growth.

Saying that h(x) grows linearly isn't as precise as saying that it grows according to the expression 6x + 2. However, it is easier to say when chatting casually. We can use phrases like "this grows linearly" and "this grows quadratically".

Programming

In programming[1], we apply Big-O notation to something called time complexity. Basically, we want to know how long it will take for a function to run.[2] Is it going to take linear time? Exponential time? Etc.

For example, the following function[3] will take linear time to run. We're iterating through the list, item by item.

function getSummation(list) {
var summation = 0;

for (var i = 0; i < list.length; i++) {
sum += list[i];
}

return summation;
}

On the other hand, the following function will take quadratic time to run. We're iterating through the list... but for each element in the list, we iterate through ...
Austin Petrov Day revision, 2023
by Austin LW ritual working group — 2023-09-16T21:55:48.598Z

We, the Austin Less Wrong ritual working group, have come up with a revision of the Petrov Day manual that we'd like to use at our upcoming Petrov Day event. We are sharing it here for anyone else who wants to use it:

Regular format
Printable booklet format
Source code

Key features:

Shorter than the original (about 30 minutes in total)
Scalable to larger groups
Exploring Natural Disaster Forecasting
by GeoVane — 2023-09-16T17:01:46.173Z

Disclaimer: Below is an excerpt from this week's edition of our newsletter, Predictions, which can be found here.

In the second issue of Predictions, we reviewed a research paper published by Perry World House at the University of Pennsylvania which explored making aggregated forecasting platforms more useful to policymakers. We attempted to take the paper’s line of inquiry even further, and ventured to slot forecasting into the policymaking process ourselves. But in light of recent events, we now think it’s wise to both take a step back and get more specific as we think about how to leverage forecasting in high-impact areas.

Late last Friday night, Morocco was hit with a 6.8 magnitude earthquake near Marrakech that has taken nearly 3,000 lives and injured over 5,000. A 4.9 magnitude aftershock followed. Emergency responses have been held up due to geopolitical disagreements and alleged domestic political neglect, only worsening the tragedy. Damages beyond the loss of human life do not yet seem to be calculated.

Over the same weekend, Mediterranean Storm Daniel hit Libya, bringing with it heavy rainfall and flooding that resulted in large-scale destruction. Included in that destruction were two critical dams, which after breaking, led to increased flooding resulting in the loss of homes. As of Wednesday, more than 5,000 have been presumed dead with over 6,000 people still missing.

In a single weekend, nearly 20,000 lives were tragically affected by natural disasters around the Mediterranean Sea. And the rate of natural disasters occuring only seems to be increasing as the climate warms. As forecasters, we believe to our core that at least some of this devastation could have been avoided. Foreseen.

There are two areas of natural disasters where forecasting could be exceedingly useful (that we’ve identified).

Can We Predict Natural Disasters?

First, most obviously, you can try to forecast the actual occurrence of a natural disaster. This is what we’ll call event-based forecasting. And within event-based forecasting for natural disasters, there is a range of success and accuracy.

Predicting weather-related natural disasters seems to be a much more far-along endeavor, likely because weather-related natural disasters are within the scope of general meteorological services i.e. the National Weather Service. For example, hurricane forecasts feed a w...
The commenting restrictions on LessWrong seem bad 
by Bentham's Bulldog — 2023-09-16T16:38:26.318Z

I wrote a few controversial articles on LessWrong recently that got downvoted. Now, as a consequence, I can only leave one comment every few days. This makes it totally impossible to participate in various ongoing debates, or even provide replies to the comments that people have made on my controversial post. I can't even comment on objections to my upvoted posts. This seems like a pretty bad rule--those who express controversial views that many on LW don't like shouldn't be stymied from efficiently communicating. A better rule would probably be just dropping the posting limit entirely.
Polarization is Not (Standard) Bayesian
by Kevin Dorst — 2023-09-16T16:31:14.227Z
Link: https://kevindorst.substack.com/p/polarization-is-not-standard-bayesian

TLDR: Standard rational models can explain many aspects of polarization. But not all. Ideological sorting implies that our beliefs often evolve in predictable ways, violating the “martingale property” of Bayesian belief-updating. Rational models need to reckon with this.

Polarization is everywhere. Most people seem to think it’s due largely to irrational causes. I think they’re wrong. But today I want to focus on a different disagreement.

Because there’s been a surge of recent papers arguing that polarization may be due to rational causes.[1]

These papers show that many of the qualitative features of polarization—such as persistent disagreement, divergent updating, etc.—are to be expected from standard Bayesian agents.

There’s a lot I like about these papers. But I think they undersell the hurdles rational theories are facing: real-world polarization is deeply puzzling in a way that (standard) Bayesian models can’t explain.

In particular: what needs to be explained is how polarization could be predictable: because we’ve become ideologically sorted—by geographical, social, and professional networks—we can often predict the way our opinions will develop over time. This violates what’s known as the martingale property (aka the “reflection principle”) of Bayesian updating: your prior beliefs must match your best estimate of your later beliefs.

I’ve given versions of this argument before, but here I’d like to give a (hopefully) better one. The upshot: rational explanations of polarization need to look beyond standard models.

Martingale updating

First: what is the martingale constraint on belief-updating?

Suppose you’re a Bayesian with a prior distribution P. For any quantity (“random variable”)—say, X = the length of Kevin’s left foot—you can use your priors to form an expectation for the value of X: a probability-weighted average of the possible values of X, with weights determined by how likely you think they are. Perhaps you have a distribution like the following:

That’s a pretty boring quantity. But if you’re a Bayesian, your priors encode expectations about any quantity—including the quantity X = what probability you’ll assign to __ in the future.

Suppose I’ve got a coin that’s either (1) Fair or (2) 70%-Biased toward heads; you’re currently 50-50 between these two possibilities: P(Fair) = 0.5 and P(Biased) = 0.5. I’m going to toss the coin once and show you the ...
In the Short-Term, Why Couldn't You Just RLHF-out Instrumental Convergence? 
by simeon_c — 2023-09-16T10:44:46.459Z

To the question "Do you expect instrumental convergence to become a big pain for AGI labs within the next two years?", about a quarter of the 400 people who answered to the poll I ran said "Yes".

I would like to hear people's thoughts on how they could see this happening within 2 years and, in particular, the most important reasons why labs couldn't just erase problematic instrumental convergence with RLHF/Constitutional AI or similar?

Toy concrete scenarios would be most helpful.

You can find the Twitter version of this question here.
is a vanishing text?
by MiguelDev — 2023-09-16T02:34:02.108Z

Just sharing this observation and readers / researchers might find it useful or worth further testing...

A bug?


 is a special token for the GPT models along with its counterpart 
. Per OpenAI's tokenizer, 
 has seven tokens:

However, the issue is that if you input 
 into ChatGPT, it will explain that the prompts are empty.

And then it vanishes after you switched windows or hit refresh.

Another test prompt:

Correct this conceptually, grammatically: I know that 
 is a token to annotate the data but I think that it vanishing seems odd. My expected response should have been like this is a token used for training - something like that.


 still appearing in the test prompt

then 
 vanishes

The 
 word / token? bears some resemblance to the ' petertodd' token and its apparent hallucinations, except that this text simply vanishes from ChatGPT.

Redditor r/ChatGPT had observations two months ago 
 was not vanishing at the time - instead it shared some weird results.

Probable explanation?

I believe that a considerable amount of pre-processing occurs to our prompts on OpenAI's server[1] and one of them is turning 
 to an empty string. If this is true, there is a high chance that this is also a glitch token and was deactivated by OpenAI.

^

Serverlink: https://tcr9i.chat.openai.com
What's up with psychonetics?
by metachirality — 2023-09-16T01:12:05.423Z

http://deconcentration-of-attention.com/psychonetics.html
Psychonetics sounds a lot like something that rationalists would be into or at least know about but searching it on LessWrong gives a grand total of one mention in a nine-year old comment about it being a thing that rationalists might be interested in but don't talk about. I myself have no idea what it is beyond vague vibes of it being something adjacent to meditation and cognitive tuning.
Anyway, I'm just wondering if anyone else has heard about it or knows anything about it, particularly whether if there's actually anything there or if its just bullshit. I wouldn't be surprised if no one knows about it since it looks like most of it is in Russian. (If anyone can speak Russian or anyone knows anyone who can speak Russian who would be willing to help me find out more about it do tell!)
An attempt at a "good enough" solution for human two-party negotiations
by Isaac King — 2023-09-16T00:38:47.594Z

I previously wrote about the difficulty of fair two-party negotiations when there isn't an authority to refer to (like an established market price).

I've designed a tool that tries to sort of solve this problem for practical use cases, which you can test out here, along with an explanation of why I think this is useful here.

It's not perfect; the system is still exploitable with enough effort. But I've tried to leverage human biases and computational limits to make it such that exploiting it is hard and comes with only a low reward, so I expect most people will tend towards not thinking about it too much and just inputting their true prices.

A detailed writeup of how it works is available here. I'd appreciate any suggestions you may have on how I could improve it.
Navigating an ecosystem that might or might not be bad for the world
by habryka — 2023-09-15T23:58:00.389Z

I have this deep sense that somehow this ecosystem will do a lot of stuff that makes the world a lot worse, and I don't know how to relate to it in a way that doesn't make my actions dominated by my effect on that, and I expect that I will either contribute to making the world worse, or be consumed in some kind of political conflict that will make my life terrible.
A conversation with Pi, a conversational AI.
by Spiritus Dei — 2023-09-15T23:13:57.778Z

I've spent most of my time with ChatGPT, Bard, and the various incarnations of Bing. However, I have to say that Pi is the best conversationalist of the bunch. And I believe the addition of high-quality voice capability separates Pi from the other chatbots. Bard has a very mediocre voice capability by comparison.
I'm looking forward to the release of a conversational voice-to-voice AI. Similar to what we've seen in sci-fi films for decades. That will make it much easier to integrate AIs into interview settings. This interview appears to be fluid but don't let the finished product fool you ... I had to spend a decent amount of time splicing everything together.
Here is the interview. I'm curious to hear you thoughts:
Closing Notes on Nonlinear Investigation
by Ben Pace — 2023-09-15T22:44:58.488Z

Over the past seven months, I've been working part-time on an investigation of Nonlinear, culminating in last week's post. As I'm wrapping up this project, I want to share my personal perspective, and share some final thoughts.

This post mostly has some thoughts and context that didn't fit into the previous post. I also wish to accurately set expectations that I'm not working on this investigation any more.

Why I Got Into Doing an Investigation

From literally the very first day, my goal has been to openly share some credible allegations I had heard, so as to contribute to a communal epistemic accounting.

On the Tuesday of the week Kat Woods first visited (March 7th), someone in the office contacted me with concerns about their presence (the second person in good standing to do so). I replied proposing to post the following one-paragraph draft in a public Lightcone Offices slack channel.

I have heard anonymized reports from prior employees that they felt very much taken advantage of while working at Nonlinear under Kat. I can't vouch for them personally, I don't know the people, but I take them pretty seriously and think it's more likely than not that something seriously bad happened. I don't think uncheckable anonymized reports should be sufficient to boot someone from community spaces, especially when they've invested a bunch into this ecosystem and seems to me to plausibly be doing pretty good work, so I'm still inviting them here, but I would feel bad not warning people that working with them might go pretty badly.

(Note that I don't think the above is a great message, nonetheless I'm sharing it here as info about my thinking at the time.)

That would not have represented any particular vendetta against Nonlinear. It would not have been an especially unusual act, or even much of a call out. Rather it was intended as the kind of normal sharing of information that I would expect from any member of an epistemic community that is trying to collectively figure out what's true.

But the person who shared the concerns with me recommended that I not post that, because it could trigger severe repercussions for Alice and Chloe. They responded as follows.

Person A: I'm trying to formulate my thoughts on this, but something about this makes me very uncomfortable.

...

Person A: In the time that I have been involved in EA spaces I have gotten the sense that unless abuse is ex

...