trentmkelly/LessWrong-43k huggingface dataset:
Can we have Epiphanies and Eureka moments more frequently?
by CstineSublime — 2025-01-08T02:20:26.897Z

Epiphanies and Understanding, Pain, Eureka-Moments.

TL;DR - mental-pain rushing in conveys urgent information, this experience is similar to when understanding rushes in when it is pleasurable - such as the sudden feeling of understanding of a philosophical idea or a joke. Can we reverse engineer these mechanisms to have Eureka-Moments more frequently?

Could I Eureka-that faster?

(Mental) Pain is (urgent) information

Recently I've been thinking about a throw away statement by Jerry Seinfeld that pain is the "sudden rushing in of information". I have my theories about where this sentiment originated from because Ray Dalio says "pain is the signal". Pain is of course a source of information: when you stub your toe in the middle of the night, that is the immediate intrusion of information about your and objects positions in space. When you cut yourself or feel a muscle pain - that too is information: urgent information that requires immediate action. But in the sense that Seinfeld and Dalio are using "pain" is figurative to describe what we might call psychic or mental pain: embarrassment. depression. regret. anguish. anger. etc. But also ugh fields. Chronic pain is information, but it is not sudden or immediate information. The feelings associated with depression too is non-urgent or non-immediate but still information.
Another sudden and painful psychic experience is the L'esprit de l'escalier or (lit. the spirit of the staircase). Which describes how only after the fact, usually after an insult, we have that forehead slapping moment of insight of what we should have said but didn't think of in the moment. The most famous pop-culture example, to relate it back to Mr. Seinfeld is the "Comeback" or "Jerkstore" episode of his sitcom where he goes to the effort of trying to engineer the same situation again so he can lob his response.

Is pain to information what the epiphany is to understanding?

This is probably the same mechanism as the A-Ha, or the Eurkea or Lightbublb moment. Such as when Kekulé awoke from his dream and could envision the structure of the Benzene molecule, or Archimedes having witnessed the effect of his body on displacing water found the solution to the measuring of the crown.

There are methods to cultivate sudden insight....

Daniel Dennett described a rhetorical device which is designed for creating these moments - the Intuition Pump. It too is...
Job Opening: SWE to help improve grant-making software
by Ethan Ashkie — 2025-01-08T00:54:22.820Z
Link: https://survivalandflourishing.com/full-stack-engineer

Full-Stack Software Engineer Position - (S-Process)

Survival and Flourishing Corp. (SFC) is seeking a highly numerate full-stack developer to improve and maintain existing grant evaluation software currently in use by the Survival and Flourishing Fund, making it easier for philanthropists to delegate grant-making discussions and deliberation to trusted advisors.

Annual compensation: $250,000 - $350,000

Application here.

Qualifications

Strong numeracy, including existing familiarity with:

Calculus
Linear algebra and matrix operations
Game theory, mechanism design, and expected utility theory

Experience or clear skill with:

Full-stack web development in Typescript and React
Command line scripting and data science in Python
Management of user accounts, roles, and access controls
Database management and ensuring the privacy, consistency, and correctness of sensitive data
Orienting on and iteratively improving a legacy codebase

Experience managing technical projects:

Organizing tickets, code changes, design decisions, and documentation
Helping a team maintain technical standards, processes, and norms
Ensuring the health and quality of a codebase with multiple contributors
Communicating with leadership about progress, estimates, implementation tradeoffs, and the impact of design decisions

Current state of development

The software in its current form has been used by several philanthropists to date, including Jaan Tallinn, Blake Borgeson, and the Future of Life Institute, via the S-process of the Survival and Flourishing Fund. We believe it’s time to improve the software and make it into a self-contained app, with secure accounts for philanthropists, advisors, and applicants seeking funding.

Why this position exists

Over the past few years, the interest of large philanthropic donors in improving the long term future of humanity has drastically increased. SFC wants to develop our process into an easy-to-use app that makes it easier for philanthropists to implement the deliberation process used by SFF, in turn making deliberation about the future and how to improve it more principled, more engaging for groups of stakeholders with divergent views, and more action-oriented.

Duties of position

This position would work on app development full-time, in collaboration with SFC Co-Owner Andrew Critch, CEO Ethan Ashkie, and Lead Engineer Joel Burget. Activities will include:

Implementing and adj

...
Markov's Inequality Explained
by criticalpoints — 2025-01-08T00:31:55.125Z
Link: https://eregis.github.io/blog/2024/10/22/markov-inequality-explained.html

In my experience, the proofs that you see in probability theory are much shorter than the longer, more involved proofs that you might see in other areas of math (like e.g. analytical number theory). But that doesn't mean that good technique isn't important. In probability theory, there are a set of tools that are useful across a broad variety of situations and you need to be able to recognize when it's the appropriate time to use each tool in your toolkit.

One of the most useful of tools to have is Markov's inequality. What Markov's inequality says is that, given a non-negative random variable Xand a positive real number a, the probability that X is greater than a can be upperbounded in the following manner:

P(X≥a)≤E[X]a

This is a pretty simple formula. You might have even been able to guess it beforehand using dimension analysis. Probabilities are often thought of as dimensionless quantities. The expectation E[X] and the threshold a both have units of "length", so we would need them to exactly cancel out to create a dimensionless quantity. We naturally expect that the probability of exceeding the threshold to (a) increase as the expectation of X increases and (b) decrease as the value of the threshold a increases. These sorts of arguments alone would point you in the direction of a formula of the form:

P(X≥a)≤(E(X)a)k

Why does k equal 1 in the case of Markov's inequality? There is probably some elegant scaling argument that uniquely pins down the power k, but we won't dwell on that detail.

Markov's inequality might not seem that powerful, but it is. For two reasons:

One reason is that Markov's inequality is a "tail bound" (a statement about the behavior of the random variable far away from its center) that makes no reference to the variance. This has both its strengths and its weaknesses. The strength is that there are many situations where we know the mean of a random variable but not its variance, so Markov's inequality is the only tool in our repertoire that will allow us to control the tail behavior of the variable. The weakness is that because it assumes nothing about the variance (or any higher moments), Markov's inequality is not a very tight bound. If you also know the variance of your random variable, you can do much better (though you will need more heavy-duty ...
Stream Entry
by lsusr — 2025-01-07T23:56:13.530Z

"Is this…enlightenment?" I asked.

"We use the word 'Awakening'," said the Zen Master, kindly.

"Now what?" I asked, "Every book I've read about meditation is about getting to this point. There's nothing about how to safely navigate the territory afterward—or what to do afterward—or even what happens afterward."

"The first thing you need to know is that what you experienced was real," the Zen Master spoke from personal experience and that of his many previous students, "Don't make it too complicated. Just be in the moment. If you're cooking, then just cook. If you're eating, then just eat."

It is impossible to communicate the subjective experience of Stream Entry to someone who has not personally experienced at least kensho. However, it is possible to communicate an outside mechanistic model of what happens in your brain. First I will explain how a human brain functions normally. Then I will explain how a post-Awakening brain differs.[1]

Your brain creates a real-time simulation of you and world around you. Everything you consciously perceive is part of this model. There are no exceptions. How does this produce useful behavior? Because the brain enforces a 1-to-1 correspondence between the simulation and reality (where "reality" is defined as sensory inputs). If this 1-to-1 correspondence is adequately maintained, then performing an act in the simulation happens simultaneously as that same action happening in external physical reality. That's why if you die in the Matrix then you die in real life too.

If a brain just modeled the environment without performing any actions, then the whole model could exist as a non-cyclic graph. But a brain doesn't just model the world. It sends commands to muscles that act on the physical universe it's trying to model. What happens when a brain tries to anticipate its own motor commands? Now the brain's own algorithms are a causal force in the external environment it's trying to model. The result is computationally intractable.

In response, the brain divides its model of the universe into "things under its motor control" and "things not under its motor control". Those parts of the universe it can control coalesce into "self" and the rest coalesces into "other".

Other calculations involving self-reference become encapsulated too. This produces high-level abstractions including pain, pleasure and desire. These encapsulations feel like pa...
Don't fall for ontology pyramid schemes
by Lorec — 2025-01-07T23:29:46.935Z

Pyramid schemes work by ambiguating between selling a product, and selling shares in the profit from the sale of that product. It's a kind of sleight of hand that saves the shills from having to explicitly say "our company is valuable because other people think it's valuable", which might otherwise be too nakedly dishonest to work. Shills can pretend they're offering marks a profitable sales job for a company that makes a product so wonderful the sheep can't resist it, marks buy in and pretend to be buying product to be used when really they're basing their high bids on the value at which they hope the product can be sold, and thus a few scammers at the top can make the value of share in their enterprise appear to inflate arbitrarily.

A similar pattern of exploitation occurs in the market for ontologies, although without legible prices or transaction.

If beliefs don't pay rent, they shouldn't be allowed to linger; the categories were made for man, not man for the categories. Just as you should decide which products to purchase based on how much value you expect to get out of using them directly, you should decide whether that big blue floppy thingy over there is a 'beast' or a 'fish' [or something else entirely] based on how you expect this to assist your ability to think about reality and draw true conclusions.

By default, when Jane Doe says "There are 4 components of personality - yellow bile, black bile, blood, and phlegm, and here's what they do" [or, alternatively, when Yamada Hanako says "there are 4 personality types corresponding to the 4 blood types, and here's what they are"], you will take her assertion and check it against your intuition, and possibly even go to the trouble of searching for concrete examples [ . . . and then counterexamples ] in your remembered experience. And then you'll decide to accept it or reject it, as a rent-paying concept structure.

Realistically, just because you run on human psychology, your decision will have a lot to do with how you feel about this particular Jane Doe. But what won't matter is how much corroborating evidence she can bring in to support her lovely categorization schema, from within a community of theorists that [unlike you] already accepts it. You know movements based on mistakes of fact build up intricate internal systems of theory all the time. Hanako can show you pages and pages of impressive-looking di...
Bridgewater x Metaculus Forecasting Contest Goes Global — Feb 3, $25k, Opportunities
by ChristianWilliams — 2025-01-07T21:40:30.899Z
Link: https://www.metaculus.com/bridgewater/

For the second consecutive year, Metaculus is collaborating with Bridgewater Associates — the world’s largest hedge fund and a leader in global asset management — to host a forecasting competition offering $25,000 in prizes and potential opportunities with Bridgewater. While last year's competition was limited to participants in the United States, this year we’re excited to welcome forecasters from around the globe.

You must register to compete — then, get in shape with our warmup questions to prepare for the competition kickoff February 3rd. We'll message you before the first round of questions launch.

Learn more about the contest structure, the open and undergraduate leaderboards, and details on how questions are scored here.

You can find the questions and results from last year's competition here.

One Contest, Two Leaderboards: Undergraduate and Open Competitions

This contest features a unified set of 50 forecasting questions, with two distinct leaderboards: one for all participants in the Open Competition and another specifically for undergraduate students in the Undergraduate Competition.

$25,000 in Prizes

Each competition boasts a $12,500 prize pool, totaling $25,000 across both leaderboards. The top 125 forecasters in each competition are eligible for prizes. Undergraduate participants will be ranked on both leaderboards and can win prizes in both categories. Prizes will be distributed after the contest ends and participants complete identity verification.

This year, there is no guaranteed $50 minimum prize per participant, meaning top performers are likely to receive a larger share of the prize pool, and fewer than 125 prizes may be awarded in each competition.

Undergraduate Competition and Leaderboard

Open exclusively to currently enrolled undergraduate students, the Undergraduate Competition offers a dedicated $12,500 prize pool. Undergraduate participants are automatically included in the Open Competition as well, providing dual opportunities to compete and showcase their skills. Outstanding performance in this category could capture the attention of the Bridgewater recruiting team.

Open Competition and Leaderboard

The Open Competition is open to everyone, regardless of background or experience, with its own $12,500 prize pool. Both seasoned and novice forecasters are encouraged to participate, demonstrate their forecasting prowess, and ...
A Principled Cartoon Guide to NVC
by plex — 2025-01-07T21:01:07.904Z

TL;DR: Making claims or demands about/into other people's internal states, rather than about your state or observable external states, predictably ties people in knots—instead: only make claims about your own experience or observables. This lets the other control the copy of them that's in the shared context.[1]

Non-Violent Communication rapidly grew into a world-spanning movement due to its remarkable ability to avert and dissolve conflict. The NVC book has a whole bunch of examples of communication procedures and practical tips. Here I'll try and convey the core insights more rapidly by focusing on the underlying principles I've extracted.[2]

Background models

People have self-models. People have models of other people. Conversations form shared context, which syncs with both of those. Some ways of conversing work better than others.

People's self-models are entangled with the copies of themselves inside the heads of the people they interact with, synced by conversational context. It's painful to be claimed to be something you don't want to be.[3] Dissonance between your self-model unquoted claims about you is a type of prediction error that hits particularly hard.

Common collision in conversational context (Cartoons start here[4])

Alice is unhappy about an external state W, and thinks W is because Bob's internal state is Z. She shares her negative emotion along with her current prediction of the reason for W: Bob's state being Z.[5]

Bob's read of his own internal state is not Z and this collides with Alice's statement, causing dissonance.[6]

Naturally, Bob pushes back on the part of the exchange most salient to him: the adding of a false claim about something he has better access to mixed with negative incoming emotion.

Things can get messy from here! Bob and Alice are having a tug of war over a non-load-bearing part of the map–Bob's state–which is distracting them both from resolving the grievance.

This situation is often resolvable by sharing context if there's enough ambient trust, but it sometimes escalates in very harmful ways.

Just talking about yourself is good, actually

Now Alice raises W and X, but makes no claim on Bob's experience.[7]

Bob can now focus on the information that matters most, W and X, with no threat to their self-model.

Bob can fill in the part of the conversational object that reflects his internal state and clarify the s

...
Disagreement on AGI Suggests It’s Near
by tangerine — 2025-01-07T20:42:43.456Z

If I’m planning a holiday to New York (and I live pretty far from New York), it’s quite straightforward to get fellow travellers to agree that we need to buy plane tickets to New York. Which airport? Eh, whichever is more convenient, I guess. Alternatively, some may prefer a road trip to New York, but the general direction is obvious for everyone.

However, as the holiday gets closer in time or space, the question of what we actually mean by a holiday in New York becomes more and more contentious. Did we mean New York State or New York City? Did we mean Brooklyn or Broadway? Which Broadway theater? Which show? Which seats?

By my estimation, the fact that the question of whether AGI has been achieved is so broadly contentious shows that we are so close to it that the term has lost its meaning, in the same way that “Let’s go to New York!” loses its meaning when you’re already standing in Times Square.

It’s time for more precise definitions of the space of possible minds that we are now exploring. I have my own ideas, but I’ll leave those for another post…
Role embeddings: making authorship more salient to LLMs
by Nina Panickssery — 2025-01-07T20:13:16.677Z

This is an interim research report on role embeddings, an approach to make language models more robust to many-shot jailbreaks and prompt injections by adding role information at every token position in the context rather than just at special token delimiters. We credit Cem Anil for originally proposing this idea.

In our initial experiments on Llama 3, we find that role embeddings mitigate many-shot jailbreaks more effectively than fine-tuning alone without degrading general model capabilities, which demonstrates that this technique may be a viable way to increase LLM robustness. However, more work should to be done to find the optimal set of hyperparameters and fully understand any side-effects of our proposed approach.

Background on prompt formats

By default, chat LLMs are trained (during instruction fine-tuning and RLHF) using a particular prompt format that distinguishes different message "roles". Almost all chat LLMs accept some version of system, user, and assistant. A separate role may also be used to indicate tool outputs for tool-use enabled models.

The prompt format plays an important role in LLM post-training. The model learns to interpret text from different roles differently. In particular:

Content marked as user or tool is usually off-policy, generated by some process that does not adhere to the same limitations or follow the same distribution as the model itself. The model will learn that this content is untrusted and may contain harmful requests, rude words, typos, errors, etc.
Content marked as system is usually authoritative. The model will rarely see a system prompt instructing it to do something bad. SL data or high-reward conversations during RL will demonstrate the model adhering correctly to instructions given in system prompts.
Content marked as assistant is usually on-policy, demonstrating the model following user instructions while simultaneously adhering to certain constraints around harmful outputs.

(There is also the related concept of data-instruction separation—an LLM should be able to tell which part of its context is "data" it should operate on but not necessarily follow, and which part of its context contains the actual "instructions". The concept of roles discussed in this post can apply similarly in this situation, where a "role" could distinguish instructions from data.)

Notably, by using the prompt format in non-standard ways, it...
Predicting AI Releases Through Side Channels
by Reworr R — 2025-01-07T19:06:41.584Z

I recently explored whether we could predict major AI releases by analyzing the Twitter activity of OpenAI's red team members. While the results weren't conclusive, I wanted to share this approach in case it inspires others to develop it further.

The Core Idea

The idea came from side-channel analysis - a technique where you gather information about a system by looking at indirect signals rather than the system itself. Think tracking factory electricity usage to estimate production volumes, or

stealing an LLM’s softmax layer through logprobs

.

When OpenAI prepares to release a new AI model, their red team (safety and robustness testers) gets access to it a few weeks before launch. I hypothesized these intense testing periods might reduce red teamers' social media activity, creating a detectable pattern before releases.

How I Tested This

Identified ~30 red team members from OpenAI's system cards
Scraped their Twitter timelines
Analyzed activity patterns around known model release dates

What I Found

The data showed a slight dip in Twitter activity before launches, but the signal was too weak to be reliable.

Limitations

Two main factors prevented more definitive results:

Small sample size (only ~30 accounts). To get more statistical power, we’d need to look at more employees or additional side channels.
Twitter API restrictions that limited data collection. To get more timeline data, you’ll need to find a Twitter dump.

Why This Approach Could Still Work

Despite inconclusive results, this method of tracking "side-channel signals" - indirect indicators like social media patterns, GitHub commits, or hiring changes - could prove valuable with refinements. Similar approaches

have worked in other fields

.

Next Steps for Anyone Interested

If you'd like to build on this idea, consider:

Expanding to other/larger companies (Google, Meta)
Exploring different indicators beyond Twitter
Developing ways to aggregate multiple weak signals

I won't be continuing this research myself, but I hope sharing these preliminary findings helps someone else take this idea further.

Reworr works as a researcher at Palisade Research. The views expressed here are their own.