trentmkelly/LessWrong-43k huggingface dataset:
Reliable Sources: The Story of David Gerard
by TracingWoodgrains — 2024-07-10T19:50:21.191Z

This is a linkpost for https://www.tracingwoodgrains.com/p/reliable-sources-how-wikipedia-admin, posted in full here given its relevance to this community. Gerard has been one of the longest-standing malicious critics of the rationalist and EA communities and has done remarkable amounts of work to shape their public images behind the scenes.

Note: I am closer to this story than to many of my others. As always, I write aiming to provide a thorough and honest picture, but this should be read as the view of a close onlooker who has known about much within this story for years and has strong opinions about the matter, not a disinterested observer coming across something foreign and new. If you’re curious about the backstory, I encourage you to read my companion article after this one.

Introduction: Reliable Sources

Wikipedia administrator David Gerard cares a great deal about Reliable Sources. For the past half-decade, he has torn through the website with dozens of daily edits—upwards of fifty thousand, all told—aimed at slashing and burning lines on the site that reference sources deemed unreliable by Wikipedia. He has stepped into dozens of official discussions determining which sources the site should allow people to use, opining on which are Reliable and which are not. He cares so much about Reliable Sources, in fact, that he goes out of his way to provide interviews to journalists who may write about topics he’s passionate about, then returns to the site to ensure someone adds just the right quotes from those sources to Wikipedia articles about those topics and to protect those additions from all who might question them.

While by Wikipedia’s nature, nobody can precisely claim to speak or act on behalf of the site as a whole, Gerard comes about as close as anyone really could. He’s been a volunteer Wikipedia administrator since 2004, has edited the site more than 200,000 times, and even served off and on as the site’s UK spokesman. Few people have had more of a hand than him in shaping the site, and few have a more encyclopedic understanding of its rules, written and unwritten.

Reliable sources, a ban on original research, and an aspiration towards a neutral point of view have long been at the heart of Wikipedia’s approach. Have an argument, editors say? Back it up with a citation. Articles should cover “all majority and significant minority views” from Reliable ...
Managing Emotional Potential Energy
by adamShimi — 2024-07-10T18:20:45.640Z
Link: https://epistemologicalfascinations.substack.com/p/managing-emotional-potential-energy

I really need a holiday.

I’m feeling stressed and trapped by my responsibilities, I get angry at people and at what they ask of me, I feel miserable at the end of most days.

Now, even just a year ago, I would have looked for a cause outside of myself: my job is unsatisfying, my boss is a dick, my colleagues are terrible, stuff like that… But as I got better and better at reflecting on how I feel, I started to notice again and again the old stoic wisdom that you hurt yourself by your own judgements.

Choose not to be harmed—and you won’t feel harmed. Don’t feel harmed—and you haven’t been.

\- Marcus Aurelius,

Meditations

, 4.7

In almost all cases of my frustrations lately, I’m the one who causes the problem: I’m the one burdening myself with more responsibilities, I’m the one turning asks into shoulds, I’m the one taking badly small comments or decisions that in truth have nothing to do with me…

So there is a mechanism. Yet what is missing from that explanation, and from Stoicism in general, is a further explanation of why sometimes this psychological self-harm ramps up enormously. Many times, I’m fine handling massive workloads and worrying world events, but last sunday I got overwhelmed and almost had a panic attack while trying to bake foccacia… talk about

giving way too many fucks when none are warranted

.

My pet theory for explaining this effect is “emotional potential energy”.

In classical mechanics (and most of physics),

potential energy

is a trick we use to ensure

conservation of energy

: in a non-dissipative system (where there is no loss of energy, for example through friction), the potential energy is where the energy is stored when the system slows down (that is, when its kinetic energy decreases).

So if you throw a ball upward, its kinetic energy is getting converted into potential energy, until it reaches the highest point of its trajectory, where all the energy is potential (there is no movement and so no kinetic energy); and when it goes down, it starts converting its potential energy back into kinetic energy (accelerating).

I say potential energy is a trick, because the exact value doesn’t matter: what count is that the sum of potential and kinetic energy stays the same. This creates a really nice degree of freedom (what value to give to this sum), which is generally exploited to make the state of rest the one with 0 potential energy, thus simplifying comp...
[EAForum xpost] A breakdown of OpenAI's revenue
by dschwarz — 2024-07-10T18:09:20.017Z
Link: https://forum.effectivealtruism.org/posts/ksdSjT7wREu6vBhYE/a-breakdown-of-openai-s-revenue

We estimate that, as of June 12, 2024, OpenAI has an annualized revenue (ARR) of:

$1.9B for ChatGPT Plus (7.7M global subscribers),
$714M from ChatGPT Enterprise (1.2M seats),
$510M from the API, and
$290M from ChatGPT Team (from 980k seats)

(Full report in https://app.futuresearch.ai/reports/3Li1, methods described in https://futuresearch.ai/openai-revenue-report.)
We looked into OpenAI's revenue because financial information should be a strong indicator of the business decisions they make in the coming months, and hence an indicator of their research priorities.

Our methods in brief: we searched exhaustively for all public information on OpenAI's finances, and filtered it to reliable data points. From this, we selected a method of calculation that required the minimal amount of inference of missing information.

To infer the missing information, we used the standard techniques of forecasters: fermi estimates, and base rates / analogies.

We're fairly confident that the true values are relatively close to what we report. We're still working on methods to assign confidence intervals on the final answers given the confidence intervals of all of the intermediate variables.

Inside the full report, you can see which of our estimates are most speculative, e.g. using the ratio of Enterprise seats to Teams seats from comparable apps; or inferring the US to non-US subscriber base across platforms from numbers about mobile subscribers, or inferring growth rates from just a few data points.

Overall, these numbers imply to us that:

Sam Altman's surprising claim of $3.4B ARR on June 12 seems quite plausible, despite skepticism people raised at the time.
Apps (consumer and enterprise) are much more important to OpenAI than the API.
Consumers are much more important to OpenAI than enterprises, as reflected in all their recent demos, but the enterprise growth rate is so high that this may change abruptly.
Solving Pascal’s Wager using dynamic programming
by Paul Wilczewski — 2024-07-10T18:09:18.014Z

In Infinite Ethics Nick Bostrom argues against using discount factors to resolve problems created by the temporal and spatial infinities that arise in moral theories. While this argument is compelling in the context of aggregative consequentialism it is less compelling under ethical egosim. Incorporating discount factors into ethical models handles many of the problematic infinities and allows us to analyze individual ethical decisions mathematically. In this post I present a model for decision making under uncertainty that incorporates the possibility of eternal reward or punishment.

Motivation

Material conditions for humanity have never been so good yet so many people focus primarily on worldly rewards. Many of these worldly rewards even have diminishing marginal utility. With such abundance for so many I would expect humanity to become more religious and more concerned about what happens after we die. Why has the opposite happened? Perhaps we lack the models needed for evaluating existential matters. Every day we face ethical choices that may or may not have eternal consequences. Despite belief in these consequences even deeply religious people sin sometimes. They risk eternal punishment for some worldly benefit - we should be able to model this behavior. Maybe sin is sometimes rational. Maybe even Faust's bargain was utility optimizing.

In his work Bostrom identifies two types of discount factors, spatial and temporal. The former is easily addressed through ethical egoism. If I am unconcerned about the well-being of others, to the extent it does not impact my well-being, then my spatial discount factor is zero and one class of infinities is resolved. The later can be addressed with temporal discount factors less than one. In economics temporal discount factors are widely used and empirically verified. While Bostrom writes temporal discount factors are "viewed with great suspicion" empirical studies show that people use discount factors when making decisions. These discount factors are commonly used to evaluate potential reward and punishment in the economics of crime literature. For example in their work Crime and Human Nature (1985) Wilson and Herrnstein argue that individuals with high temporal discount factors are more prone to criminal behavior. If sins are crimes that are only punished after we die then we can apply the same decision theoreti...
Fluent, Cruxy Predictions
by Raemon — 2024-07-10T18:00:06.424Z

The latest in the Feedback Loop Rationality series.

Periodically, people (including me) try to operationalize predictions, or bets, and... it doesn't seem to help much.

I think I recently "got good" at making "actually useful predictions." I currently feel on-the-cusp of unlocking a host of related skills further down the rationality tech tree. This post will attempt to spell out some of the nuances of how I currently go about it, and paint a picture of why I think it's worth investing in.

The takeaway that feels most important to me is: it's way better to be "fluent" at operationalizing predictions, compared to "capable at all."

Previously, "making predictions" was something I did separately from my planning process. It was a slow, clunky process.

Nowadays, reasonably often I can integrate predictions into the planning process itself, because it feels lightweight. I'm better at quickly feeling-around for "what sort of predictions would actually change my plans if they turned out a certain way?", and then quickly check in on my intuitive sense of "what do I expect to happen?"

Fluency means you can actually use it day-to-day to help with whatever work is most important to you. Day-to-day usage means you can actually get calibrated for predictions in whatever domains you care about. Calibration means that your intuitions will be good, and you'll know they're good.

If I were to summarize the change-in-how-I-predict, it's a shift from:

"Observables-first". i.e. looking for things I could easily observe/operationalize, that were somehow related to what I cared about.

to:

"Cruxy-first". i.e. Look for things that would change my decisionmaking, even if vague, and learn to either better operationalize those vague things, or, find a way to get better data. (and then, there's a cluster of skills and shortcuts to make that easier)

Disclaimer:

This post is on the awkward edge of "feels full of promise", but "I haven't yet executed on the stuff that'd make it clearly demonstrably valuable." (And, I've tracked the results of enough "feels promise" stuff to know they have a <50% hit rate)

I feel like I can see the work needed to make this technique actually functional. It's a lot of work. I'm not sure if it's worth it. (Alas, I'm inventing this technique because I don't know how to tell if my projects are worth it and I really want a principled way of forming belie

...
Antitrust as Controlled Creative Destruction
by Martin Sustrik — 2024-07-10T16:40:01.503Z

Standard Oil, Refinery No. 1

Splitting large companies is an antitrust measure which, in its essence, is meant as an act of controlled creative destruction. (Read more about controlled creative destruction here.) When a company achieves monopoly status, it often becomes ridden with different inefficiencies and perverse incentives and does not serve its customers very well. By splitting such a company, the aim is to create smaller, more efficient entities driven by the competition in a free market. What is not obvious, but may actually be the case, is that the shakeup of the management hierarchy caused by the split can disrupt extant patronage networks or break different suboptimal equilibria within the company. In this sense it is similar to democracy where such a shakeup happens each time there's a change in government.

An interesting question is what would happen if such splitting was made automatic: When company exceeds certain size, it will split. Period.

This is clearly a candidate for the "The Most Terrible Measure that Should Never have been Implemented" prize, but let's treat it as a harmless thought experiment and think about the possible consequences.

First, it would be nice for large companies to have certainty instead of playing whack-a-mole with the regulators as is the case today. The future would be predictable and the company would be in control. They could choose to grow and split or stay within the size limit and remain intact.

Now, introducing an incentive for limiting growth sounds like a terrible idea. But what it really means depends on how "size" is measured.

If the "size" is based on expenditures, the real incentive would be to maintain current expenses while increasing revenue, effectively boosting productivity. That sounds much better!

Management and shareholders would have the option to enhance revenue through increased productivity or, if that's not possible, to split the company.

From the point of view of the market as a whole, it's actually pretty nice. Companies are pushed to invest in research and development to increase productivity. Those that can't adapt, maybe because they've got too ossified and dysfunctional, may need to split.

Splitting, in turn, causes a shakeup in the management structure and maybe - only maybe! - the new child companies would avoid some of the problems of the old company and thus become more efficient. But gi...
New page: Integrity
by Zach Stein-Perlman — 2024-07-10T15:00:41.050Z

There's a new page collecting integrity incidents at the frontier AI labs.

Also a month ago I made a page on labs' policy advocacy.

If you have suggestions to improve these pages, or have ideas for other resources I should create, let me know.

Crossposted from AI Lab Watch. Subscribe on Substack.
AirBnB Baking
by jefftk — 2024-07-10T12:50:03.381Z

I like making food at home, where I know my kitchen and ingredients.
I have a several things I

make often, and while
I sometimes make new things it's all a bit familiar. So when I'm on a
vacation in a rental house, one thing I enjoy is cooking. It's a bit
of a challenge, perhaps suitable for a low-stakes cooking show, but
it's fun to see what I can make with the constraints:

What pans are there? Do they have any sneaky downsides?

What non-perishable ingredients are left over from previous
visitors?

What's the minimum I can buy at the store to combine with
what's already there and make something good?

If we went to the store a few days ago and it's not worth going
again, what's a good fit for the collection of things we have left?

What can I substitute for measuring cups? How can I calibrate
them?

If I would normally make a recipe one way, but we don't have
some of the ingredients, what would make good substitutes?

One constraint I don't enjoy is knives that have been treated quite
badly. One way I'll "cheat" is that (when I remember) I'll bring
along a knife sharpener. I'll also sometimes bring an instant read
thermometer, and somewhat unusual dry ingredients (ex: flax [1]) that I
need small quantities of.

This morning I made raspberry chocolate chip muffins (recipe):

This involved using an unlabeled measuring cup that might have been
either 1/4C or 1/3C (which I treated as 1/4C and used for everything,
even the 2.5C flour), substituting extra baking powder for the baking
soda, and bigger chocolate chips than usual. They were tasty!

[1] Lately I've been using flax as an egg substitute a lot; Lily
doesn't eat eggs anymore. I used 2.5T water and 1T flax per egg. It
works well in the raspberry muffins, ok in crepes, and very poorly in
brownies.
DIY RLHF: A simple implementation for hands on experience
by Mike Vaiana — 2024-07-10T12:07:03.047Z

Many thanks to Diogo de Lucena, Cameron Berg, Judd Rosenblatt, and Philip Gubbins for support and feedback on this post.

TL;DR

Reinforcement Learning from Human Feedback (RLHF) is one of the leading methods for fine-tuning foundational models to be helpful, harmless, and honest. But it’s complicated and the standard implementation requires a pool of crowdsourced workers to provide feedback. Therefore we are sharing our implementation of RLHF with two goals in mind:

It should be easy for a single user to complete an RLHF process end-to-end
It should be easy for students and researchers to understand the code (and modify it if they want).

Intro

Reinforcement Learning from Human Feedback (RLHF) is a popular technique to fine-tune language models that is reasonably effective at producing a final model which is generally helpful, harmless, and honest. But RLHF is not a perfect solution, and it is well documented [e.g. here, here, and here] that it is fairly easy to prompt an LLM to generate harmful or dishonest content. Despite these problems RLHF is still the de facto standard method for aligning large language models and therefore the method has garnered interest from the alignment community. However, RLHF has a reasonably complicated implementation with large scale training processes that also require a large team of human labelers.

In this report, our aim is to enhance the accessibility of RLHF so that students and researchers can engage with an implementation to gain deeper understanding and insight of how the full process works, end-to-end. We hope this will encourage further research and help overcome existing challenges in safety fine-tuning. To facilitate this, we are making available a code base that executes the complete RLHF process within the Atari gym environment, identical to the setting used in the original RLHF paper.

Note that we do not aim to completely explain RLHF (although we’ll discuss parts of it) but rather we seek to introduce a working implementation of the technique. There are several articles that provide more details and refer you to the original paper for an introduction.

Preference Learning

There are two motivations for learning from human preferences. First, it may be impossible to specify a reward function for a given task. For example, RLHF has been used to train a simulated ...
Usefulness grounds truth
by invertedpassion — 2024-07-10T07:58:03.095Z

Crosspost from https://invertedpassion.com/usefulness-grounds-truth/

Are LLMs intelligent?

Debates on this question often, but not always, devolve into debates on what LLMs can or cannot do. To a limited extent, the original question is useful because it creates an opening for people to go into specific. But, beyond that initial use, the question quickly empties itself because (obviously) the answer to the question if X is intelligence depends on how you define intelligence (and how you define X).

Even though it is clear that words are inherently empty, internet is full of such debates. People focus on syntax, when semantics is what runs the world.

There’s no

Platonic realm

teeming truth that’s disconnected from the world we inhabit. If it existed, the debate on what’s true would shift to the question of who has access to that realm. Is it the scientists? Is it the Pope? Or is it your neighbourhood aunty?

We, fortunately, live in the modern world where everyone is entitled to their opinions. Someone says God exists. The other person says it’s clear that God doesn’t exist. (I say it depends on the definition but that’s boring and nobody wants to hear that)

So, in a sea of opinions, how do you distinguish truth?

The trick is to reframe the question: instead of asking what’s true, ask what’s useful.

The kind of usefulness I’m talking about here is like, but not limited to, the usefulness of a kitchen-knife. Just like a knife helps you slice tomatoes to make a sandwich for yourself when you’re hungry, “truths” are different tools in your arsenal that you could use to (potentially) make a difference in your life or the world at large (if that’s the kind of thing you care about).

We know 1+1 is 2 because it enables us to do simple accounting of objects and get ahead of other animals who can’t count. We know the sun rises from the east because this knowledge enables us to build houses with windows that stream sunlight into our bedroom just as we’re waking up (and, of course, also launch satellites).

I am walking in the footsteps of William James who founded

Pragmatism

. Breaking away from the philosophical tradition of swimming in abstractions, he preached asking whether something makes a real difference or not. Without a focus on real world impact, questions and debates often remain circular. Take, for example, the innocuous question: “do you love me?” It’s an empty question...
On passing Complete and Honest Ideological Turing Tests (CHITTs)
by Aryeh Englander — 2024-07-10T04:01:33.567Z

[Cross-posted from Facebook]

"In the moment when I truly understand my enemy, understand him well enough to defeat him, then in that very moment I also love him. I think it’s impossible to really understand somebody, what they want, what they believe, and not love them the way they love themselves."

- Ender Wiggin (from Ender's Game, by Orson Scott Card)

Maybe the following is obvious, but it wasn't initially obvious to me so I'm sharing it anyway.

The term Ideological Turing Test (ITT) was coined by Bryan Caplan and refers to whether someone can articulate and argue the views of an opposing ideology or belief system so convincingly that others cannot tell the difference between the person's articulation and that of a true believer.

Let's use the term Complete and Honest Ideological Turing Test (CHITT) to refer to an ITT where not only can you "fool" observers, but also:

You can do so without resorting to any rhetorical tricks such as leaving out relevant information;
You can do so without making the other side look irrational;
You can do this even for very long debates with expert observers who grill you closely;
You can do this in response to any evidence or argument or counterargument that you yourself might use to argue for your true position.

Claim: It's impossible to pass a CHITT unless you actually agree that from some perspective the other side is rational. Furthermore, you must not have any knockdown arguments for why that other perspective is incorrect. That doesn't mean you need to *agree* with the other perspective, just that you don't have a knockdown argument against it.

Reasoning: By the specified criteria, you need to think that no matter what arguments you yourself would throw against the other side, you would not make the other side look irrational even to expert observers. You can of course still have some set of priors that lead you to your true position, while the other side has different priors that lead to their position. But why do you go with your priors and not theirs? If you have a knockdown argument for why your priors are correct, then that's part of the arguments you could throw at the other side, and we're back to where we were before. So it must be that you don't have any knockdown arguments for why you should go with your priors over theirs.