trentmkelly/LessWrong-43k huggingface dataset:
There is way too much serendipity
by Malmesbury — 2024-01-19T19:37:57.068Z

Crossposted from substack.

As we all know, sugar is sweet and so are the $30B in yearly revenue from the artificial sweetener industry.

Four billion years of evolution endowed our brains with a simple, straightforward mechanism to make sure we occasionally get an energy refuel so we can continue the foraging a little longer, and of course we are completely ignoring the instructions and spend billions on fake fuel that doesn’t actually grant any energy. A classic case of the Human Alignment Problem.

If we’re going to break our conditioning anyway, where do we start? How do you even come up with a new artificial sweetener? I’ve been wondering about this, because it’s not obvious to me how you would figure out what is sweet and what is not.

Look at sucrose and aspartame side by side:

I can’t imagine someone looking at these two molecules and thinking “surely they taste the same”. Most sweeteners were discovered in the 20th century, before high-throughput screening was available. So how did they proceed?

Let’s look into these molecules’ origin stories.

Aspartame was discovered accidentally by a chemist researching a completely unrelated topic. At some point, he licked his finger to grab a piece of paper and noticed a strong sweet taste.

Cyclamate was discovered by a grad student who put his cigarette on his bench, then smoked it again and noticed the cigarette was sweet.

(I know what you’re thinking. The kind of guy who lights up cigarettes in a chemistry lab and places them in the middle of uncharacterised compounds before taking them to his mouth again, must have died young of an interesting death. I checked – he proceeded to live to the old age of 87.)

Saccharine was discovered by a researcher who ate bread without washing his hands and noticed the bread was sweet.

Acesulfame K was also discovered serendipitously by a chemist licking his fingers, although the legends don’t specify the exact circumstances behind the finger-licking.

There’s an exception: sucralose was actually the fruit of rational, deliberate design. The researchers reasoned that, if you do slight modifications to sucrose, you could find a molecule that is no longer metabolized but still activates the sweetness receptors. So they started from the formula for sucrose, then made carefully-designed chemical modifications to the structure until –

Haha, just kidding:

While researching novel uses of sucrose and i...
Estimating efficiency improvements in LLM pre-training
by Daan — 2024-01-19T19:32:45.124Z

TL/DR:

GPT-3 was trained on 10,000 NVIDIA V100’s for 14.8 days. Based on my - very uncertain - estimates, OpenAI should now be able train a comparably capable model on an equivalent cluster in under an hour. This implies a drop in hardware requirements of more than 400x.

Factoring out increases in hardware efficiency, I arrive at a central estimate of 170x fewer FLOP to reach GPT-3 level performance today.

By comparison, GPT-3 to GPT-4 was an estimated 40x increase in pf16-equivalent GPU-hours, and an estimated 68x increase in FLOP. I expect OpenAI to have access to significantly more resources today though, than when pre-training GPT-4. This suggests that resources and resource efficiency are growing at roughly similar speeds.

Introduction

Suppose we’re comparing two base models. Using the same inference setup, we find that Model A outperforms Model B across a wide range of benchmarks. What might have caused this difference in performance? Presumably, Lab A beat Lab B on at least one of these six factors:

Cluster size. Example: Model A was trained on 25,000 GPUs whereas Model B was trained on only 10,000 GPUs.
GPU performance. Example: Model A was trained on H100’s whereas Model B was trained on A100’s.
Training duration. Example: Model A was trained for 100 days whereas Model B was trained for only 30 days.
Hardware efficiency. Example: Lab A achieved 40% pre-training MFU whereas Lab B only achieved 30% MFU[1].
Algorithmic efficiency. Example: Model A was trained using a Chinchilla-optimal number of parameters, whereas Model B was not.
Data quality. Example: Model A was trained on a highly curated dataset whereas Model B was trained on some random web-scrape.

Note that data quantity is implicit in this taxonomy: it is determined by factors 1 through 5.

Here's how I like to think about this framework:

Factors 1, 2 and 3 determine the hardware an AI lab has access to.
Factors 4, 5 and 6 determine the hardware requirements - that is, the hardware necessary to reach some performance threshold.
Factors 1 through 4 determine the compute used to train the model.
Factors 5 and 6 determine the compute requirements - that is, the compute necessary to reach some performance threshold.
Finally, the six factors combined determine the effective compute used to train a model.

I'm also going to introduce two umbrella terms: resour...
I Want XMP But I Know Why I Can't Have It
by jefftk — 2024-01-19T15:30:07.492Z

When writing text that will be displayed to users as HTML I write
HTML. This seems like it would be the normal way to do it, though
it's unusual these days. Either you draft in a

fancy content editor
or you write

Markdown. HTML
authoring is a bit

retrogrouch but it
suits me. Except when I need to write about HTML itself.

In my previous post I was talking
about how I write recipes in HTML:

<li>2 eggs (or 2T flax and 5T water)
<li>2/3 C oil
<li>1C greek yoghurt
<li>1/4 cup milk, more if needed

Except in the blog post itself the ampersands needed to be escaped, so
on my screen it looked like:

<pre>
&lt;li&gt;2 eggs (or 2T flax and 5T water)
&lt;li&gt;2/3 C oil
&lt;li&gt;1C greek yoghurt
&lt;li&gt;1/4 cup milk, more if needed
</pre>

This is pretty painful. What if I could just write HTML, but mark it
off as an example of how to write HTML so the browser would ignore it?
This was exactly the problem Berners-Lee
had when initially documenting HTML, and if you look at early
pages you'll see he introduced an <xmp> tag:

The title of a document is given between title tags:
<XMP><TITLE> ... </TITLE></XMP>

This is great! Can I just write:

<xmp>
<li>2 eggs (or 2T flax and 5T water)
<li>2/3 C oil
<li>1C greek yoghurt
<li>1/4 cup milk, more if needed
</xmp>

Let's go look up the documentation
first:

Oh dear. Deprecated since at least 1995.
Now, it does still work when I try it in Chrome and Firefox (would
they ever break the original HTML pages?) but I'm not going to use it
here in case it doesn't work in RSS readers. But why would they take
away this great feature?

When I look at it today, the obvious answer is security. Someone is
going to generate a page with:

print("You wrote <xmp>" +
userString +
"</xmp>")

And someone will give the string "</xmp><script>...".
That's an XSS vulnerability.

But when this was deprecated (1995 or before) XSS wasn't possible: JS
only came out in December
1995. XSS didn't become something people were really thinking
about until more like 2000. [1]

Instead, it was an SGML
issue:

An earlier HTML specification included an XMP element whose syntax is
not expressible in SGML. Inside th

...
Arguments for Robustness in AI Alignment
by Fabian Schimpf — 2024-01-19T10:24:38.507Z

Most popular robustness papers address short-term problems with robustness failures in neural networks, e.g., in the context of autonomous driving. I believe that robustness is also critical for the long-term success of AI alignment and ending the discussion with a statement along the lines of "Meh, this doesn't seem solvable; also, our benchmarks are looking good and more capable models won't be fooled like this" doesn't cut it.

Reasons to be concerned:

Epistemic status: I thought a lot about robustness for the past half year or so and found the arguments convincing. However, there are certainly more arguments, and I'll keep updating the list. Ideally, this will eventually turn into a quick survey of the arguments to quantify what people are actually concerned about.

- Remote code execution through jailbreaks seems obviously bad and is treated as a crucial threat in IT security.

- Failure to manage and maintain the resulting complexity when AI are part of larger systems deployed in complex and potentially adversarial settings, e.g., with other AIs.

- There is evidence that humans react to similar attacks like ensembles of CV models, which could enable the creation of super-stimuli.

- Transfer attacks like GCG will likely continue to be possible as there aren't many internet-scale text datasets. Hence, it seems plausible that multiple models will share similar failure modes and use the same non-robust predictive features.

- We don't have an angle on the problem that addresses the cause at scale, only empirically useful treatment of symptoms like adversarial training outside the scope of lp-attacks, which can be handled with certifiable methods.

- Just because we fail to find adversarial examples empirically doesn't mean there aren't any. Humans have been able to semantically jailbreak LLMs with patterns that no optimizer found yet. Thus, it seems plausible that we don't even know how to test if LLMs are robust.

- It might not matter if advanced AI systems share our values and act in humanity's best interest if models are vulnerable to "erratically" misbehaving badly and are leveraged sufficiently.

- As long as LLMs become more capable while remaining vulnerable, it seems likely that Yudkowsky's quip of Moore's Law of Mad Science: "Every 18 months, the minimum IQ to destroy the world drops by one point" might be accurate.
Acknowledgements:
Thanks t...
What rationality failure modes are there?
by Ulisse Mini — 2024-01-19T09:12:57.924Z

How do people fail to improve their rationality? How do they accidentally harm themselves in the process? I'm thinking of writing a post "How not to improve your rationality" or "A nuanced guide to reading the sequences" that preempts common mistakes, and I'd appreciate hearing people's experiences. Some examples:

It took me an absurdly long time (like, 1-2yr in the rat community) before I realized you don't correct for cognitive biases, you have to "be introspectively aware of the bias occuring, and remain unmoved by it" (as Eliezer put it in a podcast)

More generally, people can read about a bias and resolve to "do better" without concretely deciding what to do differently. This typically makes things worse, e.g. I have a friend who tried really hard to avoid the typical mind fallacy, and accidentally turned off her empathy in the process.

The implicit frame rationalists push is logical and legible, and can lead to people distrusting their emotions. And I think it's really important to listen to listen to ick feelings when changing your thought processes, as there can be non obvious effects.

E.g. My friend started thinking about integrity in terms of FDT, and this disconnected it from their motivational circuits and they made some pretty big mistakes because of it. If they'd listened to their feeling of "this is a weird way to think" this wouldn't have happened.

(I think many people misinterpret sequence posts and decide to change their thinking in bad ways, and listening to your feelings can be a nice emergency check.)
What’s up with online media and our ability to get sh*t done?
by TeaTieAndHat — 2024-01-19T09:12:14.600Z

Does being online destroy our attention? Have our attention spans been declining?

What I’ve read on the topic says "Not really". More specifically:

First, there’s that SMTM post (published as part of their Mysteries contest and actually written by this person). What do we learn from this one?
First, there are various kinds of attention: sustained attention, the ability to keep your mind on a specific task for about 10 minutes [no one knows where the "10 min" comes from]; selective attention, the ability to resist distractions that pop up while you’re focusing on something (yes, the gorilla thing); and divided attention, the ability to multitask. But we also learn that "attention span" isn’t really a constant the way, say, Dunbar’s number is supposed to be, but is instead very task-dependent. We also learn that all existing research on attention spans is pretty terrible. But, according to a couple of those terrible studies: sustained attention is (read "might be". Terrible study.) a little worse in young people and people who are online a lot. Selective attention maybe is lower maybe not, and divided attention (multitasking) is ("might be") actually better in young people and people who are online a lot. According to other studies, the time spent on a given website, or the time a hashtag stays popular, have fallen. Which may or may not have much to do with what we’re actually interested in, and, like the multitasking thing, may or may not actually be a problem (is it that we can’t stay focussed, or that we’re better at prioritising tasks and shifting from one to the other?). Apparently, some better-quality research also tells us that the length of each shot in films or TV commercials has declined. I’m not sure whether to see it as proof that we’re now more interested in faster-paced stuff because our attention is worse, or if the greater availability of faster-paced stuff overwhelms us in some way and contributes to making us think our attention spans have declined whether or not it actually is the case, or something else entirely. In any case, people certainly believe attention spans have been declining, but we knew that already. So, there aren’t masses of convincing evidence either way here. The author of that blog post gives ≈65% to the hypothesis that there’s been a decline in the last couple decades.

More information on the topic is provided in a...
Logical Line-Of-Sight Makes Games Sequential or Loopy
by StrivingForLegibility — 2024-01-19T04:05:44.782Z

In the last post, we talked about strategic time and the strategic time loops studied in open-source game theory. In that context, agents have logical line-of-sight to each other and the situation they're both facing, which creates a two-way information flow at the time each is making their decision. In this post I'll describe how agents in one context can use this logical line-of-sight to condition their behavior on how they behave in other contexts. This in turn makes those contexts strategically sequential or loopy, in a way that a purely causal decision theory doesn't pick up on.

Sequential Games and Leverage

As an intuition pump, consider the following ordinary game: Alice and Bob are going to play a Prisoners' Dilemma, and then an Ultimatum game. My favorite framing of the Prisoners' Dilemma is by Nicky Case: each player stands in front of a machine which accepts a certain amount of money, e.g. $100.[1] Both players choose simultaneously whether to put some of their own money into the machine. If Alice places $100 into the machine in front of her, $200 comes out of Bob's machine, and vice versa. If a player withholds their money, nothing comes out of the other player's machine. We call these strategies Cooperate and Defect respectively.

Since neither player can cause money to come out of their own machine, Causal Decision Theory (CDT) identifies Defect as a dominant strategy for both players. Dissatisfaction with this answer has motivated many to dig into the foundations of decision theory, and coming up with different conditions that enable Cooperation in the Prisoners' Dilemma has become a cottage industry for the field. I myself keep calling it the Prisoners' Dilemma (rather than the Prisoner's Dilemma) because I want to frame it as a dilemma they're facing together, where they can collaboratively implement mechanisms that incentivize mutual Cooperation. The mechanism I want to describe today is leverage: having something the other player wants, and giving it to them if and only if they do what you want.

Suppose that the subsequent Ultimatum game is about how to split $1,000. After the Prisoners' Dilemma, a fair coin is flipped to determine Alice and Bob's roles in the Ultimatum game. The evaluator can employ probabilistic rejection to shape the incentives of the proposer, so that the proposer has the unique best-response of offering a fair split. ...
Are there high-quality surveys available detailing the rates of polyamory among Americans age 18-45 in metropolitan areas in the United States?
by Evan_Gaensbauer — 2024-01-18T23:50:52.053Z

In recent years, there has been more public attention paid to seemingly increasing rates of less common identity characteristics or alternative lifestyle choices. This has provoked widespread media speculation that the trends in question are social contagions being spread by the communities or subcultures where such trends are most apparent. Examples of such trends include dramatically increased rates of reported self-identification with sexual or gender minority status; various kinds of neurodivergence, and polyamory/non-monogamy. Without denying the possibility that 'social contagion' may be a minor causal factor among many for increased rates of self-identification with the above characteristics, there are of course better explanations. Yet the narrative that these trends are becoming more prominent mostly as a consequence of social pathology persists. This specious presumption could be significantly mitigated by evaluating general causes, which in turn might be identified through knowing more about the base rates, and increase in the rate of change, for these personal/social characteristics among the general population.

Of course, on LessWrong, this will bring to mind how higher rates of polyamory among the rationality community, and related communities like effective altruism, are exploited to stigmatize them by various hostile parties. Yet I've noticed this is an even greater issue faced by larger social classes among younger generations. While public discourse has also provoked some defence of various subcultures embracing polyamory, I've noticed almost nobody on any side of the conversation bothers evaluating whether there's a general increase in rates of non-monogamy uncorrelated with increased rates of membership in alternative subcultures. To have such information would be useful for answering many unresolved questions about the status of non-monogamy in society at large.

I request data on a population sample with the particular characteristics mentioned in the title because those are the common characteristics of the non-monogamous people generally subject to so much public speculation. I'm not super well-versed in the finer details for what counts as a "higher-quality" survey, so I welcome you to make that judgment for yourself. Suggestions for other online spaces where I could ask this question which may have more insight are also welcome.
Manifund: 2023 in Review
by Austin Chen — 2024-01-18T23:50:13.557Z
Link: https://manifund.substack.com/p/manifund-2023-in-review

Manifund is a new funding org that experiments with systems and software to support awesome projects. In 2023, we built a website (

manifund.org

) and donor ecosystem supporting three main programs: impact certificates, regranting, and an open call for applications. We allocated $2m across dozens of charitable projects, primarily in AI safety and effective altruism cause areas. Here’s a breakdown of what Manifund accomplished, our current strengths and weaknesses, and what we hope to achieve in the future.

If you like our work, please consider

donating to Manifund

. Donations help cover our salaries & operating expenses, and fund projects and experiments that institutional donors aren’t willing to back — often the ones that excite us most!

At a glance

Here are some high-level stats that provide a snapshot of our 2023 activities:

$2.06M sent to projects: $2.012M to grants & $45K to impact certificates
Of the totals above, $95K that went to grants and $40K that went to certs came from unaffiliated donors/investors, rather than regrantors.
88 projects were funded: 54 grants & 34 certs
$2.22M has been deposited into Manifund, and $1.62M has been withdrawn so far.
Below are the top cause areas of projects that got funded. Note that these are overlapping, that is, one project can be filed under multiple cause areas.

Technical AI Safety: 27 projects funded, $1.57M dispersed
Science and Technology: 9 projects funded, $118K dispersed
AI Governance: 10 projects funded, $112K dispersed
Biosecurity: 4 projects funded, $97K dispersed
Honorable mention to Forecasting, which only received $76K total, but encompassed 35 projects. This is because our two biggest impact certificate rounds so far—ACX Mini-Grants and the Manifold Community Fund—were centered around forecasting and funded lots of small projects.

2023 Programs

Impact certificates

Summary: Impact certificates are venture funding for charitable endeavors. Investors fund founders by buying shares (”certs”) in their projects, which pay out if the project later receives a retroactive prize. See also

our

and

ACX’s

explainers.

Assessment: 7/10

Impact certs have been discussed as a much more efficient system of funding charitable projects. Luminaries such as

Paul Christiano, Katja Grace

,

Vitalik Buterin

and

Scott Alexander

have championed the idea, but apart from a

few small

experiments

with retroactive funding, no notable ...
The Underreaction to OpenAI
by Sherrinford — 2024-01-18T22:08:32.188Z

TL; DR

Seeing OpenAI's impressive technology should make media commentators think it more likely that AGI is possible and existentially dangerous. Some do, but overall this shift in views does not seem consistent enough; the default reaction still seems to be to view the whole AI thing as just another profitable technology development.

People should be very impressed by recent AI developments

If you did not pay much attention to AI before ChatGPT and then started, you were very likely surprised and impressed. This may even be true if you closely followed AI developments, and even if GPT-3.5 did not impress you, GPT-4 likely did. This also seems to be true of many journalists; AI reporting has become a topic for mainstream media. LLMs (a word that only AI experts knew 2 years ago) are very powerful, and even if you don't immediately see how they will transform the economy, you feel their power if you use them. The same is true of image generation or voice generation AI. The power of translation and other applications like protein folding may be less immediately obvious, but they add more facets of AI power.

Yet there is a strange missing reaction of society's mainstream to these developments, which became particularly visible in the reporting and commentary on the firing of Sam Altman as CEO of OpenAI.

The magician who creates AI

Imagine a person came to you and claimed to be a magician. She had created a magic workshop in order to summon a super-powerful demon, and on the way there she will create smaller versions of the demon. You laugh because demons don't exist. But she creates the small demons. These demons are not super-powerful, but you would not have expected them to be possible. How should the small demons change your mind about the possibility of a super-powerful one?

OpenAI claims that it can create and also tame the demon called AGI. But media coverage usually ignores that it has demonstrated the ability to create small ones explicitly as steps toward creating the super-powerful one.

Consider the following claims that you could have had opinions on in January 2019:

Within five years, someone can create an AI that can write texts that pass as term papers, and that can generate images that will look like photos or complex software-generated graphics, in reaction to commands entered in dialogues with the AI that feel like talking.
Someone can creat

...