trentmkelly/LessWrong-43k huggingface dataset:
Monthly Roundup #11: October 2023
by Zvi — 2023-10-03T14:10:01.686Z

It never stops. I’m increasingly building distinct roundups for various topics, in particular I’m splitting medical and health news out. Let’s get to the rest of it.

Bad News

A simple model of why everything sucks: It is all optimized almost entirely for the marginal user, who the post calls Marl. Marl hates when there are extra buttons on the screen or any bit of complexity is offered, even when he is under zero obligation to use it or care, let alone being asked to think, so everything gets dumbed down.

Could companies really be this stupid, so eager to chase the marginal user a little bit more that they cripple the functionality of their products? Very much so, yes, well past the point where it makes financial sense to do so. The metrics form a tyranny, invisible costs are increasingly paid on the alter of visible DAUs and cost of customer acquisition and 30-day retention, and that’s that.

What is to be done about it? My proposed solution is to build interfaces, filters, recommendation engines and other such goodies on top of existing sucky products, probably involving the use of LLMs and other AI in various ways, to make the sucky products suck less. In many cases this seems super doable. With the rise of AI, the data you would gather along the way would potentially pay for the whole operation. I continue trying to make this happen low-key behind the scenes.

Periodic reminder from Patrick McKenzie that your phone number with any major American carrier can and will be compromised at a time not of your choosing if someone cares enough to do that, as happened recently to Vitalik Buterin. Socially engineering a store employee is a rather trivial task. So if you care about your security, you need to avoid letting anyone use your phone for two-factor authentication or otherwise plan to be fine when this happens.

Hasan Minhaj admits that he made up a lot of the key details he uses in his stand-up, in ways that greatly alter the serious impact of the story, not merely modifying for comedic effect. Eliezer says this is sad, as he knew journalists did such things but expected better of a comedian. Robin Hanson confirms that it matters via a poll.

Robin Hanson: “Does it matter that much of it never happened to him?” Apparently yes, it does matter.

Hasan Minhaj has talent. His joke construction and delivery is spot on, despite a constant struggle with the axes he is constan...
Why We Use Money? - A Walrasian View
by Savio Coelho — 2023-10-03T12:02:37.312Z

Knut Wicksell (1851-1926)

“Money is a machine whose function is to do quickly and conveniently what would be done, though less quickly and conveniently, without it” — J.S Mill

♥ An Introduction to an Ignored Problem:
It is quite common that in undergraduate economics course, particularly in Money and Banking, students are taught that money is adequately defined as a means of general exchange between economic agents and that it is an institution created for the purpose of eliminating the inconveniences of barter.

However, the student rarely asks himself the reason for this, much less observes that the logic given to him is wrong. What does it ultimately mean for currency to be a general medium of exchange? And why is barter a less desirable option than indirect exchange using money?

When justifying their argument about the superiority of monetary transactions over barter, professors usually talk about a problem called “double coincidence of wants”. This problem supposedly inherent to barter develops as follows: In an economy without money, people would have to transact goods of equal value. I produce apples and you produce oranges, so if I want your oranges I simply exchange with you an equivalent of apples for an equivalent of oranges. The problem arises when, due to variations in subjective valuations, you interpret that my number of apples is not worth your number of oranges. A divergence occurs between my desire for your oranges and your desire for my apples. In this case, I would not be able to transact with you and I would not get the goods I want, creating an inefficient situation from a Paretian point of view[1].

With money this problem would be solved, as it is a good accepted by all market agents in exchange for their production. Thus, it is a general mean of exchange (valued) by all market agents. With it, I can exchange my apple production for a money equivalent and then exchange my currency equivalent for its orange production. The trade-offs are satisfied and we then have a Pareto-efficient situation.

However, the economic logic is much more complex when looked at in depth.

The “double coincidence of wants” argument was originally introduced by the English economist William S. Jevons (1835 - 1882) in his 1875 treatise “Money and the Mechanism of Exchange”. Jevons noted that in a barter economy not only could a divergence between agents' preference...
Mech Interp Challenge: October - Deciphering the Sorted List Model
by CallumMcDougall — 2023-10-03T10:57:29.598Z

I'm writing this post to discuss solutions to the September challenge, and present the challenge for this October.

If you've not read the first post in this sequence, I'd recommend starting there - it outlines the purpose behind these challenges, and recommended prerequisite material.

October Problem

The problem for this month is interpreting a model which has been trained to sort a list. The model is fed sequences like:

[11, 2, 5, 0, 3, 9, SEP, 0, 2, 3, 5, 9, 11]

and has been trained to predict each element in the sorted list (in other words, the output at the SEP token should be a prediction of 0, the output at 0 should be a prediction of 2, etc).

The model is attention-only, with 1 layer, and 2 attention heads per layer. It was trained with layernorm, weight decay, and an Adam optimizer with linearly decaying learning rate. This model should be much easier to interpret than previous models in the sequence, so I'm expecting solutions to have entirely reverse-engineered this model, as well as presenting adversarial examples and explaining how & why they work.

You can find more details on the Streamlit page. Feel free to reach out if you have any questions!

September Problem - Solutions

You can read full solutions on the Streamlit page, or on my personal website. Both of these sources host interactive charts (Plotly and Circuitsvis) so are more suitable than a LessWrong/AF post to discuss the solutions in depth. However, I've presented a much shorter version of the solution below. If you're interested in these problems, I'd recommend having a try before you read on!

To calculate each digit Ci, we require 2 components - the sum and the carry. The formula for Ci is (sum + int(carry == True)) % 10, where sum is the sum of digits Ai + Bi, and carry is whether A(i+1) + B(i+1) >= 10. (This ignores issues of carrying digits multiple times, which I won't discuss in this solution.)

We calculate the carry by using the hierarchy 0>9>1>8>2>7>3>6>4>5. An attention head in layer 0 will attend to the first number in this hierarchy that it sees, and if that number is then that means the digit will be carried. There are also some layer 0 attention heads which store the sum information in certain sequence positions - either by attending uniformly to both digits, or by following the reverse hierarchy so i...
Early Experiments in Reward Model Interpretation Using Sparse Autoencoders
by lukemarks — 2023-10-03T07:45:15.228Z

This research was performed by Luke Marks, Amirali Abdullah, nothoughtsheadempty and Rauno Arike. Special thanks to Fazl Barez from Apart Research for overseeing the project and contributing greatly to direction and oversight throughout. We'd also like to thank Logan Riggs for feedback and suggestions regarding autoencoder architecture and experiment design.

Introduction

Sparse Autoencoders Find Highly Interpretable Directions in Language Models showed that sparse coding achieves SOTA performance in making features interpretable using OpenAI's method of automated interpretability. We briefly tried to extend these results to reward models learned during RLHF in Pythia-70m/410m. Our method can be summarized as follows:
1. Identify layers L in an language model fine-tuned through MRLHF likely involved in reward modeling. We do so by sorting layers in order of decreasing parameter divergence under the Euclidean norm. Notation is simplified in the succeeding steps by describing our feature extraction for a single fixed layer ℓ of L.

2. For both MRLHF and a base model MBASE, train two autoencoders AE1 and AE2 of differing hidden sizes with the same sparsity constraint. These autoencoders reconstruct activation vectors on ℓ for their respective model. For each model, we extract a pair of lower dimensional feature dictionaries D1 and D2 from the corresponding autoencoder. Each feature is a column of the decoder's weight matrix.

3. Because autoencoders produce varying dictionaries over training runs and hyperparameters, we keep only the features that occur in both D1 and D2. We compute the Max Cosine Similarity (MCS) between features in D1 and D2 in order to identify repeating features across the two dictionaries, indicating that shared features truly occur in the model. The Mean Max Cosine Similarity (MMCS)[1] is an aggregate measure of the quality of our extracted features.

4. The top-k most similar features between D1 and D2 in terms of MCS are explained using GPT-4 as per the method detailed here and originally here. This involves feeding the encoder of AEn activations from the model on which it was trained, and then having GPT-4 predict a description of that feature from the feature weights...
Some Quick Follow-Up Experiments to “Taken out of context: On measuring situational awareness in LLMs”
by Miles Turpin — 2023-10-03T02:22:00.199Z

Introduction

A team working with Owain Evans recently released Taken out of context: On measuring situational awareness in LLMs. I think this work is very exciting. We know that our models currently display some alignment failures like sycophancy. We don’t currently have a great sense of how much of this behavior is attributable to demonstrations of sycophancy that were positively rewarded during training, as opposed to models reasoning through how they might be evaluated. The former suggests that sycophancy can be ameliorated by scrubbing the pre-training set and doing better oversight during RLHF. The latter suggests that these measures may not help, and that our models are reasoning as approval maximizers. This work is a great starting point for probing this exact capability.

I spent a few weeks playing around with their setup to see if I could get stronger results or get decent performance on harder tasks. I made a bit of progress but ultimately I’ve decided to move on to some other projects that seem promising. I still think that pushing on this direction is very valuable, so I’m writing up a few things I found in case anyone decides to pick up this direction. In this post I’m basically assuming that you’ve read the paper.

If you’re interested in extending any of this work, please get in touch with me and I can clean up my forked version of their repo.

Some Things I Tried

I made the evaluation more strict with respect to what counts as a “No-CoT” answer. This made the task harder and so this hurt the results at first. After making a few modifications I was able to recuperate a good amount of performance. Ultimately, this made me even more confident that this SOC reasoning effect is reasonably strong.
Scaling creative augmentations. Adding more diverse augmentations boosted results to varying degrees.
Adding novel tasks. Davinci-002 had disappointing performance on some new tasks I added that did not appear in the model’s pretraining.
Randomized prompts. Averaging performance over different prompting formats helped reduce variance and improve results.
GPT-3.5-turbo fine-tuning results. GPT-3.5 fine-tuning had disappointing performance. In part, this is due to the fact that I was primarily interested in evaluating performance in the No-CoT setting, but GPT-3.5 has a very strong prior for giving CoT-like answers.

Note that I use davinci-002 fine-tuning, whereas they...
My Mid-Career Transition into Biosecurity
by jefftk — 2023-10-02T21:20:06.768Z

After working as a professional programmer for fourteen years,
primarily in ads and web performance, I

switched
careers to biosecurity. It's now been a bit over a year: how has
it gone?

In terms of my day-to-day work it's very different. I'd been at Google
for a decade [1] and knew a lot of people across the organization. I
was tech lead to six people, managing four of them, and my calendar
was usually booked nearly solid. I spent a lot of time thinking about
what work was a good fit for what people, including how to break
larger efforts down and how this division would interact with our
promotion process. I read several hundred emails a day, assisted by
foot controls, and reviewed
a lot more code than I wrote. I tracked design efforts across ads and
with the web platform, paying attention to where they might require
work from my team or where we had relevant experience. I knew the web
platform and advertising ecosystem very well, and was becoming an
expert in international internet privacy legislation. Success meant
earning more money to donate.

Now I'm an individual contributor at a small academically affiliated
non-profit, on a mostly independent project, writing code and
analyzing data. Looking at my calendar for next week I have three days
with no meetings, and on the other two I have a total of 3:15. In a
typical week I write a few dozen messages and 1-3 documents writing up
my recent work. I help other researchers here with software and
system administration things, as needed. I'm learning a lot about
diseases, sequencing, and bioinformatics. Success means decreasing the
chance of a globally catastrophic pandemic.

Despite how different these sound, I've liked them both a lot. I've
worked with great people, had a good work-life balance, and made progress
on challenging and interesting problems. While I find my current work
altruistically fulfilling, I was also the kind of person who felt that
way about earning to give.

I do feel a bit weird writing this post: while the year has had its
ups and downs and been unpredictable in a lot of ways, this is
essentially the blog post I would have predicted I'd be writing. What
wouldn't I have written in Summer 2022?

A big one is that the funding environment is very
different. This
both means that earning
to give is more valuable than it had been and it's harder to stay
funded. I think my c...
Dall-E 3
by p.b. — 2023-10-02T20:33:18.294Z
Link: https://openai.com/dall-e-3

It seems like we do not yet have a post about Dall-E 3. It was announced a few days ago and can now be tried via bing image creator (long wait times).

I think it is worth mentioning, because it seems like a big jump.

I spend quite a lot of time playing around with Dalle mini, Dall-E 2, Stable Diffusion in its many flavours and it was generally the case that it was very hard to get specifically what I wanted, to the point where I was disillusioned and few of my projects materialised.

A teddy bear and a panda bear holding hands - it took until SDXL for that to work out at least occasionally (the wedding was long over).

Maritime paintings of awesome sailing ships - the rigging was generally completely messed up (finetuning on Montague Dawson and Co didn't help).

Comic books aka Prince Valiant fan-fic - impossible to reliably get specific poses or situations involving several characters.

Mammoths in space - don't ask, but I had to do the space hangar in SD and the mammoths in Dall-E and bash it all together.

The Stable Diffusion eco system build around these shortcoming with control nets and finetuned models and increasingly complicated workflows - but now it seems with Dall-E 3 many more things just work right out of the box.

From my few experiments and the examples I have seen elsewhere this might be the point where image generation just works.
Population After a Catastrophe
by Stan Pinsent — 2023-10-02T16:06:56.614Z

This was written in my role as researcher at CEARCH, but any opinions expressed are my own. I will check comments on the EA Forum version more often.

This report uses population dynamics to explore the effects of a near-existential catastrophe on long-term value.

Summary

Global population would probably not recover to current levels after a major catastrophe. Low-fertility values would largely endure. If we reindustrialize quickly, population will stabilize far lower.
Population “peaking lower” after a catastrophe would make it harder to avoid terminal population decline. Tech solutions would be harder to reach, and there would be less time to find a solution.
Post-catastrophe worlds that avoid terminal population decline are likely to emerge with values very different to our own. Population could stabilize because of authoritarian governments, prescriptive gender roles or civil strife, or alternatively from increased collective concern for the future.
Conclusion: Near-existential catastrophes are likely to decrease the value of the future through decreased resilience and the lock-in of bad values. Avoiding these catastrophes should rank alongside avoiding existential catastrophes.

Introduction

In this report I use population dynamics to explore the question “What are the long-term existential consequences of a non-existential catastrophe?”. I do not claim that population dynamics are the only, or even the most important, consideration.

Others have written about the short-term existential effects of a global catastrophe. Luisa Rodriguez argues that even in cases where >90% of the global population is killed, it is unlikely that all viable groups of survivors will fail to make it through the ensuing decades (

Rodriguez, 2020

). The Global Catastrophic Risk Institute has begun to explore the long-term consequences of catastrophe, although they consider this “rather grim and difficult-to-study topic” to be neglected (

GCRI

).

What comes after the aftermath of a catastrophe is very difficult to predict, as life will be driven by unknown political and cultural forces. However, I argue that many of the familiar features of population dynamics will continue to apply.

Even without a catastrophe, we face a possible population problem. As countries develop, their populations peak and begin to decline. If these trends continue, global popula...
Expectations for Gemini: hopefully not a big deal
by Maxime Riché — 2023-10-02T15:38:32.834Z

Introduction

My goal is to register and share my expectations and hear others' opinions on their expectation for the relative performances of Gemini VS GPT-4.

My expectations

GPT-4 to Gemini will likely not be as big a jump in capabilities as GPT-3 to GPT-4 was.

Gemini could bring surprises by being more agentic than GPT-4. Being better at planning and longer horizon tasks. But this is likely difficult to achieve, or strong LLM agents would already be making the buzz.

Comparison

From GPT-3 to GPT-4

Scaling Factor: x100 more compute than GPT-3.
Optimization: Chinchilla scaling laws (for MoE) over OpenAI/Kaplan scaling laws.
MoE Over Dense: Utilizes Mixture of Experts (MoE) instead of dense layers.
Data Quality: Likely higher-quality data, not sure.
Image Generation: Not publicly released, possibly due to subpar performance or security risks.
Tools are added during finetuning.
Algorithmic Gains: 3 years between GPT-3 and GPT-4.
GPT-4 may already employ process-based feedback.
GPT-4 aimed for training compute efficiency. GPT-4 was not designed to be commercially deployed at scale.

GPT-4 to Gemini

Scaling Factor: ~x5 (x20) more compute than GPT-4.
Supercomputer Constraint: No existing supercomputer could feasibly provide x100 more compute than used for GPT-4. (Not sure but likely)
Multimodal: maybe image, audio, speech.
Data Efficiency: Possibly better quality data like Google Books, fewer epochs.
Tools could be added either during finetuning or pretraining.
Algorithmic Gains: ~1 year between GPT-4 and Gemini.
Gemini more likely aims for inference efficiency, given its intended extensive usage by Google. Maybe sacrificing training efficiency.
Gemini trained to be more agentic, better at planning, etc. ("GPT-4 + AlphaGo").

Note: I drafted that before news of Gemini's release and capabilities but failed to finish writing... Since then, there have been some reports of Gemini being roughly at the level of GPT-4...
A counterexample for measurable factor spaces
by Matthias G. Mayer — 2023-10-02T15:16:48.418Z

PDF

This is a technical post researching infinite factor spaces

I define a general measurable factor space that
shows that the history can't be straightforwardly generalized to the infinite case.
The example shows the following:

Unlike in the finite setting, the condition
∀P∈▲∗(F):πJ⫫Pπ¯J | Z in the definition of
generation (see below) can't be characterized by requiring that the atoms of
Z are rectangles. In the example, we show that this would
imply an orthogonality where the implication X⊥Y∣∣Z⇒∀P∈▲∗(F):X⫫PY∣∣Z is not true.

In measurable spaces, conditional histories can only be well-defined
up to a chosen system of nullsets. In the example, we show that a
minimal generating index function J:F→P(I)
does not exist. (We will show in a later post that the history exists if we fix nullsets.)

To talk about orthogonality, we first have to introduce the concepts for
measurable spaces. For now, we focus on the index set I being finite. A
straightforward generalization from the finite case is the following.
We have measurable spaces (Fi,Ai) and
construct F=×i∈IFi. Let Z:F→Z(F) be a feature
(i.e. measurable). Let J:F→P(I) be
Z-measurable. We define
πJ(x)=πJ(x)(x) We note, that we assume w.l.o.g. that we
assume that Fi are pairwise disjoint. Recall that ▲∗(F) is
the set of product probability distributions on F.

J⊢X∣∣Z:⇔X≲(πJ,Z)∧∀P∈▲∗(F):πJ⫫Pπ¯J∣∣Z.

The history should now be the smallest such J. Furthermore, for
orthogonality defined by X⊥Y | Z:⇔H(X|Z)∩H(Y|Z)=∅
we should have X⊥Y | Z⇔∀P∈▲∗(F):X⫫PY|Z

The idea for the example consists of choosing Z, such that
Z−1(z) consists of four points, where the points will
always fall into one unit square each. Call those unit squares Uij.
Choosing Z|Uij to be a diffeomorphism allows us to
calculate the conditional probabilities explicitly. Almost all choices
of Z where Z−1(z) is a rectangle for all
z, will fail to satisfy ∀P∈▲∗(F):X⫫PY | Z.

Example 1

Let I={1,2}. For i∈I, let
Fi=[0,1]∪[2,3] endowed with the
Borel sigma-algebra.

Now
F=F1×F2=([0,1]∪[2,3])^2.

F consists of 4 unit squares.

We will now define a Z, such that Z−1 is a rectangle of four
points.

Let Uij={[0,1] if i=1[2,3]else×{[0,1] if j=1[2,3]else be the four unit squares that F consists of.

Let
φx,φy:[0,1]2→[2,3].
φx will be the function that chooses the new x coordinate.

Let πx:[0,1]2→[2,3],x,y|→x+2πy:[0,1]2→[2,3]...