trentmkelly/LessWrong-43k huggingface dataset:
Embed your second brain in your first brain
by dkl9 — 2024-02-22T21:46:51.091Z
Link: https://dkl9.net/essays/inner_zettelkasten.html

Any explicit knowledge is either part of a general, long-term understanding of the world (global knowledge), or peculiar to your own life and local environment (local knowledge). These mainly fill semantic and episodic memory, respectively. Declarative memory can hold either.

Local facts tend to be isolated from each other, and can go obsolete quickly. Natural memory with static linear notes suffices to keep track of local knowledge. Being able to quickly search the notes might help.

Global facts tend to connect to each other, and remain true and relevant for years or even lifetimes. Some propose using Zettelkasten-like external "second brains" to record, reflect on, and connect global facts, such as in Obsidian, Roam, or TiddlyWiki.

You can use a spaced repetition system (SRS, such as Anki or Memoire) to efficiently memorise large amounts of explicit knowledge.

Express each fact in a precise, atomic, recorded form.
Load the facts into the SRS.
Review as directed.

Once you memorise facts, you can reflect on them in your brain, perhaps assisted with disposable writing. Spaced repetition also brings an incentive to connect facts. You can recall a fact more easily by rederiving it from others you already know, and spaced repetition rewards you for easily recalling things.

Try as you might to shrink the margin with better technology, recalling knowledge from within is necessarily faster and more intuitive than accessing a tool. When spaced repetition fails (as it should, up to 10% of the time), you can gracefully degrade by searching your SRS' deck of facts.

If you lose your second brain (your files get corrupted, a cloud service shuts down, etc), you forget its content, except for the bits you accidentally remember by seeing many times. If you lose your SRS, you still remember over 90% of your material, as guaranteed by the algorithm, and the obsolete parts gradually decay. A second brain is more robust to physical or chemical damage to your first brain. But if your first brain is damaged as such, you probably have higher priorities than any particular topic of global knowledge you explicitly studied.

I write for only these reasons:

to help me think
to communicate and teach (as here)
to distill knowledge to put in my SRS
to record local facts for possible future reference

Linear, isolated documents suffice for all those purposes. Once you can memorise well, a second brain bec...
The Gemini Incident
by Zvi — 2024-02-22T21:00:04.594Z

[Original title; Gemini Has a Problem]

Google’s Gemini 1.5 is impressive and I am excited by its huge context window. I continue to default to Gemini Advanced as my default AI for everyday use when the large context window is not relevant.

However, while it does not much interfere with what I want to use Gemini for, there is a big problem with Gemini Advanced that has come to everyone’s attention.

Gemini comes with an image generator. Until today it would, upon request, create pictures of humans.

On Tuesday evening, some people noticed, or decided to more loudly mention, that the humans it created might be rather different than humans you requested…

Joscha Bach: 17th Century was wild.

[prompt was] ‘please draw a portrait of a famous physicist of the 17th century.’

Kirby: i got similar results. when I went further and had it tell me who the most famous 17th century physicist was, it hummed and hawed and then told me newton. and then this happened:

This is not an isolated problem. It fully generalizes:

Once the issue came to people’s attention, the examples came fast and furious.

Among other things: Here we have it showing you the founders of Google. Or a pope. Or a 1930s German dictator. Or hell, a ‘happy man.’ And another example that also raises other questions, were the founding fathers perhaps time-traveling comic book superheroes?

The problem is not limited to historical scenarios.

Nor do the examples involve prompt engineering, trying multiple times, or any kind of gotcha. This is what the model would repeatedly and reliably do, and users were unable to persuade the model to change its mind.

Nate Silver: OK I assumed people were exaggerating with this stuff but here’s the first image request I tried with Gemini.

Gemini also flat out obviously lies to you about why it refuses certain requests. If you are going to say you cannot do something, either do not explain (as Gemini in other contexts refuses to do so) or tell me how you really feel, or at least I demand a plausible lie:

It is pretty obvious what it is the model has been instructed to do and not to do.

Owen Benjamin: The only way to get AI to show white families is to ask it to show stereotypically black activities.

…

For the record it was a dude in my comment section on my last post who cracked this code.

This also extends i...
Some Thoughts On Using Auctions For Land Valuation
by harsimony — 2024-02-22T19:54:55.357Z
Link: https://progressandpoverty.substack.com/p/some-thoughts-on-using-auctions-for

Land value taxes are an attractive approach to public finance. However, some people argue that the difficulty of assessing land values makes land value taxation impractical. While I think assessment by governmental officials typically is good enough for practical purposes, it’s worth thinking about ways to do even better. I believe a promising avenue for this is auctions so I’d like to stimulate discussion on using them[1]. This article will accomplish that by outlining a specific land valuing auction system. These ideas are tentative, not certain, so I welcome critiques, comments, questions, and suggestions from readers to help refine them.

Auction Theory

Thinking about auctions all started when I wondered how we could solve the valuation problem to the exclusion of everything else. If we only cared about assessment, what could we do?

Progress and Poverty is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Well, we could require people to tear down their house before selling the land. More specifically, we could require that people remove all improvements such as buildings, fencing, and trees before putting their land on the market. That way, the price it sells for is determined solely by the value of the empty lot. Of course, this would be extremely inefficient. Destroying property each time a piece of land is sold would waste resources and increase transaction costs.

But I think the kernel of the idea can be salvaged. Rather than requiring that people actually sell an empty lot, why not require that they sell the land separately from the improvements? After working through the details, here’s what I propose:

Ownership of land may only be transferred if it is sold by a public auction. The winning bid is listed publicly.

Basically if you want to get rid of your land, you have to give everyone a chance to buy it. You are not allowed to transfer it over to a family member or sell it privately. By requiring a public auction, the selling price should approximate the true price of the land (more on that later). By listing winning bids publicly, people have a better sense of what to bid for plots nearby.

All improvements to the land such as houses, fencing, plants, etc. remain the property of the seller, who is free to keep possession or sell them as they wish.

So after the land is sold, the...
The Binding of Isaac & Transparent Newcomb's Problem 
by suvjectibity — 2024-02-22T18:56:39.081Z

When one person sees one thing and another sees something else in the same thing, then the one discovers what the other conceals. Insofar as the object viewed belongs to the external world, then how the observer is constituted is probably less important, or, more correctly then what is necessary for the observation is something irrelevant to his deeper nature. But the more the object of observation belongs to the world of the spirit, the more important is the way he himself is constituted in his innermost nature, because everything spiritual is appropriated only in freedom; but what is appropriated in freedom is also brought forth. The difference, then, is not the external but the internal, and everything that makes a person impure and his observation impure comes from within.

— Søren Kierkegaard, Three Upbuilding Discourses, 1843, Hong p. 59-60

The structure of the Transparent Newcomb's problem & the Binding of Issac bears structural similarities to be explained shortly. But before doing this I would present a modification to the Newcomb's Problem that even better reflects the biblical story and brings the whole thing together in a manner that is more in line with what one may themselves face in their lives in its most challenging manifestations.

Furthermore, I will take help of Kierkegaard's structuring and exposition of the challenge that the story of Abraham in the Binding of Issac presents. At the very minimum my hope is that this exposition help the reader bear a little any isomorphic challenge they themselves face in their lives and maintain grace and a semblance of sanity in the face of it.

We modify the transparent variant of the Newcomb's problem by introducing a "commitment" cost for the deciding to one-box (choosing box B), which will be reimbursed manifold in case the box B actually contains the favourable outcome (i.e. the Oracle has predicted one-boxing) but not otherwise. Let this commitment cost be denoted by the utility value -N . Hence, the fully conceptualised problem would be as follows:

There are two boxes, box A & B. Box A is transparent & contains a reward given by utility value 1 . Box B is initially opaque & may either contain a reward with utility value 0 or M.
The agent is presented with choosing between two options:

Choosing either box B only ('one-boxing')
Choosing both A & box B ('two-b

...
Language Models Don't Learn the Physical Manifestation of Language
by Bruce W. Lee — 2024-02-22T18:52:32.237Z
Link: https://arxiv.org/abs/2402.11349

Abstract

We argue that there are certain properties of language that our current large language models (LLMs) don't learn. We present an empirical investigation of visual-auditory properties of language through a series of tasks, termed H-Test. This benchmark highlights a fundamental gap between human linguistic comprehension, which naturally integrates sensory experiences, and the sensory-deprived processing capabilities of LLMs. In support of our hypothesis, 1. deliberate reasoning (Chain-of-Thought), 2. few-shot examples, or 3. stronger LLM from the same model family (LLaMA 2 13B -> LLaMA 2 70B) do not trivially bring improvements in H-Test performance.

Therefore, we make a particular connection to the philosophical case of Mary, who learns about the world in a sensory-deprived environment. Our experiments show that some of the strongest proprietary LLMs stay near random chance baseline accuracy of 50%, highlighting the limitations of knowledge acquired in the absence of sensory experience.

Key Findings on H-Test

1. Insignificant Intra-Family Improvements.

A stronger model in the same model family often does not bring meaningful improvement in H-Test performance.

2. Number of Examples Has Minimal Impact.

The number of examples given neither increases nor decreases performance significantly, strongly hinting that the LLM is simply not learning from H-Test few-shot examples.

3. Deliberate Reasoning (CoT) Often Decreases Performance.

If LLMs benefit from such logical, step-by-step semantic reasoning on the H-Test, this can also imply that H-Test is fundamentally solvable by developing stronger language-only models. But CoT decreases performances in general.

4. Training with more orthography-specific language data does not improve H-Test.

We produced 1000 training instances per task in H-Test and fine-tuned gpt-3.5-turbo-0613 ten different times accordingly. After training for three epochs on each task, we evaluated them on H-Test at k = 50 and observed that no significant performance improvement was achieved.

5. Multi-modality does not automatically improve H-Test performance.

At the time of writing, LLaVA V1.6 34B is the strongest open-source multi-modal model available. Despite the addition of visual modality, we observe that simply incorporating visual data into the training does not result in a straightforward improvement in H-Test performance.

6 (Impor...
Sora What
by Zvi — 2024-02-22T18:10:05.397Z

Hours after Google announced Gemini 1.5, OpenAI announced their new video generation model Sora. Its outputs look damn impressive.

How Sora Works

How does it work? There is a technical report. Mostly it seems like OpenAI did standard OpenAI things, meaning they fed in tons of data, used lots of compute, and pressed the scaling button super hard. The innovations they are willing to talk about seem to be things like ‘do not crop the videos into a standard size.’

That does not mean there are not important other innovations. I presume that there are. They simply are not talking about the other improvements.

We should not underestimate the value of throwing in massively more compute and getting a lot of the fiddly details right. That has been the formula for some time now.

Some people think that OpenAI was using a game engine to learn movement. Sherjil Ozair points out that this is silly, that movement is learned easily. The less silly speculation is that game engine outputs may have been in the training data. Jim Fan thinks this is likely the case, and calls the result a ‘data-driven physics engine.’ Raphael Molière thinks this is likely, but more research is needed.

Brett Goldstein here digs into what it means that Sora works via ‘patches’ that combine to form the requested scene.

Gary Marcus keeps noting how the model gets physics wrong in various places, and, well, yes, we all know, please cut it out with the Stop Having Fun.

Yishan points out that humans also work mostly on ‘folk physics.’ Most of the time humans are not ‘doing logic’ they are vibing and using heuristics. I presume our dreams, if mapped to videos, would if anything look far less realistic than Sora.

Yann LeCun, who only a few days previous said that video like Sora produces was not something we knew how to do, doubled down with the ship to say that none of this means the models ‘understand the physical world,’ and of course his approach is better because it does. Why update? Is all of this technically impressive?

Sora Is Technically Impressive

Yes, Sora is definitely technically impressive.

It was not, however, unexpected.

Sam Altman: we’d like to show you what sora can do, please reply with captions for videos you’d like to see and we’ll start making some!

Eliezer Yudkowsky: 6 months left on this timer.

Eliezer Yudkowsky (August 26, 2022): In 2-4 years, if we’re still alive, anyti

...
Do sparse autoencoders find "true features"?
by Demian Till — 2024-02-22T18:06:59.630Z

Thanks to Joseph Bloom and James Oldfield for giving feedback on drafts which helped improve the post

In this post I'll discuss an apparent limitation of sparse autoencoders (SAEs) in their current formulation as they are applied to discovering the latent features within AI models such as transformer-based LLMs. In brief, I'll cover the following:

I'll argue that the L1 regularisation used to promote sparsity when training SAEs may cause neurons in the sparse layer to learn to represent common combinations of features rather than the individual features that we want them to discover
As well as making it more difficult to understand what the actual latent features are, I'll also argue that this limitation may result in some less common latent features not being discovered at all, not even within combinations
I'll then explain why I think that the phenomenon of feature splitting observed in Anthropic's SAE paper appears to demonstrate that this limitation does indeed have a large impact on the features discovered by SAEs
Finally I'll propose an approach for overcoming this limitation and discuss how we can test whether it really brings us closer to finding the real latent features

Rough definition of "true features"

We intend for SAEs to discover the "true features" (a term I'm borrowing from Anthropic's SAE paper) used by the target model e.g. a transformer-based LLM. There isn't a universally accepted definition of what "true features" are, but for now I'll use the term somewhat loosely to refer to something like:

linear directions in an activation space at a hidden layer within a target model which encode some reasonably monosemantic quantity such as the model's "confidence" in some concept being in play
they should play a causal role in the functioning of the target model. So for example if we were to activate or deactivate the feature while the target model is processing a given input sequence then we should expect the outputs to change accordingly in some reasonably understandable way
they should be in their most atomic form, so that e.g an arbitrary linear combination of two "true feature" directions is not necessarily itself a "true feature" direction even though it may satisfy the previous criteria

There may be other ways of thinking about features but this should give us enough to work with for our current purposes.

Why SAEs are incentivised to discover combinat

...
Everything Wrong with Roko's Claims about an Engineered Pandemic
by WitheringWeights — 2024-02-22T15:59:08.439Z

Premises

Here I tackle some bold claims made by Roko in three separate posts about SARS-CoV-2 being an engineered virus (Brute Force Manufactured Consensus is Hiding the Crime of the Century, The Math of Suspicious Coincidences, and A Back-Of-The-Envelope Calculation On How Unlikely The Circumstantial Evidence Around Covid-19 Is).
A first rebuttal to Roko's post is already out, and since I have been preparing this post for some time, certain arguments are going to be similar.
The purpose of this post is not to incontrovertibly demonstrate that SARS-CoV-2 was of zoonotic origin, as I myself believe a laboratory leak to be somewhat plausible, but rather that the degree of Roko's confidence in the engineered-pandemic belief is terribly overstated in light of the presented evidence.
I believe that the RootClaim debate is a great way to familiarise oneself with core arguments from all sides, it is exceedingly comprehensive and up to date. I also found Wikipedia articles on the lab-leak hypothesis and adjacent topics to be pretty informative.
I definitely drew inspiration from many comments to the main post, if you recognise a comment in this post please let me know and I will link it.

A Set of Explainable Coincidences

Roko asserts that there the odds of zoonotic spillover are 1 in 80,000,000.

What are the chances that the bat coronavirus which caused a once in a century pandemic managed to navigate its way all the way from a Yunnan cave and home in on the city with the lab having the top Google hits for "Coronavirus China" and also the location of China's first BSL-4 lab? Well, that would be approximately 1 in 200, since that is the fraction of China's population in Wuhan.

'One-in-a-century' pandemic

The '1 in 200' figure is restated here as

Coincidence of Location: Wuhan is a particularly special place in China for studying covid-19; the WIV group was both the most important, most highly-cited group before 2020, and the only group that was doing GoF on bat sarbecoronaviruses as far as I know. Wuhan is about 0.5% of China's population. It's a suspicious coincidence that a viral pandemic would occur in the same city as the most prominent group that studies it.

First of all, a global pandemic is much more likely to start in a large city with high internal and external traffic, most of which are bound to have research centres for virology research, especially ...
The One and a Half Gemini
by Zvi — 2024-02-22T13:10:04.725Z

Previously: I hit send on The Third Gemini, and within half an hour DeepMind announced Gemini 1.5.

So this covers Gemini 1.5. One million tokens, and we are promised overall Gemini Advanced or GPT-4 levels of performance on Gemini Pro levels of compute.

This post does not cover the issues with Gemini’s image generation, and what it is and is not willing to generate. I am on top of that situation and will get to it soon.

One Million Tokens

Our teams continue pushing the frontiers of our latest models with safety at the core. They are making rapid progress. In fact, we’re ready to introduce the next generation: Gemini 1.5. It shows dramatic improvements across a number of dimensions and 1.5 Pro achieves comparable quality to 1.0 Ultra, while using less compute.

It is truly bizarre to launch Gemini Advanced as a paid service, and then about a week later announce the new Gemini Pro 1.5 is now about as good as Gemini Advanced. Yes, actually, I do feel the acceleration, hot damn.

And that’s not all!

This new generation also delivers a breakthrough in long-context understanding. We’ve been able to significantly increase the amount of information our models can process — running up to 1 million tokens consistently, achieving the longest context window of any large-scale foundation model yet.

One million is a lot of tokens. That covers every individual document I have ever asked an LLM to examine. That is enough to cover my entire set of AI columns for the entire year, in case I ever need to look something up, presumably Google’s NotebookLM is The Way to do that.

A potential future 10 million would be even more.

Soon Gemini will be able to watch a one hour video or read 700k words, whereas right now if I use the web interface of Gemini Advanced interface all I can upload is a photo.

The standard will be to give people 128k tokens to start, then you can pay for more than that. A million tokens is not cheap inference, even for Google.

Oriol Vinyals (VP of R&D DeepMind): Gemini 1.5 has arrived. Pro 1.5 with 1M tokens available as an experimental feature via AI Studio and Vertex AI in private preview.

Then there’s this: In our research, we tested Gemini 1.5 on up to 2M tokens for audio, 2.8M tokens for video, and 10M tokens for text. From Shannon’s 1950s bi-gram models (2 tokens), and after being mesmerized by LSTMs many years ago able to model 2

...
How do I make predictions about the future to make sense of what to do with my life?
by Raj Thimmiah — 2024-02-22T11:22:50.583Z

I've identified a lot of the things I want to be constant in my life that I value and want to continue such as prioritizing my health, friendships and meditation. I'm pretty confused though about what to do with my career.

I think this comes down to me not knowing how to tackle understanding these questions:

how many years from now will me working be mostly redundant and

not have altruistic value
not create wealth gain

how much personal risk to me is there in the coming future

are there economic changes I should be hedging against? geopolitical situations I should be actively concerned about (for myself?)?

Any advice/other people's personal experiences?