trentmkelly/LessWrong-43k huggingface dataset:
Passages I Highlighted in The Letters of J.R.R.Tolkien
by Ivan Vendrov — 2024-11-25T01:47:59.071Z

All quotes, unless otherwise marked, are Tolkien's words as printed in The Letters of J.R.R.Tolkien: Revised and Expanded Edition. All emphases mine.

Machinery is Power is Evil

Writing to his son Michael in the RAF:

[here is] the tragedy and despair of all machinery laid bare. Unlike art which is content to create a new secondary world in the mind, it attempts to actualize desire, and so to create power in this World; and that cannot really be done with any real satisfaction. Labour-saving machinery only creates endless and worse labour. And in addition to this fundamental disability of a creature, is added the Fall, which makes our devices not only fail of their desire but turn to new and horrible evil. So we come inevitably from Daedalus and Icarus to the Giant Bomber. It is not an advance in wisdom! This terrible truth, glimpsed long ago by Sam Butler, sticks out so plainly and is so horrifyingly exhibited in our time, with its even worse menace for the future, that it seems almost a world wide mental disease that only a tiny minority perceive it. Even if people have ever heard the legends (which is getting rarer) they have no inkling of their portent. How could a maker of motorbikes name his product Ixion cycles! Ixion, who was bound for ever in hell on a perpetually revolving wheel! Well, I have got over 2 thousand words onto this little flimsy airletter; and I will forgive the Mordor-gadgets some of their sins, if they will bring it quickly to you

The Sam Butler reference is to Samuel Butler's Erewhon, a science fiction novel which is the first known work to warn about self-replicating machines and the possibility of machine consciousness. The 'Butlerian Jihad' from Dune is another famous reference to Butler. A quote from Erewhon:

I fear none of the existing machines; what I fear is the extraordinary rapidity with which they are becoming something very different to what they are at present. No class of beings have in any time past made so rapid a movement forward. Should not that movement be jealously watched, and checked while we can still check it? And is it not necessary for this end to destroy the more advanced of the machines which are in use at present, though it is admitted that they are in themselves harmless?

Tolkien again, writing to his son Christopher during the war:

the whole human race (as each individual) is free not to rise again but to

...
Decorated pedestrian tunnels
by dkl9 — 2024-11-24T22:16:03.794Z
Link: https://dkl9.net/essays/artistic_streets.html

Moving thru a visually-varied environment helps you remember what you think, say, and hear. Adding visual variety to an area thus aids the memories of those passing thru it, and, on average, makes it look nicer. Dense cities, especially those with mixed-use zoning, already have many mutually-distinct buildings. Some other small areas, like parks, are similarly intricate. Suburbs and blander cities, however, are mostly dull arrays of streets and houses. We can do better.

Visual art can be almost arbitrarily detailed and unique. If we cover walls around streets with visual art, the problem is solved. However, most walls around streets are the outer walls of buildings, most of which are privately owned, only some of which would approve of adding lots of art. Putting art on the street-ground itself helps, but people moving outside look to the sides more than down (citation needed), and getting walked on, cycled on, and driven over could wear away the paint (or equivalent).

In much of the US, urban and (even more so) suburban streets are, regrettably, dominated by automobiles. Ideally, we would use automobiles a lot less. But, for a probably-effective stopgap: invite walking and cycling in an underground network of tunnels. Artificial underground space can get dreary, but a tunnel necessarily has walls, all of which could be delightfully coated with a wild assortment of images, completely solving my current concern. Tunnels would also be thermally insulated from the above world, and so have more stable, pleasant temperatures — especially important when moving without enclosed vehicles.

Much as it may be putting pedals before handlebars to design them in such detail:

the tunnels should form a grid, ideally triangular, or else in squares or parallelograms
the above world should connect to the tunnels near grid intersections, and maybe other spots
intersections should cut rounded corners for visibility, especially if tunnels allow cycling
tunnel-segments should extend 20 to 100 metres between grid-intersections
tunnels should be 1.5 to 5 metres wide

Good art — at least, good-enough-art — could come from graffiti, if legalised with the right caveats, like

at most so much detail in one spot (encourage spreading out)
at most so close two repeats of a design (encourage variety)
keep it family-friendly to at least such a standard

Enforcing elaborate rules on graffiti is harder and mo...
Are You More Real If You're Really Forgetful?
by Thane Ruthenis — 2024-11-24T19:30:55.233Z

It's a standard assumption, in anthropic reasoning, that effectively, we simultaneously exist in every place in Tegmark IV that simulates this precise universe (see e. g. here).

How far does this reasoning go?

Suppose that the universe's state is described by n low-level variables x1,…,xn. However, your senses are "coarse": you can only view and retain the memory of m variables y1,…,ym, where m≪n and each yi is a deterministic function of some subset of x1,…,xn.

Consider a high-level state Y∗, corresponding to each yi being assigned some specific value. For any Y∗, there's an equivalence class of low-level states X∗ precisely consistent with Y∗.

Given this, if you observe Y∗, is it valid to consider yourself simultaneously existing in all corresponding low-level states X∗ consistent with Y∗?

Note that, so far, this is isomorphic to the scenario from Nate's post, which considers all universes that only differ by the choices of gauge (which is undetectable from "within" the system) equivalent.

Now let's examine increasingly weirder situations based on the same idea.

Scenario 1:

Consider two Everett branches, A and B. They only differ by the exact numbers of photons in your room: A has an extra photon.
Suppose that we gave the entire history of your observations over your lifetime to AIXI, which simulates all universes consistent with your observations. Suppose that, in the end, it's only able to narrow it down to "A or B".
Does that mean you currently simultaneously exist in both branches?
Importantly, note that the crux here isn't whether you, a bounded agent, are able to consciously differentiate between A and B.

That is: Suppose that, in A, the extra photon hits your eye and makes you see a tiny flash of red. If so, then, even though you likely won't make any conscious inferences about the photons, that'd still create a difference between the sensory streams, which AIXI (an unbounded computation) would be able to use to distinguish between A and B.
Similarly, if the existence of the extra photon causes a tiny divergence ten years down the line, which will lead to a different photon hitting your eye and your seeing a tiny red flash, this will likewise create a difference that AIXI would be able

...
Perils of Generalizing from One's Social Group
by localdeity — 2024-11-24T15:31:18.332Z

I see people make statements of the form, "In my experience with people I encounter, X is correlated with ...". The problem is, there's an excellent chance that the people they deal with are very unrepresentative of the population they want to generalize about, and I rarely see them show awareness of the possibility that selection bias has created the effect they're describing.

Scott has written about the strength of social group filter bubbles. But there's a systematic effect I want to highlight: Berkson's paradox. Following Wikipedia's example:

Suppose that people become famous either by being pretty, or by being talented. Let's say these are all-or-nothing, binary traits. Then, among the population of famous people, being pretty will be anticorrelated with being talented. If a famous person is pretty, then they might or might not be talented, whereas if they're not pretty, then they must be talented—otherwise they wouldn't be famous. So this anticorrelation between talent and beauty is guaranteed to exist among famous people, no matter how highly correlated they might be among the general population.[1]

If we use numbers instead of a binary, then we might imagine that talent and beauty scores are numbers from 0 to 10, and we'll say one becomes famous if those scores add to at least 12. It follows that, if we see a famous guy and observe that his beauty is 10, then his talent could be anything from 2 to 10, but if we see his beauty is 2, then his talent must be 10. So the selection effect will likely create a big anticorrelation between the two traits.[2]

One can directly apply that model of "popularity ≈ beauty + substance ==> beauty anticorrelates with substance among the popular" to lots of areas. (In fact, if we treat it like a real equation, we can subtract and get "substance ≈ popularity - beauty".) For example, if games become popular by having a combination of good graphics and good gameplay, then, if you see a popular game with awful graphics, you know it has great gameplay. A successful movie that looks ugly probably has a great storyline. And, as Don Norman tells us, a product that looks like "it probably won a prize [for aesthetics]" may still be horribly unusable.

But the idea applies to any case where you're selecting on one trait that's (approximately) determined by ...
Disentangling Representations through Multi-task Learning
by Bogdan Ionut Cirstea — 2024-11-24T13:10:26.307Z
Link: https://arxiv.org/abs/2407.11249

Authors: Pantelis Vafidis, Aman Bhargava, Antonio Rangel.

Abstract:

Intelligent perception and interaction with the world hinges on internal representations that capture its underlying structure ("disentangled" or "abstract" representations). Disentangled representations serve as world models, isolating latent factors of variation in the world along orthogonal directions, thus facilitating feature-based generalization. We provide experimental and theoretical results guaranteeing the emergence of disentangled representations in agents that optimally solve multi-task evidence aggregation classification tasks, canonical in the cognitive neuroscience literature. The key conceptual finding is that, by producing accurate multi-task classification estimates, a system implicitly represents a set of coordinates specifying a disentangled representation of the underlying latent state of the data it receives. The theory provides conditions for the emergence of these representations in terms of noise, number of tasks, and evidence aggregation time. We experimentally validate these predictions in RNNs trained on multi-task classification, which learn disentangled representations in the form of continuous attractors, leading to zero-shot out-of-distribution (OOD) generalization in predicting latent factors. We demonstrate the robustness of our framework across autoregressive architectures, decision boundary geometries and in tasks requiring classification confidence estimation. We find that transformers are particularly suited for disentangling representations, which might explain their unique world understanding abilities. Overall, our framework puts forth parallel processing as a general principle for the formation of cognitive maps that capture the structure of the world in both biological and artificial systems, and helps explain why ANNs often arrive at human-interpretable concepts, and how they both may acquire exceptional zero-shot generalization capabilities.
The U.S. National Security State is Here to Make AI Even Less Transparent and Accountable
by Matrice Jacobine — 2024-11-24T09:36:39.768Z
Link: https://www.eff.org/deeplinks/2024/11/us-national-security-state-here-make-ai-even-less-transparent-and-accountable

The Biden White House has released a memorandum on “Advancing United States’ Leadership in Artificial Intelligence” which includes, among other things, a directive for the National Security apparatus to become a world leader in the use of AI. Under direction from the White House, the national security state is expected to take up this leadership position by poaching great minds from academia and the private sector and, most disturbingly, leveraging already functioning private AI models for national security objectives.

Private AI systems like those operated by tech companies are incredibly opaque. People are uncomfortable—and rightly so—with companies that use AI to decide all sorts of things about their lives–from how likely they are to commit a crime, to their eligibility for a job, to issues involving immigration, insurance, and housing. Right now, as you read this, for-profit companies are leasing their automated decision-making services to all manner of companies and employers and most of those affected will never know that a computer made a choice about them and will never be able to appeal that decision or understand how it was made.

But it can get worse; combining both private AI with national security secrecy threatens to make an already secretive system even more unaccountable and untransparent. The constellation of organizations and agencies that make up the national security apparatus are notoriously secretive. EFF has had to fight in court a number of times in an attempt to make public even the most basic frameworks of global dragnet surveillance and the rules that govern it. Combining these two will create a Frankenstein’s Monster of secrecy, unaccountability, and decision-making power.

While the Executive Branch pushes agencies to leverage private AI expertise, our concern is that more and more information on how those AI models work will be cloaked in the nigh-impenetrable veil of government secrecy. Because AI operates by collecting and processing a tremendous amount of data, understanding what information it retains and how it arrives at conclusions will all become incredibly central to how the national security state thinks about issues. This means not only will the state likely make the argument that the AI’s training data may need to be classified, but they may also argue that companies need to, under pen

...
Mechanistic Interpretability of Llama 3.2 with Sparse Autoencoders
by PaulPauls — 2024-11-24T05:45:20.124Z
Link: https://github.com/PaulPauls/llama3_interpretability_sae

I recently published a rather big side project of mine that attempts to replicate the mechanistic interpretability research on proprietary and open-source LLMs that was quite popular this year and produced great research papers by Anthropic[1][2], OpenAI[3][4] and Google Deepmind[5] with the humble but open source Llama 3.2-3B model.

The project provides a complete end-to-end pipeline for training Sparse Autoencoders to interpret LLM features, from activation capture through training, interpretation, and verification. All code, data, trained models, and detailed documentation are publicly available in my attempt to make this as open research as possible, though calling it an extensively documented personal project wouldn't be wrong either in my opinion.

Since LessWrong has a strong focus on AI interpretability research, I thought some of you might find value in this open research replication. I'm happy to answer any questions about the methodology, results, or future directions.

^

https://www.anthropic.com/research/mapping-mind-language-model

^

https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html

^

https://openai.com/index/extracting-concepts-from-gpt-4/

^

https://arxiv.org/abs/2406.04093

^

https://arxiv.org/abs/2408.05147
SB-1047, ChatGPT and AI's Game of Thrones
by Rahul Chand — 2024-11-24T02:29:34.907Z

Part-I (The Sin of Greed)

On 30 November 2022, OpenAI released ChatGPT. According to Sam Altman, it was supposed to be a demo[1] to show the progress in language models. By December 4, in just 5 days it had gained 1 million users, for comparison it took Instagram 75 days, Spotify 150 days and Netflix 2 years to gain same no. of users. By January 2023, it had 100 million users, adding 15 million users every week. ChatGPT became the fastest growing service in human history. Though most people would only come to realize this later, the race for AGI had officially began.

Days it took popular services to get to 1M and 100M users

Too Soon, Too Fast, Too Much - Timeline of AI

November 2022

At first, ChatGPT was seen more of a fun toy to play with. Early uses of ChatGPT mostly involved writing essays, reviews, small snippets of code etc. It was buggy, not very smart and even thought it became an instant hit, it was seen more as a tool that can help you draft your mail but nothing more serious. Skeptics brushed it aside as being dumb and not good enough for anything that involved any level of complex reasoning.

The next 2 years however saw unprecedented progress that would change the outlook of LLM chatbots from being tools that can help you draft a mail to tools that could replace you. The models kept getting bigger, they kept getting better and the benchmarks kept getting climbed. The interest in ChatGPT also saw billions poured into to other competetitors Anthropic, Mistral, Gemini, Llama trying to come up with their own version of an LLM powered chatbot.

February 24, 2023

Soon Open-source models started to catch-up as well, the number of model's hosted on HuggingFace increased 40x in just 1 year, from Jan 2023 to Jan 2024. Companies that had missed the boat but had enough talent drove the progress of Open-source models, on Feb 24, 2023, Meta reluctantly released the weights of their LLM Llama after someone had uploaded its weight on 4chan a week back. This kick-started the open-source progress. Llama was the first good competent open source LLM. It reportedly cost Meta 20 million dollars to train it, a huge sum that the open source community couldn't have come up with but it represents less than 0.015% of Meta's annual revenue (~130 billion). For what is chump change, Meta had successfully created an alternative to proprietary models and...
Beyond Gaussian: Language Model Representations and Distributions
by Matt Levinson — 2024-11-24T01:53:38.156Z

In January 2023, beren and Eric Winsor

cataloged basic distributional properties

of weights, activations, and gradients in GPT-2 models, providing a systematic view of model internals (thanks to Ryan Greenblatt for the pointer). This post extends their investigation in two directions.

First, examining their characterization of transformer activations as "nearly Gaussian with outliers," I conducted detailed distributional analyses of post-residual activations. Myfindings align with observations made in comments and unpublished work: the distributions are better described by heavier-tailed distributions, dominated by the logistic distribution. What can appears as outliers or artifacts manifests in my analysis as consistent mixture distributions, with minor modes appearing systematically to both sides of the primary distribution.

Second, prompted by Buck at Redwood Research (who shared initial code), I investigated how different aspects of language - subjects, attributes, and styles - interact in the model's representational space. I analyzed the relative distances between representations of sentences varying in subject matter and attributes (like tone or language) and found some patterns in how these characteristics compete for dominance in the model's internal representations. Perhaps most interesting, there is a consistent diminishing of these distances through the model's layers. Overall I had ten subjects, twelve attributes, and 200 sentences for each pair generated by Claude 3.5 Sonnet (before the update a few weeks ago). This is the dataset used for all of the work presented here.

Below I’ll run through some of the most intriguing patterns I found and some thoughts on potential future directions. I do not expect distributional analysis to become a primary tool in safety or interpretability. My goal is to contribute to the higher level characterization of transformer based LLMs. My hope is that such characterization may suggest directions for future work that can contribute directly to safety and interpretability goals. I’d be interested to hear from the community if you have any questions or thoughts about future work.

Logistic Dominated Mixture Distribution

The logistic distribution dominated. I fit a range of distributions to the concatenation of the post-residual activations at each layer for each subject:attribute pair for GPT2-Small, GPT2-Medi...
Compute and size limits on AI are the actual danger
by Shmi — 2024-11-23T21:29:37.433Z

Epistemic status: rather controversial and not very well researched :) Not super novel, I assume, but a cursory look did not bring up any earlier posts, please feel free to link some.

Intuition pump: bigger brain does not necessarily imply a smarter creature. Apes are apparently smarter than elephants and dolphins appear smarter than blue whales. There is definitely a correlation, but the relationship is far from certain.

Starting point: intelligence is roughly equivalent to the degree of abstraction of the world models (detecting Dennett's "real patterns", at increasingly higher level). Humans are much better at abstract thought than other animals, and one can trace the creature's ability to find higher-level patterns in the world (including themselves) with higher intelligence throughout the natural and artificial world.

A non-novel point: Abstraction is compression. Specifically, abstraction is nothing but a lossy compression of the world model, be it the actual physical world, or the world of ideas.

An obvious point: generating good abstractions is expensive. If you have enough resources to use your existing mental capacity, there is no reason to expend resources on generating better abstractions. If you have room to grow your brain to add more of the same-level patterns, it is cheaper than building better abstractions in the same brain size.

A less obvious point is how hard building good abstractions is. This is what theoretical research is and what separates the likes of Einstein, Dawkins and Nash from the rest of us.

An implication: size and compute restrictions while facing the need to cope with novel situations facilitate abstraction building.

A just so story: human brain size is (currently) constrained by the head size, which is constrained by the hip size due to having to walk upright, which is constrained by the body mass due to resource availability and, well, gravity, resulting in abstraction building being a good way to deal with the changing environment.

Current AI state: the LLMs now get smarter by getting larger and training more. There are always compute and size pressures, but they are not hard constraints, more like costs. Growing to get more successful, the elephant way, not the human way, seems like a winning strategy at this point.

Absolute constraints spark abstraction building: the vetoed California bill SB 1047 "covers AI models...