trentmkelly/LessWrong-43k huggingface dataset:
Testing "True" Language Understanding in LLMs: A Simple Proposal
by MtryaSam — 2024-11-02T19:12:34.691Z

The Core Idea

What if we could test whether language models truly understand meaning, rather than just matching patterns? Here's a simple thought experiment:

Create two artificial languages (A and B) that bijectively map to the same set of basic concepts R'
Ensure these languages are designed independently (no parallel texts)
Test if an LLM can translate between them without ever seeing translations

If successful, this would suggest the model has learned to understand the underlying meanings, not just statistical patterns between languages. Theoretically, if Language A and Language B each form true mappings (MA and MB) to the same concept space R', then the model should be able to perform translation through the composition MA·MB^(-1), effectively going from Language A to concepts and then to Language B, without ever seeing parallel examples. This emergent translation capability would be a strong indicator of genuine semantic understanding, as it requires the model to have internalized the relationship between symbols and meanings in each language independently.

Why This Matters

This approach could help distinguish between:

Surface-level pattern matching
Genuine semantic understanding
Internal concept representation

It's like testing if someone really understands two languages versus just memorizing a translation dictionary.

Some Initial Thoughts

Potential Setup

Start with a small, controlled set of basic concepts (colors, numbers, simple actions)
Design Language A with one set of rules/structure
Design Language B with completely different rules/structure
Both languages should map clearly to the same concepts without ambiguity

Example (Very Simplified)

Concept: "red circle"

Language A: "zix-kol" (where "zix" = red, "kol" = circle)
Language B: "nare-tup" (where "nare" = red, "tup" = circle)

Without ever showing the model that "zix-kol" = "nare-tup", can it figure out the translation by understanding that both phrases refer to the same concept?

Open Questions

How do we ensure the languages are truly independent?
What's the minimum concept space needed for a meaningful test?
How do we efficiently validate successful translations?

Limitations

As an undergraduate student outside the AI research community, I acknowledge:

This is an initial thought experiment
Implementation would require significant resources and expertise
Many practical challenges would need to be addressed

Call for Discussion

I'...
Fragile, Robust, and Antifragile Preference Satisfaction
by adamShimi — 2024-11-02T17:25:55.986Z
Link: https://formethods.substack.com/p/fragile-robust-and-antifragile-preference

What do I want to do?

This sounds like a question with an obvious answer, but the more I reflect on it, the more I notice complexity and subtleties.

First, there's a difficult tradeoff between what I want locally and globally: I regularly wish to do thing in the moment that would go against my global preferences (eat too much vs being healthy, watching random youtube videos vs reading the books I care about...), and I regularly don't want to do in the moment things that would push my global goals (study a boring topic, cook dinner, empty the dishwasher...)

But in addition, my preferences are also daily and hourly perturbed by external forces:

Easy empty distractions, like youtube, social media, junk food...
Internalized social norms, such as valuing depth and expertise and specialization, or feeling obligated to help people and contribute to the world[1]
Stimulus I get from the environment, such as my wife proposing some outing or me seeing a book title or a blog post that sounds interesting

In light of these difficulties, I've been searching for a practical model of preference satisfactions that helps me actually by happy about how I spent my day, without undue anxiety or regret.

I think I've found a starting point while reflecting on Taleb's triad of fragility, robustness, and antifragility.

Taleb's Triad

Taleb introduces his triad in his book

Antifragile

.

Notably, he asks what is the opposite of fragility. If fragile things get worse/break from randomness/variation/noise, then the opposite of fragile is not robust (things which resist randomness/variation/noise), but instead antifragile (things which benefit from randomness/ variation/noise).

And a big point of Taleb's book is that the same triad (fragile, robust, antifragile) reappears in many different settings, from the medical to the scientific- technological and the socio-political.

Indeed, it even applies to preference satisfaction.

Preference Fragility

As an anxious person, I am very good at fragilizing my preferences; that is, I'm very good at making myself less tolerant to noise/uncertainty/variations.

Which also means that I'm a decent exemplar from which to explore the mechanisms of preference fragilization.

Say I wake up tomorrow, and decide that I want to go to my favorite bakery in London, sit in my favorite spot, and eat some of their delicious almond financiers.

Have you noticed how I just innocently fragil...
Higher Order Signs, Hallucination and Schizophrenia
by Nicolas Villarreal — 2024-11-02T16:33:10.574Z
Link: https://nicolasdvillarreal.substack.com/p/higher-order-signs-hallucination

Picture yourself on a beach. The beach isn’t on Earth, it’s on a planet that’s going to be colonized by humans several thousand years from now. Let’s say the water is a little different color than what we’re used to, somewhat greener. Whatever you just experienced was not a phenomena which you’ve actually been exposed to in real life. You just used your imagination, and extrapolated from what experiences you have to figure what it might be like. But you know this could never happen to you in real life. Now try to imagine yourself going and getting a glass of water. That’s something most people reading this could do in just a few minutes. You could probably get up right now and do it. Because this is a rather familiar experience, you can extrapolate each aspect of the experience to a high degree of confidence. How do you know you didn’t already do it, just now? That’s a serious question. Likewise, you could imagine someone else drinking water, in another building, perhaps one you know. How do you know you’re not that person, somewhere else, drinking water? That’s also a serious question.

People with schizophrenia will sometimes speak about things that haven’t happened to them as if they had really happened to them. Sometimes mentally healthy people will do the same in more specific circumstances, for example, think that a dream from several weeks or months ago had actually happened to them in their waking world, or otherwise modify memories after the fact through new associations. When we predict or anticipate things, we’re also imagining what’s going to happen, and we may or may not be right. From our perspective before seeing the result of the prediction, there is no difference in experience between the right or wrong prediction. The thing about imagination, prediction, anticipation, hallucinations, delusions and dreams is that they are experiences that aren’t directly communicated to us as a real experience of the external world, or otherwise correlated to the material phenomena they invoke/stand in for.

Recently, I’ve been writing a bit about semiotics and AI, as well as reading Steps to an Ecology of Mind by Gregory Bateson which, among other things, provides some theoretical speculation about the causes of schizophrenia. Ruminating on these topics, I feel that there is something here that is crucial to understanding the development of artifici...
Is OpenAI net negative for AI Safety?
by Lysandre Terrisse — 2024-11-02T16:18:02.859Z

I recently saw a post arguing that top AI labs should shut down. This let me wonder whether the AI Safety community thinks OpenAI is net negative for AI safety. I chose OpenAI because I consider it as the most representative top AI lab (in the sense that, if we ask someone to think about an AI lab, they would probably think about that one), but if you want, you can also talk about other AI labs as well.
Two arguments against longtermist thought experiments
by momom2 — 2024-11-02T10:22:11.311Z

Epistemic status: shower thoughts.

I am currently going through the EA Introductory Course and we discussed two arguments against longtermism which I have not seen elsewhere.

So goes a thought experiment: imagine you have toxic waste at hand, which you can process right now at the cost of 100 lives, or bury it so it'll have no effect right away but poison the land, at the cost of 1000 lives in 100 years. Should you do it? Should you do the opposite tradeoff?
The basic intuition of longtermism is that clearly, the 1000 lives matter more than the 100, regardless of their position in time.

From Introduction to longtermism:

Imagine burying broken glass in a forest. In one possible future, a child steps on the glass in 5 years' time, and hurts herself. In a different possible future, a child steps on the glass in 500 years' time, and hurts herself just as much. Longtermism begins by appreciating that both possibilities seem equally bad: why stop caring about the effects of our actions just because they take place a long time from now?

Faced with this tradeoff, I'd save the 100 immediate lives. More than that, longtermism as assigning-significant-value-to-far-future-things has almost nothing to do with this thought experiment.

The first reason is a matter of practical mindset which does not undermine longtermist principles but I feel like it's overlooked.

The second reason is more central to deprioritizing directly far-reaching actions in general.

My criticisms basically don't matter for practical caring-about-far-future-people, but I still find it annoying that the thought experiments used to build longtermist intuitions are so unrelated to the central reasons why I care about influencing the far future.

Choose actions, not outcomes

The first reason is that in practice, we face not a direct choice about outcomes (100 vs 1000 lives), but over actions (processing vs burying the waste) and so the hypothetical is fraught with assumptions about the irrelevance of indirect consequences by abstracting away the causal pathways through which your action has an impact.

For example, the 100 people we save now will have a lot of impact in 100 years, which will plausibly compound to more than 10 future life saved per living person today.

This could make sense if the population over the next 100 years is more than 10 times today's population, and you assign an equal share of re

...
Both-Sidesism—When Fair & Balanced Goes Wrong
by James Stephen Brown — 2024-11-02T03:04:03.820Z
Link: https://nonzerosum.games/bothsidesism.html

In a few days time, voting will close for millions of Americans in one of the most contentious and globally consequential elections in world history. And while this week’s subject; both-sidesism, is ‘evergreen’ in that the topic will continue to be relevant long into the future, this election has highlighted its significance and introduced some new turns.

A Poisoned Chalice

In the quest for a fair and informed society, the notion of presenting both sides of an argument has been deeply ingrained in our understanding of balance. This principle was institutionalised in the US with the introduction of the Fairness Doctrine in 1949.

“The Fairness Doctrine was rooted in the notion that the broadcast spectrum is a public resource, and those who are licensed to use it have an obligation to serve the public interest by facilitating informed debate.”
-

Encyclopædia Britannica

While well-intentioned, the doctrine inadvertently codified the fallacy of both-sidesism, where equal representation is given to opposing viewpoints regardless of merit or evidence. This has had lasting implications on public discourse, science communication, and the health of our democracy.

Equivocation and False Equivalence

Both-sidesism often relies on two logical fallacies: equivocation and false equivalence. Equivocation uses ambiguous language to mask the truth, while false equivalence presents two unequal things as equal.

There Really is a Lesser of Two Evils

We see false equivalence playing out in the political arena when commentators claim there is “radicalism” on both sides in reference to radical woke leftists quibbling about pronouns in relation to a mainstream right-wing presidential candidate promising to use the military against the “enemy within”.

Presently, radical left-wing ideas are simply not comparable to radical right-wing ideas. A recent

study in Nature

found that…

… conservatives were less supportive of political equality and legal rights and guarantees and more willing to defect from democratic “rules of the game” and vote for anti-democratic candidates, even after adjusting for political extremism.
-Débora de Oliveira Santos & John T. Jost

So, by implying that extreme views are equally distributed across the political spectrum, both-sidesism masks real threats to democratic values.

Science is not Democratic

Equivocation is often used to undermine scientific discourse, the term ‘theo...
What can we learn from insecure domains?
by Logan Zoellner — 2024-11-01T23:53:30.066Z

Cryptocurrency is terrible. With a single click of a button, it is possible to accidentally lose all of your funds. 99.9% of all cryptocurrency projects are complete scams (conservative estimate). Crypto is also tailor-made for ransomware attacks, since it makes it possible to send money in such a way that the receiver has perfect anonymity.

typical web3 experience

Similarly, Cyber Security is terrible. Basically every computer on the internet is infected with multiple types of malware. If you have ever owned a web-server with a public IPV4 address, you undoubtedly have had the pleasure of viewing a log file that looks like this:

Every IPV4 address on earth is under constant attack by malware

In a few months, the world is about to be introduced to a brand new insecure by design platform, the LLM agent:

No one worth taking seriously believes that Microsoft Copilot (or Anthropic, or any other LLM agent) is going to be remotely secure against prompt injection attacks.

One fascinating thing (to me) about these examples is that they all basically work fine[1]. Despite being completely broken, normal people with normal intelligence use these systems routinely without losing 100% of their funds. This happens despite the fact that people with above-average intelligence have a financial incentive to take advantage of these security flaws.

One possible conclusion is along the lines of "everything humanity has ever built is constantly on fire. We must never built something existentially dangerous or we're already dead."

However we already did:

And like everything else, the story of nuclear weapons is that they are horribly insecure and error prone.

What I want to know is why? Why is it that all of these systems, despite being hideously error prone and blatantly insecure by design somehow still work?

I consider each of these systems (and many like them) a sort of standing challenge to the fragile world hypothesis. If the world is so fragile, why does it keep not ending?

^

If anyone would like to make a bet, I predict 2 years from now LLM agents:

will be vulnerable to nearly trivial forms of prompt-injection
Millions of people will use them to do things like spend money that common-sense tells you not to do on a platform this insecure by design
Science advances one funeral at a time
by Cameron Berg — 2024-11-01T23:06:19.381Z

Major scientific institutions talk a big game about innovation, but the reality is that many of the mechanisms designed to ensure quality—peer review, funding decisions, the academic hierarchy—

explicitly

incentivize

incremental

rather than revolutionary progress; are far more oriented towards 'day science' than 'night science.'[1]

Thomas Kuhn's now-famous notion of

paradigm shifts

was pointing at precisely this phenomenon. When scientists work within what Kuhn called "normal science," they're essentially solving low- to medium-stakes puzzles within their field's accepted framework. While it's fairly easy to evaluate the relative quality of work that occurs within any given paradigm, Kuhn argued it's nearly impossible for scientists to reason about the relative power of different paradigms for a given field—especially when they have already drank the paradigmatic kool-aid.

Max Planck captured this idea succinctly in his biting statement that "science advances one funeral at a time."[2]

There is no shortage of examples of this occurring throughout the history of science:

Ignaz Semmelweis suggested that doctors wash their hands between patients. The medical establishment ridiculed and ostracized him until his career was destroyed. Today hand washing is basic medical practice.
Barbara McClintock discovered genes could jump between chromosomes in maize in 1948. Geneticists dismissed her work for decades as it didn't fit their tidy theory of inheritance. She won the 1983 Nobel Prize for the same research they rejected.
Barry Marshall grew confident that ulcers came from bacteria, not stress. When no one would listen, he drank H. pylori in 1984 to prove it. Won the 2005 Nobel Prize for work doctors had called absurd.
Katalin Karikó lost her position and funding at UPenn in 1995 for pursuing mRNA research. She kept working on the "fringe" technology despite the setbacks. That same technology became the basis for the COVID-19 vaccines.
Alfred Wegener proposed continents move across the Earth. Geologists mocked him until his death in 1930. Plate tectonics became accepted theory in the 1950s when the evidence became overwhelming.
Lynn Margulis argued mitochondria evolved from ancient bacteria. Multiple journals rejected her paper before its 1967 publication. Her "crazy" theory is now the cornerstone of cell biology.
Dan Shechtman discovered quasicrystals in 1

...
The Cartesian Crisis
by mindprison — 2024-11-01T23:02:45.775Z

The Cartesian Crisis, as detailed in this more verbose essay, represents an unprecedented existential threat to humanity's ability to discern reality from fiction. We stand at a critical juncture where the foundations of truth are being systematically dismantled by a perfect storm of technological and social forces, leaving civilization adrift in a sea of manufactured illusions.

This crisis emerges from multiple vectors of attack on our collective ability to reason. The institutional pillars of knowledge have succumbed to ideological corruption, while our communication channels are now dominated by algorithmic manipulation that distorts the natural flow of human discourse. Perhaps most alarmingly, artificial intelligence has emerged as the ultimate weapon in this war against truth, capable of generating convincing yet entirely fabricated realities at a scale that overwhelms human discernment.

The infiltration of AI-generated content has reached staggering levels, leading many to question just how much of the internet remains real. This has led to the concept of the "dead internet," where all interaction and content becomes bot-driven.

The algorithmic takeover of the creation of all content is rapidly accelerating while we are simultaneously loosing information amidst all the noise.

Digital Decay Of The Internet:

38% of webpages that existed in 2013 are no longer available today
Digital decay is accelerating from 3.4% per year (2013-2018) to 6% per year recently
49% of links cited in Supreme Court decisions are now broken

AI Content Generation:

77% of Americans have been misled by AI-generated content online
Over 5% of newly created English Wikipedia articles are flagged as AI-generated
Students submitted more than 22 million potentially AI-generated papers in the past year

Academic Research:

AI-generated research papers increased from 3.61% to 6.22% between Nov 2022-2023
15.8% of peer reviews were written using AI assistance
AI-assisted reviews had 14.4% higher odds of assigning higher scores than human reviews

Educational:

Teachers report spending more time detecting AI plagiarism than providing actual writing feedback
Concerns about students skipping crucial brain development stages by relying on AI

Job Applications:

AI is being used to automatically create resumes and apply to 1000s of jobs for a single individual
Recruiters are using AI avatars to interview applicants to fil

...
Composition Circuits in Vision Transformers (Hypothesis)
by phenomanon — 2024-11-01T22:16:11.191Z

Idea

I just wanted to make a quick post to get this idea out there, I'll hope to do a more thorough explanation later, and experiments after that.

Basically, I hypothesize that Vision Transformers have mechanisms that I call composition circuits, which operate like induction circuits in text transformers. They begin with a mechanism akin to the previous token head in text transformers. I call this a modular attention head. It works like the previous token head, except it attends to patch embeddings (~aka "tokens") around the candidate token in multiple directions around the token of interest. We know heads like this exist in early ViT layers. We don't necessarily know they perform the functionality I propose.

A modular attention head in ViT, specifically timm/vit_base_patch32_224.augreg_in21k_ft_in1k

The second portion of the composition circuit is the, you guessed it, composition head (if you don't why this is supposedly obvious, the second head of an induction circuit is called the induction head. I'm not sure who would be reading this post who doesn't already know about induction heads, but if anyone is reading this post, I'll be happy). I haven't looked for these, but what they would do is combine the results compiled by the modular attention head. For example, if the modular attention head grabs (x-1,y-1) ... (x+1,y+1) w/ (x,y) the center token coord, the composition head just moves all the relevant features from those embeddings to the embedding at (x,y).

Why do you suspect these exist?

The primary fact motivating this hypothesis is the embedding layer of the Vision Transformer. I recently started doing some work training SAEs/etc, for Prisma, and I figured I should start thinking about what to expect from ViTs relative to text transformers. The primary observation I made was that whereas text transformers use a tokenizer/embedding matrix to convert a sentence into the embedding sequence/residual stream, a vision transformer uses d_resid convolutional kernels (of a size in (inverse) proportion to the desired patch size). So, whereas text is defined relativistically for TeTs (yep), for ViTs, the network essentially has to pick the d_resid most useful convolutional kernels it can think of and begin with all patches as a vector of features indicating their responsiveness to these kernels. So you begin the network with some very low-level information communicati...