trentmkelly/LessWrong-43k huggingface dataset:
Why comparative advantage does not help horses
by Sherrinford — 2024-09-30T22:27:57.450Z

This post discusses what statements about comparative advantage say and what they do not say, and why comparative advantage does not save horses from getting sent to glue factories. It is only marginally about AI.

Eliezer Yudkowsky, in "The Sun is big, but superintelligences will not spare Earth a little sunlight", explains Ricardo's Law of Comparative Advantage and then writes:

Ricardo's Law doesn't say, "Horses won't get sent to glue factories after cars roll out."

Ricardo's Law doesn't say (alas!) that -- when Europe encounters a new continent -- Europe can become selfishly wealthier by peacefully trading with the Native Americans, and leaving them their land.

Their labor wasn't necessarily more profitable than the land they lived on.

Comparative Advantage doesn't imply that Earth can produce more with $77 of sunlight, than a superintelligence can produce with $77 of sunlight, in goods and services valued by superintelligences. It would actually be rather odd if this were the case!

These (negative) statements are true, but they may also create confusion. Why? Because they suggest something unclear or wrong about what Ricardo's insight does and does not apply to - i.e., what comparative advantage is about.

Eliezer presents a standard example of comparative advantage, two countries trading, and he says that "Ricardo's Law of Comparative Advantage, ... shows that even if the country of Freedonia is more productive in every way than the country of Sylvania, both countries still benefit from trading with each other." But what does this require?

It is useful to consider Ricardo's example. It starts from a situation in which, in "the absence of trade, England requires 220 hours of work to both produce and consume one unit each of cloth and wine while Portugal requires 170 hours of work to produce and consume the same quantities". Given their production technologies, their natural endowments, or economies of scale (depending on the aspect of reality you are focusing on, or on the trade model you are employing), the countries together can produce more if they specialize. Or, as Wikipedia puts it after presenting a “typical modern interpretation of the classical Ricardian model”: "by trading and specializing in a good for which it has a comparative advantage, each country can expand its consumption possibilities. Consumers can choose from bundles of wine and cloth th...
Intelligence explosion: a rational assessment.
by p4rziv4l — 2024-09-30T21:17:35.675Z
Link: https://docs.google.com/spreadsheets/d/1_k_4dV3OTBf2D8mHr1uDNvNu39A4HJRqV-S-BbQ44Bs/edit?usp=sharing

Is there a more detailed analysis of the reasons for and against the intelligence explosion?
Also interested in other analytic timeline predictions.
Peak Human Capital
by PeterMcCluskey — 2024-09-30T21:13:30.421Z
Link: https://bayesianinvestor.com/blog/index.php/2024/09/30/peak-human-capital/

TL;DR: AI will soon reverse a big economic trend.

Epistemic status: This post is likely more speculative than most of my
posts. I'm writing this to clarify some vague guesses. Please assume
that most claims here are low-confidence forecasts.

There has been an important trend over the past century or so for human
capital to increase in value relative to other economically important
assets.

Context

Perplexity.ai says:

A 2016 economic
analysis
by Korn Ferry found that:

Human capital represents a potential value of $1.2 quadrillion to
the global economy.

This is 2.33 times more than the value of physical capital, which
was estimated at $521 trillion.

For every $1 invested in human capital, $11.39 is added to GDP.

I don't take those specific numbers very seriously, but the basic
pattern is real.

Technological advances have reduced the costs of finding natural
resources and turning them into physical capital.

Much of the progress of the past couple of centuries has been due to
automation of many tasks, making things such as food, clothing,
computers, etc. cheaper than pre-industrial people could imagine. But
the production of new human minds has not at all been automated in a
similar fashion, so human minds remain scarce and valuable.

This has been reflected in the price to book value ratio of stocks. A
half century ago, it was common for the S&P 500 to trade at less than 2
times book value. Today that ratio is close to 5. That's not an ideal
measure of the increasing importance of human capital - drug patents
also play a role, as do network effects, proprietary data advantages,
and various other sources of monopolistic power.

AI-related Reversal

AI is now reaching the point where I can see this trend reversing, most
likely by the end of the current decade. AI cognition is substituting
for human cognition at a rapidly increasing pace.

This post will focus on the coming time period when AI is better than
humans at a majority of tasks, but is still subhuman at a moderate
fraction of tasks. I'm guessing that's around 2030 or 2035.

Maybe this analysis will end up only applying to a brief period between
when AI starts to have measurable macroeconomic impacts and when it
becomes superintelligent.

Macroeconomic Implications

Much has been written about the effects of AI on employment. I don't
have much that's new to say about that, so I'll just make a few
predictions tha...
Sam Altman's Business Negging
by Julian Bradshaw — 2024-09-30T21:06:59.184Z
Link: https://www.bloomberg.com/opinion/authors/ARbTQlRLRjE/matthew-s-levine?cmpid=BBD093024_MONEYSTUFF

Quoting from Matt Levine's Money Stuff newsletter:

The place you want to reach in your career is where you work for a company and you are like “you know what, I am just so rich, I don’t want you to pay me anymore, it’s fine, I’ll work for free,” and your bosses are like “nope, sorry, we insist, we cannot allow you to work here for less than $10 billion.” And then you’re like “ohhhhhhhh fine, fine, fine, I do love working here, and if there’s really no other way, I guess I will take the $10 billion.” Nothing remotely like this has ever happened to me in my life but here’s Sam Altman:

> On Thursday, Altman told some employees that there were “good reasons” he shouldn’t take equity, though he didn’t elaborate. And he said investors were pushing for an equity grant to align his financial interests with those of OpenAI, said someone who heard the comments.

> Altman also said a Wednesday news report that he might get a 7% stake in the new OpenAI was “ludicrous.”

We talked about this last week: There have been reports that, as OpenAI becomes a for-profit company, it might give Altman (its co-founder and chief executive officer, who currently owns no equity) a 7% stake worth about $10 billion. I surmised that this was not something he wanted, but something the investors wanted: “He is the founder and CEO of a hot startup, and the founder and CEO of a hot startup is supposed to own equity. Not just for his sake — not just so that he can be rich — but to align incentives.” And here is Altman saying that: He doesn’t want the $10 billion, but the investors are insisting.

Nobody in history has ever been better at, like, business negging than Sam Altman. He got OpenAI to a $150 billion valuation in part by going around saying “oh no, nobody should allow us to build our product, we’re going to destroy humanity,” and now he is allegedly going to get handed a $10 billion stake in OpenAI because he’s going around saying “oh no, nobody should give me equity, that’s ludicrous.”
In-Context Learning: An Alignment Survey
by alamerton — 2024-09-30T18:44:28.589Z
Link: https://docs.google.com/document/d/1qZCfqzVhJOgWQ3ggR204CeVj9WJ7zyE2u0KT9EVbUS0/edit?usp=sharing

Epistemic status

: new to alignment; some background. I learned about alignment about 1.5 years ago and spent the last ~1 year getting up to speed on alignment through 12 AI safety-related courses and programmes while completing an artificial intelligence MSc programme. Ultimately this post is conjecture, based on my finite knowledge of the alignment problem. I plan to correct errors that are pointed out to me, so I encourage you to please point out those errors!

(Full version available here)

TL;DR

Much research has been conducted on in-context learning (ICL) since its emergence in 2020. This is a condensed survey of the existing literature regarding ICL, summarising the work in a number of research areas, and evaluating its implications for AI alignment. The survey finds that much of the work can be argued as negative from the perspective of alignment, given that most work pushes model capabilities without making alignment progress. The full survey, containing annotations for each paper, can be found at the Google Drive link above.

Introduction

Since 2020, large language models (LLMs) have started displaying a new emergent behaviour – in-context learning (ICL) – the ability to learn tasks from prompting alone, with no updates to the LLM’s parameters. Explanations for ICL differ, and whether learning is taking place in any meaningful way is an unanswered question. ICL is a significant phenomenon and is important for the future of AI research and development.

Previous Work

Some work has surveyed the ICL landscape from different angles.

Dong et al. (2023)

summarise progress and challenges of ICL from a general perspective, mostly placing emphasis on improving ICL rather than evaluating alignment. The survey is now 18 months old, and while AI safety is mentioned, it is not the focus.

Zhou et al. (2024)

provide a more recent survey, focusing on interpretations and analyses of ICL from both a theoretical and empirical perspective. They provide an up-to-date categorisation of ICL, but not from the perspective of AI alignment.

ICL Impacts Alignment

ICL was recently highlighted by

Anwar et al. (2024)

as a foundational challenge in the safety of LLMs, due to its currently uninterpretable nature, and the lack of conclusive explanatory theories regarding how it works. This survey addresses the lack of discussion in the current literature about ICL from the perspec...
Not Just For Therapy Chatbots: The Case For Compassion In AI Moral Alignment Research
by kenneth_diao — 2024-09-30T18:37:20.409Z

Purpose and Introduction

Much AI Safety research has focused on ensuring AIs conform to the intentions implicitly given by humans, but this approach alone may be insufficient (Ji et al., 2024, Section 4.3). Some recent research has explicitly used human values and preferences as benchmarks for AI moral reasoning (e.g., Hendrycks et al., 2020; Jin et al., 2022), and concluded that the study of human moral psychology could be instrumental for improving AI moral reasoning and behavior. While identifying human values and evaluating AIs against them is important, it may not be the only way in which human psychology can be applied to AI moral alignment. I think it is important to evaluate the underlying processes which inform moral cognition and action, and not just the outcomes themselves. This is especially important because historical and contemporary evidence suggests that our values are still in development (Singer, 1981), so that aligning to our current values may be suboptimal. The question this article has been motivated by is this: what cognitive processes are required in an agent for that agent to develop good moral reasoning and be open to further moral development?

Unfortunately, I am not able to fully answer this question here. However, I argue that one key component of morality in both humans and AIs is compassion, which includes an emotional component. My intuition is that many people in AI Safety and adjacent communities hold some form of belief that true morality is derived from reason, and that affective empathetic feelings—like the anguish we feel when we see an emaciated child or an oil-drenched duckling—interfere with our capacity to be impartially moral. As Nate Soares famously said, “my care-o-meter is broken.” The evidence I have seen leads me to agree that this is true. Yet there is also compelling evidence that affective empathy is important for moral development, and that it contributes to impartial moral regard. To me, this does not necessarily result in a contradiction; rather, it suggests that both affect and reason are important for morality, and that the way in which they interact with one another is especially important for moral development.

Ideally, there might be some term which takes exactly all the nodes and relationships we want from this web of concepts and sub-concepts and leaves the others, but I do not know of such a term. I...
Exploring Decomposability of SAE Features
by Vikram_N — 2024-09-30T18:28:09.348Z

TL;DR

SAE features are often less decomposable than the feature descriptions imply. By leveraging a prompting technique to test potential sub-components of individual SAE features (for example (using the analogy from the linked post) decomposing Einstein into “German”, “physics”, "relativity” and “famous”), I found very divergent behaviour in how decomposable these features were. I built an

interactive visualization

to explore these differences by feature. The key finding is that although many features can be decomposed in a human-intuitive way such as in the Einstein example above, many cannot, and these indicate more opaque model behaviour.

Motivation

The goal of this writeup is to explore the atomicity and decomposability of SAE features? How precisely do they describe the sets of inputs that will cause them to activate? Are there cases where inputs that activate SAEs are non-intuitive and unrelated to concepts that we might expect to be related?

Apart from being an interesting area of exploration, I think this is also an important question for alignment. SAE features represent our current best attempt at inspecting model behaviour in a human-interpretable way. Non-intuitive feature decompositions might indicate the potential for alignment failures.

Prior Work

I was inspired by

this work

on “meta-SAEs” (training SAEs on decoder directions from other SAEs) because it clearly demonstrated that SAE features aren’t necessarily atomic, and it is possible to decompose them into more granular latent dimensions. I was curious as to whether it was possible to come at this problem from a different direction. Given a particular SAE feature, can we generate inputs that “should” activate this feature in areas that a human would think of as related, and observe the feature activations that we see.

Methodology

I used the pretrained Gemma Scope SAEs and randomly sampled features from layer 20 of the 2B parameter model. The choice of layer 20 was somewhat arbitrary – as a layer deeper in the model the hope was to be able to work with more high-level and abstract features. I prompted ChatGPT to produce, for each feature:

A set of five sub-components that “comprised” that feature, as discussed in the Einstein example above.
For each subcomponent, a set of three “activating phrases” – sentences that we would expect to activate this subcomponent, if the subcomponent wer

...
Knowledge Base 1: Could it increase intelligence and make it safer?
by iwis — 2024-09-30T16:00:14.236Z

This series of posts presents the idea of building a knowledge database that allows for the collection and exchange of reliable information between people, computers, and between people and computers. It also discusses how this database could increase the intelligence of both people and computers, and increase the safety of AGI.

Usefulness of credible information for people and computers

Correct information often lets us make better decisions. AI will also perform better if it has access to a larger amount of credible information, both during learning and when answering questions. Therefore, this database could increase the collective intelligence of people and the intelligence of computers.

Independent functioning of AI in the information space

The database would be a vast and credible model of a significant part of the physical world we live in, understandable to people, computer programs, and AI. The fact that the model is understandable means that both people and computers can interact with it intelligently.

A child learns about the world through interactions resulting from being "immersed" in the physical environment. One of the primary goals of a living organism is survival, and the absence of pain is an important indicator of whether it is achieving this goal well.

How could AI independently understand the world and function within it? Instead of immersing AI-controlled robots in the physical environment of people, I propose to immerse intelligent agents in an information space that reflects our world. Interactions with this space would involve exchanging information (i.e., reading from and writing into this space) and evaluating its correctness. One of the primary goals of an intelligent agent would be to answer questions correctly, and its reputation would be an indicator of achieving this goal well. With this defined goal, AI could autonomously pursue objectives important for people.

AI safety through human-readability

Because the information space would be human-readable, the actions of AI would be more transparent and, therefore, safer. AI and people could interact with the database in the same way, but using unique abilities of their intelligence, complementing and correcting their actions and learning the most effective reasoning techniques from each other.

The beginning of the idea

The idea originated from fundamental considerations rather than by analo...
Point of Failure: Semiconductor-Grade Quartz
by Annapurna — 2024-09-30T15:57:40.495Z
Link: https://jorgevelez.substack.com/p/point-of-failure

ChatGPT 4o’s interpretation of semiconductor grade quartz

We rarely think about where our stuff comes from or how it’s made. We go through our lives expecting that the things we consume are easily acquired. That is the beauty of modern society: supply chain logistics work so well that we seldom think about the consequences if these systems are disrupted. I think many of us thought about this for the first time during the COVID-19 pandemic. It was a wake-up call that revealed how fragile these systems could be, as it disrupted everything from basic goods to high-tech products.

Since the pandemic, I’ve become mildly interested in other supply chain vulnerabilities that could arise. Recently, I discovered one that is particularly concerning: the supply of semiconductor-grade quartz, which virtually all comes from one place—Spruce Pine, North Carolina.

Quartz Mine in Spruce Pine, North Carolina.

Source

What is semiconductor-grade quartz?

Semiconductor-grade quartz is a highly purified form of silicon dioxide (SiO₂), essential for producing silicon wafers used in microchips. These chips power the modern world, from smartphones to cars. Although quartz is the most abundant mineral on Earth, only an extremely small amount of it can be refined to reach the 99.9999% purity (6N) required for semiconductor production. The reason? Most quartz contains trace amounts of contaminants like iron and aluminum, which make it unsuitable for high-tech applications.

Currently, the only known deposit in the world capable of consistently producing ultra-high-purity quartz for semiconductors is located in the mountains surrounding Spruce Pine, North Carolina. Only two companies,

The Quartz Corp

and

Covia Corp

,

operate in this area, tightly controlling the extraction and refinement processes.

To me, it is incredibly fascinating and at the same time concerning that such key material is only produced in one place by an oligopoly.

What are the alternatives?

As of now, there are no scalable alternatives to the semiconductor-grade quartz produced in Spruce Pine. Refining lower-purity quartz is possible but extremely expensive, requiring massive energy consumption and producing significant hazardous waste. Synthetic quartz is another option, but its production is still relatively small and expensive, with only a few companies in the U.S., Germany, Japan, and France producing it.

The Point of Failure

...
on bacteria, on teeth
by bhauth — 2024-09-30T15:56:56.830Z
Link: https://bhauth.com/blog/biology/tooth%20bacteria.html

You may have heard that tooth decay is caused by bacteria producing lactic acid. Let's consider that a little more deeply.

criteria for decay

To effectively cause cavities, bacteria must meet 4 criteria:

Anchoring (with special proteins) to either the tooth surface or something connected to it.

Biofilm production to trap nutrients and protect the bacteria.

Metabolism that produces acid, especially lactic acid.

Growth in acidic conditions.

Some of those can be done by different bacteria, but cavities are made most effectively when a single species can do all of them.

many species of bacteria exist

Converting glucose to lactic acid is one of the (biologically) easiest ways to get energy (ATP) from it. As such, lactic acid bacteria are one of the most common types, and as you may know, there are a lot of bacteria species, maybe around a million. Human mouths have over 700 known bacterial species, and probably more unknown ones.

That being the case, you should expect that multiple types of bacteria are responsible for dental caries, aka cavities.

Streptococcus mutans

If you've looked into tooth decay, perhaps you've heard of S mutans, which notably meets all of the above criteria. Here's an introductory paper on it.

If you have a sugary drink, then notice your teeth became more sticky, that's not the sugar just naturally sticking to your teeth. S mutans makes an enzyme (dextransucrase) that makes a sticky polymer (dextran) from sucrose specifically; using sucrose is more thermodynamically favorable than using glucose or fructose. But of course, there are other bacteria that make exopolysaccharides.

S mutans also:

Produces lactic acid.

Can tolerate relatively low pH.

Uses sortase enzymes to anchor cell wall proteins to teeth.

looking under the lamppost

I've heard people say that "S mutans is the bacteria responsible for tooth decay", and I was immediately suspicious. Sure, it's been found on teeth, and it can degrade enamel, but outside of a lab, there are always multiple bacteria.

S mutans grows better in lab cultures than most bacteria, which is part of why it's been focused on. Why is that?

S mutans makes mutacins, which are a type of antibiotic. (The ones produced are strain-specific.) Because of that, at high densities and without something to wash away extracellular chemicals, S mutans tends to outcompete other bacteria present in mouths. As a result, when p...
SB 1047 gets vetoed
by ryan_b — 2024-09-30T15:49:38.609Z
Link: https://www.reuters.com/technology/artificial-intelligence/california-governor-vetoes-contentious-ai-safety-bill-2024-09-29/

Just what it says on the tin. Covered most everywhere, but I found the quote in this Reuters article stuck out to me the most:

Newsom said the bill "does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data" and would apply "stringent standards to even the most basic functions — so long as a large system deploys it."

Emphasis mine. So the governor says he vetoed it for the exact reason that it would have been a good law, in my view.