trentmkelly/LessWrong-43k huggingface dataset:
AI & wisdom 2: growth and amortised optimisation
by L Rudolf L — 2024-10-28T21:07:39.449Z
Link: https://rudolf.website/wisdom2/

Written for the

AI Impacts essay competition on the automation of wisdom and philosophy

Growth modes and direct v amortised optimisation

Amortised optimisation is more important when relevant histories are long and growth is slow. If growth suddenly picks up, the flexibility of optimising on the fly with direct optimisation becomes more important.

For example, the role of previously-accumulated wisdom went down after the industrial revolution. In the pre-industrial world, change was slow, and you would do well to follow cultural traditions, old books, and grandparents, all of whom have collected and distilled a large amount of useful knowledge. With faster technological progress, change accelerated, and on-the-fly intelligence became more important and culturally valued. Historically, it was the young who asked the old for advice; the situation of today, where parents ask children for tech advice, is unusual. Current developed-world culture places more weight on intelligence and less on wisdom than most past cultures.

If AI accelerates the rate of change and growth, both the importance of direct optimisation, and the cultural weighting of it, may get more extreme. This will happen regardless of what sort of optimisation the AI itself is doing.

(The following has some maths - for the key takeaways, it's enough to look at the last figure, with the 3x3 grid of line graphs.)

We can try to roughly model this. Let I(T)represent the amount of "intelligence" - direct attempts at optimisation by actors in the world - at some time T. Historically, I(T) has probably tracked population trends, though would also have effects from education, human capital, and productivity. Let A(T) be the stock of results available for amortised optimisation - civilisation's total track record of accumulated knowledge, experience, and solutions. A simple relation would be that A(T) is the sum of all prior direct optimisation - A(T)=∫T0I(t)dt.

However, some knowledge becomes less relevant with time. We'll incorporate this by changing our model to A(T)=∫T0I(t)fT(t)dt, where fT(t) is a number between 0 and 1 measuring what fraction of the results I(t) generated at time t are still relevant at current time T. For example, here are some settings:

no decay: fT(t)=1 (to recover the case where results of direc

...
AI & wisdom 1: wisdom, amortised optimisation, and AI
by L Rudolf L — 2024-10-28T21:02:51.215Z
Link: https://rudolf.website/wisdom1/

Written for the AI Impacts essay competition on the automation of wisdom and philosophy

At least in name, AI is about intelligence. However, intelligence is not the only mental quality that matters. Many deep mental skills, in particular regarding good, deep, or long-term judgement, are bundled under the term "wisdom". If we automate and industrialise intelligence, we may not automate and industrialise other important mental qualities, similar to how industrialising food lead to a surplus of calories, but also harms from missing nutrients.

Wisdom

How does wisdom differ from intelligence? Here are some pairs of associations between intelligence and wisdom:

fast v slow

novelty vs distillation

young v old

fluid intelligence v crystallised intelligence

analysis v synthesis

legible / verifiable v illegible / hard to verify

done alone v learned from elders/culture

small steps well v large things right

(Some others that fit my latter points less well: complex v simple, solving v preventing, facts v meaning, question-answering v question-asking, specific v holistic)

Here are some examples that are more on the wisdom side than the intelligence side:

Among the Fuegians, an indigenous people in South America, creating an arrow is a 14-step process with 7 different tools and 6 different materials. The steps all improve the final product. The instructions emerged over a long time scale, are passed down from generation to generation. No single person invented it, or could invent it, except by learning it from their elders. (h/t The Secrets of Our Success by Joseph Heinrich, which has a dozen examples like this)

Research taste. Acquiring it requires significant time. Having it is critical for choosing research directions. Older researchers and PhD advisers occupy a social role not that distinct from wise gurus who live at the tops of mountains: they're hard-to-reach people, whose audience you can seek if you're part of the right group, and who will pass down cryptic advice that you would do well to heed.

The changes in the Kennedy administration between the botched Bay of Pigs invasion and the Cuban Missile Crisis. The fallout from the Bay of Pigs failure made them more pragmatic and less ideological, emphasise intellectual diversity, improve their intelligence analysis methods, and gain experience about crisis management.

Wise quotes (e.g. "do not ask if you can bear the truth; you alread

...
Finishing The SB-1047 Documentary In 6 Weeks
by Michaël Trazzi — 2024-10-28T20:17:47.465Z
Link: https://manifund.org/projects/finishing-the-sb-1047-documentary-in-6-weeks

(Crossposted from Manifund)

Summary of the proposal

We will produce a one-hour feature documentary about SB-1047, which would:

Serve as a comprehensive reference on the bill's history and implications.
Present a balanced view of perspectives from both proponents and opponents, bridging ideological divides.
Contribute to informed discussions about future AI Policy by providing in-depth, impartial analysis.
Enhance public understanding of AI regulation challenges, fostering more informed public discourse.
Offer policymakers, researchers, and the public a nuanced resource on various stakeholder perspectives, supporting well-informed decision-making processes.

Scott Wiener, Member of the California State Senate, SB-1047 sponsor
“Congress has not passed major tech regulation in more than a quarter century. And so in the absence of congressional action, California has a responsibility to lead.”

Funding

We are seeking $35,000 to finish the documentary before January 2nd, 2025 (10 weeks).
With $55,000, we would be able to finish the project in only 6 weeks (Dec 5), while the bill is still fresh in people’s minds, and do additional interviews.
The project can further benefit from additional funding up to $30,000 ($85,000 total), which would reimburse the time and money already spent on the project.

Dan Hendrycks, Director, Center for AI Safety
“Regulation shouldn't be written in blood."

Interviewees

We have currently completed 23 interviews (including 17 longform) with:

Proponents of the bill

All the sponsors and co-sponsors of the bill (longform, 1-3h)

Scott Wiener (Member of the California State Senate, SB-1047 sponsor)
Dan Hendrycks (Director, Center for AI Safety)
Nathan Calvin (Senior Policy Counsel, Center for AI Safety Action Fund)
Sunny Gandhi (VP of Political Affairs, Encode Justice)
Teri Olle (Director, Economic Security California)

Other proponents of the bill (longform, 1-3h)

Zvi Mowshowitz (Founder and CEO, Balsa Research, writer of “Don’t Worry About The Vase”)
Holly Elmore (Director, Pause AI)
Flo Crivello (Founder and CEO, Lindy)

Samuel Hammond (Senior Economist, Foundation for American Innovation)
"The bill moves so much further towards being basically toothless and yet is still vetoed. That sends a signal to the world that if you do a bill that's narrowly focused on catastrophic risk, that won't have the political support."

People who were initially critical and ended up somewhat

...
Towards the Operationalization of Philosophy & Wisdom
by Thane Ruthenis — 2024-10-28T19:45:07.571Z
Link: https://aiimpacts.org/towards-the-operationalization-of-philosophy-wisdom/

Written for the competition on the Automation of Wisdom and Philosophy.

Summary

Philosophy and wisdom, and the processes underlying them, currently lack a proper operationalization: a set of robust formal or semi-formal definitions. If such definitions were found, they could be used as the foundation for a strong methodological framework. Such a framework would provide clear guidelines for how to engage in high-quality philosophical/wise reasoning and how to evaluate whether a given attempt at philosophy or wisdom was a success or a failure.

To address that, I provide candidate definitions for philosophy and wisdom, relate them to intuitive examples of philosophical and wise reasoning, and offer a tentative formalization of both concepts. The motivation for this is my belief that the lack of proper operationalization is the main obstacle to both (1) scaling up the work done in these domains (i. e., creating a bigger ecosystem that would naturally attract funding), and (2) automating them.

The discussion of philosophy focuses on the tentative formalization of a specific algorithm that I believe is central to philosophical thinking: the algorithm that allows humans to derive novel ontologies (conceptual schemes). Defined in a more fine-grained manner, the function of that algorithm is “deriving a set of assumptions using which a domain of reality could be decomposed into subdomains that could be studied separately”.

I point out the similarity of this definition to

John Wentworth’s operationalization of natural abstractions

, from which I build the formal model.

From this foundation, I discuss the discipline of philosophy more broadly. I point out instances where humans seem to employ the “algorithm of philosophical reasoning”, but which don’t fall under the standard definition of “philosophy”. In particular, I discuss the category of research tasks varyingly called “qualitative” or “non-paradigmatic” research, arguing that the core cognitive processes underlying them are implemented using “philosophical reasoning” as well.

Counterweighting that, I define philosophy-as-a-discipline as a special case of such research. While “qualitative research” within a specific field of study focuses on decomposing the domain of reality within that field’s remit, “philosophy” focuses on decomposing reality-as-a-whole (which, in turn, produces the previously mentio...
Winners of the Essay competition on the Automation of Wisdom and Philosophy
by owencb — 2024-10-28T17:10:04.272Z

We’re delighted to announce the winners of the Essay competition on the Automation of Wisdom and Philosophy.

Overview

The competition attracted 90 entries in total (only one of which was obviously just the work of an LLM!), taking a wide variety of angles on the topic. The judges awarded the top four prizes as follows:

$7,000 to Rudolf Laine for essays on wisdom, amortised optimisation, and AI:

Part I: Wisdom, amortised optimisation, and AI
Part II: Growth and amortised optimisation
Part III: AI effects on amortised optimisation

$6,000 to Thane Ruthenis for Towards the operationalization of philosophy & wisdom
$4,000 to Chris Leong for essays on training wise AI systems:

An Overview of “Obvious” Approaches to Training Wise AI Advisors
Some Preliminary Notes on the Promise of a Wisdom Explosion

$3,000 to Gabriel Recchia for Should we just be building more datasets?

Additionally, the judges awarded ten runner-up prizes, of $500 each. These adjustments to the prize schedule were made to better reflect the judges’ assessments — see footnote for details[1].

Many of those essays which did not get prizes still had something valuable to recommend them, and we are very grateful for the deep thought participants put into engaging with these potentially-crucial topics.

In the rest of this post, we’ll link to all fourteen of the prize-winning entries, and provide judges’ commentary on these. We have made a point to include critical commentary as well as praise. In our view this is an essential part of the competition — helping to draw collective attention to the ideas people have presented, and (we hope) helping to advance the discourse on what makes for valuable work.

Judge introductions

Andreas Stuhlmüller (AS) — Hi, I'm CEO & cofounder of Elicit, an AI company working on scaling up high-quality reasoning, starting with science. I've been interested in how AI can differentially advance wisdom for a long time, and (pre LLMs) founded the non-profit Ought to work on that topic.

Linh Chi Nguyen (CN) — Hi, I have been thinking about long-term risk from AI besides alignment for a few years. I landed on thinking a bunch about AIs’ decision theories, which is what I’m currently doing. In that process, I’ve thought a bit more generally about the philosophical capabilities and inclinations of AI systems and I’m excited about more work in this area.

Bradford Saad (BS) — ...
Miles Brundage: Finding Ways to Credibly Signal the Benignness of AI Development and Deployment is an Urgent Priority
by Zach Stein-Perlman — 2024-10-28T17:00:18.660Z
Link: https://milesbrundage.substack.com/p/finding-ways-to-credibly-signal-the

Miles Brundage has a new substack and I like this post. Here's the introduction.

A simplified view of AI policy is that there are two “arenas” with distinct challenges and opportunities. In the domestic arena, governments aim to support AI innovation within their borders and ensure that it is widely beneficial to their citizens, while simultaneously preventing that innovation from harming citizens’ interests (safety, human rights, economic well-being, etc). In the international arena, governments aim to leverage AI to improve their national security, while minimizing the extent to which these efforts cause other governments to fear for their security (unless this is the explicit intent in a given context, e.g., deterrence or active conflict).

For example, in the domestic arena, a government might require that companies assess the degree to which a language model is capable of aiding in the creation of biological weapons, and share information about the process of red teaming that model and mitigating such risks. And governments could require documentation of known biases in the model. In each case, the government would be attempting to ensure, and produce evidence, that an AI system is benign, in the sense of not posing threats to the citizens of that country (although in some cases the government may also be mitigating risks to citizens of other countries, particularly for catastrophic risks that spill across borders, of companies with a global user base). Citizens may also demand evidence that a government’s own use of AI is benign — e.g., not infringing on citizens’ privacy — or that appropriate precautions are taken in high-stakes use cases, or one part of government might demand this of another part of government.

In the international arena, a military might build an autonomous weapon system and attempt to demonstrate that it will be used for defensive purposes only. A military might also state that it will not use AI in certain contexts like nuclear command and control.

In all of these cases, outside observers might be justifiably skeptical of these claims being true, or staying true over time, without evidence. The common theme here is that many parties would benefit from it being possible to credibly signal the benignness of AI development and deployment. Those parties include organizations developing and deploying AI (who want to be trusted), government

...
somebody explain the word "epistemic" to me
by KvmanThinking — 2024-10-28T16:40:24.275Z

definition from lesswrong's A-Z glossary:

Concerning knowledge.

huh? So then, what is an "epistemic state"? Is it a collection of ideas? Is it a combination of knowledge a brain can have? What is an "epistemic status"? Is it the current epistemic state someone is in? Is it their amount of knowledge about something? Is it their opinion of whether or something is true?
~80 Interesting Questions about Foundation Model Agent Safety
by RohanS — 2024-10-28T16:37:04.713Z

Many people helped us a great deal in developing the questions and ideas in this post, including people at CHAI, MATS, various other places in Berkeley, and

Aether

. To all of them: Thank you very much! Any mistakes are our own.

Foundation model agents - systems like AutoGPT and Devin that equip foundation models with planning, memory, tool use, and other affordances to perform autonomous tasks - seem to have immense implications for AI capabilities and safety. As such,

I (Rohan) am planning to do foundation model agent safety research

.

Following the spirit of an earlier post I wrote, I thought it would be fun and valuable write as many interesting questions as I could about foundation model agent safety. I shared these questions with my collaborators, and Govind wrote a bunch more questions that he is interested in. This post includes questions from both of us. We didn't worry too much about avoiding redundancy in our questions.

We’ll often use FM for foundation model and FMA for foundation model agent.

Rohan

I've bolded some of the questions I'm most interested in. I've also attempted to categorize the questions, mostly to make this a bit more digestible than 70+ randomly arranged questions.

Basics and Current Status

What exactly is a foundation model agent?

Above I describe them as "systems like AutoGPT and Devin that equip foundation models with planning, memory, tool use, and other affordances to perform autonomous tasks," but there is a lot of flexibility. I think Scaffolded LLMs as natural language computers does a good job illustrating just how flexible this framework is.

What are the right questions to be asking about FMA safety?

Hopefully the rest of this post will include some of the right questions!

What are the implications of FMAs for safety and capabilities?
How good are FMAs right now? What are they good at and what are they bad at?
What aspects of FMA safety are people already working on? E.g. top AGI labs, AI safety labs like METR and Apollo, governments (e.g. AISIs), academics.

A lot of evals I think, particularly dangerous capabilities evals
Based on the o1 safety card and Anthropic’s papers, some CoT faithfulness stuff? Though I don't know if Anthropic is still thinking about this.
What are tractable and important aspects of FMA safety for academics and independent researchers to work on?

One take I heard was that benchmarks are good thin

...
AI Safety Newsletter #43: White House Issues First National Security Memo on AI Plus, AI and Job Displacement, and AI Takes Over the Nobels
by Corin Katzke — 2024-10-28T16:03:39.258Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-43-white-house

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

Subscribe here to receive future versions.

White House Issues First National Security Memo on AI

On October 24, 2024, the White House issued the first National Security Memorandum (NSM) on Artificial Intelligence, accompanied by a Framework to Advance AI Governance and Risk Management in National Security.

The NSM identifies AI leadership as a national security priority. The memorandum states that competitors have employed economic and technological espionage to steal U.S. AI technology. To maintain a U.S. advantage in AI, the memorandum directs the National Economic Council to assess the U.S.’s competitive position in:

Semiconductor design and manufacturing
Availability of computational resources
Access to workers highly skilled in AI
Capital availability for AI development

The Intelligence Community must make gathering intelligence on competitors' operations against the U.S. AI sector a top-tier intelligence priority. The Department of Energy will launch a pilot project to evaluate AI training and data sources, while the National Science Foundation will distribute compute through the National AI Research Resource pilot.

The NSM and Framework outline AI safety and security practices. The AI Safety Institute at the National Institute of Standards and Technology becomes the primary U.S. Government point of contact for private sector AI developers. The AI Safety Institute must:

Test at least two frontier AI models within 180 days of the memorandum
Issue guidance for testing AI systems' safety and security risks
Establish benchmarks for assessing AI capabilities
Share findings with the Assistant to the President for National Security Affairs

The Department of Energy's National Nuclear Security Administration will lead classified testing for nuclear and radiological risks, while the National Security Agency will evaluate cyber threats through its AI Security Center.

The framework prohibits government AI use for:

Nuclear weapons deployment without human oversight
Final immigration determinations
Profiling based on constitutionally protected rights
Intelligence analysis based solely on AI without warnings

National security agencies must also:

Appoint Chief AI Officers with

...
Death notes - 7 thoughts on death
by Nathan Young — 2024-10-28T15:01:13.532Z
Link: https://nathanpmyoung.substack.com/p/death-notes

Both my grandmother and my great aunt passed away recently, which has caused me to muse upon death.

Slowly, then all at once

What is death? I guess it's the ending of the process of consciousness. Not that we really know what consciousness is?

In that sense, do I die when I go to sleep? My consciousness in the morning often seems pretty unconnected to the consciousness of the night before. Things that seem so important when I go to bed seem trivial when I wake up. What does the waking Nathan owe the man from the night before? I don't believe in souls, so what connects me to future Nathan?

I was a Christian five years ago. What would that Nathan think about how I use the knowledge, resources and pattern of consciousness he bequeathed to me? Would he consider it a death? The light has been refracted so many times as to be going in a very different direction. Why should I entrust my possessions to future Nathan, rather than giving them away[1]?

I guess that people have the sense that only the ending of the pattern of consciousness matters, rather than its shift over days or weeks. That I am still, legally and fundamentally “Nathan”, regardless of what past Nathan might think about me.

I find this becomes more difficult when someone is slowly dying. My great uncle is losing his memory and his ability to understand speech. He sometimes becomes violent, despite having been a fairly easy going man in his time. Is this not a kind of death - his consciousness slowly being railroaded by unfortunate genes? I have heard accounts (n = 1-3) of dementia patients becoming entirely unlike their previous selves. To see someone you love become someone you don’t care for seems a torture almost deliberately selected for cruelty.

All at once, then slowly

In Robin Hanson’s Age of Em, our minds have been all uploaded and whatever causes them to decay has long since been solved. There is likely no upper bound to how long a mind can exist, so it becomes a question of who can afford to. But rather than poverty leading to death, one can only afford to run on slower and slower servers (as compared to the breakneck pace of the technological elite[2]). Perhaps a group of us retire together, opening the morning paper each morning to discover what the last 3 years of elite history have been like.

I find this both comforting and brutal.

It’s comforting that there is functionally no death -...