trentmkelly/LessWrong-43k huggingface dataset:
Workshop Report: Why current benchmarks approaches are not sufficient for safety?
by Tom DAVID — 2024-11-26T17:20:47.453Z

I’m sharing the report from the workshop held during the AI, Data, Robotics Forum in Eindhoven, a European event bringing together policymakers, industry representatives, and academics to discuss the challenges and opportunities in AI, data, and robotics. This report provides a snapshot of the current state of discussions on benchmarking within these spheres.

Speakers: Peter Mattson, Pierre Peigné and Tom David

Observations

Safety and robustness are essential for AI systems to transition from innovative concepts and research to reliable products and services that deliver real value. Without these qualities, the potential benefits of AI may be overshadowed by failures and safety concerns, hindering adoption and trust in the technology.
AI research and development have transitioned from traditional engineering methodologies, which rely on explicitly defined rules, to data-driven approaches. This shift highlights the need to leverage extensive datasets and computational power to train models, underscoring the complexity of developing systems that operate effectively without predefined logic.
The opaque nature of deep learning models, often described as "black boxes," presents significant challenges in understanding these models. This necessitates rigorous research into interpretability and transparency, ensuring that stakeholders can trust AI systems, particularly in critical applications where safety and reliability are paramount.
Current benchmarking practices face significant challenges, such as the tendency for models to memorize benchmark data. This memorization can lead to misaligned metrics that do not accurately reflect a model's real-world capabilities. Additionally, the sensitivity of benchmarks to prompt variations introduces inconsistencies in evaluation, undermining the reliability of results and making it difficult to assess model capabilities across different scenarios.
From a safety perspective, existing benchmarks may inadvertently exploit vulnerabilities within AI models in a biased manner. This bias can lead to skewed assessments that fail to address critical safety concerns, resulting in AI systems that may perform well under benchmark conditions but exhibit unsafe behaviors in real-world applications.

Recommendations

To facilitate the transition towards the development of useful and safe products and services that effectively benefit soci

...
You are not too "irrational" to know your preferences.
by DaystarEld — 2024-11-26T15:01:42.996Z

Epistemic Status: 13 years working as a therapist for a wide variety of populations, 5 of them working with rationalists and EA clients. 7 years teaching and directing at over 20 rationality camps and workshops. This is an extremely short and colloquially written form of points that could be expanded on to fill a book, and there is plenty of nuance to practically everything here, but I am extremely confident of the core points in this frame, and have used it to help many people break out of or avoid manipulative practices.

TL;DR: Your wants and preferences are not invalidated by smarter or more “rational” people’s preferences. What feels good or bad to someone is not a monocausal result of how smart or stupid they are.

Alternative titles to this post are "Two people are enough to form a cult" and "Red flags if dating rationalists," but this stuff extends beyond romance and far beyond LW-Rationalism.

I saw forms of it as a college student among various intellectual subcultures. I saw forms of it growing up around non-intellectuals who still ascribed clear positives and negatives to the words "smart" and "stupid." I saw forms of it as a therapist working with people from a variety of nationalities. And of course, my various roles in the rationalist and EA communities have exposed me to a number of people who have been subject to some form of it from friends, romantic partners, or family... hell, most of the time I've heard it coming from someone's parents.

What I'm here to argue against is, put simply, the notion that what feels good or bad to someone is a monocausal result of how smart or stupid they are. There are a lot of false beliefs downstream of that notion, but the main one I'm focusing on here is the idea that your wants or preferences might be invalid because someone "more rational" than you said so.

Because while I've taught extensively about how to defend against "dark arts" emotional manipulation in a variety of flavors, I especially dislike seeing "rationality" used as an authoritative word to shame others into self-coercive narratives.

Rationality, as I use the word, refers to an epistemology that minimizes false beliefs and a course of action that maximally fulfills one’s goals.

If someone else tells you that something you’re doing or thinking is irrational, they need to first demonstrate that they understand your goals, and second dem...
AI & Liability Ideathon
by Kabir Kumar — 2024-11-26T13:54:01.820Z
Link: https://lu.ma/sjd7r89v

Overview

Join us for the AI & Liability Ideathon, a two-week event on December 7, 2024, at 3:00 PM BST.

https://lu.ma/sjd7r89v
Join lawyers, researchers and developers to create solutions for AI Liability. Propose, develop and refine ideas with a team, ending in a presentation evening where you can share the final version of your proposal.

All the final proposals will be published on AI-Plans, with the top 3 being selected by peer review after the presentation evening.

The presentation evening is open to everyone, including those who didn't take part in the Ideathon.

The Ideathon, including the Presentation Evening, the Speakers and the Kick Off Call will be primarily taking place in the AI-Plans Discord: https://discord.gg/X2bsw8FG3f

What is an Ideathon?

An Ideathon is a brainstorming event designed to allow individuals to combine collective multidisciplinary knowledge, experience, and creativity to tackle specific topics.. Participation is open to all interested individuals, including students, academics, civil society, non-profit organizations, lawyers, law professors, AI/ML engineers, developers, and product leaders. All are welcome, including those interested in AI safety and liability issues.

For this AI Liability Ideathon, team proposals may be technology-based, policy-based, a combination of both, or otherwise related to the topic.

Examples of Potential Ideas:

Autonomous Legal Continuum: Develop a framework for determining liability for different types of systems, with the level of autonomy as a critical component. For less autonomous systems, greater human liability may be appropriate, while more autonomous systems might have liability regimes similar to those of corporations.
Legal Entities: Explore the concept of granting AI systems legal personhood, similar to the current status of corporations, to clarify liability issues.
Use Cross-Coders to Identify Duty of Care: Employ cross-coders to identify differences between base models and fine-tuned models, potentially reducing liability if it can be demonstrated that a duty of care has been met.
AI Agents as Subcontractors: Identify relevant pre-existing laws and regulations concerning business agents and examine the implications of treating an AI as a business agent with the authority to act on behalf of a corporation, employee, or subcontractor.
Map AI Dev Pipeli

...
Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?
by Bogdan Ionut Cirstea — 2024-11-26T09:58:44.025Z
Link: https://arxiv.org/abs/2411.16679

Authors: Sohee Yang, Nora Kassner, Elena Gribovskaya, Sebastian Riedel, Mor Geva.

Abstract:

We evaluate how well Large Language Models (LLMs) latently recall and compose facts to answer multi-hop queries like "In the year Scarlett Johansson was born, the Summer Olympics were hosted in the country of". One major challenge in evaluating this ability is that LLMs may have developed shortcuts by encounters of the head entity "Scarlett Johansson" and the answer entity "United States" in the same training sequences or merely guess the answer based on frequency-based priors. To prevent shortcuts, we exclude test queries where the head and answer entities co-appear in pretraining corpora. Through careful selection of relations and facts and systematic removal of cases where models might guess answers or exploit partial matches, we construct an evaluation dataset SOCRATES (ShOrtCut-fRee lATent rEaSoning). We observe that LLMs demonstrate promising latent multi-hop reasoning abilities without exploiting shortcuts, but only for certain types of queries. For queries requiring latent recall of countries as the intermediate answer, the best models achieve 80% latent composability, but this drops to just 5% for the recall of years. Comparisons with Chain-of-Thought composability highlight a significant gap between the ability of models to reason latently versus explicitly. Analysis reveals that latent representations of the intermediate answer are constructed more often in queries with higher latent composability, and shows the emergence of latent multi-hop reasoning during pretraining.

I've only skimmed for now, but seems relevant to Chain-of-Thought alignment and out of context reasoning (OOCR) as a threat model.
Should you increase AI alignment funding, or increase AI regulation?
by Knight Lee — 2024-11-26T09:17:01.809Z

I recently wrote A better “Statement on AI Risk?” an open letter I hope AI experts can sign. One commenter objected, saying that stopping the development of apocalyptic AI is a better focus than asking for AI alignment funding.

My boring answer was that I think there is little conflict between these goals, and the community can afford to focus on both.

This answer is boring, and won't convince everyone since maybe people think AI regulation/pausing is so much more important, that focus on AI alignment funding distracts away from it, and is therefore counterproductive.

The Question

So how should we weigh the relative importances of AI alignment funding and AI regulation/pausing?

For humanity to survive, we either need to survive ASI by making it aligned/controlled, or avoid building ASI forever (millions of years).

Surviving ASI

To make ASI aligned/controlled, we either need to be lucky, or we need to get alignment/control right before we build ASI. In order to get alignment/control right, we need many trained experts working on alignment times a long enough time working on alignment.

Which is more important? In terms of raw numbers, I believe that a longer time is more important than the number of trained experts:

No matter how great the talent or efforts, some things just take time. You can't produce a baby in one month by getting nine women pregnant.

Warren Buffett

Alignment work is a bit more forgiving than having babies, and more people might work faster. There is an innovative process to it, and sometimes twice the number of innovative people are twice as likely to stumble across a new idea. My very rough estimate is this:

A Spherical Cow Approximation

If we have twice as much time, we can make twice as much progress (by our definition of progress).
If we have twice as many trained experts working on alignment, we can make √2 times as much progress.

The total alignment progress A can be very roughly approximated as

A=√N∫T0fT(t)dt

where T is the duration, N are the trained experts working on alignment, and fT(t) is how productive alignment work is, given the level of AI capabilities at time t.

If you don't like integrals we can further approximate it as A=T√N

Regulation

Regulating and pausing AI increases T, and will also increase N because new people working on alignment can become trained experts. ...
Mitigating Geomagnetic Storm and EMP Risks to the Electrical Grid (Shallow Dive)
by Davidmanheim — 2024-11-26T08:00:04.810Z

Executive Summary

This initial investigation begins to examine strategies to mitigate and respond to risks posed by high-impact geomagnetic events, which can severely damage electrical infrastructure. This is split into four sections. The first set are ordered from least promising to most promising targets for investment; recovery, adaptation, and withstanding events. Finally, risk reduction is noted as a highly speculative but high value-of-information area for research.

Recovery approaches involve replacing broken equipment and infrastructure after an event, and focus on the logistical and financial challenges of replacing key infrastructure, notably the costly and rare Ultra-High Voltage (UHV) transformers. Adaptation strategies, including the use of Ground-Induced Current (GIC) blocking devices, are identified as viable and potentially partially adopted means for system operators to prevent damage and reduce restoration costs. Next, withstanding involves protocols for what power systems can do during geomagnetic events. This emphasizes proactive grid shutdowns and sectional isolation, which work by leveraging the short warning period before a CME reaches Earth, or plausibly in advance of nuclear war. These are very promising, but should be pursued by governments vand industry. Finally, risk reduction options are currently limited, but potential exists for longer term, highly speculative approaches for minimizing geomagnetic vulnerabilities at the global level.

Specific high-impact or very impactful mitigations are not explored, but because private companies have some incentive to address the risk, policy approaches involving insurance and regulation are noted. Further work could address this, and will be outlined in the conclusion.

Background

The electrical grid is critical infrastructure, and if electrical systems were destroyed at a national or global level, it could plausibly be or lead to a global catastrophe, especially given the fragility and interconnectedness of other systems. A brief (8 minute) video overview from

Kurzgesagt

from 2020 explains the risk of solar storms. Given that the lack of sufficient backup transformers was

recently highlighted by J.D. Vance on Joe Rogan’s podcast

, I wanted to double check my current understanding of the risk and mitigations available. Rather than focusing on risk estimation, which has been done before, I...
Filled Cupcakes
by jefftk — 2024-11-26T03:20:08.504Z

Cake is tasty, pre-divided cake is convenient, and food is often
better with a combination of textures and flavors. And so we have
frosted cupcakes:

(source: generated by Gemini)

While these look good, they're bad food. The problem is they're very
hard to eat while getting a combination of cake and frosting in each
bite. Vertical slice bites are impractical for people with
non-pythonic jaws, and it's hard to alternate without undermining the
frosting. Also the frosting needs to be quite firm to stand up to
this stacking, limiting texture options.

You can make a jaw-compatible cupcake by restraining your frosting
application to a thin layer, but this is stoichiometrically unsound.

Instead of putting goop on top of the cupcake, it should go in the
middle. Here's one I made, piping plum into the center of a chocolate
cupcake:

(source: messy reality)

Each bite is balanced, you have full artistic freedom when it comes to
texture, and no one gets trismus. You can even put a (thin!) layer of
frosting on top if you're using a subtle flavor, the filling injection
wounds aren't appetizing, or just two components wouldn't sufficiently
impress your guests.

Food should be optimized for eating.

Comment via: facebook, mastodon, bluesky
notes on prioritizing tasks & cognition-threads
by Emrik — 2024-11-26T00:28:03.400Z

...tfw[1] a top promising-looking plan wic wud hv been post-empted by future events if we had pursued it, is pre-empted ind, bc we carefwly plan it out in advance and find the inevitable crux bfr we even begin.

phew! dodged a bullet I might've otherwise had to bite. 😌

the frequency of events like this is one of the main ways we intuitively track & sanity-check our progress wrt our prioritization-framework. it's rly hard to verify if our chosen course is any *good*, but it's much easier to verify that the plan we wud hv prioritized (if not for framework) is any *terrible*.

if we increasingly cut branches we were otherwise tempted to pursue, and that isn't explained by just having sampled more plans or from a different distribution, then it has to be the result of us getting better at finding their flaws (or better alternatives). either way, even if it's just the sampling-more one, it's a fairly reliable metric for concentrating possible selves into better strategic branches.

succinctly:

track u's speed to victory bon the rate of branches u cut (feasible-to-verify via inside-model consistency), not bon the weight of considerations ifo the one u climb (infeasible-to-verify bc outside-model uncertainty).

if I compare myself_now vs myself_4-months-ago, I subjectively notice finding solid reasons to cut tasks (or cognition-threads) fm my tasklist (or cognition-stack) much faster per cut, and w greater frequency of cuts. I now sorta know the questions to ask, and the intangible haze of "uhh mby it seems good to do? who knows, I shud keep it in there jic" is more quickly dispelled.

incomplete list of especially branch-cutty checks in my prioritization-checklist[2]

stack-trace the causality that led u to consider the task itfp

especially check the motivations that led u to want consider it; j bc those is especially fast to check.
but also check the events that caused u's info abt the task itfp.
why is this important? smth smth the bottom-line. u's decisions is only ever as wise as the processes that aly caused u to decide those things as opposed to all other possible things u cud decide. most of u's options is pre-filtered bfr u can filter it ato u's own criteria.

do limiting-case analysis (LCA): assume the best-case scenario for the task-result, then compare its expected Δutility coto the task's counterfactua

...
Why are there no interesting (1D, 2-state) quantum cellular automata?
by Optimization Process — 2024-11-26T00:11:37.833Z

You know elementary cellular automata, where each of the boolean-valued cells evolves according to

x(k)t+1=f(x(k−1)t,x(k)t,x(k+1)t)

where f:{0,1}3→{0,1}.

I think the natural quantum-mechanical extension of this is:

there are 2(N := tape size) basis states: |00⋯00⟩ through |11⋯11⟩
its time-evolution is given, of course, by a unitary operator U, which, expressed in that basis, is:

⟨y|U|x⟩=∏kf(x(k−1),x(k),x(k+1),y(k))

...where f:{0,1}4→C.

You can take any elementary cellular automaton and quantum-ize it: just choose fquantum(a,b,c,z)=( if fclassical(a,b,c)=z then 1 else 0 ); then that product is 1 exactly when y is the classical evolution of x. (Not every fclassical gives rise to a unitary U, though; only the reversible ones.)

But... are there other unitary operators of this form, which aren't basically equivalent to reversible classical CAs? I think not, disappointingly, but I'm not sure, and I don't understand why not.

Bounty: $100 if you make me feel like I have a significantly deeper understanding of why all quantum elementary CAs are basically equivalent to classical elementary CAs (or show me I'm wrong and there actually is interesting behavior here). Partial payouts for partial successes.

My current understanding (the thing you have to enhance or beat) is:

Any choice of f is equivalent to a choice of eight complex two-vectors →λ000,⋯,→λ111, each describing roughly "how (0/1)ish the next state of a cell should be given its current neighborhood."
For unitarity, we want ⟨Ux|Uy⟩=⟨x|y⟩ for all x,y. If you bang through some math, I think this inner product turns out to equal the product of all 64 possible inner products of the →λabc s, raised to various powers:

⟨Ux|Uy⟩=(→λ000⋅→λ111)N000,111⋯(→λ000⋅→λ111)N000,111

...where N000,111 is the number of locations where the neighborhood on tape x is 000 and the neighborhood on tape y is 111. For x=y, we want this product to be 1; for x≠y, we want this product to be 0, meaning at least one of the inner products with a nonzero N must be zero.

(Sanity check: if x=y, then Nabc,abc=1 and all the other Ns are 0. The inner products raised to power 0 disappear; the remaining ones are →λabc⋅→λabc, which is 1 as long as the&n

...
Counting AGIs
by cash — 2024-11-26T00:06:17.845Z

“The resources used to train the model can be repurposed to run millions of instances of it (this matches projected cluster sizes by ~2027), and the model can absorb information and generate actions at roughly 10x-100x human speed. … We could summarize this as a ‘country of geniuses in a datacenter’.”

Dario Amodei, CEO of Anthropic, Machines of Loving Grace

“Let’s say each copy of GPT-4 is producing 10 words per second. It turns out they would be able to run something like 300,000 copies of GPT-4 in parallel. And by the time they are training GPT-5 it will be a more extreme situation where just using the computer chips they used to train GPT-5, using them to kind of run copies of GPT-5 in parallel, you know, again, each producing 10 words per second, they’d be able to run 3 million copies of GPT-5 in parallel. And for GPT-6, it’ll just increase again, there’ll be another factor of 10 at play, and so it’ll be 30 million copies in parallel.”

Tom Davidson, researcher at OpenPhil, Future of Life Institute interview

“Once we get to AGI, we won’t just have one AGI. … given inference GPU fleets by then, we’ll likely be able to run many millions of them (perhaps 100 million human-equivalents, and soon after at 10x+ human speed).”

Leopold Aschenbrenner, Situational Awareness pg. 47

Table of Contents

Section I - The Question

Section II - The Scenario

Section III - Existing Estimates

Section IV - Compute

Section VI - Inference

Section V - Human Equivalents

Section VII - The Estimates

Method 1: Total training to inference per token ratio

Method 2: Flat inference costs

Method 3: Human brain equivalent

Method 4: Chip capabilities

Method 5: Adjusting for capabilities per token

Section VIII - Implications

Acknowledgements

Section I - The Question

What will the initial AGI population be?

Artificial intelligence (AI) systems have become significantly more capable and general in the last decade, especially since the launch of ChatGPT in December 2022. Many people believe that the technological trajectory of AI will lead to the advent of artificial general intelligence (AGI), an AI system that can autonomously do virtually anything a human professional can do. Leading AI scientists, like Geoffrey Hinton, Yoshua Bengio, and Shane Legg, are publicly raising the alarm that such a system is incoming. There are several AI enterprises premised on the business model of creating AGI (Anthropic, OpenAI, ...