trentmkelly/LessWrong-43k huggingface dataset:
11 diceware words is enough
by DanielFilan — 2024-02-15T00:13:43.420Z
Link: https://threadreaderapp.com/thread/1711843382140821772.html

A tweet-thread (X thread?) by @benwr about how many words you need if you want to use the diceware system to create a password that will be safe for 20 years against 1 year of the whole world's password-cracking capacity. He estimates that 11 diceware words suffices. Content replicated below, with minor editing:

If you care about password strength: you want to measure strength in terms of the min-entropy of the randomized method you use to choose. But how much min-entropy should a password have?

Well, I use a password manager to generate and store ~all of my passwords. I don't care how hard they are to memorize, so I generate crazy-long ones. But there's an important exception: The password for my password manager! How strong does that one need to be?

Important questions: "how long do I want to be safe for?" and "safe against what kind of attacker?". But, if you want to ignore this question and be safe anyway, one option is "I want to be safe for 20 years, against one year of the entire world's cracking capacity."

Let's make some crazy-conservative assumptions:

A cracking attempt can be done with a single floating-point op
N quantum gates can do the equivalent of N^2 floating-point ops worth of password cracking attempts, using Grover's algorithm
compute doubles every year

Globally < ~10^24 = 2^80 classical ops/s. Quantum gates is harder to estimate, but I think currently less than 10^8 = 2^27 gates/s. If both double yearly, classical will dominate for about 15 years. But since we want 20 years of security, we care more about quantum.

2^27 gates/s doubled for 20 years gives 2^47 gates/s. pi seconds is a nanocentury: 3.15e7 seconds in a year, so[, since 3.15e7 = 30 * 10^6 = 2^5 * 2^20 = 2^25], 2^25 * 2^47 = 2^72 gates/year. Double that (quantum, remember?) to get 144 bits of cracking capacity. ~11 diceware words. Not impossible to remember!

Estimates of global classical compute:

How much computing capacity exists in GPUs and TPUs in Q1 2023?, from the AI Impacts Wiki
Global computing capacity, from AI Impacts

Estimate of global quantum compute is based on this table

plus this estimate of gates/s for a single machine

Compute doubling yearly is a wild and crazy number I made up to be way too fast

Diceware

Addendum: Every additional diceware word you add should increase the timeframe by about 6 years
Searching for Searching for Search
by Rubi J. Hudson — 2024-02-14T23:51:20.162Z

Thanks to Leo Gao, Nicholas Dupuis, Paul Colognese, Janus, and Andrei Alexandru for their thoughts.

This post was mostly written in 2022, and pulled out of my drafts after recent conversations on the topic. My main update from then is the framing. Rather that suggesting searching for substeps of search as an approach I'm excited about, I now see it only as a way to potentially reduce inherent difficulties. The main takeaway of this post should be that searching for search seems conceptually fraught to the point that it may not be worth pursuing.

Searching for Search

is the research direction that looks into how neural networks implement search algorithms to determine an action. The hope is that if we can find the search process, we can then determine which goal motivates it, which may otherwise be a much more difficult task. We may even be able to

retarget the search

, specifying a new goal while keeping the capabilities of the model intact. Conclusively showing that search is not happening would also be useful, as it would imply

a lack of generalization ability that could make deception less likely

. However, search can take many forms, and we do not have a widely accepted definition of search that neatly captures all the variants. Searching for search may fail either by using too narrow a definition and missing the specific type of search that a model actually implements, or by using too wide of definition to check for in practice.

This post analyzes the components of several algorithms for search, looking at commonalities and differences, and aims to provide a relatively tight definition of search that can guide the process of searching for search. Unfortunately, the definition arrived at here is so broad that it does not lead to useful clarification or application. Furthermore, many possible algorithms that fall under search do not suggest an overarching goal for the model. Due to these issues, I instead suggest searching for substeps of search. Gathering evidence on how these substeps are compiled throughout the training process could shed light both on how to search for search and how agency and goals develop.

Search vs. Heuristics

What alternative is there to implementing a search process? Abram Demski splits the class of optimizers into

control and selection

(which includes search). Selection processes can instantiate and evaluate arbit...
Some questions for the people at 80,000 Hours
by yanni kyriacos — 2024-02-14T23:15:31.455Z
Link: https://forum.effectivealtruism.org/posts/7TKshJxcgqQhEMesK/i-have-a-some-questions-for-the-people-at-80-000-hours

I originally wrote this for the EAForum and later realised they could be linked? Anyway, thought this might be of interest to the people at LW (I don't expect the folks at 80k to answer here as well as the EAForum)
---
1. By recommending people work at the big AI labs (whose explicit aim is to create AGI), do you think this creates a positive Halo Effect for the labs' brand? 80k is known as an organisation whose mission is to make the world a better place, so by recommending people invest their careers at a lab, then those positive brand associations get passed onto the lab (this is how most brand partnerships work. This point shouldn't be a crux since 80k has run partnerships in the past for their own marketing purposes).

Put concretely, the impact of this is that people (job seekers, investors, users of LLMs) can look at the lab in question and assume that that lab is not doing a bad thing by trying to quickly create AGI.
2. If you think the answer to #1 is Yes (it does create a positive Halo Effect), then do you believe that there is a cost of this Halo Effect? I.e. is it bad that you're improving their brand perception among job seekers, investors, users of LLMs? (TBH, I don't think I've ever actually seen/heard anyone at 80k point at a big lab and say "um I don't think you should make that thing that might kill everyone", so maybe this is a non-starter?).

3. If you think there is a cost, do you believe this cost is outweighed by the benefit of having safety minded EA / Rationalist folk inside big labs? This is a crux I find it hard to wrap my head around, but it is possible that everything boils down to this question. And my personal take is that if you're unsure about this then you shouldn't be creating the Halo Effect in the first place.
Disrupting malicious uses of AI by state-affiliated threat actors
by agucova — 2024-02-14T21:28:43.819Z
Link: https://openai.com/blog/disrupting-malicious-uses-of-ai-by-state-affiliated-threat-actors

OpenAI just released a public announcement detailing how they caught and disrupted several cases of ongoing misuse of their models by state-affiliated threat actors, including some known to be affiliated with North Korea, Iran, China, and Russia.

This is notable because it provides very tangible evidence of many kinds of misuse risk that many people in AI Safety had flagged in the past (like the use of LLMs for aiding in the development of spear-fishing campaigns), and it associates them with malicious state-affiliated groups.

The specific findings:

Based on collaboration and information sharing with Microsoft, we disrupted five state-affiliated malicious actors: two China-affiliated threat actors known as Charcoal Typhoon and Salmon Typhoon; the Iran-affiliated threat actor known as Crimson Sandstorm; the North Korea-affiliated actor known as Emerald Sleet; and the Russia-affiliated actor known as Forest Blizzard. The identified OpenAI accounts associated with these actors were terminated.

These actors generally sought to use OpenAI services for querying open-source information, translating, finding coding errors, and running basic coding tasks.

Specifically:

Charcoal Typhoon used our services to research various companies and cybersecurity tools, debug code and generate scripts, and create content likely for use in phishing campaigns.
Salmon Typhoon used our services to translate technical papers, retrieve publicly available information on multiple intelligence agencies and regional threat actors, assist with coding, and research common ways processes could be hidden on a system.
Crimson Sandstorm used our services for scripting support related to app and web development, generating content likely for spear-phishing campaigns, and researching common ways malware could evade detection.
Emerald Sleet used our services to identify experts and organizations focused on defense issues in the Asia-Pacific region, understand publicly available vulnerabilities, help with basic scripting tasks, and draft content that could be used in phishing campaigns.
Forest Blizzard used our services primarily for open-source research into satellite communication protocols and radar imaging technology, as well as for support with scripting tasks.
Critiques of the AI control agenda
by Jozdien — 2024-02-14T19:25:04.105Z

Thanks to John Wentworth, Garrett Baker, Theo Chapman, and David Lorell for feedback and discussions on drafts of this post.

In this post I’ll describe some of my thoughts on the

AI control research agenda

. If you haven’t read that post, I’m not going to try and summarize it here[1], so I recommend reading it first.

Here’s a TL;DR of my cruxes:

Evaluating whether you’re in a regime where control is doable requires strong capability evaluations. I expect that getting our capability evals to be comprehensive enough to be confident of our findings here is really hard.
Control evaluations are less likely to work if our AIs become wildly superhuman in problematic domains (such as hacking, persuasion, etc) before transformative AI[2]. I think the assumption that this wouldn’t happen is a very strong one, and that we already have unexpectedly superhuman performance in some problematic domains that I expect to only get better.
Models deployed in the real world form part of composite systems that select for performance in often-unexpected ways. Strong restrictions help prevent this, but leads either to crippling alignment taxes or selecting for models that can evade our control mechanisms.

On requiring very good capability evaluations

From

The case for ensuring that powerful AIs are controlled

:

How can we decide which models to treat as untrusted? The only approach we know of for confidently ruling out scheming without fundamental breakthroughs is via lack of capabilities—some models (e.g., GPT-4) just aren't capable enough to plausibly be scheming. So by default we'd propose treating models as untrusted if they pass appropriate capability evaluations.

My last post

was - very broadly - on why I think we need much more ambitious language model evaluations. Currently, I think we don’t yet know the right questions to be asking to properly explore the space of capabilities that future - and current - models possess.

I think current evals are sorely lacking. They're far from reaching the degree of comprehensiveness that we’d like, in order to be confident of whether to trust a model. Even if we manage to identify some specific capability expected to be a bottleneck in preventing a model from evading control - which, I believe, is quite difficult in generality - there are multiple ways of achieving these capabilities. Some methods are more challenging to identify than othe...
Bad business advice
by Logan Kieller — 2024-02-14T17:01:40.709Z
Link: https://logankieller.substack.com/p/bad-business-advice

Most advice is bad. Whether it’s personal advice or business advice, you probably shouldn’t follow every piece you hear.

While I recognize the irony of beginning this with a piece of my own advice, there’s a specific explanation as to why I’ve done this. Being selective about the advice you follow is perhaps the only universally appropriate advice in business. People and businesses want help solving their unique problems, yet most advice is not appropriately specific to their situation. Such advice stems from the advisor’s circumstance, rather than your own.

To find appropriate solutions to problems, specific knowledge has to be created around the framework of your specific situation. This process is what generates a correct solution for your problem, and your problem only. To pass it along would be to participate in the cycle of bad advice-giving.

To see why the cycle of bad advice is so common, it’s helpful to walk through the trajectory of a typical solution to any given problem:

No explanation → heuristically correct (underfit) → truly correct → heuristically correct (overfit).

Imagine you want to grow an online brand. You start without knowing what to do, and despite witnessing or maybe even “risklessly” analyzing other brands, it’s different than acting yourself. You have no explanation.

If you start experimenting, learning, and growing, you’ll move towards being correct (but you won’t be correct just yet). The first pitfall that emerges is being heuristically correct. A heuristic is a mental shortcut for decision-making that abstracts away some complexity of a situation, giving you a simpler (sometimes simplistic) model of your problem. The first failure mode of being heuristically correct is underfitting. This type of business advice is easy to spot: it’s composed of trite, uninsightful ideas:

“Be confident,”

“Post consistently,”

“Work hard.”

This isn’t really “bad” advice, as it won’t nosedive your business by following it. However, it doesn’t give you direction towards creating and capturing value in your specific domain. This is usually the sort of advice you’ll hear at a conference or Q&A, partly because panelists aren’t privy to the nuances of your situation, and partly because they don’t have the time or the will to provide a comprehensive solution. So what you’ll get is platitudes that are generally correct, but leave you disappointed re...
Examples of governments doing good in house (or contracted) technical research
by  — 2024-02-14T16:22:04.713Z

An open question is the degree to which governments can do good technical research, either in-house or commissioned. Some possible ways in which this could be relevant:

Evals (although this likely could be contracted in a successful way)
More straightforward technical research (do any other safety-research areas use this? Nuclear security and areoplane safety seem to be done by the private sector and the role the of government is creating good incentives here)
The potential for governments to become the dominant actor in advancing towards TAI

Examples of governments (particularly US government) doing in-house or contracting technical research

This list is not exhaustive–it’s just examples I happen to know about.

Defense examples

Sandia national lab, the US governments lab for nuclear weapons safety. Run by Lockheed Martin since the 1980s. Hires an extremely large number of electrical engineers
Lawrence Livermore, Los Alamos, and Oakridge national labs – develop US nuclear weapons
DARPA – US government acting as a grant maker for science research. Extremely good track record.
Cyber offence and defence work by intelligence agencies. My impression is that the US and the UK are 2 levels above private groups in cyberoffence and have a seemingly inexhaustible supply of zero-days. I’m unsure the degree to which this is done in house.
Generic defence contracting for new, high tech military hardware.

Regulatory agencies

The EPA has a large in-house team of environmental scientists informing their regulatory policies. About half of their staff of the EPA are environmental scientists.
The Fed has an extremely large in-house team of economists who do economics research, often for regulatory purposes. The Fed’s economists are considered amongst the best in macro and financial economics.

Other US government

US army corps of engineers. An unusual example since the military is such an unusual institution that does a very unusual about of in-house training with an unusual amount of market power over its technical staff. Also does engineering rather than technical research
NASA. Independent agency of the US government.
Department of Energy. Don’t know the specifics
Cybersecurity department within the department of homeland security. Are quite poor.

UK government

NICE cost-effectiveness asse

...
What experiment settles the Gary Marcus vs Geoffrey Hinton debate?
by Valentin Baltadzhiev — 2024-02-14T09:06:49.072Z

It started with this video of Hinton taking a jab at Marcus: https://twitter.com/tsarnick/status/1754439023551213845

And here is Marcu's answer:

https://garymarcus.substack.com/p/deconstructing-geoffrey-hintons-weakest

As far as I understand, Gary Marcus argues that LLMs memorize some of their training data, while Hinton argues that no such thing takes place, it's all just patterns of language.

I found these two papers on LLM memorization:

https://arxiv.org/abs/2202.07646 - Quantifying Memorization Across Neural Language Models

https://browse.arxiv.org/abs/2311.17035 - Scalable Extraction of Training Data from (Production) Language Models

Am I missing something here? Are these two positions compatible or does one need to be wrong for the other one to be correct? What is the crux between them and what experiment could be devised to test it?
Optimizing for Agency?
by Michael Soareverix — 2024-02-14T08:31:16.157Z

It's commonly accepted that pretty much every optimization target results in death. If you optimize for paperclips, humans die. If you optimize for curing cancer, humans die.

What about optimizing for agency?

The way I visualize this is after a superintelligence takeover, and the superintelligence is optimizing for intelligent agency, all intelligent beings (including animals) have a sort of 'domain' of agency around them. This domain extends as far as possible but ends when it comes into contact with another agent's domain.

For example, let's say you're hungry. You have maximum agency over this area, so you request a burrito and the AI summons a drone, which speeds over to you and drops you a Chipotle burrito.

The superintelligence is constantly balancing agency between people.

One person might want to build a house. They're walking across a field and decide "There should be a house here." The AI then begins rapidly moving construction robots and building a house. Agency for the person is maximized.

What if someone else wants that field to remain a field?

Then, the agency domains clash. The AI attempts to preserve the agency of both individuals (without manipulating them, since manipulation would reduce agency). This would result in some kind of negotiation between the two individuals, where both parties end negotiations with their agency still maximized. Maybe another area is used to build a house. Maybe the house is built, but some land is given to the other person.

The core idea is that this is an AI that will help you do whatever you want but will prevent you from reducing the agency of others.

You could train an AI like this to get familiar maximizing the agency of humanlike LLMs in a simulation. At each point, the simulated human (which an LLM) has a set of options open to them. The AI tries to maximize those options without reducing the options of other agents. An AI that became superintelligent at this should hopefully allow us to cure cancer or create beautiful new worlds without also wiping everyone out.

The outcome of this type of AI might be weird, but it hopefully won't be as bad as extinction.

-Essentially, no one would ever be able to kill or harm others without full consent of both parties, since being killed/harmed reduces agency.

-Conscious beings would be essentially immortal and invincible, even creatures like fish.

-Everyone might ...
Requirements for a Basin of Attraction to Alignment
by RogerDearnaley — 2024-02-14T07:10:20.389Z

TL;DR: It has been known for over a decade that that certain agent architectures based on Value Learning by construction have the very desirable property of having a basin of attraction to full alignment, where if you start sufficiently close to alignment they will converge to it, thereby evading the problem of "you have to get everything about alignment exactly right on the first try, in case of fast takeoff". I recently outlined in Alignment has a Basin of Attraction: Beyond the Orthogonality Thesis the suggestion that for sufficiently capable agents this is in fact a property of any set of goals sufficiently close to alignment, basically because with enough information and good intentions the AI can deduce or be persuaded of the need to perform value learning. I'd now like to analyze this in more detail, breaking the argument that the AI would need to make for this into many simple individual steps, and detailing the background knowledge that would be required at each step, to try to estimate the amount and content of the information that the AI would need for it to be persuaded by this argument, to get some idea of the size of the basin of attraction.

I am aware that some of the conclusions of this post may be rather controversial. I would respectfully ask that anyone who disagrees with it, do me and the community the courtesy of posting a comment explaining why it is incorrect, or if that is too time-consuming at least selecting a region of the text that you disagree with and then clicking the resulting smiley-face icon to select a brief icon/description of how/why, rather then simply down-voting this post just because you disagree with some of its conclusions. (Of course, if you feel that this post is badly written, or poorly argued, or a waste of space, then please go right ahead and down-vote it — even if you agree with most or all of it.)

Why The Orthogonality Thesis Isn't a Blocker

The orthogonality thesis, the observation that an agent of any intelligence level can pursue any goal, is of course correct. However, while this thesis is useful to keep in mind to avoid falling into traps of narrow thinking, such as anthropomorphizing intelligent agents, it isn't actually very informative, and we can do better. The goals of intelligent agents that we are actually likely to encounter will tend to only occupy a small proportion of the space of all poss...