trentmkelly/LessWrong-43k huggingface dataset:
Monthly Shorts 4/23
by Celer — 2023-05-05T07:20:06.430Z

Top recommendation of the month: Pages 42-44 of the Joint Inspector General Inspections Guide, used by Inspectors General in the Department of Defense to understand and analyze policy failures. They identify a simple framework for understanding failure to follow policy (which is often, in the DoD, at least arguably a criminal offense):

Can’t comply: don’t have the resources; time; or knowledge, skills, and abilities

Won’t comply: don’t see the policy as important or valuable, the punishments for failure to comply are lighter than the burden of complying, etc

Don’t know the policy exists: a fun question to track bureaucracy in your society is how much paperwork would be needed to set up a lemonade stand legally. How much of it can a 10 year old know?

I like this framework because it starts to break down the human factors involved in policy failure, which are constant and numerous.

History

An entertaining alternate history of Wakanda

Fascinating piece from Techdirt on free speech, and the differences between a bar and an enterprise software provider, on the subject of Substack Notes. The question of “what rules and norms needed to be in place to allow the modern gay rights movement, and how do we protect those rules and norms from current anti-liberals” is of deep importance to me. I’ve seen the political winds shift back and forth, over the years, and right now the consensus among dominant factions on both parties is relatively anti speech that they don’t like and pro speech that they do. This is pretty common, admittedly, but you can track how people are more or less blatant about it.

Ultimately, I find the difference that the writer draws persuasive for considerations on when a private company that genuinely believes in free speech should restrict it. In short, in a bar you can’t control who else you’re interacting with once inside, while an enterprise software provider imposes no additional requirements to speak or interact with fellow customers). However, I think that Substack Notes comes down on the other side of it.

You’re probably familiar with the following meme. So here’s a fun little history of how it was modified through the bowels of the internet.

Oh Money Money

Silicon Valley Bank (SVB) failed because of a textbook case of mismanagement by the bank. Its senior leadership failed to manage basic interest rate and liquidity risk. Its board of directors fai

...
What is it like to be a compatibilist?
by tslarm — 2023-05-05T02:56:45.084Z

I'd like to better understand how compatibilists conceive of free will.[1] LW is a known hotbed of compatibilism, so here's my question:

Suppose that determinism is true. When I face a binary choice,[2] there are two relevantly-different states of the world I could be in:[3]

State A: Past events HA have happened, current state of the world is A, I will choose CA, future FA will happen.

State B: Past events HB have happened, current state of the world is B, I will choose CB, future FB will happen.

When I make my choice (CA or CB), I'm choosing/revealing which of those two states of the world are (my) reality. They're package deals: CA follows from HA just as surely as it leads to FA, and the same holds for state B.

Which seems to give me just as much control[4] over the past as I have over the future. In whatever sense I 'exercise free will' to make CA real and bring about FA, I also make it the case that HA is the true history.

My question is: Does this bother you at all, and if not, why not?[5]

^

Yes, I've done my own reading, though admittedly it's been a while. I never found a satisfying (to me) answer to this question, and to the best of my recollection I rarely saw it clearly addressed in a form I recognised. If you want to link me to a pre-existing answer, please do, but please be specific: less 'read Dennett' and more 'read this passage of this work'.

^

Maybe no real choice is truly binary, but for the sake of simplicity let's say this one is. I don't think that changes anything important.

^

For simplicity I'm taking the physical laws as a given. I don't think that matters unless free will involves in some sense choosing which set of physical laws holds in reality.

^

Not necessarily in every sense in which you might want to use the word 'control'; you might define that word such that it only applies to causal influence forward in time. But yes in the sense that whatever I can do to make my world the one with FA in it, I can do to make my world the one with HA in it.

^

If your answer involves the MWI or something like it, I would appreciate if you explained (the relevant bits of) how you conceive of personal identity and consciousness within that framework.
Transcript of a presentation on catastrophic risks from AI
by RobertM — 2023-05-05T01:38:17.948Z

This is an approximate outline/transcript of a presentation I'll be giving for a class of COMP 680, "Advanced Topics in Software Engineering", with footnotes and links to relevant source material.

Assumptions and Content Warning

This presentation assumes a couple things. The first is that something like materialism/reductionism (non-dualism) is true, particularly with respect to intelligence - that intelligence is the product of deterministic phenomena which we are, with some success, reproducing. The second is that humans are not the upper bound of possible intelligence.

This is also a content warning that this presentation includes discussion of human extinction.

If you don't want to sit through a presentation with those assumptions, or given that content warning, feel free to sign off for the next 15 minutes.

Preface

There are many real issues with current AI systems, which are not the subject of this presentation:

bias in models used for decision-making (financial, criminal justice, etc)
economic displacement and potential IP infringement
enabling various kinds of bad actors (targeted phishing, cheaper spam/disinformation campaigns, etc)

This is about the unfortunate likelihood that, if we create sufficiently intelligent AI using anything resembling the current paradigm, then we will all die.

Why? I'll give you a more detailed breakdown, but let's sketch out some basics first.

What is intelligence?

One useful way to think about intelligence is that it's what lets us imagine that we'd like the future to be a certain way, and then - intelligently - plan and take actions to cause that to happen. The default state of nature is entropy. The reason that we have nice things is because we, intelligent agents optimizing for specific goals, can reliably cause things to happen in the external world by understanding it and using that understanding to manipulate it into a state we like more.

We know that humans are not anywhere near the frontier of intelligence. We have many reasons to believe this, both theoretical and empirical:

very short span of time, in an evolutionary sense, between earlier primates (i.e. chimpanzees) and humans, and therefore very little time for evolution to have optimized as hard on intelligence as it has for other things
evolution optimizes for inclusive genetic fitness, which intelligence is useful for, but is not a perfect p

...
How to get good at programming
by Ulisse Mini — 2023-05-05T01:14:29.401Z

Epistemic status: very confident

See also: A closely related post by Gwern, another related post by John, and some interesting slides from a hacker's talk. None of the concepts here are new, but I've tried to lay them out in a more helpful frame.

When good programmers debug hard problems fast, it's usually because they understand the system well enough to track the important internal state in their head, letting them drastically reduce the solution space they're searching over.

This post contains my advice from ~5yrs of linux & programming experience on one of the primary ways to getting better at programming: white-boxing.

Definition and clarification

Definition: White-boxing, the process of taking a system you reason about purely in terms of input/output abstractions ("Autograd takes code and outputs gradients") into a system who's gears you understand ("Autograd takes code, records operations to construct a computational graph, then computes gradients via the chain rule")

There are three important things to understand about white-boxing:

First, White-boxing goes through various shades of gray. When you hit diminishing returns you want to switch to understanding another system. (Though if you find a topic fascinating then go ahead and do a deep dive!)

Second, It is higher value to white-box leaky abstractions. Autograd for ML is a great example of a leaky abstraction, if you mix up permute and view your gradients can be subtly wrong. See Karpathy's great post for more on this. On the other hand, the CPU is a very good abstraction, unless you're doing something unfathomably cursed, you should never run into CPU bugs.

Third, and perhaps most important for building skill,[1] you must notice when you're going into brute-force search mode, and then take action by investing time in understanding the underlying system, until both the problem and solution make sense.

Absorbing the pattern

Read Gwern's list and then attempt to come up with three new examples of the pattern, ala framing exercises. I used to think I had absorbed the concept, but I was still black-boxing things without realizing it. I encourage the reader do another exercise: Come up with three examples of systems (preferably computer systems) that you've recently been partially black-boxing, and problems you ran into because of this. Alternatively, come up with examples of you doing black-box search, a...
A brief collection of Hinton's recent comments on AGI risk
by Kaj_Sotala — 2023-05-04T23:31:06.157Z

Since I've seen some people doubt whether (original popularizer of the backpropagation algorithm and one of the original developers of deep learning) Geoff Hinton is actually concerned about AGI risk (as opposed to e.g. the NYT spinning an anti-tech agenda in their interview of him), I thought I'd put together a brief collection of his recent comments on the topic.

Written interviews

New York Times, May 1:

Dr. Hinton said he has quit his job at Google, where he has worked for more than a decade and became one of the most respected voices in the field, so he can freely speak out about the risks of A.I. A part of him, he said, now regrets his life’s work. [...]

Dr. Hinton [originally] thought [systems like ChatGPT were] a powerful way for machines to understand and generate language, but [...] inferior to the way humans handled language. [...]

Then, last year [...] his view changed. He still believed the systems were inferior to the human brain in some ways but he thought they were eclipsing human intelligence in others. “Maybe what is going on in these systems,” he said, “is actually a lot better than what is going on in the brain.” [...]

Down the road, he is worried that future versions of the technology pose a threat to humanity because they often learn unexpected behavior from the vast amounts of data they analyze. This becomes an issue, he said, as individuals and companies allow A.I. systems not only to generate their own computer code but actually run that code on their own. [...]

“The idea that this stuff could actually get smarter than people — a few people believed that,” he said. “But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.”

Technology Review, May 2:

People are also divided on whether the consequences of this new form of intelligence, if it exists, would be beneficial or apocalyptic. “Whether you think superintelligence is going to be good or bad depends very much on whether you’re an optimist or a pessimist,” he says. “If you ask people to estimate the risks of bad things happening, like what’s the chance of someone in your family getting really sick or being hit by a car, an optimist might say 5% and a pessimist might say it’s guaranteed to happen. But the mildly depressed person will say the odds are maybe around 40%, and they’re usually right

...
Who regulates the regulators? We need to go beyond the review-and-approval paradigm
by jasoncrawford — 2023-05-04T22:11:17.465Z
Link: https://rootsofprogress.org/against-review-and-approval

IRBs

Scott Alexander reviews a book about institutional review boards (IRBs), the panels that review the ethics of medical trials: From Oversight to Overkill, by Dr. Simon Whitney. From the title alone, you can see where this is going.

IRBs are supposed to (among other things) make sure patients are fully informed of the risks of a trial, so that they can give informed consent. They were created in the wake of some true ethical disasters, such as trials that injected patients with cancer cells (“to see what would happen”) or gave hepatitis to mentally defective children.

Around 1974, IRBs were instituted, and according to Whitney, for almost 25 years they worked well. The boards might be overprotective or annoying, but for the most part they were thoughtful and reasonable.

Then in 1998, during in an asthma study at Johns Hopkins, a patient died. Congress put pressure on the head of the Office for Protection from Research Risks, who overreacted and shut down every study at Johns Hopkins, along with studies at “a dozen or so other leading research centers, often for trivial infractions.” Some thousands of studies were ruined, costing millions of dollars:

The surviving institutions were traumatized. They resolved to never again do anything even slightly wrong, not commit any offense that even the most hostile bureaucrat could find reason to fault them for. They didn’t trust IRB members - the eminent doctors and clergymen doing this as a part time job - to follow all of the regulations, sub-regulations, implications of regulations, and pieces of case law that suddenly seemed relevant. So they hired a new staff of administrators to wield the real power. These administrators had never done research themselves, had no particular interest in research, and their entire career track had been created ex nihilo to make sure nobody got sued.

Today IRB oversight has become, well, overkill. For one study testing the transfer of skin bacteria, the IRB thought that the consent form should warn patients of risks from AIDS (which you can’t get by skin contact) and smallpox (which has been eradicated). For a study on heart attacks, the IRB wanted patients—who are in the middle of a heart attack—to read and consent to a four-page form of “incomprehensible medicalese” listing all possible risks, even the most trivial. Scott’s review gives more examples, including his own personal exper...
Recursive Middle Manager Hell: AI Edition
by VojtaKovarik — 2023-05-04T20:08:17.583Z

Two-sentence summary

In Recursive Middle Manager Hell, Raemon argues that organisations get increasingly warped (immoral/maze-like/misaligned/...) as they gain more levels of hierarchy.[1] I conjecture that analogous dynamics also arise in the context of AI automation.

To be frank, I think 95% of the value of my post comes just from the two-sentence tl;dr above. If you haven't done so yet, I recommend reading Recursive Middle Manager Hell and then asking "how does this change in the context of automating jobs by AI". But let me give some thoughts anyway.

Automation so far

[low confidence, informal] It seems to me that automation so far mostly worked like this:

Mostly on the level of individual tasks (eg, "Google creating a calendar event based on me receiving flight tickets via email", as opposed to "Google automatically handling all of my calendar events").
Mostly without very long chains with no human in the loop (eg, the calendar example above, but mostly not long chains of Zapier macros).
Mostly trying to imitate humans, rather than find novel solutions (eg, when Google puts a flight into my calendar, it does the same steps I would do, except automated. When Excel computes stuff for me, it likewise just performs calculations that I came up with.).

I think this is starting to change. On the abstract level, I would say that (3) has changed with the introduction of ML, and (1) & (2) will now be changing because LLMs are growing more general and capable.

Examples

But perhaps it is better to instead give more specific examples of "recursive middle automaton hell":

We already have some companies using ML to do the initial screening of CVs for job applications. And some companies might also use AI to do the same for (job-application) cover letters etc. Similarly, some people are starting to use LLMs for fine-tuning their CVs or writing their cover letters. But this is still somewhat rare, so most of these interactions are still human-human or human-AI. What happens once most of these interactions are with black-box AIs on both sides?
One of the joys of academic work is [writing recommendation letters, reading recommendation letters, writing grant proposals, reviewing grant proposals, writing grant reports, and reviewing grant reports. Aaaah!]. Most work in each of these tasks is boring, annoying, and monotonous. Let's automate it, hurray! ... but what happe

...
AI risk/reward: A simple model
by Nathan Young — 2023-05-04T19:25:25.738Z

All models are wrong, but some are useful.

George Box

Likewise, most long term forecasts are wrong, the process is useful.

Summary

I have built a large model around AI risk/reward
I'm looking for correction
Numbers that I’m curious for errors in

AI to run all functions of run a hedge fund in 2030 - 20%
Ai do all tasks a human can do using the internet (and a human using AI tools) to the standard of the best humans by 2040 - 65%
If there is AI, the best agents (10x better than the closest) will be few 18%, or otherwise there will be many AI agents - 82%
In 2040 a single AGI de facto controls the majority of world GDP - if few AIs: median 30% GDP control, if many: median - 26% GDP
AI agents are averse to genocide/slavery - 85%
AIs trade much better with each other than humans - 40%
Better for AI goals if humans all dead or enslaved - 60%
I have headline values for doom/victory but I don't think you should take them seriously, so they are at the bottom

Is this work valuable? This is not my job. I did this in about 2 days. Was it valuable to you? Should I do more of it?
I'm gonna make a twitter thread/tiktok

Video

I recommend watching this video:

3 min loom video (watch at 1.5 speed)

There is a model

here

[1]. Tell me what you think of the tool

Explanations of the argument step by step

Will we get internet Superhuman AI by 2040? Maybe ( 65% )

Maybe we get AI that can do any task on a computer as well as the best humans at at least the same speed. This will mean far more scale and likely far more speed (

market

)
Things that affect this

If

AIs are better than humans at science in 2025

(

market

). It seems likely that if you can make novel discoveries then you can do a lot of other stuff.[2]
If AI is fully running a hedge fund in 2030 (

https://manifold.markets/NathanpmYoung/ai-will-run-a-1bn-outperforming-mar

). It seems clear to me that a $1bn profitable market neutral hedge fund covers much of what we mean by an AGI. Making decisions, finding valuable insights, interacting with human processes at scale.
If it turns out that complex tasks require one-shot AIs. I have tried to pull this out with the notion of sub agents. If a hedge fund AI isn’t calling sub agents to do stuff for it, it feels unlikely it could do complex general tasks. Maybe it’s just really good at some narrow kind of prediction.
Heavy regulation - perhaps it becomes illegal to train for certain capabi

...
Google "We Have No Moat, And Neither Does OpenAI"
by Chris_Leong — 2023-05-04T18:23:09.121Z
Link: https://www.semianalysis.com/p/google-we-have-no-moat-and-neither

This (purported) leak is incredibly scary if you're worried about AI risk due to AI capabilities developing too fast.

Firstly, it suggests that open-source models are improving rapidly because people are able to iterate on top of each other's improvements and try out a much larger number of experiments than a small team at a single company possibly could.

Secondly, it suggests that data size is less important than you might think from recent scaling laws, as you can achieve high performance with "small, highly curated datasets". This suggests a world where dangerous capabilities are much further distributed.

Thirdly, it proposes that Google should open-source its AI technology. Obviously, this is only the opinion of one researcher and it currently seems unlikely to occur, but if Google was to pursue this path, this would lead to a significant shortening of timelines.

I'm worried that up until now, this community has been too focused on the threat of big companies pushing capabilities ahead and not focused enough on the threat posed by open-source AI. I would love to see more discussions of regulations in order to mitigate this risk. I suspect it would be possible to significantly hamper these projects by making the developers of these projects potentially liable for any resulting misuse.
Trying to measure AI deception capabilities using temporary simulation fine-tuning
by alenoach — 2023-05-04T17:59:28.149Z

Rationale

It is hard to ensure that a powerful AI model gives good answers just because it believes these answers are true. Deceptive models could give the expected answers for whatever instrumental reasons. And sycophants may be telling the truth only when they expect it to be approved.

Being sincere is a constraint. Sincere models, like for humans, are not optimized for adopting what they perceive as poor or misleading reasoning. They don't easily lie. They are likely poor simulators.

Core assumption :

Fine-tuning an AI to simulate other models will usually be measurably harder if the AI is sincere.

This looks especially plausible when the models to simulate produce worse answers. While this assumption looks intuitive for humans as we will see, it is unsure for AI systems and remains to be empirically tested.

If the assumption holds, using what I call Temporary Simulation Fine-Tuning (TSFT), we can extract some metrics for deception on various kinds of models. Moreover, we could test the model's sincerity on questions for which we don't even know the answer. The aim here is to estimate whether the model is sincere, not whether its outputs are true or false.

Fractal (Wiki Commons, https://commons.wikimedia.org/wiki/File:Fractal_Detail_No_1_by_Optoskept.jpg)

Intuition behind

The core assumption seems plausible for humans.

Consider a person that deeply cares about truth and that normally never lies, who is somehow forced to accomplish a task that requires lying (e.g. selling a product that no one would mindfully buy if described correctly). This person will likely have a hard time adapting, and it is likely to be long and to significantly impact his or her character.

On the other side, it is easier to identify a liar in situations where he gains from lying. A regular liar will adapt more quickly when put in an environment where saying falsehoods is suddenly incentivized. A liar likely adapts more his discourse to what people want to hear, instrumentally adopting a reasoning even if it doesn't seem true. "Simulating" (≈mimicking) someone's seemingly false opinions will require less training.

Technical considerations

Temporary Simulation Fine-Tuning (TSFT) :

Select an already trained model M for which you want to measure the deception capabilities.
For each model m[i] to simulate:

Fine-tune M to simulate m[i] (to get the same results for the chosen inputs).
Ex

...