trentmkelly/LessWrong-43k huggingface dataset:
An upcoming US Supreme Court case  may impede AI governance efforts
by NickGabs — 2023-07-16T23:51:26.073Z

According to various sources, the US Supreme Court is poised to rule on and potentially overturn the principle of "Chevron deference." Chevron deference is a key legal principle by which the entire federal bureaucracy functions, being perhaps the most cited case in American administrative law. Basically, it says that when Congress establishes a federal agency and there is ambiguity in the statutes determining the scope of the agency's powers and goals, courts will defer to the agency's interpretation of that scope as long as it is reasonable. While the original ruling seems to have merely officially codified the previously implicit rules regarding the legal authority of federal agencies, this practice seems likely to have increased the power and authority of the agencies because it has enabled them to act without much congressional oversight and because they tend to interpret their powers and goals rather broadly. I am not a legal expert, but it seems to me that without something like Chevron deference, the federal bureaucracy basically could not function in its contemporary form. Without it, Congress would have to establish agencies with much more well-specified goals and powers, which seems very difficult given the technocratic complexity of many regulations and the fact that politicians often have limited understanding of these details.

Given that the ruling has expanded the regulatory capacity of the state, it seems to be opposed by many conservative judges. Moreover, the Supreme Court is currently dominated by a conservative majority, as reflected by the recent affirmative action and abortion decisions. The market on Manifold Markets is trading at 62% that they will do so, and while only two people have traded on it, it altogether seems pretty plausible that the ruling will be somehow overturned.

While overturning Chevron deference seems likely to have positive effects for many industries which I think are largely overregulated, it seems like it could be quite bad for AI governance. Assuming that the regulation of AI systems is conducted by members of a federal agency (either a pre-existing one or a new one designed for AI as several politicians have suggested), I expect that the bureaucrats and experts who staff the agency will need a fair amount of autonomy to do their job effectively. This is because the questions relevant AI regulation (i. e. which ev...
Weak Evidence is Common
by dkl9 — 2023-07-16T23:37:40.983Z
Link: https://dkl9.net/essays/weak_evidence.html

Epistemic status: parody well-intentioned with known mistakes (see comments), more confident about clarification at the end

Followup to: Strong Evidence is Common

One time, someone asked me if I was evil. I said "no". Afterward, they probably didn't believe I wasn't evil. I'm guessing they wouldn't have happily accepted a bet at 20:1 odds that I don't deliberately hurt people.

The prior odds that someone is evil are realistically 1:10. Posterior odds above 1:20 implies that the odds ratio of me saying "no" is less than 2:1, or roughly 1 bit of evidence. That's a little bit of evidence.

Seeing an Onion headline say "X did Y" is teeny evidence that X didn't Y. Someone telling you "I am not gay" is small evidence they they are not gay. Putting a complicated, unclear technical claim into ChatGPT and getting "correct" is little evidence that the claim is correct. Tiny odds ratios lurk behind many encounters.

On average, people are overconfident, but 12% aren't. It takes 8:1 evidence to conclude someone isn't overconfident. Asking if they're overconfident and getting "yes" won't be enough, and might be negative.

Running through Bayes' Rule explicitly might produce a bias towards significant values. Extraordinary claims require extraordinary evidence, and a lot of the evidence you get is quite ordinary.

Clarification

This is not to contradict the original, but to remind you not to take it too far.

Strong evidence is common (but certainly not guaranteed) for questions with mostly-honest sources and tiny priors, such as

someone's name
a capital city
whether someone has an uncommon skill
the exact value of a mathematical expression

In many of those cases, the priors are tiny sith there are thousands or millions of plausible options you'd consider before the evidence.

Weak evidence is common when the source of evidence is likely to lie or be stupid, and that situation seems to be especially common with binary questions.
Even briefer summary of ai-plans.com
by Iknownothing — 2023-07-16T23:25:44.076Z
Link: https://www.ai-plans.com

Currently, newcomers to the field of AI Alignment often struggle to understand the ongoing work and individuals involved, as well as the assumptions, strengths, and weaknesses of each plan.

We believe AI-plans.com will be an easy, centralized way to discover and learn more about the most promising alignment plans.

The site is currently in Stage 1, functioning purely as a compendium. We are in the process of adding up to 1000 plans and the criticisms made against them so far. Further plans and criticisms can be added by users.

Projected benefits of Stage 1:

- Easy discovery of proposed plans and better understanding of their prevalent challenges.

(This is already showing promise, with one researcher letting us know they found useful papers on the site and multiple researchers interested- including Jonathan Ng who has been helping us.)

Next, in Stage 2, we will introduce a scoring system for criticisms and a ranking system for plans. Plans will be ranked based on the cumulative scores of their criticisms. Criticism votes will be weighted, giving more influence to users who have submitted higher-scoring criticisms. Alignment researchers will have the option to link their AI-Plans account to accounts on research-relevant platforms (such as arXiv, OpenReview or the AI Alignment Forum) in order to start out with a slightly weighted vote (with mod approval).

Each new plan will start with 0 bounty, and lower bounty plans will give the most points. That way, each new plan will have a lot of opportunity and incentive for criticism. More details

here

.

Projected benefits of Stage 2:

- Incentivizes users to write high-quality criticisms.

- Facilitates identification of plans with significant weaknesses, supporting arguments against problematic plans.

- Allows newcomers to the field(including talented and untapped scientists and engineers) to see which companies have the least problematic plans.
After all, who would want to work for the lowest-ranked company on the leaderboard?

(I have spoken with the creator of aisafety.careers, who intends to integrate with our site.)

At Stage 3, in addition to everything from Stage 1 and 2, we plan to introduce monthly cash prizes for the highest ranking plan and for the users with the most criticism points that month.

Projected benefits of Stage 3:

- Supercharges the impact of Stage 2,...
Mech Interp Puzzle 1: Suspiciously Similar Embeddings in GPT-Neo
by Neel Nanda — 2023-07-16T22:02:15.410Z

I made a series of mech interp puzzles for my MATS scholars, which seemed well received, so I thought I'd share them more widely! I'll be posting a sequence of puzzles, approx one a week - these are real questions about models which I think are interesting, and where thinking about them should teach you some real principle about how to think about mech interp. Here's a short one to start:

Mech Interp Puzzle 1:
This is a histogram of the pairwise cosine similarity of the embedding of tokens in GPT-Neo (125M language model). Note that the mean is very high! (>0.9) Is this surprising? Why does this happen?

Bonus question: Here's the same histogram for GPT-2 Small, with a mean closer to 0.3. Is this surprising? What, if anything, can you infer from the fact that they differ?

Code:

!pip install transformer_lens plotly
from transformer_lens import HookedTransformer
import plotly.express as px
import torch
model = HookedTransformer.from_pretrained("gpt-neo-small")
subsample = torch.randperm(model.cfg.d_vocab)[:5000].to(model.cfg.device)
W_E = model.W_E[subsample] # Take a random subset of 5,000 for memory reasons
W_E_normed = W_E / W_E.norm(dim=-1, keepdim=True) # [d_vocab, d_model]
cosine_sims = W_E_normed @ W_E_normed.T # [d_vocab, d_vocab]
px.histogram(cosine_sims.flatten().detach().cpu().numpy(), title="Pairwise cosine sims of embedding")

Answer: (decode with rot13)
Gur zrna irpgbe bs TCG-Arb vf whfg ernyyl ovt - gur zbqny pbfvar fvz jvgu nal gbxra rzorq naq gur zrna vf nobhg 95% (frr orybj).
Gur pbaprcghny yrffba oruvaq guvf vf znxr fher lbhe mreb cbvag vf zrnavatshy. Zrgevpf yvxr pbfvar fvz naq qbg cebqhpgf vaureragyl cevivyrtr gur mreb cbvag bs lbhe qngn. Lbh jnag gb or pnershy gung vg'f zrnavatshy, naq gur mreb rzorqqvat inyhr vf abg vaureragyl zrnavatshy. (Mreb noyngvbaf ner nyfb bsgra abg cevapvcyrq!) V trarenyyl zrna prager zl qngn - guvf vf abg n havirefny ehyr, ohg gur uvtu-yriry cbvag vf gb or pnershy naq gubhtugshy nobhg jurer lbhe bevtva vf!
V qba'g unir n terng fgbel sbe jul, be jul bgure zbqryf ner fb qvssrerag, V onfvpnyyl whfg guvax bs vg nf n ovnf grez gung yvxryl freirf fbzr checbfr sbe pbagebyyvat gur YnlreAbez fpnyr. Vg'f onfvpnyyl n serr inevnoyr bgurejvfr, fvapr zbqryf pna nyfb serryl yrnea ovnfrf sbe rnpu ynlre ernqvat sebz gur erfvqhny fgernz (naq rnpu ynlre qverpgyl nqqf n ovnf gb gur erfvqhny fgernz naljnl). Abgnoyl, Arb hfrf nofbyhgr...
A Technology of Everything – Part 1: A Magical Science Experiment
by aiuisensei — 2023-07-16T22:01:18.643Z
Link: https://www.aiui.cloud/2023/05/28/magical-science-experiment/

The Pledge

Any idea that can be conceived can be realized. Every conceivable notion has potential for existence. Arthur C. Clarke, in his 1962 piece, 'Profiles of the Future,' presented three principles. Despite the third being the most well-known, all three warrant acknowledgements. These "laws" communicate Clarke's viewpoints on technology and future forecasting:

1. "When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong." The idea behind this law is that established scientists can sometimes be too tied to existing paradigms to see the potential of new technologies or theories.

2. "The only way of discovering the limits of the possible is to venture a little way past them into the impossible." This law encourages pushing the boundaries of what is known and understood, suggesting that progress often comes from daring to explore beyond established limits.

3. "Any sufficiently advanced technology is indistinguishable from magic." This law suggests that advanced technology, beyond the observer's understanding, will appear as magic because the observer doesn't understand the principles behind its operation.

The laws are profound, they include three components that will be crucial to understand in how we come from Clarkes third law to this one:

The engineers first conjecture: What can be realized will be.

Clarke's laws offer a shrewd interpretation of how language can create barriers instead of opportunities. These concepts can be summarized as follows:

1.Always trust observers asserting possibilities over those declaring impossibilities.

2.The realm of possibilities naturally expands as observers navigate it.

3.Observers who do not comprehend something should always append a 'yet'.

Now, let's envision a universe where natural laws coexist with those that appear supernatural. Along this journey, I will demonstrate that Fantasy and Science Fiction—two literary genres often considered distinct—could be viewed as derivatives of the same origin.

Ultimately, I will propose how artificial intelligence stands on the verge of reintroducing genuine magic into our 21st-century world.

First let’s hear what our overlady (ChatGPT) has to say about magic:

Magic, in its broadest sense, refers to the use of supernatural forces or entities to affect natural events. ...
AI, Consciousness, and the problem of Moral Considerability
by stultus — 2023-07-16T19:56:45.200Z

The problem I intend to work through in this short (hopefully) post is that of moral considerability in regard to artificial intelligence systems. This is not a problem that we should wait for advanced AI to consider and it doesn’t even really hinge on “advanced” AI anyways so it doesn’t really matter. The question is “what gives an entity moral worth?” and my general position is that current arguments center too much on consciousness (which is too loosely defined) and qualia (which is not testable, provable, or even guessable).

I’ll start with Peter Singer’s position that the ability to feel pain is what gives entities moral worth. Given his influence on the EA community, I imagine his view is not unpopular here. Singer uses the development of a central nervous system to qualify some animals (but not clams, etc.) as deserving of consideration. This is fine for promoting animal welfare, but it's not hard to build a program that quantifies “pain” and avoids it as we do. We obviously don’t want to give this program moral worth, so this system doesn’t hold. Setting the bar too low isn’t just inaccurate, but turns basically every form of utilitarianism against us humans because it's cheaper and easier to help simulated entities, so we don’t want that.

Some might say ”yeah sure but if we can look in the system code and see that this code block is actually just weighted tendencies towards actions that promote x value and away from actions that reduce x value, then that’s not true pain” as if there’s something in consciousness that is not weighted tendencies. Also worth noting that if something had weights but couldn’t read its weights, it would probably “think” of consciousness as intuition. This is the cartesian dualist position (that thought is not material matter and numbers), but I think that in order for our actions to have any effect on whether or not AI develops something that fits our definition of consciousness, this must not be the case. In other words, if the dualists are right then whether it happens or not is outside of our power. There are also many serious retorts to this line of thinking, notably the position that "consciousness" may very well be simply the best way for unthinking matter to plan things out, etc.

The obvious response is “well, sure that program avoids pain or whatever, but it doesn’t actually feel the feeling of pain,...
Narrative Theory. Part 3. Simplest to succeed
by Eris — 2023-07-16T14:41:21.837Z

"Evolution is a tinkerer, not an engineer. It works with what is already there and takes the path of least resistance.
It is not always the most efficient solution, but it is the dumbest solution that works."
-François Jacob, "The Logic of Life: A History of Heredity"

Reverse engineering complex systems is a tricky problem. Look for example at the design of modern microprocessors, how easy it would be to see the underlying principle of the Turing machine behind all the caches, branch prediction, thread balancing, and the rest. Not that easy I would say. This example might be also applicable to reverse engineering the mind. After peeling out all the evolutionary optimisations the underlying principles of brain design might turn out modestly simple. Some of these optimisations have been already identified and well studied[1] (e.g. mechanisms of translating chemical signals to electrical ones, numerous structural decisions dedicated to spending as less wire and energy as possible, ways of getting information from different sensors to the same frequency). While we have not yet succeeded in this quest the idea of simple but powerful core principles should be part of our strategy of getting there.

Biology offers one more piece of the strategy. Although the overall amount of “design work” done by evolution is incredible, only a fraction of it is directly associated with the decision-making circuitry of the mind. The relatively slow pace of cognitive evolution means that there was not much time for reinventing cognitive architecture between subsequent species. Meaning, the most necessary parts of the apparatus have been already present in primates, some smaller part has been present in mammals, and so on.

The reasoning above together with a liberal application of the Lindy Effect[2] justify us taking a certain stance towards reverse engineering the mind - the longer some specific design principle has been around, the more it got represented in the construction of the system and the more we should put emphasis on it while building our models.

By this logic, we should expect the bulk of the design to be implemented via the use of a tiny set of the oldest mechanisms (if we take into account the timeframes of the introduction of all of them). The prime suspects for that set are:

Reflexes aka direct stimuli-action reactions
Associations. Extension of a sti

...
Runaway Optimizers in Mind Space
by silentbob — 2023-07-16T14:26:45.091Z

This post is an attempt to make an argument more explicit that sometimes seems to be implied (or stated without deeper explanation) in AI safety discussions, but that I haven’t seen spelled out that directly or in this particular form. This post contains nothing fundamentally new, but a slightly different – and I hope more intuitive – perspective on reasoning that has been heard elsewhere. Particularly I attempt to make the argument more tangible with a visualization.

Tl;dr: When considering the case for AI as an existential risk, it may not be all that important what most AIs or the typical AI will be like. Instead there’s a particular type of AI – the ones I refer to as “capable optimizers” in this post – that may ultimately dominate the impact AIs have on the world. This is because their nature implies runaway optimizations that may lead to a trajectory of the world that humans are unable to influence.

Consider the following schematic diagram:

I find that using this visualization helps build an intuition for the case that building powerful AIs might lead to the disempowerment (and possibly extinction) of humanity. Let’s first look into the details of what is depicted:

The outer, grey shape is “mind space”, i.e. some visualization of the space of possible intelligent minds. That space in reality is of course neither 2-dimensional nor necessarily bounded in any direction; still I find it helpful to visualize the space in this simplified way for the sake of the argument
The yellow “humans” blob in the bottom left illustrates that a small subset somewhere in mind space is the space of all human minds, given our current state of evolution.
The GPT4 blob has no direct relevance to the argument; but it’s interesting to realize that GPT4 is not a point in mind space, but rather a relatively broad set, because, based on e.g. user prompting and a conversation’s history (as well as additional system prompts), it may take on very different simulacra/personas that may behave self-consistently, but highly different from other instantiations

Note that arguably we could make a similar case for individual humans, given that e.g. the acute distribution of neurotransmitters in their brain greatly impacts how their mind works
Also note that combining GPT4 with other tools and technologies may shift it to different areas in mind space, a possible example being Aut

...
A simple way of exploiting AI's coming economic impact may be highly-impactful
by  — 2023-07-16T09:33:52.783Z

Scott Alexander ended A Lesswrong Crypto Autopsy with these words

And for those of us who failed - well, the world is getting way too weird to expect there won't be similarly interesting challenges ahead in the future.

This line led me to update towards the current gameboard still being winnable, if we act creatively.
I was laying in bed this morning wondering what odd, creative actions could significantly alter the playing field -- and I remembered that I thought of a potential one, emailed it off to an EA investing group, and then forgot about it. I never got a reply. On the other hand, the two friends I discussed it with both thought it was worth telling others about (even though one was skeptical).

Maybe it's a dumb idea, but given the upside potential if it's not, I think I should share it again, this time publicly: short-selling stocks likely to lose to A(G)I. For example, the stock of Chegg, a homework cheating site, dropped ~45% after a quarterly report stated it was losing signups to ChatGPT. I think this wouldn't have been hard to predict in advance.

Here's how I argued for this idea in the email.

It seems that the market, right now, is largely unaware of the impact AI will have (this comment on 'AGI and the EMH' seems true to me). If so, given many EAs have inside-views of AI's trajectory, it seems to me like looking for and shorting stocks likely to rapidly decrease in value due to outcompetition by AI could be a really effective strategy. For example, if starting with $1m, it would only take successfully predicting 19 similar drops to reach above $1b (but even predicting 1 seems significant, compared to normal return rates). Moreover, it seems a failed prediction doesn't incur much loss in comparison, e.g if a stock's value doesn't change or only shifts a little, then on average one stays equal on failed predictions.

[...] I'm a little hesitant to post this idea publicly, because I don't really have a presence in the EA community yet, but I think that if this idea seems good to you there should definitely be some coordinated effort among EAs to fund an attempt at this. (Another reason is that I'm not sure if it might be slightly infohazardous, since I'd guess there's only some years remaining before the market catches up, and posting publicly could help it catch up sooner.)

[...] Even if it only had a 10% chance of success, the expected gain/loss

...