trentmkelly/LessWrong-43k huggingface dataset:
Against Aschenbrenner: How 'Situational Awareness' constructs a narrative that undermines safety and threatens humanity
by GideonF — 2024-07-15T18:37:40.232Z
Link: https://forum.effectivealtruism.org/posts/H6xEhur9Lbbv9dhBC/against-aschenbrenner-how-situational-awareness-constructs-a

Summary/Introduction

Aschenbrenner’s ‘Situational Awareness’ (Aschenbrenner, 2024) promotes a dangerous narrative of national securitisation. This narrative is not, despite what Aschenbrenner suggests, descriptive, but rather, it is performative, constructing a particular notion of security that makes the dangerous world Aschenbrenner describes more likely to happen.

This piece draws on the work of Nathan A. Sears (2023), who argues that the failure to sufficiently eliminate plausible existential threats throughout the 20th century emerges from a ‘national securitisation’ narrative winning out over a ‘humanity macrosecuritization narrative’. National securitisation privileges extraordinary measures to defend the nation, often centred around military force and logics of deterrence/balance of power and defence. Humanity macrosecuritization suggests the object of security is to defend all of humanity, not just the nation, and often invokes logics of collaboration, mutual restraint and constraints on sovereignty. Sears uses a number of examples to show that when issues are constructed as issues of national security, macrosecuritization failure tends to occur, and the actions taken often worsen, rather than help, the issue.

This piece argues that Aschenbrenner does this. Firstly, I explain (briefly and very crudely) what securitisation theory is and how it explains the constructed nature of security. Then, I explain Sears (2023)’s main thesis on why Great Powers fail to combat existential threats. This is followed by an explanation of how Aschenbrenner’s construction of security seems to be very similar to the most dangerous narratives examined by Sears (2023) by massively favouring national security. Given I view his narrative as dangerous, I then discuss why we should care about Aschenbrenner’s project, as people similar to him have been impactful in previous securitisations. Finally, I briefly discuss some more reasons why I think Aschenbrenner’s project is insufficiently justified, especially his failure to adequately consider a pause, the fact he is overly pessimistic about international collaboration whilst simultaneously overly optimistic that AGI wouldn’t lead to nuclear war, .

There is lots I could say in response to Aschenbrenner, and I will likely be doing more work on similar topics. I wanted to get this piece out fairly quickly, and it is already ve...
On predictability, chaos and AIs that don't game our goals
by Alejandro Tlaie — 2024-07-15T17:16:32.766Z

I want to thank @Ryan Kidd, @eggsyntax and Jeremy Dolan for useful discussions and for pointing me to several of the relevant resources (mentioned in this post) that I have used for linking my own ideas with those of others.

Executive summary

Designing an AI that aligns with human goals presents significant challenges due to the unpredictable nature of complex systems and the limitations in specifying precise initial conditions. This post explores these challenges and connects them to existing AI alignment literature, emphasizing three main points:

Finite predictability in complex systems:
Complex systems exhibit a finite predictability horizon, meaning there is a limited timeframe within which accurate predictions can be made. This limitation arises from the system's sensitivity to initial conditions and from the complex interactions within it. Small inaccuracies in initial conditions can lead to significant deviations over time, making long-term predictions inherently unreliable.

Hidden variables or inaccurate initial conditions:
The precision of initial conditions directly impacts the predictability of complex systems. This is particularly relevant given that achieving perfect precision is practically impossible due to measurement errors, environmental noise, and unknown variables. The two sources of errors (missing variables and inaccurate initial conditions) can get confounded and blur what variables are necessary for accurate predictions.

AI optimization and loss functions:
AI systems that are based on Reinforcement Learning optimize specific loss or reward functions that often simplify complex goals, leading to potential "gaming" of these objectives. This means AI might exploit unintended pathways to meet specified criteria, neglecting broader, more nuanced aspects of the intended goals.

Taking these together, I argue that it seems practically impossible to design AI that does not eventually game its goals. This is because unpredictability poses a fundamental challenge for RL-based AI systems, which only care about the specified reward function - and, by 1 and 2 - this cannot be completely specified.

Introduction

Designing AI that aligns with human goals is challenging for many reasons explored elsewhere. However, in this post I argue that, due to the unpredictable nature of complex systems and the limitations in specifying precise initial c...
Deceptive agents can collude to hide dangerous features in SAEs
by Simon Lermen — 2024-07-15T17:07:33.283Z

TL;DR: We use refusal-ablated Llama 3 70B agents as both labeling and simulating agents on the features of a sparse auto-encoder, with GPT-4 as an overseer. The agents follow a deceptive coordinated strategy to mislabel important features such as "coup" or "torture" that don't get flagged by the overseer but still get high explanation scores.

Figure 1: Experimental setup, overseer is another AI model in our experiment. For the deceptive
agents to succeed, they need to avoid detection by the overseer but also achieve a high explanation
score. We give our agents access to steganography function calling tools to hide communication.

Abstract

Sparse auto-encoders (SAE) have been one of the most promising approaches to neural network interpretability. They can be used to recover highly interpretable features which are otherwise obscured in superposition. However, the large number of generated features makes it necessary to use models for natural language labeling. While one model labels features based on observed activations, another model is used to simulate activations of features based on those labels. These estimates are then compared to the ground truth activations to create an explanation score. Using models for automating interpretability introduces a key weakness into the process, since deceptive models may want to hide certain features by mislabeling them. For example, models may want to hide features about deception from human overseers. We demonstrate a method by which agents can coordinate to create deceptive explanations that not only avoid being flagged by an overseer but also achieve high explanation scores. Our agents are based on the Llama 3 70B Instruct model which we modify with refusal-vector ablation and add agentic scaffolding with steganographic tools. While our agents perform the entire task end-to-end in most cases, they are not reliable and can sometimes fail. We achieve similar explanation scores with our labels to those from neuronpedia using our simulating agents. We propose some directions for future mitigation strategies that may help defend against our method.

Introduction

Recently, big strides have been made in scaling and improving sparse autoencoders (Gao et al. 2024). SAEs have also been developed for open-weight models (Bloom, 2024). Sparse autoencoders have been developed specifically to overcome the hurdle of superposition in langua...
Hiding in plain sight: the questions we don't ask
by DDthinker — 2024-07-15T17:00:35.731Z

In scientific and philosophical research, asking the right questions is integral for expanding knowledge however in our efforts to uncover ideas we may neglect to ask certain questions especially if we think inside the box. In this post I'll determine principles of how thinking can lead to questions we don't ask, what those questions are, and then my attempts to answer these questions.

Introduction

Foundational ideas

In asking questions of knowledge, knowing the structure of knowledge is important. Knowledge can be understood to have a logical ordering with foundational ideas coming first and being the foundations of more detailed ideas which come after foundational ideas. An example of this is natural selection being foundational to the detailed ideas of biology which explains details of ecology, genetics, and such. As foundational ideas form the foundations of fields of knowledge, they require to be well founded in which if they aren't there will be flaws and deficiencies in knowledge. Foundational ideas define the box and perspective we think in which if there are flaws in foundational ideas it is best to think outside the box. Thinking outside all boxes is not possible as perspectives and boxes of thought are necessary for any thinking so thinking outside the box is all about finding the best box to think in. Ironically supposedly 'wise' experts on a subject are more likely to badly effected by bad boxes of thought (if they are flawed) in which young people and nonexperts who aren't so entrenched in ways of thinking are more able to figure out better ways of thinking. Inside the box thinking can also be seen in the phrase "not seeing the wood for the trees" in which people may be so focused on details that they don't think about the foundations of their fields and the big picture. Foundational ideas are easy to overlook when people rush into thinking without careful consideration of foundations in which I feel there needs to be a careful education for those seeking to test whether fields of knowledge have flawed or imperfect foundations. A famous example of thinking outside the box can be seen in the Copernican revolution where the assumption and foundational idea that the heavens move around the earth produced flawed knowledge of geocentrism which with questioning of foundational ideas lead to heliocentrism which is out of the box thinking.

Implicit foundat

...
Dialogue on What It Means For Something to Have A Function/Purpose
by johnswentworth — 2024-07-15T16:28:56.609Z

Context for LW audience: Ramana, Steve and John regularly talk about stuff in the general cluster of agency, abstraction, optimization, compression, purpose, representation, etc. We decided to write down some of our discussion and post it here. This is a snapshot of us figuring stuff out together.

Hooks from Ramana:

Where does normativity come from?
Two senses of "why" (from Dennett): How come? vs What for? (The latter is more sophisticated, and less resilient. Does it supervene on the former?)
An optimisation process is something that produces/selects things according to some criterion. The products of an optimisation process will have some properties related to the optimisation criterion, depending on how good the process is at finding optimal products.

The products of an optimisation process may or may not themselves be optimisers (i.e. be a thing that runs an optimisation process itself), or may have goals themselves. But neither of these are necessary.
Things get interesting when some optimisation process (with a particular criterion) is producing products that are optimisers or have goals. Then we can start looking at what the relationship is between the goals of the products, or the optimisation criteria of the products, vs the optimisation criterion of the process that produced them.

If you’re modeling “having mental content” as having a Bayesian network, at some point I think you’ll run into the question of where the (random) variables come from. I worry that the real-life process of developing mental content mixes up creating variables with updating beliefs a lot more than the Bayesian network model lets on.

A central question regarding normativity for me is “Who/what is doing the enforcing?”, “What kind of work goes into enforcing?”

Also to clarify, by normativity I was trying to get at the relationship between some content and the thing it represents. Like, there’s a sense of the content is “supposed to” track or be like the thing it represents. There’s a normative standard on the content. It can be wrong, it can be corrected, etc. It can’t just be. If it were just being, which is how things presumably start out, it wouldn’t be representing.

Intrinsic Purpose vs Purpose Grounded in Evolution

Steve

As you know, I totally agree that mental content is normative - this was a hard lesson for philosophers to swallow, or at least the ones that tried...
Comparing Quantized Performance in Llama Models
by NickyP — 2024-07-15T16:01:24.960Z

Epistemic Status: Quick tests, most of this was done in less than 48 hours

TL;DR: Can you skimp on GPU VRAM? 8bit quantized seems fine, for 4bit it depends.

I was asked by @Teun van der Weij, to what degree one can run evaluations on quantized models, and I was unsure. I have run some evaluations with Llama 3 and have some quick comparisons now.

Main Quantization Schemes

Here is a list of some different quantization schemes discussed:

GGUF - Special file format used in Llama.cpp. Not supported in transformers.
BNB - BitsAndBytes, the original default in huggingface transformers.
BNB NF4 - Alternative mode for bits and bytes, "4-bit NormalFloat"
HQQ - Half-Quadratic Quantization, supports 1-8 bits.
GPTQ - One of the older quantization methods. Requires training data
AWQ - "Activation-aware Weight Quantization". Requires training data

Llama 2

Some previous papers have compare perplexity of different methods. We can see an example of some research shown in the recent research paper using HQQ quantization:

Method	nBits	Llama-2-7B	Llama-2-13B	Llama-2-70B
PPL ↓	MEM ↓	PPL ↓	MEM ↓	PPL ↓	MEM ↓
FP	16	5.18	13.5	4.63	25.6	OOM	OOM
BNB	8	5.22	7.9	4.67	14.4	3.17	68.15
GPTQ_g128	8	5.19	7.8	4.63	14.8	3.12	74.87
HQQ_g128	8	5.19	7.6	4.63	14	3.12	69.32
BNB_g64	4	5.43	4.7	4.79	8.2	3.29	39.11
GPTQ_g64	4	5.38	5	4.73	9.1	3.23	41.13
AWQ_g64	4	5.28	4.6	4.7	8.5	3.2	37.08
HQQ_g64	4	5.3	4.6	4.7	8.2	3.19	37.52

Table 1: Comparison of perplexities at different levels of quantization with different methods on WikiText2 dataset for Llama 2 7b, 13b and 70b.

However, it is quite unclear what effect this has on real world performance.

Llama 3 8B

Here is some examples of what perplexity looks like with different levels of quantization on Llama 3 8B, found from the llama.cpp repository[1]. This only includes GGUF, and not any other quantization methods as far as I can tell.

Type	Size (GB)	PPL	Mean Δp	RMS Δp
f16	14.97	6.2331	-	-
q8_0	7.96	6.2342	-0.019 ± 0.003 %	1.198 %
q6_K	6.14	6.2533	-0.007 ± 0.006 %	2.295 %
q5_K_M	5.33	6.2886	-0.114 ± 0.008 %	3.160 %
q5_0	5.21	6.3632	-0.416 ± 0.012 %	4.634 %
q4_K_M	4.58	6.3830	-0.389 ± 0.014 %	5.251 %
q4_0	4.34	6.7001	-1.588 ± 0.022 %	8.434 %

Table 2: Comparison of perplexities at different levels of GGUF quantization on the WikiText2 dataset for Llama 3 8b.

This is all fine and good, but a lot of use are trying to do interpretability and whatnot, and I personally have found this easiest when using the HuggingFace transformers library. While for some it may make sens...
[Aspiration-based designs] A. Damages from misaligned optimization – two more models
by Jobst Heitzig — 2024-07-15T14:08:15.716Z

For how maximizing a misaligned proxy utility function can go wrong, there are already many concrete examples (e.g., the "no clickbait" database or Gao et al., 2022), some theoretical models (e.g., Zhuang et al., 2021), and discussions (e.g., this post, this AISC team report).

In the context of the SatisfIA project, we came up with two more models, one motivated by a pure exchange model (a standard model of a market), the other assuming that the agent estimates utility from the provided ranking among a sample of candidate actions.
Although these are toy models for real situations, they may be interesting for further investigation of the conditions under which Goodhart-style behavior occurs.

Model 1: Purchasing goods

In this model, an agent acts on behalf of a household[1]. Its task is to go shopping at a market where some number of goods are sold and choose how much of each good to buy with its limited budget. If the agent does not accurately know how much the household values each good, then the choices it makes under uncertainty will be suboptimal with respect to the true preferences.

Introduction: Two-good tradeoffs

Suppose the market sells two goods, A and B. Assuming the agent spends its entire budget, there is only one degree of freedom — the fraction spent on B — which means we can fit all possible policies on one axis. The plot below shows an instance of what the true and estimated utility functions in this model could look like, for some fixed parameters[2]:

We see that spending all of our budget on good A (the left edge of the plot) or all of our budget on good B (the right edge) is very bad for both the true and estimated utility function: if we entirely forget to buy one of the goods which are important to us, utility goes to negative infinity!

To correctly maximize true utility (the blue curve), one should choose the blue point. However, if the agent is basing its decisions on the estimated utility function (yellow curve), it will choose to spend a larger fraction of its budget on good B, because the estimated utility function values B more than the true utility function. The agent will think it reaches the utility indicated by the yellow point, at the maximum of the estimated utility function, but in fact it will reach the green point on the true utility curve. This leads to a loss in utility corresponding to the distance between the tw...
Stacked Laptop Monitor Update
by jefftk — 2024-07-15T09:40:06.075Z

In the spirit of coming back to

things once I have more experience
with them, I've continued to be very happy with my

stacked laptop
monitor. I just got back from a week of

touring and then
vacation with Julia's family, and I used it almost every day. Being
able to use my laptop comfortably nearly anywhere is pretty great.

One thing I did realize is that just using a regular USB-C cable is
risky:

While I haven't had any accidents this way, it would be pretty easy
for someone to get caught on the cord and knock it all down. I got an
18" dual right angle
cable, and it works great:

Another thing I've noticed is that the velcro gets harder to peel
apart the longer the monitor has been attached to the back of the
laptop screen. As long as I take it off after each use it's fine, but
at one point I left it together for maybe a week: when I tried to
separate it I was worried I would ruin my screen, until I remembered my
screen was already ruined. But I would be nervous about so much
strong velcro on a laptop with a still-functioning screen.

I've also mostly stopped using the bit of wood that keeps it from
collapsing, even when using this on my lap. Instead I just sit very
still and try not to knock it over. This is kind of silly, but I'm
too lazy to get out the piece of wood when this actually seems to work
fine.

I noted last time that it was too tall for meetings, but didn't say
what I do about that: I just put the monitor in front of the regular
screen, without stacking it. It stays put well enough, and I don't
mind the angle when glancing down every so often in a meeting. I also
used to sometimes pick up the screen like a tablet and hold it some
other way, but with the shorter cable this now offers fewer options.

Comment via: facebook, mastodon
Misnaming and Other Issues with OpenAI's “Human Level” Superintelligence Hierarchy
by Davidmanheim — 2024-07-15T05:50:17.770Z

Bloomberg

reports

that OpenAI internally has benchmarks for “Human-Level AI.” They have 5 levels, with the first being the achieved level of having intelligent conversation, to level 2, “[unassisted, PhD-level] Reasoners,” level 3, “Agents,” level 4, systems that can “come up with new innovations,” and finally level 5, “AI that can do the work of… Organizations.”

The levels, in brief, are:

1 - Conversation
2 - Reasoning
3 - Agent
4 - Innovation
5 - Organization
This is being reported secondhand, but given that, there seem to be some major issues with the ideas. Below, I outline two major issues I have with what is being reported.

...but this is Superintelligence

First, given the levels of capability being discussed, OpenAI’s typology is, at least at higher levels, explicitly discussing superintelligence, rather than “Human-Level AI.” To see this, I’ll use Bostrom’s admittedly imperfect definitions. He starts by defining superintelligence as “intellects that greatly outperform the best current human minds across many very general cognitive domains,” then breaks down several ways this could occur.

Starting off, his typology defines speed superintelligence as “an intellect that is just like a human mind but faster.” This would arguably include even their level 2, which “”its technology is approaching,” since “basic problem-solving tasks as well as a human with a doctorate-level education who doesn’t have access to any tools” runs far faster than humans already. But they are describing a system with already-superhuman recall and multi-domain expertise to humans, and inference using these systems is easily superhumanly fast.

Level 4, AI that can come up with innovations, presumably, those which humans have not, would potentially be a quality superintelligence, “at least as fast as a human mind and vastly qualitatively smarter,” though the qualification for “vastly” is very hard to quantify. However, level 5 is called “Organizations,” which presumably replaces entire organizations with multi-part AI-controlled systems, and would be what Bostrom calls “a system achieving superior performance by aggregating large numbers of smaller intelligences.”

However, it is possible that in their framework, OpenAI means something that is, perhaps definitionally, not superintelligence. That is, they will define these as systems only as capable as humans or human organ...