trentmkelly/LessWrong-43k huggingface dataset:
Assessment of intelligence agency functionality is difficult yet important
by trevor — 2023-08-24T01:42:20.931Z

Summary: When it comes to observing intelligence agencies, it's hard to see the hardened parts and easy to observe the soft corrupt parts. This leads to a bias where very large numbers of people overestimate how prevalent the easily-observed soft and harmless parts are. This can sometimes even result in a dangerous and prevalent estimation, among people whose careers are much further ahead than yours, that the entire intelligence agency is harmless and irrelevant, when it actually isn't. Intelligence agencies are probably a mix of both less-functional, less-relevant parts, and also more-functional, more-relevant parts that have a disproportionately large influence over governments and policies; and it is a mistake to assume that intelligence agencies are homogenously composed of non-functional non-relevant parts that aren't worth paying any attention to, even if such a belief is a popular norm.

Why intelligence agencies are dangerous

There are a wide variety of situations where intelligence agencies suddenly becomes relevant, without warning. For example, most or all of the US Natsec establishment might suddenly and unanimously change its stance on Gain of Function research, such as if US-China relations or US-Russian relations once again hit a new 25-year low (which has actually been happening very frequently over the last few years).

Either the leadership of an agency, or a powerful individual in an agency with authority to execute operations, or a corrupt clique, might personally make a judgement that the best way to expedite or restart GOF research is to target various people who are the most efficient or effective at opposing GOF research.

This need not be anywhere near the most effective way to expedite or protect GOF research, it just needs to look like that, sufficiently for someone to sign off on that, or even for them to merely thing that it would look good to their boss.

Competent or technologically advanced capabilities can obviously be mixed with incompetent administration/decisionmaking in the mixed competence model of intelligence agencies. An intelligence agency that is truly harmless, irrelevant, and not worth paying attention to (as opposed to having an incentive to falsely give off the appearance of harmlessness, irrelevance, or not being worth paying attention to) would have to be an intelligence agency that is both technologically...
China's position on autonomous weapons
by bhauth — 2023-08-23T22:20:14.112Z
Link: https://academic.oup.com/cjip/article/16/1/106/6976053

While continuing to endorse the necessity to develop a legally binding instrument with respect to LAWS, in 2018 China also proposed a somewhat narrow definition of LAWS comprising five essential features:

The first is lethality, which means sufficient payload (charge) for it to be lethal. The second is autonomy, which means absence of human intervention and control during the entire process of executing a task. Third is impossibility of termination, meaning that once activated there is no way of terminating the device. Fourth is indiscriminate effects, meaning that the device will execute the task of killing and maiming regardless of conditions, scenarios, or targets. Fifth is evolution, meaning that through interaction with the environment the device can learn autonomously and expand its functions and capabilities in a way that exceeds human expectations.

[...] China’s restrictive definition sets an extremely high threshold in regard to the kinds of technologies that may be eligible for legal regulation.

Chinese delegates to the GGE that we interviewed, however, do not view this definition as “narrow.” For them, rapid technological advancements could soon make weapons with zero human oversight a reality, especially in technologically advanced countries like the USA.

Military UAVs with autonomous targeting already exist. Turkey probably used one. Israel has also been developing autonomous weapons. The Switchblade 600 isn't autonomous, but US defense contractors are interested in adding autonomous targeting to such products and seem to be lobbying against US regulations on that. The Chinese government seems to be strongly in favor of autonomous weapons, but also strongly against literal Terminators a la the movies.

Thus far, the focus has been on autonomous targeting for small short-range electric UAVs, but autonomy is easier for larger UAVs that can carry more sensors and computing, and datalinks are harder over long distances - especially if satellites were targeted in eg a war over Taiwan between the US and a China-Russia alliance.
Diet Experiment Preregistration: Long-term water fasting + seed oil removal 
by lc — 2023-08-23T22:08:49.058Z

I am intrigued by exfatloss's recent theory that the primary cause of the obesity epidemic is slow, subtle damage from linoleic acid over many years. Specifically, I want to test the broader possibility that the composition of adipose tissues is what's causing the problem. Since we do not have a decade to test his hypothesis naturally, I'll retcon my current eating habits for a modified experiment.

I am just now transitioning from a very low calorie diet to a water fast, having lost around fifteen pounds in the last few weeks. I am going to attempt to continue my water fast for the next thirty days, or until I reach a BMI of <25. After the end of my water fast, I am going to rigorously exclude linoleic acid and other PUFAs from my diet; my meals will consist mostly of vegetables, rice, specially ordered low-PUFA pork, and fish. Then I'm going to eat to satiety for another month.

Normally what is expected to happen after a water fast is that you regain all of the weight you just lost, because you haven't modified your body's dietary "set point" and you wasted all of your mental strength trying to keep yourself from eating. However, if the linoleic acid hypothesis is true, all of the PUFA I've consumed over my many years as a good American are what's causing my set point to remain so high in the first place. So for this self-experiment I'm going to deliberately purge most of my remaining fat stores, fill them up with a traditional Japanese diet, and see if my "natural weight" changes.

It won't invalidate the lineolic acid hypothesis if the damage is just irreversible, but I can at least test out the hypothesis that the running composition of my fat is what causes my metabolism to freak out.
The Low-Hanging Fruit Prior and sloped valleys in the loss landscape
by Dmitry Vaintrob — 2023-08-23T21:12:58.599Z

You can find code for the referenced experiments in this GitHub repository

Many have postulated that training large neural networks will enforce a simplicity, or

Solomonoff prior

. This is grounded in the idea that simpler solutions occupy expansive regions in the weight space (there exist more generalization

directions in weight space along which loss does not increase or increases very little

), translating to a broad attractor basin where perturbations in weight adjustments have a marginal impact on the loss.

However, stochastic gradient descent (SGD), the workhorse of deep learning optimization, operates in a manner that challenges this simplicity-centric view. SGD is, by design, driven by the immediate gradient on the current batch of data. The nature of this process means that SGD operates like a greedy heuristic search, progressively inching towards solutions that may be incrementally better but not necessarily the simplest.

Part of this process can be understood as a collection of "grokking" steps, or phase transitions, where the network learns and "solidifies" a new circuit corresponding to correctly identifying some relationships between weights (or, mathematically, finding a submanifold). This circuit then (often) remains "turned on" (i.e., this relationship between weights stays in force) throughout learning.

From the point of view of the

loss landscape

, this can be conceptualized as recursively finding a valley corresponding to a circuit, then executing search within that valley until it meets another valley (corresponding to discovering a second circuit), then executing search in the joint valley of the two found circuits, and so on. As the number of circuits learned starts to saturate the available weight parameters (in the underparametrized case), old circuits may get overwritten (i.e., the network may leave certain shallow valleys while pursuing new, deeper ones). However, in small models or models not trained to convergence, we observe that large-scale circuits associated with phase transitions largely survive to the end.

This gif shows the Fourier modes of the learned embeddings in our modular addition MLP model. Circles correspond to fully grokked circuits of the kind found in

Progress measures for grokking via mechanistic interpretability

A greedier picture

This idea aligns with what we call the low-hanging fruit pr...
Governing, Fast and Slow
by Carson — 2023-08-23T20:01:41.504Z

Systems and their Mechanisms

A system's mechanisms are fast or slow. Fast mechanisms are inherent and thus adopted by default. Slow mechanisms are formed by actors and take time, energy and cooperation. Actors are things which knowingly or unknowingly take advantage of fast mechanisms and are composed of slow mechanisms.

Fast or Inherent Mechanisms

Fast mechanisms are fast because they are static in the system. Some systems heavily comprised of fast mechanisms are nature, unregulated markets and pure democracies. All of these systems can thrive under the correct circumstances, but their inherent problems are insurmountable. Natural selection is a fine mechanism for change if you're willing to wait for millions of years. Unregulated markets are bountiful until monopolies inevitably form. Pure democracies are effective until size makes discourse impossible.

Accepting fast mechanisms (aka doing nothing) with no other action is taking a leap of faith that the system will not collapse. All systems collapse eventually from the perspective of a status quo, thus humans are always fighting to keep a set of status quos. Examples of status quos many take for granted are a breathable atmosphere, farmable soil and potable water. None of the aforementioned things would exist for long in a world dependent only on fast mechanisms, thus durable human systems employ slow mechanisms in order to protect the status quos which people depend on.

Another important thing about fast mechanisms is that many exist unbeknownst to the actors within the system, thus effective actors take time to discover the fast mechanisms within their system. Fast mechanisms can also be greatly misunderstood and entire disciplines can be formed around faulty understanding (e.g. alchemy, witchcraft, religious practices). The process of discovering fast mechanisms is what humans call science.

Examples of fast mechanisms:

Physics: fundamental constants, entropy, conservation laws.
Economics: global market forces, exogenous shocks, and price levels.
Political Science: public opinions, cultural inertia, black swan events.

Slow or Composed Mechanisms

Slow mechanisms are slow because they require pre-meditation and time for integration. There is an infinite set of slow mechanisms to employ and this is part of what makes them slow: time must be allocated for searching the space of mechanisms. Another reason sl...
A problem with the most recently published version of CEV
by ThomasCederborg — 2023-08-23T18:05:46.658Z

This post outlines a thought experiment, designed to illustrate a problem with the Parliamentarian version of CEV (PCEV). It then shows that the problem is not inherent in the scenario of the thought experiment, by describing a large space of other designs, that does not suffer from this problem (based on modifying PCEV, in a way that gives each individual, meaningful influence, over the adoption of those preferences, that refer to her).

The most recently published version of CEV, uses Parliamentarianism to deal with disagreements, among individuals, that disagree on how to deal with disagreements. Let's refer to the AI, that implements this Parliamentarian definition, of the CEV of humanity, as PCEV. The basic idea of PCEV, is that extrapolated delegates, representing billions of humans, will negotiate and vote, on what PCEV will do.

PCEV is described in an arbital entry from 2016.

It links to an OB post by Bostrom, from 2009, on Parliamentarianism.

The 2009 OB post, proposes to use Parliamentarian negotiations, for a different purpose. The 2016 arbital entry, proposes to re purpose the idea for CEV.

The idea, that was only briefly mentioned in the short OB post, was not written up until 2021, when Toby Newberry and Toby Ord published: The Parliamentary Approach to Moral Uncertainty.

One central aspect of Parliamentarianism, in all three references above, is that delegates must negotiate and vote, under the assumption, that votes will be settled stochastically (votes are, however, not actually settled stochastically). Without this feature, the thought experiment below does not work. However, removing this feature would result in a fundamentally different type of AI, with a number of other problems (for example: any solid majority, no matter how narrow, would be able to do, literally, anything that they want, to everyone else).

The 2021 paper describes more than one version of Parliamentarian negotiations. In a version with multiple separate votes, on separate decisions, the resulting AI would sometimes prefer A to B, B to C, and C to A (see for example 3.2, in the 2021 paper). Thus, the thought experiment below uses the single vote version of Parliamentary negotiations (so, basically, the extrapolated delegates will vote, on what successor AI, PCEV should build).

The thought experiment:

Assume that each individual has a delegate that will represent the interests of ...
Which paths to powerful AI should be boosted?
by Zach Stein-Perlman — 2023-08-23T16:00:00.790Z

Some AI safety methods/mechanisms can be tacked onto many kinds of AI systems. But separately, some paths to powerful AI are safer or more alignable than others:

Maybe whole-brain emulations are safer than de novo AI
Maybe feedforward-y systems are safer than recurrent-y systems
Maybe LM agents (LM-based systems with sophisticated scaffolding) are safer than similarly powerful base models[1]
Maybe process-based systems are safer than outcome-based systems

WBE seems very unlikely to appear before strong de novo AI. But other relatively-safe-paths may be competitive (i.e. not require much extra-cost and capabilities-sacrifice relative to unsafe paths). This has important implications-- it means that AI developers should prioritize those paths, and especially should differentially publish research on those paths to differentially boost others on those paths.[2]

Which paths to powerful AI are relatively safe and potentially competitive, and thus should be boosted?

This question is a more focused successor to Which possible AI systems are relatively safe?

^

Paul

says

"My guess is that if you hold capability fixed and make a marginal move in the direction of (better LM agents) + (smaller LMs) then you will make the world safer. It straightforwardly decreases the risk of deceptive alignment, makes oversight easier, and decreases the potential advantages of optimizing on outcomes."

^

There's a quote I'm forgetting on differential technological development like if there’s an unsafe path and a safer path and the unsafe path is ahead (in terms of capabilities), we should rush to make progress on the safer path so that it gets ahead and even non-safety-motivated researchers switch to the safer path.
A Theory of Laughter
by Steven Byrnes — 2023-08-23T15:05:59.694Z

(Last revised: December 2024. See changelog at the bottom.)

1. tl;dr

There should be parallel explanations for laughter at two levels.

At the brain level, there should be some mechanism / algorithm that produces laughter, and it should fit the data of when people laugh in practice.
At the evolution level, there should be some explanation for why this mechanism exists in the first place. Why was it adaptive in our ancestors? And where did it come from—are there homologues in other animals?

I’ll summarize my proposals for both of these, in the opposite order:

1.1 First half of the tl;dr: Laughter in terms of evolution

I endorse the popular theory that laughter is an indicator of “play”, homologous to the play-related vocalizations and body language in other animals (e.g. the dog’s “play bow”).

The evolutionary purpose of play is “practice for future high-stakes situations”. For example, a wolf pup that engages in play-fighting and play-chasing would presumably be more skilled in its future real-life fights and chases.

“High-stakes situation” usually means “immediate physical danger”. But situations can also be evolutionarily high-stakes for other reasons—think of behaviors related to reproduction (asking your crush on a date), or kin selection (threats to one’s children), or long-term access to resources (threats to one’s social status), etc.

The evolutionary purpose of innate communicative play signals, like laughter in humans and play-bows in dogs, is to reduce the probability of accidental escalation from practice to serious. For example, if a play-fight between two wolf-pups escalates into a real fight between the pups, that’s dangerous for both pups. If the pups are emitting and responding to communicative play signals, then that kind of escalation is much less likely to happen. It’s kinda the same idea as

“safewords” in fight-related sports

(among other places).

1.2 Second half of the tl;dr: Laughter in terms of brain algorithms

My (oversimplified) pseudocode brain

“business logic”

for laughter is something like:

PROPOSED BRAIN PSEUDOCODE FOR LAUGHTER

:

(A) IF my hypothalamus & brainstem are getting some evidence that I’m in a high-stakes (e.g. dangerous) situation

(the “evidence” here would presumably be some of the same signals that, by themselves, would tend to cause physiological arousal / increase my heart rate / activate my sympat

...
Why Is No One Trying To Align Profit Incentives With Alignment Research?
by Prometheus — 2023-08-23T13:16:41.927Z

A whole lot of Alignment work seems to be resource-constrained. Many funders have talked about how they were only able to give grants to a small percentage of projects and work they found promising. Many researchers also receive a small fraction of what they could make in the for-profit sector (Netflix recently offered $900k for an ML position). The pipeline of recruiting talent, training, and hiring could be greatly accelerated if it wasn’t contingent on continuing to receive nonprofit donations.

Possible Ideas

AI Auditing Companies

We’ve already seen a bit of this with ARC’s eval of GPT4, but why isn’t there more of this? Many companies will/are training their own models, or else using existing models in a way beyond what they were intended. Even starting with non-cutting-edge models could provide insight and train people to have the proper Security Mindset and understanding to audit larger ones. Furthermore, there has been a push to regulate and make this a required practice. The possibility of this regulation being made into law will likely be contingent on the infrastructure for it already existing. And it makes sense to take action toward this now, if we want those auditing teams to be as useful as possible, and not merely satisfy a governmental requirement. Existential concerns would also be taken more seriously by a company that has already built a reputation for auditing models.

Evals reporting

Companies don’t want to see their models doing things that weren’t intended (example, giving people credit card information, as was just recently demonstrated). And as time goes on, companies will want some way of showcasing their models have been rigorously tested. Audit reports covering a large, diverse set of vulnerabilities is something many will probably want.

Red teaming

Jailbreaking has been a common practice, done by a wide number of people after a model is released. Like an Evals Report, many will want a separate entity that can red team their models, the same way many tech companies hire an external cybersecurity company to provide a similar service.

Alignment as a service

This could bring in new talent and incentives toward building better understanding and talent to handle alignment. These services would be smaller scale, and would not tackle some of the “core problems” of alignment, but might provide pieces to the puzzle. Solving alignme...
Exploring the Responsible Path to AI Research in the Philippines
by MiguelDev — 2023-08-23T08:44:18.657Z

Introduction

The transformative potential of artificial intelligence (AI) is reshaping our world, presenting both thrilling opportunities and profound challenges. In the Philippines, where Dominic Ligot (Founder, AI expert & communicator)[1]and I have delved into AI's possibilities, the integration of this technology into our unique cultural fabric can unleash exponential productivity. This post explores how we (as a country) can harness AI to enhance our local community without losing our distinct identity. With our shared fascination as a spark, I'll examine the broader implications of AI, guided by the imperative to understand its complexities and the need to proceed with caution and adherence to alignment research[2].

Transformative Technologies and AI Research

The Transformative Potential of AI

AI stands as a transformative technology, holding the potential to reshape our world in the coming decade. The failure to understand and integrate AI could leave a country behind in a rapidly changing landscape.

Historical Precedence of Technological Transformations

Technological advancements have driven significant transformations. Whether through the application of cutting-edge physics in warfare or the discovery of new scientific fields, technological innovations have often led to pivotal moments in human history.

The Complexity and Urgency of AI Research

AI research is neither simple nor conceptually straightforward. The very complexity of the field underscores the importance of pursuing a deep understanding of it. While the path towards mastering AI may seem daunting and even wild, the stakes are too high for complacency.

The Inevitable Direction of AI Research

Our current situation leaves us with few choices but to embrace AI research. It's not merely an option; it's an essential direction we must take to ensure our survival in a future increasingly shaped by artificial intelligence. The question is not whether we should pursue AI research, but how we can do so most effectively and responsibly.

Having explored the immense transformative power of AI and its historical parallels, it's essential to consider both the challenges and promises this complex field presents to us today.

Upsides and Downsides

AI Research Challenges and Uncertainties

AI research at present resembles alchemy more than mathematics. We still lack a formal theory on the malleability of d...