trentmkelly/LessWrong-43k huggingface dataset:
What is the foundation of me experiencing the present moment being right now and not at some other point in time?
by MvB — 2023-06-17T20:47:32.175Z

This question may be either trivial to answer and I just don‘t get it, or it may be highly controversial.
I am trying to pose my problem as shortly as I can.

What is responsible for my subjective experience of the present moment (or rather, the immediate past, taking information processing into account)?
Why is now actually now?

Under the premise of spacetime being a static and eternal thing, doesn‘t any line of thought trying to answer this question necessarily make any intuitive notions of identity and the passing of time illusionary?

The only answer that I can come up with is that every conscious moment exists, despite any causal connections, for itself in some eternal sense. In this sense, „I“ right now am just existent at some specific point in spacetime, and before and after that there are, with regard to consiousness, separated and slightly different „mes“ which just happen to be in the closest proximity possible to the „me“ in that moment. Is there any more intuitive way of thinking about this?

Probably relates to timeless physics and quantum immortality.
Adventist Health Study-2 supports pescetarianism more than veganism
by Elizabeth — 2023-06-17T20:10:06.161Z

Or: how the Adventist Health Study-2 had a pretty good study design but was oversold in popular description, and then misrepresented its own results.

When I laid out my existing beliefs on veganism and nutrition I asked people for evidence to the contrary. By far the most promising thing people shared was the 7th Day Adventist Health Studies. I got very excited because the project promised something of a miracle in nutrition science: an approximate RCT. I read the paper that included vegan results, and while it’s still very good as far as nutrition studies go it’s well below what I was promised, and the summaries I read were misleading. It’s not a pseudo-RCT, and even if you take the data at face value (which you shouldn’t) it doesn’t show a vegan diet is superior to all others (as measured by lifespan). Vegan is at best tied with pescetarianism, and in certain niche cases (e.g. being a woman) it’s the worst choice besides unconstrained meat consumption.

I’m going to try not to be too sarcastic about this, the study really is very good data by nutrition science standards, but I have a sour spot for medical papers that say “people” when they mean “men”, so probably something will leak out. Also, please consider what the state of nutrition science must be to make me call something that made this mistake “very good”.

Background

The 7th Day Adventists are a fairly large Christian sect. For decades scientists have been recruiting huge cohorts to study their diet, and publishing a lot of papers.

The Adventists are a promising group to use to study nutrition for lots of reasons, but primarily because the Church discourages meat, smoking, and drinking. So you lose the worst confounders for health, and get a population of lifelong, culturally competent vegetarians, which is a pretty good deal. Total abstinence from meat isn’t technically required – you’re allowed to eat kosher meat – but it’s heavily discouraged. 7DA colleges only serve vegetarian meals, and church meals will typically be vegetarian.

Some popular descriptions say that rules vary between individual churches, which could give you an almost RCT effect. AFAICT this isn’t true, and the paper I read never claimed it was. Both the internet and my ex-Adventist friend say that individual churches within the US (where the study took place) vary a little in recommendations, but most of the v...
The environment as infrastructure
by jasoncrawford — 2023-06-17T18:42:35.200Z
Link: https://rootsofprogress.org/environment-as-infrastructure

A good metaphor for the ideal relationship between humanity and the environment is that the environment is like critical infrastructure.

Infrastructure is valuable, because it provides crucial services. You want to maintain it carefully, because it’s bad if it breaks down.

But infrastructure is there to serve us, not for its own sake. It has no intrinsic value. We don’t have to “minimize impact” on it. It belongs to us, and it’s ours to optimize for our purposes.

Infrastructure is something that can & should be upgraded, improved upon—as we often improve on nature. If a river or harbor isn’t deep enough, we dredge it. If there’s no waterway where we want one, we dig a canal. If there is a mountain in our way, we blast a tunnel; if a canyon, we span it with a bridge. If a river is threatening to overflow its banks, we build a levee. If our fields don’t get enough water, we irrigate them; if they don’t have enough nutrients, we fertilize them. If the water we use for drinking and bathing is unclean, we filter and sanitize it. If mosquitoes are spreading disease, we eliminate them.

In the future, with better technology, we might do even more ambitious upgrades and more sophisticated maintenance. We could monitor and control the chemical composition of the oceans and the atmosphere. We could maintain the level of the oceans, the temperature of the planet, the patterns of rainfall.

The metaphor of environment as infrastructure implies that we should neither trash the planet nor leave it untouched. Instead, we should maintain and upgrade it.

(Credit where due: I got this idea for this metaphor from Stewart Brand; the elaboration/interpretation is my own, and he might not agree with it.)
A summary of current work in AI governance
by constructive — 2023-06-17T18:41:13.187Z
Link: https://forum.effectivealtruism.org/posts/8KhGio2rhgHgsBoZ6/a-summary-of-current-work-in-ai-governance

A summary of current work in AI governance

If you’d like to learn more about AI governance, apply to the

AI Safety Fundamentals: Governance Track

, a 12-week, part-time fellowship before June 25.

Context

For the past nine months, I spent ~50% of my time upskilling in AI alignment and governance alongside my role as a research assistant in compute governance.

While I discovered great writing characterizing AI governance on a high level, few texts covered which work is currently ongoing. To improve my understanding of the current landscape, I began compiling different lines of work and made a presentation. People liked my presentation and suggested I could publish this as a blog post.

Disclaimers:

I’ve only started working in the field ~9 months ago
I haven’t run this by any of the organizations I am mentioning. My impression of their work is likely different from their intent behind it.
I’m biased toward the work by GovAI as I engage with that most.
My list is far from comprehensive.

What is AI governance?

Note that I am primarily discussing AI governance in the context of preventing existential risks.

Matthijs Maas defines

AI long-term governance as

“The study and shaping of local and global governance systems—including norms, policies, laws, processes, politics, and institutions—that affect the research, development, deployment, and use of existing and future AI systems in ways that positively shape societal outcomes into the long-term future.”

Considering this, I want to point out:

AI governance is not just government policy, but involves a large range of actors. (In fact, the most important decisions in AI governance are currently being made at major AI labs rather than at governments.)
The field is broad. Rather than only preventing misalignment, AI governance is concerned with a variety of ways in which future AI systems could impact the long-term prospects of humanity.

Since "long-term" somewhat implies that those decisions are far away, another term used to describe the field is “governance of advanced AI systems.”

Threat Models

Researchers and policymakers in AI governance are concerned with a range of threat models from the development of advanced AI systems. For an overview, I highly recommend

Allan Dafoe’s research agenda

and

Sam Clarke’s "Classifying sources of AI x-risk"

.

To illustrate this point, I will briefly describe some of the main threat ...
[Linkpost] Rosetta Neurons: Mining the Common Units in a Model Zoo
by Bogdan Ionut Cirstea — 2023-06-17T16:38:16.906Z

This is a linkpost for https://arxiv.org/abs/2306.09346.

Do different neural networks, trained for various vision tasks, share some common representations?
In this paper, we demonstrate the existence of common features we call "Rosetta Neurons" across a range of models with different architectures, different tasks (generative and discriminative), and different types of supervision (class-supervised, text-supervised, self-supervised). We present an algorithm for mining a dictionary of Rosetta Neurons across several popular vision models:
Class Supervised-ResNet50, DINO-ResNet50, DINO-ViT, MAE, CLIP-ResNet50, BigGAN, StyleGAN-2, StyleGAN-XL.
Our findings suggest that certain visual concepts and structures are inherently embedded in the natural world and can be learned by different models regardless of the specific task or architecture, and without the use of semantic labels. We can visualize shared concepts directly due to generative models included in our analysis. The Rosetta Neurons facilitate model-to-model translation enabling various inversion-based manipulations, including cross-class alignments, shifting, zooming, and more, without the need for specialized training.
Partial Simulation Extrapolation: A Proposal for Building Safer Simulators
by lukemarks — 2023-06-17T13:55:03.438Z

Introduction

This document outlines a proposal for building safer simulators/predictive models, hereafter ‘Partial Simulation Extrapolation’ (PSE). Without requiring Nice Things like flawless interpretability tools or undiscovered theoretical results, a successful implementation of PSE could help prevent outer alignment failures in simulators (namely the prediction of misaligned systems). Moreover, I suggest PSE will enforce a minimal alignment tax and can be considered ‘self-contained’ in that application to already trained (as well as yet to be trained) models might be possible without architectural or training data modifications. The caveat is that even a completely ideal implementation doesn’t provide a theoretical safety guarantee, and only addresses select outer alignment failures.

As a primer, I recommend reading the preceding post

Higher Dimension Cartesian Objects and Aligning ‘Tiling Simulators’

, and if you are interested in the math component of the theory, also the

introductory post to Scott Garrabrants Cartesian Frames sequence

and section one and two of MIRI’s

Tiling Agents for Self-Modifying AI, and the Löbian Obstacle

(although neither of these will be necessary to obtain a conceptual grasp). Whilst the previous post focused quite heavily on the mathematical side of Cartesian objects, I intend to make this proposal as accessible as possible, and will then associate its claims with more rigorous theory in later posts.

Recap

This is roughly what a Cartesian object looks like:

Where the axis A1 represents an agent, which we formalize as a set of actions that agent can take. A2 is much the same but for a different agent, and E is the environment (which in accordance with the Cartesian frame paradigm, can also be modeled as a set of actions except taken by the environment in place of an agent). The twenty-seven translucent cubes represent potential worlds. In theory a Cartesian object could have any number n∈N of dimensions, and the above is just one potential configuration used as a toy model to introduce the concept. Most models will involve far more than three actions per axis and many more than two agents.

Quoting Higher Dimension Cartesian Objects and Aligning ‘Tiling Simulators’:

To make Cartesian objects useful for thinking about simulators, do the following:

Imagine a box. This

...
Alewife Train is Now Arriving
by jefftk — 2023-06-17T13:20:02.499Z

A few months ago

I
complained that the text version of the MBTA train arrival
announcements include a not-needed-for-text "attention passengers"
which delayed the key information to the second screen. I'm happy to
report that they've fixed it!

While the vocal announcement is still "Attention passengers, the next
Red Line train to Alewife is now arriving", the text version is the
simple "Alewife train is now arriving". They've only fixed it for the
second of the two announcements, when the train is arriving instead of
when it's approaching, but it's still great!

(They actually changed this a few months ago, mid-April,
but at the time I was waiting to see if they would fix it for the
other one, and then forgot to write about it.)
Are Bayesian methods guaranteed to overfit?
by Ege Erdil — 2023-06-17T12:52:43.987Z
Link: https://www.yulingyao.com/blog/2023/overfit/

Yuling Yao argues that Bayesian models are guaranteed to overfit. He summarizes his point as follows:

I have a different view. Bayesian model does overfit.

Moreover, Bayes is guaranteed to overfit, regardless of the model (being correct or wrong) or the prior ( “strong” or uninformative).

Moreover, Bayes is guaranteed to overfit on every realization of training data, not just in expectation.

Moreover, Bayes is guaranteed to overfit on every single point of the training data, not just in the summation.

He uses the following definition of "overfitting": a model "overfits" some data if its out-of-sample log loss exceeds its within-sample log loss. Interpreted in a different way, this is equivalent to saying that the model assigns higher probability to a data point after updating on it than before. Andrew Gelman makes the point that any proper fitting procedure whatsoever has this property, and alternative methods "overfit" more than ideal Bayesian methods.

I think the proper way to interpret the results is not that Bayesian methods are guaranteed to overfit but that the definition of "overfitting" used by Yuling Yao, while intuitively plausible at first glance, is actually poor. Still, proving the fact that Bayesian methods indeed must "overfit" in his sense is an interesting exercise. I tried understanding his derivation of this and gave up - I present an original derivation of the same fact below that I hope is clearer.

Derivation

Suppose we have a model parametrized by parameters θ and the probability of seeing some data y according to our model is P(y|θ). Now, suppose we draw n independent samples y1,y2,…,yn. Denote this whole data vector by y, and denote the data vector with the ith sample omitted by y−i. Under Bayesian inference, the within-sample probability of observing the value yi in the next sample we draw is

P(yn+1=yi|y)=∫θP(θ|y)P(yi|θ)dθ

On the other hand, Bayes says that

P(θ|y)=P(θ|y−i,yi)=P(θ|y−i)P(yi|y−i,θ)P(yi|y−i)=P(θ|y−i)P(yi|θ)P(yi|y−i)

Plugging in gives

P(yn+1=yi|y)=∫θP(θ|y−i)P(yi|θ)2P(yi|y−i)dθ

or

P(yn+1=yi|y)P(yi|y−i)=Eθ∼P(θ|y−i)[P(yi|θ)2]

We can decompose the expectation of the squared probability on the right hand side using the definition of variance as follows:

P(yn+1=yi|y)P(yi|y−i)=Eθ∼P(θ|y−i)[P(yi|θ)]2+varθ∼P(θ|y−i)(P(yi|θ))=P(yi|y−i)2+varθ∼P(θ|y−i)(P(yi|θ))

where I've used the fact that

Eθ∼P(θ|y−i)[P(yi|θ)]=∫θP(θ|y−i)P(yi|θ)dθ=P(yi|y−i)

to get r...
The AI governance gaps in developing countries
by ntran — 2023-06-17T02:50:43.599Z

As developed countries rapidly become more equipped in the governance of safe and beneficial AI systems, developing countries are slackened off in the global AI race and standing at risk of extreme vulnerabilities. By examining not only “how we can effectively govern AI” but also “who has the power to govern AI”, this article will make a case against AI-accelerated forms of exploitation in low- and middle-income countries, highlight the need for AI governance in highly vulnerable countries, and propose ways to mitigate risks of AI-driven hegemons.

This report was written as a custom case of the AI governance hackathon[1].

Executive Summary

Advanced AI may accelerate forms of exploitation e.g. surveillance capitalism[2] and data colonialism, making low- and middle-income countries the most vulnerable in the process of AI development and adoption.
Developing countries may face unique challenges, such as a lack of technological capability and governance frameworks, but they are also vulnerable to the potential negative impacts of AI systems exacerbating existing power structures, infrastructures, and inequalities.
To address these challenges, there is a need for increased transparency, regulation, and coordination between countries at both the global and local levels.
This includes incentivizing companies to prioritize transparency in their engagement with foreign countries, contextualizing international initiatives and guidelines to national AI policies, and mobilizing resources to enhance global equity in AI governance.

Introduction

With the exponential growth of AI systems and the potential emergence of superintelligence, our “risk society” preoccupied with the potential of a safe future[3], is urged to figure out how to navigate ourselves in this continuously expanding black box. Thus, the coordination and response of the global political economy to this important time of human history will shape the state of the world and impact future generations to come. As developed countries rapidly become more equipped in the governance of safe and beneficial AI systems, developing countries are slackened off in the global AI race and standing at risk of extreme vulnerabilities.

This article brings attention to the impact of transformative artificial intelligence on developing countries and where they stand in the rising AI race dynamics. It also raises questions about whether...
June and Mulberries
by jefftk — 2023-06-17T01:30:01.274Z

Around here the

Juneberries and

Mulberries
are ripe.

The berries are delicious, but a lot of people don't know that they're
safe to eat.

I like the mulberries best when they still have a hint of red, before
they get too sweet.

We've been walking over to the neighborhood tree with a stepladder
every other day or so and enjoying them.

Comment via: facebook, mastodon
Updating Drexler's CAIS model
by Matthew Barnett — 2023-06-16T22:53:58.140Z

Eric Drexler's report Reframing Superintelligence: Comprehensive AI Services (CAIS) as General Intelligence reshaped how a lot of people think about AI (summary 1, summary 2). I still agree with many parts of it, perhaps even the core elements of the model. However, after looking back on it more than four years later, I think the general picture it gave missed some crucial details about how AI will go.

The problem seems to be that his report neglected a discussion of foundation models, which I think have transformed how we should think about AI services and specialization.

The general vibe I got from CAIS (which may not have been Drexler's intention) was something like the following picture:

For each task in the economy, we will train a model from scratch to automate the task, using the minimum compute necessary to train an AI to do well on the task. Over time, the fraction of tasks automated will slowly expand like a wave, starting with the tasks that are cheapest to automate computationally, and ending with the most expensive tasks. At some point, automation will be so widespread that it will begin to meaningfully feed into itself, increasing AI R&D, and accelerating the rate of technological progress.

The problem with this approach to automation is that it's extremely wasteful to train models from scratch for each task. It might make sense when training budgets are tiny — as they mostly were in 2018 — but it doesn't make sense when it takes 10^25 FLOP to reach adequate performance on a given set of tasks.

The big obvious-in-hindsight idea that we've gotten over the last several years is that, instead of training from scratch for each new task, we'll train train a foundation model on some general distribution, which can then be fine-tuned using small amounts of compute to perform well on any task. In the CAIS model, "general intelligence" is just the name we can give to the collection of all AI services in the economy. In this new paradigm, "general intelligence" refers to the fact that sufficiently large foundation models can efficiently transfer their knowledge to obtain high performance on almost any downstream task, which is pretty closely analogous to what humans do to take over jobs.

The fact that generalist models can be efficiently adapted to perform well on almost any task is an incredibly important fact about our world, because it impli...
Avoiding metaphysics means giving bad philosophy a free pass
by Aditya — 2023-06-16T20:54:33.799Z

I have noticed a culture among AI Alignment field builders, a certain reluctance to let conversations lead to metaphysical underpinnings. The framing and unsaid assumptions beneath models that predict the existential risk from AI is not explained, rather selection pressures strongly favor people who already grokked these ideas.

So I worry that this is leading to a homogeneity in the kind of people that are surviving the funnel to reach the point where they start contributing to the community - by deconfusing concepts, starting new research directions, communicating the risk to others etc. It seems worthwhile at this point to try and increase variance here.

This current strategy is totally understandable because people do not react favorably to their metaphysics being challenged. Philosophy needs careful rigorous thinking, it is extremely easy to get mired up in discussions that lead nowhere, with ill defined words thrown around regarding non falsifiable questions.

So most people choose to ignore these questions about consciousness, free will, qualia, sentience, etc and try to stick to the standard pitch. The standard pitch involves extrapolating that generally capable systems do not need any of these human-like properties to be dangerous. The bar is quite low to logically make a case for x risk from intelligent systems that mechanically follow goals.

But logic is not sufficient here, there are emotional knots to untangle, there are cognitive defense mechanisms in all of us which protect our sanity. These series of compromises that our psyche has set up to compartmentalize inconsistent beliefs need to be carefully undone, it is especially important to have empathy when spreading this meme.

All of our axioms are based on intuitions. The initial friction that I see when trying to communicate the existential threat from generally intelligent AI systems is that people have strong priors on the inherent limitations of such “software” running on electronics. They put AI systems into a different category than humans.

So I try to poke at these beliefs and try to understand how they are so confidently brushing away concerns about agentic systems and instead focusing on the risks of narrowly intelligent tools being misused by human actors. It turns out that is a more concrete narrative which is easier to hold in their heads.

&nb...