trentmkelly/LessWrong-43k huggingface dataset:
How should we compare neural network representations?
by jsteinhardt — 2021-11-05T22:10:18.677Z

Cross-posted from the BAIR Blog.

To understand neural networks, researchers often use similarity metrics to measure how similar or different two neural networks are to each other. For instance, they are used to compare vision transformers to convnets [1], to understand transfer learning [2], and to explain the success of standard training practices for deep models [3]. Below is an example visualization using similarity metrics; specifically we use the popular CKA similarity metric (introduced in [4]) to compare two transformer models across different layers:

Figure 1. CKA (Centered Kernel Alignment) similarity between two networks trained identically except for random initialization. Lower values (darker colors) are more similar. CKA suggests that the two networks have similar representations.

Unfortunately, there isn't much agreement on which particular similarity metric to use. Here's the exact same figure, but produced using the Canonical Correlation Analysis (CCA) metric instead of CKA:

Figure 2. CCA (Canonical Correlation Analysis) similarity between the same two networks. CCA distances suggest that the two networks learn somewhat different representations, especially at later layerss.

In the literature, researchers often propose new metrics and justify them based on intuitive desiderata that were missing from previous metrics. For example, Morcos et al. motivate CCA by arguing that similarity metrics should be invariant to invertible linear transformations [5]. Kornblith et al. disagree about which invariances a similarity metric should have, and instead argue that metrics should pass an intuitive test - given two trained networks with the same architecture but different initialization, layers at the same depth should be most similar to each other - and their proposed metric, CKA, performs the best on their test [4].

Our paper, Grounding Representation Similarity with Statistical Testing, argues against this practice. To start, we show that by choosing different intuitive tests, we can make any method look good. CKA does well on a "specificity test" similar to the one proposed by Kornblith et al., but it does poorly on a "sensitivity test" that CCA shines on.

To move beyond intuitive tests, our paper provides a carefully-designed quantitative benchmark for evaluting similarity metrics. The basic idea is that a good similarity metric should correlate w...
Drug addicts and deceptively aligned agents - a comparative analysis
by Jan — 2021-11-05T21:42:48.993Z
Link: https://universalprior.substack.com/p/drug-addicts-and-deceptively-aligned

Co-authored by Nadia Mir-Montazeri and Jan Kirchner.

Addicts and AIs

A young man, let’s call him Dave[1], starts consuming different kinds of illegal drugs (mostly heroin) as early as age 14, as a reaction to the divorce of his parents. Even though he is from an otherwise stable family, he becomes homeless after fights about his drug consumption. Repeatedly, Dave returns to his family's home, crying and asking to sleep on the couch for a few nights. Repeatedly, the family takes him in, believing that this time he finally has a change of heart. And time and time again, Dave will disappear after a few days, taking with him all money or any other valuable thing that is not nailed down.

As a doctor on the psych ward, I (Nadia) have encountered situations like this over and over. The first few times as a health professional, you might get fooled by a patient with addiction. But soon, you learn to be skeptical and stop paying attention to what they say they did and how they appeal to your empathy. Instead, you start to pay close attention to what you can observe people doing (and what others tell you they did). Luckily, most doctors are smarter than addicts, especially after the addict’s cognitive ability starts to decline due to substance abuse. Still, treating addicts requires constant vigilance and attention to details and potential contradictions. This is a sad state of affairs, but the simple fact that “addicts lie”[2] has become one of the central axioms of psychiatric practice.

Intriguingly, a very similar situation has emerged in AI Safety[3], albeit from a very different direction. One of the central intuition pumps of AI Safety is that of the single-minded, utility-maximizing artificial agent, whose actions end up being harmful due to a misalignment of values. While (luckily!) no pure version of this agent exists yet, theoretical studies allow us to make inferences about properties that such an agent is likely to have. This includes f.e. the properties incentivized by instrumental convergence or the property of nontrivial inner alignment. However, theory is theory, and some have questioned the likelihood of instrumental convergence actually occurring. Examples of instrumental convergence remain “hypothetical” and we only know of one reported example of inner alignment failure in a computer model. We believe that this is problematic, as reality is complex a...
Modeling the impact of safety agendas
by Ben Cottier — 2021-11-05T19:46:04.656Z

This post, which deals with how safety research - that is, technical research agendas aiming to reduce AI existential risk - might impact risks from AI, is part 6 in our sequence on Modeling Transformative AI Risk. In this series of posts, we are presenting a preliminary model of the relationships between key hypotheses in debates about catastrophic risks from AI. Previous posts in this sequence explained how different subtopics of this project, such as AI takeoff and mesa-optimization, are incorporated into our model.

We caution that this part of the model is much more of a work in progress than others. At present, it is best described as loosely modeling a few aspects of safety agendas, and the hope is that it can be further developed to be similar quality to more complete portions of the model. In many cases, we are unclear about how different research agendas relate to specific types and causes of risk. While this is partly because this part of the model is a work in progress, it is also because the theory of impact for many safety agendas is still unclear. So in addition to explaining the model, we will highlight things that are unclear and what would clarify those things.

Modeling of different research areas is contained in the green-colored modules, circled below.

Key questions for safety agendas

We encountered several uncertainties about safety agendas which have made modeling them relatively difficult. While most of these uncertainties are explained throughout the next section, the following points seem like the most important questions to ask about a safety agenda:

What is the theory of change? What does success look like, and how does that reduce AI risk? What aspects of alignment is the agenda supposed to address?

What is assumed about the progression of AI? How much does the agenda rely on a particular AI paradigm?

What is the expected timeline of the research agenda, and how much does that depend on additional funding or buy-in?

What are likely effects from work on this agenda even if it doesn’t succeed fully? Are there spillover effects for AI capabilities or for other safety agendas? Are there beneficial effects from partial success?

Besides just helping our model, it's worth us highlighting all the benefits of understanding the above for various safety agendas:

the community can better understand what constitutes success and provide better

...
Some Remarks on Regulator Theorems No One Asked For
by Past Account — 2021-11-05T19:33:22.774Z

Epistemological Status: Correct, but lazily written.

Basically, a while ago John made a post attempting to fix the Good Regulator Theorem. I've refrained from writing this simply because I know it may come off as "why does anyone care isn't this trivial"? However, more recently John has set up a research program centered around creating more results of this type so perhaps it's worth introducing a new opinion.

I guess I'm also slowly becoming a cynical grad student. Since I'm at UIUC and Conant/Ashby were affiliated I simply went around and asked a few ECE friends about the internal model principle. While it's a skewed sample, the friend and teacher I know didn't recognize it. When I explained it further, my teacher told me the result is trivial and that's probably why no one knows about it.

I personally felt like the result is obviously just a corollary of the minimal map theorem. The internal model principle is a special case. There is no contradiction between the internal model principle and the existence of model-free RL because we only care about modeling outcomes. I'll spend the rest of this post trying to articulate these intuitions. While I'm going to use some math, I personally didn't think these results justified rigor so you may notice holes in some of the reasoning.

I'll suppose we have some sort of decision process that depends on history arbitrarily. I'll abstract most of that away and propose something like,
s0→πs∞
We'll take in an initial state and then our agent will propose a policy $\pi $ that acts in the environment as a function of its observation history. This ultimately will yield some terminal state. Specifically, we'll suppose the terminal state is equivalent to a finite collection of outcome features that can be used for evaluation using a function u:S∞→[n]. Note, I've restricted to a finite outcome space out of laziness, not necessity.

Minimal Model Lemma: Up to isomorphism, any information in collectively observing the outcome of χQ=⟨χQ1(o),…,χQn(o)⟩ for a given o∈O is equivalent to P(χQ(o)|o). Moreover, this measure is minimal in the sense that any other representation gQ(o) is either equivalent in information content or lossy.

This is a simple corollary of the Minimal Map Lemma so I'm not going to prove this here. This is just an alternative way of saying that probability is a minimal sufficient statistic for describing outcomes. Whe...
Summary of the sequences / Lesson plans for rationality
by Space L Clottey — 2021-11-05T17:22:49.439Z

Does anyone know where I can find a summary of the sequences, at any level of quality? (Has anyone made their own they're willing to share?)

Or any materials prepared to help teach the sequences or similar content to an unfamiliar audience? I am starting a rationality society for high schoolers and would love if I could save time.
[External Event] 2022 IEEE International Conference on Assured Autonomy (ICAA) - submission deadline extended
by Aryeh Englander — 2021-11-05T15:29:58.224Z

This is a linkpost for https://iaa.jhu.edu/icaa/

This conference may be of interest to some, since the subject matter overlaps very heavily with AI safety research (see Topics of Interest below). The deadline for paper submissions has been extended to this coming Monday November 8.

Important Dates:

Paper submission deadline: 11/08/2021 (Anywhere on Earth)
Acceptance notification: 12/06/2021
Publication-ready Papers Due: 01/06/2022
Conference: March 22 – 24, 2022, Puerto Rico (Hybrid)

Overview:

The IEEE International Conference on Assured Autonomy (ICAA) plans to address the gap that exists between theory-heavy autonomous systems and algorithms and the privacy, security, and safety of their real-world implementations. Advances in machine learning and artificial intelligence have shown great promise in automating complex decision-making processes across transportation, critical infrastructure, and cyber infrastructure domains. Practical implementations of these algorithms require significant systems engineering and integration support, especially as they integrate with the physical world. This integration is wrought with artificial intelligence (AI) safety, security, and privacy issues.

The primary focus of this conference is the: (1) detection of, (2) response to, and (3) recovery from AI safety, security, and privacy violations in autonomous systems. Key technical challenges include discriminating between application-layer data breaches and benign process noises, responding to breaches and failures in real-time systems, and recovering from decision making failures autonomously.

Topics of Interest:

ICAA seeks contributions on all aspects of AI safety, security, and privacy in autonomous systems. Papers that encourage the discussion and exchange of experimental and theoretical results, novel designs, and works in progress are preferred. Topics of interest include (but are not limited to):

Autonomous System and AI Safety
Detecting dataset anomalies that lead to unsafe AI decisions
Evaluating safety of autonomous systems according to their potential risks and vulnerabilities
Resilient, explainable deep learning, and interpretable machine learning
Verification, testing and acceptance of machine learning models and autonomous systems
Autonomic computing for autonomous system safety
Standards, ethics, and policies for autonomy and AI safety
Ethics of autonomous system behaviors

...
Disagreeables and Assessors: Two Intellectual Archetypes
by ozziegooen — 2021-11-05T09:05:07.056Z
Link: https://forum.effectivealtruism.org/posts/PwH5hKfAdwYGE8Jdh/disagreables-and-assessors-two-intellectual-archetypes

Epistemic status: uncertain. I removed most weasel words for clarity, but that doesn't mean I'm very confident. Don’t take this too literally. I'd love to see a cluster analysis to see if there's actually a trend, this is a rough guess.[1] I'm interested in feedback on this; if it matches the intuitions of others reading this, and if there are important aspects I've missed or really messed up.

I separate a lot of interesting intellectuals into disagreeables and assessors.[2]

Disagreeables are highly disagreeable. They're quick to call out bullshit and are excellent at coming up with innovative ideas. Unfortunately, they produce a whole lot of false positives; they're pretty overconfident and wrong a great deal of the time. They're "idea detectors" with the sensitivity turned up to 11.

Disagreeables often either work alone or on top of a structure that believes (almost) everything they say.

Assessors are well-calibrated. If/when they participate in forecasting tournaments, they do really well. Assessors don't try to come up with many brilliant new ideas and don't spend particularly much effort questioning the most deeply held group assumptions, but they're awfully good at not being wrong. When they say X is true, X is indeed very likely to be true.

Both disagreeables and assessors are frustrated by mainstream epistemics, but for different reasons.

Disagreeables tend to make dramatic statements like,

“There’s a big conspiracy that everyone is in on”
“Everyone is just signaling all the time, they’re not even trying to be right”
“This organization is a huge Ponzi scheme”

Assessors would make more calm clarifications like,

“Yea, disagreeable person said X is a huge Ponzi scheme. There’s some truth there, but it’s a big oversimplification”
“I’m quite sure that Y’s paper is unlikely to replicate, after closely looking at the related papers”

If the emperor really has no clothes, disagreeables would be the first to (loudly) call that out. If the emperor has a large set of broadly reasonable policies, but a few that are subtly mistaken, the assessors would be a better fit to diligently identify and explain these.

Some disagreeables include Socrates, Nietzsche, Wittgenstein, Nassim Taleb, Robin Hanson, early Steve Jobs, other tech founders, angry public figures, professional gurus of all kinds.

Assessors include David Hume[3], Bertrand Russell, Robert Caro, Scott Alexander, Superfore...
Y2K: Successful Practice for AI Alignment
by Darmani — 2021-11-05T06:09:18.591Z

This is a linkpost for Martyn Thomas's What Really Happened in Y2K and its associated talk recording.

Some spoke of a terrible catastrophe coming inevitably.

They said the death toll could be in the millions, if not higher.

An alarm was raised, but few paid attention.

But soon it became too big to ignore.

It felt like the entire world was focused on the problem.

And then....nothing happened.

Or so they say.

Y2K often gets portrayed as a false crisis turned into a boondoggle. But, but in truth quite a lot of critical infrastructure failed due to the bug – and we should be glad it got caught in testing.

My career is in programming tools and software maintenance, and one of my mentors built program transformation tools in that era. I've known for a long time that Y2K was a real problem – and one that didn't end in 2000, with many companies implementing hacks to put it off by 20 years.

Today, I watched Martyn Thomas's excellent talk What Really Happened in Y2K . Thomas knows what he's talking about: he was on the ground as the leader of consulting teams addressing Y2K, and is now a professor of software engineering. Never have I seen such a clear and detailed description of the many things that almost went wrong.

As I watched, something struck me:

Like many others here, I watched in March 2020 as COVID presented humanity a practice run in global coordination to avoid scientific catastrophe, and humanity received a solid F. Yet Y2K is an example of humanity succeeding against a similarly inevitable catastrophe, with the IY2KCC (International Y2K Coordination Committee) recommending sound countermeasures in exactly the way the WHO didn't. As I and many readers are concerned that humanity may soon face another international coordination problem in the race for AGI, this story serves as an example of history that we should wish to repeat.

And we should ask: What made Y2K different from COVID? Could it be just that it wasn't politicized? I was not reading blog posts op-eds in 1999 and don't know how strong the Y2K naysayers were, so I'm not even sure that's true.
Below are highlighted quotes from the article. Let's hope that preventing unaligned AGI is more like Y2K and less like COVID.

There were early warnings

Some early failures helped to raise the alarm. It was said that sometime in the late 1980s the UK supermarket Marks and Spencer rejected a deli

...
Covid 11/4: The After Times
by Zvi — 2021-11-04T15:20:01.362Z

The hope is that the pandemic will be fully and truly over because case counts will be sufficiently low, and vaccination rates sufficiently high, that we can all agree to move on and resume our lives.

The fear is that this will never happen. Either cases will climb back up and be sufficiently high to justify a continued emergency state, or they won’t but people will react in a nonsensical and disproportionate way to a tiny risk, forever damaging or even destroying much of our way of life.

At this point, that potential reaction is the true risk factor. Children as young as five can be vaccinated, and anyone who wants one can effectively get a booster shot. There’s no risk left in the room that is different from many other background risks we all take every day.

Meanwhile, case counts stopped declining this week outside of the South, so the strategy of ‘wait until cases are much lower’ is looking like a less promising strategy than it did before.

For you, in your life, outside of official meddling, the pandemic is over for you, if and when you decide it is over.

If you want them to, your After Times can start today.

As far as my personal life is concerned, the After Times started last week. Pandemic over. I’ll still have to flash my vaccination card and toggle my mask on and off as required, but that’s all for show. It’s over. If my building keeps requiring masks and keeps refusing to let delivery people go upstairs, that’s annoying, but so is it when one’s day sadly requires pants.

This past week I dove a bunch into the logistical situation at the Port of Long Beach, and the Tweetstorm that helped change the container stacking rule in the city. That first post was my most widely read post ever by a wide margin. I wrote a follow-up, and notice that the logistics issues seem urgent in a way that Covid issues increasingly do not seem urgent. Perhaps I can continue to work on transitioning away from a Covid focus towards a focus on things that now matter more, on a variety of fronts. The more of these posts I can keep short, the better.

Executive Summary

Child vaccinations for ages 5-11 good to go.
Case counts may no longer be declining.
Vaccine mandate compliance very high when mandates are actually enforced.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 400k cases (-9%) and 8,600 deaths (-10%).

Results: 442,620 ca...
Baby Sister Numbers
by jefftk — 2021-11-04T01:10:09.551Z

A few days ago, Lily (7y) told me about some

Nora-inspired numbers:

The largest number is Noranoo.

If you try and make any larger number, you still get
Noranoo. For example, Noranoo + 1 =
Noranoo, and Noranoo * 2 =
Noranoo.

Otherwise, it behaves normally. You can have
Noranoo - 1, dubbed
"Norklet". This means Noranoo - 1 +
1 = Noranoo, while Noranoo + 1 -
1 = Norklet. This didn't bother her.

Noranoo * -1 is Norahats.
It is the smallest number, and like Noranoo any
attempt to go lower keeps you at Norahats.

These are very large numbers: much bigger than a googol.

This is a kind of saturation
arithmetic, more of a computersy approach than a mathy one, since
you give up associativity, distributivity, the successor function
being an injection, and all that.

On the other hand, it's slightly more elegant than a typical
computational implementation of saturation, because it is symmetric
around zero. Normally, you are using some number of bits, which gives
you 2^N distinct values, and so an even number of integers. Typically
we set the minimum integer to be one larger, in absolute value, than
the maximum one. In this case, though, there are an odd number of integers.
I asked whether perhaps Norahats * -1 * -1 * -1
could be Norklet and not
Noranoo, but Lily insisted that
Noranoo and Norahats were
equal in magnitude.

Comment via: facebook