trentmkelly/LessWrong-43k huggingface dataset:
Research ideas (AI Interpretability & Neurosciences) for a 2-months project
by flux — 2023-01-08T15:36:12.984Z

My university (EPFL) organizes a "Summer in the Lab" internship for interested Bachelor students. The idea is to send them to a lab for a ~2-months period, so they can begin engaging with the research community and develop research skills. I have been selected for the internship and the Alexander Mathis Lab accepted me. They use a mix of neuroscience and AI/ML in their research. (e.g. their most famous paper: DeepLabCut: markerless pose estimation of user-defined body parts with deep learning)

As I am interested in studying AI interpretability later in my career, I am now looking for project ideas that combine this with neuroscience. I read a bit of research in this area, such as what is depicted in the Intro to brain-like AGI post series, or the Shard theory.

My question is: which ideas (in the theories above or any others) you think are worth diving into for this 2-months research internship ? What could help advance towards a safer future (even though it is at best a very small step)?
200 COP in MI: Image Model Interpretability
by Neel Nanda — 2023-01-08T14:53:14.681Z

This is the eighth post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started. Look up jargon in

my Mechanistic Interpretability Explainer

Motivating papers: Thread: Circuits, Multimodal Neurons in Artificial Neural Networks

Disclaimer: My area of expertise is language model interpretability, not image models - it would completely not surprise me if this section contains errors, or if there are a lot of great open problems that I’ve missed!

Background

A lot of the early work in mechanistic interpretability was focused on reverse engineering image classification models, especially

Inceptionv1 (GoogLeNet)

. This work was largely (but not entirely!) led by Chris Olah and the OpenAI interpretability team. They got a lot of fascinating results, most notably (in my opinion):

Finding a technique called

Feature Visualization

to visualize what neurons are “looking at”, essentially creating a picture that represents what most activates a given neuron

Intuitively, this technique exploits the fact that each neuron is basically a function that maps an image to a scalar (the neuron activation). Images live in a continuous space (we can vary a pixel by an infinitesimal amount to slightly change the image), so we can do gradient descent on the image to find a max activating image

Curve Circuits

- they reverse engineered a 50,000 ish parameter circuit used to form curve detecting neurons, and understood it well enough that they could hand-code the weights to the neurons, insert in these hand-coded neurons, and (mostly) recover the original performance.

Multimodal Neurons

- they looked at CLIP (a model that takes in an image and a caption and outputs a score for how well they match) and found a bunch of fascinating abstract neurons that seemed to activate on concepts and on things related to that concept - eg a Donald Trump neuron that actives on his picture and on MAGA hats, or neurons corresponding to Halloween, or anime, or Catholicism

Motivation

I think the image interpretability results are awesome, and one of the main things that convinced me that reverse engineering neural networks was even possible! But also, very few people worked on these questions! There was ...
Dangers of deference
by TsviBT — 2023-01-08T14:36:33.454Z

[Written September 02, 2022. Note: I'm likely to not respond to comments promptly.]

Sometimes people defer to other people, e.g. by believing what they say, by following orders, or by adopting intents or stances. In many cases it makes sense to defer, since other people know more than you about many things, and it's useful to share eyes and ears, and coordination and specialization are valuable, and one can "inquisitively defer" to opinions by taking them as challenges to investigate further by trying them out for oneself. But there are major issues with deferring, among which are:

Deferral-based opinions don't contain the detailed content that generated the opinions, and therefore can't direct action effectively or update on new evidence correctly.

Acting based on deferral-based opinions is discouraging because it's especially not the case that the whole of you can see why the action is good.

Acting based on deferral-based opinions to some extent removes the "meaning" of learning new information; if you're just going to defer anyway, it's sort of irrelevant to gain information, and your brain can kind of tell that, so you don't seek information as much. Deference therefore constricts the influx of new information to individuals and groups.

A group with many people deferring to others will amplify information cascades by double-triple-quadruple-counting non-deferral-based evidence.

A group with many people deferring to others will have mistakenly correlated beliefs and actions, and so will fail to explore many worthwhile possibilities.

The deferrer will copy beliefs mistakenly imputed to the deferred-to that would have explained the deferred-to's externally visible behavior. This pushes in the direction opposite to science because science is the way of making beliefs come apart from their pre-theoretical pragmatic implications.

Sometimes the deferrer, instead of imputing beliefs to the deferred-to and adopting those beliefs, will adopt the same model-free behavioral stance that the deferred-to has adopted to perform to onlookers, such as pretending to believe something while acting towards no coherent purpose other than to maintain the pretense.

If the deferred-to takes actions for PR reasons, e.g. attempting to appear from the outside to hold some belief or intent that they don't actually hold, then the PR might work on the deferrer so that the deferrer sys

...
Could evolution produce something truly aligned with its own optimization standards? What would an answer to this mean for AI alignment?
by No77e — 2023-01-08T11:04:40.642Z

If Evolution was allowed to continue undisturbed, it could, conceivably, one day, produce a pure inclusive genetic fitness maximizer by reworking our base desires. So the path would have been:

First replicator -> first brains -> base desires (roughly first reinforcement learners) -> first unaligned (by Evolution's standards) consequentialists -> aligned consequentialists by reworking base desires -> Evolution cannot optimize further because it reaches the global optimum or at least a very steep local optimum.

Does that mean that with sufficiently long training time and being careful not to produce some agent that gets stuck with an imperfectly aligned goal mid-training, we could also create, artificially, a perfectly aligned (to some goal specified by us) consequentialist?

How likely are these things to happen? Both during Evolution and during training of an ML model?

Thoughts inspired by Thou Art Godshatter.
AI psychology should ground the theories of AI consciousness and inform human-AI ethical interaction design
by Roman Leventov — 2023-01-08T06:37:54.090Z

This post is a follow-up of Buck's The case for becoming a black-box investigator of language models. Here, I want to highlight two further reasons for studying AI psychology that Buck didn't mention:

the evidence from AI psychology will be important for checking theories of AI consciousness, and
AI psychology should inform the practical design of human-AI interaction interfaces, their limitations and restrictions, rules of conduct, guidelines, etc.

AI consciousness

It is possible for neuroscientists without education in psychology to discuss human consciousness because they themselves are conscious. All people (including consciousness researchers) are at least partly psychologists because they have to deal with their own psyche and people around them throughout their everyday lives, and therefore they must have “their own” psychological theory that explains and helps them predict their own behaviour and the behaviour of others.

Therefore, the role of psychology in the study of consciousness is not evident. However, this is a methodological lapse. Zoopsychology (or, more generally, ethology), for instance, is a crucial source of data for reasoning about the consciousness in animals.

This will be very important in relation to AI. Theories of AI consciousness must be grounded in the wealth of data about AI psychology. Which must be a new field with new methods of work. The methods of AI psychology should be distinct from the methods of human psychology because of the lack of the first-person perspective that every human psychologist has, and the fact that the phenotype and the ecological niche of AI agents are so different from the human phenotype, which present completely different demands to their respective psyches. Likewise, the methods of AI psychology should be distinct from the methods of zoopsychology because we can use language for both probing AIs and receiving responses from them, whereas animals almost never can respond to zoopsychologists in language.

Interaction design

Safron, Sheikhbahaee et al. (2022) and Friston et al. (2022) have already indicated the need for the deliberate design of ecosystems of natural and artificial intelligences. Obviously, these interactions should have some guardrails, ranging from codes of conduct to hard limitations in the interaction interfaces.

For the emergent activity in these ecosystems to benefit all participants, the ...
Stop Talking to Each Other and Start Buying Things: Three Decades of Survival in the Desert of Social Media
by the gears to ascension — 2023-01-08T04:45:11.413Z
Link: https://catvalente.substack.com/p/stop-talking-to-each-other-and-start

edit: I share this post because it discusses the history of social networks getting bought out in a way that I found insightful. be aware that it is quite angry; I generally have a fairly strong "you are angry, and I am over here" response, so I am able to comfortably take in another's anger without automatically agreeing; I fairly soberly agree with the direction of anger, and it is the direction I have interest in sharing. In particular, it has a call to action at the end that I think is sweet and good. Folks in comments suggested not copying full posts (contrary to the instructions on the linkpost form?) so I edited that out; recommend reading the last quarter particularly much more, though the whole post is interesting to me.
Can Ads be GDPR Compliant?
by jefftk — 2023-01-08T02:50:01.486Z

I think the online ads ecosystem is most likely illegal in Europe, and
as more decisions come out it will become clear that it can't be
reworked to be within the bounds of the GDPR. This is a strong claim,
but before I get into backing it up here's some background on me:

I'm not a lawyer or an expert in privacy regulation; this is
something I follow because I'm interested in it.

I worked in ads until
June 2022, but I'm speaking only for myself. I don't expect to go
back into the industry.

So, how are sites not compliant?

When you visit a site in Europe, or an international site as a
European, you'll typically see a prompt like this:

In this screenshot El País is asking for permission to use
cookies and use your data to personalize ads.

Why are they asking you? A combination of two regulations:

The ePrivacy
Directive (2002) which requires the site to get your consent
before using cookies or other storage on your device unless they're
strictly necessary to
provide a service you requested.

The GDPR
(2016) which tightly limits what companies can do with your data
without your consent.

The idea is, if you click "accept" then they can say they had your
consent for all the advertising stuff they do. But I think
it's very unlikely this is compliant with the GDPR.

For example, in a recent case France's data privacy regulator CNIL
recently fined
Microsoft €60M (full
text) for a similar popup on Bing. I'm going to come back to this
decision later because it has other implications, but in paragraph 65
the CNIL ruled that their cookie banner was not collecting valid
consent because it took more clicks to refuse cookies than to accept
them.

The principle here is that for consent to be valid under the GDPR it
needs to be just as easy to give consent as it is to refuse it.

This is not widely respected today, since for most companies it's
going to be much more profitable to put up a not-really-legal banner
that heavily pushes users towards saying yes and hope they don't get
in trouble, but as the data protection agencies continue their
enforcement I think this will become less practical.

Another approach you see on a few sites is the one that Der Spiegel takes:

They offer a choice between accepting their standard ad stuff or
paying to subscribe to the site (more
details). I'm glad they're giving us...
Feature suggestion: add a 'clarity score' to posts
by LVSN — 2023-01-08T01:00:29.029Z

Add a voting mechanism for posts, beside the karma, that indicates whether you

understood the post well enough to agree or disagree with it (upvote), or
cannot understand the post with a reasonable time investment (downvote)

I suggest that this determines the post's clarity score (we could name it something else, like the 'simplicity' score).

Then, allow users to sort posts by clarity.

Especially: allow users to sort posts by clarity within a topic. That way, people can read posts in a topic in order of simplicity descending/complexity ascending. I suspect that this would make familiarization with a topic a rather linear and efficient process.
How do I better stick to a morning schedule?
by Randomized, Controlled — 2023-01-08T00:52:52.791Z

I've always struggled with keeping my morning routine to a reasonable length of time. I'm spacey and unfocused in the morning, and it can be hard to stay aware of "when" I am in the morning.

I don't want to come down too hard on this part of the day, because it also feels healthy and important, but I also want to rein it in a bit.

I'm looking for suggestions that might help with keeping my meandering mornings to a reasonable length of time. Currently they often run up to around four hours before I "clock in" and start my productive work. I'm working alone right now, so there's not a lot of outside pressure, although sometime I take morning meetings which does help with this problem.

A small part of this problem is maintaining a consistent bedtime, but that isn't the crux, I'm generally reasonably good at that part.
Protectionism will Slow the Deployment of AI
by Ben Goldhaber — 2023-01-07T20:57:11.644Z

I believe I’m more optimistic than the average LWer that regulation will be passed that slows down AI capabilities research. The current capabilities of language models threaten the interests of the politically salient professional managerial middle class, which will cause constituencies in the government to pass protectionist, anti-AI policies.

I don’t think they will be well targeted or well designed regulation from an alignment point of view, but they will nevertheless slow capabilities research and deployment and alter the strategic AI development landscape.

—

ChatGPT was a fire alarm for AI and,

contra the original post

, people have noticed. The capabilities of ChatGPT, and the accessible interface, mean many people have or will experience it. This is already causing a general reaction among the chattering classes that I’d sum up as awareness that this is cool and yet also terrifying.
The government is a combination of elected politicians and non-elected ‘civil servant types’ (bureaucrats, NGOs, the permanent government)
The government is ineffective at long term strategic thinking that maximizes the general good of the public, no argument there. However, that’s the wrong lens to view things through - from a public choice/class perspective, the government is quite responsive to certain specific interest groups.
The professional managerial class (PMCs) is a term of analysis/too-online slur[1] for the ‘class’ of professionals and managers who, in the late 2010s and 2020s America, tend to be distinguished by things like having gone to university, have a generic technocratic liberal bent, work in high status professions (tech, academia, journalism, legal, medicine parts of finance, etc.).
The government is very responsive to PMC interests; the civil service, almost by definition, entirely comes from the PMC class.

Democratic and Republican congresspeople are primarily from professional managerial backgrounds

. The donor and chattering classes also come from or are of this class.
ChatGPT is, rightfully, very scary to PMCs. Headlines about replacing writers, serving as medical counsel, lawyers; right now, these are easily tuned out because there are always these stories (there were tons of op eds seven years ago talking about political truck drivers being replaced).

However, I think this will be different: ChatGPT gives that exper

...
David Krueger on AI Alignment in Academia, Coordination and Testing Intuitions
by Michaël Trazzi — 2023-01-07T19:59:09.785Z
Link: https://theinsideview.ai/david

David Krueger is an assistant professor at the University of Cambridge and got his PhD from Mila. His research group focuses on aligning deep learning systems, but he is also interested in governance and global coordination. He does not have an AI alignment research agenda per se, and instead tries to enable his seven PhD students to drive their own research.

Below are some highlighted quotes from our conversation (available on Youtube, Spotify, Google Podcast, Apple Podcast). For the full context for each of these quotes, you can find the accompanying transcript.

On Academia and Coordination

Building A Research Team, Not Following An Agenda

"I think agenda is a very grandiose term to me. It's oftentimes, I think people who are at my level of seniority or even more senior in machine learning would say, "oh, I'm pursuing a few research directions." And they wouldn't say, "I have this big agenda." And so I think my philosophy or mentality, I should say, when I set up this group and started hiring people was like, let's get talented people. Let's get people who understand and care about the problem. Let's get people who understand machine learning. Let's put them all together and just see what happens and try and find people who I want to work with, who I think are going to be nice people to have in the group who have good personalities, pro-social, who seem to really understand and care and all that stuff." (full context)

On Coordination Between Academia And The Broader World

"There's a lack of understanding and appreciation of the perspective of people in machine learning within the existential safety community and vice versa. And I think that's really important to address, especially because I'm pretty pessimistic about the technical approaches. I don't think alignment is a problem that can be solved. I think we can do better and better. But to have it be existentially safe, the bar seems really, really high and I don't think we're going to get there. So we're going to need to have some ability to coordinate and say let's not pursue this development path or let's not deploy these kinds of systems right now. And for that, I think to have a high level of coordination around that, we're going to need to have a lot of people on board with that in academia and in the broader world. So I don't think this is a problem that we can solve just with the die hard people

...
Looking for Spanish AI Alignment Researchers
by Antb — 2023-01-07T18:52:08.375Z

I am fairly new to the AI alignment/safety community and I am looking to step up my game by getting a Ph.D. in the field. Problem is, I have not found any academic AI alignment research activity here in Spain, where I currently live.

I am aware that this community is largely based on the States and the UK, but it would be far too difficult for me to move there for a wide variety of reasons, and so I am instead trying to exhaust all local possibilities I have available before turning my life upside-down.

I am interested in just about any alignment research, but my speciality is software security, BCI and computational neuroscience. Contact me if you're interested in having a collaborator/sparring partner/mentee or simply someone to talk about AI stuff.

Let's get to making AI alignment an international movement ;)
Nothing New: Productive Reframing
by adamShimi — 2023-01-07T18:43:35.617Z
Link: https://epistemologicalvigilance.substack.com/p/nothing-new-productive-reframing

In a comment on my last post, someone deplored the tendency of LW bloggers to rediscover ideas of famous philosophers and pretend that they discovered it first. This made me think of an interesting question of epistemology: is there value in reformulating and/or rediscovering things?

A naive answer would be no— after all, we already have the knowledge. But a look at the history of science brings up many examples showing otherwise..

In physics, the most obvious case comes from the formulations of classical mechanics. Newton gave the core of the theory in his original formulation, but what gets applied in modern science an play a key part in the revolutions of quantum mechanics and quantum electrodynamics among others, are the other two formulations: Lagrangian and Hamiltonian. And what these two, particularly the Lagrangian, pushed and pioneered the least action principle in physics, but capturing every system through an action that was maximized or minimized for every trajectory. This principle is one of the core building blocks of modern physics, and came out of a pure compression and reformulation of Newtonian mechanics!

Similarly, potential energy started as a mathematical trick, the potential function of Lagrange which captured all gravitational forces acting on a system of N-bodies, which were the coefficients of its partial derivatives. Once again, there was nothing new here; even more than the Lagrangian and the action, this was a computational trick that sped up annoying calculations. And yet. And yet it’s hard to envision modern physics, even classical mechanics at all, without potential.

In modern evolutionary biology, many advances like exaptation can be brought back to Darwin’s seminal work. And yet they often reveal different aspects of the idea, clarifying and bringing new intuitions to bear that make our models of life and its constraints that much richer.

Again in theoretical computer science, reframing of the same core ideas is at the center of so many productive inquiries:

all the models of computation unified by the Church Turing thesis, all equivalent yet providing different angles on questions of computations
the non-deterministic complexity classes like NP reframed as proving/certifying classes, leading to far better intuitions and to interactive proofs and IP=PSPACE
the Curry-Howard isomorphism shows that proof-calculi like Natural Deduc

...