trentmkelly/LessWrong-43k huggingface dataset:
The Role of Transparency and Explainability in Responsible NLP
by RAMEBC78 — 2024-09-01T20:08:37.743Z

Dr. Ramesh Babu Chellappan I AI Researcher I Senior Director – Process Excellence, Thomson Reuters Corporation

(https://www.linkedin.com/in/dr-ramesh-babu-chellappan-0624848/)

As Natural Language Processing (NLP) technologies continue to advance, their applications are becoming increasingly integrated into various aspects of our lives, from virtual assistants and automated customer service to medical diagnostics and financial forecasting. However, as these systems grow more complex, so too do the challenges associated with understanding how they work. This has led to a growing emphasis on transparency and explainability in NLP models, which are critical for building trust with stakeholders, ensuring ethical use, and complying with regulatory standards. This blog explores the importance of transparency and explainability in NLP, examines popular techniques like LIME, SHAP, and attention mechanisms, and provides actionable steps for implementing these principles in AI projects.

Why Transparency and Explainability Matter in NLP

Transparency and explainability in NLP are not just buzzwords; they are foundational to the responsible development and deployment of AI systems. Here’s why they matter:

Building Trust with Stakeholders: Transparency helps stakeholders—including developers, users, and regulators—understand how an NLP model makes decisions. This understanding is crucial for building trust, particularly in high-stakes applications where decisions can significantly impact individuals and society (Rudin, 2022).
Ensuring Ethical and Fair AI: Explainable models help identify and mitigate biases, enhancing fairness and reducing the risk of harm from biased or unethical decisions. By making models more interpretable, developers can ensure that AI systems align with ethical guidelines and societal values (Kim, 2023).
Facilitating Regulatory Compliance: Regulatory bodies worldwide are increasingly mandating explainability as a requirement for AI systems, especially in sensitive sectors like finance, healthcare, and criminal justice. Explainable models help organizations comply with these regulations and avoid legal repercussions (Goodman & Flaxman, 2021).
Enhancing Model Robustness and Debugging: Understanding the inner workings of NLP models enables developers to identify potential flaws or vulnerabilities, making it easier to debug models and improve their robustnes

...
Book Review: What Even Is Gender?
by Joey Marcellino — 2024-09-01T16:09:27.773Z

I submitted this review to the 2024 ACX book review contest, but it didn't make the cut, so I'm putting it here instead for posterity.

Conspiracy theories are fun because of how they make everything fit together, and scratch the unbearable itch some of us get when there are little details of a narrative that just don’t make sense. The problem is they tend to have a few issues, like requiring one to posit expansive perfectly coordinated infosecurity, demanding inaccessible or running contrary to existing evidence, and generally making you look weird for believing them.

We can get our connecting-the-dots high while avoiding social stigma and epistemic demerits by instead foraging in the verdant jungle of “new conceptual frameworks for intractable debates.” Arguments about gender tend to devolve, not just for lack of a shared conceptual framework, but because the dominant frameworks used by both defenders and critics of gender ideology are various shades of incoherent.

To the rescue are R. A. Briggs and B. R. George, two philosophers of gender promising a new approach to thinking about gender identity and categorization with their book What Even Is Gender? I appreciate that I’m probably atypical in that my first thought when confronting a difficult conceptual problem is “I wonder what mainstream analytic philosophy has to say about this?”, but What Even Is Gender? is that rare thing: a philosophical work for a popular audience that is rigorous without sacrificing clarity (and that’s clarity by normal-human-conversation standards, not analytic philosophy standards). Let’s see what they have to say.

Why I Picked This Book

BG are writing for two primary audiences in What Even Is Gender? First are people trying to make sense of their own experience of gender, especially those who feel the existing conceptual toolbox is limited, or doesn’t exactly match up with their circumstances. The second, in their words, are:

“people who, while broadly sympathetic (or at least open) to the goals of trans inclusion and trans liberation, harbor some unease regarding the conceptual tensions, apparent contradictions, and metaphysical vagaries of the dominant rhetoric of trans politics. This sort of reader might feel the pull of some of the foundational concerns that they see raised in “gender critical” arguments, but is also trying to take their tran

...
Can a Bayesian Oracle Prevent Harm from an Agent? (Bengio et al. 2024)
by mattmacdermott — 2024-09-01T07:46:26.647Z
Link: https://yoshuabengio.org/2024/08/29/bounding-the-probability-of-harm-from-an-ai-to-create-a-guardrail/

Yoshua Bengio wrote a blogpost about a new AI safety paper by him, various collaborators, and me. I've pasted the text below, but first here are a few comments from me aimed at an AF/LW audience.

The paper is basically maths plus some toy experiments. It assumes access to a Bayesian oracle that can infer a posterior over hypotheses given data, and can also estimate probabilities for some negative outcome ("harm"). It proposes some conservative decision rules one could use to reject actions proposed by an agent, and proves probabilistic bounds on their performance under appropriate assumptions.

I expect the median reaction in these parts to be something like: ok, I'm sure there are various conservative decision rules you could apply using a Bayesian oracle, but isn't obtaining a Bayesian oracle the hard part here? Doesn't that involve various advances, e.g. solving ELK to get the harm estimates?

My answer to that is: yes, I think so. And I think Yoshua would probably agree.

Probably the main interest of this paper to people here is to provide an update on Yoshua's research plans. In particular it gives some more context on what the "guaranteed safe AI" part of his approach might look like -- design your system to do explicit Bayesian inference, and make an argument that the system is safe based on probabilistic guarantees about the behaviour of a Bayesian inference machine. This is in contrast to more hardcore approaches that want to do formal verification by model-checking. You should probably think of the ambition here as more like "a safety case involving proofs" than "a formal proof of safety".

Bounding the probability of harm from an AI to create a guardrail

Published 29 August 2024 by yoshuabengio

As we move towards more powerful AI, it becomes urgent to better understand the risks, ideally in a mathematically rigorous and quantifiable way, and use that knowledge to mitigate them. Is there a way to design powerful AI systems based on machine learning methods that would satisfy probabilistic safety guarantees, i.e., would be provably unlikely to take a harmful action?

Current AI safety evaluations and benchmarks test the AI for cases where it may behave badly, e.g., by providing answers that could yield dangerous misuse. That is useful and should be legally required with flexible regulation, but is not sufficient. These tests only tell us one side of the story: ...
Forecasting One-Shot Games
by Raemon — 2024-08-31T23:10:05.475Z

Cliff notes:

You can practice forecasting on videogames you've never played before, to develop the muscles for "decision-relevant forecasting."
Turn based videogames work best. I recommend "Luck Be a Landlord", "Battle for Polytopia", or "Into the Breach."
Each turn, make as many Fatebook predictions as you can in 5 minutes, then actually make your decision(s) for the turn.
After 3 turns, instead of making "as many predictions as possible", switch to trying to come up with at least two mutually exclusive actions you might take this turn, and come up with predictions that would inform which action to take.

Don't forget to follow this up with practicing forecasting for decisions you're making in "real life", to improve transfer learning. And, watch out for accidentally just getting yourself addicted to videogames, if you weren't already in that boat.

This is pretty fun to do in groups and makes for a good meetup, if you're into that.

Recently I published Exercise: Planmaking, Surprise Anticipation, and "Baba is You". In that exercise, you try to make a complete plan for solving a puzzle-level in a videogame, without interacting with the world (on levels where you don't know what all the objects in the environment do), and solve it on your first try.

Several people reported it pretty valuable (it was highest rated activity at my metastrategy workshop). But, it's fairly complicated as an exercise, and a single run of the exercise typically takes at least an hour (and maybe several hours) before you get feedback on whether you're "doing it right."

It'd be nice to have a way to practice decision-relevant forecasting with a faster feedback loop. I've been exploring the space of games that are interesting to "one-shot". (i.e. " try to win on your first playthrough"), and also exploring the space of exercises that take advantage of your first playthrough of a game.

So, an additional, much simpler exercise that I also like, is:

Play a turn-based game you haven't played before.
Each turn, set a 5 minute timer for making as many predictions as you can about how the game works, what new rules or considerations you might learn later. Then, a 1 minute timer for actually making your choices for what action(s) to take during the turn.

And... that's it. (to start with, anyway).

Rather that particularly focusing on "trying really hard to win", start with just m...
On epistemic autonomy
by sanyer — 2024-08-31T18:50:43.377Z

TL;DR: Epistemic autonomy refers to the degree to which you are able to change your mind without letting outside sources, such as social and financial pressure, affect it. To increase your epistemic autonomy, you can refrain from publishing your opinions, reduce your conflicts of interest, or change your environment, among other things. However, there are usually tradeoffs, and increasing epistemic autonomy might not always be worth it.

The politician

Imagine you are a politician who, in the middle of her term, experiences a genuine change of mind. Perhaps a world event or personal reflection leads you to reject a stance you once held. After adjusting your political behavior based on the newly held belief, you find that your fellow party members start to dislike you. You also find people mocking you on the internet, vowing to not vote for you anymore, since you don’t represent their views anymore.

The fast fashion retailer

Imagine you work for a fast fashion retailer. You have worked in the company for many years, and are in a high position in the company. In the past few years the public has raised concerns regarding the environment and labor exploitation, and you have also been challenged for furthering these effects. However, you continue to believe your work is valuable for society, and don’t see anything wrong with your company’s impact on the environment and workers.

Religious family

Imagine you were raised in a religious family in a small village. Everyone in the village goes to church, and the church has a lot of power over villagers’ lives. You aren’t totally isolated though, because you have a TV and internet. The people in movies don’t always practice religion and sometimes they even say heretical things. To you, these people seem weird, and you couldn’t see such people in your village.

Epistemic autonomy

In all of the above situations, there are powerful forces which push you away from reaching true beliefs autonomously. In other words, your epistemic autonomy is hampered. The concerns of humiliation, shame, and simple inconvenience could render it difficult to even start questioning your beliefs.

Maintaining and increasing your epistemic autonomy

As we are all a part of a social network, our epistemic autonomy is somehow limited. However, there are ways of increasing your epistemic autonomy.

The most obvious, but possibly not the mos

...
Epistemic states as a potential benign prior
by Tamsin Leake — 2024-08-31T18:26:14.093Z
Link: https://carado.moe/epistemic-state-benign-prior.html

Malignancy in the prior seems like a strong crux of the goal-design part of alignment to me. Whether your prior is going to be used to model:

processes in the multiverse containing the AI which does said modeling,

processes which would output all of some blog so we can make it output more of it,

processes which match an AI chatbot's hypotheses about who/what it's talking with,

then you have to sample hypotheses from somewhere; and typically, we want to use either solomonoff induction or time-penalized versions of it such as levin search (penalized by log of runtime), or the implicit prior of neural networks (large sequences of multiplying by a matrix, adding a vector, and ReLU, often with a penalty related to how many non-zero weights are used).

And the solomonoff prior is famously malign.

(Alternatively, you could have knightian uncertainty about parts of your prior that aren't nailed down enough, and then do maximin over your knightian uncertainty (like in infra-bayesianism), but then you're not guaranteed that your AI gets anywhere at all; its knightian uncertainty might remain so immense that the AI keeps picking the null action all the time because some of its knightian hypotheses still say that anything else is a bad idea. Note: I might be greatly misunderstanding knightian uncertainty!)

(It does seem plausible that doing geometric expectation over hypotheses in the prior helps "smooth things over" in some way, but I don't think this particularly removes the weight of malign hypotheses in the prior? It just allocates their steering power in a different way, which might make things less bad, but it sounds difficult to quantify.)

It does feel to me like we do want a prior for the AI to do expected value calculations over, either for prediction or for utility maximization (or quantilization or whatever).

One helpful aspect of prior-distribution-design is that, in many cases, I don't think the prior needs to contain the true hypothesis. For example, if the problem that we're using a prior for is to model

processes which match an AI chatbot's hypotheses about what it's talking with

then we don't need the AI's prior to contain a process which behaves just like the human user it's interacting with; rather, we just need the AI's prior to contain a hypothesis which:

is accurate enough to match observations.

is accurate enough to capture the fact that the user (

...
My Model of Epistemology
by adamShimi — 2024-08-31T17:01:45.472Z
Link: https://epistemologicalfascinations.substack.com/p/my-model-of-epistemology

I regularly get asked by friends and colleagues for recommendation of good resources to study epistemology. And whenever that happens, I make an internal (or external) "Eeehhh"pained sound.

For I can definitely point to books and papers and blog posts that inspired me, excited me, and shaped my world view on the topic. But there is no single resource that encapsulate my full model of this topic.

To be clear, I have tried to write that resource — my hard-drive is littered with such attempts. It's just that I always end up shelving them, because I don't have enough time, because I'm not sure exactly how to make it legible, because I haven't ironed out everything.

Well, the point of this new blog was to lower the activation energy of blog post writing, by simply sharing what I found exciting quickly. So let's try the simplest possible account I can make of my model.

And keep in mind that this is indeed a work in progress.

The Roots of Epistemology

My model of epistemology stems from two obvious facts:

The world is complex
Humans are not that smart

Taken together, these two facts mean that humans have no hope of ever tackling most problems in the world in the naive way — that is, by just simulating everything about them, in the fully reductionist ideal.

And yet human civilization has figured out how to reliably cook tasty meals, build bridges, predict the minutest behaviors of matter... So what gives?

The trick is that we shortcut these intractable computations: we exploit epistemic regularities in the world, additional structure which means that we don't need to do all the computation.[1]

As a concrete example, think about what you need to keep in mind when cooking relatively simple meals (not the most advanced of chef's meals).

You can approximate many tastes through a basic palette (sour, bitter, sweet, salty, umami), and then consider the specific touches (lemon juice vs vinegar for example, and which vinegar, changes the color of sourness you get)
You don't need to model your ingredient at the microscopic level, most of the transformations that happen are readily visible and understandable at the macro level: cutting, mixing, heating…
You don't need to consider all the possible combinations of ingredients and spices; if you know how to cook, you probably know many basic combinations of ingredients and/or spices that you can then pimp or adapt for different dishes.

All of t...
Verification methods for international AI agreements
by Orpheus16 — 2024-08-31T14:58:10.986Z
Link: https://arxiv.org/abs/2408.16074

TLDR: A new paper summarizes some verification methods for international AI agreements. See also summaries on LinkedIn and Twitter.

Several co-authors and I are currently planning some follow-up projects about verification methods. There are also at least 2 other groups planning to release reports on verification methods. If you have feedback or are interested in getting involved, please feel free to reach out.

Overview

There have been many calls for potential international agreements around the development or deployment of advanced AI. If governments become more concerned about AI risks, there might be a short window of time in which ambitious international proposals are seriously considered. If this happens, I expect many questions will be raised, such as:

Can compliance with international AI agreements be robustly verified?
What tactics could adversaries use to try to secretly develop unauthorized AI projects or unauthorized data centers?
What assumptions do various verification methods rely on? Under what circumstances could they be deployed?

Our paper attempts to get readers thinking about these questions and considering the kinds of verification methods that nations could deploy. The paper is not conclusive– its main goal is to provide some framings/concepts/descriptions/examples that can help readers orient to this space & inspire future research.

I'd be especially interested in feedback on the following questions:

New verification methods. What are verification methods that are missing from our existing list?
Evasion strategies. What kinds of things could adversaries do to hide AI development or data centers?
Technical advances. What kinds of technical advances could make verification easier or harder? (For example, distributed training could make verification harder; LLMs that could securely scan code could make verification easier.)

Abstract

What techniques can be used to verify compliance with international agreements about advanced AI development? In this paper, we examine 10 verification methods that could detect two types of potential violations: unauthorized AI training (e.g., training runs above a certain FLOP threshold) and unauthorized data centers. We divide the verification methods into three categories: (a) national technical means (methods requiring minimal or no access from suspected non-compliant nations), (b) access-dependent meth...
Fake Blog Posts as a Problem Solving Device
by silentbob — 2024-08-31T09:22:54.513Z

This is a very brief post about a simple problem solving strategy I sometimes find useful, that may be worth trying for people who have never done it.

This is the strategy:

When struggling with some difficult problem X, I often find it helpful to write a blog post titled “How I Solved X” or “How I Managed to Overcome X” from a hypothetical future perspective, explaining the path from my current situation to the solution.

It’s not necessary to ever publish this writing or show it to anyone, but I still try to at least entertain the possibility that this might happen.

Naturally it makes sense to avoid putting too much effort in the quality of the writing itself, and rather focus on the content.

Dressing up like a burglar is optional

Why?

This approach has a lot of overlap with e.g. journaling, pair debugging or coaching conversations. Some of the benefits are:

It pushes you to think systematically.
Writing things down may help avoid flinching away from difficult or uncomfortable aspects of the problem.
Targeting the writing at a hypothetical wider audience might sharpen the senses and increase the quality of reasoning.
Taking the vantage point of a future where the problem has already been overcome may unlock some creative solutions that would be otherwise hard to come up with. Somewhat like the opposite of

Murphyjitsu

.
Personally, I also find that this approach decreases my anxiety and increases motivation; it gives me a convincing feeling that the problem is indeed solvable and I’m now on track to do so. Somehow I mysteriously get this comforting sensation of not being alone in struggling with the problem.
It helps put the focus on actual solutions; to actually

cut through the problem

rather than attacking it ineffectively.
It’s free and pretty low-effort.
In some cases it may actually lead to a publishable artifact that other people may find insightful.
As a side effect, it may serve as writing practice and help overcome perfectionism.

Some Examples

Here are some things I already have or eventually might write such a fake blog post about:

Achieving my desired body weight, something I failed to do for many years
Decluttering my flat
Overcoming the problem of perpetually feeling busy and as if there’s never enough time to do all the things I’d like to do
Being able to emotionally feel secure and content in a polyamorous relationship
Growing an AI Safety discussion group I

...