trentmkelly/LessWrong-43k huggingface dataset:
When to mention irrelevant accusations?
by philh — 2023-01-14T21:58:08.015Z

I expect in the near-ish future to publish a thing. That thing references an idea that someone had and published on LW. I believe (having seen a credible accusation) that person has acted unethically and illegally, in ways irrelevant to the idea.

I currently intend to mention this belief, either in a footnote or a comment, along the lines of "this isn't relevant but I believe _" (and link to why I believe that). But I feel like doing so has the potential to stir up drama, and have a bunch of people telling me I shouldn't have said anything?

(Maybe relevant: in a kinda-similar situation, I recently had someone tell me I shouldn't have brought something up. But the situations were different in that I thought the thing clearly was relevant, that time, and bringing it up was clearly reasonable of me. It was frustrating, and I might have over-updated on how likely that kind of thing is to happen.)

So in the spirit of why I'm punching you, here's a space for

Me to explain my current thinking, which I'll do in an answer;

Others to give their own thoughts, and perhaps persuade me that I shouldn't say anything.

Assuming I do mention the thing, I can link here when I do and hopefully avoid the drama that might not have happened anyway.

Note that a blanket "do/don't mention", based on what I've said so far, feels unlikely to be very helpful here. I think there are probably situations compatible with what I describe where mentioning is good, and situations compatible with what I describe where mentioning is not good. The question is how we decide which is which.
World-Model Interpretability Is All We Need
by Thane Ruthenis — 2023-01-14T19:37:14.707Z

Summary, by sections:

Perfect world-model interpretability seems both sufficient for robust alignment (via a decent variety of approaches) and realistically attainable (compared to "perfect interpretability" in general, i. e. insight into AIs' heuristics, goals, and thoughts as well). Main arguments: the NAH + internal interfaces.
There's plenty of reasons to think that world-models would converge towards satisfying a lot of nice desiderata: they'd be represented as a separate module in AI cognitive architecture, and that module would consists of many consistently-formatted sub-modules representing recognizable-to-us concepts. Said "consistent formatting" may allow us to, in a certain sense, interpret the entire world-model in one fell swoop.
We already have some rough ideas on how the data in world-models would be formatted, courtesy of the NAH. I also offer some rough speculations on possible higher-level organizing principles.
This avenue of research also seems very tractable. It can be approached from a wide variety of directions, and should be, to an extent, decently factorizable. Optimistically, it may constitute a relatively straight path from here to a "minimum viable product" for alignment, even in words where alignment is really hard[1].

1. Introduction

1A. Why Aim For This?

Imagine that we develop interpretability tools that allow us to flexibly understand and manipulate an AGI's world-model — but only its world-model. We would be able to see what the AGI knows, add or remove concepts from its mental ontology, and perhaps even use its world-model to run simulations/counterfactuals. But its thoughts and plans, and its hard-coded values and shards, would remain opaque to us. Would that be sufficient for robust alignment?

I argue it would be.

Primarily, this would solve the Pointers Problem. A central difficulty of alignment is that our values are functions of highly abstract variables, and that makes it hard to point an AI at them, instead of at easy-to-measure, shallow functions over sense-data. Cracking open a world-model would allow us to design metrics that have depth.

From there, we'd have several ways to proceed:

Fine-tune the AI to point more precisely at what we want (such as "human values" or "faithful obedience"), instead of its shallow correlates.

This would also solve the ELK, which alone can be used as a lever to solve the rest of alignment.
Alterna

...
Current AI Models Seem Sufficient for Low-Risk, Beneficial AI
by harsimony — 2023-01-14T18:55:13.691Z

Note: Mostly opinion/speculation, hoping to encourage further discussion.

Summary: Current AI models have demonstrated amazing capabilities. It seems feasible to turn them into usable products without much additional scaling. These models are probably safe, and have the potential to do a lot of good. I tentatively argue that their deployment should be encouraged, with reservations about accelerating risky AI development.

Current AI has potential

It feels like there is a new AI breakthrough every week. Driven by advances in model architecture, training, and scaling, we now have models that can write code, answer questions, control robots, make pictures, produce songs, play Minecraft, and more.

Of course, a cool demo is far from a product that can scale to billions of people; but the barriers to commercialization seem surmountable. For example, though cost of a ChatGPT query is 7x that of a Google search, but it seems feasible to bring inference costs down by an order of magnitude. In fact, there are already several companies building products based on transformer models.

I see potential for existing AI to have a large positive impact. Here are just a few possibilities:

Digital Media Creation: Diffusion models and language models can already be used to generate text, images, audio, video, 3D objects, and virtual environments. These tools can be used to allow anyone to generate high-quality digital media and augment human creativity.

Search: With all of the digital content being produced using AI we will need a way to sift through to find things people like. Language models can describe various forms of media using text, compress those descriptions, and use semantic search to find content that a particular person would enjoy.

Education: Language models can act like a tutor, teaching children any topic at an appropriate level, producing educational materials, and creating Anki-style question sets to cement understanding. A similar approach can be used to get researchers up to speed on any topic.

These are just some of the possibilities commonly discussed today; I expect people to come up with even more ingenious uses for modern AI.

Current AI seems pretty safe

Despite their impressive capabilities, it seems unlikely that existing models could or would take over the world. So far, they (mostly) seem to do what they're told. Even AI researchers with particularly short ti...
Basic Question about LLMs: how do they know what task to perform
by Garak — 2023-01-14T13:13:31.197Z

My understanding of LLMs:
I have some very basic understanding of LLM and the underlying transformer architecture. As I understand it, these LLMs are, as an analogy, basically a much-improved version of autocomplete. During training they are shown a word-sequence with some words masked out and the LLM is asked to guess what the correct word is. If it gets the word wrong the model gets adjusted to better predict the correct word the next time.
The attention mechanism is trained to give the LLM an idea which previous words are most important to pay attention to when predicting the masked word.
That’s all the LLM learns: what previous words to pay attention to and in that context to predict the next/masked word.
During inference/deployment the user prompts the LLM with some words and the model outputs a hopefully sensible response.
In some models there is a separate attention mechanism 1) attention to the user prompt 2) attention to the so far generated model output sequence.

Question:
How does the LLM know what task the prompt is asking it to do?
I could want the model to continue/extend my prompt-sentence with OR I could wand the model to answer a question.

Examples:
* Prompt: what is the best car
- If this was intended to be a question the answer might be: Toyota or Tesla - maybe used in a whole sentence.
- If this was intended to be a prompt to be continued it might be: "for big families" or "going on camping vacation"

*Prompt: Harry Potter walked into the halls of Hogwarts and guess who he found there sitting at the stairs.
- If this was intended to be a question the answer might be: Hermine - maybe used in a whole sentence.
- If this was intended to be a prompt to be continued it might be followed up by a page of Harry Potter fan fiction.
I do understand that prompt engineering is an important part of using these LLMs. But in the above examples there is no refined prompting it’s just a one sentence, unstructured text - no preceding examples (that would be few shot) or keywords like "please translate the following" given.
And even with "few shot" examples in the prompt I don’t have a good intuition what is happening inside the model that makes it do the expected task. I mean LLMs are just trained to do next word prediction. How does this understanding of the intended task still work so well? What is happening inside the LLM when it "swit...
Aligned with what?
by Program Den — 2023-01-14T10:28:10.929Z

I'm assuming there are other people (I'm a person too, honest!) up in here asking this same question, but I haven't seen them so far, and I do see all these posts about AI "alignment" and I can't help but wonder: when did we discover an objective definition of "good"?
I've already mentioned it elsewhere here, but I think Nietzsche has some good (heh) thoughts about the nature of Good and Evil, and that they are subjective concepts. As ChatGPT has to say:
Nietzsche believed that good and evil are not fixed things, but rather something that people create in their minds. He thought that people create their own sense of what is good and what is bad, and that it changes depending on the culture and time period. He also believed that people often use the idea of "good and evil" to justify their own actions and to control others. So, in simple terms, Nietzsche believed that good and evil are not real things that exist on their own, but are instead created by people's thoughts and actions.
How does "alignment" differ? Is there a definition somewhere? From what I see, it's subjective. What is the real difference between "how to do X" and "how to prevent X"? One form is good and the other not— depending on what X is? But again, perhaps I misunderstand the goal, and what exactly is being proposed be controlled.
Is information itself good or bad? Or is it how the information is used that is good or bad (and as mentioned, relatively so)?
I do not know. I do know that I'm stoked about AI, as I have been since I was smol, and as I am about all the advancements us just-above-animals make. Biased for sure.
A general comment on discussions of genetic group differences
by anonymous8101 — 2023-01-14T02:11:51.890Z

(content warning: discussion of racially motivated violence and coercion)

I wanted to share that I think it's not bad to think about the object level question of whether there are group differences in intelligence rooted in genetic differences. This is an empirical claim, and can be true or false.

My moral beliefs are pretty rooted in egalitarianism. I think as a matter of policy, but also as a matter of moral character, it is good and important to treat the experience of strangers as equally valuable, regardless of their class or race. I do not think more intelligent people are more worthy of moral consideration than less intelligent people. I think it can be complicated at the extremes, especially when considering digital people, animals, etc., but that this has little bearing on public policy when concerning existing humans.

I don't think genetic group differences in intelligence are likely to be that relevant given I have short AI timelines. If we assume longer timelines, I believe the most likely places they would be important in terms of policy would be in education and reproductive technology. Whether or not there are such differences between groups now, there could easily come to be large differences through the application of embryo selection techniques or other intelligence enhancing technologies. From an egalitarian moral framework, I suspect it would be important to subsidize this technology for disadvantaged groups or individuals so that they have the same options and opportunities as everyone else. Even if genes turn out to not be a major cause of inegalitarian outcomes today, they can definitely become a major cause in the future, if we don't exercise wisdom and thoughtfulness in how we wield these technologies. However, as I said, I don't expect this to be very significant in practice given short AI timelines.

Most importantly, from my perspective, it's important to be able to think about questions like this clearly, and so I want to encourage people to not feel constrained to avoid the question because of fear of social censure for merely thinking about them. For a reasonably well researched (not necessarily correct) discussion of the object level, see this post:

[link deleted at the author's request; see also AnonymousCommentator's note about the racial IQ gap]

I think it's important context to keep in view that some of the worst human beha...
Abstractions as morphisms between (co)algebras
by Erik Jenner — 2023-01-14T01:51:45.622Z

In a previous post, I suggested we should think of abstractions/ontologies as maps that make a certain commutative diagram commute:

The downward arrows are the abstraction, which throws away some information about the world state while keeping other parts. The horizontal arrow at the top is time evolution. The diagram says that there should be a way to "mirror" the time evolution inside the high-level model (that's the dashed arrow at the bottom). Commutativity of this diagram means that we can either do time evolution in the actual world and then abstract, or first abstract and then do time evolution in the model. If this doesn't make sense, the original post has more details and examples.

It turns out that this idea is very, very far from new.[1] I'll hopefully write a post on some related literature soon, but here I want to discuss one connection in particular: category theory, and more specifically (co)algebras of endofunctors. Using coalgebras for abstractions is an idea from bisimulations in computer science. See nlab if you just want a quick version. This post essentially just adds lots of examples and some reframing—the computer science discussions I've seen tend to be heavily focused on the special case of abstractions of transition systems.

Motivation

There are two main things the (co)algebra framework will give us:

It could be a useful way to formalize the slightly vague description of abstractions-via-commutative-diagrams from my previous post. (In particular, it addresses one open question that I'll discuss in a moment.)
The original post had a few examples for commutative diagrams of the type above, but most of them focused on deterministic time evolution. This (co)algebraic framework lets us easily generate tons of examples in a natural way, from group theory to general-purpose optimizers. This gives some support both to this specific framework but also to the more general idea of using these commutative diagrams to define "good abstractions".

I'll briefly elaborate on why the description in my earlier post was a bit vague. I had another example of the commutative diagram above with a somewhat different flavor:

The abstraction M maps from real numbers to a representation using 11 decimal digits (the details don't matter here). The horizontal arrow isn't time evolution of a physical system in this example, instead it's addition of real n...
Concrete Reasons for Hope about AI
by Zac Hatfield-Dodds — 2023-01-14T01:22:18.723Z

Recent advances in machine learning—in reinforcement learning, language modeling, image and video generation, translation and transcription models, etc.—without similarly striking safety results, have rather dampened the mood in many AI Safety circles. If I was any less concerned by extinction risks from AI, I would have finished my PhD[1] as planned before moving from Australia to SF to work at Anthropic; I believe that the situation is both urgent and important.[2]

On the other hand, despair is neither instrumentally nor terminally valuable.[3] This essay therefore lays out some concrete reasons for hope, which might help rebalance the emotional scales and offer some directions to move in.

Background: a little about Anthropic

I must emphasize here that this essay represents only my own views, and not those of my employer. I’ll try to make this clear by restricting we to actions, and using I for opinions to avoid attributing my own views to my colleagues. Please forgive any lapses of style or substance.

Anthropic’s raison d’etre is AI Safety. It was founded in early 2021, as a public benefit corporation,[4] and focuses on empirical research with advanced ML systems. I see our work as having four key pillars:

Training near-SOTA models.

This ensures that our safety work will in fact be relevant to cutting-edge systems, and we’ve found that many alignment techniques only work at large scales.[5] Understanding how capabilities emerge over model scale and training-time seems vital for safety, as a basis to proceed with care or as a source of evidence that continuing to scale capabilities would be immediately risky.

Direct alignment research.
There are many proposals for how advanced AI systems might be aligned, many of which can be tested empirically in near-SOTA (but not smaller) models today. We regularly produce the safest model we can with current techniques,[6] and characterize how it fails in order to inform research and policy. With RLHF as a solid baseline and building block, we're investigating more complicated but robust schemes such as constitutional AI, scalable supervision, and model-assisted evaluations.

Interpretability research.
Fully understanding models could let us rule out learned optimizers, deceptive misalignment, and more. Even limited insights would be incredibly valuable as an independent check on other alignment efforts, and mi

...
Negative Expertise 
by Jonas Kgomo — 2023-01-14T00:51:30.280Z
Link: https://twitter.com/jonas_kg/status/1614060219281063937

Marvin Minsky's theory of Negative Expertise: Knowledge is typically considered in positive terms ,but can also be viewed in negative terms, a negative way to seem competent is to never make mistakes. The idea is that experts can be seen as those who know what not to do

Most of human knowledge is negative.

We can only be certain about things we create for ourselves like logic and mathematics. But we can reduce the chances of making mistakes by learning two different types of knowledge: Find "islands of consistency" and their boundaries. "Rule-based-systems" can contribute to negative expertise. Minsky argues that avoiding actions that could cause trouble is often more important than taking positive measures to prevent accidents. This is similar to the "info-hazards" theory.

RL Agents are by definition negative experts.

They can be considered as "negative experts" in some sense, in that they are trained to avoid certain actions that lead to negative outcomes or penalties, but also taking specific actions that lead to positive rewards.

Fault-tolerant systems

very large neural networks could be prone to accumulate too many interconnections and become paralyzed by oscillations or instabilities. One might have to provide a variety of alternative sub-systems. Perhaps we need a call for "Insulationists" researchers. This systems should be able to shut down themselves and rehash in new forms using some tripwire

Insulationists focus on designing AI systems that are insulated from one another, while interpretability researchers focus on understanding how AI systems make decisions and making them more transparent and understandable. It would be interesting to see an appetite for this.

FunnyGPT Models: Minsky suggests that jokes, like negative expertise, serve a cognitive function of helping people to think of ideas outside the overton-window and navigate their mental "censors" in a safe way. He believes that jokes can be seen as a form of negative expertise.

GPT-3, as a language model, can be seen as a "negative expert" in the sense that it has been trained on a large dataset of text and can identify and avoid certain patterns of language through negative sampling. Curious, to what extend is this theory studied today?
Smallpox vaccines are widely available, for now
by David Hornbein — 2023-01-13T20:02:44.972Z

In response to the 2022 monkeypox outbreak, public health departments throughout the USA and Europe are offering free smallpox vaccines to the public. (The diseases are closely related and the same vaccine protects against both.) Demand for these vaccines was high when they first became available, but has since plummeted as monkeypox cases fell, and the vaccine is now very easy to get. I have no idea how long this situation will last.

Eligibility varies by city. In some cities, including most of California, the vaccine is freely available to anyone. In other cities the vaccine is available to anyone who has (or claims to have) traits associated with risk of monkeypox exposure, such as male homosexuality or sexual contact with prostitutes.

Most likely, getting this vaccine will never matter. However, if there is ever an accidental or deliberate outbreak of smallpox or a closely derived disease, then being vaccinated ahead of time will be very good. Considering the operational difficulty of mass vaccination campaigns, getting a vaccine before any outbreak will both protect yourself and also ease the vaccination bottleneck of society at large.