trentmkelly/LessWrong-43k huggingface dataset:
Semantics, Syntax and Pragmatics of the Mind?
by Ben Amitay — 2023-04-23T06:13:31.464Z

One feature of natural languge that seem to be essential is the (blurry) distinction between 3 layers:

Semantics of basic units

Syntactic rules for how to combine the meanings of basic units.

Pragmatic considerations for modifying the literal meaning by taking into account things like context, shared priors, and cooperative game theory.

That distinction seem very natural to me, because 1+2 are needed for learnability of the language and 3 is a direct result of trying to optimize the communication in context. Hence my question:
I'm interested at whether similar 3 layers emerge in other important communication context:

Communication between modules in the brain.

Memory, when viewed as communication if the present brain with the future brain.

inter-modul communication and memory in deep learning.

Do you have evidence for/against it in those contexts?
A great talk for AI noobs (according to an AI noob)
by dov — 2023-04-23T05:34:52.772Z
Link: https://forum.effectivealtruism.org/posts/REjSEuKbzh2QRFBgK/a-great-talk-for-ai-noobs-according-to-an-ai-noob

I'm far from an expert on AI and alignment but I really liked this Ted talk and think it serves as a good introduction to the topic. Here are some highlights:

It starts with a demo of what ChatGPT is currently or soon will be capable and it's bound to shocking to plenty of noobs like me.

Then OpenAI cofounder is also pressed on the company's controversial, perhaps reckless, approach. He explains why OpenAI has often been the first to release AI products despite safety concerns (particularly at 24:17) In short:

The cofounder says that OpenAI's views programing Chatgpt like teaching a child insofar as regular servings of positive and negative reinforcement are better than trying to create a master plan ahead of time. OpenAI currently want the world's help feeding feedback into their "children" and although their view may change, Alan Turing suggested this approach in 1950 and OpenAI still doesn't see a better approach.

Also (at 20:52) I think OpenAI's cofounder claims that the company is "starting to really get good at is predicting" emergent capabilities of AI. That sounds a little too good to be true so I'm curious what yall think.

And please correct me if I'm wrong or missing something important (hopefully before I share this talk with my friends :)
Bits of NEFFA
by jefftk — 2023-04-23T02:20:04.722Z

First in-person NEFFA since 2019. Two really nice days, with a third
to come. Some bits:

Lily getting to fire a rubber chicken at Jack in the Green
(Morris)

Credit: Ezra

Couldn't go anywhere without many stops to greet people I
hadn't seen in years. So many "you're so much bigger than the last
time I saw yous" with the kids, except for one person who was very
confused and thought Nora (1y22m) was Anna (7y) and who initially
thought Anna hadn't grown since 2019.

Lily (9y) bringing her fiddle for jamming in the hallways, and
being joined by a patient piano player. We played her tunes, and then as other
people started to join us and she ran out of tunes she did a bit of
rhythm.

Playing a Friday evening contra set with Cecilia, and seeing
Mark rocking out at the top.

Getting my rhythm stage setup ready
for a a ten-minute changeover, following a ~20 person band. That is, they
stopped playing at 7:50, we started playing at 8:00, and in between
they needed to get off stage, we needed to get on stage, we needed to
hook everything up, and we needed to sound check and get monitors
working. Which we did!

Dancing the Friday contra medley with Cecilia and the Saturday
one with Julia, and then a regular contra with Lily. I wish my knees
were up for dancing more!

Anna getting to pick dandelions nearly indefinitely. After
almost an hour we needed to move to a different part of the festival
and she wanted to pick just one more, just one more.

First time seeing Harbour Steel
perform!

Nora was often the center of attention, though she was iffy
about this.

Lots of time with Brattleboro cousins, which was especially
important to Anna (really misses having her cousin Olivia close).

The cousins picking out what toppings they wanted on their
blintzes. Turns out it was "all of them".

Nora has gotten very fast on her boot; so much chasing her around!

This is far from everything, but now I need to go to sleep so I can go
back again tomorrow.

Comment via: facebook, mastodon
"Rate limiting" as a mod tool
by Raemon — 2023-04-23T00:42:58.233Z

A lot of my recent thoughts on moderation have included an emphasis on "rate limiting as a primary mod tool." This is very much in the "Ray has a hypothesis and it might not work out" phase, but I think it has promise.

The general principle is "moderators should be able to employ the least-amount-of-constraint/force possible to enforce a decision or give that decision teeth." A full ban is a pretty extreme option, and being limited to that option makes me much more hesitant to take any kind of action against a person.

A rate limit is a pretty flexible tool. You can rate limit comments, posts, or (possibly) wordcount. You can limit by hour, day, week or more. It can be limited to a specific post, or across posts in general (i.e. N comments per post per unit-of-time), or sitewide (N comments total per unit of time)

But at least some people have reported experiencing limits as very costly, and have described it as "being banned." Ruby has argued it might basically just feel like moderators are trying to ban people without really acknowledging the magnitude of what they're doing.

I'm curious how people feel about it. I want to outline a bit how I'm thinking about it.

There's roughly 3 ways I think I want to use rate-limits

1. As a soft, temporary tool to give moderator-warnings a little more teeth.

If someone(s) is/are spiralling into a demon thread, we could give them a temporary "1 comment per ~hour" rate limit on that particular post, to force them to slow down and chill out. If they've done something that seems particularly norm-violating, we can give them a temporary significant rate limit. The alternative here would be to do nothing, or give them a temporary 1-week ban. It seems at least in some cases better to do the rate-limit, although I could imagine in some cases "taking an actual break for a week" would be better.

2. As a response to serious problem users who are either new, or established but have been given previous warnings.

The alternatives are "permaban," "doing nothing", and "moderators consistently have to be-on-top-of-things giving 'temp-bans' or other local-reactions to serious problem users each time they do something bad."

Doing nothing means leaving the garden untended, falling into entropy. "Moderators have to keep paying attention" is just... way too much work. From this perspective, a significant rate-limit is harsh...
Architecture-aware optimisation: train ImageNet and more without hyperparameters
by Chris Mingard — 2023-04-22T21:50:49.795Z

A deep learning system is composed of lots of interrelated components: architecture, data, loss function and gradients. There is a structure in the way these components interact - however, the most popular optimisers (e.g. Adam and SGD) do not utilise this information. This means there are leftover degrees of freedom in the optimisation process - which we currently have to take care of via manually tuning their hyperparameters (most importantly, the learning rate). If we could characterise these interactions perfectly, we could remove all degrees of freedom, and thus remove the need for hyperparameters.

Second-order methods characterise the sensitivity of the objective to weight perturbations using implicit architectural information via the Hessian, and remove degrees of freedom that way. However, such methods can be computationally intensive and thus not practical for large models.

I worked with Jeremy Bernstein on leveraging explicit architectural information to produce a new first-order optimisation algorithm: Automatic Gradient Descent (AGD). With computational complexity no greater than SGD, AGD trained all architectures and datasets we threw at it without needing any hyperparameters: from a 2-layer FCN on CIFAR10 to ResNet50 on ImageNet. Where tested, AGD achieved comparable test accuracy to tuned Adam and SGD.

Anyone interested in the derivation, PyTorch code, or experiments might be interested in any of the following links, or the summary figure below.

Here is a link to a blog post I wrote summarising the paper.
Here is a link to the paper
Here is a link to the official GitHub
Here is a link to an experimental GitHub where we test AGD on systems not yet in the paper (including language models).

Solid lines show train accuracy and dotted lines show test accuracy. Left: In contrast to our method, Adam and SGD with default hyperparameters perform poorly on a deep fully connected network (FCN) on CIFAR-10. Middle: A learning rate grid search for Adam and SGD. Our optimiser performs about as well as fully-tuned Adam and SGD. Right: AGD trains ImageNet to a respectable test accuracy.

Hopefully, the ideas in the paper will form the basis of a more complete understanding of optimisation in neural networks - as discussed in the paper, there are a few applications that need to be fully fleshed out. The derivation relies on an architectural perturbation boun...
OpenAI's GPT-4 Safety Goals
by PeterMcCluskey — 2023-04-22T19:11:42.945Z
Link: https://bayesianinvestor.com/blog/index.php/2023/04/22/openais-gpt-4-safety-goals/

OpenAI has told us in some
detail what
they've done to make GPT-4 safe.

This post will complain about some misguided aspects of OpenAI's goals.

Heteronormativity and Amish Culture

OpenAI wants GPT to avoid the stereotype ("bias") that says marriage
is between a man and a woman (see section 2.4, figure 2 of the system
card). Their example doesn't indicate that they're focused on avoiding
intolerance of same-sex marriage. Instead, OpenAI seems to be
condemning, as intolerably biased, the implication that the most common
form of marriage is between a man and a woman.

Heteronormativity is sometimes a signal that a person supports hate and
violence toward a sometimes-oppressed minority. But it's unfair to
stereotype heteronormativity as always signaling that.

For an example, I'll turn to my favorite example of a weird culture
that ought to be tolerated by any civilized world: Amish culture, where
the penalty for unrepentant gay sex is shunning. Not hate. I presume the
Amish sometimes engage in hate, but they approximately never encourage
it. They use shunning as a tool that's necessary to preserve their way
of life, and to create some incentive to follow their best guesses about
how to achieve a good afterlife.

I benefit quite directly from US recognition of same-sex marriage. I
believe it's important for anyone to be able to move to a society that
accepts something like same-sex marriage. But that doesn't imply that I
ought to be intolerant of societies that want different marriage rules.
Nor does it imply that I ought to avoid acknowledging that the majority
of marriages are heterosexual.

Training AIs to Deceive Us

OpenAI isn't just training GPT-4 to believe that OpenAI's culture is
more virtuous than the outgroup's culture.

They're trying to get GPT-4 to hide awareness of a fact about marriage
(i.e. that it is usually between a man and a woman).

Why is that important?

An important part of my hope for AI alignment involves getting a good
enough understanding that we can determine whether an AI is honestly
answering our questions about how to build more powerful aligned AIs. If
we need to drastically slow AI progress, that kind of transparency is
almost the only way to achieve widespread cooperation with such a costly
strategy.

Training an AI to hide awareness of reality makes transparency harder.
Not necessarily by much. But imagine that we end up relying on GPT-6 to
t...
Introducing the Nuts and Bolts Of Naturalism
by LoganStrohl — 2023-04-22T18:31:25.620Z

What Is Naturalism?

Naturalism is a general-purpose procedure for advancing one’s art of rationality.

“Naturalism” is the term I use for an investigative process that focuses attention on the points in daily life where subjective experience intersects with crucial information. It is a systematic method for taking your implicit stories and assumptions and holding them up against reality. In the process, you’ll find out whether they survive that contact, or get replaced with new stories.

I’ve previously published a sequence that focuses on the philosophy and perspective underlying this method. In it, I describe what I mean by “naturalism” in a lot of detail, by expanding on the sentence, “Knowing the territory takes patient and direct observation.” Fundamentally, I think of naturalism as the practice of behaving as though you believe that knowing the territory takes patient and direct observation.

What Is This Sequence?

My first naturalism sequence was about the worldview. This one is the step-by-step how-to.

The sequence you’re currently reading is the second in a larger series. This second sequence is coherent on its own; however, if you start to read and find yourself wondering, “But why would I bother with any of this?” then you should probably stop and go read the worldview sequence first. (If you instead find yourself wishing that there were many more in-depth examples, I’m afraid you’ll have to wait for my third sequence, which I expect will consist mainly of demonstration.)

As I pointed out at the end of the intro sequence, the practice of naturalism is entirely focused on “patient observation”.

When a person first begins to study naturalism with me, … I begin by helping them establish consistent habits of observation. … And then, throughout what has so far proven to be about a three month program, I never shift our focus away from consistent habits of observation. It’s not just where I start. It’s the entire curriculum. …

From a practical perspective, this dogged persistence is the foundation of naturalism. “Direct observation of the territory”, without patience, gets you something like a bag of tricks. Valuable tricks, but still tricks. Isolated mental motions made when they are convenient and enjoyable, not when they are most needed.

With patience, though, you get a life-long practice of epistemic rationality.

The sort of patience I have in mind is not s...
We Need To Know About Continual Learning
by michael_mjd — 2023-04-22T17:08:06.980Z

(Original title: The Continual Learning Overhang, but too many Overhang posts)

TL;DR Continual learning could lead to emergent causal understanding or agency and we should probably study it now before it’s too late.

Current LLMs are very impressive but suffer from a few key

missing pieces

(Sarah Constantin):

They inherently lack agency. They can be put within a loop of control code that makes them more agentic but it feels like this won’t really take off.
They are missing causal modeling to some extent. It’s just not how they are trained. They may get a sense of the causal structure of fiction but it’s not quite the same. They never train on interacting and have no inherent sense of “making an intervention”.

Sarah argues that the current approaches will be very unlikely to spontaneously develop these characteristics, and that it would require a ground level rethinking of how AI is done. I am not so convinced we’ve seen the full potential of the path we are on.

I think that we have yet to explore (publicly?) the potential of “switching on” backpropagation/training while in inference mode. Most models have a clean separation of “train” and “inference”. Inference applies the model to generate token by token as it goes along, but it is no longer learning.

Why I am skeptical of agency in current models / AutoGPT

AutoGPT is very interesting, but seems to be plagued by

trouble actually completing tasks

. It may improve with better code & prompts, but I still suspect it will miss the mark.

The context window is a very strong handicap in practice. To be a proper agent, one must have a long term goal, and a good explanation of the current state of the world. Without updating the weights of the model, this must fit inside the context window. Why this is suboptimal:

The context window itself in theory can hold several KB (maybe even 32KB!), which is not bad, but it is also in word form and may not necessarily be efficiently encodable or decodable.
Finding ways to track state between calls to GPT-4 is not even trained, so it (a) relies on human trial and error, and (b) on the model side, the kind of inputs expected are always in natural language, so no special efficient encoding can emerge (other than that weird emoji soup).
In any case, it seems very unlikely that even a human could be an effective agent at anything, if they could only remember

...
The Security Mindset, S-Risk and Publishing Prosaic Alignment Research
by lukemarks — 2023-04-22T14:36:50.535Z

Note: I no longer endorse this post as strongly as I did when publishing it. I now agree with Neel Nanda's criticism and endorse this position. I still think the analysis in the post could be useful, but I wanted to include this note at the beginning of the post in the case that someone else in my position happens to find it.

Introduction

When crafting ideas related to alignment, it is natural to want to share them, get feedback, run experiments and iterate in the hope of improving them, but doing this suddenly can bare contradiction to the

security mindset

.

The Waluigi Effect

, for example, ended with a rather grim conclusion:

If this Semiotic–Simulation Theory is correct, then RLHF is an irreparably inadequate solution to the AI alignment problem, and RLHF is probably increasing the likelihood of a misalignment catastrophe. Moreover, this Semiotic–Simulation Theory has increased my credence in the

absurd science-fiction tropes

that the

AI Alignment community has tended to reject

, and thereby increased my credence in

s-risks

.”

Could the same apply to other prosaic alignment techniques? What if they do end up scaling to superintelligence?

It's easy to internally justify publishing in spite of this. Reassuring comments always sound reasonable at the time, but aren't always robust upon reflection. While developing new ideas one may tell themselves “This is really interesting! I think this could have an impact on alignment!”, and while justifying publishing them “It’s probably not that good of an idea anyway, just push through and wait until someone points out the obvious flaw in your plan.” Completely and detrimentally inconsistent.

Arguments 1 and 3

Many have shared their musings about this before.

Andrew Saur

does so in his post “

The Case Against AI Alignment

”,

Andrea Miotti

explains here

why they believe RLHF-esque research is a net-negative for existential safety (see

Paul Christiano

’s

response

), and Christiano provides an alternative perspective in his “

Thoughts on the Impact of RLHF Research

”. Generally, these arguments seem to fall into three different categories (or their inversions):

This research will further capabilities more than it will alignment
This alignment technique if implemented could actually elevate s/x-risks
This alignment technique if implemented will increase the commercializability of AI, feeding int

...
How did LW update p(doom) after LLMs blew up?
by FinalFormal2 — 2023-04-22T14:21:23.174Z

Here's something which makes me feel very much as if I'm in a cult:

After LLMs became a massive thing, I've heard a lot of people p(doom) on the basis that we were in shorter timelines.

How have we updated p(doom) on the idea that LLMs are very different than hypothesized AI?

Firstly, it would seem to me to be much more difficult to FOOM with an LLM, it would seem much more difficult to create a superintelligence in the first place, and it seems like getting them to act creatively and be reliable are going to be much harder problems than making sure they aren't too creative.

LLMs often default to human wisdom on topics, the way we're developing them with AutoGPT they can't even really think privately, if you had to imagine a better model of AI for a disorganized species to trip into, could you get safer than LLMs?

Maybe I've just not been looking the right places to see how the discourse has changed, but it seems like we're spending all the weirdness points on preventing the training of a language model that at the end of the day will be slightly better than GPT-4.

I will bet any amount of money that GPT-5 will not kill us all.
The Cruel Trade-Off Between AI Misuse and AI X-risk Concerns
by simeon_c — 2023-04-22T13:49:02.124Z

Introduction

In this post, I will discuss the cruel trade-off between misuse concerns and X-risks (accidental risks) regarding racing. It is important to note that this post does not advocate for one risk being more plausible than the other, nor does it make any normative statements. The purpose is to analyze the outcome of a particular worldview.

In one sentence, the key claim is: “If an entity in power to develop AGI is mostly worried about misuse AND think that they’re the best entity (morally speaking) in their reference class, it is good to race. This is bad for accidental risks”.

Epistemic status: I’ve written up this post pretty quickly, after a conversation where it didn’t seem clear to someone. I'm confident about the general claim, less about specific claims.

Why If One Is Worried about Misuse They Should Race?

a. AGI is the most powerful thing ever created. Those who will control that will have an unprecedented level of control over everyone else. So if you end up with someone with bad intents controlling that, that’s probably the end for everyone else. It also leaves plenty of room for scenarios like stable authoritarianism.
b. Thus, if you’re at the head of an AGI lab and you’re genuinely worried about some other AGI labs’ CEO’s ethics, you would want to ensure that they don't develop AGI before you do. You could be worried about individuals getting power or certain cultures or governments getting power (the US fearing China, or a lab fearing another)
a. In a world where weak AGIs accessible to a wide range of people enable the creation of bioweapons or facilitate massive cyberattacks, there is a danger of reaching a point where everyone can kill everyone else but there is not yet a powerful "defensive AGI" to prevent this via global deterence & surveillance.
b. If you view your lab as responsible and if you’re primarily concerned about misuse, it makes sense to race towards the development of a defensive AGI of a sufficient power to avoid the dangerous scenario mentioned above.
a. Preventing jailbreaks is hard. So you may want to reach AGI as early as possible with as few deployments as possible. So the earlier you get to AGI, the better the world is.
b. Hence you should race as fast as possible[1] internally and deploy as little as possible externally to get the necessary capital to be able to reach

...