trentmkelly/LessWrong-43k huggingface dataset:
Critique of some recent philosophy of LLMs’ minds
by Roman Leventov — 2023-01-20T12:53:38.477Z

I structure this post as a critique of some recent papers on the philosophy of mind in application to LLMs, concretely, on whether we can say that LLMs think, reason, understand language, refer to the real world when producing language, have goals and intents, etc. I also use this discussion as a springboard to express some of my views about the ontology of intelligence, agency, and alignment.

Mahowald, Ivanova, et al., “Dissociating language and thought in large language models: a cognitive perspective” (Jan 2023). Note that this is a broad review paper, synthesising findings from computational linguistics, cognitive science, and neuroscience, as well as offering an engineering vision (perspective) of building an AGI (primarily, in section 5). I don’t argue with these aspects of the paper’s content (although I disagree with something about their engineering perspective, I think that engaging in this disagreement would be infohazarous). I argue with the philosophical content of the paper, which is revealed in the language that the authors use and the conclusions that they make, as well as the ontology of linguistic competencies that the authors propose.
Shanahan, “Talking About Large Language Models” (Dec 2022).

Dissociating language and thought in large language models: a cognitive perspective

In this section, I shortly expose the gist of the paper by Mahowald, Ivanova, et al., for the convenience of the reader.

Abstract:

Today’s large language models (LLMs) routinely generate coherent, grammatical and seemingly meaningful paragraphs of text. This achievement has led to speculation that these networks are—or will soon become—“thinking machines”, capable of performing tasks that require abstract knowledge and reasoning. Here, we review the capabilities of LLMs by considering their performance on two different aspects of language use: ‘formal linguistic competence’, which includes knowledge of rules and patterns of a given language, and ’functional linguistic competence’, a host of cognitive abilities required for language understanding and use in the real world. Drawing on evidence from cognitive neuroscience, we show that formal competence in humans relies on specialized language processing mechanisms, whereas functional competence recruits multiple extralinguistic capacities that comprise human thought, such as formal reasoning, world knowledge, situation modeli

...
Preface
by iy3d — 2023-01-20T12:38:25.643Z

Hello LessWrong World !

Welcome to my personal regular writing challenge on LessWrong!

This post serves as a Preface to the start of a personal blogging experience I'm very excited (synonym of apprehension) to start. Inspired by Eliezer Yudkowsky similar challenge, this is an attempt to freely write and form a habit of writing down on a public platform.

I'm looking forward to think about both the form and the aim of this writing experience.

As to the specifics, the underlying goal of this challenge is to encourage myself to engage in the practice of clear and effective communication of ideas, and as a byproduct, deep dive into technical subjects related to my center of interests.

As we strive to understand the world as best as we can, it is not enough to simply read about good ideas : you must share them ! I must also be able to communicate them effectively, structurally to others. This is a skill that I aim to practice and improve; this writing challenge is an opportunity to do just that.

Each week, I will provide myself a topic for me to write about in a given timeframe. The range of topics and, my center of interest per se, range from biosciences to cinematography, and are a challenge to review existing informations about a topic.

Furthermore, I believe it's important to point out that a blog article, written following scientific paper guidelines, could be considered a self-published paper rather than a mere blog article.

Scientific paper guidelines involve a specific format, such as an abstract, introduction, methods, results, and discussion sections, and a rigorous review process to ensure the quality and accuracy of the research presented. As a general outline, following those specific set of guidelines would be considered a scholarly work, regardless of the fact that it was published on a blog or in the margin of your library book.

It's worth noting that without peer-review, there is no credible way of quantifying your writings, as the peer-review process is meant to ensure the quality and accuracy of the research. (link to my next writing about quantifying researchers)

Nobody likes studying, but everyone wants to be smarter. Learning, getting insights and understanding academic subject are immensely rewarding. However, my real motivation is to show people that researching faster is possible and that it can be fun at the same time.
Without a cl...
Lost in Innovation: The Case of Phlogiston
by adamShimi — 2023-01-20T12:19:00.603Z
Link: https://epistemologicalvigilance.substack.com/p/lost-in-innovation-the-case-of-phlogiston

Innovation and progress make the world a better place. They unlock new possibilities, correct past mistakes, and realize untapped potential. It’s not a coincidence that these words can have such positive connotations.

Yet to reap these benefits, progress must be carefully integrated into the old order. For innovators often turn their back on all that existed before, throwing away valuable past insights.

The best example I know of is the so-called Chemical Revolution at the end of the 18th century.

In most popular books and histories of science, this shift is presented as Lavoisier discovering oxygen, and with it the compositional approach to chemistry, replacing the old alchemy-imbued idea of phlogiston. The prevalent explanation of combustion at the time, phlogiston was a weightless fluid whose movement in and out of objects ruled combustion, respiration, and the properties of metals. But to the modern eye, it looks nothing like a scientific explanation — just a made-up fake substance, a fake explanation. It clearly couldn’t compete with Lavoisier’s composition of elements.

Or so we read in the modern treatment.[1]

On the other hand, a quick look at our current best models of combustion (so called oxido-reduction reactions) reveal a weightless “fluid” whose movements capture the essence of the reaction: electrons. Sure, electrons are not a fluid; but they’re not really particles either according to quantum mechanics, and the fluid/substance frame fits with the concepts available at the end of the 18th century. Properties of metals also emerge from free electrons, confirming a key prediction of phlogistonist chemistry that Lavoisierian chemistry couldn’t explain.

Gilbert Lewis, of the Lewis diagram fame, makes this point in 1926 in The Anatomy of Science:

The phlogistonists were not content with the idea alone, but must add a mechanism, the hypothetical phlogiston, so that every process of the type we are discussing was supposed to involve the gain or the loss of this almost imponderable substance. By this mechanism they fell. They had not yet recognized that the air is a chemical reagent, and thought that the process of burning was merely the loss of phlogiston. When it was found that substances in burning gain in weight they were obliged to retreat before the proponents of the oxygen theory. If they had only thought to say “The substance burning gives up its phlo

...
Generalizability & Hope for AI [MLAISU W03]
by Esben Kran — 2023-01-20T10:06:59.898Z
Link: https://newsletter.apartresearch.com/posts/compiling-code-to-neural-networks-w03

Welcome to this week’s ML & AI Safety Report where we dive into overfitting and look at a compiler for Transformer architectures! This week is a bit short because the mechanistic interpretability hackathon is starting today – sign up on

ais.pub/mechint

and

join the Discord

.

Watch this week's MLAISU on

YouTube

or listen to it on

Spotify

.

Superpositions & Transformers

In

a recent Anthropic paper

, the authors find that overfitting corresponds to the neurons in a model storing data points instead of features. This mostly happens early in training and when we don’t have a lot of data.

In their experiment, they use a very simple model (a so-called

toy model

) that is useful when studying isolated phenomena in detail. In some of the visualizations, they train it from 2D data with T training examples. As seen below, the feature activations (blue) look very messy while the activations to the data points (red) look very clean.

Going deeper in the paper, they find that this generalizes to larger dimensions (10,000D) and that the transition from overfitting on smaller datasets’ data points to generalizing to the actual data features seems to be the reason for the famous

double descent phenomenon

where a model sees a dip in performance but then becomes better afterwards.

And on the topic of toy models, DeepMind releases

Tracr

, a compiler that can turn any

RASP human-readable code

into a Transformer architecture. This can be useful for studying how algorithms represent themselves in Transformer space and to study phenomena of learned algorithms in-depth.

Other research news

In other news…

Demis Hassabis, the CEO of DeepMind, is warning the world on the risks of artificial intelligence in

a new Time piece

. He mentions that the wealth arising from artificial general intelligence (AGI) should be redistributed throughout the population and that we need to make sure it does not fall into the wrong hands.
Another piece reveals that OpenAI contracted Sama to use Kenyan workers with less than $2 / hour wage ($0.5 / hour average in Nairobi) for toxicity annotation for ChatGPT and undisclosed graphical models, with reports of employee trauma from the explicit and graphical annotation work, union breaking, and false hiring promises. A serious issue.

Jesse Hoogland releases

an exciting piece exploring why and how

...
What’s going on with ‘crunch time’?
by rosehadshar — 2023-01-20T09:42:53.215Z

Also on EAForum here.

Imagine the set of decisions which impact TAI outcomes.

Some of the decisions are more important than others: they have a larger impact on TAI outcomes. We care more about a decision the more important it is.

Some of the decisions are also easier to influence than others, and we care more about a decision the easier it is to influence.[1]

Decisions in the past are un-influenceable
Probably some decisions in the future are too, but it may be hard to identify which ones
How easy it is to influence a decision depends on who you are

People have access to different sets of people and resources
Influence can be more or less in/direct (e.g. how many steps removed are you from the US President)

The influenceable decisions are distributed over time, but we don’t know the distribution:

Decisions might happen earlier or later (timing/timelines)
There might be more or fewer decisions in the set
Decisions might be more or less concentrated over time (urgency/duration)
There might be discontinuities (only one decision matters, at the right of the distribution suddenly there are no more decisions because takeover has happened…)

It’s possible that in the future there might be a particularly concentrated period of important decisions, for example like this:

People refer to this concentrated period of important decisions as ‘crunch time’. The distribution might or might not end up actually containing a concentrated period of important decisions - or in other words, crunch time may or may not happen.

Where the distribution does contain a concentrated period of important decisions (or at least is sufficiently likely to in expectation), crunch time might be a useful concept to have:

It’s a flag to switch modes (e.g. aiming for direct effects rather than getting ourselves into a better position in future)[2]
It’s a way of compressing lots of information about the world (e.g. these 15 things have happened with various levels of confidence, which makes us think that these 50 things will happen soon with various probabilities)
It’s a concept which can help to coordinate lots of people, which in some crunch time scenarios could be very important

But we don’t know what the underlying distribution of decisions is. Here are just some of the ways the distribution might look:

The fact that the underlying distribution is uncertain means that there are several ways in whi...
Shard theory alignment has important, often-overlooked free parameters.
by Charlie Steiner — 2023-01-20T09:30:29.959Z

A delayed hot take. This is pretty similar to previous comments from Rohin.

Shard theory alignment requires magic - not in the sense of magic spells, but in the technical sense of steps we need to remind ourselves we don't know how to do. Locating magic is an important step in trying to demystify it.

"Shard theory alignment" means building an AI that does good things and not bad things by encouraging an RL agent to want to do good things, via kinds of reward shaping analogous to the diamond maximizer example.

How might the story go?

You start out with some unsupervised model of sensory data.
On top of its representation of the world you start training an RL agent, with a carefully chosen curriculum and a reward signal that you think matches "goodness in general" on that curriculum distribution.
This cultivates shards that want things in the vicinity of "what's good according to human values."
These start out as mere bundles of heuristics, but eventually they generalize far enough to be self-reflective, promoting goal-directed behavior that takes into account the training process and the possibility of self-modification.
At this point the values will lock themselves in, and future behavior will be guided by the abstractions in the learned representation of the world that the shards used to get good results in training, not by what would actually maximize the reward function you used.

There magic here is especially concentrated around how we end up with the right shards.

One magical process is how we pick the training curriculum and reward signal. If the curriculum is only made up only of simple environments, then the RL agent will learn heuristics that don't need to refer to humans. But if you push the complexity up too fast, the RL process will fail, or the AI will be more likely to learn heuristics that are better than nothing but aren't what we intended. Does a goldilocks zone where the agent learns more-or-less what we intended exist? How can we build confidence that it does, and that we've found it?

And what's in the curriculum matters a lot. Do we try to teach the AI to locate "human values" by having it be prosocial towards individuals? Which ones? To groups? Over what timescale? How do we reward it for choices on various ethical dilemmas? Or do we artificially suppress the rate of occurrence of such dilemmas? Different choices will lead to different shards. We w...
Solving For Meta-Ethics By Inducing From The Self
by  — 2023-01-20T07:21:29.267Z

How do we determine what is good? By what method/metric? What is the argument? In this paper I will attempt to argue why and how I ground my ethics in an understanding of the self and it's values. To give a very broad and insufficient tl;dr: I'll try to create a descriptive moral foundation that describes oughts without needing to justify them. A hypothetical imperative with presatisfied, 'given', "ifs".

I will try to justify my ethics in this paper–I will try to ground my “first principles”/so that they are not merely chosen–‘external’–arbitrary:

You--your values, wants, and even actions, can be understood to really be two things. an external self--a self that is cultivated depending on and from the environment, and an internal self--a self that exists regardless of environment.

Then we use the hypothetical of a you that has existed in every possible world with every possible external self, and we can reasonably deduce that the self would be most satisfied in a world in which its internal wants are manifested rather than just any external want. You would yourself choose to align with your internal wants, desires, actions, and so on and so forth. (This is the best possible worlds hypothetical thought up by many). You yourself would be maximally satisfied in a world in which your internal self is maximally manifested and realized in the environment–tangentially for humans this even means some variation and development OF external wants and hobbies. Your specific family for example is an external value, but cultivating an external specific family is itself internal. Also you can’t argue against this using internal intuitions–regardless of external beliefs and even internal intuitions ABOUT goodness (the human sense of morality), what you already value and would by yourself choose and affirm is what you have to work with. All else, is not only just "relativism", but it's from a human/subjective perspective nonsensical to use. It’s delusional, literally in the same vein as imagining a giant squirrel in your backyard.

From that meta-ethical foundation, (which you can note would actually first seemingly justify egoism,) you work with other humans and engage, and from that arises a generally compatible sense of what makes life worth living all based around a shared (or at least compatible) understanding of what constitutes a ma...
Vegan Nutrition Testing Project: Interim Report
by Elizabeth — 2023-01-20T05:50:03.565Z

Introduction

Reducing consumption of animal products is a choice with both moral and practical consequences. Last summer I found myself in contact with many vegans who cared a lot about the moral consequences, but had put little effort into learning about or managing the practical consideration of removing animal products from their diet. I’ve suffered a lot due to bad nutrition, so this made me very concerned. With a grant from the Survival and Flourishing Fund, I launched small a pilot project to give nutritional tests to 5 vegans and near-vegans from the Lightcone Office, which they could use to choose supplements that would hopefully improve their health.

My long-term goal was for everyone to have accurate information on their personal nutritional costs of veganism and make informed choices about how to handle them, with the first line solution being supplements. My goal for the pilot was to work out practical issues in testing, narrow the confidence interval on potential impact, and improve the nutrition of the handful of people. This report is on phase 1: getting the testing done and supplements started. It is aimed at people who might want to run a similar program at scale; if you are interested in running this for yourself I recommend checking out Tuesday’s post on iron deficiency.

Tl;dr: I found rampant iron deficiencies, validating the overall concern. The procedure I used has a lot of room for improvement.

The Experiment

I gave nutrition tests to 6 people in the Lightcone office.

The ideal subject was completely vegan, had never put any effort or thought into their diet, and was extremely motivated to take a test and implement changes. This person does not volunteer for studies, so I ended up with 4 vegans or near-vegans who had put somewhere between 0 and a lot of thought into their diet, 1 vegetarian, and 1 extremely motivated omnivore I used to test out the process. In addition, one hardcore vegan contributed results from private testing. I did not poll the ~vegans on their exact diets.

Unless otherwise stated the results exclude the omnivore.

I gave each of these six people a Genova Metabolomix+ test, ordered from walkinlabs.com, with the iron add-on. This test was selected for being recommended by doctors I trust (in part because they prefer urine to blood testing), having extremely easy-to-read results, being nearly comprehensive (wit...
Maybe you can learn exotic experiences via analytical thought
by Q Home — 2023-01-20T01:50:48.938Z

The contents of this post:

I argue that you can create exotic experiences at will via mere analytical thought. I think you can easily enough learn the weirdest types of synesthesia. Without any substances.
This post touches on the binding problem and (a little bit) on the solvability of the hard problem of consciousness.
This post describes unusual possible properties of experience. You're free to disagree with those properties, but they are new ideas.
I try to give examples of my idea.

My arguments are mostly based on my models. The arguments repeat the same idea 9 times from 9 different angles.

1. Experience is prediction

A simple argument that experiences can be learned:

Each human thought carries at least some experience.
A new type of thoughts = a new type of experience.
"Predictions" are a very sharp and repeatable and verifiable type of thoughts.
Learning a new type of prediction = learning a new type of sharp, repeatable and verifiable experience.

Imagine you can't "experience" sentences, you experience only particular words. You can reason about a sentence only after studying every particular word. I come along and say "hey, I can teach you to predict words in a sentence before you read them".

If you learned this - you learned to experience sentences. Even though your experience of sentences can still be weird and not equivalent to the normal experience.

2. Experiences = situations

Every important situation/context affects, at least a little bit, the way you experience the smallest things. So, in order to create a new type of an experience (A)... you just need to experience A in a new important situation/context! It's that easy.

Smells trigger memories, the perception of time is 100% tied to situations, music easily connects to situations (take any popular tune and people are going to describe memories and elaborate associations: HOME - Resonance, Mareux - The Perfect Girl, Donkey Kong Country 2 - Stickerbush Symphony, Chrono Trigger - Secret of the Forest)...

You can't separate the perception of people from situations and relationships. Sadly, traumatic events can turn the simplest experiences into triggers. Liminal spaces is the most popular/relatable/developed meme about subjective experience.

I know, "experience is tied to situations/contexts" sounds extremely trivial. Of course it is!

But it's really strange, actually. Why even the simplest experiences link...