trentmkelly/LessWrong-43k huggingface dataset:
Violin Supports
by jefftk — 2023-05-07T12:10:03.849Z

Violins are a bit of a strange design. Classically you hold them up
by pinching them between your chin and shoulder (restricting head
motion), though in more casual contexts people sometimes use their
left arm (restricting hand motion). Neither of these are great! Why
can't the violin stay up on its own?

It can! People have made all sorts of straps, ribs, and
counterweights for violins. Some examples:

Wickelbuckle:

NS
Design Balanced Shoulder Rest:

Aceto
Violin Neck Strap:

Violin Valet:

VioStrap:

I wanted something similar for my Stroh violin,
and played around a bit. After trying a few things I ended up with a
truly silly counterweight design:

Probably one of the other designs would have worked better, but this
does have the advantage of a long lever. Those are literal rocks
taped to the end.

I still probably won't play my Stroh very much, but I'm now much less
likely to hurt myself when I do.

Comment via: facebook, mastodon
Properties of Good Textbooks
by niplav — 2023-05-07T08:38:05.243Z

Heuristics for choosing/writing good textbooks (see also
here):

Has exercises

Exercises are interspersed in the text, not in large chunks (better at the end of sections, not just at the end of chapters).

Solutions are available but difficult to access (in a separate book, or on the web), this reduces the urge to look the solution up if one is stuck.

Of varying difficulty (I like the approach Concrete Mathematics takes: everything from trivial applications to research questions).

I like it when difficulty is indicated, but it's also okay when it's said clearly in the beginning that exercises are not marked for difficulty (making them mystery boxes).

Takes many angles

Has figures and illustrations. I don't think I've encountered a textbook with too many yet. (See Visual Complex Analysis for an example of doing this well.)

Has many examples. I'm not sure yet about the advantage of recurring examples. Same point about amount as with figures.

Includes code, if possible. It's cool if you tell me the equations for computing the likelihood ratio of a hypothesis & dataset, but it's even cooler if you give me some sample code I can use and extend along with it.

Uses typography

You can use boldface and italics and underlining for reading comprehension, example here.

Use section headings and paragraphs liberally.

Artificial Intelligence: A Modern Approach has one- to three-word side-notes describing the content of each paragraph. This is very good.

Distinguish definitions, proofs, examples, case-studies, code, formulas &c.

Dependencies

Define terms before they are used. (This is not a joke. Population Genetics uses the term "substitution" on p. 32 without defining it, and exercise 12-1 from Naive Set Theory depends on the axiom of regularity, but the book doesn't define it.)

If the book has pre-requisites beyond what a high-schooler knows, a good textbook lists those pre-requisites and textbooks that teach them.

Indicators

Multiple editions are an indicator for quality.

Ditto for multiple authors.

A conversational and whimsy style can be nice, but shouldn't be overdone.

Hot take: I get very little value from proofs in math textbooks, and consider them usually unnecessary (unless they teach a new proof method). I like the Infinite Napkin for its approach.

Wishlist

Flashcard sets that come together with textbooks. Please.

3blue1brown style videos

...
Against sacrificing AI transparency for generality gains
by Ape in the coat — 2023-05-07T06:52:33.531Z

One of the cornerstone issue of AI alignment is the black box problem. Our machine learning models are inscrutable tables of floating point numbers and we don't know how to decpher them to understand what is actually going on inside. This is bad. Everyone seem to agree that this is bad. And for some reason we keep making it worse.

I've heard about some efforts to increase AI transparency. To look inside the black box and figure out some of the gears inside of it. I think they are noble and important. And yet they are absolutely inadequte. It's incredibly hard to decipher human-understandable insights from the ML model. While, encoding the approximation of an existent algorithm in one is easy. Interpretability research in the current climate is not just going against a flow, it's an attempt to row a boat against a tsunami.

So maybe we should, at first, stop decreasing the AI transparency: making even more complex, more general models, encrypting even somewhat known algorithms into inscrutable float tables? This seems as an obvious thought to me. Let our AI tools consist of multiple ML models connected by inputs and outputs in an actual human readable code. Multiple black boxes with some known interactions between them is superior transparency-wise to one huge black box where nothing is known for sure. And yet we seem to continue moving in the opposite direction.

More and more general models are developed. GATO can do multiple different tasks with exactly the same weights. When CICERO was released and it was revealed that it consists of a language model working in conjunction with a strategy engine, it was casually mentioned that we should expect a unified model in the future. GPT-4 can take images as an input. I hope OpenAI is using a separate model to parse them, so it's a two black boxes scenario instead of one, but I wouldn't be surprised if it's not the case. Why are we accepting this for granted? Why are we not opposing these tendencies?

I expect that if humanity in our folly creates a true AGI via blind search by gradient decent it will more likely happen with a deliberate attempt to increase the generality of some model from just its capabilities in one single domain.

Recently, LessWrong adjacent circles started more vocally fighting against capability gains in AI. This is a hard fight that is going to cost us quite some social capital, an...
TED talk by Eliezer Yudkowsky: Unleashing the Power of Artificial Intelligence
by bayesed — 2023-05-07T05:45:17.345Z
Link: https://www.youtube.com/watch?v=7hFtyaeYylg

Note: The video is no longer available. It has been set to private. It'll eventually be released on the main TED channel.

This is a TED talk published early by a random TEDx channel.

Tweet by EY: https://twitter.com/ESYudkowsky/status/1655232464506466306

Looks like my Sudden Unexpected TED Talk got posted early by a TEDx account.

YouTube description:

Eliezer Yudkowsky is a foundational thinker on the long-term future of artificial intelligence.

With more than 20 years of experience in the world of AI, Eliezer Yudkowsky is the founder and senior research fellow of the Machine Intelligence Research Institute, an organization dedicated to ensuring smarter-than-human AI has a positive impact on the world. His writings, both fiction and nonfiction, frequently warn of the dangers of unchecked AI and its philosophical significance in today's world.

Yudkowsky is the founder of LessWrong, an online forum and community dedicated to improving human reasoning and decision-making, and the coinventor of the "functional decision theory," which states that decisions should be the output of a ﬁxed mathematical function answering the question: "Which output of this very function would yield the best outcome?"
Thinking of Convenience as an Economic Term
by ozziegooen — 2023-05-07T01:21:30.797Z
Link: https://forum.effectivealtruism.org/posts/o4e6PsdPQE7rRjzWy/convenience-a-term-for-discussing-incentives-in

Epistemic Status: Similar to my recent two posts about criticism and comfort, much of my thinking here comes from introspection and experience, as opposed to formal literature reviews.
TLDR: We can define "convenience" as an economic term to roughly mean "specific local benefit", then use it to clarify incentives around things like communication and belief. I lay out how this can work, how we can use "convenience accounting" at a large (but expensive) scale, attempt to formalize the concept, and more.

Introduction

Incentives are important, but discussing them can feel amorphous.

There’s a lot of scattered terminology. We have terms like incentives, incentive gradients, signaling, motivated reasoning, cognitive dissonance, bias, and even flexing.

I find this terminology to be frustratingly vague in practice. I like modeling decisions in terms of estimatable costs and benefits, but I haven’t had success putting these concepts into satisfying cost-benefit models.

After playing around with different ideas, I’ve so far settled on the concept of convenience as a helpful construct.

Simply put, a convenience is a specific local/private benefit. They’re typically intangible.

To many, this definition might closely match their existing understanding of the word convenience, enough that much of this essay might seem trivial.

I think the concept is a decent mix of straightforward and valuable. Actually estimating the full costs and benefits of conveniences would be a lot of work, but at least it can be clear what the path forward would be.

I’m particularly interested in conveniences in communication and thought and present examples accordingly. Our work at QURI is about building effective evaluations and estimates, so it’s important to understand where these might be hindered or assisted based on different pressures.

Since formalizing the concept, I’ve noticed many of these incentives in the wild. I’ve built further terminology on top of it and intend to expand on this in future posts.

As always, I’m eager to get feedback.

Someone in 2030 using special VR glasses to list all of the conveniences in social settings they come across. It’s a valuable lens!

Examples of convenient actions for myself

Let’s start with examples of conveniences in my life. I think these examples should generally match some of people’s existing intuitions on the word.

Convenient (beneficial) th

...
Corrigibility, Much more detail than anyone wants to Read
by Logan Zoellner — 2023-05-07T01:02:35.442Z

Corrigibility has been variously defined, for example here as:

We say that an agent is “corrigible” if it tolerates or
assists many forms of outside correction, including at
least the following: (1) A corrigible reasoner must at
least tolerate and preferably assist the programmers in
their attempts to alter or turn off the system. (2) It
must not attempt to manipulate or deceive its program-
mers, despite the fact that most possible choices of util-
ity functions would give it incentives to do so. (3) It
should have a tendency to repair safety measures (such
as shutdown buttons) if they break, or at least to notify
programmers that this breakage has occurred. (4) It
must preserve the programmers’ ability to correct or
shut down the system (even as the system creates new
subsystems or self-modifies). That is, corrigible reason-
ing should only allow an agent to create new agents if
these new agents are also corrigible

Or here as:

I would like to build AI systems which help me:

Figure out whether I built the right AI and correct any mistakes I made
Remain informed about the AI’s behavior and avoid unpleasant surprises
Make better decisions and clarify my preferences
Acquire resources and remain in effective control of them
Ensure that my AI systems continue to do all of these nice things

Regardless of the definition, there is a fundamental tension between two things that we want a Corrigible AGI to do:

We want the AGI to take useful actions which affect the future
We want the AGI to avoid taking actions which limit our possible futures

Corrigibility is frequently described in terms of the ability to "shut down" the AGI, but this is a oversimplification. For example, an AGI which spawns a world consuming nanobot swarm and then shuts down obviously satisfies the "user can shut down the AGI" condition, but not the "AGI prevents the user from losing control of the future" condition.

The tension between "does useful stuff" and "doesn't affect the far future" is inherent. For example, as pointed out in this impossibility proof.

The correct way to frame corrigibility is therefore not in terms of binary conditions such as "the user can shut down the AGI" but rather in terms of the tradeoff between fulfilling the user's objective function and limiting the user's possible reachable futures.

A Continuous Definition of Corrigibility

Suppose we have some metric F(a), which describes the...
Residual stream norms grow exponentially over the forward pass
by StefanHex — 2023-05-07T00:46:02.658Z

Summary: For a range of language models and a range of input prompts, the norm of each residual stream grows exponentially over the forward pass, with average per-layer growth rate of about 1.045 in GPT2-XL. We show a bunch of evidence for this. We discuss to what extent different weights and parts of the network are responsible.

We find that some model weights increase exponentially as a function of layer number. We finally note our current favored explanation: Due to LayerNorm, it's hard to cancel out existing residual stream features, but easy to overshadow existing features by just making new features 4.5% larger.

Thanks to Aryan Bhatt, Marius Hobbhahn, Neel Nanda, and Nicky Pochinkov for discussion.

Plots showing exponential norm and variance growth

Our results are reproducible in this Colab.

Alex noticed exponential growth in the contents of GPT-2-XL's residual streams. He ran dozens of prompts through the model, plotted for each layer the distribution of residual stream norms in a histogram, and found exponential growth in the L2 norm of the residual streams:

We had GPT-4 generate dozens of strings which "look like they could have been in GPT-2's training corpus", in addition to a few hand-written strings. We ran these strings through the model and recorded the norms of each residual stream, across layers and sequence positions (except for position 0, which is EOS padding, discussed later).
GPT2-XL has 48 layers in total.

Here's the norm of each residual stream for a specific prompt:

Position 0 (
) behaves differently, which is why we exclude it from the averaged plots.

Stefan had previously noticed this phenomenon in GPT2-small, back in MATS 3.0:

Note that Stefan originally used the standard deviation, but it is proportional to the norm because our models have zero mean.[1]

Basic Facts about Language Model Internals also finds a growth in the norms of the attention-out matrices WO and the norms of MLP out matrices Wout ("writing weights"), while they find stable norms for WQ, WK, and Win ("reading weights"):

Comparison of various transformer models

We started our investigation by computing these residual stream norms for a variety of models, recovering Stefan's results (rescaled by √dmodel=√768) and Alex's earlier numbers. We see a number of straight lines in these log...
On the Loebner Silver Prize (a Turing test)
by hold_my_fish — 2023-05-07T00:39:12.607Z

The Metaculus question "When will the first weakly general AI system be devised, tested, and publicly announced?" has four resolution criteria, and in my opinion, reliably passing a Loebner-Silver-prize-equivalent Turing test is the hardest, since it is the only one that is adversarial. (See Wikipedia for background on the Loebner Prize.)

What's involved in winning the Loebner Silver Prize?

Though the Loebner Prize is no longer being run, we can get information about its format from the Wayback Machine.
https://web.archive.org/web/20190220020219/https://aisb.org.uk/events/loebner-prize

The link covers up to the 2018 prize. According to Wikipedia, the competition was held once more in 2019, with a dramatically different format (notably not involving judges), and then discontinued. I think the resolution criterion is intended to refer to a more traditional format with judges.

There are two parts to the contest: the selection process and the finals. The selection process is only relevant for deciding which bots to include in the finals, but it's nonetheless interesting to read its transcript: https://web.archive.org/web/20181022113601/https://www.aisb.org.uk/media/files/LoebnerPrize2018/Transcripts_2018.pdf. The bot that would go on to win the bronze medal, Mitsuku, by today's standards would be considered very bad. The selection process consists of 20 pre-decided questions with no follow-up, and my opinion is that an appropriately fine-tuned GPT-4 would likely be indistinguishable from human on these questions.

However, the finals format is much more difficult. Format details:

Four judges, four bots, four humans, four rounds.
In each round, a judge is paired with one bot and one human, and there is 25 minutes of questioning.
The questioning is in instant-messaging style, much like https://www.humanornot.ai/.
To win the silver medal, the system must fool half the judges. (It seems to me that even a perfect bot would lose sometimes due to random chance, and I don't know how they account for that.)

It is not clear how much the judges and humans know about each other. If the judges know who the human confederates are, that would make the contest considerably harder.

Mitsuku's creator, Steve Worswick, wrote a recap of his 2018 win, and also posted Mitsuku's finals transcripts.

How hard is it to win?

Though Mitsuku won the bronze prize in 2018, it was clearly ...
Time and Energy Costs to Erase a Bit
by DaemonicSigil — 2023-05-06T23:29:42.909Z

Landauer derived a limit on the amount of energy dissipated in order to erase a bit. This limit is kTlog2, where k is Boltzmann's constant and T is the temperature of the environment.

However, recently some questions have been raised as to what situations this limit applies to. Maybe using kTlog2 energy only gives you a 50% chance of erasing the bit, which is nearly useless. Does kTlog2 suffice to allow us to erase a bit with high reliability, or is much more energy (eg. 50kT) required to do this? We'll analyze the energy cost to reliably erase a bit in this post, while also considering the time taken to do so.

The argument for ~50kT

First let's examine the argument for 50kT. Consider that the bit needs to represented with two possible states. For simplicity, let's say we have a particle which can be in either of the two states. To keep the particle from jumping from one state to the other, there needs to be an energy wall between the two states. And this wall needs to be high enough that the particle doesn't spontaneously jump between them due to thermal noise. If we want a decent reliability (say <10−15 probability of error), then giving the wall a height of something like 50kT should be more than sufficient, since the chance of jumping the wall goes with e−Ewall/kT.

Now to erase a bit, we raise the energy of the state on one side of the wall. If the bit is a 1, then the energy of the particle increases until it is able to jump the wall and land on the other side. (Or if the bit is a 0, then the particle just stays on the other side and nothing happens.) Since we have to raise the particle quite high for it to be able to jump the wall, nearly the entire height of the wall, the energy cost is equal to nearly the height of the wall. So if the bit had a 50% chance of being a 1, then the expected energy cost to erase a bit in this scenario is about 25kT (we don't have to pay any energy cost to raise the energy of a state if the particle is not currently occupying that state).

Simple straightforward argument for ~kT

First note that we only need the energy wall between states to be high while we care about the value of the bit and are doing computations with it. Once it comes time to erase a bit, we no longer care at all about whether the bit gets flipped by thermal noise. We're going to be erasing it in any case.

So consider a computer made out of 2 parts: The ...
How much do you believe your results?
by Eric Neyman — 2023-05-06T20:31:31.277Z
Link: https://ericneyman.wordpress.com/2023/05/05/how-much-do-you-believe-your-results/

Thanks to Drake Thomas for feedback.

I.

Here’s a fun scatter plot. It has two thousand points, which I generated as follows: first, I drew two thousand x-values from a normal distribution with mean 0 and standard deviation 1. Then, I chose the y-value of each point by taking the x-value and then adding noise to it. The noise is also normally distributed, with mean 0 and standard deviation 1.

Notice that there’s more spread along the y-axis than along the x-axis. That’s because each y-coordinate is a sum of two independently drawn numbers from the standard normal distribution. Because variances add, the y-values have variance 2 (standard deviation 1.41), not 1.

Statisticians often talk about data forming an “elliptical cloud”. You can see how the data forms into an elliptical shape. To put a finer point on it:

Why an ellipse — what’s the mathematical significance of this shape? The answer pops out if you look at a plot of how likely different points on the plane are to be selected by the random generation procedure that I used.

The highest density of points is near (0, 0), and as you get farther from the origin the density decreases. The green ellipse on the scatter plot is a level set of equal probability: if you were to select a datapoint using my procedure, you’d be more likely to land in any square millimeter inside the ellipse than in any square millimeter outside the ellipse — and you’d be equally likely to land in any location on the ellipse as on any other location on the ellipse.

The line of best fit is a statistical tool for answering the following question: given an x-value, what is your best guess about the y-value?

What is the line of best fit for this data? Here’s one line of reasoning: since the y-values were generated by taking the x-values and adding random noise, our best guess for y should just be x. So the line of best fit is y = x.

Huh, weird… this line is weirdly “askew” of the ellipse, and it doesn’t reflect the fact that the y-values are more dispersed than the x-values. Maybe the line of best fit instead passes from the bottom-left to the top-right of the ellipse, along its major axis. It sure looks like the points are on average closer to this line than to the previous one.

Which line is the line of best fit, and what’s wrong with the other line? I recommend pondering this for a bit before reading on.

&...