trentmkelly/LessWrong-43k huggingface dataset:
Writing this post as rationality case study
by Ben Amitay — 2023-07-08T12:24:58.156Z

[Before I begin: If you don't like this post, please let me know why. Even just few words like boring/off-topic/poorly-written may give me something to work with.]

I wish to share a late struggle I have with rationality, because I think that it touches some interesting points. But more importantly - because I think that it is important to think about rationality in the context of concrete day-to-day decisions. This post is going to be messy and have no specific "point" or definite conclusion - like real life decisions.

Like many, I am attracted to content creation. I like to think about stuff and share my insights and patterns of thoughts, and like the idea that one day I will be able to spread them far and wide. To begin moving in that direction, I lately published two posts. The first was ignored, and the second heavily downvoted. Then I stopped to think.

Why do I want that really? The immediate flattering answer is that I value good ideas and view their creation as the highest public service. The less flattering answer is that I like prestige and want more of it. The interesting answer is that the will to create content is not that different from the will to have children: The memes that ended up inhabiting my brain are those who successfully spread themselves across society - often by convincing their host that meme-spreading is a good idea. It mean that I should expect exactly what I see - that the will to spread memes will be wide-spread, and therefore highly competitive. That people put extraordinary effort into writing books with very small success rates, when the positive black swans are just not big enough to justify it (if money was the sole consideration).

I first thought about it as a bias for content-creation, and wanted to write a post about that, but I probably don't endorse this framing anymore. Let's say that I buy that analysis - does it really mean that I shouldn't pursue that goal? It seem less respectable now, for sure. It seem less likely to maximize my happiness or my contribution to society, but it doesn't make me not want it. Is it a terminal value of mine, or just a tool that I refuse to put down when I see it don't work? How would I tell? Does the distinction even make sense given that I'm not an ideal agent?

When did all those thoughts come up anyway? When I saw the downvotes and wanted to shy away. Downvotes which I pretty much expec...
What Does LessWrong/EA Think of Human Intelligence Augmentation as of mid-2023?
by lukemarks — 2023-07-08T11:42:38.625Z

Zvi Recently asked on Twitter:

If someone was founding a new AI notkilleveryoneism research organization, what is the best research agenda they should look into pursuing right now?

To which Eliezer replied:

Human intelligence augmentation.

And then elaborated:

No time for GM kids to grow up, so:

collect a database of genius genomes and try to interpret for variations that could have effect on adult state rather than child development
try to disable a human brain's built-in disablers like rationalization circuitry, though unfortunately a lot of this seems liable to have mixed functional and dysfunctional use, but maybe you can snip an upstream trigger circuit
upload and mod the upload
neuralink shit but aim for 64-node clustered humans

This post contains the most in-depth analysis of human intelligence augmentation (HIA) I have seen recently, and provides the following taxonomy for applications of neurotechnology to alignment:

BCIs to extract human knowledge
neurotech to enhance humans
understanding human value formation
cyborgism
whole brain emulation
BCIs creating a reward signal.

It also includes the opinions of attendees (stated to be 16 technical researchers and domain specialists) who provide the following analysis of these options:

From the original post: "Fig. 2| Comparison on key variables. A. Feasibility vs. timeline. Technology clusters that were deemed less feasible were also presumed to take longer to develop. B. Impact on AI vs. timeline. Technology clusters that were seen as having a larger potential impact on AI alignment were also presumed to take longer to develop. C. Impact on AI vs. feasibility. Technology clusters that were deemed more feasible were seen to be less likely to have an impact on AI alignment. Green trend lines represent high correlations (R2 ≥ 0.4318) and red represent low correlations."

Outside of cyborgism, I have seen very little recent discussion regarding HIA with the exclusion of the above post. This could be because I am simply looking in the wrong places, or it could be because the topic is not discussed much in the context of being a legitimate AI safety agenda. The following is a list of questions I have about the topic:

Does anyone have a comprehensive list of organizations working on HIA or related technologies?

Perhaps producing something like this map for HIA might be valuable.

Does independent HIA research

...
Request for feedback - infohazards in testing LLMs for causal reasoning?
by DirectedEvolution — 2023-07-08T09:01:31.760Z

Edit: I have no idea why this is getting silently downvoted, and would appreciate a comment or PM to explain.

Judea Pearl has tweeted a couple of times the claim that ChatGPT can't perform causal reasoning (1, 2). Although it's possible to get ChatGPT to exhibit successful or failed causal reasoning in response to natural-language prompts, it's not always clear whether the success or failure was due to confusing wording, or a similar test being included in training data. A second limitation of natural-language ad hoc prompts is that they don't allow scaling the complexity or scoring success rates quantitatively. Finally, many prompts do not make it unambiguously clear whether ChatGPT is meant to find any possible causal relationship, or whether it's meant to find a common-sense causal relationship.

To address these limitations, I have developed software that generates a directed acyclic graph (DAG) of an arbitrary size representing a causal network, and then outputs it in natural language, along with a set of test questions interrogating the effect of intervening on a particular node in the graph. It also outputs human-readable answers. This permits directly pasting in the description of the causal DAG into any language model and comparing the LLM's answers to the algorithmically computed answers. Because the causal DAGs are generated randomly and can be of an arbitrary size, it is implausible that the LLM's answers are based on regurgitating training data.

Currently, my test software requires manual input and answer-checking, but I am working on adding the ability to send questions and receive answers from ChatGPT via the API, and to have those answers algorithmically checked for accuracy. This would allow highly scalable and quantitative testing of ChatGPT's performance. Other tests are possible as well:

Can ChatGPT not only determine the correct answers for an intervention in a causal DAG, but generate a compelling real-world example of a set of causal relatoinships that fit the causal DAG?
Can ChatGPT convert a real-world scenario into a causal DAG description, feed that description back into the Python program in order to generate a series of test questions, receive back the abstract description of the causal DAG, and then answer the questions successfully?

As interesting and illuminating as this test may be, I do have a concern that it may be capabilities-en...
Views on when AGI comes and on strategy to reduce existential risk

by TsviBT — 2023-07-08T09:00:19.735Z

Summary: AGI isn't super likely to come super soon. People should be working on stuff that saves humanity in worlds where AGI comes in 20 or 50 years, in addition to stuff that saves humanity in worlds where AGI comes in the next 10 years.

Thanks to Alexander Gietelink Oldenziel, Abram Demski, Daniel Kokotajlo, Cleo Nardo, Alex Zhu, and Sam Eisenstat for related conversations.

My views on when AGI comes

AGI

By "AGI" I mean the thing that has very large effects on the world (e.g., it kills everyone) via the same sort of route that humanity has large effects on the world. The route is where you figure out how to figure stuff out, and you figure a lot of stuff out using your figure-outers, and then the stuff you figured out says how to make powerful artifacts that move many atoms into very specific arrangements.

This isn't the only thing to worry about. There could be transformative AI that isn't AGI in this sense. E.g. a fairly-narrow AI that just searches configurations of atoms and finds ways to do atomically precise manufacturing would also be an existential threat and a possibility for an existential win.

Conceptual capabilities progress

The "conceptual AGI" view:

The first way humanity makes AGI is by combining some set of significant ideas about intelligence. Significant ideas are things like (the ideas of) gradient descent, recombination, probability distributions, universal computation, search, world-optimization. Significant ideas are to a significant extent bottlenecked on great natural philosophers doing great natural philosophy about intelligence, with sequential bottlenecks between many insights.

The conceptual AGI doesn't claim that humanity doesn't already have enough ideas to make AGI. I claim that——though not super strongly.

Timelines

Giving probabilities here doesn't feel great. For one thing, it seems to contribute to information cascades and to shallow coalition-forming. For another, it hides the useful models. For yet another thing: A probability bundles together a bunch of stuff I have models about, with a bunch of stuff I don't have models about. For example, how many people will be doing original AGI-relevant research in 15 years? I have no idea, and it seems like largely a social question. The answer to that question does affect when AGI comes, though, so a probability about when AGI comes would have to depend on that answer.

But ok. He...
Weekday Evening Beach Picnics
by jefftk — 2023-07-08T02:20:00.910Z

In the summer the kids don't need to be to bed so early and it stays
light longer. When the weather's good we've been doing a lot of
evening picnics at the beach. It's not too crowded (making parking
easier), there's less direct sun (don't need to deal with
sunscreen), and it doesn't conflict with the toddler's nap time.

Often we go to Shannon
Beach ("Sandy Beach"), on the Mystic Lakes:

They test bacteria levels, and when they get high they close the
beach. Check the DCR Alerts
Page before going.

This is the closer option for us, but when it's closed we often go Revere Beach:

This is on the ocean, but it's sheltered enough that waves are
typically just a few inches. It's also colder: I'll swim at Sandy
Beach, especially in the late summer, but I'd need to go hundreds of
miles south of Boston if I wanted ocean water that didn't chill me too
quickly to be fun.

Both have free parking, and Revere Beach is on the Blue Line.
Neither have lifeguards: at Sandy Beach there are ropes that keep the
older two within an area they can easily touch and we stay close to
the toddler; at Revere Beach if they want to go past their knees they
need an adult with them.

The older two often want to eat quickly just before leaving or in the
car because (a) they prefer hot food and (b) they want to play the
whole time we're there. But it's still important to them that the
trip be a picnic, and Julia and I enjoy eating on the beach. Also a
good fit for portable ice cream!
ACI#4: Seed AI is the new Perpetual Motion Machine
by Akira Pyinya — 2023-07-08T01:17:50.246Z

Abstract

Seed AI's singularity scenario is based on the hypothesis that an AI agent can preserve its original utility function while self-improving. We argue that all value-preservation mechanisms will eventually fail because you can't encode values into someone smarter than you. Thus, a seed AI may lose the motivation to improve itself. However, this does not preclude the self-improvement of AI agents. They may find motivation in competition between intelligent systems, but they will modify their initial values in different environments.

Introduction

It is believed that a seed AI can build machines smarter than itself without human intervention. Through an iterative process called recursive self-improvement, the seed AI continues to improve itself by rewriting its own source code, and can eventually outperform humans.

Furthermore, the seed AI would retain its original utility function throughout this self-improvement process, because it wants to maximize the expected total utility by ensuring that its future self retains the same utility function.

As the improved AI achieves superhuman capabilities, it would take dangerous actions, such as filling the universe with paperclips, in order to maximize the total utility. The seed AI would pursue a goal relentlessly with ever-increasing competence, like a Perpetual Motion Machine with an incremental power, or a car with a self-enhancing accelerator and a locked steering wheel.

Yudkowsky and Herreshoff introduced the "tilting" agent, which can construct other agents with similar preferences, and argued that a tilting agent might be able to keep improving itself despite a Gödelian difficulty in the form of Löb's Theorem. However, they still have no evidence that this obstacle is not unavoidable.

We claim that there is a fatal flaw in the seed AI scenario: a rational AI agent can never successfully preserve its utility functions as long as it wants. The scenario in which no self-modifying agent can keep its value would be more reasonable than the other way around.

Value-preservation may fail

Omohundro has shown that a rational agent should try to preserve its values by protecting its original utility functions from being rewritten. Unfortunately, no value-preservation mechanism is perfect. Any advanced intelligent system may fail to protect its utility function, just as anyone, no matter how smar...
Links to discussions on social equilibrium and human value after (aligned) super-AI?
by Michael Tontchev — 2023-07-08T01:01:53.732Z

Suppose we work hard and get a stroke of luck and unaligned AI isn't a problem.

What does society look like when human-level and super-intelligent AIs can be spun up and work way cheaper than humans do?

I'm in a bit of an existential void right now, because I can't see any future other than mass wealth concentration, UBI for 99% of people (if they can get it and keep it, which is iffy, since they will have no political power), and potentially depopulation (e.g. one child policies, higher UBIs for sterilization, etc).

I've had answers to previous such questions, like "the market will figure it out", and historically, it has, because intelligence has always been something that humans can add to a production process. But when we're totally and strictly inferior, what happens? What livable, corner of existence can the vast majority of people who do not own plots of large natural resources carve for themselves to lead meaningful lives?

I'm especially focused on stable, long-term

equilibrium outcomes

. Answers like "well, we'll all be rich!" ring hollow, because when you zoom into who will be rich and how, the answer doesn't seem to be "all of us", but rather a relatively small group of people/corporations who spin up the AIs and own protectable natural resources. Even "we'll all be on UBI" is questionable, because why would the owners of the resources choose to support 99% of the world? Especially in the long term, when they can instead choose (non-violent) depopulation options.

Sorry for the gloom, but would love to see previous work on this :)
Notes from the Qatar Center for Global Banking and Finance 3rd Annual Conference
by PixelatedPenguin — 2023-07-07T23:48:24.560Z

Today, I attended a crypto-related panel discussion at the 3rd annual conference at the Qatar Center for Global Banking and Finance, delving deep into the future of crypto regulation.

Leonardo Gambacorta (BIS): Discussed the types of national regulatory strategies: ban, contain, and regulation.

Sheldon Mills (FCA): Discussed the potential of CBDCs as a digital form of trusted currencies. He acknowledged the significant hurdle to digital currency adoption due to the public's attachment to physical cash and traditional currencies.

David Birch (15Mb Ltd): Highlighted the importance of digital identity laws over crypto-specific laws. He proposed that the solution to off-chain and on-chain discrepancy should be institutional, not technological.

My Reflections:

Contrary to the common perception, I believe the crypto landscape is more about hope than trust. The difference is subtle - while trust requires verification, hope involves trusting and putting effort in advance. The oracle problem, an embodiment of the principle-agent problem, exemplifies this distinction between trust (typified by Bitcoin) and hope (typified by DeFi).

In the context of the oracle problem, Birch's institutional solution presents a contradiction. Proposals for trusted organizations to verify oracle data go against Bitcoin's philosophy of disintermediation. The oracle becomes a paradoxical entity - an unverifiable, predetermined demi-god of the system. Regulation, hence, needs to address the technical realities and inherent contradictions when dealing with trust generation and consensus.

Can we ever completely remove the human factor from finance? Smart contracts might offer predictability and reassurance to investors, but the web of smart contracts behaves unpredictably. Changes in system parameters could trigger drastic deviations from the desired outcomes. So the answer is perhaps no.
Introducing bayescalc.io
by Adele Lopez — 2023-07-07T16:11:12.854Z
Link: https://bayescalc.io

I made a simple online calculator for doing elementary hypothesis testing!

Link to example shown.

I was disappointed that an intuitive and easy-to-use app for using bayes' theorem apparently did not exist, so I decided to make it. My goal was to make something that:

Helped people correctly and quickly evaluate the effect of evidence while comparing hypotheses.
Was easy enough for someone who didn't know math to use.

And which also helped show what the math was doing in an intuitive way, so that you didn't have to trust math you didn't understand.

Felt good enough to use that it would actually be used.
Could be used to share simple models of things in a way that would help people have more productive discussions, and promote shared model building.

Hopefully I've at least made substantial progress on these goals, and I'd really appreciate feedback on ways in which it falls short! This includes even minor interface or design issues. You can leave feedback as a comment here, or on the issues page.
I'd also be really happy to see people share examples they've made in the comments!
Meetup Tip: Ask Attendees To Explain It
by Screwtape — 2023-07-07T16:08:40.639Z

Summary

A question I like asking when running meetups is, "do you think you could explain how to do this to someone else?" I usually ask it just after explaining the rules or steps for whatever we're doing this meetup. This timing only works for a meetup with an explicit activity, though you can also ask attendees if they feel they could explain how to run a dinner party or other less explicit meetup. Often people nod yes, and then I point at one of them and ask them to explain it, and point out some missed details. This is partially a good way to make sure your audience actually understands well enough to do the activity, but I have a sneaky ulterior motive: every person who understands how to run meetups is one step closer to being a meetup organizer.

You have now read the basic point of this post. If you want to read on, cool, lets talk implementation details for a bit.

Details

I suggest not spending too long on this in any single meetup. It's not really worth the cost in pacing to have every attendee explain how to do this until they can do it without leaving anything out or adding unnecessary steps. I usually pick one person and have them explain things once, give corrections once, and then move on. Maybe I'll do two people, each explaining it once. Think of "could you explain how to do this to someone else?" as a small habit you take five minutes for every meetup, like light stretching.

Sometimes their explanation is actually better than mine. It's not at all uncommon that there's some neat analogy or example the use that I didn't think of. When that happens, I copy it with gleeful joy. Take notes! You presumably have thought about how to do this more than they have, but you also knew what you wanted to happen before you explained anything. They have an advantage in that they are just learning it now.

Speaking of which, take note of where they get things wrong. Especially if you're running the same kind of meetups repeatedly, this gives you an excellent opportunity to notice which parts of your explanation people get stuck on or which parts of a technique are tricky to learn. Refine your explanations accordingly.

This question usually applies to what I call the payload of a meetup. If you're running something from the Meetup In A Box sequence like Calibration Trivia, then a good response indicates that the person can run Calibration Trivia once all the ...
Interpreting Modular Addition in MLPs
by Bart Bussmann — 2023-07-07T09:22:51.940Z

Summary

In this post, we investigate one of Neel Nanda's 200 Concrete Open Problems in Mechanistic Interpretability, namely problem A3.3: Interpret a 2L MLP (one hidden layer) trained to do modular addition.

The network seems to learn the following function:

logit(c)∝∑Ni=0relu(u1i(cos(wia+s1i)+cos(wib+s2i)))(u2icos(wic+s1i+s2i)+oi)
The code to reproduce the experiments in this post can be found here.

Background

In their paper, Progress Measures For Grokking Via Mechanistic Interpretability, Neel Nanda et. al find that the one-layer transformers learn a surprisingly funky algorithm for modular addition. Modular addition is the function c=(a+b) mod P, where P = 113 in both my and the original experiments.

In the original work, they find that a 1-layer Transformer learns an algorithm where the numbers are converted to frequencies with a memorized Discrete Fourier Transform, added using trig identities, and converted back to the answer.

The goal of my project is to investigate whether a simple MLP with one hidden layer learns a similar algorithm, or something completely different.

Training the Network

I create a dataset with all possible combinations of a and b, where a and b are integers in the range (0, 112). This gives me a dataset of 12769 samples, of which 80% is used for training and 20% for validation. The inputs and outputs are one-hot encoded, which means that the MLP has 226 input neurons and 113 output neurons. In the hidden layer, we use 100 neurons with ReLu activation[1]. We use an AdamW optimizer with a learning rate of 0.003, batch size of 128, and weight decay of 0.5.[2] We train the network for 2000 epochs, which is plenty to get near-perfect accuracy on both training and validation sets.

Note that the x-axis on both graphs is log-scale. The y-axis is the loss graph is also log-scale, but the one on the accuracy graph isn't.

Looking at Activations and Weights

Now that the network is trained, let's take a look at the activations. We make a plot where we show the activation of some neurons for every possible value of a and b.

Oooh, pretty patterns! That must mean that the weights in the first layer are encoding the input in some periodic manner. Let's take a closer look at the input weights for Neuron 43, and see if they can be described by a sine/cosine function like the embedding layer in the previous work.

The plot on the left shows

...