trentmkelly/LessWrong-43k huggingface dataset:
Forum Proposal: Karma Transfers
by DirectedEvolution — 2023-04-30T00:34:55.318Z

On LessWrong, we can weak- and strong-upvote posts, which adds to the other user's karma without impacting our own. As we gain karma, we are able to award more karma. I have nearly 10,000 LessWrong Karma, and my strong upvote awards 8 karma at no cost to me.

Many users offer monetary bounties or after-the-fact money awards for posts or comments that they find especially useful. These contests are often fun, motivating, and a costly signal of genuine appreciation. Over $500 in prize money has accumulated for fact-checking Jake Cannell's Brain Efficiency, and I offered a $10 micro-bounty for answers to Vanessa Kosoy's questions about technologies stuck on initial adoption.

I think the benefits of a bounty or award offering extend come from the structure of the reward rather than the fact that it is offered with cash. And so I think we could achieve a similar bounty/award-like result if it were possible for LessWrong users to directly transfer some of their LessWrong karma to other users.

For example, if I liked your response to my question, what if I could transfer 50 karma points from me (-50) to your comment or post (+50)?

Benefits

It expands the ability to use the "bounty" format, which seems to be generally considered to be a sound incentive structure
Bounties are fun and seem to be considered generally good for epistemics, are akin to bets
They allow people with less money to participate in bounty-making
I personally am uncomfortable with the way monetary bets on beliefs make for "pay to play credibility," and because I don't know how much money any particular user has, I don't know what's a "laughably small" bet to propose and what's a "laughably large" bet. But I know how much karma they have, and I wouldn't feel bad about proposing karma bets to anybody.
Sometimes, a post gets a very useful comment either from a latecomer or deep in a subthread. These comments will remain low-visibility because so few users will bother to scour the bottom of the comments section or 5 comments deep in a thread. The ability for the participants to spend karma to drastically increase the visibility of a comment to promote these neglected comments to wider visibility and appreciation seems like a very useful feature for which there is currently there is no good solution.

Refinements

We could make it cost 2 karma to transfer 1 karma to increase the costliness of the signal.

We could le...
Accuracy of arguments that are seen as ridiculous and intuitively false but don't have good counter-arguments
by Christopher King — 2023-04-29T23:58:24.012Z

I think one of the biggest reasons I am worried about AI doom is that there doesn't seem to be very good counter-arguments. Most of them are just bad, and the ones that aren't a bit vague (usually something along the lines of "humanity will figure it out" or "maybe LLMs will scale in nice ways").

However, I'm curious as to how accurate this heuristic is. My question: What examples in the past are there of "argument is widespreadly seen as ridiculous and intuitively false, but the argument is pretty solid and the counter-arguments aren't". (Sorry if that's a bit vague, use your best judgement. I'm looking specifically for examples that are similar to the AI x-risk debate.) And did they turn out true or false? Try to include reasons why the argument was so strong, a typical counter-argument, and the strongest counter-argument.

Please use spoiler text for the final answer so that I can try to predict it before seeing it!
A Guide to Forecasting AI Science Capabilities 
by Eleni Angelou — 2023-04-29T23:24:46.579Z

The following contains resources that I (Eleni) curated to help the AI Science team of AI Safety Camp 2023 prepare for the second half of the project, i.e., forecasting science capabilities. Suggestions for improvement of this guide are welcome.

Key points and readings

for forecasting in general:

What is a vignette?:

https://www.lesswrong.com/posts/jusSrXEAsiqehBsmh/vignettes-workshop-ai-impacts

Start using

Metaculus

and

Manifold

(if you haven’t already)

Book review of Superforecasting

Actually possible: thoughts on Utopia

for forecasting AI in particular:

What is a

hard take off

?: “A hard takeoff (or an AI going "FOOM"

[2]

) refers to AGI expansion in a matter of minutes, days, or months. It is a fast, abruptly, local increase in capability. This scenario is widely considered much more precarious, as this involves an AGI rapidly ascending in power without human control. This may result in unexpected or undesired behavior (i.e.

Unfriendly AI

). It is one of the main ideas supporting the

Intelligence explosion

hypothesis.
Read more:

https://www.lesswrong.com/posts/tjH8XPxAnr6JRbh7k/hard-takeoff

What is a

soft take off

?: “A soft takeoff refers to an AGI that would self-improve over a period of years or decades. This could be due to either the learning algorithm being too demanding for the hardware or because the AI relies on experiencing feedback from the real-world that would have to be played out in real time.”
What is a

sharp left turn

?: the transition from a slower to a faster scaling regime as defined

here

(6:27 to 7:50).

Pivotal acts

: acts that we, humans, take that make a big difference in terms of making x-risk less likely.

Pivotal acts from Math AIs

: use AI to solve alignment at the formal/mathematical level.
Objection: AI alignment doesn’t seem like a purely mathematical problem - it requires knowledge about different aspects of reality that we haven’t been able to formalize (yet) e.g., how do agents (both human and artificial) represent values.
Follow-up objection: if we knew how to formalize these problems then we wouldn’t need much help from AI science models, i.e., if we knew what questions to ask and how to ask them.
Such questions would be: “can you solve this [input difficult for humans yet solvable mathematical puzzle e.g., a matri

...
(A Failed Approach) From Precedent to Utility Function
by Akira Pyinya — 2023-04-29T21:55:16.192Z

(Sorry this post has many errors, so I rewrite this chapter here)

In the previous chapter, we have introduced how ACI circumvents the is-ought problem: an ought can’t be derived from an is, but what is the right thing to do can be derived from examples of doing the right thing.

In evolutionary history, the histories are strained through a sieve called natural selection, thus organisms receive and remember the right things. That's how living organisms, as examples of natural intelligence, can learn to behave in the right way from evolutionary history.

This chapter will provide a more detailed analysis of ACI. We argue that it is possible to derive utility functions from ACI's first principle, so that ACI can be compared with other intelligent models, such as the rational agent, active inference, and value learning approaches. We will also unpack how ACI acquires the motivation to construct models of the world and explore it.

The optimal policies’ distribution

ACI learns about policy from history. ACI argues that if the past actions and observations are known, it is possible to establish a posterior probability distribution for an agent’s policy. However, before this, it is necessary to assign a prior probability distribution.

According to the ACI model, an agent’s policy can be of various forms, ranging from simple lookup tables and goal-directed policies to more complex ones that aim to maximize expected total utility or reward. Because ACI is a universal intelligence model, not bound by the limitations of rational behavior, all these types of policies should be considered.

Without loss of generality, a policy π∈Π , can be defined as a map from histories h to a probability distribution over actions a∈A (following Stuart Armstrong):

π:H→ΔA

A history h∈H , is a sequence of observations and actions of an agent, while H is the set of histories.

Using Solomonoff induction, we can have a universal prior that assigns a probability distribution to the policy of an unknown agent, denoted as P(π) , with a sample space consisting of all possible policies. Roughly speaking, this universal prior is related to the length of the shortest Turing machine that can output the policy.

After that, we can have a posterior probability distribution of ...
Biological DOOM: a brief overview of biological computation
by Metacelsus — 2023-04-29T20:02:25.257Z
Link: https://denovo.substack.com/p/biological-doom

(no, not that kind of biological doom)

DOOM is a classic first-person shooter game released in 1993 by id Software. Because it’s from 1993, it doesn’t require much computing power compared to modern games. Additionally, the code (written in C) is easy to compile to run on a variety of processors.

Over the years, hackers have made DOOM run on things such as an ATM, a touchbar of a MacBook, a Porsche 911, and even a TI-84 calculator powered by potato batteries.

But what about cells?

Requirements for DOOM

The inputs to DOOM are based on button presses, traditionally on a keyboard. 9 keys in total are required (assuming “switch weapon” is implemented as one key that cycles through weapons).

For computation, the original 1993 release required:

4 MB of RAM and 12 MB of hard-drive storage
Intel 386 (bare minimum) or 486 processor.

There is some flexibility regarding the processor, but slower processors will have worse frame-rates. The Intel 386 had 275,000 transistors in its most basic configuration.

DOOM also requires a graphical output. The smallest resolution I’ve seen is 128x32 pixels, and that was cutting it a bit close. We’ll assume we need 4096 black-and-white pixels for the display.

Finally, DOOM has audio. For the purposes of this thought experiment, we can ignore this output. Although the soundtrack is great, it’s not strictly required to play the game.

Approaches to biological computation

So, how could we potentially run DOOM? Biological systems can perform computations in several ways:

Nucleic acid hybridization

These logic gates are based on strand displacement between complementary DNA sequences.[1] A recent paper demonstrated a set of DNA-based logic gates that could add two 6-bit binary numbers.

A DNA-based AND gate, from the paper. The output Oab is released only if both inputs A and B are present. This can also be reconfigured to work as an XOR gate.

Pros and cons:

Memory capacity is good (encoded in DNA or RNA)
Switching speed is OK (rate constants vary by design but are typically around 106M−1s−1)
Visual output could be provided by fluorophore/quencher conjugated oligonucleotides, but . . .
Coupling to a macroscopic output display would be far too slow, because it would have to rely on diffusion (taking a few minutes to cover a millimeter-scale distance). So, the game would have to be played using a microscope.
It’s hard to “reset” gates after using t

...
Horizontal vs vertical generality
by tailcalled — 2023-04-29T19:14:35.632Z

This is a note/definition I want to put out there. While the term may sound similar to breadth vs depth of knowledge, it is importantly distinct.

Let's define intelligence as the ability to process information and make decisions. (This is a very broad definition and includes not just humans but also e.g. most common computer programs.) General ability is intelligence that is applicable to many different tasks. Consider a key distinction in intelligence: horizontal generality vs vertical generality.

Horizontal generality: the ability to contribute to many different tasks. Calculators, search engines, etc. are horizontally general.
Vertical generality: the ability to efficiently complete tasks with minimal outside assistance. Animals, plants, etc. are vertically general.

Humans could arguably be said to be on the simultaneous frontier of horizontal and vertical generality.

You might think horizontal vs vertical generality is the same as breadth vs depth of knowledge, but I don't think it is. The key distinction is that breadth vs depth of knowledge concerns fields of information, whereas horizontal vs vertical generality concerns tasks. Inputs vs outputs.

Some tasks may depend on multiple fields of knowledge, e.g. software development depends on programming capabilities and understanding user needs, which means that depth of knowledge doesn't guarantee vertical generality.

On the other hand, some fields of knowledge, e.g. math or conflict resolution, may give gains in multiple tasks, which means that horizontal generality doesn't require breadth of knowledge.
Which technologies are stuck on initial adoption?
by Vanessa Kosoy — 2023-04-29T17:37:34.749Z

Video calls have been with us for a while. Except, they were rarely used. IME, people sometimes had Skype calls with relatives abroad and that's about it. And then, COVID happened. Suddenly, Zoom skyrocketed, with Google Meet not far behind. The reason is obvious.

Now, the time of lockdowns and restrictions on gatherings is over, the incentives to do video calls are (AFAICT) more or less the same as pre-COVID, and yet video calls persist. They became a completely routine way of doing business meetings, academic seminars and occasional social events. Why? AFAICT it's just the initial adoption barrier: once everyone did lots of video calls, and realized they are actually pretty convenient, they just kept using them.

So, here's a fun question: What other things are like video calls in the pre-COVID era? That is, the technology exists (more or less: maybe the UX needs some trivial improvements), the use-cases exist, only nobody uses it just because they're unaware or because it's not a "normal" thing everyone does. Given something to create initial adoption (like COVID did for video calls), everyone would start using it and never go back.
Will GPT-5 be able to self-improve?
by Nathan Helm-Burger — 2023-04-29T17:34:48.028Z

I want to try to go over some of the objections imagine people are having. I don't think I fully understand the opposing viewpoint here, so hopefully it will be clarified for me in the comments.

1. LLMs are not truly generating novel ideas, they are just interpolating between existing ideas via memorized statistical patterns.
- I think this is true to some degree. I don't think that this prevents those interpolations from being testable hypotheses which turn out to be useful. I think there's enough scientific literature and enough relevant open-source code available on the internet that remixing and integrating will be sufficient for the first few cycles of improvement. And after that, perhaps the resulting LLM++ will be better able to devise truely novel insights.

2. It is too hard to come up with any improvements on LLMs. Lots of people have tried and failed.
- In Sam Altman's interview with Lex Fridman, Sam says that there have been lots of small improvements between GPT-3 and GPT-4. The improvements don't have to each be huge leaps in order to make a noticeable difference as they accumulate.

3. LLMs hallucinate too much. They are unreliable and much of the code they produce doesn't work.
- My observation has been that when I ask GPT-4 for code relating to an easy and common problem, the sort of thing talked about in a lot of online beginner coding tutorials, it does great. Something like a 95% success rate. When I ask GPT-4 for something novel and challenging, about a specific edge case at which I've already tried and failed to do an internet search for the answer, then it does poorly. Around a 5% success rate.
- I expect that coming up with a set of code suggestions which result in enough successes to constitute a cycle of self-improvement will be even harder. I'm guessing something like a 0.1% success rate. I think this is sufficient for success if you have automated the process and can afford to run the process enough to generate and test millions of possibilities. This is a largely parallelizable process, so it doesn't necessarily take much wall clock time.

4. LLMs are currently insufficiently agentic to spontaneously do this.
- I agree. I'm discussing a large well-funded effort by the owners of a SotA LLM who have ac...
Research agenda: Supervising AIs improving AIs
by Quintin Pope — 2023-04-29T17:09:21.182Z

[This post summarizes some of the work done by Owen Dudney, Roman Engeler and myself (Quintin Pope) as part of the SERI MATS shard theory stream.]

TL;DR

Future prosaic AIs will likely shape their own development or that of successor AIs. We're trying to make sure they don't go insane.

Summary

There are two main ways AIs can get better: by improving their training algorithms or by improving their training data.

We consider both scenarios, and tentatively believe that data-based improvement is riskier than architecture based improvement. Current models mostly derive their behavior from their training data, and not training algorithms (meaning their architectures, hyperparameters, loss functions, optimizers or the like)[1]. So far, most improvements to AI training algorithms seem 'value neutral'[2]. Also note that most of human value drift currently derives from cultural shifts changing the 'training data' available in the environment, not biological evolution over the brain's base learning algorithms[3].

We imagine a future where AIs self-augment by continuously seeking out more and better training data, and either creating successor AIs or training themselves on that data. Often, these data will come from the AIs running experiments in the real world (doing science), deliberately seeking data that would cover a specific gap in its current capabilities, analogous to how human scientists seek data from domains where our current understanding is limited. With AI, this could involve AgentGPT-like systems that spin up many instances of themselves to run experiments in parallel, potentially leading to quick improvements if we are in an agency overhang.

We want to find methods of ensuring such 'automated science' processes remain safe and controllable, even after many rounds of self-directed data collection and training. In particular, we consider problems such as:

Preventing self-training from amplifying undesirable behaviors
Preventing semantic drift in concept representations during self-training
Ensuring cross-modality actions (such as a generated image for a text-to-image model, or robot movement for a text-and-image-to-actuator-motion model) remain grounded in their natural language descriptions after self-training in a non-lingual modality
Preventing value drift during multiple, iterated steps of self-retraining

Currently, we're focusing on scalable methods o...
The Many Stories of Science
by amayhew — 2023-04-29T16:49:08.964Z
Link: https://naiveskepticblog.wordpress.com/2023/04/28/the-many-stories-of-science/

Epistemic Status: Not exhaustively researched, dates and particular events are selected based on my incomplete and biased memory. My point in this post also rest on me not having seen particular versions of the story of science in the wild as well as repeated encounters with people seeming uncomfortable with the more abstract components of science. YMMV.

When the story of science is relayed it is often in terms of the development of that body of knowledge. We celebrate milestones of understanding like Galileo and Heliocentrism, Newton and Universal Gravitation, and Darwin and Evolution by Natural Selection.

An alternative story that is sometimes told is that of science as social activity and the role of power: The influence of funders politics and societal values on the direction of research and technology. The idea of progress vs stagnation. How the actions of individual scientists ripples outwards affecting generations. Science as part of the wider cultural fabric.

But I think it is worth reflecting on how science as a method has also changed over time. This development is

fairly well documented

but it is not presented as a narrative nearly as often in my experience. My guess is that the triumphs of such a story are too abstract to be as immediately compelling and were often unrecognized at the time as separate accomplishments from their associated advancements in the scientific body of knowledge.

Presented here is a chronological list of the major conceptual milestones in the development of the scientific method. This is not a list of scientific discoveries or theories, this is a list of the conceptual tools that enabled such discoveries. Not all of these were recognized for what they were at the time, but have since become integral to the method of scientific investigation and inquiry.

Conceptual Tool	Date	People Involved
Materialism	600 BC	ancient Greek philosophers, Thales, Heraclitus, Democritus
Theoretic models of the world	400 BC	Plato, Aristotle, Galen
Observation, Notice regularities	100 AD	Ptolemy
Scholastic tradition (move away from apprenticeships and mystery cults)	1200 AD	Duns Scotus, Roger Bacon
Parsimony	1300 AD	William of Ockham
Mathematical models	1610 AD	Kepler
Observational experiments	1620 AD	Francis Bacon
Revolutionary ideas (overturning the old)	1630 AD	Galileo, Bruno
Peer review and scholarly community	1660

...
One bit of observation can unlock many of optimization - but at what cost?
by dr_s — 2023-04-29T10:53:03.969Z

This question by johnswentworth nerd sniped me, so I ended up thinking a lot about the relationship between information and control over the world in the simplified scenario of a single tape of bits. The question asked how many bits of optimization one could unlock with a single bit of observation; the answer ended up being "arbitrarily many", as proven by a simple example: suppose you have a rule that says that if the first bits of your action Amatch those of your observation O, then the rest of A gets copied to your target value Y; then there's no limit to how many bits you can copy, and the only information you need to leverage is the knowledge of the secret "key". We know this works because this is exactly how locks work in real life too. If I possess someone's bank account password, or the combination to a safe, or the US nuclear codes, then I'm able to produce disproportionate effects to the tiny size of that knowledge, just because I can also rely on the state of the world and its laws being set up in such a way that I can use that knowledge as a pivot to trigger much bigger effects.

The thing I wanted to focus on then was those conditions: even given that I know the "password", what else do I need to know about the world at large, and what limits are there on my power to optimize the final state? I decided to focus on the following model:

a world string W of N bits, prepared in some initial state W0, with some regions known and some randomized;
a discrete map f that determines the evolution of this world, such that W1=f(W0), W2=f(f(W0)) and so on. The map is reversible, such that there exists an inverse Wi−1=f−1(Wi);

The focus on the map being reversible is because in the real world the laws of physics are time symmetric too and microscopically should not destroy information. Irreversible computing allows the deletion of information, which reduces the entropy of the system. In a computer embedded in a larger world this can be compensated by creating entropy somewhere else, but if our string has to represent the entire world, then it should preserve information. The world string has several identifiable regions:

an action region A, within which we can set up bits arbitrarily in the initial state;
an observation region O, which we can't affect but whose contents we know

...