trentmkelly/LessWrong-43k huggingface dataset:
Subdivisions for Useful Distillations?
by Sharat Jacob Jacob — 2023-07-24T18:55:05.801Z

I am new to the field of AI safety and am testing my fit for being a communicator because, with the current climate, I think it would do a lot of good to bring clarity to the mind of the everyday person.

I started off with writing distillations as was recommended and suggested and I wrote one on the first post of "Risks From Learned Optimization" by Evan Hubinger et al. Filled with analogies and humor, cause that's my style of writing.

But I recently was part of the SERI MATS workshops, in the Agent Foundations stream headed by John Wentworth and realized I had a bunch of questions regarding writing distillations in general.

What's the appropriate length?
How much should you cut out / add to the distillation?
How do you assess whether your distillation adds anything of value to sites that alignment researchers/laypersons frequent? (Other than the obvious value of perceiving the material through a clearer lens?)

John's opinion (in quick response to my query) was that, whilst he firmly believed that distillations contained value and awareness was key, over the past time period, there were a growing number of distillations that just regurgitated what the original article/paper/post said or cut down and just gave the insights. Upon additional queries, he believed a good percentage of the time, the distillation in question should ideally be larger than the source material.

With that in mind, I think distillations could be divided into two categories (which can also be categorized as two others, but I will get to that) would provide a better categorization to go through for incoming would-be alignment researchers:

-> Expansive Distillations

This would be a space for distillations that expand and bring to proper clarity the source material so that each paragraph of the original can be digested more easily, and serve as a reference for anyone who seeks explanation.

-> Contractive Distillations

Working on a better name, cause that sounds pretentious to me, these would be distillations that work on just summarizing the key points of the source material for easy perusal. One could easily cover a larger amount of papers and this would pave the way to identify the more confusing posts that could use expansive distillations.

I think these distillations could also be classified into two different categories:

-> Explanatory Distilla...
Optimizing For Approval And Disapproval
by Thoth Hermes — 2023-07-24T18:46:15.223Z
Link: https://thothhermes.substack.com/p/optimizing-for-approval-and-disapproval

How to Proceed in a (Semi-)Adversarial Social Environment

Suppose you are a member of a relatively large intellectual community, which has frequent public discussions, a high level of interaction between members, and some mechanism for recording the level of engagement plus feedback on content submitted to the record by its membership.

Usually, new members of such a community spend most of their beginning hours “lurking” rather than contributing material. This is normal - most anyone has to spend time gathering data when being introduced to a new community. But suppose the time comes in which you’d first like to engage with the community. How do you know when to do so? What do you decide to write / talk about first? What do you decide to do after your first round of feedback?

Many communities ask their new members to lurk, too. Usually, you’re told something like “when in doubt, lurk more.” In other words, air on the side of not posting, if you’re not sure if you should yet. Chances are, you’re still not completely up to speed on the material.

Okay, so suppose you’ve been lurking for roughly eight years or so, but you still don’t know if you should post yet. At this point, I would go ahead and say that the data is probably telling you something else. You may have also noticed that it’s not so obvious to tell how long anyone else has lurked, either. In fact, most of the top community members were kind of there the entire time, from the beginning. They just kind of showed up with their first posts receiving wide acclaim.

The first time you feel the urge to post something substantial, it’s at the moment you have constructed (inside your mind, at least) enough arguments that you can build something larger than a comment. Moreover, you feel the urge to post this particular thing because according to your lights, you feel that the community has a blind-spot over a particular thing, which you feel to be kind of odd. You figure your arguments are at least pretty well-articulated. Therefore, if you’re wrong, you’ll receive a missing piece of the puzzle right away. If you’re right, everyone will thank you and you’ll gain lots of new friends and-or poly relationships.

But the most likely outcome - at least from my experience - is that you will not even know if you’re wrong, perhaps ever. You will be very lucky to receive attention at all, positive or negative. But even still...
An Opinionated Guide to Computability and Complexity (Post #0)
by Noosphere89 — 2023-07-24T17:53:18.551Z

Today, I'll try to introduce certain things in this sequence I will make that I found interesting of computability and complexity theory is, and in particular show how many problems can be solved by computers, in both a philosophical and practical sense. I will mostly not be reviewing the history of computability theory and complexity theory, except as a brief aside.

Along the way, I'll introduce some commentary of my own, so this will at least be somewhat interesting, as it would be boring to only talk about it, without interacting with the subject.

I will mostly be using the terms computability and complexity theory for historical reasons, but note that this will not correspond to decidable and practically solvable problems always, and we must always pick a machine and constraints on the machine before we can talk about whether a problem is solvable at all.

Things I will cover include:

Why the definition of decidable problems is machine and constraint dependent, and why picking different machines will get you different results in what you can compute, and why picking different time and space requirements will affect the sets of things you can compute.

Why Hilbert was right, in a manner of speaking, to hope for an algorithm that could decide at least major parts of mathematics, and why Church and Turing's thesis that the Turing Machine could capture every computer and algorithm was false.

Why Cobham's thesis that the practical problems we can solve correspond to P in complexity theory in our specific universe is unlikely to be true, because of the Time Hierarchy theorem as well as issues with the constant potentially being arbitrarily large.

What philosophy can gain from understanding complexity and computability theory, and addressing Aaronson's paper to point out where it goes right and wrong.

The underappreciated amount of things a Universal/Complete Turing Machine can simulate, and that it can actually emulate a halting oracle, even if it couldn't solve the halting problem on it's own, and the implications of that observation. Equivalently, I'm trying to say that computability and simulatability are different concepts, and one can simulate something without you being able to solve the computational problem it's doing.

What problems could we solve over the very long future, and what would we need to be able to do or know in order to solve diffic

...
Slowing down AI progress is an underexplored alignment strategy
by Norman Borlaug — 2023-07-24T16:56:25.604Z

The emotional burden of watching the world end

My current beliefs about AI timelines have made my life significantly worse. I find thoughts about the ever-shrinking timeline to AGI invading nearly every aspect of my life. Every choice now seems to be affected by the trajectory of this horrible technology.

Based on the responses in this post and others, many if not most people on this forum have been affected in a similar way. AI is the black hole swallowing all other considerations about the future.

Frankly, I feel a good deal of anger about the current state of affairs. All these otherwise thoughtful and nice people are working to build potentially world-ending technology as fast as possible. I'm angry so few people are paying any attention to the danger. I'm angry that invasive thoughts about AGI destroying the world are making it harder for me to focus on work that might have a non-zero chance of saving it. And I'm frustrated we've been so closed-minded in our approach to solving this problem.

The current situation is almost exactly analogous to the creation of the atomic bomb during World War 2. There's a bunch of very smart people gathered together, supplied with a fuckton of money by powerful groups, working towards the same destructive goal. There is social proof all around them that they are doing the right thing. There are superficially plausible reasons to think that everything will turn out fine. There is token engagement with the concerns raised by people concerned about the implications of the technology. But at the end of the day, the combination of personal incentives, social proof and outside impetus make everyone turn their head and ignore the danger. On the rare occasion that people are convinced to leave, it's almost always the most conscientious, most cautious people, ensuring the remaining team is even less careful.

There is so much magical thinking going on among otherwise intelligent people. Everyone seems to be operating on the assumption that no technology can destroy us, that everything will magically turn out fine, despite the numerous historical examples of new inventions destroying or nearly destroying the world (see the Cuban Missile Crisis or the great oxidation event when oxygen-producing bacteria extincted themselves and most other life on earth and caused a 300 million year ice age).

Frankly I've found the response from the ...
Anticipation in LLMs
by derek shiller — 2023-07-24T15:53:07.076Z

Anticipation in LLMs

This post describes some basic explorations of predictive capacities in LLMs. You can get the gist by reading bolded text. Thanks to Gemma Moran, Adam Jermyn, and Matthew Lee for helpful comments.

Autoregressive language models (LLMs) like GPT-3 & 4 produce convincingly human-like texts one word (or token) at a time. At each step, they estimate the probabilities of the next word, take a sample from that distribution, add it to their input, and repeat.

The text used to train them was written by humans. We don’t approach writing one word at a time. Instead, we generally have a rough idea in mind that we want to express before we start to type it. We anticipate where we’re going and the words we produce follow a plan. To be really good at estimating the probabilities of the next words in human-generated text, you might think that LLMs need to think ahead too. They need to anticipate what they may say later, if not explicitly form plans about it.

‘Anticipation’ is the process of reasoning about possible continuations in formulating next token predictions. Extreme forms of anticipation would involve formulating complete responses at the start, but to count as anticipating its responses, a model would only need to think ahead about some aspect of later text. Anticipation doesn’t require that LLMs have any ability or tendency to implement their expectations. There are reasons to plan now even if you expect your plans may change later, because anticipating the future can help you optimize your next move.

Anticipation also seems like it should be an accessible subject to study. You don’t have to make surgical modifications of the layers of a network. All it takes to confirm anticipation is one clear example generated by a clever prompt that demonstrates that the model is aware of how it needs to continue. This is more challenging than it may first seem, because LLMs might be very good at choosing ways that will make it easy to continue without having any foresight on how they will do it. I have tried to find compelling examples of anticipation, but I have so far failed. I’m relatively confident that recent models can produce surprisingly sophisticated text without engaging in anticipation.

I started this project not knowing what I thought about anticipation in LLMs and used the writing process as a way to figure it out. It is quite possible that t...
The cone of freedom (or, freedom might only be instrumentally valuable)
by dkl9 — 2023-07-24T15:38:54.687Z
Link: https://dkl9.net/essays/freedom_cone.html

Many policies with nontrivial points in their favour — as e.g. thorough environmental protection, public-dominant transportation, communal acoustic improvement — are swiftly rejected sith They Restrict Freedom, and People Value Freedom. (This objection is especially common in the US, but there's nothing intrinsically US-specific about this.) I don't intend to justify or criticise any particular such proposals. I target that swift general rejection, for it is, in many cases, based on a fallacious pattern.

In some cases, people are really rejecting the policy sith they tricked themselves into consistently supporting the opinions of one side, regardless of those opinions' basis in truth, and so I don't expect to fix those errors by reason. Those in other cases should consider the following.

Upon suggesting lesser freedom, some people assume the path towards particular low-freedom societies. They point to those dreadful examples of reduced freedom — (contemporary) China, North Korea, etc — and from there argue that any sufficient reductions of freedom lead to a miserable, enslaving society.

Some such objectors explicitly appealed to a spectrum of freedom, with total anarchy on one end and something like Nineteen Eighty-Four on the other end. Therein lies the confusion. Freedom and the lack thereof is not a line, but a cone. (This analogy isn't quite right. The correct analogy involves annoying combinatorics and isn't helpful enough to make up for it.)

At the tip of the cone is total anarchy. At the base disk is a range of many types of restrictive societies, arising from many sets of things to restrict. Such objectors think of a single path from the tip to one point on the base. We who promote a restrictive policy intended a point in the cone entirely off that path.

That is, some reject restrictive-but-beneficial policies sith they notice that other salient restrictive policies have been reliably bad. They are told to consider reduced freedom, and jump to a particular bad form of reduced freedom.

Restriction directed by a "good" (citizen-aligned) government can be better than freedom. If you still think freedom is intrinsically valuable: what if you were allowed to do all that which you would want to do, but no more? Under such law, you would be clearly non-free, yet satisfied — perhaps more than in the current world — no? If you object sith that neglects the preferen...
A reformulation of Finite Factored Sets
by Matthias G. Mayer — 2023-07-24T13:02:25.382Z

I present my bachelor's thesis that reformulates Finite Factored Sets:

Causality with Deterministic Relationships

Abstract:
We present a reformulation of Finite Factored Sets that uses functions and cartesian products instead of partitions and factorizations. In the new setting, we motivate assuming conditional orthogonality from observed conditional independence and show that inference is decidable. We show that Pearlian causal structures have analogs in the developed framework and propose an extension to the infinite unconditional case using measurable spaces.

I will quote Section 1.3 to give an overview of the additions to the original paper.

Most of the concepts in this work were first introduced in [Finite Factored Sets (FFS)], where the author starts with a set S and factorizes it using a set of partitions {Fi:i∈I} of S. Then partitions on S are analyzed using this factorization. This is called a (finite) factored set.

In this work, we instead start from the factors Fi as sets and construct F=×i∈IFi as a cartesian product. Here we call F a (finite) factor space. Instead of a partition on S, we analyze features X:F→ΩX, where a feature would correspond to the partition {X−1(ω):ω∈ΩX} of F.

In the same manner, we can translate all concepts using functions. In the authors' opinion, this is a more natural way to frame things. We hope that this reframing is helpful to understand the framework and for doing further work.

To compare this work to the original paper [FFS], we give a detailed list of the main additions, apart from the reformulation.

In Section 2.4, we provide an alternative way to define conditional histories using a factor space induced on a subset A⊆F.
This is helpful to understand the concepts more deeply and leads, in some cases,
to shorter and more intuitive proofs.
In Section 3.3, we generalize the examples provided in [FFS] (Example 1 and 2)
to groups.
In Section 3.2 and the appendix, we spell out the implications carried by
the fundamental theorem of finite factored sets [FFS] (Theorem 3)
to justify assuming conditional orthogonality from observed conditional independence.
In Section 3.4, we prove that inference is decidable, thus solving
this open problem raised in [FFS] (Section 7.1).
In Section 4, we convert Pearlian causal structures and models to factor spaces

...
Brain Efficiency Cannell Prize Contest Award Ceremony
by Alexander Gietelink Oldenziel — 2023-07-24T11:30:10.602Z

Previously Jacob Cannell wrote the post "Brain Efficiency" which makes several radical claims: that the brain is at the pareto frontier of speed, energy efficiency and memory bandwith, that this represent a fundamental physical frontier.

Here's an AI-generated summary

The article “Brain Efficiency: Much More than You Wanted to Know” on LessWrong discusses the efficiency of physical learning machines. The article explains that there are several interconnected key measures of efficiency for physical learning machines: energy efficiency in ops/J, spatial efficiency in ops/mm^2 or ops/mm^3, speed efficiency in time/delay for key learned tasks, circuit/compute efficiency in size and steps for key low-level algorithmic tasks, and learning/data efficiency in samples/observations/bits required to achieve a level of circuit efficiency, or per unit thereof. The article also explains why brain efficiency matters a great deal for AGI timelines and takeoff speeds, as AGI is implicitly/explicitly defined in terms of brain parity. The article predicts that AGI will consume compute & data in predictable brain-like ways and suggests that AGI will be far more like human simulations/emulations than you’d otherwise expect and will require training/education/raising vaguely like humans1.

Jake further has argued that this has implication for FOOM and DOOM.

Considering the intense technical mastery of nanoelectronics, thermodynamics and neuroscience required to assess the arguments here I concluded that a public debate between experts was called for. This was the start of the Brain Efficiency Prize contest which attracted over a 100 in-depth technically informed comments.

Now for the winners! Please note that the criteria for winning the contest was based on bringing in novel and substantive technical arguments as assesed by me. In contrast, general arguments about the likelihood of FOOM or DOOM while no doubt interesting did not factor into the judgement.

And the winners of the Jake Cannell Brain Efficiency Prize contest are

Ege Erdil
DaemonicSigil
spxtr
... and Steven Byrnes!

Each has won $150, provided by Jake Cannell, Eli Tyre and myself.

I'd like to heartily congratulate the winners and thank everybody who engaged in the debate. The discussion were sometimes heated but always very informed. I was wowed and amazed by the extraordinary erudition and willingness ...
[Crosspost] An AI Pause Is Humanity's Best Bet For Preventing Extinction (TIME)
by otto.barten — 2023-07-24T10:07:40.473Z
Link: https://time.com/6295879/ai-pause-is-humanitys-best-bet-for-preventing-extinction/

Otto Barten is director of the Existential Risk Observatory, a nonprofit aiming to reduce existential risk by informing the public debate.

Joep Meindertsma is founder of PauseAI, a movement campaigning for an AI Pause.
The existential risks posed by artificial intelligence (AI) are now widely recognized.

After hundreds of industry and science leaders warned that “mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war,” the U.N. Secretary-General recently echoed their concern. So did the prime minister of the U.K., who is also investing 100 million pounds into AI safety research that is mostly meant to prevent existential risk. Other leaders are likely to follow in recognizing AI’s ultimate threat.
In the scientific field of existential risk, which studies the most likely causes of human extinction, AI is consistently ranked at the top of the list. In The Precipice, a book by Oxford existential risk researcher Toby Ord that aims to quantify human extinction risks, the likeliness of AI leading to human extinction exceeds that of climate change, pandemics, asteroid strikes, supervolcanoes, and nuclear war combined. One would expect that even for severe global problems, the risk that they lead to full human extinction is relatively small, and this is indeed true for most of the above risks. AI, however, may cause human extinction if only a few conditions are met. Among them is human-level AI, defined as an AI that can perform a broad range of cognitive tasks at least as well as we can. Studies outlining these ideas were previously known, but new AI breakthroughs have underlined their urgency: AI may be getting close to human level already.
Recursive self-improvement is one of the reasons why existential-risk academics think human-level AI is so dangerous. Because human-level AI could do almost all tasks at our level, and since doing AI research is one of those tasks, advanced AI should therefore be able to improve the state of AI. Constantly improving AI would create a positive feedback loop with no scientifically established limits: an intelligence explosion. The endpoint of this intelligence explosion could be a superintelligence: a godlike AI that outsmarts us the way humans often outsmart insects. We would be no match for it.
A godlike, superintelligent AI
A superintelligent AI co...