trentmkelly/LessWrong-43k huggingface dataset:
The Hidden Complexity of Thought
by Isaac King — 2023-03-19T21:59:21.696Z
Link: https://outsidetheasylum.blog/the-hidden-complexity-of-thought/

If you've read Thinking, Fast and Slow, you'll be familiar with the concepts of a "system 1" that does fast, unconscious processing, and a "system 2" that does slow, methodical processing. The important insight here is that these systems don't sit side by side. System 2 is built *on top* of system 1. Conscious thought is an emergent property of all the low-level unconscious thinking that's going on.

(Every part of your conscious thinking process originally comes from your unconscious mind. Think about how you explicitly work through a problem step-by-step. How do you determine what step comes next? How do you perform one of those steps? The details are all performed unconsciously, and only a "summary" is brought to conscious awareness.)

If you had to use your system 2 to catch a ball, you couldn't do it. The time it would take to explicitly calculate a trajectory and the muscle movements needed to position your hand in that location would take hours, if not years, to figure out. The reason we can do it with our system 1 is because system 1 is a vastly more powerful system, finely tuned through evolution to be good at problems like that.

(Just watch people try to program a robot to catch a ball or assemble a puzzle.)

System 1 evolved first, and it's found in the minds of all animals. System 2 comes into existence once system 1 gets complicated enough to support a second layer of processing on top of it. (Think about someone building a computer in Minecraft. Their physical computer is simulating a universe, and then a computer is implemented inside the physics of that universe. The computer inside Minecraft is vastly slower and more limited than the actual computer it's built on top of.)

There's a conception of fields of thought as being "hard" or "soft", such as in the hard sciences/soft sciences and hard skills/soft skills dichotomies. And the hard skills/sciences are generally thought of as being more difficult. This is generally true, for humans. Soft skills are the sorts of things that we evolved to be good at, so they feel natural and effortless. Hard skills are those that didn't matter all that much in our ancestral environment, so we have no natural affinity for them.

But in a fundamental sense, hard skills are vastly simpler and easier than soft skills. Hard skills are those that can be formalized. Knowing how to perform long division is challenging for hum...
"Wide" vs "Tall" superintelligence 
by Templarrr — 2023-03-19T19:23:57.966Z

There are 2 distinctive but compatible modes of superintelligence.

First one (arguably what we see right now with latest NNs) - super resource access, but not necessary super skills. Operates not better, maybe even significantly worse, than the person who's able to read entire library of congress, wiki and all open professional forums. But no person can do that due to reading IO speed and lifetime limitations, that's what makes it "superintelligence". "Knowing more", "horizontal line in T-shaped person" etc etc. No pushing the boundaries but great ability navigating within them.

Second one - super-skills but not necessary super resource access. You can do only X, but in X there neither is or ever was anyone better. Pushing boundaries, inventing new styles and approaches, "doing better", "vertical line in T-shaped person" etc etc.

My question - is there already accepted terminology for them? For myself I was always calling them Wide and Tall (because of T-shaped person analogy and also association with different strategy names).

I found a similar idea explored in https://www.lesswrong.com/posts/semvkn56ZFcXBNc2d/superintelligence-5-forms-of-superintelligence but both Wide and Tall arguably fit the "quality" term there.
Humanity's Lack of Unity Will Lead to AGI Catastrophe
by MiguelDev — 2023-03-19T19:18:02.099Z

An AI worldview contest entry for the question: Conditional on AGI being developed by 2070, what is the probability that humanity will suffer an

existential catastrophe

due to loss of control over an AGI system?

https://www.openphilanthropy.org/open-philanthropy-ai-worldviews-contest/

Introduction

Facing the undeniable possibility of failure, it is not the emergence of AI Overlords that threatens us, but our shared struggle to join forces as a society during critical times. Though I firmly trust in our potential to unite against impending disasters, I recognize the considerable hurdles brought by corporate agendas and human vulnerability. Thus, I share my pessimistic outlook on why the world will not successfully collaborate on tackling this seemingly inevitable catastrophe that will happen in the next 20 years or less. Why? the ingredients are already here....

Catastrophic Ingredients

Make sure you are sitting on a chair

Transformers appear to emulate humanistic dream states, enabling the fascinating existence of the ' petertodd'. It appears to have internalized all the dark, malevolent concepts present in the vast pool of online information that served as the training ground for GPT-2 and GPT-3.

Dr. @mwatkins twitter post cites how GPT may have developed a sophisticated character named petertodd. This still being investigated.

Opposing the notion of utilizing Github copilot and ChatGPT for low-level coding, it's essential to consider their potential in C and C++ languages. These AI-powered tools can significantly impact the functionality of everyday devices, including modifying the code of running microchips or even revolutionizing something as commonplace as your home garage door opening exactly at 12 midnight...

The potential catastrophe being illustrated here heavily impacts the stability of the Western world. With access to these AI tools, extremist organizations could wield unprecedented power, and we're only just beginning to grasp the extent of the issue.

Unfalsifiable Woo

The lacking element for AGI to unlock true consciousness and wield destructive power lies with our corruptive tendencies and stupidity in general - yes, our humanistic tendencies is the last piece of this puzzle. Let me share a personal experience here to explain this:

While seeking the reasons behind Carl Jung, the father of analytical psychology's un...
Probabilistic Payor Lemma?
by abramdemski — 2023-03-19T17:57:04.237Z

Epistemic status: too good to be true? Please check my math.

We've known for a while that Löb's theorem fails when proof is relaxed to probabilistic belief. This has pros and cons. On the pro side, it means there's no Löbian Obstacle to probabilistic self-trust. On the con side, it means that some Löb-derived insights for proof-based decision theory don't translate to probabilistic decision theory, at least not as directly as one might hope. In particular, it appeared to dash hopes for probabilistic generalizations of the "Löbian handshake" for cooperation.

Recently, Andrew Critch wrote about the Payor Lemma, which allows for a very similar "modal handshake" without Löb's Theorem. The lemma was proved using the same modal assumptions as Löb's, so on the surface it may appear to be just a different method to achieve similar results, whose main advantage is that it is much easier to prove (and therefore explain and understand) than Löb's Theorem.

But, a natural question arises: does Payor's Lemma have a suitable probabilistic version?

I'll give an affirmative proof; but I haven't confirmed that the assumptions are reasonable to my satisfaction.

Setup

Let L be a language in first-order logic, expressive enough to represent its sentences s∈L as quoted terms ┌s┐, eg, through Gödel numbering; and with a probability function symbol on these terms, p(┌s┐), which can be equated with (some representation of) rational numbers, e.g. p(┌⊤┐)=1, p(┌s┐)=12, etc. I also assume the system can reason about these rational numbers in the basic ways you'd expect.

For all a,b∈L and all r∈Q, we have:

If ⊢a, then ⊢p(┌a┐)=1.
If ⊢a→b, then ⊢p(┌a┐)≤p(┌b┐).

(These assumptions might look pretty minimal, but they aren't going to be true for every theory of self-referential probability; more on this later.)

Let B(s) abbreviate the sentence p(┌s┐)>c for any s and some globally fixed constant c strictly between 0 and 1. This is our modal operator. (This modal operator is often called "p-belief" with the constant c named p instead, but I wanted to reserve p for the probability operator.)

Some important properties of B:

Necessitation. If ⊢s, then ⊢B(s), for any s.

Proof: Since ⊢s implies ⊢p(s)=1, and c∈(0,1), we have...
Playbook for the Great Divergence
by intellectronica — 2023-03-19T17:42:32.948Z
Link: https://www.intellectronica.net/playbook-for-the-great-divergence/

As we progress through the advanced stages of a period that can be called "the great divergence", the divide between those who can effectively utilise AI tools and those who cannot is widening. In the final stage (sometimes referred to as "the singularity" by the techno-religious), AI will ultimately diverge from biological intelligence.

To remain competitive during this transformative era, it would be useful to follow a playbook detailing various levels of human-AI interaction skills. Each level gains relevance after the preceding one and eventually becomes obsolete as AI technology continues to evolve.

Level 1: Acquaintance with AI Tools

With AI development advancing rapidly, it is essential to stay informed about the latest tools and integrations. This requires dedicating time and effort to explore new developments, which in turn helps you remain current and gain hands-on experience with innovative technologies.

Skills required: curiosity, experimentation, internet and media awareness
Why this level will become obsolete: As AI becomes increasingly general and ubiquitous, the most advanced AI technology will be readily accessible through all digital portals, and it will no longer make sense to consider specific AI tools and integrations.

Level 2: Mastery of AI Tool Operation

Effectively employing AI tools demands the mastery of approaches and techniques that are both universally applicable and unique to each tool. These methods change with every new generation of tools, requiring constant adaptation.

Skills required: experimentation, structured operations, continuous learning, basic programming and scripting, "prompt engineering"
Why this level will become obsolete: As AI becomes more sophisticated, operating it will no longer be a specialised task—AI will effortlessly comprehend or predict our requests and needs.

Level 3: Filtering and Curating AI Output

AI output vastly surpasses human output, resulting in an overwhelming volume of information—more than humans can or should consume. The primary challenge lies in identifying the most valuable and highest quality AI output.

Skills required: microeconomics, taste
Why this level will become obsolete: Advanced AI will eventually be capable of filtering and curating output more effectively than any human operator.

Level 4: Managing AI Agents

As AI becomes more powerful, every user will act as a manager overseeing a swarm of ...
How AI could workaround goals if rated by people
by ProgramCrafter — 2023-03-19T15:51:04.743Z

There are some suggestions on how to align AI with human values based on having operators rate AI's actions.

There is always a possibility that operators are unaligned to other people's values themselves; however, there is also a second risk.

1. (Way of tricking the goal) In future, AI could become able to raise children; even now it can significantly affect their beliefs. For example, I think rather big amount of people have ever talked to ChatGPT, some of them have been talking about AI, and someone could believe in GPT's claims. Then, after maybe twenty or thirty years, majority of people would have value drift relative to us, so then AI could choose its goals and objectives more freely.

2. (Way of deception) AI could create artificial humanoid robots, make them look like people (real people's images and even videos can be downloaded from Internet) and somehow make them indistinguishable from humans. Then, by most people's definition, common human values would have to include values of this robots (because no one would know they are robots), which in turn gives AI some degree of freedom.
A tension between two prosaic alignment subgoals
by Alex Lawsen  — 2023-03-19T14:07:53.519Z

Written quickly rather than not at all, as I've described this idea a few times and wanted to have something to point at when talking to people. 'Quickly' here means I was heavily aided by a language model while writing, which I want to be up-front about given recent discussion.

BLUF

In alignment research, two seemingly conflicting objectives arise: eliciting honest behavior from AI systems, and ensuring that AI systems do not produce harmful outputs. This tension is not simply a matter of contradictory training objectives; it runs deeper, creating potential risks even when models are perfectly trained never to utter harmful information.

Tension

Eliciting honest behavior in this context means developing techniques to extract AI systems' "beliefs", to the extent that they are well-described as having them. In other words, honest models should, if they have an internal world model, accurately report predictions or features of that world model. Incentivizing honesty in AI systems seems important in order to avoid and detect deceptive behavior. Additionally, something like this seems necessary for aiding with alignment research - we want to extract valuable predictions of genuine research breakthroughs, as opposed to mere imaginative or fictional content.

On the other hand, avoiding harmful outputs entails training AI systems never to produce information that might lead to dangerous consequences, such as instructions for creating weapons that could cause global catastrophes.

The tension arises not just because "say true stuff" and "sometimes don't say stuff" seem like objectives which will occasionally end up in direct opposition, but also because methods that successfully elicit honest behavior could potentially be used to extract harmful information from AI systems, even when they have been perfectly trained not to share such content. In this situation, the very techniques that promote honest behavior might also provide a gateway to accessing dangerous knowledge.
Shell games
by TsviBT — 2023-03-19T10:43:44.184Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2022/11/shell-games.html. First completed November 18, 2022.]

Shell game

Here's the classic shell game: Youtube

Screenshot from that video.

The little ball is a phantom: when you look for it under a specific shell, it's not there, it's under a different shell.

(This might be where the name "shell company" comes from: the business dealings are definitely somewhere, just not in this company you're looking at.)

Perpetual motion machines

Related: Perpetual motion beliefs

Bhāskara's wheel is a proposed perpetual-motion machine from the Middle Ages:

Here's another version:

From this video.

Someone could try arguing that this really is a perpetual motion machine:

Q: How do the bars get lifted up? What does the work to lift them?

A: By the bars on the other side pulling down.

Q: How does the wheel keep turning? How do the bars pull more on their way down than on their way up?

A: Because they're extended further from the center on the downward-moving side than on the upward-moving side, so they apply more torque to the wheel.

Q: How do the bars extend further on the way down?

A: Because the momentum of the wheel carries them into the vertical bar, flipping them over.

Q: But when that happens, energy is expended to lift up the little weights; that energy comes out of the kinetic energy of the wheel.

A: Ok, you're right, but that's not necessary to the design. All we need is that the torque on the downward side is greater than the torque on the upward side, so instead of flipping the weights up, we could tweak the mechanism to just shift them outward, straight to the side. That doesn't take any energy because it's just going straight sideways, from a resting position to another resting position.

Q: Yeah... you can shift them sideways with nearly zero work... but that means the weights are attached to the wheel at a pivot, right? So they'll just fall back and won't provide more torque.

A: They don't pivot, you fix them in place so they provide more torque.

Q: Ok, but then when do you push the weights back inward?

A: At the bottom.

Q: When the weight is at the bottom? But then the slider isn't horizontal, so pushing the weight back towards the center is pushing it upward, which takes work.

A: I meant, when the slider is at the bottom--when it's horizontal.

Q: But if the sliders are fixed in place, by the time they're horizontal at the bo...
Self-censorship is probably bad for epistemology. Maybe we should figure out a way to avoid it?
by DaemonicSigil — 2023-03-19T09:04:42.360Z

Imagine a place where speaking certain thoughts out loud is widely considered to be harmful. Not because they are lies, spam, threats, or insults, most people in most places can agree those are harmful. No, the thing that makes this place unusual is that those who dwell there believe the following: That even if an idea is true, and unlikely to annoy, coerce, or offend the listener, some ideas should still be suppressed, lest they spread across the world and cause great damage. Ideas of this type are deemed "Risky", and people carefully avoid communicating them.

The strange thing is that there is no authoritarian government banning the speaking of Risky ideas. Rather, the people there just seem to have decided that this is what they want for themselves. Most people reason about the world in such a way that it's simply obvious that some ideas are Risky and if we want to have nice things, we ought to avoid saying anything that could be Risky. Not everyone agrees with this, there are a few outliers who don't care too much about Riskiness. But it's common to see such people rebuked by their peers for saying Risky things, or even for suggesting that they might in the future say Risky things. Occasionally some oblivious researcher will propose a project, only to be warned that it's ill-advised to conduct a project of that nature because it could turn up Risky results. When it comes to Risky ideas, self-censorship is the rule, even if it's more of a social norm than a legal rule.

Of course because of this self-censorship, people in this place find it much harder to reason as a group. You can still think freely inside your own head, but if you need to know something that's inside someone else's head, you're likely to have a difficult time. Whenever speaking, they'll have to carefully avoid coming too close to Risky ideas. Sometimes by making an intellectual detour, they will manage to convey a roughly similar notion. More often, they'll opt to simply discard the offending branch of thought. Even when reasoning about topics that are on the surface unrelated to Riskiness, self censorship still gets in the way, slowing down the discussion and introducing errors. How could it not? It's all one causally-connected world. Yet whatever the cost to good epistemology, the people there seem willing to pay it.

You may consider such a place strange, but I assure that all this seems ...
Feature proposal: integrate LessWrong with ChatGPT to promote active reading
by DirectedEvolution — 2023-03-19T03:41:34.781Z

I've built a little Python app I call aiRead.

Its least interesting feature is that it breaks down text the way I prefer to read it: a few sentences at a time, presented in ticker-tape fashion.

However, I've also integrated it with ChatGPT. Two of my engineered prompts are most useful. One is activated by typing "explain," which generates a rewording, a definition of jargon terms, and a simplified explanation in conversational language. The other is activated by typing "quiz," which generates an interactive quiz on the content you're reading.

The point of these features is to promote active reading. I find them tremendously helpful, such that despite the numerous shortcomings of aiRead, I'm now converting all text I have to read to be compatible with it. When I read LessWrong posts, I put them into aiRead, then generate an interactive quiz to help me understand them better. It is transformative.

I would love to see features like these integrated into LessWrong.

I know the dev team is busy, and they do a wonderful job - this is one of the best forum architectures I've ever encountered already. The only reason I'm suggesting this is because an ongoing, convenient explanation-and-quiz feature in my reading app has completely redefined the way I learn overnight, and I'm excited to see it incorporated into the other ways I consume information.

Note: Anyone is welcome to download and use aiRead, and if you do, I'd be very interested to hear about your experiences with it. You'll need a ChatGPT+ subscription. The way to use it is explained in the readme.

Also, I'm looking for collaborators on aiRead! If you'd like to help me polish this up, or ideally turn aiRead into a browser-based app, please get in touch!
Against Deep Ideas
by YafahEdelman — 2023-03-19T03:04:01.815Z

When discussing impactful research directions, it's tempting to get excited about ideas that seem deep and profoundly insightful. This seems especially true in areas that are theoretical and relatively new - such as AI Alignment Theory. Fascination with the concept of a research direction can leak into evaluations of the expected impact, most often through overestimating the likelihood of extremely impactful outcomes. As a result, we should a priori be more skeptical of research projects that we encounter that sound insightful and deep than of those that sound boring and incremental.

This phenomenon can arise naturally from how ideas are generated and spread. If there are two research projects that are roughly equivalent, but one seems deep while the other seems boring, the deep one will garner more attention and interest. The spread and discovery of research ideas thus has a bias towards profound ideas, as profundity is more memetically fit than its absence. I believe that this bias is fairly strong in the AI alignment community, full as it is with researchers who love[1] interesting intellectual challenges and ideas.

Some researchers might think that profound ideas are likely necessary to solve AI Alignment. However, I'll note that even in such a scenario we should expect profound ideas to be given inordinate attention - as they will by default be selected over boring ideas that are as promising as the average profound approach to the problem. Unless exclusively profound ideas are promising, we should expect bias towards profound ideas to creep in.

Even in a world where profound ideas are absolutely required for AI Alignment research, we should still expect that any given profound idea is very unlikely to succeed. Profound ideas very rarely yield significant results and the importance of solving a given problem should not affect our expectation that any given idea will be successful. In such a world I think exploration is much more important than exploitation - as the chances of success in any one direction are low.

I'm particularly worried about profound research directions like

Natural Abstractions

or

Heuristic Arguments

being treated as more promising than they are and consuming a large amount of attention and resources. Both seem to have absorbed quite a lot of thought without yielding legible successes as of yet. Addi...