trentmkelly/LessWrong-43k huggingface dataset:
Agents synchronization
by Ben Amitay — 2023-03-11T18:41:29.734Z

There is a line in alignment-related thinking, of looking for ways that agents will tend to be similar. An early example is convergent instrumental goals, and a later is natural abstractions. Those two ideas share an important attribute - trying to think on something "mental" (values, abstractions) as at least partially grounded in the objective environment. My goal in this post is to present and discuss another family of mechanisms for convergence of agents. Those mechanisms are different in that they arise from interaction between the agents and make them "synchronize" with each other, rather than adapt to similar non-agentic environment. As a result, the convergence is around things that are "socially constructed" and somewhat arbitrary, rather than "objective” things. I’ll then shortly touch another family of mechanisms and conclude with some short points about relevance to alignment.

Instrumental Value Synchronization

Depending on their specific situations and values, agents may care about different aspects of the environment to different degrees. A not-too-ambitious paperclip maximizer (Pam for short) may care more about controlling metal on earth than it cares about controlling rocks on the moon. A huge-pyramids-on-moon maximizer (Pym) may have the opposite priorities. But they both care about uranium to energize their projects and get into conflicts around it. It then makes sense for Pym to care about controlling metal too – to have some leverage on Pam, that may then be used to achieve more uranium. It may want the ability to give Pam metal in exchange for uranium, or promise to create paperclips in exchange for uranium, or to retaliate with paperclip-destruction if Pam try to take its uranium. Anyway, the result for Pym is that it is now more like Pam in that it care about metal and paperclips, and about things that are relevant for those.

Money

Money is a strange thing, leading people to say strange things about how it works. The most common ones are "It only has value because we believe it does" and "It only has value because we all agree to accept it". But I do not believe that money have intrinsic value, nor do I recall anyone asking me to consent it’s having value. My belief that it has value is only my well-founded expectation that people will want it in exchange for goods, and my agreement to accept it for my work is solely based on ...
Against Complete Blackout Curtains For Sleep
by jp — 2023-03-11T18:29:29.063Z

Epistemic status: moderately confident that something like this is directionally true, but good data or arguments could change my mind. Note the lack of citations — this is based on memory.

Most rationalists I meet believe the following:

Sleep is very important
To get the best sleep, install blackout curtains strong enough to prevent any sunlight from coming in in the morning

I disagree with them. I believe point 2 is false for most people, and that natural light in the morning is helpful.

In my model of the world:

Your circadian rhythm figures out what time of day it is based on the amount of blue light hitting your pupils, and picks a time it thinks you should wake up
It’s important to wake up at a consistent time each day, so your circadian rhythm is your friend

By blocking out all light in the morning, you might get a short term increase in the number of hours you sleep, but I predict you will hurt your sleeping consistency (and potentially soon you will start staying up later, negating your benefits entirely).

If you are the sort of person whose body wants to wake up 3 hours after sunrise, let your body have a little bit of natural light to let it know that sunrise is coming. If you’re the sort of person who's body wants to wake up with sunrise, I predict you’ll do best by waking up with sunrise.

My advice is to take the energy you were going to spend on blackout curtains, and instead spend it on smart lighting that automatically gets redder in the evening.

Exception: if your circadian rhythm is incompatible with the schedules of other people, you may want to black out your windows and use artificial lighting to simulate a later sunrise. For example, you may work in the UK, have coworkers in California, but be a morning lark.
The Power of Intelligence - The Animation
by Writer — 2023-03-11T16:15:16.705Z
Link: https://youtu.be/q9Figerh89g

This video is an animation of The Power of Intelligence, by @Eliezer Yudkowsky

The Sorting Pebbles Into Correct Heaps video, coupled with this video, make a very basic case for the importance of AGI alignment.
Here's most of the pinned comment under the video, also in the description:

The script used for this video is an essay published by Eliezer Yudkowsky in 2007.

Now, a few points:

Sorting Pebbles Into Correct Heaps was about the orthogonality thesis. A consequence of the orthogonality thesis is that powerful artificial intelligence will not necessarily share human values.

This new video is about just how powerful and dangerous intelligence is. These two insights put together are cause for concern.

If humanity doesn't solve the problem of aligning AIs to human values, there's a high chance we'll not survive the creation of artificial general intelligence. This issue is known as "The Alignment Problem". Some of you may be familiar with the paperclips scenario: an AGI created to maximize the number of paperclips uses up all the resources on Earth, and eventually outer space, to produce paperclips. Humanity dies early in this process. But, given the current state of research, even a simple goal such as “maximize paperclips” is already too difficult for us to program reliably into an AI. We simply don't know how to aim AIs reliably at goals. If tomorrow a paperclip company manages to program a superintelligence, that superintelligence likely won't maximize paperclips. We have no idea what it would do. It would be an alien mind pursuing alien goals. Knowing this, solving the alignment problem for human values in general, with all their complexity, appears like truly a daunting task. But we must rise to the challenge, or things could go very wrong for us.

You can read The Power of Intelligence and many other essays by Eliezer Yudkowsky on this website:

https://www.readthesequences.com/
Hoarding Gmail-accounts in a post-CAPTCHA world?
by Alexander Gietelink Oldenziel — 2023-03-11T16:08:34.659Z

You may have noticed that CAPTCHAs have become harder and harder. With the imminent advent of powerful multimodal AI, CAPTCHAS might soon be no longer be able to tell human from AI. The era of easy signup and free accounts might be at an end. It might consequently be much harder to get accounts set up and new cumbersome human-verification procedures may become the norm. Many current security protocols used in banking are already not AI-safe anymore.

One consequence would be that the internet might become less anonymous and less anonymous as accounts will have to be linked to real-world authentication. We already see a centralization of online identities around Google accounts.

A more interesting phenomena might be that accounts made before 2023 may become quite valuable. Does this mean you should stock up on gmail (and other internet services) accounts?

What do you think is the best way to prepare for a post-CAPTCHA world?
Will the Bitcoin fee market actually work?
by TropicalFruit — 2023-03-11T00:02:22.485Z

Based on other posts I've seen on here, many of you are Bitcoiners. While I clearly see the value proposition, I have a few questions about the protocol that haven't been answered, and I figure the lesswrong audience is my best change.

1.) Will the fee market actually work?

Is there any evidence of a fee market working, incentive wise? Gold used inflationary proof-of-work indefinitely. Miners extract value from the monetary network by making more gold. They can increase or decrease their mining efforts due to supply and demand. This yielded a 1-2% inflation rate into perpetuity, and acted as a stable monetary system for many centuries.

Bitcoin... is trying to replace the model of inflation-paid-to-the-miners with one of voluntary fees. Has that every been tried? Are there economic reasons to strongly believe the network will remain secure as the block subsidy goes down?

2.) Will lightning ever work?

My understanding is that the current lightning network is more of a proof-of-concept, rather than an actually useful, scalable solution today. It's almost impossible to settle on the main chain, because the TPS of the main chain is so low. Additionally, there are denial of service security issues and logistical overheads like watchtowers that make it relatively impractical in its current state.

Bitcoiners don't seem bothered by this. Is there a good reason to believe this is just technology in its infancy, and these structural problems will slowly be solved?

3.) Can deflationary money work?

Human beings are loss averse, and most of the monetary systems throughout the world have had slight inflation to offset this. Slight loss aversion = spending too little compared to what you should be. Slight inflation is a pressure to spend more than you'd like, so it offsets the loss aversion.

I'm not sure a human biased economy can actually function on a deflationary standard, because people actually won't spend on things that would economically benefit them, because they're so attached to the appreciating currency.

Trying to get answers to these questions on reddit was abysmal... but we've got a different audience here, so hopefully someone can help me out. Thanks.
Rationalism and social rationalism
by philosophybear — 2023-03-10T23:20:54.513Z
Link: https://philosophybear.substack.com/p/rationalism-and-social-rationalism

As with all my blog posts, these are only very preliminary thoughts. I worry perhaps their scope gives them a sense of self-seriousness I don’t intend. This piece is about floating ideas that might be worth discussing, rather than strong claims to truth.

Cate Hall tweeted:

“In 2017 I was convinced AI timelines were <5 years so I cashed in my 401k and blew away the money and let me tell you this particular form of intellectual consistency is Not Recommended”

I didn’t believe that superintelligent AGI was less than <5 years away in 2017, but I can imagine many reasons a smart person could have thought that. I can also see the argument that, if you believe something, you should act on it. After all, not acting on your beliefs is a form of hypocrisy. Since at least Jesus, there has been a tradition of treating hypocrisy as among the very worst of sins. Hannah Arendt wrote:

As witnesses not of our intentions but of our conduct, we can be true or false, and the hypocrite's crime is that he bears false witness against himself. What makes it so plausible to assume that hypocrisy is the vice of vices is that integrity can indeed exist under the cover of all other vices except this one. Only crime and the criminal, it is true, confront us with the perplexity of radical evil; but only the hypocrite is really rotten to the core.

In the present day, there is a tradition of demanding people who make unusual claims be willing to place bets on them. If the asserter will not take the bet, then presumably they don’t really believe what they’ve claimed, or so the rhetoric goes. Sometimes people will preemptively offer to sake such bets.

But I think Cate Hall is right that the kind of intellectual consistency she engaged in here was a mistake. In showing why that’s the case, I propose to overturn several common and fundamental beliefs about how the human mind and intellectual inquiry work. I also plan to critique (in a broadly friendly way) the social and intellectual movement sometimes called rationalism or internet rationalism. I suppose I am on the margins of rationalism, but I have never felt comfortable identifying with it, at least in part because of the ideas I will develop in this piece.

Aside: Also, a general sense that rationalists have a tendency to discount the importance of hard, slogging, domain expertise, and a sense that rationalists are insufficiently attentive ...
Meetup Tip: Nametags
by Screwtape — 2023-03-10T21:00:49.546Z

Summary

Nametags are nice to have at meetups. When I was planning my second big meetup, I typed “nametag” into Amazon and got

these

. They’ve worked great but whatever your local stationary store or online shopping service suggests is probably just as good. The summary of this post is that if you run meetups I think it’s worth buying a roll of nametags and a couple markers and bringing them to the meetups.

You have now read the basic point of this post. If you want to read on, cool, lets talk about nametags for more words than are strictly necessary.

Detail

The easiest nametags to use are adhesive stickers with a blank front, which people can write their name on using a marker and then stick on the front of their shirt. Variations on this include a printed sheet with people’s names that can be put into a plastic sheath and hung around their neck on a lanyard, or a badge with their name embossed on it which they pin to their shirt. Nametags can also be created by combining an index card and a safety pin. If you’re at all unsure, I suggest you use the adhesive sticker version.

In all cases, nametags help remember names. Typical conversational norms might have an introduction where you say your name at the start, but then you don’t repeat your name throughout. Nametags help associate your name with your face as people can check the tag when they forget.

Nametags can include information other than just names. Pronouns are common in the social circles I frequent, as are online handles like your Discord nickname. In professional circles nametags often also mention what company someone is from. When I run meetups, sometimes I put "Organizer" under my name. I recall one person at a rationalist gathering wrote their name on their nametag and then, underneath, wrote “Ask me about insects.”

For meetups that are open to anyone who shows up, nametags are good for seeing who is a part of the group. If there is a big meetup in a park, someone standing nearby might be part of the meetup or might just be enjoying the day, but if they're wearing a nametag they're probably here for the meetup. They’re also great for introductions; I’ve overheard more than one conversation start with a variation on “that’s a cool name. Where’s it from?”

For meetups that are mostly the same people every time (the “regular crowd”) with occasional newcomers, nametags help the newcome...
Is ChatGPT (or other LLMs) more 'sentient'/'conscious/etc. then a baby without a brain? 
by M. Y. Zuo — 2023-03-10T19:00:36.011Z

I'm curious to see what the views of various community members are in regards to those attributes we typically ascribe to humans when put in comparison to the edge case where it's clearly absent, i.e. babies born without a brain (Anencephaly)

See: https://en.wikipedia.org/wiki/Anencephaly

I personally am not well informed enough about their inner workings to have decided.

The reason for comparing to the edge case is because it's a very low bar to clear, so even minute differences should be more easily discernable.
The humanity's biggest mistake
by RomanS — 2023-03-10T16:30:09.037Z

It seems that the alignment problem is unlikely to be solved before the emergence of the first misaligned AGI.

This means, humanity has made a deadly mistake.

But what is the nature of the mistake, and when did it happen?

From my understanding of the history of the field, I draw the following conclusions.

Unavoidable mistakes

Firstly, there wasn’t a single event, a single mistake that has created the current perilous situation. Several trends contributed to it, each of them seems to be inevitable:

Researchers are fascinated with the idea of AGI (and want to brag about their research)
Gamers love realistic 3d graphics, driving progress in GPUs
The massively parallel hardware for 3d graphics can be easily utilized for artificial neural networks, due to the similar nature of the computation
The Internet generates massive amounts of training data for the AI, and makes it profitable.
As it turned out, AGI is surprisingly easy to create. Feed enough data and compute to a relatively simple artificial neural network, and it becomes smart enough and general enough to beat humans on many cognitive tasks. And it’s trivial to make it smarter: just give it more data and compute.

It means, there is no realistic possibility to significantly slow down the AGI progress by fixing some big mistake, or by slowing down one of the trends.

Secondly, some people and companies have had a disproportionately big impact on accelerating the progress towards AGI. Among them:

Hardware: Nvidia, Google...
Algorithms: Schmidhuber, Hutter, Hochreiter, LeCun, Krizhevsky…
Funding and management: Page, Brin, Musk, Hassabis, Altman…
Cultural influence: Vinge, M.Banks, M.More, Kurzweil…

While AGI was inevitable with or without them, the collective action of these guys probably has shortened the timelines by years, if not by decades, without contributing much to alignment research. And now we're running out of time. As much as I respect many of them, they deserve some blame.

Thirdly, the progress towards the alignment solution was mostly driven by a few random nerds of the “herding cats” variety, with some assistance from a few random philanthropists. There is no virtuous cycle where alignment research gives you gold nuggets which you invest into hiring more alignment researchers. There is no "Google-Inc-for-Alignment". The alignment research community is likely to remain as a "bunch of random nerds", ev...
Operationalizing timelines
by Zach Stein-Perlman — 2023-03-10T16:30:01.654Z

If you're forecasting AI progress or asking someone about their timelines, what event should you focus on?

tl;dr it's messy and I don't have answers.

AGI, TAI, etc. are bad. Mostly because they are vague or don't capture what we care about.

AGI = artificial general intelligence (no canonical operationalization)

This is vague/imprecise, and is used vague/imprecisely
We care about capability-level; we don't directly care about generality
Maybe we should be paying attention to specific AI capabilities, AI impacts, or conditions for AI catastrophe

TAI = transformative AI, originally

defined

as "AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution" (see also discussion

here

)

"a transition comparable to . . . the agricultural or industrial revolution" is vague, and I don't know what it looks like
"precipitates" is ambiguous. Suppose for illustration that AI in 2025 would take 10 years to cause a transition comparable to the industrial revolution (if there was no more AI progress, or no more AI progress by humans), but AI in 2026 would take 1 year. Then the transition is precipitated by the 2026-AI, but the 2025-AI was capable enough to precipitate a transition. Is the 2025-AI TAI? If so, "TAI" seems to miss what we care about. And regardless, whether a set of AI systems precipitates a transition comparable to the industrial revolution is determined not just by the capabilities and other properties of the systems, but also by other facts about the world, which is weird. Also note that some forecasters believe that current Al would be "eventually transformative" but future Al will be transformative faster, so under some definitions, they believe we already have TAI.
This is often used vaguely/imprecisely

HLAI = human-level AI (no canonical operationalization)

This is kinda vague/imprecise but can be operationalized pretty well, I think
This may come after the stuff we should pay attention to

HLMI = high-level machine intelligence,

defined

as "when unaided machines can accomplish every task better and more cheaply than human workers"

This will come after the stuff we should pay attention to

PONR = (AI-induced) point of no return, vaguely

defined

as "the day we AI risk reducers lose the ability to significantly reduce AI risk"

But we may lose that ability gradually rather than in a binary, threshold-y

...
Dice Decision Making
by Bart Bussmann — 2023-03-10T13:01:19.060Z

Epistemic status: I have been using dice on and off in my decision-making processes for a few years and am confident it has been a net positive for me (85%). I believe that experimenting with dice decisions would be beneficial for most people who struggle with decision-making (60%). I have a very limited understanding of IFS and quantum mechanics, so take these connections with a grain of salt. All quotes interleaved are from Luke Rhinehart.

I often find myself struggling to make decisions. It can be overwhelming to weigh the pros and cons of each option and make a choice that feels right. A while ago, I came across the concept of using dice to make decisions. At first, it sounded crazy, but the more I thought about it, the more it made sense. I read the book "The Dice Man" by Luke Rhinehart, which tells the story of a psychiatrist who uses dice to live his life. He assigns various actions to each number on the die and follows through with whatever the die tells him to do. Although the book is pretty terrible, this crazy form of decision-making made me wonder if leaving some choices up to chance could eliminate the stress and anxiety that comes with decision-making.

For example, this morning I gave the die the following six options:

Improve my draft of A Framework of Possible Minds
Improve my draft of A Framework of Possible Minds
Write a post about Dice Decision Making
Call a friend I haven’t spoken to in a long time
Apply for a job
Think of six new options and throw again

(Note that the first option appears twice. I control both the options and the probabilities.)

I believe dice decision-making relates interestingly to Multiagent Models of Mind and Internal Family Systems (IFS). These models view the mind as a system of parts, with each part having its own thoughts, emotions, and desires. To promote healing and personal growth, IFS encourages individuals to identify and communicate with their parts. When making decisions, the different options we consider can represent the preferences of different parts of ourselves. Using the dice as a coordination mechanism, we can assign the preferences of different parts to a number on the die and roll it. In this way, we can give the preference of each part a fair chance. This can help us make decisions that align with our true desires and values, allowing us to coordinate our various parts and promote inner harmony. I often fe...