trentmkelly/LessWrong-43k huggingface dataset:
Proposing the Post-Singularity Symbiotic Researches
by Hiroshi Yamakawa — 2024-06-20T04:05:38.644Z

The great concern that had long smoldered in the depths of people's hearts—that humanity would be displaced by beings surpassing us in intelligence, which would eventually come into existence. This is now beginning to take on sufficient reality. At this juncture, we must have the resolve to accept this risk and find hope for survival.

Abstract

This paper discusses post-singularity symbiosis (PSS) against the backdrop of the rapid progress of artificial intelligence (AI) technology and the challenge of the arrival of superintelligence, which humanity has never experienced before. I propose the establishment of PSS. In a post-singularity world, superintelligence will likely prioritize self-preservation without considering the values ​​of humanity, and if this scenario becomes reality, it could have devastating consequences for humanity. Therefore, I propose to develop PSS as a preventive and constructive research field that can be comprehensively advanced to increase the probability of survival and welfare of humankind, even under the assumption that humanity cannot control superintelligence as desired. PSS is an interdisciplinary study that does not rely on any particular culture or ideology but has a universal goal of the survival and development of humanity. Its research areas are wide-ranging, including analysis of superintelligence, guidance, and promotion of human empowerment. In this paper, I will discuss in detail the specific research themes in these research areas and their relationship with previous research. Furthermore, I emphasize that respecting cultural diversity and building a global cooperative system are essential to realizing PSS. This article emphasizes the importance of PSS in gathering wisdom and carving out a future for humanity in the face of the singularity, the greatest challenge in human history.

1. Introduction

Humanity is at a critical crossroads with the advent of artificial intelligence (AI) technology, which is developing at an unprecedented rate [Kurzweil, 2005][Bostrom, 2003][Aschenbrenner, 2024]. Rapid advances in AI are making the emergence of superintelligence that will rule the world a reality. Although it is extremely difficult to predict the behavior of superintelligence in a post-singularity world accurately, it is likely that superintelligence will prioritize self-preservation and put consideration for humanity second [Ch...
Week One of Studying Transformers Architecture
by JustisMills — 2024-06-20T03:47:12.509Z
Link: https://justismills.substack.com/p/week-one-of-studying-transformers

I asked Claude already, but he's kind of a suckup, so if any experts find themselves reading, please tell me if anything here is super wrong.

I’ve been writing a lot about the meta level, which I think can be a trap for blogs in general, and blogs in the rationalist diaspora specifically. So this post is going to go into the weeds. Specifically, I’m going to write down all the stuff I’ve figured out about transformers architecture in a week of study. I may do more of these as my knowledge crystallizes.

You probably shouldn’t use this post to learn perfectly reliable information about transformers - I’m a beginner, and am happy to be corrected by readers for stuff I get wrong. But you might find it interesting if:

You’re curious how difficult it would be to get up to speed on transformers architecture yourself
You want a really high level view from a fellow beginner
You’re interested in learning processes more broadly

Or, of course, if you just like reading technical blog posts for its own sake. To these readers, especially, I bid you a warm welcome!

What are transformers?

A transformer is a kind of deep learning network, first introduced in 2017 by the paper

Attention Is All You Need

. Traditional transformers, of the type in that seminal paper, are good for sequence to sequence (or seq2seq) tasks, where both the input and the output are sequences. For example, translation, where the input is some words in one language, and the output is some words in another.

The big innovation transformers brought to the table was leaning harder on an already-existing piece of machine learning architecture called self attention. I’ll get to attention mechanisms later on, but for now just think of them as the transformers secret sauce. Some earlier models used attention in tandem with various other techniques; Attention Is All You Need showed that you could get better performance relying exclusively on attention mechanisms.

Most of the splashiest AI progress of the last 7 years has been on top of transformers architecture. Modern AI chatbots are transformers, as are LLMs (large language models) more broadly. In general, most of the recent gigantic burst of investment into AI has been a race to build ever larger and more sophisticated transformers.

What sort of transformer is an LLM?

I’m glad you asked! LLMs are autoregressive transformers, meaning (roughly) that they take in a sequence ...
What are things you're allowed to do as a startup?
by Elizabeth — 2024-06-20T00:01:59.257Z

The internet has previously seen Things You're Allowed To Do (as an adult) and Things You're Allowed To Do: University Edition. The idea is to list things that are totally possible, often easy, but people often fail to think of because not one of the highlighted options. Things like "run mturk surveys" or "dissect a cadaver".

What belongs on a similar list for startups, founders, and early employees?

Some ideas I've heard so far:

host a podcast so you can invite on people you want to meet
any relevant factory will be happy to talk about how they would make your product.

[Feels like good practice to acknowledge I was commissioned to create this new list. I wouldn't have put this on LessWrong if I didn't think it was relevant and interesting to users, but I'm open to hearing I was wrong]
Chronic perfectionism through the eyes of school reports
by Stuart Johnson — 2024-06-19T17:46:02.229Z

A short post for a short thought. I was sifting through old school reports, and there's only one comment which is repeated in pretty much every report from 3 years old all the way through to 16.

3 years old: "Has a hard time dealing with disappointments. Tends to cry and is hard to console. Always wants to be first and if he is not sometimes he is very disappointed."

5 years old: "Stuart sometimes gets frustrated with his finished results, but he tries his best."

10 years old: "Stuart is mostly a thoughtful and friendly boy, however sometimes his rather volatile temper can get the better of him when he gets frustrated."

12 years old: "[Stuart] is a fragile pupil who is reluctant to make mistakes which may prevent him from putting pen to paper."

14 years old: "Stuart should [...] have the confidence to ask teachers for support when he is not sure. It is vital that Stuart can convey his ideas in writing if he wishes to show the extent of his potential."

It's sort of telling, really. My parents weren't tigers who would berate my every failure - to the contrary, they learned this was counterproductive because I was usually harder on myself.

There's a lot of talk about unlearning perfectionism in the world. It's not such an easy thing to get away from when it's an inherent part of your personality rather than a learned behaviour. I think perhaps the message is to treat it as nature rather than nurture, and lean in, rather than away.

I wonder how many traits there are that, for most people, are acquired responses to their environment, but for a minority are essential qualities of their nature.
Ilya Sutskever created a new AGI startup
by harfe — 2024-06-19T17:17:17.366Z
Link: https://ssi.inc/

[copy of the whole text of the announcement on ssi.inc, not an endorsement]

Safe Superintelligence Inc.

Superintelligence is within reach.

Building safe superintelligence (SSI) is the most important technical problem of our​​ time.

We have started the world’s first straight-shot SSI lab, with one goal and one product: a safe superintelligence.

It’s called Safe Superintelligence Inc.

SSI is our mission, our name, and our entire product roadmap, because it is our sole focus. Our team, investors, and business model are all aligned to achieve SSI.

We approach safety and capabilities in tandem, as technical problems to be solved through revolutionary engineering and scientific breakthroughs. We plan to advance capabilities as fast as possible while making sure our safety always remains ahead.

This way, we can scale in peace.

Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.

We are an American company with offices in Palo Alto and Tel Aviv, where we have deep roots and the ability to recruit top technical talent.

We are assembling a lean, cracked team of the world’s best engineers and researchers dedicated to focusing on SSI and nothing else.

If that’s you, we offer an opportunity to do your life’s work and help solve the most important technical challenge of our age.

Now is the time. Join us.

Ilya Sutskever, Daniel Gross, Daniel Levy

June 19, 2024
Beyond the Board: Exploring AI Robustness Through Go
by AdamGleave — 2024-06-19T16:40:06.594Z
Link: https://far.ai/post/2024-06-go-defense/

Last year, we showed that supposedly superhuman Go AIs can be beaten by human amateurs playing specific “cyclic” patterns on the board. Vulnerabilities have previously been observed in a wide variety of sub- or near-human AI systems, but this result demonstrates that even far superhuman AI systems can fail catastrophically in surprising ways. This lack of robustness poses a critical challenge for AI safety, especially as AI systems are integrated in critical infrastructure or deployed in large-scale applications. We seek to defend Go AIs, in the process developing insights that can make AI applications in various domains more robust against unpredictable threats.

We explored three defense strategies: positional adversarial training on handpicked examples of cyclic patterns, iterated adversarial training against successively fine-tuned adversaries, and replacing convolutional neural networks with vision transformers. We found that the two adversarial training methods defend against the original cyclic attack. However, we also found several qualitatively new adversarial strategies that can overcome all these defenses. Nonetheless, finding these new attacks is more challenging than against an undefended KataGo, requiring more training compute resources for the adversary.

For more information, see our blog post, project website or paper.
A study on cults and non-cults - answer questions about a group and get a cult score
by spencerg — 2024-06-19T14:30:03.818Z
Link: https://www.guidedtrack.com/programs/zpdhrqg/run?source=LessWrong

Hi all! We're running a study examining the differences between cults and non-cults. It would be great if you'd take part! It involves answering a bunch of questions about a group/community/org you are in or used to be in.

In the end, you'll get to see a (very rough) cult score for the group. This study is aimed at helping make our cult score system much more accurate.

Here's the link to participate:

https://www.guidedtrack.com/programs/zpdhrqg/run?source=LessWrong
Thanks!
Workshop: data analysis for software engineers
by Derek M. Jones — 2024-06-19T14:20:57.092Z

I'm giving a hands-on workshop: Data analysis for software engineers this coming Saturday, in central London+online.

The material is based on my book Evidence-based Software Engineering which discusses what is known about software engineering, based on all the publicly available data pdf+code+all data freely available here: http://knosof.co.uk/ESEUR/

The workshop is a British Computer Society event, and since we are charging a minimal amount to cover lunch we cannot offer online for free (you can blame HMRC or the BCS accounting software for having to pay £2 for online).

Register: https://www.eventbrite.co.uk/e/data-analysis-for-software-engineers-tickets-896080399597
Surviving Seveneves
by Yair Halberstadt — 2024-06-19T13:11:55.414Z

Contains spoilers for the first couple of chapters of Seveneves

Highly speculative on my part, I know very little about most of these topics

In Seveneves Neal Stephenson does the classic sci-fi trick of assuming that exactly one thing in the universe is different, and seeing where that takes us. In his case that one thing is the moon has somehow exploded.

And where that takes us is the complete destruction of the earth. As the initially huge chunks of moon rock hit into each other they break into smaller and smaller pieces, and take up more and more space. Eventually this process increases exponentially, the loosely held collection of rocks that was the moon disperses into a planetary ring, and earth is bombarded by lunar leavings for 5000 years:

There will be so many [meteors] that they will merge into a dome of fire that will set aflame anything that can see it. The entire surface of the Earth is going to be sterilized. Glaciers will boil. The only way to survive is to get away from the atmosphere. Go underground, or go into space.

They have only two years to prepare. Which option should they take? The choice seems obvious!

But they respond with the absolutely batshit insane solution. They go into space. And not to mars, or some other friendly location. Low Earth Orbit..

This is a terrible choice for all sorts of reasons:

They are even more at risk of meteor collision there, since all meteors that hit earth pass through LEO, but at least the atmosphere protects earth from the small ones.
There's simply no way to get people up there at scale. No matter how you slice it, at most an insignificant fraction of people can get to LEO. We simply don't have the capacity to send rockets at scale, and two years is not enough time to develop and scale the technology enough to make a dent in the 7 billion people on earth.
To prepare as well as possible in two years, the earth economy will have to keep running and sending stuff up to space. But if people know they are going to die, and don't have any real chance of being one of the lucky survivors, why would they bother? I would expect the economy to collapse fairly rapidly, followed by looting, and the collapse of government structures.
There's a thousand things that can kill you in space, and just staying alive requires lots of advanced technology. If society isn't able to keep a highly technologically advanced society going in

...
Self responsibility 
by Elo — 2024-06-19T10:17:35.268Z

I see a modern puzzle of "self responsibility" which gets tricky quickly. At all times a person has responsibility to themselves. From child to adult, you are “self responsible”. It always applies.

When a person isn't of a sound mind, they are still expected to maintain their responsibility but they may simply be unwell. Unable to be responsible for themselves or their actions.

As a society we prepare for a few times when people don’t have responsibility, for example being drunk or a minor. For minors, we expect parents to care for them until they are old enough to be responsible for themselves, we build schooling institutions to guard children for a while too. For drunks, we make duty-of-care (temporarily) as part of the deal for the people who serve alcohol.

But we as a society also are not prepared to deal with people who are not drunk, not underage and also not self responsible. People who might be able to be self responsible but just aren't. Immature or having a mental break, Society isn’t ready for them.

At this moment in time, there's no one else who automatically assumes personal responsibility for you. There's no service or system that is designed to stand in for self responsibility, nor is it something we can expect to systematise permanently or somehow magically assign to a service.

We can implement some solutions. We can set up expectations. We can plan to support people to live a neutral/positive existence, force them away from negative existences (by outlawing drugs, gambling, violence etc). But ultimately pushing people into neutral/positive existences (aside from being an unhealthy attitude of control over people's autonomy to make bad choices) would be leveraging goodwill from the humans whose job it is to provide services like medical or psychological counselling or other support services. If there exists in the world, a black hole of self responsibility, it's very hard to fill for an individual and especially for a society to do it on behalf of an individual.

Unless someone else comes along and says, "I'll do responsibility for you for a time, until you can take it on". But people who do that are likely to want to benefit from their efforts.

This is independent of the specific condition that may cause a lack of self responsibility.

Beyond basic responsibility, a more mature adult can take more ...
Gizmo Watch Review
by jefftk — 2024-06-18T20:00:02.247Z

Summary: the watches do the basic things they need to and have let
us give our kids more freedom, but there are also a bunch of annoying
limitations and they're 36% more expensive than they look. I'm overall
happy we got them.

Our older two kids, Lily and Anna, are ten and eight, and are mature
enough that they're able to cross streets and handle unusual
situations. They can go to the park or a nearby friends house on their own, but we
do need to know where they are. Over the past three years we've
mostly used three strategies. They can:

Tell us where they're going, and if they want to go
somewhere else they need to come home and tell us first.

Bring a walkie-talkie.

Get a friend's parent to text or call us.

Initially they were very excited about the independence of the
walkie-talkies, but over time they got frustrated with them. While
Julia made them pouches to make carrying them easier, they're still
heavy (for a child) and hard to run with. They also make annoying
noises, because even with the squelch
codes we do sometimes hear static or bits of someone else's
conversation. They're also range limited, about 1/4mi, and some of
their friends or favorite parks are outside that range.

In mid-May we revisited this:

Positive: a friend had been playing at the park, and Lily
invited her to come with us to a family dance. The friend had a
smartwatch and called her parent (with me listening in) and got
permission to come.

Negative: we had a miscommunication where one of the kids went
from the park to a friend's house without letting us know, where they
thought the friend's parent had texted us.

After looking at some options, we decided to get the older two
smartwatches. The key features we wanted were:

GPS: see where they are

Communication: send and receive messages

Other things, like a calculator or clock, are nice to have but not
critical. We talked to other parents, read reviews, and settled on
the Verizon
Gizmo 3. The bottom line is that while there are a bunch of
annoyances we're happy with them, and they do what we need them to
do. You can buy them online, and pay $18.30/month for the watch and
service, [1] with an extra $35 for the first month.

I set up a place to recharge them, on our charging shelf:

The chargers are attached with adhesive velcro:

Advantages:

Easy

...
Boycott OpenAI
by PeterMcCluskey — 2024-06-18T19:52:42.854Z
Link: https://bayesianinvestor.com/blog/index.php/2024/06/18/boycott-openai/

I have canceled my OpenAI subscription in protest over OpenAI's lack of
ethics.

In particular, I object to:

threats to confiscate departing employees' equity unless those
employees signed a life-long non-disparagement contract

Sam Altman's pattern of lying about important topics

I'm trying to hold AI companies to higher standards than I use for
typical companies, due to the risk that AI companies will exert unusual
power.

A boycott of OpenAI subscriptions seems unlikely to gain enough
attention to meaningfully influence OpenAI. Where I hope to make a
difference is by discouraging competent researchers from joining OpenAI
unless they clearly reform (e.g. by firing Altman). A few good
researchers choosing not to work at OpenAI could make the difference
between OpenAI being the leader in AI 5 years from now versus being,
say, a distant 3rd place.

A year ago, I thought that OpenAI equity would be a great investment,
but that I had no hope of buying any. But the value of equity is heavily
dependent on trust that a company will treat equity holders fairly. The
legal system helps somewhat with that, but it can be expensive to rely
on the legal system. OpenAI's equity is
nonstandard
in ways that should create some unusual uncertainty. Potential employees
ought to question whether there's much connection between OpenAI's
future profits and what equity holders will get.

How does OpenAI's behavior compare to other leading AI companies?

I'm unsure whether Elon Musk's xAI deserves a
boycott, partly because I'm unsure whether it's a serious company.
Musk has a history of
breaking
contracts
that bears some similarity to OpenAI's attitude. Musk also bears some
responsibility for SpaceX requiring non-disparagement
agreements.

Google has shown some signs of being
evil.
As far as I can tell, DeepMind has been relatively ethical. I've heard
clear praise of Demis Hassabis's character from Aubrey de Grey, who
knew Hassabis back in the 1990s. Probably parts of Google ought to be
boycotted, but I encourage good researchers to work at DeepMind.

Anthropic seems to be a good deal more ethical than OpenAI. I feel
comfortable paying them for a subscription to Claude Opus. My evidence
concerning their ethics is too weak to say more than that.

P.S. Some of the better sources to start with for evidence against Sam
Altman / OpenAI:

a lengthy Zvi
post
about one week's worth of evidence

Leopold
As

...