skysight-inc/HackerNews huggingface dataset:
Truly, amazing work! Not only because of the final results, but also because of the whole process it took the author to bring this to life. If I could upvote this by giving points from my karma, I wouldn't hesitate to easily give a hundred points. Without a doubt, I would classify this on par with "40k HN comments mentioning books, extracted using deep learning" (https://news.ycombinator.com/item?id=28595967), which is the highest-voted "Show HN" project related to hacker news so far with 1359 points.

I'm not in the ML/AI arena yet, so I couldn't fully understand the second half of the article except for having a general idea about Embeddings and their potential, but the first part is what interests me as a software engineer.

Following are some of the challenges the author came across, was able to overcome each of them, and published the full source code.

Downloading HN database

> There's also a maxitem.json API, which gives the largest ID. As of this writing, the max item ID is over 40 million. Even with a very nice and low 10 ms mean response time, this would take over 4 days to crawl, so we need some parallelism.

> I've exported the HN crawler [1] (in TypeScript) to its own project, if you're ever in need to fetch HN items.

Fetching and parsing linked URLs' HTML for metadata and text

> For text posts and comments, the answer is simple. However, for the vast majority of link posts, this would mean crawling those pages being linked to. So I wrote up a quick Rust service [2] to fetch the URLs linked to and parse the HTML for metadata (title, picture, author, etc.) and text. This was CPU-intensive so an initial Node.js-based version was 10x slower and a Rust rewrite was worthwhile. Fortunately, other than that, it was mostly smooth and painless, likely because HN links are pretty good (responsive servers, non-pathological HTML, etc.).

Recovering missing/dead links

> A lot of content even on Hacker News suffers from the well-known link rot: around 200K resulted in a 404, DNS lookup failure, or connection timeout, which is a sizable "hole" in the dataset that would be nice to mend. Fortunately, the Internet Archive has an API that we can use to use to programmatically fetch archived copies of these pages. So, as a final push for a more "complete" dataset, I used the Wayback API to fetch the last few thousands of articles, some dating back years, which was very annoying because IA has very, very low rate limits (around 5 per minute).

Finding a cost-effective cloud provider for GPUs

> Fortunately, I discovered RunPod, a provider of machines with GPUs that you can deploy your containers onto, at a cost far cheaper than major cloud providers. They also have more cost-effective GPUs like RTX 4090, while still running in datacenters with fast Internet connections. This made scaling up a price-accessible option to mitigate the inference time required.

This is the type of content that makes HN stands out from the crowd.

_____________________________

1. https://github.com/wilsonzlin/crawler-toolkit-hn/

2. https://github.com/wilsonzlin/hackerverse/tree/master/crawle...
One of the appeals of doing MQ in Postgres is being able to submit events atomically in same db transaction as the stuff that raised the event

Looking at https://github.com/tembo-io/pgmq/tree/main/tembo-pgmq-python ...how do I integrate the queue ops with my other db access code?

Or is it better not to use the client lib in that scenario and use the SQL functions directly?
https://github.com/tembo-io/pgmq?tab=readme-ov-file#send-two...
It's been ages since I've clicked on The Verge. Surprised its still around.
The easiest way is to change a commit you've already pushed, and that something somewhere else references, and then force push it.
What an untasteful advert.
Huge congratulations to the Chips and Cheese team on what they have achieved. The quality and volume of posts on the site - particularly from clamchowder - has been stunning.

IMHO sustainability of the site should be a primary concern. The team should really do what they think is best, given their time and resources, to enable them to continue to deliver on their mission.
Mushishi is so atmospheric and just amazing

Also for some madcap anime FLCL ... that is quintessential anime for me.
> Most software outside Fedora's own repositories isn't available in 64-bit ARM (or "AArch64"), at least not without some deeper compilation and configuration work. That means no Slack or Steam for the time being.

Really?! Is Slack — a glorified web browser — what people trying to run Linux on Apple hardware are looking for? Personally, I'm happy running Slack in Firefox.

My main concern with the Asahi project is Apple's reluctance to help out. Personally I'd love to give them my money if I they would just stop pushing MacOS and force feeding their app store ecosystem.

I don't mind well intended UX beloved by many consumers, but please make life a little easier on developers.
To be pedantic the Holy Roman Emperors claimed that title before Constantinople fell to Mehmed, which makes the claim even less legitimate.

The Byzantine emperors carried on many Roman traditions, including wearing purple. The imperial birth chambers were decorated in purple (actually a purple stone), giving rise to the term “born in the purple” to describe their legitimate heirs, these days used to describe any old toff.

https://en.m.wikipedia.org/wiki/Born_in_the_purple
Active noise cancellation speakers. You can create small quiet bubble for your head, where you sleep.
Yikes, that was really offbrand for Nintendo also, but it fits within their 90s "Play It Loud" marketing strategy wherein they tried to compete with ow-the-edge Sega and later Sony.
Is the largest root of a random real polynomial more likely real than complex?
Psht, then just make _smart_ AI, instead of this stupid mutual-distructive one.
For the longest time the common advice was that using a database as a message queue/broker was a bad idea. Now, everyone seems keen to use a DB for this instead of tools dedicated to this purpose. Why?
> For that, NPCs need their own motivations, so they essentially play the game too. With needs like maintain their smithy, so they can build quality armor, so they can make customers happy, so they can make money, so they can feed their family. I.e. not just being props

So you want a system like rimworld, and a LLM doesn't really help on that front. It can power the dialogues, but that's it, all the logic you describe would need to be encoded somewhere else.
Yes. But it’s relatively easy to train a model as there is a huge quantity of before text and after PDF available.
It looks animated to me, I don't think a lot of real objects were destroyed. (Not an expert though)
Matrix got ripped apart a couple of years ago and there are some questionable aspects https://securitycryptographywhatever.com/2022/11/02/Matrix-w...
The last octet with ipv4 and the last 72 bits in IPv6 should be ok.
> UniSuper said it lost cloud infrastructure in “two geographies”, a configuration it believed should have kept it safer in an outage situation.

But in the end, it's still a single system. As can be seen here. So it might help you when the DC burns down but it won't protect you from software bugs of your cloud provider.
They probably get a ton of GCP credits.
Animal Well Released
I know your statement is sarcastic, but it's disturbing how common this sentiment actually is.

It assumes that humans are unchanging, inflexible automata whose actions can be predicted entirely by what they have said or done, let alone that it doesn't consider the fact that they may simply be a facade. In reality, it is their unstated framework of thinking that guides their speech and actions.
It was totally incongruous when it was made as well. Buying a computer from a company was never an act of rebellion.
Berty: Secure P2P message app that needs no internet access or trust in network
> I can recycle it in my fireplace in about 30 seconds

not disagreeing w/ anything else (beside responding to a sarcastic comment, usually 'caring for the children' [esp. out of place] is a decent giveaway + repeating 'inexpensive'); however burning stuff is not recycling. If anything it releases all the carbon (CO2 + CO) in the atmosphere, compositing in the ground is a tad better option, but the lacquer might prevent that part... for a while.
It says that in the article, that in F1 they just pee in their suits if they have to.
Probably a mix. The paint is likely real. The eye popping rubber ball emoji is likely CGI.

Edit: I take that back. My money is on full CGI. I can’t imagine how a brass trumpet would break when placed above an arcade with no physical deformation on the arcade below it.
Now just imagine what we could do with a purpose-built ML model!!!
> "1984" I'd say was edgy in a rebellious way as you point out, which I'd argue gives it more substance. The sledgehammer hitting the screen isn't even the focus, it's the climax to a sequence that carries more of a meaningful message than "wow look how much functionality we fit into this thin shell."

Well, it just wasn't focusing on empty slowmo destruction of a big screen. It had an emotional message behind it. A bit like original Star Wars vs Rebel Moon.
Tangential to this, there's also https://native.live/.
The second example was more unfamiliar, especially with the bespoke `{|h|...}` syntax. But yes, the first example looks...the same as a bash for loop.
Trip C++Now 2024 – think-cell
A common mistake:

Exactly once delivery is not the same as exactly once processing.

Exactly once delivery is an impossibility; see the two generals problem. Exactly once processing is usually at-least-once delivery coupled with deduplication or an otherwise idempotent action
I didn't understand why people kept recommending this until I spotted that the director is Mamoru Oshii, director of GitS. It's perfectly reasonable as a standalone film without having seen 1. And it's good in the same ways as GitS; mix of action, political thriller, and lingering artistic set piece.
SpaceX's satellite internet surprises analysts with $6.6B revenue
Indeed.

Anecdotal example: I'm currently living above a cafe, with an entire floor (with an office in it) between us and them.

When they drag the tables and chairs around, we can hear it easily, because it sets up a vibration through the structure of the building. Took us a long time to realise where the noise came from — it sounds like it comes from above, but the flat above us has been unoccupied on some of the occasions we heard the noise, and we were only able to confirm the real origin by me being outside the cafe watching them while my partner was inside, and with the two of us on a call so we could directly observe that what the cafe staff were doing correlated with the noises he was hearing.
Ask HN: Plagiarism Horror Stories
The recently released, controversial iPad "Crush" ad strongly resembles LG's 2008 advertisement [1]. Yesterday, there was a post [2] from a YC backed company that submitted a product that, many say, looked like a carbon copy of Linear, and even in their own words:

> We, too, are fans of Linear and appreciate its user experience (UX). After speaking with a lot of people, we concluded that it’s top-notch. Consequently, we wanted to give that experience rather than attempting to reinvent the wheel. [3]

Too many "plagiarism" (alleged or real, not for me to judge) stories within a day, so I wanted to use this occasion, to ask for your plagiarism horror stories:

* when you were shamelessly copied

* when you accidentally designed something that looks very much like an already existing design

* when taking inspiration from someone else vs copying elements of design became murky at your job

* when someone asked you to copy a competitors approach and you felt conflicted, or you had to reject doing so

* when you were accused of stealing a design even though you haven't

* when following what works for others is just the name of the game ("stories" and "shorts" taking over Instagram, YouTube, Facebook, and even LinkedIn within a matter of months, spreadsheet apps looking the same, etc)

[1] https://news.ycombinator.com/item?id=40314122

[2] https://news.ycombinator.com/item?id=40290735

[3] https://news.ycombinator.com/item?id=40306240
Absolutely nothing the author said is even remotely controversial in the cryptographic community.

MTProto is weird and countless choices made in its design are bizarre with no clear rationale. Throwing in confusing and cryptographically unnecessary steps with thin rationalizations is par for the course.

Its authors have specifically chosen an approach that all but guarantees lots of “innocent mistakes”.
Brain-inspired computing with fluidic iontronic nanochannels
Alienating artists should be off the fucking table for Apple, for whom creatives are the core customer base.

But it's 2024, everything is enshittified, God has forsaken us.
Compromise at scale, or how all work is flawed
Is this a “we didn’t consider it” problem, or a “it’s part of how we build these days” problem?
I remember back at the 2017 IAC, Elon saying on stage that "we think we've figured out how to pay for it", referring to how they would fund Starship flights to Mars. Starlink was the solution to pay for the Mars settlement, so exciting to see it has been successful at generating revenue for this cause.
So, it's not for me, especially given the examples. As usual with these kinds of things (fish, zsh, etc) this really has everything to do with taste, especially since bash has such a difficult syntax...at least for those who aren't used to it. Even to this day, years later, I have to look up bits and pieces here and again if I don't use some of the ${}-type substitutions frequently.
Indeed. But then they’d be ‘sticking it’ to China too. A welcome outcome, but naive expectation.
Even if you do nothing 'fancy' like Nagle, corking, or userspace building up the complete buffer before writing etc., at the very least the above should be using a vectored write (writev() ).
> Is it really wrong to lie for the benefit of society?

In ideal world or in real life? I don't believe anyone who speaks from society's point of view.
Alice and Bob are not inclusive
> Similar to the PR descriptions, I'm concerned that a tool like this would create an illusion of security against the "simple" problems (leaving reviewers to focus on the high level), whereas I'd hope that the human reviewer would still read every line carefully.

PR descriptions should be about intention, not what changed (what changed is in the PR itself), so I agree there.

With the other stuff though, if you treat it as a linter, that might be fine? The problem is when you justify getting it as it'll save you engineering time, but it won't actually do that significantly unless you have a lot of people using it.
I’m referring to RenTech’s well known Medallion fund, which I believe is now only available internally to longtime employees. Even in the article you linked, it says this fund has “continued to shine”.
The urination system is similar for technical divers, and its viewed as part of the safety system. Trying to hold it in adds stress which in turn could lead to very deadly mistakes.

If you ever see a diver with a drysuit, look towards one of the legs. If there is a plastic nob there then it is likely the pee valve.
> not requiring a hosted service
All of them, due to usb hub issues.
The scary thing with the future of weapons is that they seem to increase plausible deniability thus changing the game theory equation to one of more aggression.

The basic tenant of a lot of the games we play are of attributing past actions to another agent. If there is no attribution, playing fair makes less sense.

Hacking, media manipulation ops, and commodity drones can be used in a way that does not reveal the origin. That means that there cannot be retribution or, more perversely, it can mean incorrect attribution and retribution.

This lowers the cost of aggressive acts substantially.

I don't have a solution to it but I hope there is one.
Sioyek developer here, we don't have the exact feature you ask for, but we have an alternative.

Instead of modifying the PDF file (which kind of is against the point of PDF files in the first place, the point is the file looks the same in all devices) we have a ruler which can automatically highlight the line being read. If the line is larger than the window, then we move horizontally so the next part of the line is shown. If the line is such that it is larger than the current window but is smaller than the next horizontal movement, a red notch is displayed in the ruler which marks the location of the line which will be moved to the leftmost side of the window.

I know I explained it poorly, here is a video which should clarify: https://www.youtube.com/watch?v=ZYUYMhvwN3U
You do if you think the gap between this cliff and the next can be jumped
Now this is a matter of taste, but I love the slow moments where the film just pans over city scenes with no dialogue. Just a little time to breathe and look at the art in the middle of the action. It's a film that posits a bunch of questions about human vs machine and then declines to answer them in favor of letting you think about them.
ditch the google oauth only signup
Do you have a source for him living in Russia? He may be an honorary researcher, which is more likely, considering the two of them escapes Russia years ago with no intention to return.
If you had money there, did you file a claim with the law firm that emailed you?
That doesn't work since you cannot pass that as a single argument to a function.
Creating a Muti-Algorithm Simple Proof-of-Work Library in Rust
> nor is [the iPad] sufficient to make music.

I mean that's just a nonsense statement. You can say "make music (that I don't like)" but you 100% cannot say that an iPad is insufficient to make music when thousands of people do that every day and tens of thousands of people enjoy their output.
Cowboy Bebop: The Movie

Your Name

Violet Evergarden

Perfect Blue

Paprika

Metropolis

There's a bunch of great animated movies.

If you're referring to series then among the best Seinen classics are shows like:

Berzerk (1997)

Monster (2004) [My personal favorite]

Cowboy Bebop

Steins:Gate

Vinland Saga (newer)

Frieren (newer)

There's a lot of Shounen classics as well, my favorite probably being

Full Metal Alchemist: Brotherhood

granted there's several tropes in it that aren't going to be as enjoyable to watch on repeat for older folks. But the storytelling and plot is practically like Breaking Bad but in anime form.
I think there's a reasonable C) which acknowledges that there are multiple jobs where people have an important reason to just have to pee on the spot, and also ridicule gamers for implying that their game is comparable to that.
why did you stop playing?
This scenario does not make any sense. Should the pandora's box of nuclear use be opened, it's not getting closed. And in this context, large scale conventional forces aren't much more than sitting ducks that would just be met with further nuclear strikes. It's for this reason that there's few, to no, scenarios involving nuclear weapons that don't result in global nuclear war, and thus the end of the developed world, if not of humanity.
Exactly my thoughts - this ad does very little to invoke the desire for the product, unlike many other Apple ads.

It's not like Apple has forgotten how to make such ads - the recent one for iPhones with family members asking to not be let go while the owner tries to delete photos represented a familiar experience of people trying to free up storage, and how they wouldn't have to do that if they bought a new iPhone.

On the other hand, this ad just shows stuff being destroyed, just like some of those useless Youtube videos which shows perfectly usable stuff being destroyed under the pretext of "ASMR" or whatnot. Not only is it very difficult to watch as someone who didn't have a lot of money and was taught to make careful use of it from an early age, it just invokes negative vibes, as if possessing a musical instrument is something to be ashamed of.
> If 80% of the tests are passing, then another week of work get 2 more passing, that's progress.

Sure. But that is the sort of information you get by ... talking to the responsible dev.

You can get the dev to describe their progress in any one of a myriad of ways. But you aren't escaping from the fundamental issue here of the dev self-reporting and management ultimately having to have confidence in the dev's ability to self-report. There is no more information for non-technical management in this metric than if the dev says "I think I'm about 60% done".

I'd have more respect for a dev that estimated this way than one who just guessed at random, but it isn't making it easier for non-technical people to gauge progress. It is actually running a serious risk of backfiring, because on the date that the metric suggests the project should complete, the dev might quite reasonably discover something that needs another X amount of time to work on something that the test cases didn't cover. Then management may well feel deceived because it turns out the metric wasn't tracking progress accurately.
You've missed my point if you think I've made yours for you.

I'm not saying closed source is a silver bullet.

I'm saying OSS also isn't a silver bullet, it doesn't find everything because there's not enough interest in doing this work.

The Log4j example alone, given it took 8 years, is enough to demonstrate that.

Everything is an illusion of trust, nothing is perfect; all we can do is try to align the interests of those working on projects with the interests of society — which is so hard that it's an entire field of study called "politics".
This is all really silly. Don't people have anything better to get upset about? I wish companies/Apple didn't cave every time a couple of people start crying.

It must be really annoying to work in such industries, you spend a large amount of time making something and then have no idea if people will get upset about it or not.
MITM isn't offensive either.

It's bikeshedding, we have climate change, rise of extremism, wars around the world etc.

BTW do all people identify as person?
You’re just looking for things to feel emotional about now. “Even if it was CGI”. That’s really just leaning into an uncharitable interpretation for the sake of it.
This is the exact same thing except you're visiting two arguments at a time.

Guess what, the same syntax I gave supports exactly that as well.
These particular hair cells dont die, so they think the treatment will persist, see https://news.ycombinator.com/item?id=40313256
and perhaps that's why today's shells are not very far from the shells 20 years ago.
This is a post without a clickable link.
Why the hostile tone? I don’t see anything on the website saying you should do it. You could, though, up to you. I doubt the authors had you in mind specifically when they wrote the tool.

They made something new and non-detrimental and are sharing it with the world. It’s fine if it’s not something you’re going to use. I won’t either, but I’m failing to see the point of ridiculing their effort.

Your example seems unfair, too. It’s not like that is the only use-case on the homepage.
No I didn’t, I provided two options and asked which case it was. I could’ve assumed OP had used the wrong abbreviation.
There was very little "animé" in North America. If you prefer your animation with an acute accent, you're better off looking in France.
I think the ad is great and funny.

It says: you don't need those anymore. Just buy the iPad. Look at all this space you'll save. All the items you won't need to buy. Minimalism, utility, fun.

I think people who get offended by seeing objects being destroyed are being hyper-hypocritical, because I bet you all replace your phones every few years, replace your cars every few years, replace all of your stuff even if the older versions work fine. But here you get offended. Take a good look in the mirror is all I can say.
This is a market where shame and apologies still have significance.