skysight-inc/HackerNews huggingface dataset:
I think the best way of looking at this is arrow-format = feather-format

I think the format layouts are the same, the main difference between them being compression. Compressed data can in many cases be faster than uncompressed data to read/scan, (https://stackoverflow.com/questions/48083405/what-are-the-di...) so the cpu transform above is simply an uncompress step, which is notably simpler than what would go on with Sqlite to transform the data to a sqlite structure.

What we're looking at here in this test is a direct-data-access pattern (DDA), which is important as you can avoid ETL caching steps eg. Parquet->Postgres (which have an ingestion time) if you can access the data quick enough for your use case, and if the data is on say s3, you can have multiple (parallel) readers onto the same data rather than a connection pool for databases.

It also allows joining different larger-than-memory datasets efficiently, and avoids much of the infrastructure costs for something like Presto (Athena has per-query costs instead of infrastructure costs).

What I'd have liked to see, would be a Vaex/DuckDB benchmark, as the main differentiation between them appears to be SQL vs df/linq/dyplr/Rx semantics.
Hm, that seems roughly analogous to ArgoCD?
Well, it describes people like GP as "writing a recipe for a horrendously unimaginative cook." It's telling them that they're writing clunky, minutia-obsessed instructions, instead of elegantly orchestrating a collection of mathematical functions.
We're exposed to billionaires and things priced in the billions all the time through news and media, but we forget that it's still an exceptional amount of money for a very, very limited few. Like how Instagram shows you all the exceptionally lucky people on holidays and with expensive cars and incredible beauty, so you get to feel like it's ubiquitous and you're just missing out. But the truth is that almost everyone just goes to work everyday, looking normal, driving normal cars, even the influencers.
So that you can prove that your institutions and systems are not discriminating.

How do you know that all members of a given minority aren't being pressured to quit if you don't ever look at the numbers?
They could probably have a much bigger effect by working to stop cryptocurrencies instead of supporting them.
Just because it's cleaner doesn't make it clean. My point is that we should be asking questions about the impacts so we can minimize the negative ones.

For example, where is the energy coming from to charge those batteries? Even if we say wind or solar, those devices end up in landfills. Maybe we should look at making these devices recyclable as an integral part of the strategy.
See fracking. Unfortunately, it was so profitable they irreversably poisoned the ground water to do it.
Hello HN! A while ago, I built a tool to show the top 100 stocks discussed on Wallstreetbets and posted here. That went on the front page and gave me motivation to build more of these stock market analytical tools.

I know there are too many out there, but I plan to build things I would use myself. As part of this, a friend and I created this tool to answer questions like "What If I Invested in GME in 2019 etc". You can use this tool to check historical returns for any stock in the last 20 years.

Also taking requests for any stock analytical tools you think are missing in the market, we are happy to build it for you :)
> HN is one of the few (maybe the only) forum that still tolerates some intellectual discussion about issues that other sites would ignorantly label as conspiracy / misinformation and ban discussion

What kind of places do you mean?

I've been part of many online communities, and never once has one banned any type of discussion. At most, they would ask for off-topic conversations to be restricted to specific forum threads.

HN is probably the most heavily-moderated community I've been a part of. I'm grateful for that, frankly--I don't think letting every comment section devolve into a political debate is healthy for any individuals, let alone the community here.

I've actually noticed lately that, even here, more threads feel like arguments (name calling, derision of opinions, etc), and fewer feel like discussions. Everywhere else (FB & Reddit, in particular), I feel like there's virtually no discussion, and almost entirely arguments.
Not going into specifics, but some of my early career was working for the people I described. One of them was part of the mafia. They weren't actually that bad to work for but they screwed over a lot of companies and organizations. There were far worse narcissistic sociopath criminals and as one could expect they had zero ethics and all that entails. In some cases these leaders moved in from acquisitions and mergers so I didn't really get to pick who I worked for but rather who I left. With time I learned to research companies and their leadership before starting and this vastly improved my well being.
facebook is a great place to work if you’re a sociopath, you’re ambivalent about this whole ‘human family’ thing and you really just want to make a lot of money not working too much, or being too productive. facebook has to massively overpay if they need anyone other than junior.
Also don't forget, many people don't want to live in a crowded, noisy, dirty, disease ridden city. You can count me as part of that group.
I enjoyed this comparison, thanks! Here is a related generally R-centric comparison that you might enjoy of DuckDB, dplyr, data.table, etc. applied to five data-sciency problems I wrote up a few months ago: https://github.com/bwlewis/duckdb_and_r
Honestly, even though I don't like the insane power usage of PoW cryptos, I don't like the waste of the normal economy either. There's so much waste ... but anyways that would require its own thread I guess.
...that's a joke, right?

Have you seen the average enterprise's stack?
Exactly: Bitcoin is using as much energy as the entire country of Sweden in order to do work that could be done by a single desktop PC from 2004.
When Facebook/Zucker said in around 2019:

"I don't think most people want to live in a world where you can only post things that tech companies judge to be 100% true," Zuckerberg said during a nearly 40-minute speech at Georgetown University.

He was right. He then said:

"People no longer have to rely on traditional gatekeepers," Zuckerberg said. "I actually believe that the much bigger story is how much these platforms have decentralized power by putting it directly into people's hands."

That is what made facebook the devil. The media have since then been manufacturing facebook's demise. However, outside the mass media, none of this is actually happening. There's no exodus away from facebook. Facebook hasn't even seen a blip or anything. It steadily has been rising in popularity.

What you're seeing is an inflection point of the media. They no longer can peddle lies and conspiracy theories. Instead of coming back with journalistic standards, they have instead decided to just attack the entity that's revealing their lies. They want to make themselves the arbitors of truth on Facebook so they can keep going.
I've always thought that the Missionaria Protectiva was coolest concept in Dune. The "religious engineering".

The idea that over thousands of years the Bene Gesserit have seeded fake prophecy and religion throughout the universe to "primitive worlds". A Bene Gesserit visiting one of these worlds is trained to fulfill these prophecies and basically weaponize the locals.

I don't know why it blows my mind but it does. Has modern scifi used this idea anywhere?
So, let's play that out. Suppose 90% of miners (hash power) collude (as GP posited), for example validating blocks with a larger coinbase (mining reward). The "good" nodes don't accept these (but the "bad" miners can trivially have many nodes that do accept them). But now the "good" hash power drops to 10%, so only every 100 minutes a "good" block is mined (suppose this happened just after a difficulty adjustment), and that goes on for 20 weeks. In the meantime, the bad nodes and miners carry on as before, pretty much, but with their malicious change incorporated. If they're bored, they can devote some of their hash power to double spend attack the "good" chain. Which chain will come out on top?

Sure, it's unlikely. But not impossible. Just as runaway inflation in a reasonably managed fiat currency is not impossible. Just unlikely. (And, yes, hyper inflation has happened historically. Similarly, rewriting of the immutable Ethereum chain and BTC forks have happened historically.)
It's always a pendulum. When we're lucky it's a "spiral", where when we swing back to the first pole we do it better than the time before. We're not always lucky.
Bitcoin uses a lot of energy. Power is (arguably, and not always)[0] gained or exercised through violence, if not of the state then of the large entities that generate that power, be it coal, natural gas, hydro or even solar.

[0] https://hbr.org/2021/05/how-much-energy-does-bitcoin-actuall...
Brings back fond memories of computing at the time. I remember saving up for a 3dfx card and being amazed at how much of an improvement there was.
> More than 80% of Americans have coronavirus antibodies acquired through infection or vaccination, according to a new study of over 1.4 million blood donations across the U.S. [0]

Just to add some additional detail: The article explains that 83% had antibodies as of May 2021, and that number should be expected to have increased based on additional vaccinations and delta wave infections. Their research will continue until December 2021.

I agree with you that the response of many governments to this does not appear to be following the science.

[0] https://www.miamiherald.com/news/coronavirus/article25398704...
How to Do the Most Good
I agree, it's too narrow to be used for general remote procedure calls, and an incredibly primitive database compared to the alternatives. Imagine having a web socket directly to an RDBMS being able to call any stored procedure, you would do backflips around teams arguing over REST API:s.
Zillow's Confusing Keyword UX
The worst enterprise stack you have ever seen is most likely at least a million times more efficient than bitcoin.

That is not an exaggeration, and is more likely to be a severe understatement.
Same here. Another advantage I've found is that I can have a virtualenv with my particular pelican version and never feel the need to update. There are no moving parts on the server, so as long as the HTML / RSS output works, everything is fine. At this point I don't like dealing with the (necessary, expected, I'm not critizising that) bits that break between versions.
Show HN: Video Generation from a Text
You would think such blatant contradictions work as an argument. It doesn't if it attacks belief and dogma.
It's not really a NDA government default response, it is also the de facto default response for any alleged government mishap. It straight up dates back to the Emergency days.
The weight issue is a mixed bag. It can improve handling in some conditions such as bad weather. Regenerative braking makes up for the energy cost fairly well.

The cost is higher up front but lower overall. The comparison is also only fair if you compare similar cost/luxury classes of car. It's not fair to compare a Tesla Model S to a Honda Civic. Instead you'd compare a Model S cost to a mid-high end BMW, Mercedes, Audi, etc. You'd compare a Honda Civic to a Nissan Leaf or a Chevy Bolt.
> the gravitas house Atreides' arrival seemed to have, when they came to Arrakis

In the modern age we don't have much 'pomp and circumstance', but in an aristocratic society that generally tends to be more prevalent. It helps to show people's place in the hierarchy of things.

See for example the "Sir Jean" scene from The Last Duel (set in 1386):

* https://www.youtube.com/watch?v=lEoUgrOxHfc
That is true. However much of that stuff can basically be piled and is not really damaging to the environment. Lithium is usually only about <5% of any given rock.

That is far better then blowing tons of things into the air. Its just a stack of crushed rocks.
Microsoft ramps up plans to make its data centers less thirsty
> Great Depression could have been averted by throwing $100 bills from helicopters

A lot of people don't know this, but the main reason Calvin Coolidge suddenly, and without warning, decided not to seek reelection in 1928 was that he couldn't find any helicopters. Hoover was set up.
What if the majority of people running nodes disagrees with the majority of hash power holders? Running a node is cheap.
The Ownership Dilemma with ML Pipelines in Production
That's actually quite an insightful piece. As it says, “Strategy” is a nebulous concept, but so are many articles about how to think strategically.

I find asking senior people about their vision (no, not their eyesight) is a good way to understand the background and context to many low level decisions.
You setup a read-replica so that the production database isn't hit by analytical queries.

And then ingest that read-replica to a column store data warehouse so that you don't have to think about reliability until someone starts yelling about speed or cost.
> This was never the intention of REST.

Yeah so REST doesn't support what people are doing or what people want. So let's not use it if we want to succeed.
Because mining bitcoin alone consumes more power than the entirety of the Netherlands. And all that for a very low amount of transactions.
gRPC if you need some sort of static typing over https and don't want vendor lock-in. SOAP/XML works too. It's been a while since I looked which open source toolkit supported it. Problem with SOAP/XML is that although its not supposed to be "Java specific" it is mostly "Java specific".
The Soviets didn't have pandemics, but look at the measures imposed by the Chinese for a hint.
It would be funny if Facebook just launched their own news organization and used it to drag MSM through the mud all day. There's endless content to work with to that end. I'd rather see them eliminate each other.
> nearly complete vaccination

Are you being dishonest or are you really that poorly informed?
> from whatever language and framework you like

Eh, not really. Python SOAP clients are bad and have always been. In reality, SOAP works well only when using platforms sold by commercial vendors, who invested heavily in producing tools for it in an attempt to generate lock-in for their tools. Which means interop issues between platforms, for anything beyond the trivial case, are rife.

Whereas REST is so basic that you literally need nothing more than a standard HTTP client (and a JSON parser, these days; back then it would be some hand-rolled XML or even just encoded form-like data).
There is something very similar between HATEOAS/smart-clients and "semantic web", right? They are both part of the same kind of vision/dream. Or maybe semantic web is actually an attempted implementation of HATEOS?

This connection just occurred to me today in these comments; maybe because I hadn't thought about HATEOS in a while.

I still work with a lot of people who don't think semantic web is dead...
Maybe you misread me.

I said the the protocol can be stateless but the app itself, towards the user, should carry state – where this makes sense. E.g. in the cases I described.
For me its just that you have to get through a lot of world/universe building before anything really happens. I struggle with books that want me to learn about insert alien history before telling me why i need to care about this specific alien race.

But I'm going to read the book now. Normally having a visual to put to the words helps me get through that stuff.
> The Fed cannot do helicopter money. It is not connected to the "real" economy directly, only through the banks as middle-men. Helicopter money comes via fiscal policy.

Exactly. All those people talking about money printing don't understand that money doesn't exist in a vaccum. Every dollar created is backed by one dollar of debt which ads up to a net worth of $0.

So the Fed cannot helicopter money for the obvious reason that they cannot ever pay that money back. The government can borrow and spend money into the economy because it can tax its citizens. The money will come back one day.
Afghanistan’s Pivot to Crypto
Have used Octal for over two years--been mostly happy with it.

* High-quality reading experience. Appends the title and other helpful info to the message if you want to share something, and provides easy options to share the HN discussion or the original article. The UI also prioritizes reading the discussion first which, let's be honest, many of on HN are known to do. ;)

* It mostly stays in sync with the front page of HN (and you can keep scrolling down, making it easier to read stuff that has fallen off the front page but is still valuable).

* The search function finds what I want about 80% of the time.

* You can log in with your HN account and see account stats, I sometimes see issues with the account stats (recent submissions, karma, etc.) staying up-to-date, but that's not a big deal to me.
Why don't we have signing keys for publishers yet?
That stopped with 9/11.
> People like to map REST to CRUD because GET looks like a SELECT, POST looks like an INSERT, PUT looks like an UPDATE and DELETE looks like a DELETE.

> But there's nothing set in stone about those mappings.

People use those mapping because that's what is actually happening further down the stack, when you call your RDBMS in the end.
You have a delusional believe in the power of lobbying. Establishing a mine in the US is incredibly hard. Even in places with existing mining, establishing a new mine is a major undertaking.

Mines even in mining friendly places is very, very hard. Doing it mining unfriendly places where you have real popular opposition is bordering on impossible.
Who would have thought a sci-fi film with aliens could be about motherhood/parenting? :)
> This is incorrect. Replace "miners" with "mining pools" and they're closer to the mark.

It is correct. They are talking about miners not mining pools.
I love annotating shows and movies, several of my friends looked at me so strangely when I was taking notes while watching NGE but sometimes the plot is so complicated it necessitates it to fully understand and appreciate it. The first watch is always regular though, if I come out confused I'll rewatch it.
The UK considers the USA a strong ally in intelligence and doesn't view it as a threat.
Even if you have a top 1% home theater set up (and I do), the HBO Max release uses the cinematic aspect ratio (2.40:1?) with giant letterboxes top and bottom throughout, which cuts out a giant proportion out of the scenes filmed on IMAX cameras. I watched both a day apart, and the home video release is barely the same film (I exaggerate, but it's almost true).
Where did you read that? Methane which already powers 40% of the US electrical grid can be created and then turned back in electricity in a Electricity→Gas→Electricity process with a round trip efficiency of 30–38% [0]. There is no way you'd need 7x production when you can use surpluses to make methane or hydrogen.

[0]: https://en.wikipedia.org/wiki/Power-to-gas
> Currently, you can only chop them up into atoms worth 1e-8 BTC (satoshis), but there isn't much that needs be changed in the code and the protocol to get that to much finer grained.

Such a change would be a hard fork and, as you know, we don't do that in Bitcoin. But, no worries because we have millisatoshis in Lightning. Because Bitcoin scales in layers, each layer can implement more and more divisibility as required.

https://bitcoin.stackexchange.com/a/69704

As you said, bitcoin is infinitely divisible, therefore the world economy can run on 1 btc or 1 satoshi.
I think that Intel is using TSMC because Intel having issues with their smaller process nodes. It probably helps that their lead GPU guy, Raja Koduri[0], worked at ATI (later AMD), so he's familiar with how TSMC does things (because TSMC has manufactured all of ATI/AMD's GPUs).

[0] https://en.wikipedia.org/wiki/Raja_Koduri
Actually getting COVID can cause permanent damage to the heart, lungs and blood vessels of some people, which is thought to be the cause of so-called "long covid" which some people experience.
we don’t have a democracy. the will of the people no longer holds.
With RPC you can call any function by arbitrary name. That is by definition the most powerful protocol ever. Any restricted grammar will take away from that.
When to Skip UX Research
I don't think it progresses the conversation to refer to cogs in the wheel as sociopaths, as if they, as individuals, have any ability to alter the behavior of the company they work for. What is the most compelling argument that individuals are ethically or morally bound to something other than a transactional relationship with their employer?
> I ended up with my current IDE font

Same for me, even the same font as for you, Fira. I just find it very clear and simple and it supports a great set of ligatures. A lot of other fonts I didn't like at all. So maybe I subconciously chose this, but I feel it's also just a reflection of my demands for a coding font.
> war

Given some folks are complaining about the "white saviour" trope in Dune, it would be interesting to see Dune Messiah be made. It's not like Herbert wasn't aware of the idea, given what he knew about (e.g.) T. E. Lawrence ('of Arabia').
These QR codes are the most obvious things to make, but they're also least useful as proof you've got the keys.

Certainly if you have a private key and you want to prove that you know the key, you can trivially make documents that only somebody with the key could make, that aren't documents any system would give people who don't have the key and yet also aren't useful fictitious documents if you're acting as a whistleblower.

That's what was done when a certificate reseller emailed the private keys of their customers to the CA they were reselling - for some reason that's unclear. The CA minted CSRs that showed they now knew the private key, without revealing what it is, and so we could all see that yup, somebody sent this CA the private keys, game over for those certificates.

[ PSA: They're called private keys, not secret keys or shared keys for a reason. Where possible you should choose your own private keys randomly and never reveal them to anybody ]
I wouldn't call "sprinkled undertones" a few visions describing very explicitly a holy war, but to each their own.
The Dollar isn't money, it doesn't function as such. It can't store value very well. Its a federal instrument, and its value is adjusted by the Federal Reserve, which pretty much everyone agrees is incredibly fraudulent.
> 60 to 70 percent threshold that Fauci and other public servants claimed as the target over the last year.

That number changed because the delta variant is far more infectious now. It's closer to 90 %. But you already knew that, which is why you mentioned the time frame of those statements. If I know that you're making disingenuous arguments, and you know that you're making disingenuous arguments, the healthy reaction would be to change your position until you know longer have to walk around compartmentalizing schizophrenic beliefs.

As for that meaning the pandemic has ended: it just hasn't. 1,400 people died every day over the last two week (https://www.nytimes.com/interactive/2021/us/covid-cases.html). Now I don't necessarily care about the death of people who themselves don't care, although even idiots don't deserve to die for their stupidity. But the toll on HCWs is something I would rather avoid.
To me Sicario was a masterpiece. But Dune ... I could never get warm with it. Neither the book, nor the first movie nor the recent one.
SF Mono should also be on that list.
Finding your stolen shit has been possible before; is there going to be any further reason for the police to care when you do?

Way, way back when; I had an acquaintance who fenced stolen electronics. He often sold things back to the people whose "Property of" tags appeared on things. I never heard of the police troubling him.