skysight-inc/HackerNews huggingface dataset:
I am working on a project [0] to generate 1 billion rows in SQLite under a minute and inserted 100M rows inserts in 33 seconds. First, I generate the rows and insert them in an in-memory database, then flush them to the disk at the end. To flush it to disk it takes only 2 seconds, so 99% of the time is being spent generating and adding rows to the in-memory B Tree.

For Python optimisation, have you tried PyPy? I ran my same code (zero changes) using PyPy, and I got 3.5x better speed.

I published my findings here [1].

[0] - https://github.com/avinassh/fast-sqlite3-inserts

[1] - https://avi.im/blag/2021/fast-sqlite-inserts/
I think the effects of madvise primarily crop up in extremely I/O-saturated scenarios, which are rare. Reads primarily incur latency, with a good SSD it's hard to actually run into IOPS limitations and you're not likely to run out of RAM for caching either in this scenario. MADV_RANDOM is usually a pessimization, MADV_SEQUENTIAL may help if you are truly reading sequentially, but may also worsen performance as pages don't linger as long.

But as I mentioned, there's caching upon caching, and also protocol level optimizations, and hardware-level considerations (physical block size may be quite large but is generally unknown).

It's nearly impossible to benchmark this stuff in a meaningful way. Or rather, it's nearly impossible to know what you are benchmarking, as there are a lot of nontrivially stateful parts all the way down that have real impact on your performance.

There are so many moving parts I think the only meaningful disk benchmarks consider whatever application you want to make go faster. Do the change. Is it faster? Great. Is it not? Well at least you learned.
That's generally true of any change that Apple makes. They telegraph the change directly, yet it's largely ignored, usually wilfully, and the caterwauling starts as soon as they take the action that they say they are going to take. Take the M-series transition. When M1 was announced (mid 2020), Cook said that they expected to complete the transition in 2 years. If you have a pre-2020 mac, expect the next macOS to be the last.
How does this system perform compared to modern laser/fiber based gyros?
No, both of you are strawmanning my argument. I said "No True Scotsman isn't a valid argument" and you got "moral relativism is OK" from that.

There's a vast difference between "saying 'no sane person wants to do X, therefore if you want to do X, you're insane' is wrong" and "everything is morally OK".
If there was any anxiety in the 8-bit era it came from disk-born viruses (although in the C-64 era I don't recall even that as a worry). No log-in, no passwords, no cookies, no spyware. (I suppose the BBS introduced most of us to passwords for the first time?)

C-64 + 2FA feels sort of ... blasphemous? Besmirching?
Same here. After getting used to the "automatic" way of editing and writing s-expressions, I can't believe something similar doesn't exists for the C-like code most people write today, but then I remember how much syntax all those languages really contain. Rust is like a trivia answer for "How much syntax could you possibly put into one language?".
There's already some high level people lobbying to update interpretation of GDPR in a way that it would require proof of actual harm to user for any fines to be issued - essentially completely neutering the regulation.
Verifying identity is highly intrusive and would hinder quite a lot of desirable activity. I don't think bots are such a huge problem as to be worth paying that steep of a price.
For sure, skip ahead :) Math textbooks are not generally meant to be read front-to-back (and math papers even less so). This one lends itself well to a straight-line readthrough, but what's important is what you get out of it.

As the sibling comment notes, if you skip too far ahead, the technical material will make you very well aware of that fact. But I think chapter 8 is a good place to jump in -- the material up to that point is mostly setting up intuitions and analogies. I think those are important to pick up, but they're also the kind of thing you pick up from sufficient exposure to mathematics of any flavor, so there's no harm.

Chapters 9 through 13 show how lots of other things can be seen as categories, so if you're already aware of some of them from undergrad math, you're probably good to go.
Not sure why you get downvoted, maybe bots problem is huge problem that should make users concerned? As for me, when facebook tried to force me to send scan of my ID over email in order to unblock my account I decided I benefit more by not unblocking it :) Facebook way was probably the wrong way to ensure your users are real human beings, but either way I like not to be verified when online.
What Simon said! I've found that blogging hugely helps me clarify my own thinking on a topic, and flush out areas of ignorance.

Another excellent option for making getting going with blogging simple is Docusaurus - makes it very easy to get up and running!

https://docusaurus.io
It's nothing, but once one of their customers gets a 5 millioj euro fine for using Office365 for sensitive data, the impact will be significantly higher. Microsoft can take the hit but most of its customers can't.

Microsoft's incompatibility with the GDPR puts some of its customers at risk. A fine or two and businesses might stop paying for those lucrative cloud subscriptions.
>And if you wanna go fast instead of writing maintainable code you surely can with stuff like unwrap(), clone(), Copy.

That's a good approach to learning the language, but it breaks down as soon as async comes into the picture. It's basically impossible to be fluent writing async rust without a very healthy understanding of complex types, how the borrow checker works passing things across closures, etc.
How I made the best map of William and Mary
I think in a sense this adds its own sort of artistic value: life and time do not wait for us.
Not being able to use Office is a death sentence?
I've written extensively in about 10 languages including the two you mentioned.

The current project I'm writing is migrating from Python to Rust. Python is not appropriate for large projects IMO and this one I wrote and am now porting to Rust is no exception. I do love Python for things up to 1000 lines or so, but after that you really need to test every single line, which is just a PITA. Mypy and now others help, but the typing isn't strong enough (although I've heard good things about some newer static typing addons).

I have a love/hate relationship with Go, but after I found Rust I'll never touch it again. It simply isn't expressive enough which leads to code that is overly verbose. The lack of a decent enum, repeating nil pointer mistake, verbose and error prone error handling, the constant copy/paste (now semi fixed with generics). I do love that it strives to be a simple language, but strangely the Rust tooling is much easier to use, so I'm not convinced it hits the mark. I do love how fast it compiles, and that is the one thing I hate about Rust.
Twitter and the Perils of Learning from Users
If you're interested in this one, NoStarch is having a Black Friday sale now, I think the discount is 35% off using code `HOLIDEALS` (not just this book, but all of them afaik)

https://nostarch.com/Effective_C
> I get ~30 GiB/s for threaded sequential memory reads, but ~4 GiB/s for SSD. However, I think the SSD number is single-threaded and not even with io_uring—so I need to regenerate those numbers. It's possible it could be 2-4x better.

Assuming that you run the experiments on NVMe SSD which is attached to PCIe 3.0, where theoretical maximum is around 1GB/s per each lane, I am not sure I understand how do you expect to go faster than 4 GiB/s? Isn't that already a theoretical maximum of what you can achieve?
A well written article on the origins of Python without ever mentioning Monty Python[1].

1. https://en.wikipedia.org/wiki/Monty_Python
Or it will force the European market to innovate instead of directly sending cash to US Megacorps.
The Go code is reasonably efficient at avoiding allocations, though it's hard to avoid some. Without thinking too hard, it's also going to be hard to avoid those same ones in C, and some possible improvements would apply to both (e.g. defining a maximum word length).

"Lowercase word count" is a surprisingly difficult case in this regard, because you need to check and potentially transform each character individually, and also store a normalized form of each word. Probably some smart SIMD lowercase function could help here but I don't think any language is going to offer that out of the box. It's also defined in a way I think detaches a bit much from real-world issues - it's handling arbitrary bytes but also only ASCII. If it had to handle UTF-8 it would be very different; but also if it could make assumptions that only a few control characters were relevant.
Yeah, they get cheap publicity for it, as the topic is currently hot, and they can always return later (with less fanfare than their "quitting"). They win both ways...
Large code bases are always going to be hard to read for beginners.

As an introduction language, I don't see why you would need to use decorators, generators or anything? You don't need to learn control flow, functions, variables, etc.
You can perfectly write python without all the fancy stuff. In fact, there are quite a lot of data scientists who are paid to write python and don't know the first thing about concepts as seemingly basic as classes.
inter-process communication has its own overhead though.
That's a good one too.
> That is of course an easy solution but I would argue that this is just throwing more resources at the problem. Not a very impressive optimization.

You could say the same about the existing implementation as that reads the whole file into memory instead of processing it in chunks.
I mean, Go is closer to C than Python is. But Go is still a garbage collected language with a runtime. It's not going to come close to the limits of performance that C offers
Why don’t you have an opinion? They’re treating women so bad, their policies are stuck in 1200s, and we are in 2022. Does that not bother you at least a bit?

Honest question about the argument - how would you phrase it (I wrote that comment)?
There is nothing wrong with the namespace. I just never found situations where it was useful.

And I see I accidentally wrote "the problem" instead of "this problem". I fixed that.
Hydrogen already is needed today in some processes and some processes can only be made clean with clean hydrogen.

The hydrogen ladder offers a good overview of where hydrogen makes sense and where it doesn’t: https://www.linkedin.com/pulse/clean-hydrogen-ladder-v40-mic...

For me methane pyrolysis from biogas seems like the most realistic way of carbon capture (while also producing hydrogen).
>Most Westerners could witness the change with their own eyes in 2019 when Saudi Arabia began issuing tourist visas for the first time.

The internet is a big place and most people lurking around aren't "westerners". Most people aren't arabs nor Westeners. We are also interested in Saudia Arabia and also find their extreme gender segregation shocking. I also see this when they talk about, say, democracy in China. "We Westeners have democracy so that's why we oppose China".

Am I being too sensitive? Maybe, but it makes me feel excluded and makes me think that a bit percentage of Westeners think it's them vs the topic at hand (whether it be Saudi Arabia, China, etc).
> I don't think anyone expects AI to displace "name brand" art

The problem isn't for existing artists (except in terms of ethical issues)—it's for new/budding artists, who will have to contend with challenges to the authenticity of their work. How can they prove that they put their blood and sweat into making a piece of artwork by hand when an AI could've generated something equally passable?
And yet, none of those are exclusive to the US. The original argument is that the US is more willing to incorporate large technological changes than Japan, so please show me a large technological change that the US has adopted that Japan hasn't.
Related :

Deploying IPv6-mostly access networks : https://news.ycombinator.com/item?id=33694293

(I can't for the life of me find the comment chain where this link was posted ~4 days ago ??)

RFC 8925 – IPv6-Only Preferred Option for DHCPv4 (2020) : https://news.ycombinator.com/item?id=33697978 (posted on HN by me)
Ability to run {X} locally is _the_ problem with building atop paid services, not the dreaded vendor lock-in.
Last time I checked it out it depended on JS-specific behaviour which had to be replicated in other languages. It seemed like something was written first, and then they decided to extract it into a spec.

Maybe this has been cleaned up in the interim, but at the time a bunch of apps for it already existed so it was probably to late to go back.
Does anyone have a good resource for reasoning about how to avoid allocations in JavaScript?
"Hydrogen, which releases zero carbon emissions"

Except the article already stated the Co2 is already emitted as part of the process of breaking down these hydrocarbons (club being in their name) -

"microbes began breaking down the residual oil hydrocarbons in there—dregs that would be unprofitable to extract—to generate hydrogen and CO2."
thank you for not downvoting me. i am less arrogant than you think
i am a true effective altruist. hahahahaha
Part of it is that for users IRC definitely presents as centralised. You don't usually connect to a specific server, but rather a network that did some load balancing in the background. Like you typically connect to irc.efnet.org and not one of the sixty servers specifically.
The problem is, Arithmetic is taught like math, instead of being taught like it is a language, a rich, beautiful language, with nuance and poetry.
He gets quite a lot wrong about Wahhabism (which I am not a fan of). To his credit he does mention he didn't bother going deep into trying to understand it, and his research is very superficial relying on all the well-known debunked cliches about it.

First there are 5 prayers a day not 6.

Also the official doctrine of SA’s head imams is not that the earth is flat and doesn't rotate.

On the earth is flat this has been widely mis-reported and debunked, see (https://www.reddit.com/r/islam/comments/2hc96u/shaykh_abd_al...)

On the earth not rotating he links to a random cleric not an official position. This is no different from citing any of the random nutty clerics in the US who every once in a while make absurd scientific claims (including Ben Carson) and then projecting it as an official position of the entire evangelical movement.

An easier way to debunk the above is to note that newtonian mechanic and astronomy are taught in the Saudi curriculum in a way that is virtually identical to any western curriculum.

I think it is true that Wahhabism is one of the least progressive and anti-science mainstream modern islamic movements, but not to the extent that the author presents.

A better illustration of Wahhabism's anti-science stance is its semi-hostile position on darwinism, but this is true to many mainstream movements in abrahamic religions (including christian and jewish movements).

It is also amusing to note that Wahhabism has a more progressive stance on abortion than most mainstream American christians.
Well Im european and I can tell you from the inside, it's not the same mindset at all. We dont want to grow companies, in fact we barely give a fuck at all. It s hard to understand for capitalists, and I disliked this mindset so much I moved to Hong Kong, but that s what the people vote for: they d rather have no growth and no Microsoft 365 than put their data there.

People in most of Europe are truly convinced finance, money and growth are mirages made to enslave them in eternal pursuit of an unreachable state, and instead prefer to cool it down. It's not a pragmatic strategy because it ignores we re not alone.
I don't disagree that finding alternatives will be expensive, but I think this is the same harmful thinking we have in the US where people disagree with regulation that adds necessary protection at the cost of business. So we have a "regulation is bad" mindset. Most prominently I wish we could convince companies here to believe handling/retaining unnecessary data is like handling something radioactive. Until we convince these companies it's a danger to themself, we won't see change. Sure, in the near future US companies will have an advantage continuing to use Microsoft 365, but harm to our privacy and beyond that is demonstrable. I haven't used a computer like its my own private space to think in decades because of what I know is collected with telemetry. My creativity and passion for computing is harmed by what Microsoft engineers its products to do, and glean from my daily use. If Microsoft wants access to a large customer base in Europe, they should make changes to their products that respect consumer privacy laws in that area. I hope we benefit tangentially.

Something-something auto makers conform to California emission laws, same argument.
You're welcome. Transmitting one's true personality across TCP/IP is harder than it looks.
> You may very well be correct, but there's no formal result that proves that, right? So your assertion could be wrong.

The entire thread started off by you rejecting the formalism provided by folks like Turing in favor of the practical argument. I'm speaking from a practical perspective of building actual static analysis tools. When writing real systems, considering things like "oh well actually there is a stack depth limit" is zero help at best.

> If everyone believed that the Halting problem is undecidable, then nobody would even try to find cycle detection algorithms.

No that's not true. Not at all. We write static analyses that operate on abstractions of a program, which will generate some false positives or false negatives for whatever thing we are looking for but end up with a system that works well in practice. We have not "solved the Halting Problem" nor do we believe that it is actually decidable when we build these systems.

There is no cycle detection algorithm (of actual program semantics - not finding loops in CFGs or whatever) that will never produce a false positive or a false negative that runs on all possible programs and terminates in anything resembling a reasonable time. Actual deployed algorithms have some number of false positives or false negatives (or both).
The author isn't dismissing your existence as a non-westerner. They're describing how a foreign culture looks to someone who's a westerner.
>If halt can be implemented by using X that means you can reduce X to halt.

Replace X with .length and read your statement again.

"If halt can be implemented by using .length that means you can reduce .length to halt."

Please just answer whether you believe that statement is true or false (or undecidable).
> - a court forces a public institution to cease using Office365 (no fines possible there)

AFAIK, in Norway, most fines have been directed at public institutions.
Visibility and analytics is a key component of zero trust... but the comment above seems to forget that the core concept of zero trust is about any person, device, or application trying to access a network cannot be trusted until authenticated and verified... this should mean using strong identity (e.g., x509) and authentication-before-connect, not trusting or allowing network identity such as IP/DNS/ports.
finally, this is really great news for anyone european, I hope it won't take long to determine there are a whole lot of other MS products that should also be illegal
Why not? I got an M1 Max for the RAM, but the relatively powerful GPU is a nice bonus. It's also kinda natural to want to use a general-purpose device you already own for more purposes than you intended when you were buying it.
What about caching to reduce ci time? GH setup scripts cache dependencies in a way that would seem hard to replicate in a make file.
How is Speakeasy (http://alpha.speakeasy.tech) [1] same and different?

## Same

No sign-up required

Client-side encryption (and server side mixing after that)

Anonymous Web authorization requires a unique URL and a password

App itself can run on local LAN (data is encrypted but isn't stored on the app server, see below)

Single button click to clean browser cache and "forget" encrypted data cache

## Different

Speakeasy Web app runs in the browser (i.e. can run on LAN) but the content of messages to self or others is encrypted on the client, and then mixed by a network of distributed nodes. If CryptBoard runs on LAN, you can't access its data from the outside. (Encrypted) Speakeasy data is stored on the external mixing nodes for 21 days, and therefore can be accessed from any location with proper URL and credentials. If you need sharing on multiple locations, Speakeasy is better. If you need it on LAN, CryptBoard may be better.

Speakeasy does not yet support attachments (coming in Beta, around Dec 22/Jan 23), so at this time it's good for passwords, creds and short config files.

Speakeasy is also open source, but it doesn't have a thick stack like CryptBoard, so it is probably more attack resistant. With Speakeasy two people can each run Speakeasy at localhost:80 (attack-resistance, no NAT, etc.), and still share data.

CryptBoard's official app uses centralized server to store data; Speakeasy's "official site" is identical to what you can run on localhost - data is always stored on "mixing servers".

Speakeasy cannot relate users metadata with data, and mixing servers cannot either. CryptBoard's official server can correlate user's metadata (client, IP, etc.) with data, probably also in the case of access by multiple users it is possible to find who's sharing with whom (although not what is being shared, it may still be undesirable).

[1] Remember to disable browser redirection with Alpha version can't yet support HTTPS. HTTPS and attachments are coming in Beta.
On Writing: A Memoir of the Craft
Because before digital projectors were a thing we had slide projectors. Much easier to create physical slides and flip through those with a slide projector than create a scroll of film and project that. PowerPoint and other presentation programs were the digital version of slides. For awhile you'd still have physical slides made from your PowerPoint slides. My father did that up to around 2000.

It's also just less fiddly to flip through pages or slides than scroll around.
> Just because something is different doesn't mean it is wrong or the makers are incompetent. Some of these e-commerce firms have thousands of good developers. Designs are extensively A/B tested. They are common because they work.

Everyone in Rakuten repeats this. It's not true. A/B testing between bad and bad still results in bad. You just get which bad works better.

If Japanese design was awesome, it would be pretty obvious to anyone how awesome it was. This sort of rationalization of bad quality is the mark of today's Japanese mediocre output.

Luckily there are strong local players, like Recruit or TeamLab, who refuse this idea of "it's good because I say so", and are actually pushing for better products.
Show HN: The Newbies Guide to Crypto – a notion style e-book built with nextJS
I’m also European. Thanks for the insight.

I don’t agree with you. So think about that next time you say “most Europeans”.
user space code will not use the kernel and there will be no context switch. it will call a function such as malloc, which is a user-space function. malloc will then go on to interact with the memory allocation sub-system, which may occasionally need to ask the os for more memory via a system call, if the allocator is any good.
My personal favorite outcome of this would be a joint public and corporate funded leap in open source development. This would do much for the budget, privacy and probably also security of businesses and private users. A good example where this principle is already in use is the Matrix protocol.
Read your favourite blogs, articles and other study materials faster with AI