skysight-inc/HackerNews huggingface dataset:
I was curious about the performance difference between these shell scripts and sqlite3, single threaded but fast and hardened database.

I don't have forever, so I modified the linked-to scripts to make only 12,345 customer records (two orders of magnitude slower).

I wrote a python importer to sqlite, which is largely CPU bound on the python side -- go would be much faster, but I wanted to stick with his other tools. 20 lines of code.

Times for 'setting up the data': in his case, catting everything into one file. In mine, splitting the data up into rows and inserting into sqlite:

#His Data Prep
time ls | parallel -m -j 32 "cat {} >> ../transactions_cat"
real 0m1.218s

#My Data Prep
time pypy import.py
real 1m54s

#His Count unique products (Note that as written it overcounts)
time cut -d ' ' -f 5- ../transactions_cat | tr -d \" | tr ',' '\n' | parallel --pipe --block 1M sort -u | sort -u | wc -l
real 1m52s

#My Count Unique products
time sqlite3 transaction.db "select count(distinct item) from txs"
real 0m3.033s

#Transactions per day
#(his script -- note that as written it adds up the wrong columns)
real	0m8.976s

#My transactions per day.
time sqlite3 transaction.db ' select substr(date, 1, 10) as day, count(*) from txs group by day'
real	0m8.625s

#Transactions per store month
# I used the single threaded oneliner from the article, he noted it was fastest.
time cut -d ' ' -f 2,3,5 transactions.csv | tr -d '[A-Za-z\"/\- ]' | awk '{print (substr($1,1,5)"-"substr($1,6,6)), length(substr($1,14))+1}' | awk '{a[$1]+=$2;}END{for(i in a)print i" "a[i];}' | sort
real 0m50.886s

#Mine
time sqlite3 transaction.db ' select substr(date, 1, 7) as month, store, count(*) from txs group by month, store'
real 0m16.908s

In each case except for import sqlite is faster. To me the biggest win is not just the speed or the added expressivity (it's a real difference though for many of these queries), but the simplicity with which correctness can be assessed. Two of his three scripts did not do what they said on the tin, and the third one (which seems to), had a large disclaimer on it that he wasn't sure he'd properly optimized it.
With more integrations. Put the two together though...
Great! Now just drop the embarrassing man(1) page reference, and you can call it modernized.

Wow, I'm surprised that the people whose buttons this pushes are able to make(1) a HN account, let alone have enough points to downvote.

Think about it. There is only one man page for xterm. I fyou type "man xterm" with no section number you get that man page. If there existed an xterm(7) page, you'd still get the xterm(1) man page by default. So why the hell write the (1) notation every time you type the word xterm?

Man page section numbers are not useful or relevant, by and large and mentioning them only adds noise to a paragraph.

Even stupider is when the worst of the Unix wankers write man page section numbers after ISO C function names. Example sentence: "Microsoft's malloc(3) implementation is found in MSVCRT.DLL". #facepalm#
AIUI your blog post was nonetheless wrong. As the OP explains, there are multiple layers of encryption.

The first is at the hardware level, performed by the SSD itself.[0] It encrypts all data on it with a key stored in rewriteable memory. By changing the key, you can "erase" the drive. The FBI could backup that key if they wanted to undo the "secure erase" feature.

However, this is not the only layer of encryption. The second is done in software, performed by iOS. It encrypts all user data with a key derived from the user's passcode and a unique key burned into the chip. The FBI cannot get past this layer, and this is what they wanted Apple's help for.

You were right in that the FBI could trivially get past the first layer of encryption, but it's not the one we care about. There's more than one layer.

[0] Or rather, by the flash controller. At least, I assume it is. It might actually be an iOS software feature too, but the OP's description reminds me very much of how some SSDs implement secure erase. Whether it's actually done in hardware or software is immaterial, anyway.
What the quoted study showed was a lot of "did" and "didn't" behavior, but the article phrased it as "can't".

There are some poor of whom it is literally true that they can't spend $24 on toilet paper. My local food bank gives out rolls of TP, and keeps nearly-empty rolls in the public bathrooms because rolls that are above half get stolen. Some of the clientele literally have every dollar for the month already spoken for, and $24 of TP is not in the budget.

But for a lot of the poor, it's more accurate to say that spending $24 on toilet paper would interfere with other things they've chosen to spend on instead. It's not impossible to come up with $24, but they'd rather spend it in other ways when they do. Now, it's not my place to say that a poor family should or shouldn't use a small surplus to become more efficient and generate bigger surplus later, when they could spend it on something with a more immediate psychological benefit (like a nicer-than-their-average meal). This isn't a judgment, it's just an observation. But I think it's an important observation to make, because the hidden implication is that an extra $20 a month would allow for a new efficiency (like buying TP in bulk/on sale) which would then snowball. But the reality is that an extra $20 a month rarely has anything close to its full potential effect on efficiency, for psychological reasons. Attempts to ease the inefficiencies of poverty need to account for this.
If you read the article, it clearly states: "What works better than traditional homework at the elementary level is simply reading at home". I think where the author would disagree with you is how you encourage them to read. By forcing reading, you're turning it into a chore and presumably get worse results.
The title should read: "Thumbprint thing may be useless for authentication". There are other uses for it outside of security. I have a cydia tweak on my phone that will open different apps based on which finger I press to the home button.
+1 for Detroit. Low cost of living, downtown life is happening and lot's of startup and co-working spaces doing well.
I think people are figuring this out will and use biometrics as a second, or ideally third factor for authentication.

As far a bad authentication, this bothers me far less than when people (and this includes banks!) use my birthday, residence history, or other easily mine-able information to establish identity.
I agree. As the father of an 11-year-old 5th grader, I have absolutely no qualifications to say that, except for how I see the excessive homework they pile on ruining her childhood.

She should be out of doors riding her bike, exploring the woods, discovering weird new bugs in the dirt, interacting with friends, reading books, building stuff, play-acting, playing music, dancing, and generally being the amazingly creative kid that she is, that most of us should have been, had that creative spark not been quantified and tested and pigeonholed out of us by years of dreary deadlines and shouting matches and power struggles.

The poor kids. I hated homework, school in general actually, for most of my student life, and now I get to watch it crush the next generation. Very jealous of the families who home school.
Your comments make sense in the real world, where the big threat is "criminal enterprise looking to make an illicit buck".

I worry that people are too obsessed about the hypothetical specter of tremendously skilled and bored black-hats who will ruin lives for fun, rather than for a pay-off, e.g. ZF0.
It's easy to imagine that being poor just means you have to forgo luxury. If you have $100/month after rent, saving up doesn't mean forgoing luxury, it means forgoing food. If you provide for a family, it means forgoing food for your children.

And then what are you saving for? Not to rent a car or buy one with a loan, because you can't afford the payments. So you save up until you have enough to buy a car outright. Maybe the cost of fuel won't outweigh the savings you can now make in getting to cheaper out-of-town stores[1]. And how long does that take? A $1k second-hand car will require 10 months without food, and something that cheap is probably expensive to run.

Maybe you're saving to move house. Of course you could move somewhere out-of-town. Now you have cheap shopping. But can you find work there? Or, if you keep your old job, how are you getting there? And, again, how long do you survive on air before you've saved enough to move house?

[1] Cheaper because out-of-town stores have to compete for middle-class shoppers who can afford to travel further to shop.
"You've read your limit of free articles for this month." So.... how to the poor read the Washington Post?

I was talking to a recruiter today. He asked me if I could start a job immediately instead of the usual two week notice. I wondered how I was going to buy gasoline to get to work. Guess I better start saving up.
As somebody who has never even experimented with this stuff, I have a pretty fundamental question:

How do the people who submit these reports know what they're using, and how much?

The stuff you're after might be cut with other stuff. What you call X might be completely different from what somebody else calls X, or maybe your seller didn't have any X, gave you Y, and told you it was X.

Even if you know what you're using, how do you figure out the purity so you can determine the dose? Even relatively unprocessed drugs, like cannabis, vary wildly in THC content.

It seems to me like there's a veneer of rigor here that's probably deceiving.
According to your post, the FBI would only need to do that if they were "completely incompetent." You wrote, "unless they are completely incompetent, having read out the contents of the chip they should be able to decrypt its contents in a matter of minutes if not seconds." Details matter, and it was fair to call you on your mistake. You shouldn't get all upset now that the ACLU has published a post which gets the details right.
The 5c doesn't have a secure enclave. That came with 6.
> I do it because it reduces physical code size

And aren't the number of bugs loosely related to the number of lines of code? A darling has fewer chances of a typo bug.
It's like leaving your house key at a restaurant and someone saying all locks are now useless because someone might have your key.
I've found that employees who are able to discuss the benefits of certain data structures and their associated time complexity are generally able to solve problems quicker than those who struggle to discuss these fundamentals. That said, the thing that matters most to me when hiring programmers is proof that they can write decent code.
That's a double edged sword. Now, your code has to compile...
I learned a similar line: "I'm too poor to be cheap."
I grew up in cuse and went to school at RIT.

Why I'd never go back: 1. Snow, 2. Taxes

Why I'd go back: 1. Insanely cheap 2. Hot plates

Thus is life.
being destitute indeed is expensive and low income + high costs let you down the spiral
Thumb/finger prints aren't useless for authentication, they just limited and shouldn't be used as the primary method. They can form a valid part of a security plan as they can protect against certain types of attacks with minimal intrusion.
"Because I want my IDE to be named after someone who tortured people in a basement before he had them shot."

Do we have Eclipse Pol Pot to look forward to?

At least the corporate sponsorships hopefully have him spinning in his grave like an ultracentrifuge.
I'd say their highest point in software was Xenix, but I'd get downvoted into oblivion.
I haven't seen anyone specifically comment on Nashville so I figure I'll chime in and give you my $.02.

I did my undergrad at Vanderbilt in Nashville, moved to Mountain View for two years to work at Google and moved back a year ago after realizing living in California didn't work for me.

Pros:
- No state income tax in Tennessee
- Tons of stuff to do. There's way more than just Honky Tonks and country music. Professional football and hockey teams and a decent minor league baseball team. People go to the symphony and see shows at TPAC regularly. The Frist usually has interesting art exhibits, as does Cheekwood (also a botanical garden). If you prefer to be outdoors, there are plenty of places to hike, climb, kayak, etc. It might not be quite as striking as Yosemite or Marin, but the nature here is still quite beautiful.
- Housing is cheap (relatively). A lot of people (short and long timers) have taken to complaining about housing prices recently, and to be fair they have gone up quite a bit from where they were. It's not as cheap as Chattanooga or other smaller cities but it's still _very_ affordable compared to California. I recently closed on a house just using money I saved from working in CA for my down payment.
- Growing tech scene. It's really night and day even from when I graduated 3 years ago. Tons of meet ups and the community is really awesome. A growing number of companies - and the companies here tend to be more focused on actually growing a business instead of just hitting a valuation.
- Food. Awesome restaurant scene. Obviously more BBQ and Southern focused though.
- Friendly. People in Nashville are very friendly. I think this is immediately noticeable, but I enjoy having conversations with random people I run into.
- Centrally located. They've added direct flights to the Bay Area recently, and you're a day's drive from a good chunk of the eastern half of the country (you can get to Chicago, Oklahoma City, New Orleans, Orlando, or Buffalo within a day's drive).

Cons:
- Davidson County schools aren't great. Lots of people with kids live in other counties and commute in to get better schools.
- Traffic. This isn't as bad as other major metro areas, but depending on where you live it can be significant.
- Not very walkable. There are a few areas that are walkable (12 South, Hillsboro, places in East Nashville) but you need a car. Public transit is pretty bad.

Neutral:
- More conservative state politics. Nashville is fairly liberal, but there's definitely still a conservative bent at the state level. I put this in neutral because I don't think it's all bad (remember, no state income tax) but you may end up getting upset about state legislation.
- Weather. Really hot summers. Winters aren't too bad. Occasional snow, and it can actually get chilly. Lots of rain (compared to CA). I don't think this is bad because I actually enjoy having seasons.
- Big healthcare focus (both small and large companies).

tl;dr - Nashville is awesome, and I'm really happy I moved back. It's not as easy as the Bay Area to job hop, but I'm not worried about finding work when I do decide to make a jump.
Maybe some misguided "SQL is black magic, avoid at any cost" bias?

I'd even pull out PostgreSQL with its mighty scripting and analyzing capabilities. Ok, speed may come at cost of HD space and memory consumption and would require a not so fast import...
There are a lot of startups in Salt Lake, UT. Billboards everywhere looking for devs. I really am surprised it never gets mentioned on posts like this. Salt lake area cost if living is about the same as Austin I believe l, but based on the fact that it isn't as well known that it is truly a tech hub, cost probably won't rise nearly as quickly as the other places.
It has recently been called *Silicon Slopes" by Google (when they installed Google fiber), if that's not enough proof idk what is. Also Utah might be the first state to have Google fiber in two different cities. (Provo, and talk of Salt lake)
This is what I call my "cheater bike": https://www.dropbox.com/s/nu2q0srtj62w3gl/cheater-bike.JPG?d...

It's a 250W rated bunch of Chinese-sourced parts on a $50 eBay steel framed budget mountain bike.

It's _so_ hilarious to ride, a friend says "it make you feel like Lance Armstrong!". It just makes everything too easy. I barely ever take it out of top gear - I just lazily pedal up hills, zoom along the flat, and have to brake to stop feeling suicidal down hills. I'm an almost 50 year old overweight and out of shape guy, and I commute to work on it keeping up easily with the lycra-clad mamils on what look like $10K+ carbon-everything bikes - I've been doing it for a few years and it's _still_ so much fun passing them going up hills.

I can _easily_ believe even a fraction of the power I've got there would turn a midfield racer into a winner.

(For the record, I've got a 7 cell LiPo battery powering mine, and off a full charge I see 470W or so coming out of the battery accelerating up a medium incline. It tops out at 25kmh, and pulls barely 100W to do that on the flat, but it's _so_ noticeable accelerating away form stationary or up hills. It's _ really_ fun :-)
I've never been able to articulate my own feeling about wealth and poverty. I've been objectively poor at some points in my life. I became vegetarian in my early 20s when I was travelling because I couldn't afford meat. Even then there were some weeks where I didn't eat for a few days because I had spent all my money for the week. It sounds bad, but being hungry taught me how to prioritise and I don't regret it.

In other times of my life, I have quit my job and spent half a year writing free software, or training karate -- without any income. I also spent 5 years living in a run down shack spending less than $10K a year, when I was teaching English in Japan.

By many measures, I was living in poverty because I had almost no income, or I was spending almost no money. But at no time in my life have I ever thought of myself as poor. I have always had money in the bank and I have never really been in debt. When I went hungry it was because I stupidly spent my food budget on beer (or something similar). I didn't have money for food, but I did have money.

Money is an option to buy things and for me the feeling of being rich is the feeling of always having that option. You can be happy with less things, I think. Nobody actually needs a yacht to be happy, or a big house, or a collection of vintage wine. But I think it is really hard to be happy when you have no options. It is even harder to be happy when you spend your options in the future and have to suffer with crushing debt -- possibly enslaved forever.

I could choose to be hungry. I could choose to give up my job and do something that paid no money. I could choose to move half way around the world and live in a shack teaching English. That's incredible wealth for me. That kind of wealth is available to many people, but I think that most people can't see it, unfortunately.
If anything, I think low-level programming has become an underappreciated skill.

If you work on kernel development, be as snooty as you want.

These days, those jobs that require real expertise and in-depth knowledge aren't nearly as heavily compensated as your run-of-the-mill Rails dev Stackoverflow pasting position.
I think it underscores how much we (and other animals at high tropic-levels) directly or indirectly rely on extracting energy from other species within a large area.
Could be, depends on how you value it.
Meet the First Woman to Win Math’s Most Prestigious Prize
This is a fantastic, amazing, wonderful, and never mentioned point. I am beat after 8 hours of mental output. I mean, I sit in a chair (or stand) at my desk most of the day, and take frequent breaks, and I'm still tired.

Why would I expect more of a child?
lol indeed. ;) i'm an ex denverite who keeps going back at least once a year
Plus a daily testing regime, incentive structure for kids, and feedback loop for lectures
Toaster.io
Repping Pittsburgh and Vegas! The Burgh has Uber ATC, Oculus Research, and Google now, not to mention CMU.
I was going to recommend traffic manipulation and traffic shaping with netem and tc. I set this up at my last employer in the lab for use with our networked IP radios and such. Stimulating poor network conditions has been a mainstay in my testing ever since, even now that I work on web apps and distributed services.
What a great article! My head is buzzing, but to start:

1) You mention that some people never used the invitation or used it but didn't complete the initiation. "Why were some of us drawn in like moths to a flame, while others reacted zero?" You also mention some trepidation during your own initiation ritual. What made you go through with it? I'm a healthy young male, and I feel nontrivial trepidation at the thought of going somewhere unknown, alone, knowing I'm being watched, not knowing if I can get out, surrendering my phone and wallet, seeing the glass of who-knows-what...I imagine these fears would reasonably be stronger for many others. Is it reasonable for the society to ask such trust from people who haven't yet interacted with it? Presumably the danger and the fun are intertwined. Is that necessarily a good thing? Does it have negative filtering effects on society membership?

2) What happened if someone going through the initiation ritual didn't play along - didn't go down the slide or leave, didn't move on from the library at the appointed time, hid their phone then tried to record the Fable, etc?
> The other thing that helps me in interviews is to just talk through everything I am thinking. ... If you just stand there silently decoding a problem in your head, the interviewer can't help you and has no idea how you'd approach a real problem on the job.

This.

Most important thing for me as an interviewer is for you to talk out loud about what you're doing and what you're thinking.

Just let it flow - all of it.

Programming is often a solitary thing - we go through a lot in our own heads when writing code; e.g. say I ask you to write some code to iterate through some website log files and build a site-map from the URL paths in the log entries. I bet that your mind - even as just a reader of this comment - has already started working out some of the steps to do this ... "load the files, iterate through line by line, some sort of map or DB to store the files, some way to handle duplciates" etc. You might even already have some questions lined up about number of files, frequency of running, shell-scripts vs "real" code etc. Great - but I need to know that you thought about that stuff and I wont unless you talk about it out loud.

Standing silently at the board is not giving me the interviewer as much to go on. Even if you get stuck and/or make a mess of it and dont have a working solution at the end, if you've talked out loud about what you're doing, why you're doing it (and why you're NOT doing something else etc) might be enough on its own to get you pass that interview despite your solution not working (everyone has off days, but show me your thoughts!)