skysight-inc/HackerNews huggingface dataset:
Ask HN: What are the smartest trade-offs you know in software?
Okay, I've really struggled to make title both short and descriptive enough, but here is a longer story.

I'm really fascinated by reading posts from dev blogs where a team facing some problem or need for optimization, took very smart trade off, based on data analysis or just doing a lot of brainstorming and putting a lot of thought into understanding the problem. Trade offs that are specific for their domain and allowed them to choose non-obvious decision to drastically improve their software design.

Examples:

- Twitter redesigned their media storage system, based on data analysis. They found that images older than 20 days has very low probability of being accessed, so it's ok to precompute thumbnails on the fly. Sort of storage/cpu tradeoff, saved them 4TB per day.

http://highscalability.com/blog/2016/4/20/how-twitter-handles-3000-images-per-second.html

- Ravelin was about to use heavy graph databases for the fraud detection purposes, but after thorough analysis of the problem they come up with the solution that only uses Union Find datastructure.

https://skillsmatter.com/skillscasts/8355-london-go-usergroup

So I'm looking for more examples like these, when understanding the data you work with lead to not very obvious/popular tradeoffs, that simplify or improve the overall software design drastically.

If you know such examples, links, talks, blog posts, or can tell your story, please share!
LOL!!!!!!!!!!!
Books I'll treasure forever:

- Advanced Programming in the Unix Environment by Stevens
- Operating Systems: Design and Implementation by Tanenbaum
- The Art of Unix Programming by ESR
- Parsing Techniques by Grune and Jacobs
- Applied Cryptography by Schneier
And usually when a submission is old it is marked as such with [year]. I incorrectly assumed that was widely known.
Haha... "workstation mode". Didn't everyone who tested "Game Mode" on Windows 10 creators update come to the conclusion that it didn't do anything?
"The Design of Design: Essays from a Computer Scientist" by Frederick P. Brooks [1] is language-agnostic and worth reading.

It's about software engineering but also about hardware and some different kinds of design outside of IT.

From an interview about the book [2]:

> Eoin: Your new book does talk about software design in places, but it’s really about design generally, and the case studies span buildings, organizations, hardware and software. Who is the book aimed at? Are you still writing primarily for people who design software or are you writing for a broader audience?

> Fred: Definitely for a broader audience. I have been surprised that The Mythical Man-Month, aimed at software engineers, seems to have resonated with a broader audience. Even doctors and lawyers find it speaks to some of their team problems. So I aimed this one more broadly.

Brooks is also the author of The Mythical Man-Month which is often mentioned on HN.

[1]: http://www.informit.com/store/design-of-design-essays-from-a...

[2]: http://www.informit.com/articles/article.aspx?p=1600886
ISRO GSAT-19 satellite launch live updates
Crypto Anarchism
The headstart that both Tesla and Google have is test data. Tesla has more, because they've collected data from production cars from the get go. Google likely has more mature software (pure speculation), and it has comprehensive mapping data, but it doesn't have 100k customers providing them with useful real-world information. It does, however, have millions of Android users, but that's not the same. I believe all new Tesla vehicles send back autopilot data to the mothership.

There's some interesting comparisons. On the one hand you have Google, which is primarily a software company with hardware interests. Then you have GM which is an auto company with software interests. And you have Tesla which positioned itself in the middle, aiming to both build cars and software.

As a result, Google has a lot of core software expertise and infrastructure (particularly for machine learning). GM has the production capacity and the ability to roll out features to potentially millions of new vehicles. Google had to buy in automotive people. GM had to buy in software people. Tesla had to buy in both, but they did it from the very beginning with clear goals in mind.

From an outsiders' perspective, building a car that can drive by itself isn't terribly difficult nowadays. If you have the cash, you can buy a team that will build it. The challenge is getting from that stage to a car that can reliably drive better than a human in all driving conditions and deal with unexpected events. That will take much more time, and much more data.

There are other aspects like getting the sensors down to a price that people can actually afford.
[Screencast] Autocomplete with HTML Results
Do you want potentially sensitive information leaving your machine? That's basically the counterargument.

https://en.wikipedia.org/wiki/Windows_Error_Reporting#Privac...
Agreed. Scheme itself is "not much of s language." It's like a language-agnostic language.
"Resilient file system"... sorry, if you want professionals to use and trust your new filesystem you should give it to end users first; wider audience. Then the pros will pick it up.
Consumers are already willingly installing telescreens in their homes.
Mission creep.

"Maybe if I invent more Windows editions my salary will also increase because of the increase in responsibility as well as the higher revenue from the nickle and diming customers?"

I imagine that would be her line of thought.
ISRO GSAT-19 live launch stream
Which programming language was begat by FORTRAN

Fortran->Matlab->Julia is a pretty obvious family tree. Much of how numpy and similar libraries do numerics is also quite obviously influenced by Matlab (and thus Fortran).
moby dick by herman mellvile

the bible / unknown author

the new testament / various
You can get Linear -> SVM -> Neural Networks on Andrew Ng's Machine Learning course at Coursera CS229A. You could then go advanced with CS229 from Stanford "Academic Earth".
That's a great answer thank you for your response, do you think it would be possible to write a plugin that auto comments code and updates the comments for particularly tricky parts say something that comments [n] then tacks on what n does at the end of the code. Then in the background keep a running list of what got referenced where and updates those references as needed?
Oh, my gods.

> a young family without a lot of money.

...

> Our budget was $560,000, but nothing came on the market at that price

wat.

> We sold our two-bedroom rat trap for $635,000

wat.

> Luckily, we still owned the two-bedroom condo at King and Bathurst.

Well, that's okay then.

Seriously, these people live in a completely different universe from me.
Show HN: Javalin REST Library – Rewritten in Kotlin
Both of the investments I referenced were solid until they weren't.

My adult life has seen 3 major market crashes and a terrible recession. I've watched neighborhoods go into foreclosures on 80% of their homes. I've had friends go unemployed for over a year.

When I see people spread investment advice as if there's no downside, I have to shake my head. There's money to be made in the market, but it's not without risk.

The Nasdaq Composite lost 78% of its value as it fell from 5046.86 to 1114.11 in the dotcom crash.

The S&P 500 declined 57% from its high in the housing crash.

The Dow lost more than 5% in a single day at least 5 times in 2008. The next time one of the major indexes drops 700 points in a day look around and tell people "you'll be fine if you don't panic sell. Think about it in 20 year increments. It'll all average out. You're money is all in an index fund, right? Those are totally safe. You didn't want to spend anything during this presidential cycle anyways."

Where was the index fund advice in '98? Where were all the people who had well balanced / well hedged portfolios in '07-'09? I'll tell you where they weren't. Every house with brown grass that the banks had foreclosed on and decided not to keep up.

There was an army of day-traders once upon a time not all that long ago that blew all kinds of sunshine up people's asses about investments. There's no reason to. People who have the money to can invest if they want and if they do they should learn about the risks and ask the people who are giving them advice how they fared during the crashes. If those advisors don't admit it was hairy and ridiculously stressful, they are lying through their teeth.

Index fund advice became widespread after '08. Now Black Rock, State Street, and Vanguard own majority stock in 440 S&P companies. You aren't invested in the S&P500 anymore. You are invested in BR/SS/VG management. Who aren't incentivized towards the interest of any particular firm. What happens as these funds grow? If investors herd to passively managed funds, what's the outcome? Faster cycles. The indexes do more securities lending, which makes them less liquid come crash time. Do investors know that these funds will have liquidity problems if the market goes belly up? A whole shitload of people are going to learn the term "halting redemptions" really quickly. Do investors realize that the growth of passive investment increases anti-competitive behavior? Which isn't good for the economy and sure as shit isn't good for small investors.

But by all means, keep shilling the passive investment advice as if you've been doing it for 40 years. Then complain with the rest of the country that the funds should have been regulated more and how people should have known not to give that much power to so few companies in the financial sector.
5 Steps to Successful Android Testing
http://norvig.com/spell-correct.html
Maybe they should go back to good old days of Windows XP when they've asked if I want to send them crash report.
They didn't do the work themselves. Construction workers did. Also the "neighbourhood asset" was not donated to the neighborhood.
In my experience, there's definitely a point where poll() beats select() when handling lots of connections, long-lived or short-lived. But at that point you are much better off moving to epoll() or kqueue() depending upon the OS. And if you support those syscalls, there's little point falling back to select().

For most people, if your program is handling 10s or 100s of concurrent connections, select() should be just fine.

If you are going to be handling more, it's worthwhile looking into the other syscalls to improve performance. In any case, I'd recommend abstracting away the event handling so that you can switch between different syscalls and can use benchmarking to try to replicate the traffic you want to handle. There's lots of libraries that will do this work for you, unless you need to get into the low-level stuff.

Trying to judge the syscall performance based on reasoning about amount of data transferred between your program and the OS, number of syscalls etc, is very difficult. You are much better off measuring the actual behaviour. For example, epoll() seems like a terribly designed API to me as it involves making many syscalls (whereas kqueue() is just one per loop). However, I found epoll() was very high performance. I guess the cost of syscalls on Linux can be very low in some cases.
This almost answers a question I've often wondered.

Whenever I see research on calorie restriction, I wonder whether it's directly caused by eating fewer calories, or indirectly caused by eating fewer bad things. For example, methionine, an essential amino acid, is linked to cancer growth. In other words, maybe we don't need to eat less calories, but less of specific nutrients.

Anyways, this study seems focused on metabolic diseases and doesn't touch on cancer or longevity in general. So, if anyone has any thoughts, I'd like to know. It's been bugging me for a while.
On that note if they would do updates without forcing restarting it wouldn't be as horrible. Maybe a softer approach is "logout and log back in to see updates" on Linux unless it's a Kernel update I usually don't need to restart, just logout (in some cases that's not even needed just reopen the program and keep going). Of course if you're on Slackware you may just sit with the same files for years :) At least I did, (it was my first distro) so Windows needs updating improvements heavily. Also don't slow down my whole OS while they're downloaded / being processed.
There are two main arguments:

- You can't disable telemetry completey. Some people want it, some don't. Let the user decide.

- Microsoft does collect an insane amount of data.

https://docs.microsoft.com/en-us/windows/configuration/basic...
Research identifies methods to protect against online privacy attacks
There is nothing wrong with telemetry - if it's opt-in.
That's all.
Well if you google for "no evidence collusion", and if you get evidence for collusion there is something wrong with the algorithm don't you think?
ISRO’s heaviest rocket with GSAT-19 satellite takes off into space
>As someone who has been on the publication side of digital media: Never ascribe to propaganda what you can explain by greed.

A great documentary on the interplay between investigative journalism, government lies, and infotainment (greed based media guised as "news" or "journalism" is

All Governments Lie: Truth, Deception and the Spirit of IF Stone
https://www.youtube.com/watch?v=C_cKC0_sGu0
Got some good feedback here last time, so posting again. I rewrote the library to Kotlin, but I had to leave a few things java, mainly the SAM interfaces and the Javalin class iteself. This had to be done due to limitations in Kotlin (https://youtrack.jetbrains.com/issue/KT-14151, https://devnet.jetbrains.com/thread/461516)

The rest of the library will be ported to Kotlin if this issue is resolved, or and alternative solution is discovered.

I guess the biggest change for Kotlin devs is that Request and Response are now Kotlin.

Feedback/criticism is welcome!
Code complete

Refactoring

Patterns of enterprise architecture
Top tools for your new marketing stack recommended by Product Hunt users
Fuck selling pens.

Give him the pen, then sell his mailing address to a paper manufacture.

Then cap his refills at 1 per 3 months, but for a low monthly subscription offer to give him all the ink he wants, and all the new pen models as they rollout.

Hard selling of ubiquitous consumables in highly competitive markets has no place in the SaaS world.
InterSystems, Epic take the lead in global EHR market share
Fabien Sanglard wrote great reviews of various codebases, including git, Doom 3 and others.

http://fabiensanglard.net/
Isn't BitLocker (https://en.wikipedia.org/wiki/BitLocker) how Windows supports full disk encryption by default (shipped with Vista and later)?

From an end-user perspective it seems to behave very transparent as well. In what way does it come up short?
Also "Software Foundations"
Only an Id10t would think it's actually deleted from FB-data logs.
You're right. Effectively the kernel is having to setup the event listeners for each FD on every select() call, whereas with epoll it can be done once and the state is preserved, leading to less overall work. Also, the epoll syscalls are surprisingly fast and lightweight.
Rational decision-making involves second guessing or even countering your genetic programming.

The "my genes made me do it" argument is no less flawed in a discussion about terraforming than one about what to order at McDonald's.
Looking for tech co-founder to join Cool Startup
Trying to bring all of Africa's info to the internet. First by getting every small {business} website. Looking for a tech co-founder to help me build a website builder with a twist? Next big thing.
>the way those results are ordered isn’t exactly organic. Alexa ranks the NYT at 120 globally; WaPo at 190. Now, what about the illustrious townhall.com, which had not one but two hits on page one? It’s ranked at 9,109. In other words, those first four pages (four full pages of synchronized bullshit) are evidence of a massive and centrally managed strategic misinformation campaign being waged on your brain.

Granted.

>These dozens of sites are all peddling the same lie...

Uh, the headlines provided range from "No evidence of trump-russia collusion" (page 1) all the way to... "lack of evidence on trump-russia collusion" (page 4).

>If you read closely (which they’re betting you won’t) they all include the proper, honest caveats: no evidence YET; no one says they’ve SEEN evidence; etc.

The jump from that, to "No evidence" in the headline, is the most humdrum of spin used in every media article written every day, it's the background noise of journalism.

You're onto something big, don't mire your articles in blue tribe whining.
I dare you to go explore their ERP offerings. It is impossible to grasp, and if you ask for help, your support request will be redirected to one of your local third-party Microsoft partners.
"Falsehood flies, and truth comes limping after" - Jonathon Swift

http://www.bartleby.com/209/633.html
Operating system is not website. I don't want my system to be sanding any data anywhere without my knowledge. Currently operating systems are used for handling various of very sensitive and potentially dangerous information (credit card details, bank accounts, SSNs, tax returns etc). There is tons of informations on computers that can cause a lot of problems if used by bad people. Blocking unknown and unnecessary traffic and strong encryption is our best bet against cyber criminals.
Your points are valid, but I'm not sure about the strategy. It would definitely make sense for left wing adversaries to use the same techniques, ending in total media pollution. Even if that helps everyone distinguish fact from spin, Google's mission would inevitably have to shift towards estimating the credibility of such organized mis-information.
I think they could have avoided almost all issues if they would have paid market rate from the start..
I mean, some additional stuff needed to be done (stripping carpet, re-finishing floors, replacing bathroom and kitchen fixtures). But there was no call to "gut it to the studs".
Structure and Interpretation of Computer Program (it's written in scheme but it's mostly for convenience and its lack of syntax).

Definitely the best book I've read on programming.
The only version of Windows 10 I'm interested in is the one where it is I who is mostly in control, or, you know, be the power user, not just some power sheep.
I would recommend "Sorting and Searching" Volume 3 from Donald Knuth's "The Art of Computer Programming". Fantastic, in-depth read.
A short book (25 pages), but I find it worth mentioning here -
6 Things About Programming That Every Computer Programmer Should Know -
https://www.amazon.co.uk/Things-Programming-Computer-Program...

A really well written short glimpse of things every programmer must know.
Saving someone a Google.

http://course.fast.ai
Yes, they are both detriments to the environment, but precious metals provide value. Not only do they provide value, but in many cases, once they are used, they may be recycled and used again. You get to amortize the pollution over the lifetime of utility of the material until it can no longer be recovered.
Funny that you should mention it, because there was an article here some time ago that basically said that 90% of buildings in New York wouldn't be allowed to be built today, they wouldn't get the permits. I've moved several times in my life and never lived in a house/flat that was newer than 50 years old, and older houses/flats were built small - although over here in UK I would argue that if you want space you need to look at older houses, everything new that I looked at was absolutely tiny, 3-4 bedroom houses with bedrooms that would fit a bed and literally nothing else.

According to this, 85 square meters is enough for 6 people to live in:

http://www.privatehousinginformation.co.uk/site/files/LHMO%2...

I meant a family of 2 adults + 4 children, in which case yes, 85sqm is ample. For just 2 adults + 2 children it's way more than enough.
Let Q(p) be quantity demanded at price p, and c be marginal cost. Then profit is Q(p)*(p-c).

If e.g. Q(p) = 1/p^2 (price elasticity of demand of -2), then profit is 1/p - c/p^2, which is maximized at p = 2c.

Note how optimal price depends on cost.
Cache-Mirror DockerHub for speed
If you liked Clean Code, read Clean Coder. A quick summary:

Robert C. Martin introduces the disciplines, techniques, tools, and practices of true software craftsmanship. This book is packed with practical advice–about everything from estimating and coding to refactoring and testing. It covers much more than technique: It is about attitude.
Pet peeves of little experiments like this of carb cutting. Cutting carbs causes you to shed a lot of water weight. Also if the exercise is new it is quite possible that is the bigger factor in improving perceived cognitive ability. If you changed multiple things and started paying closer attention to diet it is really hard to pin down what actually helped the most.

Vigorous exercise helps clear waste along with sleep, not much else really clears things out -- brain seems more and more to be a part of, or similar to, the lymph system.

edit: also most people who do 18 hour fasts report feeling energized, better concentration, etc,. especially if they have caffeine on an empty stomach.
How a Chinese Tech Firm Borrowed $2.1B in a Hurry
"Left wing media" lost me a few years ago to no credit of "Right wing media". They're shamelessly embracing polarization and every time I decide to take another peek at it, it seems it's sunken further.

Journalism is broken, completely broken, and as long as people are trying to figure out how to fling mud at other publications rather than rise above the sea of trash they'll keep losing to ever less credible sources.
- Computer Systems: A Programmer's Perspective
Hey, would like to know more,am in Uganda, tell me about it.
info@payrainbow.com
Who knew renovating a house could be so complicated?
Is it the "Game Mode" that enables the Xbox DVR by default? If so, it did do something, it locked my FPS at 30.

It is such a big problem that Valve had to build in detection into their game where they warn you about it.

And in order to turn it off, you either have to login to their Xbox App, or play around with regedit, Valve have documented it here: https://support.steampowered.com/kb_article.php?ref=6239-DZC...

And it seems like updates are enabling it again automatically, because I have had to do it twice.
You can customize a workspace, or create your own.
https://www.freecadweb.org/wiki/Customize_Toolbars
Why Isro's launch of heaviest rocket GSLV MK-III is a big deal for India
Because telemetry is potentially not only the core dumps and performance metrics, but also all sorts of potentially sensitive information - like a data derived from the user input (keyboard , voice or pen - this data is said to be anonymized but AFAIK no one had audited this) or list of installed applications. Knowing that something is crashing or performing badly is one thing, knowing what I'm doing with my computer is another.

Telemetry can be done right by letting end-user to see an example of the data that's going to be sent and then also audit the actual reports that were sent. Transcribed into a human-readable format with some comments about what the values mean - I can see the binary data myself, thanks. And a way to opt out if the data is found to be actually sensitive.

Basically, an opt-out switch and "sure, but give me a full transparency report" mode that a "power user" can optionally access if they happen to care. This is rarely the case, and not the case with Windows.
I remember having a few aha moments when reading through the Redis code.
> Would fellow Toronto residents prefer the squatters in the neighborhood to this couple?

I live in Brooklyn. But if I had to choose between a rooming house and these insufferable adult-children for my neighbors I would opt for the former.