skysight-inc/HackerNews huggingface dataset:
> Until the 1900's starch fed the world, it still does, some how what is succesfully feeding the 5.5-6bln poor people of the world is no longer good for you.

What are you even talking about? We have a public health epidemic of obesity, especially among the "poor people of the world." They are literally the most affected group by low-cost availability of processed foods. This is undisputed and self-evident.

Of course it is worth it to revisit our recommendations and practices given what it looks like when we look out the damned window. You think someone just decided well, starch has had a good run, time to move on based on no scientific basis?

Your commentary in this thread is actively harmful in its complete substitution of science and understanding for your own personal gut feeling. You sure are saying a lot and ignoring the requests for citations.
Might this be used to read the famous burned scrolls of Pompeii and Herculaneum? (Although according to this article, current thinking is they might be able to read them with X-rays: http://www.smithsonianmag.com/history/ancient-scrolls-blacke...)
I laugh to myself whenever I see weight loss articles like this.

Just Eat Less! Worked for me*

Shameless plug

My Weight Loss Story 40kg in 40 Weeks (90lb in 9 months) Without Exercise

http://40in40book.com

The book is more for motivational reading of my story.

*DISCLAIMER: The information and opinions set out in this publication are provided only to convey the author’s personal experiences and opinions for information purposes only and are not intended to constitute medical advice, guidance or recommendations as to the reader’s actions or omissions.
Disclaimer truncated for brevity of comment. The rest of the disclaimer is available here: http://40in40book.com/disclaimer
That's the point: the entire welfare maze does not make sense, so UBI does not either.
I am not talking about maintaining order (by force or not), but about extorting people to pay from their work to give to others. There are many arguments around pro and con, some are simply ridiculous but there are around for so many time and inoculated so well, it works like a religion now: beliefs trumps reason.
There isn't exactly a direct relationship between the size of the data set and the amount of memory required to process it. It depends on the specific reporting you are doing.

In the case of this article, the output is 4 numbers:

games, white, black, draw

Processing 10 items takes the same amount of memory as processing 10 billion items.

If the data set in this case was 50TB instead of a few GB, it would benefit from running the processing pipeline across many machines to increase the IO performance. You could still process everything on a single machine, it would just take longer.

Some other examples of large data sets+reports that don't require a large amount of memory to process:

* input: petabytes of web log data. output: count by day by country code
* input: petabytes of web crawl data. output: html tags and their frequencies
* input: petabytes of network flow data. output: inbound connection attempts by port

Reports that require no grouping (like this chess example) or group things into buckets with a defined size (ports that are in a range of 1-65535) are easy to process on a single machine with simple data structures.

Now, as soon as you start reporting over more dimensions things become harder to process on a single machine, or at least, harder to process using simple data structures.

* input: petabytes of web crawl data. output: page rank
* input: petabytes of network flow data. output: count of connection attempts by source address and destination port

I kinda forget what point I was trying to make.. I guess.. Big data != Big report.

I generated a report the other day from a few TB of log data, but the report was basically

for day in /data/* ; do #YYYY-MM-DD
echo $day $(zcat $day/*.gz | fgrep -c interestingValue)
done
Or Google Translate
> In theory voting is compulsory. In practice it is compulsory to show up and get your name marked off.

Since we have a secret ballot, it would be unenforceable for them to say that you had to submit a valid ballot.

> What you do after that in the little booth with just your conscious and a pen is up to you. (I usually take the pen, that's all I've ever got out of politics.)

Odd, my voting place last election only had pencils. I carried a pen with me (voting with a pencil is asking for trouble).
> DON'T POP A NEWSLETTER FORM OVER THE TEXT I AM TRYING TO READ.

I'm running NoScript in default deny all JS here - I read the whole article without ever realizing there was a "newsletter form" until after I saw your comment above.
I like this notion of "political compass" : https://www.politicalcompass.org/analysis2 (not unique to this author), in which liberal vs conservative (left vs right) is an axis , but with a second one authoritarian vs libertarian i.e. more government intervention vs less government intervention is another axis.

Essentially this highlights the difference between your particular policy opinions and your enthusiasm for having a government enforce those policies on everyone.
Putting a 100TB of storage on your network isn't hard though. There are off-the-shelf NAS servers that big - eg http://www.ebuyer.com/671448-qnap-ts-ec2480u-rp-144tb-24-x-6...
AWQL.me – A web AWQL console to easily run requests on your Adwords accounts
useful technology for the MIT blackjack team
My personal take is that people have difficulty with change in computing because computers are so damn good at irrecoverably destroying data.
Kinda stopped reading when he described "the four things states are proxy insurers for", demonstrating a pretty drastic misrepresentation of how modern functional democratic social policies can be grouped and analyzed.
Having worked at a large company and extensively used their Hadoop Cluster, I could not agree more with you.

The author of the blogpost/article, completely misses the point. The goal with Hadoop is not minimizing the lower bound on time taken to finish the job but rather maximizing disk read throughput while supporting fault tolerance, failure recovery, elasticity, and the massive ecosystem of aggregations, data types and external integrations as you noted. Hadoop has enabled Hive, Presto and Spark.

The author completely forgets that the data needs to transferred in from some network storage and the results need to be written back! For any non-trivial organization ( > 5 users), you cannot expect all of them to SSH into a single machine. It would be an instant nightmare. This article is essentially saying "I can directly write to a file in a local file system faster than to a database cluster", hence the entire DB ecosystem is hyped!

Finally Hadoop is not a monolithic piece of software but an ecosystem of tools and storage engine. E.g. consider Presto, software developers at Facebook realized the exact problem outlined in the blogpost but instead of hacking bash scripts and command line tools, they built Presto. Which essentially performs similar functions on top of HDFS. Because of the way it works Presto is actually faster than "command line" tools suggested in this post.

https://prestodb.io/
Comments moved to https://news.ycombinator.com/item?id=12470703.
If you are reading this thread, you will probably also find the comment thread on the Economist article somewhat informative. (They seem to be moderating it quite well too)
I dunno. Robots already can make artistic artifacts, but certainly it wouldn't really be considered "good", but I would still stay it's interesting to us. http://arstechnica.com/the-multiverse/2016/06/an-ai-wrote-th...
I hope you realize that starch has fed the world only in a post agricultural world. That may be thousands of years, but we spent hundreds of thousands of years living without agriculture. That argument does not hold water.

Secondly, eating fruit and eating sugar are two very different things. Fruit has things like nutrients and fiber that is lost in processing, which is likely the root of the disorder. The fiber, for example, slows down the absorption of sugars. The modern world is full of highly refined sugars that are missing the goodness found in nature (that our bodies adapted to).
I didn't know this, thanks! It looks like one can have composition after all. http://ftp.squeak.org/docs/Self-4.0-UI-Framework.pdf
Well the full message is

"436F6E67726174756C6174696F6E732C20796F7520736F6C766564207468652070757A7A6C65212057687920646F6E927420796F75206170706C7920746F206A6F696E206F7572207465616D3F206D69352E676F762E756B2F6361726565727320"

"Congratulations, you solved the puzzle! Why don�t you apply to join our team? mi5.gov.uk/careers "

So i doubt there is much more to it ;)
I think you might be replying to my comment. We just took delivery of a 20K WGS callset that is 30TB gzip compressed (about 240TB uncompressed) and expect something twice as big by the end of the year. We're trying to build something pretty general for variant level data (post calling, no reads), annotation and phenotype data. Currently we focus on QC, rare and common variant association and tools for studying rare disease. Everything is open source, we develop in the open and we're trying hard to make it easy for others to develop methods on our infrastructure. Feel free to email if you'd like to know more.
1. Open PNG in text editor, see comment:

As I read, numbers I see. 'Twould be a shame not to
count this art among the great texts of our time

2. Threshold so some of the pixels are black and others are white. Save as ASCII PBM.

3. Strip the PBM header lines and remove newlines so that it's just 0's and 1's.

4. Now feed it through this program:

#include <stdio.h>
main() {
int c, l=0, n=1;
while(EOF != (c=getchar())) {
if(c == l) n++;
else {
putchar(n);
n=1;
}
l = c;
}
putchar('\n');
return 0;
}

Output is the following:

43-6F-6E-67-72-61-74-75-6C-61-74-69-6F-6E-73-2C-20-79-6F-75-20-73-6F-
6C-76-65-64-20-74-68-65-20-70-75-7A-7A-6C-65-21-20-57-68-79-20-64-6F-
6E-92-74-20-79-6F-75-20-61-70-70-6C-79-20-74-6F-20-6A-6F-69-6E-20-6F-
75-72-20-74-65-61-6D-3F-20-6D-69-35-2E-67-6F-76-2E-75-6B-2F-63-61-72-
65-65-72-73-20

5. This smells like ASCII, let's decode it with tr -d - | xxd -p -r

> Congratulations, you solved the puzzle! Why dont you apply to join our team? mi5.gov.uk/careers

From: https://archive.rebeccablacktech.com/g/thread/52888920#p5289...
I've been meaning to write up the adventure I had setting up my Yubikey 4 together with GnuPG. Most of my work was cribbed off of this guide: https://www.jfry.me/articles/2015/gpg-smartcard/

But there were some important differences. Newer GnuPG versions have simplified how gpg-agent takes the place of ssh-agent. Nowadays, it's enough to create an SSH_AUTH_SOCK environment variable that points to ~/.gnupg/S.gpg-agent.ssh

Also, I found the air-gapped system setup described there and elsewhere to be excessively difficult. Far and away the easiest way to create an air-gapped key generating machine was to install OpenBSD to a USB key (you can boot the mini install image and overwrite the same device). Installing the gpg2 package gives you a complete gnupg environment for interacting with OpenPGP smart cards. By contrast, there were a bunch of packages to install with Ubuntu / Debian.

It was a little hairy to set up in total, but I really love my Yubikey-mediated GPG setup. I also now use password-store for passwords, complete with dmenu integration.

I'm not super happy that the Yubikey 4 isn't 100% open hardware though. If someone has a recommendation for something that is, and supports 4096 bit keys, I'd gladly hear it.
I love this. I hope the 'COST' measurement takes off.

I'm not going to go as far as some in condemning the latest frameworks, but I do agree that they are often chosen with no concept of the overhead imposed by being distributed.

Is there anything similar comparing 'old school' distributed frameworks like MPI to the new ones like Spark. I'm curious how much of the overhead is due to being distributed, network latency and Amdel's law, versus the overhead from the much higher level, and more productive, framework itself.
>felt physically dizzy

Seriously?
ouch, dat STRETCH ;)
Yep, sending the prints over is just getting started.

Similar to razor blades and little balls that go into writing pens. Few companies can make them well. China has to import little balls from Japan (at least for high quality pens).

Sending over work instructions will help (somewhat). China will need veterans of the process to work with local Chinese vendors, explaining how to do each step.

The Ukrainian staff will resist handing over their work babies. Also, the subcontractors (heat treaters, metal workers) will not hand over their secret sauce.

China will figure it out eventually. This will help. Still many years away from mastering it though. They will have to source from Ukraine for quite a while.
These are valid points, and I agree many underestimate the cost of retooling and infrastructure. However, I am working on a team of smart engineers, but shell scripting is new to them, much less learning a full Hadoop / spark setup and associated tools. Luckily, you can often have your cake and eat it too: https://apidocs.joyent.com/manta/example-line-count-by-exten... Super useful system so far, and my goal is to allow our team to learn some basic scripting techniques and then run them on our internal cloud using almost identical tooling. Plus things like simple Python scripts are really easy to teach, and with this infrastructure it can scale quickly!
Seems to pretty much sum up the MS MO ever since they got the DOS contract with IBM.

You could observe something similar with how they dislodged Novell from the office network, and how they responded to Netscape trying the same.

I worry that we are seeing similar tactics being deployed by other entities in an effort to grab control of the Linux ecosystem.
It could be that art is highly subjective from species to species. What is stimulating to a chimpanzee may not spark any emotions in a gorilla, and vice-versa.

Just because 'bots may not create "human art" doesn't mean they'd be incapable of producing art at all.
A shuttle actually works fairly well at around 1/5 the size

As I'm sure you know, the USA actually has an operational tiny "shuttle". But for others here, I wanted to link to it, because it's cute. It looks just like a baby shuttle.

I put the word "shuttle" in quotes, because it's actually not a shuttle, but an unmanned reusable orbital spy spaceplane. Or is it?

https://en.wikipedia.org/wiki/Boeing_X-37

http://phys.org/news/2016-05-super-secret-x-37b-space-plane-...
Hmm. I recommend at least a little preparation, since there's always someone who gets it wrong and has to be rescued: https://www.theguardian.com/uk-news/2016/mar/31/woman-sorry-...

People absolutely should try, and the price of entry is really tiny, but the price of getting it wrong can be fatal. What's also possible is people simply being unnecessarily uncomfortable due to a lack of preparation and then never doing it again.

Heck, I had the latter experience myself: I'd been camping as a kid and got training at school and all that, but the first time I did it as an adult after a gap of ten years I was uncomfortable because I'd forgotten that nights are cold in summer.
Related discussion: https://news.ycombinator.com/item?id=12291529
It's not that simple, since sex is an instinct. It's hard-wired in our brains, not lile sugar or meat that are luxuries. But definitely possible with a ton of self-control.
Robert Moses and the Oxygen of Pure Competence
I get what you're saying, but the Grand Canyon and Yosemite are worth the crowds. When you walk up to the Grand Canyon the first time, your brain can't quite get a grip on the scale. And the first time I went to Yosemite I drove in at night and had no idea what I was in until I woke up inside that gigantic valley. Awe inspiring experiences.
> Who says it would cost launch mass to soundproof rooms ?

I'm open to zero-weight soundproofing ideas that doesn't require a significant amount of wasted volume.

> Everyone has their limits.

And NASA'll be picking people with higher limits. If you "can't relax or concentrate with the sounds of people doing things", sorry, but you're not gonna wind up an astronaut anytime soon.
To come at it from the other side, UBI is sold as a replacement for a number of other social programs. I find it very hard to believe that, in the US at least, any of those other social programs would be dismantled and stay dead. It is more likely that UBI would temporarily replace one or two programs in the social safety net before being supplemented with two or three more new ones.

As both US parties currently have significant trust issues with one another, I would say UBI is a complete non-starter.
Out of curiosity, how would you enforce compulsory voting, and what would you consider acceptable penalties for non-participation?
I think you should go learn history to understand when we started agriculture.

And even so how starch and simpler sugars played in our diets for the past 150,000 years.

The fruits that primates eat are "processed" they let them ferment this is one of the reasons why like alcohol so much because amongst other things we used it to tell us when the fruit is ready to eat and it's at it's highest caloric value.

We have been refining sugar for about 10,000 years if not longer, we have been using fermentation and other processes to process complex carbs into sugars for even longer.
> Wait, this isn't universal in the US?

Anything that makes it logistically easier to vote is resisted in certain states because it would increase the turnout of poor and minority voters.
Websites behaviour like this led me to use toggles for js and images which are disabled by default.

If a click on a link of an article, I might actually want to read the article and not having stuff pop in my face or the content jumping around when the stock photo in the header has finally loaded.
I personally strongly support the Australian method. Make voting mandatory. I'm sure some morons may think that "fascist" or something, but it's objectively the most democratic way to do it.
This article is a great litmus test for checking if someone has experience working at scale (Multi Terabytes, Multiple analysts, Multiple job types) or not. Anyone who has had that experience will instantly describe why this article is wrong.
It's akin to saying a Tesla is faster than Boeing 777 on a 100 meter track.
Many cheess make good alternatives to meat - cheddar, feta, mozzarella, paneer to name a few
OnTrack EasyRecovery Professional Keygen Crack and Serial Number
This reminds of a scene from I, Robot.

Detective Del Spooner: Human beings have dreams. Even dogs have dreams, but not you, you are just a machine. An imitation of life. Can a robot write a symphony? Can a robot turn a... canvas into a beautiful masterpiece?

Sonny: Can you?

Detective Del Spooner: (Doesn't respond,looks irritated)

EDIT : this a scene from the will smith movie
Thanks for writing the article. It was well thought out and pretty comprehensive.

I couldn't seem to find a clear definition of UBI in the article, which helps people discuss the concept as many people view possible UBI programs completely differently.

While I am in agreement that UBI is massively overhyped and not a true, large-scale solution to income inequality, I'm happy that it has gotten so much interest and might lead to productive conversations around tax reform.
So can you go vote near your workplace instead of near your home?

In Belgium we have to vote at a designated location (almost always a school building) but it's on a Sunday.
An established, standardised, existing platform is often more maintainable than a custom solution, even if that platform includes a bit more scalability than you actually need.
AWQL.me is a web console to query easily your Google Adwords accounts with AdWords Query Language. It is born when we developed sunnyreports, a Google Adwords reporting tool. At that time, we needed an easy way to check the AWQL queries we were using.

We developed a simple web console for that. Later, we released it as a free web service for everybody.

The primary use is to query manually the accounts but we added scheduled queries by users request.

Although it hasn't had much traction, we have some dozens of users using it hasn't had much traction, we have some dozens of users using.
Similarly, if we eliminate all possible record keeping and audits for financial fraud, we will have no evidence that it ever occurs.

That is essentially the system we currently have for electoral fraud.
Should be called "A practical guide to building modern web applications".

I see nothing here related to building native applications.
> Also, I thought that astronauts had such full schedules that extracurricular activities were limited?

That was the case until they mutinied. https://en.wikipedia.org/wiki/Apollo_7#.22Mutiny.22_in_space
Yes and no.

Yes the core is BSD derived unix, but OSX is also a massive layer of libs on top that is proprietary to OSX.

So unless the developer takes outmost care to not touch anything that makes OSX, well, OSX, its damn hard to transfer anything over to a _nix from OSX.
Bridge to nowhere shows China's failed efforts to engage North Korea
Not only exotic alloys, but single crystal blades. That's right, the blade is machined out of a single atomistically perfect crystal with no defects what so ever. The main point is that this reduces thermal creep, meaning you can have tighter tolerances and more power/efficiency.
it seems China is starting to militarize under its one dictator Xi, what with the island building in south china sea, desire to build aircraft engines, voicing economic/military threats to other countries when it cannot buy critical infrastructures in those countries (UK, Australia) and continued trade support for north korea (with its bigger nuclear bomb capability). It must be preparing for some sort of external conflicts, what with the internal economy crashing, wto membership ending this year, capital outflows, manufacturing moving out.
Why Apple software updates are much better than Google
What's the purpose of that?
I don't have references on hand, but I can tell you that US medical schools are teaching: Artificial sweeteners do not appear to promote as significant an insulin response as digestible sugars do, however they appear to stimulate gluconeogenesis and lipogensis in a similar manner.

In other words, drinking diet soda may not by itself cause problems, but it will stimulate your body to absorb even more of the burger and fries you consumed with the soda (or whatever other metabolites you currently have circulating in your bloodstream).

I'm on mobile now with limited connectivity. I'll try to edit later with a link to a source.
I think the point is if you really have big data then it makes sense, but many shops add huge cost and complexity to projects where simpler tools would be more than adequate.
ΒΓΕΙΤΕ ΠΡΩΤΟΙ ΣΤΗ GOOGLE – SEO SERVICES GREECE
They use Rust, fantastic! Will put this on my must-read-list. Based on their graphs, it makes one wonder how much literal energy has been wasted using 'scalable' but suboptimal solutions... Of course if you're wishing to start a company competing on data processing (e.g. small IoT startups), being a bit cleverer could let you have the same performance or feature set with 1/10th the overhead costs. So maybe don't let too many people know? ;)