skysight-inc/HackerNews huggingface dataset:
Yep, our Spark deployment, like many others, uses YARN and HDFS. EMR has done some great work to make YARN a great target for deployment of jobs using various technologies.

I'm very much not against the Hadoop ecosystem. The ecosystem represents very real progress for data infrastructure. But Hadoop MapReduce is just not what people should be using to build machine learning jobs at scale in 2015.

Spark makes great use of the Hadoop ecosystem, and I'm primarily interested in future innovations in the big data space that try to work with the Hadoop ecosystem instead of trying to supplant it. Modularity and composability benefit us all.
Wow, I had no idea.

I've gone to the twitter website (on my phone, especially) several times in the past to try and look up "local, now" things -- e.g. I hear a big bang/explosion, or there are fire/police in force nearby, or I thought there was a tremor, or there's a big storm.

And every time, I've sighed and been disappointed that (as I thought) I couldn't search tweets without being logged in, and I rarely ever log in to any services on my phone. And so then I clicked away, opened up facebook/NYT/boston.com/etc. instead.
> So we come to our partners and we can now say “alright, here are the things that if we both do them, this is going to be successful.” 99% of the time, people are like “thank goodness, let’s do this.”

Would be great to see some examples of these successful marketing campaigns.
Interesting article about the history and future of autism diagnosis, but it had very little to do with the title.

"Interview With the Author of a New Book on Autism" is a lot less gripping, I suppose.
Agile with a little 'a' – sometimes forget what Agile is all about
Thanks for sharing your experience, I was thinking on something on the lines of Cordova et. al. but I will take into account your experience.

I think I get what you say about 'certain types of companies making certain types of apps' right now I have a potential client that needs a pretty basic app that could be done easily in HTML5 and made 'mobile' (for his use a webpage would be fine, but they want it mobile) with Cordova.
Twitter is useful to me specifically because they do not filter the feeds. If they start removing or preventing me from receiving content I specifically request to receive, that will be the knell for twitter. I can put up with ads. Even being flooded with ads would be better than twitter deciding to hide content from me.
Predatory publishers criticised for 'unethical, unprincipled' tactics
Interestingly, ~ for logical negation is also a convention in C, which I think is more widely known among HN users. I would have predicted (wrongly, obviously) that the LSAT might use the symbol from logic, ¬. Any idea how they chose the tilde?
It's just skeuomorphism taken to the extreme.

We might not have to traverse miles of virtual hallways, but a lot of designers still seem to prefer to make us flick our fingers a lot. Each of us probably scrolls through hundreds of feet of virtual space every day. Before long, we'll probably begin to hear about whatever the index finger equivalent of carpal tunnel syndrome is.

But at least we don't have to swing our arms in midair for 8 hours a day like the poor folks in Minority Report.
Please state yours!
Well for me (and I presume at least some other people) bad science in works ruins the entertainment value.

For example, if in an otherwise realistic military thriller, the hero finds the runway blocked by a tank, so he throws his F-16 into reverse and takes off backwards, that would completely take me out of the story, destroy the suspension of disbelief, and ruin (or at least severely degrade) the book.

Some of Crichton's books do that for me, as does almost every movie in which they use computers, and the computers are all like, bleep-bleep-bleep-bleep-bleep! whzzzzt! and sounding like R2D2 with every normal user interface action, such as scrolling text and opening windows.
YAY! Japanese support for Stripe.

Unfortunately, I found a few issues:
http://imgur.com/vZbheUP

1) The Zip-Code lookup is GREAT, except it looks up the town name, not the 都道府県 (prefecture) like it should.

2) The prefecture is in english, not Japanese

3) The Country names are also in english, not Japanese

4) Was hoping to see auto-kana support for Japanese names: https://github.com/harisenbon/autokana ;)

Overall, looks amazing, and awesome that you're getting ready for a full launch in Japan.

* Running in IE11 on Windows 7 (Japanese)
Thanks for the laugh :)
It Mozilla on an upswing or a decline since the Brendan Eich / Firefox OS stuff happened?

They're kind of shifting more from technology into politics. I heard they were cutting employees and divisions a lot.

I know former employees have to sign a waiver not to talk about their layoff, but why hasn't Mozilla's layoffs gotten any press, but startups do?

http://www.glassdoor.com/Reviews/Mozilla-Reviews-E19129.htm gives a clue.
At the Hayward shoreline a few weeks back, a guy was using a drone. I ignored him until he buzzed the area where Snowy Plovers (an Endangered Species) hang out. Then I went over to talk to him.

He was very contrite. Apparently his model has a special light that is supposed to light up in a "no fly zone." I don't know what that means, but he was in a Wildlife Refuge, directly under the flightpath for Oakland International (exactly 5 miles from the landing strip) and less than 2 miles from Hayward airport [1].

Maybe the FAA needs to publish maps for drone guys to show them where it is legal. Or maybe someone should start a Wiki showing legal and interesting place to use drones as a way of diverting drone users away from silly places while raising awareness of where not to go.

[1] 37.645292 -122.155191
No, I agree. Twitter as a general-purpose communication platform is impractical. Yet a ton of industry figures have crowded into it for some reason. Twitter is the flashy, arbitrarily constrained version of what fingering other people's .plan files was in the 80s and 90s.
I've used zapier recently and I've found it pretty convenient. My only complain is that their RSS integration lacks the concept of rate limiting. In my case I was using it to process user-events, so I needed a way to introduce some delay between an event an another, but I couldn't find a way of doing it using zapier.
I've used C# but am not on windows anymore, so Visual Studio is not an option right now.

I'm digging the Cordova docs on another tab and it looks promising for my use case.
"Doesn't work reliably enough" doesn't answer the question of why Google didn't test HTTP/2 against pipelining. None of Google's performance improvement claims compared to pipelining, and they've never demonstrated or quantified an actual real-world head of line blocking problem (ironically, other than Google Maps loading very slowly in HTTP/2 because of a priority inversion).

Will Chan is the guy that wrote "it’s unclear what these intermediaries are". Oh well, there's some bad software out there, let's just make a whole new protocol /s. Fix the bad software, or at least find out what it is. If it's malware causing the problems, you don't need to make a whole new protocol you can just get rid of the malware.
"Yeah, my mom can use it. My mom can write optimizing compilers." (10:45)

Devastating comeback.
I wonder if anyone has submitted patches to dbus-daemon adding support for non-systemd "daemon managers". If the article is to be believed it should be really easy. Maybe that could have been done instead of writing a blog post? Or in addition to?
I have nothing against GCE. There's certainly innovation going on there. The new dataflow system certainly includes some very exciting and powerful ideas: https://cloud.google.com/dataflow/

The empowerment of these platforms is something that I'm very excited about. Spark will allow you to go from some fairly basic processing of small files on your laptop to a massive cluster that will process huge amounts of data very efficiently. And EMR makes all of that even easier.

This is something that I hope to convey in my upcoming book: http://www.reactivemachinelearning.com/ One of the ideas that I'm playing with is that big data and small data are basically the same. You should assume that you have an infinite amount of data, and then you'll build your system to handle whatever comes at it. Even if you end up not having a ton of data, you won't be sorry you used awesome tools like Spark.
FF users have almost halved over the last 2 years, despite what feels like solid technology - I wonder what the folks at Opera Software think?

http://core0.staticworld.net/images/article/2015/07/firefox-...
What were the drugs they wanted to combine?
> FF users have almost halved over the last 2 years,

That's not what your linked chart shows. First, the chart is a % market share, not raw user counts. Second, ~18% down to ~13% isn't really "almost half" anyway.
Sure. The story is somewhat different with Hadoop. But it's really quite feasible to adapt your systems incrementally from Hadoop to Spark. One of the details that I elided in the post is that we're just now decommissioning our last Hadoop machine learning job, nearly a year after moving to Spark. These things take time. That's why it's important that new and useful tools like Spark, Impala, etc. work with the existing Hadoop ecosystem.
Yes, but the spellings valid in the trade context may differ from the spellings valid for the other meanings of the word. Consider "dialog" vs. "dialogue", or "disc" vs. "disk" (which have different connotations for the two spellings).
They don't advertise it as complete, but the source data for this map is managed on github:

https://www.mapbox.com/drone/no-fly/
Science Crowdfunding Case Study: Sequencing Lil Bub's Genome
Another thought I just had. The original artists of Richard Prince's pieces were understandably upset because it looked like he was profiting unfairly off of their artistic effort--a reasonable assumption, and something that happens all too often with art shared online.

But the truth is actually worse. He didn't profit off of their art, didn't make money because of the quality of their photos. It could have been any photo. It could have been a frame from one of those rotating E-Mail .gifs from the '90s. It doesn't matter, because the only thing Richard Price was ever selling was his own celebrity. The image is merely the frame in which it was hung.
The fundamental problem here is plainly, and simply, why is an ipc daemon doing anything with starting and stopping processes. I could understand a bit more if something like gnome-session was poorly starting and stopping daemons, but an ipc daemon is near the bottom of the List of things that should have that functionality.
VMware vs. bhyve Performance Comparison
A Simple Secret to Lowering Your Churn Rate
Wyoming Evolving AI Lab on Visualizing--and Fooling--Deep Neural Networks
An Open Letter to YC Founders on Demo Day Eve by Startupljackson
Well, no. The blog post argues it shouldn't be used for this purpose at all and to just use a service manager directly.
The post's author writes a systemd-alike daemon manager named 'nosh'. Hence the complaints about the non-systemd behavior.

Also note that DBus runs on more than just Linux (BDSs and perhaps OS X or Windows), while systemD only runs on Linux.
It's pretty easy. The right way is to patch D-Bus, but you can just drop in a replacement for dbus-daemon-launch-helper, which is a standalone executable. I've done this on a system where interactive logins took place in temporary chroots; we were having bus-spawned daemons started outside the chroot, so I just wrote a wrapper to chroot and then exec the real dbus-daemon-launch-helper.

EDIT: Also, Ubuntu has been shipping Upstart activation support in D-Bus since 2010.
Errors and Exceptions
relevant username ;)
The code mentioned in the documentary was thrown away at a later and they started over from scratch anyway didn't they?
>you're much less likely to walk away with large sums of money

It would be great to have data on this. How many SMB owners make a few hundreds $K a year, for many years, ending up making a lot more than a very successful/lucky early employee making a couple of $M in one shot after many years. Especially in the software field.
"how will consumer like going to a sandwich restaurant to buy something that was made by a robot behind the counter?"

Bank tellers, petrol attendants, supermarket cashiers and packers are all examples of where machines have replaced people. Where time, cost and convenience are concerned it appears people are redundant with the exception of services with very high level social interactions.

"Can it match the higher level of quality provided by a local restaurant?"

Nutritional match: Let's face it, no machine will ever match the quality of of favourite pâtisserie [0] in terms of taste, style and presentation. However a machine with a suitable API and focus would be able to outdo out-scale the best trained chefs with individualised nutrition.

Given the western world has a glut of cheap, fast and fatty foods, a machine able to build a menu selection given the individual nutritional and taste of thousands of individual customers can excel in terms of choice, cost and diet.

[0] Although you cannot see sandwiches, baguettes, they are available: https://www.flickr.com/photos/bootload/albums/72157622498559...
I don't think it's a disdain for .Net. It's more of a disdain for ASP.Net Webforms. I'm definitely not a youngster (I'm 41). But he's been doing Asp.Net Webforms for 12 years. He hasn't shown a "knack for constant improvement". He hasn't been keeping up with the frameworks and technologies that MS has been pushing - Asp.Net MVC, WebApi, Entity Framework, etc.

If he has been doing WebForms instead of MVC, he's probably not up on all of the basic table stakes client side stuff like BootStrap and JQuery.

I know at 41 years old, if I want to stay in development and not go into management and command the salary I want, I can't be complacent. The minute that my company stagnates, I must find another job. That means for me, being a full stack .Net developer:

1. Web - Angular, JQuery, CSS, TypeScript, and Bootstrap
2. Server side web - Asp.Net MVC, WebApi, WCF
3. Knowing how to speak the language of an architect (DDD, Design Patterns,everything that Martin Fowler writes)
4. Database theory and maintenance and EF.
5. Testing - front end and back end automation testing.

I'm not bragging, I know lots of developers who can tick off these checkboxes. If you're not willing to aggressive learn, this isn't the field for you.
I'm also the author of a Japanese learning tool: https://github.com/mikekchar/jldrill I have some advice for you if you decide to use spaced repetition in your flashcards (not sure if you do already as I haven't had time to look in detail).

JLDrill is mostly abandonware at the moment and is written in very poor Ruby ;-). I don't actually recommend that anyone use it. It was written for me. I think the pheonix branch will actually work on modern Ruby, but I was half way through trying a different scheduling idea, so it probably has considerable bugs.

However, if you decide to do spaced repetition in your app, I would humbly suggest reading JLDrill's strategy on it: https://github.com/mikekchar/JLDrill/blob/master/web/src/Str...

Most tools (like Anki) use the SM2 algorithm, but I found that my strategy works considerably better (for me, anyway ;-) ) The latest version of JLDrill has an ability to "forget" items if you have neglected studying them for too long: https://github.com/mikekchar/JLDrill/blob/master/web/src/Usa...

Feel free to ping me if you have any questions.
How do you automate testing for Internet of Things (IoT) products
Learn how to apply automate testing for WebSocket APIs used by Internet of Things (IoT) products http://www.webtesting.io
My email address is in the Ashley Madison data dump. This is my story
agreed. I can't believe I had never seen this until a few months ago. Speaking of great tech documentaries, just got done with Jason Scott's BBS documentary, which was fantastic.
China factory activity shrinks to six and a half year low as orders tumble
Yep, and good RPG programmers can charge very high rates!
Seems like an awesome place to blow off steam. Thanks for the suggestion!
If you enjoy this, check out the PBS documentary "Silicon Valley" [1], which was based on this essay.

1: https://m.youtube.com/watch?v=kzit5-Ulx_4
Annual licensing fees (state bar fees) and professional liability insurance are things that programmers don't pay for* but lawyers do.

*Some programmers have insurance but not to the extent that lawyers are required to have it.
It's terrible. I recently started to use it and I'm shocked at how poor it is. Scroll down, see something interesting. Click. Now I've lost where I was, no way back. Going back reloads the entire page (slow) then, since content is loaded in after scrolling, scrolls one page down and stops. They're basically saying "don't engage too long with Twitter, just keep it quick and then stop".

How they have 3000 employees and no one that has fixed their main product is very strange.
This is why I almost always automatically upvote posts that were downvoted away....
What consoles were you using? The closest thing to PC hardware were the Xbox's, which was intentional. The Playstations used MIPS-based setups with tricks like scratchpad while PS3 had Cell processor. Had it been boring and PC-like, then porting to PS3 would've been a breeze.
It doesn't glorify it, it clearly shows the effect such behavior has on one's personal/family life, and how it can all be in vain anyway. It's clearly a crunch-is-problematic message, if not an anti-crunch message.
The videogame Natural Selection introduced fictional alien monsters, among them the "Onos", which resembles an elephant with the head of a rhinoceros. There were many debates about how to pluralize this - both "Onoses" and "Onii" feel improper.
And this is exactly the reason I use Gmail.
The unique ecology of human predators
How I Knew When I Was a Hacker
Indeed. I made a project a few years ago and maintained it for several years. At one point I just didn't have the time (or motivation, I suppose) to continue updates and essentially abandoned it without much official word, while putting in a good amount of work every few months. The guilt was pretty crushing, so I can understand where the author comes from.

Fortunately, someone offered to take it up recently so I feel much better now.
Logical Increments – Computer parts picker
Oh I agree. I was just pointing out it would happen. Happens every time a company jumps on a mainstream platform unless what they had before was horrific. What you say is the exact model for how device-manufacturers look at security. Well, that plus I.P. protection where appropriate. They mostly use obfuscations for that.
Is it possible to differentiate a downsampled image from an image captured natively at a given resolution? My gut tells me no, but this certainly isn't my field.
Play the game. Art vs. art. Alter the image to be yours. Outdo it. Don't just put your name real big.

Unless we're using a very broad definition of 'defacing', in which case sure go ahead and 'deface' it.
>How would one's answer change if they had never heard of these concepts?

I am not sure who 'they' are, but ignorance of the law is no excuse - especially in the bar exam.
I suppose this is true, but in my experience a low quality bug report is usually either user error or not familiar to the developer. In both of those cases more investigation would help, and often the user corrects him/herself. Only in rare cases does the developer have a magic fix for a (not even formatted) stack trace.
I'm so tired of the frame too. So, how do you define the crisis?
It creates a barrier to doing long form content although, encouraging brevity. The amount of multi-part tweets are low compared to the average tweet.
Apprentice.at – Apprenticeships, not internships
Re: Cost of lawyers, I'd love to try to get H1s. Smart, hardworking.

Why only software engineers?
MVC is a lot easier for someone who grew up developing on the web. But if you aren't skilled at hand rolling your own HTML and you don't know client side frameworks that can take the place of the bloated but easy to use WebForm controls, MVC is a lot harder. WebForms hide the real web from developers.
There's some temporal disconnect here, which I think is making the two of you talk (angrily) past each other.

The D-Bus activation system was written in 2006. So was Upstart. There wasn't a better activator to use at the time on Linux; it was certainly many years before anyone even thought of systemd. This was back when we were actively using hald. And the D-Bus developer had reservations about whether Upstart in fact covered that use case:

http://lists.freedesktop.org/archives/dbus/2006-October/0061...

https://lists.ubuntu.com/archives/upstart-devel/2006-October...

So it is reasonable that the "incorrect" activator was around. Back then, D-Bus (and hald) was our systemd.

Ubuntu added Upstart activation support to D-Bus in 2010, just a few months after systemd v1 was even released:

https://lists.ubuntu.com/archives/ubuntu-devel/2010-December...

So the article has some important omissions of chronology when it claims that "the writers of D-Bus didn't make it capable of talking to any [daemon management subsystems]". I don't think jjoonathan's reading is a misreading at all.
The cynicism that jwz wrote with in this time period about what it's like for young people working in tech was influential on me. I do believe it also comes through in some of his segments in the film.
that's Michael Toy. Along with jz, easily the most insightful person in this film ~ http://toyblog.typepad.com/about.html & https://twitter.com/mtoy

remember Andy Baio released made this film viewable in 2008 and wrote about ~ http://waxy.org/2008/06/code_rush/
I've concluded that art is anything an 'artist' can sell to someone, whether the sale price is reputation or money.
Does anyone else find it utterly amazing how inexpensive desktop computing bepower can be these days?
I feel like every three weeks I see an article like this on Hacker News and a gazillion good ideas about how to improve Twitter's service.

At the same time, there have been no core changes to Twitter in the last several years.

Is Twitter the only company in the Valley where every engineer isn't checking Hacker News every few hours!?
Michael O. Church's rap challenge
Conceded. I still think the ae ligature in "Unix daemon" is a misspelling or usage failure.
Totally agree. I stopped using Twitter because I got so tired of rewording everything to fit into 140 characters. The most interesting people on Twitter tend to use multiple tweets to broadcast one message, and join them up with "...", but when I see that I'd rather they had just written a short blog post and linked it.

In my opinion, what Twitter is most useful for is sharing links and short updates to prompt action from your followers, like reading your new blog post. And because of this, it's not the kind of place where you want to spend a lot of time, because the content isn't that engaging, it's just the news ticker that you scroll through for 5 minutes and leave.
It's madness that the Apache Foundation, the charter of which is absolutely unquestionable, must tread lightly, while the Church of Scientology enjoys (albeit different) tax-free status no matter what they do.
There wasn't a better activator to use at the time on Linux

runit, s6, depinit, initng, minit, eINIT, simpleinit-msb, plain old daemontools with UCSPI, etc. etc.

There were plenty. In fact, before Ubuntu wrote Upstart, they were considering adopting initng instead. Gentoo was experimenting with it at the time, but ultimately went on to make OpenRC.

Back then, D-Bus (and hald) was our systemd.

This statement is absolute nonsense. D-Bus, HALd and systemd are three completely orthogonal and unrelated pieces of software.