skysight-inc/HackerNews huggingface dataset:
People should keep in mind that you do not always have the benefit of green field development; sometimes you have a project already written that does not have much of server-side component, and has no budget or time left for adding real pages/do graceful degradation (much less progressive enhancement.) In these cases, using a spider is pretty much your only choice. Some notes/recommendations:

- I'd recommend PhantomJS (there are some other packages built on top of it, but for my custom needs, using the original was better)

- If you spider a whole site, especially if it's somewhat complicated, log what you're spidering and see if and where it hangs. I started getting some PhantomJS hangs after ~100 URLs. In this case, it can be a good idea to do multiple spidering runs using different entry points (I use command line options to exclude certain URL patterns I know were spidered during previous runs)

- If you're spidering sites using script loaders (like require.js), pay careful attention to console errors; if you notice things aren't loading, you may have to tweak your load timeout to compensate. Using a "gnomon" (indicator) CSS selector is very helpful here.

- Add a link back to your static version for no-JS people in case Google/Bing serves up the static version. This only seemed to be problem shortly after spidering, but it's worth doing regardless (later, search engines seemed to start serving the real version)

- For those wondering how to keep the static version up-to-date, use a cron job, then cp/rsync/whatever your latest version to your "static" directory.

One thing I'd like to add is that I wish PhantomJS would support more things that node.js does (since some of its API is modeled after it), particularly many synchronous versions of functions. That aside, it's an incredibly useful piece of software.
> Erlang is interpreted.

I was under the impression it was compiled into .beam files. Is this just a subtle point that I'm missing somewhere?
Well I guess that's the risk you take by using an off-the-shelf theme... your copy differs significantly if that's any consolation...

edit: (this is the theme https://wrapbootstrap.com/theme/beaker-responsive-working-la...)
Sweet write up. I'm curious as to how CloudFlare's needs are similar/different from other companies (i.e. Facebook, Google, Twitter etc.). It seems to me that there would already be quite a lot of iteration and therefore experience/knowledge for what the best set-up would be? Though a large portion seems to be with new hardware coming out.
It's not fine, that's why his repeated cheap shots about 'do you have kids' were removed. On the other hand, two wrongs don't make a right - the notion of permanently banning him from participating doesn't make a lot of sense either.
Without details of some kind, juicy or otherwise, this will get flagged.
Facebook remembers first chef
> when moving at the speed of light, one moves through time at a velocity of zero

When moving through space at the speed of light one moves through time at speed zero. Likewise, when moving through space at speed zero one moves through time at the speed of light, which is one second per second. This is the normal state of affairs that you experience day to day. It is not possible to achieve zero movement through spacetime. In this universe, everything is always moving through spacetime at the speed of light. The only thing that changes is the direction of motion: more through space, less through time, and vice versa.
Congrats, raganwald! Just don't forget to keep posting comments on HN every now and then ;).
We elect representatives who make educated decisions for us. At least, that's how it works in the USA.

Chance would be fine thing.
My favourite bits of the project;

Elasticsearch

ZMQ

Golang & Haskell

Unbound DNS

And a bit of old Python glue to hold everything together.

It's funny how long things can last when you think you would have outgrown them pretty quickly. A tiny round robin domain server that maxes out at around 60k domains per second on just a single core to distributed spider bots though is just about easy with Python + ZMQ. Plenty to keep a swarm of far more complex and well engineered Haskell crawlers humming away and throwing the results to Golang indexers to push into the Elasticsearch core for ...

Well, I'll leave the rest of it to mwotton's blog post, suffice to say it's been one of those life changingly awesome projects of doom. I'm sold on the benefits of embracing polyglot.
Then why did you make a new account?
launchd, but I can pretty safely assume that I am one of the few here running servers on OS X.
At JacobsParts we experienced this. Every few weeks our central API would just stop responding. We were using a threaded python zeromq server and the REP "sockets" would get stuck, even after the client was gone. Enough stuck sockets and all threads were hung. Now we use python gevent for an asynchronous server and we use ROUTER, DEALER instead of REP, REQ. Actually the clients can use either DEALER or REQ depending on if they are asynchronous or threaded. Anyway the results have been fantastic and stable for a year now. When there are many small requests the performance benefit of ZeroMQ is just amazing. Can hardly tell the difference between a remote call and a local function call!
Show HN: A Search Engine for Developers, Marketers, Web Designers, and Hackers
That was as polite and professional a take down letter as I've ever seen. Although the OP might prefer to get the request from the lead designer, that is not the designer's job. It's the lawyer's job, and this lawyer actually does seem to have way more of a clue than most that write these letters.
I can see what you mean here. I feel like this often applies to SaaS products and Network companies like Facebook, where forerunners in the field have validated the need for something in the market, but failed from a technical or a distribution angle rather than a product perspective.

Its a very good point. I think the 'lean' movement is really meaningful when you're talking about big engineering innovations (where the alternative is often between doing a big build or doing manual work) rather than a network or service business where the concern is actually around the quality of the network or service you're providing, which can't be easily faked or tested.
Deny yourself using one of the best products on the market because you overreacted? That's normal
OTOH, Clojure gets more common as more people use it in examples. Subconsciously, if you keep seeing this strange language every other example code then you start to think that maybe it's not so side-projecty after all.
Microsoft is far from failed. They might not be doing great with tablets and phones, but they have a few other businesses that happen to make billions upon billions each quarter. The PC is declining, it isn't dead. And it won't be any time soon.
MailChimp just set an example for how to craft a strong legal notice while not sounding like a dick. Color me impressed.
مسلسل عطر الجنة الحلقة 14
http://www.alhqeqa.com/?p=17509
مسلسل عطر الجنة الحلقة 14 مسلسل عطر الجنة الحلقة 14
مسلسل عطر الجنة الحلقة 14 مسلسل عطر الجنة الحلقة 14
http://www.alhqeqa.com/?p=17509
the government fears

There are (at least) 51 governments in the USA. For the nth time, it's not a monolithic entity. After all, we have a good numbers of states that have banned or imposed moratoria on the death penalty too, and those states obviously have no compelling interest in preventing scrutiny of its reliability.

massive public unrest

No chance. People (collectively speaking) are indifferent to perfectly well-documented failures of the legal system as it is.
Holy cow, the background of this website is an animation! That is really hard to read.
> However, they chose to go through their legal department, and for that I’ve lost respect for their company. I’ll no longer be managing my project’s email lists from MailChimp.

Of course they chose to go through their legal department, it's their job, not the designers. You can't seriously have lost respect for MailChimp because a lawyer was doing her job. Maybe the designers were too busy, oh I don't know, designing?
That should not be the case, as we've allowed distribution to everywhere. It could be a device issue- are you on a tablet? We're working to fix that issue at the moment. If you are on a phone, what type is it and which Android version does it run?

Feel free to email me (see profile) and we'll get you set up.
yes, I probably should have mentioned the other 90% of the project, hey :)
What a dick move by the OP.

MailChimp made it very clear they realise his intent was not malicious, and the language of the letter further shows this. He reproduced their UI (branding included) without permission and posted it online as open source. The way he has handled this (especially the ranting blog post naming MailChimp's lawyer) is extremely unprofessional. If anything needs to be boycotted it's his consulting business, not MailChimp.
The fact that we have to worry about womens' marital status when addressing them is sad.
Cory Doctorow at Comic-Con: Why You Should Care About NSA Overreach
I work for a company building servers for Facebook. They really did try to follow the Open Compute (http://www.opencompute.org/) architecture but ... um, it didn't work out particularly well. They turned to us for tweaking, Quanta for server manufacturing, and us again for rack integration and large scale testing.

<edit> We are the OEM (e.g. design and build) for large scale storage arrays for Amazon & Netflix, too, but not compute servers.
This looks more like an angry letter with childish points made.
> i hate this I hate that.
> Why do you need to mutate things in order to write a program?

To change the state of the universe from one in which my program is not written to one where it is.

Furthermore, if the program I write mutates nothing upon execution there is no point in running it.
The response to MailChimp's letter is distasteful at best. I don't know how MailChimp found out about Luke's work, but I imagine having a Dev contact someone infringing on their work is not normal procedure for any company. Lawyers are paid to do this professionally, however Ms. Danin's letter goes above and beyond, illustrating the issues they have and their reasoning behind it. They could have had court documents (in the US anyways) show up at your door, or the site / files could have been forced offline by working with the host.

Your response reflects poorly on you. I would say it was fair to mention your employer as it's listed on your public github profile, and it isn't clear that your employer isn't aware. Additionally, I'd imagine you'll be receiving more removal requests from the other sites you replicate.

Also, releasing your files under the MIT license prohibits you from restricting them to "educational purposes", at least to my understanding, but I could be incorrect here.
I want to hook this up to an Oculus Rift.
Congrats GH. raganwald is a class act.

I didn't realize that he lived in Toronto though. Awesome to see the streetcar on the GH blog!
I only have a use for a recording app occasionally, but when I needed one they were all awful. This looks great, I've installed it in case I need it - thanks for doing a nice design!

It would be cool to show the levels while recording, recorded time etc to give us more confidence that it's actually recording and the levels are good
Hey HackerNews!

We're proud to announce the launch of our one-of-a-kind source code search engine.

Our team is 2 recent graduates who are trying to undertake the herculean challenge of building a search engine.

Traditional search engines only index the text of webpages. We go deeper, and get all the HTML, CSS, Javascript, and more.

All feedback is welcome, we're new and trying to improve.
For a paid service, Xbox Live is crap.
Had someone from the MailChimp design or development team reached
out and asked me to take the resource down, I totally would have understood.
I greatly admire their work and respect their talents immensely.

It's like they teach this in asshole 101. You can tell someone is being unreasonable every time this argument arises, "I'm not upset about the bigger picture, I'm upset about meaningless-detail x!".
Two wrongs? You're really asserting that stating my opinion was at least as bad as his behavior? Granted, it's probably a good thing I don't have the power to actually ban someone, but I think I did a reasonable job of at trying to explain the rationale behind my wish. And why doesn't the notion of banning make sense in this case? People have certainly been hellbanned for less.
Wow, it would be cool (and probably wishful thinking) if Raganwald could help make a dent in the documentation universe beyond Github.

I'm thinking Sphinx/ReadTheDocs in scope, version controlled docs with pull requests, output in Jekyl to Github (or other) Pages. Etc.
I'm no authority on JavaCard, but I'd guess it's verbose like Java and expects some sort of auto-completion capable "IDE". I took a quick peek at a JavaCard standard and I could swear it said there are classes.

I don't want no stinking Java, whether it's a small subset of the language or the full blown monster. For the task at hand, I'd have more fun with assembly language than anything prefixed with "Java".

The blank smart card possibilities are enticing. If we can use our own crypto.

Doesn't OpenMoko's WikiReader run FORTH?
Hacker Nation[1]: Do things that don't scale, Hyperloop, Nate Silver
Every computer I own or use runs better with Linux Mint than with Windows.
Instead of (or in addition to) manually creating scrapers, you can use Diffbot to automatically extract this type of information from news articles using computer vision: http://diffbot.com/products/automatic/article. It also allows you to create rules with a WYSIWYG editor: http://diffbot.com/products/custom/
I found the email from MailChimp is highly professional and very reasonable, even though it's their right to get mad.
Pfft. Drove the bus off a cliff, into the ocean, through the fires of hell and into my living room. Fortunately, you can now install a few different Linux distros on their UEFI @#$% loader.
God's phone
Arguing on HN is probably a moot point, but I'll try anyways. Don't get me wrong, I'm aware they could have gone about this in a much more aggressive way. But to go through their lawyers right away was a much more aggressive response than what I would have expected from MailChimp. I guess that's the totally correct legal response, but why not just send me a quick message?
CB Insights and PrivCo absolutely have data on startups, and just because a company is a "startup" doesn't mean that it won't have any visibility in some of the other sources I listed.

As for imperfect data, billions are spent every year on all kinds of weight loss supplements of questionable value. That doesn't mean that there's necessarily going to be demand for your weight loss supplement.

Here, I see little more than an amalgamation of free data from sources like CrunchBase, AngelList, Twitter and LinkedIn along with a proprietary momentum scoring system that is designed to track a startup's growth but for which no efficacy/validation data is presented. I don't see any reference to the specific metrics that are inputs to the scoring system, but the mention of "traffic graphs, app store ranking, inbound links, social media stats" suggests that the aforementioned free data is what is being used.

Of course, "growth" in the abstract is an ambiguous concept and, I would argue, a poor one to try to measure for private companies:

1. What is growing matters. A B2C startup may have a lot more traffic and social media activity than a B2B startup, even though a B2B startup in a lucrative market could be growing its revenue at an exponentially higher rate. This is why angels and venture capitalists will always have to hit the phones and the streets: you can't find the B2B startup that inked $500,000 in deals with Fortune 500 companies in its first six months of existence if you're just looking at Alexa stats or Twitter followers. This information simply isn't being released publicly. If you want an advantage, you have to seek out non-public information. It's why some hedge funds, for instance, go to great lengths to physically monitor in real time the supply chain movements of companies they have or are considering positions in.

2. Very young companies can more easily grow at a faster rate than more mature companies. Going from 0 to 1,000 visitors in the period of a week is generally easier than going from, say, 20,000 to 30,000 in a month, but the former is arguably likely to be a lot less meaningful than the latter for most businesses.

3. How big you need to grow to have a sustainable, highly-profitable business varies substantially from company to company and market to market. In other words, it's all but impossible to distill "growth" down into a single number that allows for an apples to apples comparison of companies in different spaces and at different stages of development.
I'm off topic here, but I think this is more an artifact than common place anymore, right? While Mrs and Miss still exist, I've never seen them anywhere aside from formal invitations. (except my third grade teacher who required 'Mrs.', but that was almost 20 years ago)
Good point! We've thought about showing time before, but levels would definitely help too.
Could be, it is in front of the monitor, but it's flat on my desk in more or less the same place shown in most of their demo videos.
NASA/Industry Test 3D Printed Rocket Engine Injector (1/3rd time & 70% less $)
It doesn't help that the writing in the article is pretty bad.

I don't think M$ will go under (think: Fortune 500 + gov't), but as a consumer I don't want any of their new stuff.
60% huh? It sounds like it's time for anti-trust case to me, hey DoJ.
The massive stalling issues are probably a result of your ISP. I know that Comcast massively throttles YT video downloads after the first thirty seconds or so of the video have been downloaded. (I've verified this by directly downloading a particular video file at home, and at a Uni across the continent. The download at the Uni downloaded at maximum speed until the rather long video was downloaded.)

[download, download, download!]
Does nginx ever really crash, though? It hasn't in my experience.
The letter you got is a friendly request ("quick message"), not a legal demand. Why does it matter who it came from?
I feel like there's a lot of articles like this, which kinda make it seem like life is easy as an entrepreneur, or that successful business just happen. I feel like this mentality is behind the explosion in early stage financings relative to late stage financings.

I know that as an early stage founder that I am part of the problem, but I often stop to think about whether what I'm doing is really that valuable or if I should just join another company in growth phase. I think the 100th engineer at Facebook would have had a lot more impact than most of the failed founders today.

I just think that being a founder is a harsh experience that you have to learn from many times over in order to get right, and that even when things go well there's no guarantee of a money pot at the end of the rainbow. I think you really need to have motivations other than cash to truly convince yourself that what you're working on is worth waking up for every day.

I think if you have that motivation inside you then being a founder is a great experience. Thinking you should just leave your job for an instant moneypot is just silly though, IMHO.
I thought so too. Certainly not the technical breadth of YC, but on par in terms of uniqueness of ideas, founder quality and traction. I thought our YC batch was a force to be reckoned with, but wow, this year's FonderFuel batch was killer.
Growing up in the 70s, we looked at news from the Soviet Union and other authoritarian regimes as blatantly untrue propaganda. By comparison, our press and government practiced transparency -- they may have lied, but not on the scale the others did.

I keep slipping back into believing the United States government believes in honesty and transparency, at least relatively more than other governments. In some areas it might, but not on major issues like war.

It's painful to watch a government that instituted the Marshall Plan actively destroy its credibility like this. Or to think of the consequences. Or to see them play out.
This numbers are a bit higher

http://drones.pitchinteractive.com/

And that only counts Pakistan. Afganistan, Algeria, Iraq, Libya, Somalia and Yemen were attacked by drones too

http://www.policymic.com/articles/23708/guess-which-8-countr...
But the main point of the article was (which I strongly agree with) that running stuff like Autotools, gems, or package manager kind of sucks when you just want to install a console tool.

At least back in the "old times" we were distributing program + additional runtime, and not requiring a random unexpected runtime in proper version + whole supporting infrastructure + program source code to build. The former option greatly simplified installation.
It's a tone argument, which is toward the bottom of PG's hierarchy of disagreement, not to mention many other places (e.g. random google hit http://lucereta.wordpress.com/2011/06/30/tone-argument-as-lo...).
But system operators love to use God. It's that whole male ego thing.
Props for mentioning Fairphone! I can't wait to get mine in October.

Ubuntu Edge does look good though, mostly because it should be a viable hybrid Mobile/Desktop replacement. While I won't be supporting it, I do hope it goes ahead.
As a developer I hate that MS does not upgrade IE browser when they issue service releases. Ex: W7SP1 did not upgrade to latest IE.

This causes me to have to develop 2 versions of my product, increasing expense. Shame on you MS.
I've seen much more aggressive emails and letters, although I think this could have been better-resolved with a phone call to attempt clear up the immediate issues (e.g. proper attribution, terms and conditions) before even bringing up the word 'legal.'

MailChimp's concerns are quite valid, and to an extent you have to enforce IP issues, but OP is right that this happens all over the internet. If you lookup 'Dropbox' on Dribbble, you'll see tons of reproductions and remixes of their UI. I see OP's PSD as something made out of respect, and something provided to the community as an educational resource.

That doesn't change MailChimp's legal rights -- but regardless of who is right and wrong, I think a lot can be accomplished among adults having collaborative conversations. I think MailChimp is asking "How can we protect our IP?" instead of "How can we protect our IP while continuing to engage the designer and developer community?"
The anti-Microsoft articles on SA are pretty over the top. There's a grain -- often more than a grain -- of truth, but to lump MS in the same category as Sun, or one of any number of failed large companies, is premature.

They've got trouble. I don't know if getting rid of Ballmer would solve anything (it might be necessary to start a long road to reform). I can trace back a /lot/ of MS's bad decisions to the way they review and compensate people, and Ballmer seems dead set on keeping the current review system intact, which is a shame.