skysight-inc/HackerNews huggingface dataset:
Yes and no.

Some crawlers are most interested in freshest versions of the most inlinked articles, or in the exact HTML presentation at Wikipedia.

The monthly full raw wikitext dumps don't provide that.

And, Wikipedia's serving plant is pretty efficient, with bandwidth only being a small portion of their costs. They can afford some crawling... and correspondingly, their /robots.txt is pretty open.

Good crawlers seeking just the bulk text shouldn't try to grab the whole thing as fast as possible via the standard web URLs... but other good crawlers may want or need to visit discovered Wikipedia links, and doing so at a measured pace should be OK.
Fair point and makes sense. If you publish the rank along with the data itself that would be very useful. Perhaps having a few sets of data? 3 million top pages, 3 million deep pages etc...

Personally I would like to see around 20-100 million pages or whatever is about 500-1000GB. That's enough data to work with on a local machine and serve up some meaningful results assuming you want to build a search engine or just do some deep analysis of the web.
GNU Parallel – The command line power tool
Evernote and Telefonica announce global partnership
> Thats a huge stretch and abuse of logic IMHO. Don't build roads because criminals and terrorists will drive on them.

Just thought those two sentences were funny being right next to each other.

Let's be honest here, though: The percentage of people using his email service for illegal reasons is much higher than the percentage of people using roads for the same illegal reasons. It's the same problem that Pastebin faces,[0] and it's the reason paste.pocoo.org shut down.[1] Services that advertise complete privacy and anonymity get swamped with people who want to hide illegal activity.

[0]: http://www.tgdaily.com/security-features/62490-pastebin-to-p...

[1]: http://paste.pocoo.org/about/
To be honest a simple crawler is a very simple thing to write. If someone had issues getting that going I think they are going to have issues with the data volume anyway. LisaG answered why the 3 million data set though and I agree with the reasoning.
Yeah I agree with Ryan's point there I was just replying to the GP about how nobody cares about programmers and how programmers who whine and complain are just easy to replace like cogs in a machine.
x265 Project Announcement
Did you eventually get better at typing on the Kinesis?
Be glad for the rise of Clang/LLVM. When there is a serious competitor, quality will stay higher and the projects will evolve faster. ie. Clang will force gcc to become better. gcc will force Clang to become better. End result, we all WIN!
This.

Kids watch adults to learn the adult behaviors that lead to safety, community, and procreative success. If the only lesson they ever learn is that their behavior when bored is the center of the universe, they are going to have a hard time as adults.

Around your children, you should behave in a way that is reasonable and ethical and safe for an adult to behave. You should also make it your first and highest priority that they don't die. If you do those things, then go ahead and play with your smartphone.

Chances are, when your kids grow up, they will find work, friends, information, and entertainment valuable as well. There is no harm in teaching them that you value those things too.
It's a bit pricy compared to your standard $.99 iOS app, but next to a $9.99 ebook it seems pretty reasonable considering the quantity and quality of content and the unique interactive elements. Like other Touch Press releases I think of this as a sort of coffee table book for your iPad.
There are at least 2.5M English wikipedia pages indexed in the crawl:

$ cci_lookup org.wikipedia.en | wc -l
2516956

(See https://github.com/wiseman/common_crawl_index, but note that the index is incomplete.)
I don't care if the government asked for the data of one particular user, when investigating a serious crime and having probable cause. Did you read the documents you linked to? Are you saying that you don't believe that investigating authorities should ever be able to access the contents of an email account, even with probable cause?
Larry Ellison: Google's Larry Page acted 'evil'
The short term plan was to make sure that we got developers up and running again. I know for a fact we're going to do an internal post-mortem for sure. To be 100% clear and honest, I'm not sure if that will turn into an external post-mortem or not. I'll personally push to do one, but I can't make any guarantees. (It's not my call so I can't promise.)
Do you have some app ids so I can go and see what's up?
blekko attempted to implement crawling a local copy, and it was a PITA. We'd rather crawl the real thing with a crawl-delay of 1. Best would be if the Wikimedia Foundation made a .html dump available.
should I mention futurama?
Why would you have no idea? If you're querying by id, you have one, if you're querying by anything else, assume you have more. Once you assume you have more, you use each() to operate on all or only() to narrow the list. Those functions handle the abstraction for you, just like a jQuery wrapper does.
Dear Project Manager, I Hate You
From a quick glance, it looks like you'd need to make Gumbo a backend for lxml. Is that even possible?
I've run jsonip.com for years. It supports millions of requests a day. More than anyone else, its organic growth and scale has surprised me. The server I have costs $50. That provides me with enough capability to support jsonip and other sites. There's not a chance in hell I'm going to kill it. That being said, things have been tight for me recently and I'm trying to get this straightened out. But I figured it would be worth the 10 minute effort to see if any of the hundreds-thousands of people that use the service on some very, very high traffic sites were willing to chip in a bit since its been free, and will remain so indefinitely.
I hope more CEOs and other executives do this too.
Aaah, I see. Not sure if the page was updated, or if I overlooked that bit. But at any rate, it's good to know. I mean, I assumed it was something like that, but it's nice to get some clarity.
There is something very, very wrong with the world.
You said your approach was to "make a bunch of movies for not a lot of money." No one's arguing the "not a lot of money" part, what I'm saying is that you can't make a bunch. You can produce one really great movie every two years if you're amazing. That's it. It will take you twenty years to get to "a bunch". That's after ten years of learning how to make movies. It's that hard.

This is my friend's website. He takes his craft very seriously and is a workaholic. No day job, this is all he does:

http://www.villainpictures.com/work/index.html

Five finished, full-length projects in the last 12 years, two more in the pipe.
From Ichan's Twitter bio: "Some people get rich studying artificial intelligence. Me, I make money studying natural stupidity."

Ichan probably foresaw the impact of his tweet. He probably made millions, and will make a few more million by taking a short position on AAPL in the coming days.
You're entirely right, I was mistaken about the author. It sounds similar to a Stephen Wolfram post though; these guys really, really love Mathematica (understandably), and every post goes to great lengths to explain how they've done something incredibly novel which was only possible because Mathematica is fast and versatile, etc. The end result was really impressive, and I know they got there using Mathematica (in some cases), but I was more interested in the techniques and the approach they took than the particular language they used. All of the puffery and going out of the way to talk about how terse the code was seems excessive.
> Both terms encompass both copyleft and non-copyleft (all-permissive) licensing.

Then what's the difference between the two terms?

I've always thought that they were quite different (RMS vs ESR). Especially when capitalized as pronouns.

Edit: Nevermind. I missed ajross's response.
Sinatra: Some Questions from Ruby Newbies
That was only opened 6 days ago. Seems premature to be complaining about it.
Is there an obvious trick to 8a that I'm missing? I've been staring at it for longer than I'd care to admit...

Edit: by "8a", I mean the puzzle that leads to 8a ("get out your notepad")
I wasn't saying Steve Jobs was self-promotional like Wolfram, but a lot of Apple reveals involved 'incredible, never-before-seen' sorts of language. Language like "as far as I know we're the first people to ever do this" seems like the kind of thing Steve would say about a product; these guys are just applying it to a project they worked on.

If I'm honest, I'm probably just very cynical about any sort of salesmanship.
> If it doesn’t augment the human condition for a huge number of people in a meaningful way, it’s not worth doing.

this is the kind of arrogance i find disheartening about these kinds of lists. really - things aren't worth doing unless they clear this impossibly ambitious bar?

should i tell the owner of the corner cafe i stop into after work every other day, and that gives me a huge amount of joy nearly every time, that she should give it up because her business doesn't "augment the human condition for a huge number of people"? how about the guy who wrote the $5 clipboard app that makes my life much easier? i'm sure he has lots of customers but "a huge number" that have "augmented" their humanity because of his clipboards....

and i hate to be an ass, but ... under whose criteria does rethinkdb qualify for this, anyway? how meta do we have to go here? how many degrees of separation are we talking about exactly?

the problem with these kinds of statements is they can be rationalized by smart people into all sorts of justifications for doing and/or not doing a huge number of things.
Android is better? Not if you look deeper
A whole $2 a month! You're a god damned saint with such selfless generosity.
I'm curious how "trustworthy" this is -- i.e. how hard it would be to spoof. If I could trust that a certain configuration of ~12 estimote beacons changing signals in a certain way as I walked through my home meant I was at home, I'd be comfortable using 4-digit PINs instead of passwords for a lot of things.
Interesting enough, looks like there are changes to routeProvider that's causing injection to failed. Unfortunately, not a drop in replacement from 1.1.5. Anyone else experiencing same issue?

Failed to instantiate module editor due to:
Error: [$injector:unpr] Unknown provider: $routeProvider
You are right, my bad with the low pressure speed of sound . Anyway the high transonic flow isn't a piece of cake as well.
https://github.com/html5lib/html5lib-python/issues/105 seems to imply that such a thing is possible. I am unsure about the requirement for lxml. I was under the impression that lxml is an optional walker, the default is the slower pure python walker.
I don't think any designer feels "threatened" by Bootstrap. Almost all front-end devs use either Bootstrap or something similar (I'm a fan of Foundation personally).

The complaints probably are from devs and designers, but that doesn't mean users like Bootstrap. Devs and designers complain because we know enough to recognize, and complain about, the source of all the monotonous websites coming out these days.
That's completely irrelevant. The non-human-readable SVG format can't translate to this topic in any respect.
New York is probably even stronger than the Bay Area for this kind of service, but there are entrenched competitors there.

College campuses, big industrial parks, and maybe military bases would be my next pick after NYC and SFBA. Another interesting market might be events -- during Sturgis, or Oshkosh, or other vents, running something like Instacart would work, while it wouldn't work in the same towns otherwise.
Before the GPL life was nasty, brutish and short.

... never mind that GNU was around for 6 years, and BSD was around for 12 years before the GPL was published.
Just wanted to let you know that this comment of yours from last year was a huge help: https://news.ycombinator.com/item?id=4476649

Thanks!
v3 will ship with a v2-ish theme that you can optionally enable.
Apple acquires Matcha.TV
A closure. That's true, but it doesn't change the behavior inside a node module at all, they already have their own contained scope, you have access to the same globals `module/exports/global/require/etc`, and `this` is preserved.
I put them in the comments as you suggested. The main one was just restored 288731291242658. I don't actually know the full list of IDs, but it looks like the ones I care about were restored.
Hi, I work at Facebook along with Chris. All of the accounts and apps should now be restored. If any are not (and were disabled between 3pm and 5pm PST today), please let me know the app ID and I will investigate. My sincere apologies this happened - we are looking into ways of making sure this cannot happen again.
Due to https://github.com/angular/angular.js/commit/5599b55b, you'll now need to explicitly include angular-route.js and add a dependency on ngRoute.
You have to license any code that links with GPL code under the GPL. This does not prevent you from dual-licensing your own portion of the code base, or shipping non-GPL, non-linked code as part of the same product (Although in the case of things like DLL plugins it gets a bit hazy)

Actually, that's a common, but wrong interpretation. The whole work becomes GPL, but the code that links to GPL code doesn't have to be GPL.
He's most likely only talking about venture-backed startups or those startups that have the potential to be venture-backed but forego it for other reasons. These are obviously not "57 lessons for any business anywhere", especially since there is an entire section called Fundraising.

In that context that's a perfectly reasonable piece of advice. Without an "enormous potential for growth" you're likely not going to get substantial interest from investors or have the kind of outcome that folks joining your "startup" would expect.

If that's the case then your critique is off-point. He was deliberately excluding the those kinds of businesses from his analysis.
My suggestion for all non-fiction writing is On Writing Well by William Zinsser. It's got sections on many different topics (including some technical writing), but its stylistic advice is the best.
These should now be restored. Can you double check for us?
A 50% pay increase? Go for the money.

Don't hold onto your preferred practices so strongly that you forfeit your own market value.
The only thing you have a say in is code that you have copyright on.
Question is whether that corner cafe falls into the category of a startup or not.
Can you include a sample of a template with your custom directive? I'm not sure what the expected thing out of the box would be. (not the code of the directive, just the use)
OK, thanks. As of a couple of minutes ago we finished running the last job to restore the IDs. If you can still find one that's missing please let us know what the ID is.
Herschel Discovers Infrared Light
Is there a way to communicate that to users better? I don't think a lot of devs understand that it's easier to build up on a flat button as opposed to trying to overwrite stylesheets on a gradient button. Maybe something in the features for the final release that helps developers understand that it's not flat for flat design, it's flat because it's easier to build on top of.
You'll find me trying to walk at odd times of the day to hit my 10,000 steps.
I'm no lawyer but I think Mr. Levinson should lay low and avoid talking to the media. I think he is already in deep trouble, if he keeps talking, he is pretty much challenging federal prosecutors to have him "Swartz'd"
"[The subpoena] is extremely broad and onerous....While the [Department of Financial Services] says they want to have a conversation with the industry, their actions speak the language of aggressive litigation."

Sets that expected tone of heavy handed future `presumed guilty' confrontation for any new process remaining beyond wetting their beak or their manipulation to directly corrupt or inflate.
There is good advise in this list. But I also know counter-examples from "successful" startups for many of the points.

Given that, I'll propose a 58th: be formidable enough to overcome any of the other 57.
Apparently I lied here. Now it should be restored.
Definitely a good comparison....

And it will be an odd thing though when you have a coffee table with an ipad instead of books... I think that's where Surface will come in handy.
It has to be "GPL-compatible" though, which means whoever you give it to can convert it to the GPL by adding the extra conditions of the GPL to it.
If somebody sells me a service saying they can't provide that data, then I expect that data to not be provided. It's what I'm paying for, and it's the commitment they made to me.
What the fuck, NSLs have taken down the first and fourth amendments and are turning towards the third.
This is exactly GCC's architecture as well, where front-ends are transpiled into a generic language, which is then compiled into native code for a given platform.
>Let me tell you who the savers are: mostly retirees (401k), elderly people who use their life savings to pay the bills, etc, and others like successful businessmen who just want to save. I just don't think that it is morally ok and I know that it is not economically ok (due to moral hazard)

and don't forget bondholders. But yes, inflation (I mean, flat, across the board inflation, and it never actually happens that way) tends to hurt the rich and tends to benefit the poor. The view that the rich are rich because they are morally superior is pretty common, as far as I can tell; more common here than abroad. It's a view that I disagree with.
Why do you feel node modules are a hack? They are part of node.js core and have very well-defined behavior. Most developers prefer them over the proposed `import` syntax in ES6.

I have nothing but praise for NPM - it's simple, fast, keeps dependencies isolated (but can dedupe if wanted), and did I mention fast? It runs circles around any other package manager I've used, except maybe for homebrew.
what is protection from "pump and dump" here? He isn't allowed to dump the stock for some time? doesn't seem so. One reason Buffet, for example, has special reporting arrangement with SEC is to prevent such volatility, and as a side effect i think it also decreases potential for such pump and dump on his side (not that i imply that Buffet would succumb to such temptation :)

edit: after refreshing myself on Buffet's exception, it seems it protects his interests from volatility, not public's :).
Cashflow.
Fukushima Radiation Affecting Americans And There’s No Way To Stop It
Predictions of technology from long ago just aren't interesting, since it's trivial to fantasize about technologies without having to think about implementation.

I can predict today of future flying cars, teleporters, mind-controlled interfaces, Turing-test AIs, electrically powered autonomous airplanes, giant tubes that go through the center of the Earth and take me directly to China, a massive spaceship that takes all of Earth's 20 billion people to a new galaxy in the year 2380, etc. etc.

Doesn't mean that I know anything.
This is covered in some detail in the proposal. The capsules will be able to stop in the event of an emergency without hitting capsules in front or crushing passengers.
Interesting points you bring up.... I've never consciously noticed a general color palette trend but it is interesting to look at this sample and see how Disney has many similarities across film periods.

I'm definitely keeping an eye open for Blue/Teal/Orange now
It's funny. I _just_ got back from seeing a VC earlier today on Sandhill Road, and when I told them we were working on building out a product, they told me I shouldn't focus on that and should come back instead with a business plan. I'm fairly convinced that if I'd gone in with a business plan they would have told me to come back with a product.

Ultimately I think PG is correct. Focus on building out the business. If you spend all of your time planning instead of shipping, you're going to fail.
Both of these are back, just verified.
If only they did this for Pixar films and then you could've said that Steve actually had a hand in this...... ;)
The Jawbone Up bracelet figures out when I'm cycling. However, it doesn't understand the intensity of the activity but that's OK as I track that through other means like the PowerTap on my bike.
i use GNU parallel exclusively in place of xargs simply because it has --dry-run.
They should all be back now. If not, trouble you to post the app IDs?
Musk's design is neither a pneumatic, nor a vacuum tube system. It's more like an enclosed high altitude system.
If I give you Tivoized BSD code, this is open source but not free software, at least since GPLv3.
I have no affiliation, but I'll plug this awesome Chrome add-on[1] for finding discount codes at checkout. I'm not one to go actively hunting for discounts, and this has saved me probably tens of dollars since installing.

[1] http://joinhoney.com
Here's the original article I took inspiration from:

http://sporto.github.io/blog/2013/06/24/nested-recursive-dir...

I've modified it a bit to add a $watchCollection listener. So when a recursive array is updated, it deletes a child OL and then recompiles again:

https://gist.github.com/jchester/6227865

I don't particularly like the fact that this creates a dependency on jQuery (through .empty()).

There are other approaches where the recursion is sorta-kinda-but-not-really built into a single directive:

https://gist.github.com/furf/4331090

https://github.com/eu81273/angular.treeview

Edit: I guess what I wished for would be something like this --

<div ng-repeat="recursive items in sub_items">{{foo}}</div>

or

<div ng-recursive="items in sub_items">{{foo}}</div>

or

<div ng-repeat="items in sub_items" ng-recursive>{{foo}}</div>

Whichever is easiest to get into the mainline code.
I did not understand the 'political' issue: The GPLv3 license does not apply to executable files created by gcc; you can even use gcc to create proprietary products. What is the problem of using gcc 4.7 in order to compile BSD if the license does not apply to the executable - BSD in this case ?
> The proposed Hyperloop's operation is quite different from a pneumatic tube, which forces objects through with air pressure.

The Hyperloop is similar to a pneumatic tube in much the same way the Segway is similar to a scooter.
ah. cool. I remember seeing it somewhere. thanks for the tip.
Pseudoscience? Here?

Placebo pills seem a lot less time consuming than the suggested above...
I use GNU Parallel. I like it because its interface is simple - input is piping filenames to it, just like xargs, and output is nicely collated to the screen.

I used to use ppss, which does the core task just as well, but the interface is more complex.

I mostly use these tools to optimize large numbers of PNGs before deployment, using optipng, pngout, and/or my own lossypng. These programs take a while to run so using all my cores gets the job done a lot quicker.
This work does a great job espousing the Visual Information-Seeking Mantra: "Overview first, zoom and filter, then details-on-demand," a cornerstone of effective visualization . http://www.infovis-wiki.net/index.php/Visual_Information-See...
Except scooters are wildly more practical than Segways, and a pneumatic tube at the proposed scale simply doesn't even look good on paper.
True, we could be louder about that. I'll get a tweet out about it shortly. To be honest I totally forgot about even mentioning it anywhere but in a few @ replies and issues on GitHub.

And once we add the theme, it'll become clear :).

Thanks for the feedback! <3
Sort of a Concorde-railgun, if you will...
Or a series of third parties can compete to offer the infrastructure, which is just a CDN tied to a small computer with an HDMI out, and the former networks and new production houses can offer their programming on one, some, or all of them.