skysight-inc/HackerNews huggingface dataset:
I used to scrape web for a daily deals search engine i wrote for a client in 2010. But we scraped in realtime as the number of sites were really low (in 10s).

pre-crawled copies with distributed processing platform could be cool. you could come up with a better search engine with programmable rules that are edited collaboratively (like wikipedia)
DevOps and The Enterprise: A Match Made in Heaven?
There are also some forest paths that I'd like to track. For them I guess I need GPX tracks (they can't be seen from satelite). I've installed GPSLogger. I guess I could just upload those files to Open Street Map for use in the iD Editor.
Simon Fraser University has a well-regarded NLP group. That might be a good place to start networking if you were looking for something in Vancouver...
Twitter feeds move pretty fast. People might not notice a cry for help until it's too late.
Brutal honesty, delivered kindly
OpenTable Mobile Payments National Expansion Kicks Off in New York City
ElixirConf 2014 Videos
Life is a struggle. I'm edging more and more towards life NEEDING a struggle to reach it's potential. The struggle must be balanced, but it is necessary.
Testing iOS apps
There is a closet industry for scraping any sort of data that can move markets. Fed, crop, weather, employment,etc.

Anything that is released at a certain time on a fixed calendar, you can bet that multiple parties are trying to scrape it as fast as possible.

If you can scrape this data( the easy part), put it in a structured format( somewhat hard) and deliver it in under a few seconds(this is where you get paid) then you can almost name your price.

It's an interesting niche that hasn't been computerized yet.

If you can't get the speed then the first 2 steps can still be useful to the large number of funds that are springing up using "deep learning" techniques to build a portfolio over timelines of weeks to months.

To answer the question of:
> Wouldn't this require a huge network of various proxy IPs to constantly fetch new data from the site without being flagged and blacklisted?

This is why I gave the caveat of only looking at data that comes out at certain times. That way you only have to hit the server once, when the data comes out, or atleast a few hundred times in the seconds leading up to the data's release:)
I'm guessing there are legal requirements that prevent them from truly deleting your content.

Abusive ex-husband posts pictures of himself making threatening remarks to his wife. Other potentially incriminating documents. If he was able to delete it, then that evidence is lost. My only thought of how to preserve privacy rights is to have the "deleted"-not-deleted user files sent to a legal-compliance department and out of facebook's user database (where I have no doubt they're using the "deleted"-not-deleted user's files in analysis.
Content generation for employee acquisition
Unless they've changed their web page, I wouldn't really call this fine print. It's in the FAQ and it looks to be in massive print at the top of the Fees page.
Here's a link to how many visas these companies steal: http://abcnews.go.com/Business/top-10-companies-request-visa... . They're literally flooding it (Around 75%, from the looks of it)
Actually, this has nothing to do with surge pricing. From the article:

"Working in the Hamptons is a perk for Uber’s top drivers. Only those with the highest ratings from riders -- at least a 4.7 out of 5 stars -- can operate in the area because “Uber wants to preserve its image keeping a high-standard of service,” Cosentino said."
We do a lot of live web scraping of product information from retail sites for http://agora.sh. We basically scrape all of the essential product info and offer it to the user in an optimized view (we call it the 'product portal') that can be accessed without having to load a new page. This reduces tab sprawl and provides a general improvement to a lot of shopping workflows, such as just wanting to see larger/more images of the product (likely clothing) and being able to do so with an overlay on the existing page.
In the US, at least, I believe there is no such legal requirement unless the party doing the deleting is aware of pending legal action relevant to the documents being deleted. I'm not a lawyer though and may be wrong.
The H-1B system should at least be an auction, not a lottery. The companies that use the system to source cheaper replacements for current workers would get priced out of the market and the individuals who contribute the most to the company (and the economy) would get in. Bonus: more revenue would be raised by the government.
Nice, thx for the reply :) I actually read your blog yesterday even before posting the question on HN. I liked the small project you did with Angular and Node.
CFPB warns consumers about Bitcoin 'Wild West'
Sorry, I have no idea. It isn't anything I have experience with.

Best of luck.
Its almost as if requirements went up over time. Google Play Services has also made the problem worse over that time.
Just update my first iOS game (Orbit Birds) and would appreciate some advices
And indeed one way of making existing asbestos building materials safe (or at least, safer) is by spraying plastic on them to encapsulate them.
US agency warns consumers about Bitcoin risks – US News
I was a presenter. I wasn't entirely sure what I expected from the conference when I submitted my talk, but I was really impressed with the breadth of topics as well as some early commercial adopters of Julia. Props to the creators and maintainers for making this a reality, you all have some serious talent working behind the scenes and if that keeps up I see only good things for Julia. I look forward to the next conference, and hope after people see these videos they may be encouraged to come to the next one (how about Houston? :) )
Reverse engineering is probably not the place to learn to code.
Enabling Collaboration Across Diverse Cultures
I'm not so sure he's actually being protected in any real way by the Russian government, as you seem to claim. Although the FSB is probably keeping tabs on him too.

For the rest: what watwut said in a nearby comment. I'm just extrapolating from things like the Evo Morales incident.
The DevOps Diet (Getting Lean)
Yes get the MD/Ph.D With an EE degree you can build the next generation devices.
Thanks for your reply, how arduous is the process of hiring talent also is it easy to find decent paying jobs?
How a Pair of Jeans Could Save Our Plastic-Choked Oceans
Well, I said VPN has a catch-all for using a remote gateway. I actually use a VPS as well (with sshuttle).
Hi! I'm an MD/PhD student, in my final PhD year (6+2, Netherlands). I'm working on computer navigation in orthopedic oncology, basically objective navigation in the OR. This, together with some small sidesteps into 3d printing and computer supported diagnostic imaging. The combination of medicine and technology is an awesome field to be in. There are so many projects that you can work on.

As Xaa has written, most doctors are not focused on technology. They will use it, understand it but especially the older ones do not embrace the possibilities automation or innovation can offer. That coupled with slow development and access to the market makes us lag behind other high-tech/high-impact industries (for example aviation).

Do not underestimate how hard it is for automation to completely take over a doctors job. In 50 years we will still need radiologist. The tools you will develop will support your and others workflows, making healthcare better. And if it replaces a certain task, others will arise, as for example radio frequency ablation has a growing role in oncology. (and is often done by a radiologist!)

Furthermore there are legal implications. Surgery will not be completely automated (fire and forget)in the next decades, simply because of this. A surgeon always has to be present. For example: the most chosen approach for robotics in orthopedics is assisted surgery, where you move the tool and the robot blocks you from making bad moves.

So, as an MD/PhD you will be a bridge between two completely different cultures. I sometimes joke that the engineers we talk to have a solution for a non-existing problem and doctors no technical solution for an existing problem. It's actually really hard to understand each others fields. So in this you are valuable!

You seem very motivated! I can only draw conclusions on my choice, but I can recommend it! If you have any questions ask away.
Capture the Coins – Bitcoin Challenge Explained
As someone who prefers Python to JavaScript (as far as dynamic languages go), but who has been recently using nodejs a lot, I disagree. I've found that there is a very strong testing culture among node hackers.

In my opinion, the biggest security liabilities in Node are with npm:

1. The culture that emphasizes tons of tiny packages, each bringing in dozens of other dependencies managed by multiple maintainers. The increases the attack surface dramatically. All it will take will be one single maintainer of a small package used by dozens of other packages to expose his npm password, and exploit could easily spread to hundreds of node apps in a very short period of time. I'm surprised this hasn't happened already.

2. On a related note, npm's lack of package signing. Most of the other popular dynamic languages (Python, Ruby, Perl) offer or even require package signing. npm does require SSL, but package signing is even more important, since only package signing can prevent the scenario I describe above.

That said, I do see lots of unit testing in node packages, so I can't criticize them there.
Setting up Sublime Text 3 for Full Stack Python Development
For high level languages, you are correct. However, lots of people learn assembly as they are learning to reverse engineer software.

That being said, I don't think the $200 Chromebook is going to cut it. Eventually he's going to want to run a Windows VM.
I've heard an anecdote along these lines, too. It was about a lathe that had been exposed to carbon-fiber dust starting to turn when supposedly off.
That is also a workable solution. Thanks!
Sports scores, statistics, etc are always high in demand for scraping and great to get people interested when learning scraping techniques.
> deliver it in under a few seconds(this is where you get paid) then you can almost name your price.

Wouldn't this require a huge network of various proxy IPs to constantly fetch new data from the site without being flagged and blacklisted?

Or are you referencing from the time you scrape data to deliver it in under 3 seconds?
Been there, done that.
I had the same experience back in 2008, when the economy was on its peak --before it crashed miserably--
back then I got an offer from Microsoft to work in their new Vancouver development center aka(MCDC). The plan was to get everyone who is stuck waiting for their H1b Visas to start working immediately (because Canada is awesome) while testing their luck with the lottery and in worse case scenario working for a period of 1 year to qualify for L1 visa, the plan was working great, actually many people ended up loving Vancouver and decided to stay there and not go to US at all, however after the economy crashed, the H1b demand fell pretty sharp for the next 3 to 4 years which made the whole development center purpose unnecessary, in addition to significantly slower hiring and layoffs in that time frame.
So, now I see the same thing happening again, and I worry if this is just a déjà vu, hopefully not, fingers crossed.
Random: introduce getrandom(2) system call
Thanks for the offer, what's your best email.
> For years, Nvidia and AMD have been the only ones to provide (at least almost) the latest version of OpenGL (but only on Windows and Linux, not Mac)

The lack of up-to-date OpenGL versions on MacOS-X is not the fault of NVidia, AMD or any other GPU vendor. It's entirely Apple's fault. Apple insists on developing the MacOS-X OpenGL drivers themselves. And because OpenGL has been rooted so deep in the MacOS-X graphics stack it's virtually impossible to update without upgrading the rest of the operating system as well. Well done Apple, well done…
The quality of programmers tends to vary wildly.
There's all kinds of nasty stuff in and around aerospace / military wrecks. It always boggles my mind when I see kids playing on wrecked equipment. DU aerosols, hydrazine, CF aerosols, and who knows what else.
+1 for Ubuntu 14.04 with Chrome 36
This is changing too, with the move to cartocss. Follow the github repo perhaps!
I used to scrape a bunch of webcomics to turn them into RSS feeds. I still have one or two running, actually.
Any idea why I can't install it on my Nexus 5?
Looks really nice, guys-- congrats! Would love to see some more product screens versus "request a demo" or "try it for free" (both are time-consuming ways to learn how a product looks/works).
That is a fairly surprising opportunity. I have experience monitoring/scraping thousands government websites for a different purpose. Considering some government sites have a round trip of well over 5 seconds, seems like it'd be a fun challenge to parse, format, and deliver it that fast.

What types of data formatting are you talking about here? Would it require a unique template for each individual site?
The demands a definition of "forces".

The truth is that modern commercial media are very consciously designed to be highly addictive, including TV, Facebook, many websites, and other media.

Source: I've worked in the field for much of the past decade, and could cite media executives talking about creating addictive and compelling content, dominating "all screens" -- phone, tablet, laptop, PC, and small and large-screen TVs.

It's also very difficult to find places in public where you're not assaulted by TVs, including increasingly in bars and cafes.

The salient characteristic of peer pressure is that it's not easily ignored -- that's precisely why peer pressure and social norms are sought by marketers, propagandists, and others who seek to influence group behaviors.

See: http://www.wired.com/2014/08/i-liked-everything-i-saw-on-fac...

Or discussions I've had on "the dopamine meme", TED, and weaponized viral clickbait:

http://www.reddit.com/r/dredmorbius/comments/240xss/the_dopa...

http://gigaom.com/2012/09/02/when-did-addiction-become-a-goo...

http://www.truth-out.org/opinion/item/17021-society-of-addic...

http://www.reddit.com/r/dredmorbius/comments/1te3hz/we_need_...

http://www.reddit.com/r/dredmorbius/comments/23twec/weaponiz...

And the Medium is the Message: how incentives shape what we're offered

http://www.reddit.com/r/dredmorbius/comments/278e2o/the_medi...
You could wash one lung at a time. Or use extracorporeal membrane oxygenation.

Water in an of itself won't hurt lungs as long as the patient gets enough oxygen.
Yeah, very much agreed. I have to scrape schedules together to populate games for my ticket site (http://www.boxrowseat.com).

The popular sites are wising up to scraping of information, so it's really an exercise in futility. But when it works, it's super rewarding.
Woodcut Data Visualization
He has a chance, but I think the US would know after a few days that he has left the country, and would just raid all boats on plausible routes. The US is not all-powerful, but still pretty powerful and white-hot with rage, and willing to risk diplomatic incidents over this.

Submarines would be too expensive. He's not worth that much to Russia, Argentina or any other country.
Online personal banking
Yes it can be a good guide, but don't forget economics guided us to network address translation instead of ipv6. Maybe it just isn't done guiding.
> To be fair, most things are dangerous when burned.

That's not at all true. Some things are more, some things are less.

Incompletely burned plastic? More. Full burned? Less. Gasoline? Less. Mercury? More. Arsenic? Less.
I'd be more worried about the tank of hydrazine in an F-16.
Seven Principles of Learning Better From Cognitive Science
Understanding substitutes
Nice. I always thought Rypple was a great idea but a) early and b) was never sure if their execution would work. They were acquired by Salesforce and rebranded as Work.com in 2013.

I hope the Impraise team can make it work - waiting a year for performance feedback is a stunningly stupid notion.
L1 is almost identical to H1b except for two main things, L1 grants your spouse an L2 visa which make him\her eligible to work in US unlike H1b\H4 nightmare, while on H1b you have more freedom applying for other jobs (if the companies are willing to transfer your visa) without going through the lottery again, unlike L1 which you get stuck with the same employer unless another company is willing to go through the lottery hassle with you.
in short, Pick your poison.
Previous: https://news.ycombinator.com/item?id=8155861
hum, interesting.

Kind of minority report here.

Machine learning on depression syndrome ?

I'am a hacker and not truly white, are you not afraid that people used this to take advantage of depressed people ?

A man can use this to spot depressed women , some are manipulator born people.

I find the project interesting but it have a lot of ethic problems. I think that it will be more a tools for the people in wealth.

maybe it could be more interesting to have the global mood of people than the mood of one people.
I scrape about 60 blogs and news sites that deal with a niche topic and examine all the hyperlinks. If more than one of them links to the same page, I assume that it's a page that's generating some buzz, so I send it out in an email. It's proved to be a generally reliable assumption.
I never like anything except my friends, anywhere. Of course Facebook gleans stuff from that as well. I would pay to have a site where all you could do is interact with friends posts/photos/movies with zero commercial features but we all know FB would kill such a thing in its infancy.
The other dude in the car
There is one other option for forest paths - Strava have made available their GPS heatmap for editing OpenStreetMap:

http://labs.strava.com/heatmap/

http://strava.github.io/iD/

So, depending on where you live, you may find local trails are on there (although you've still got to go in person to confirm they do actually exist, and to know exactly what they are - mountain bike trails, or narrow footpaths, or wide tracks for horses and 4x4s). In theory, it means you can shift effort away from sorting out gpx tracks, and more into getting the contextual information correct.
You're the one who brought up "allow", so I can't say what you mean by it. What I mean is that I'm disinclined to let your dickishness go by unchallenged.

Your "OMG hypocrisy" routine is cheap and lazy rhetoric. I'm not saying a negative tone is universally bad. I'm saying that replying dickishly to a positive novice contribution harms the dialog by reducing future novice contributions. Whereas I think negative tone in reply to a dickish comment can help the dialog by reducing future frequency of dickish replies.

If you don't like being irritated by people calling you out to being a dick to noobs, maybe you could try stopping being a dick to noobs. Then we can all be less irritated.
My parents do and they are perhaps the least technical users on the planet. I didn't even tell them about it!
Try https://www.comic-rocket.com/ and see if it has all the comics you want to read. It supports either RSS, a web-based reader, or an Android app.
And google does that better? Google pretty much only shows large forrests in my area, everything else is often just grey, no matter if small wood area, grass or farmland. Maybe it's in the US because of bad imports, but in Europe OSM is usually is pretty accurate.

> lastly, oh you want to look at other countries? how about we use their language instead. ugg

Well, that's what everybody outside the US or non English speaker has to deal with all the time. OpenStreetMap simply is a global project, while google still is very US centric.
All kinds of awesome stuff in this release, thanks Ghost team!

Still sad that apps didn't make it into 0.5 as planned, but the API and the new admin screen make up for that.
Here's a video that describes the app: https://www.youtube.com/watch?v=1lrp7ctc5FM

(Had a hard time finding details on the website)
Tilt: Easily Collect ESPN Fantasy Football League Dues
Or you could roll your own MMU for the 68020, like Sun and Apollo did.
> Most of the people ignoring the LL arms race target RV opportunities on the 30s+ range. This involves taking a pretty big step back from microsctructure/toxicity models.

Would you mind explaining more about this?
This indeed looks very cool, thank you and your sibling comment very much!
This limitation is not "weird or arbitrary" but based on the technical capabilities of Android phones in these days.

And those limits were later changed and will surely be changed in the future as smartphones get more powerful.

All kinds of limitations are in place both on Android and iOS, particularly when it's about the amount of memory an application can access or the amount of background processing an app can make.

If you don't have these limitations, then apps could affect not only the functioning of other apps, by grabbing and keeping all available memory, but also that of the phone itself, by draining the battery in 1 hour.

That's not something that the average smartphone user would expect or desire his phone to do.

Edit: I'd also like to add, that almost anything a developer creates is usually created with some limit, or at least some expectancy of costs in mind.

For example I couldn't imagine that someone creates a memory cache, where the limit = infinite without a really good reason. Or an API where the request limit/sec for any client = infinite.

You usually develop something in a way, that reduces the cost of some computation as far as possible, and when it still causes problem - introduce hard limits. Especially when it's a whole ecosystem where many companies/developers participate.
An auction based on the salary of the employee. i.e. the money doesn't go to the government, it goes to the person being hired. This also removes the criticism that the H1B is being used to bring in "cheap labour".
You’re 16. You’re a Pedophile. You Don’t Want to Hurt Anyone. What Do You Do?
A Thorough History of a Lack of Peace
Exactly. And I'd add: should have he expected it to fail where he gets silently blocked but everything else works as normal?

One of the advantages of using something like Twitter is that if you're a regular user, you'll have a good idea of when it stops working altogether or is experiencing significant reliability problems. That there's a failure mode where they silently break your app but nothing else sure isn't mentioned in the TOS.
Correlated with a second or third tracking method such as cookies sent to analytics, ad and fb sites, would it be super sticky?
When I was in the Hamptons last year, getting a car was MAD expensive. This was before Uber had people come in, so they partnered with some car services so even Uber was like $150 minimum for a round-trip to go 2 miles. I'm not actually joking about that.

Even with a cut-rate car service, to go from our hotel to an event I was attending and back (and it was maybe 4 miles in distance, MAYBE), it was close to $100. And that was because we found a cheap car service.

I totally see Uber drivers who have high ratings and $15 minimums being able to clean UP over the summer.

The only thing is I wonder about the impact on the existing drivers. Part of the reason the prices are so nuts is that the town basically shuts down the other nine months of the year, so those drivers charge an arm and a leg because after that, that part of the island is just freaking dead.

Screw pissing off London cabbies, I'd seriously be scared of messing with Hamptons drivers...