skysight-inc/HackerNews huggingface dataset:
Hey, article author here.

I've done extensive scraping in both Python and Ruby, as I wrote most of the scraping / crawling code at http://serpiq.com, so I can chip in.

Overall, I prefer Python. That is pretty much solely down to the requests library though, it makes everything so simple and quick. I haven't covered it in the article yet, but you can extend the Response() class easily, so you can for example add methods like images(), links(nofollow=True), etc. Overall, I just think the requests library is much more polished than anything available in Ruby.

grequests (Python) means I can make things concurrent in a matter of minutes. However in Ruby the only capable library supporting concurrent HTTP requests that I liked was Typhoeus. It just wasn't to the same standard though, and I ran across certain issues when using proxies etc.

As far as the HTML parsing goes, I don't really have any preference. Nokogiri and lxml are both equally capable.

I think they're both perfectly capable languages though, stick with what you prefer. I've been experimenting with Go lately.
"The International Narcotics Control Board"[1] is 'calling us out' as you say, but I don't think it's quite fair to call them "the UN"[2].

[1] https://www.incb.org/incb/en/about.html
[2] http://www.ihra.net/files/2012/04/05/INCB_Briefing.pdf
Scrapy mentioned here is really easy, or as the article mentions a combination of requests, lxml and something like phantomjs if there is javascript involved makes scraping sites in python a nice experience.

I have briefly made something with Scrapy to scrape my university's websites to notify me when we get new exam results [1], and that was okay. I might be slightly abusing scrapy but it was an okay experience.

Previously I have used 'scrubyt' for Ruby to scrape things, but their homepage seems to lead to a skin-related website now. What tools/libraries do you use for scraping stuff with Ruby these days? I remember Mechanize and Nokogiri was good/decent, but it's been more than a few years since I last used Ruby.

[1] https://github.com/flexd/studweb (description in Norwegian but it's not important)
I don't think they need to get one; Take a look at the CAs are in your browser. Do you implicitly trust all of those organizations and governments? It doesn't matter if you do as your browser already does.
Agree. Scrapy is great for this sort of thing. Takes care of little details you don't want to be worrying about.

I coupled scrapy with Stem to control Tor on the odd chance your IP gets blocked. Works great.
Your wife's parents might not have known. There was an article no too long ago describing how kids in the midwest are at a disadvantage because kids never learn to perceive elite schools as substantially better than most state schools.

My opinion is that the real value of elite prep schools is that they teach (indoctrinate?) you elite values through the culture, your peers, actions of your peers etc. They tell you what to do, instill in you a belief that you are "worthy" and capable and give you the tools on how to do it.
Jcabi-http, fluent Java HTTP client
Is it time to rethink the 40-hour week?
Ingenious. thanks for the link.
eBooks are mostly about stripping out people's "design" stuff and replacing it with some fairly standard components that actually work, and adding in some various bits and pieces in XML, a hyperlinked table of contents, and stuff like that.
I know, that's why I said I can't figure out why places would be willing to pay me the equivalent rate of $100k as a subcontractor, but only $50k as an employee. It's very obviously better for me as a subcontractor.
For anyone who doesn't know what BGA stands for, it is Ball Grid Array.

http://en.wikipedia.org/wiki/Ball_grid_array
It is pretty easy. If the editor works in word then you copy the tex and past it into word. Then tell the editor to not mess with the funny symbols and images etc. Use track changes and reject any changes that would mess up the tex.

Then just compile the tex to enjoy a beautiful laid out and professionally edited document.
> I don't see a reason that docker needs to support multiple paths to a handful of syscalls, especially at this fairly-immature point in its evolution.

Adoption.

The priorities of Docker 1.0 are production quality and first class support of all major operating systems. Today, that means first-class support of lxc (because lots of Ubuntu users depend on specific lxc configurations) and first-class support of either systemd or libvirt (because lots of Red Hat users depend on specific systemd or libvirt configurations).

From the strict point of view of surface area and reliability, you're right, it would be better if we exposed a tiny core with no possible customization. But then nobody would use Docker in the first place and there would be no Docker ecosystem, making its quality irrelevant.

The way we resolve this is by 1) exposing lots of APIs to allow better customization, while 2) shrinking the core and making it more reliable. Over time this allows us to converge towards higher quality, without sacrificing adoption. It's a delicate balance, but a necessary one and ultimately I think everybody is better off.
No, Edward Snowden wasn't behind 7 proxies at SXSW
There was a recent episode of House of Cards where the protagonists were exposed in the media over something. They came up with a false cover story, but they also deliberately bungled their response (asked another party to deny the claims completely, then somewhat contracted them in their own statement). The reasoning was that people would smell a cover up if everything was too neat.

I wonder if Dorian Satoshi used the same tactic. Once a journalist discovers him, he tacitly admits to being the Bitcoiner, then in his next public statement retracts this and says it was a misunderstanding. Now everyone's going about saying the real Satoshi would never have responded to the journalist like that.

Probably not but arguably it would be the best way to handle the situation if you wanted the story to blow over.
Last I checked non-Singapore citizens and Permanent Residents need an Employment Pass (EP) to work as a skilled worker in Singapore and such a pass must be applied for by the employer. So under that scenario you need a visa and sponsorship by an employer. There is a new system called PEP which does not require sponsorship by the employer, but has additional requirements in terms of base salary and restrictions on employment, but again it still counts as requiring a visa.

So having "just an American passport" does not allow you work in a country like Singapore without applying for a visa. While the Singapore system might be more flexible than other such as the US, it certainly is not "free movement of labor".
I did not but I was only basing my findings off of the production of beer/wine, not the agriculture side.
http://upload.wikimedia.org/wikipedia/commons/thumb/f/fb/Cat...
Xembly.org, Assembly for XML manipulations
Ask HN: My 19 year old friend has cancer. How can I help him? 
Hello HN,

Very unfortunately my friend and neighbour who is just 19 years old has colorectal cancer. He was undergoing radiation therapy but his chance of survival has dropped from 90% to 35% after his tumors got infected. His doctors (in the Netherlands) say that they will try more radiation and chemotherapy but with the 35% chance of survival that his doctors told him recently he seems to have lost some hope. I was wondering if there are any options that we are not looking at.

He is a very bright kid who has the potential to be a great hacker :) HN, tell me how we can hack his chances of survival.
iOS 7.1 Real-time Design Reactions
It's not a huge improvement to me, that's why I don't use it.
This hasn't been the case since Chrome implemented certificate pinning in 2011.
This is just a guess (and so a completely worthless comment), but I'd expect that http pipelining has something to do with it.
This was a great question, plus a great answer from @bane. New technologies and new languages pop up overnight, it's overwhelming to keep up - so the next best thing for us to do was to 'hire our weaknesses' = management.
Have you tried making any graff? If not I highly suggest it!

I made a batch and it turned out to be arguably my best homebrew yet. Currently making a blueberry graff now and mead kit should be delivered today.

Any recipes you have shared on brewtoad/elsewhere of your cider?

[1] - http://www.homebrewtalk.com/f81/graff-malty-slightly-hopped-...

[2] - http://mnbeer.com/2012/09/21/brewing-tv-68-how-to-make-graf/
Google faces up to $5-billion fine in India; says cooperating in probe
... and herein lies the problem.

I know what I'm getting when I buy an iPhone. Navigating Android smartphones is a minefield.
Engineering is a suckers game. Go back into consulting, apply for an MBA from an elite school and come back as a manager at any of the companies

And if your accent is a problem then do whatever it takes: go to a speech therapist. Actors do it all the time and so do CEOs (Vikram Pandit).
Coolio. I'd use PyQuery instead of xpath.
Hey, thanks for the feedback!

I love your pgp-messenger idea - you should do it!

And you're right - the pretty front end is a big component. Two things, though - 1) we're working on open sourcing the profile crawler 2) it shouldn't be that difficult to roll your own pretty front end.
All of those UNIXen are going the way of the buffalo as the world moves to the web. I haven't seen HP-UX or AIX system in a decade or more. Admittedly, I haven't done contract work in about eight years, which is where I previously saw those beasts (and others; for a short time I specialized in migrating businesses from SCO UNIX to Linux and made a small fortune doing it), but our software gets installed in all sorts of places and we used to hear about those systems pretty regularly...but the mentions of everything other than Solaris on our mailing lists and forums and IRC has slowed to practically nothing, and Solaris is in a slow spiral downward.

I use Perl daily. Love it. But, the argument that it's available while others are not isn't really valid anymore except in a few niches. Linux is pervasive, and Python and Ruby are pervasive on Linux systems. In fact, Python is more likely to be available on the two biggest distros because many of the administrative tools provided by the distro are written in Python. RHEL/CentOS/Fedora effectively have standardized on Python as the language of choice for admin tools.
Kim Jong-Un Wins Re-election
What is a TSC and how do I get one?
Cool sites - I like this idea of sharing concepts and getting feedbacks. One other larger one is producthunt.co. I'd be curious to see a full list if you happen to do some more competitive research!
Never used it, but looks useful. I'll add a note regarding it. Thanks.
Have you ever experienced the Google interview process for SRE?

After I'd spent a couple of weeks going through phone screens which alternated between "Rah rah! Google! We solve interesting problems! We're changing the world! Work on whatever you want!" and "How do you sort a million phone numbers using the least amount of memory possible?" and that silly ping pong ball question, the recruiters started asking things like "How do you feel about getting handed code without documentation and being responsible for deploying it?" and "Is it a problem for you to be maintaining code but not writing it?"

Then again, I'd accuse myself of making this up if I hadn't experienced it more than once.
Going to add a section on PyQuery, thanks!
At Hired, we found that candidates who are on H1B visas get 1/3rd fewer job offers. So yes, technically transfers are easy, but you definitely reduce the number of potential employers.
Huh? Here's the exact scenario I responded to:

If a guy on the street asks you where you just came from, and you lie to him ("Church. I don't visit strip clubs"), that's your right. Unless he was a plainclothes LEO.

The statute he cited does not make that scenario criminal.
I've used Scrapy previously, and agree it's a good tool in the right situation.

It could just be me, but I prefer more control, and using the requests library gave me that. I ran into some obscure problems when wanting to use / change multiple proxies with Scrapy.

I think for a long running spider, then maybe scrapy is worth looking into. For simple one-off scripts / things not run very often, I'd much prefer to write something custom and not have to learn about Pipelines, Filters, etc.
The best thing about this game is that you can press the arrow keys randomly for about five minutes and anyone walking past thinks you're incredibly good at the game. Just make sure you lose after they walk past.
With my 2.1 GB of unused space I had no issue installing this update. It's a delta update so the size will vary. Mine requested about 200 some MB if I'm not mistaken.
Very cool. Would like to see proxy support on the xmlrpclib bits; http_proxy=... doesn't work at present and I don't see command line flags. (Not a hard fix, but I can't offer a patch right now.)
Well I know it's in Sendai region.

But I forgot how much Japan bends up at Tokyo. Sendai isn't North East like I imagined. It's more directly North.

Also, although Narita feels like an exhausting distance from Tokyo after you get off a plane, Sendai is of course much much farther away.
So true. Add to this that brewing to the tastes of your target market will be much less interesting than trying out that new farmhouse yeast you located at 28*C in the garage.

Who wants to brew pilsner and pale 7 days a week?
It also makes a big difference to use a headset or headphones so the other person's audio is directed into your ears, not played out loud.
Optional if you want good battery life, sure.
There are many sites like this, here's a list I started: http://appworthy.co/alternatives
I am trying to decide whether or not to pursue it commercially. Makes a fun hobby though and a great way to teach your kids about microbiology ;-)
NgTailor scaffolds out new Angularjs applications
A little bit too similar to 'threes' on the app store.. I like it, but again, it's too much of a ripoff.
Consoles Are A Dying Market
One problem with using Windows guests on Vagrant is how big they are. You can be looking at 20GB for a stripped down disk with just the OS. This can easily become prohibitive if you want to use more than a few VMs in Vagrant (not to mention the time copying all this redundant data).

Having read about differencing disks, they seem like the perfect solution to this problem, by allowing VMs to share the same parent disk, without needing to copy it (as each VM only needs to keep track of differences to the base disk). From what I've read of differencing disks, this seems like a no-brainer for Vagrant (but unfortunately my experience extends no further than what I've read).

Is there a reason Vagrant doesn't use this approach? Any plans to put this on the roadmap? It seems like this feature is supported by both VirtualBox and Hyper-v.

(This page describes differencing disks, and the first diagram under "Using multiple differencing disks with one parent disk" accurately depicts the sort of thing I have in mind here
http://technet.microsoft.com/en-us/library/cc720381(v=WS.10)... )
Starterpack.js – Simple Node.js Authentication
yes, it should prompt you to login... must be bugged.
I do not find Macs to be very "usable", and previous-gen ThinkPads outranked them in everything but the shiny-showing-off-at-starbucks metric. However, in the last generations, Lenovo has nerfed the ThinkPads (new ones don't even have a middle mouse area - no more buttons, either).
They've also even put the logo upside down so that you can show it off at cafes.
Did I just read a "because it's popular, it's better" argument? Combined with a strawman ("the poster is arguing everyone but himself and other people who think Perl ~ Ruby/Python is an idiot")?

People don't need to be particularly stupid to be driven by trends that aren't entirely founded on reason. And preferences can be both subjective and valid.

I don't have a dog in this particular fight (while I worked in Perl for at least five years, I stopped over 10 years ago and moved on), but I think it's pretty much a reasonable claim that Ruby and Python and Perl (and maybe even PHP) are all more or less in the same class in terms of capability. Python and Ruby may have some gains at certain margins (perhaps Python in particular with the focus on clarity in both the language and community), loses in others.
I have to agree that Scrapy is oriented more towards long-running spiders. For a one-off script, I'd probably do the same thing as you.
Mt. Gox files for US bankruptcy to halt class-action suit
In my experience at Stanford, this usually wasn't the case. Most people started looking for internships around February-April. Obviously not the case everywhere... but it's the case at arguably one of the closest talent pools to YC.
There's an entire other 40 hours between 6 and 10, plus the weekend.

Anyone of any skill of any appreciable amount working a 9-5 has plenty of off hours to build their thing.

My question is: who are the people who could be successful with Friday off that can't be successfull with Saturday, Sunday, and after work? My bet is you're talking about serving a null set.
Codementor.io
International "norms" are only that; norms.

It is the absolute height of arrogance to assume that other nations will follow the West's lead because we're "right" or "morally superior" or any other bullshit people want to make up.

The implication is that if NSA can engage in mass surveillance, that other nations might think that's acceptable. But the implication is already proven false; NSA wasn't even the first, and the list of countries that do engage in technical surveillance is not just Russia and China.

In the end a nation will or will not use network surveillance because it does or does not help them meet their policy and strategic goals.

After all look at the Russian response to very strongly-held "norms" about the sovereignty of borders in Ukraine. Whatever rhetoric you wish to use to defend their actions, the fact is that they're stepping on one of the few clearly-drawn lines in international law. What's more, the reason that they're doing so isn't because there isn't a "norm" against this behavior, it's because they judged there will be no appreciable penalty for their action. Putin isn't "going rogue" and he's definitely not stupid.

In case you're wondering, that is where the world will be led; power politics instead of wishy-washy Western values, because power politics is what works.
Differencing disks depend on some things in the core changing. These changes are under way already in branches of Vagrant and Vagrant 1.6 has a very good chance of properly using linked clones wherever possible.
You can view examples by clicking the links at the bottom of the page like here: http://www.gopitch.me/ideas/all
Here's a tip: never bet on something proprietary.
Wait doesn't this break the microwave? If not, I'm gonna show my kids something really neat.
Using a Bitcoin ATM is pretty expensive
Jugos para bajar de peso
Cuando te invitan a la playa o a algún evento y tienes kilos de más. Entonces el pensar solamente en esos kilos te daña inconscientemente esos planes que tienes. En este artículo te enseñaré un dieta eficaz que te ayudará a perder peso en una semana.
Yes! At the very least put up a BTC or Dogecoin address we can send donations to!
It depends a lot. This is opinion, but I would say that there are three big categories of software development skills. Of course most jobs are a mixture of these.

1. Web development (transport-level and higher networking, HTML/JS/CSS, PHP/Perl/Python/Ruby, using databases, etc.)

2. Enterprise software design (domain modeling, OO programming, system architecture, scaling, security)

3. High performance (algorithms, advanced data structures, C/C++, memory management, DB internals, internet-layer and below networking, cryptography)

It sounds like you want to get into 3. A good way to start is algos/data structures. Almost all the fields the parent listed benefit from a very strong foundation in computer science: discrete math, linear algebra, algorithms, advanced data structures, etc.

Data analysis with python/R is more for scientists and mathematicians, not software developers. If you are being paid to analyze data as a programmer, usually you are analyzing very large data sets and you will be using a much more performant language or possibly even a highly parallelizable paradigm like MapReduce.
Seems to me that if I needed to take a pay-cut then I'd be better prepared if my costs of living were lower than if I had locked myself into the expensive city house.

"Just pack up and move somewhere cheaper" is easy advice to give, but it doesn't apply to every situation. Not everyone has that luxury. By the time you reach the mid-point of your career, you may have one or more of a partner, a partner with a professional career, personal obligations in your area, a mortgage, a mortgage you can't get out of without losing a lot of money/credit rating, aging parents, children, ailing family members who live with you, medical conditions which limit where you can live, ex partner with children not willing to move, et cetera. Alternately you may already live very frugally.

When I was fresh out of college, I couldn't have imagined any of those circumstances applying to me or anyone I know. Now I've seen them happen to many of my peers and colleagues.
Programmer Step #2 - write an automated solver :)
My father-in-law, who used to be the brewmaster of one of the biggest breweries in the world, has a PhD in Chemical Engineering. I'm glad he did not end up as a developer, although perhaps my wife would understand me more!
So glad you said that! It's very challenging to find beer that's not overhopped but they do exist. Coming from Germany I'm still looking for some good lagers that aren't insanely hoppy.
I don't mean to alarm you, but I think http://medium.com stole your design.
Thanks for your feedback and for the link. Nice website!
Thirteen year old (youngest ever) builds nuclear fusion reactor.
considering that 3 is 50% more than 2, I'd say the negative is that it ISN'T a "huge" improvement.

(only a little kidding).
MelbourneIT stores domain passwords in cleartext
I ported a python2 app with py2to3 a couple months ago and it wasn't completely painless.

The worst problem I ran into was that it didn't know which strings were supposed to be bytes or unicode. And the program would throw a runtime error when it tried to do something like use a byte regex on a unicode string.
Python is the language designed by a guy that did not understand the difference between tail-call optimization (a general, widely useful optimization) and something he made up as a strawman, called "tail-recursion elimination."

PHP began life as the antithesis of "goto considered harmful." The idea to embed a language directly into a mix of HTML/CSS/JS was just profoundly awful. Yet we still bash "goto" as the spaghetti monster. I guess we just can't detect irony today.

To your #1 point, I'm going to have to say there are a large number of powerful, well-funded, well-marketed idiots in this industry. If languages really mattered, we would all be using Scheme, ML, or Haskell or something. People are just now discovering Erlang, thanks to WhatsApp and Facebook. Despite Erlang being a year older than Perl. But Perl didn't make recent headlines for a $19bn sale, so...
>encoded to CD, then FLAC'ed

You realise that's a no-op, right?

I also give you this marvellous study on "golden ears" and "high-resolution audio" versus science: "Audibility of a CD-Standard A/D/A Loop Inserted Into High-Resolution Audio Playback."

http://mixonline.com/recording/mixing/audio_emperors_new_sam...
The particular teleco I'm thinking of ... they'd have way fewer problems if they got an independent person running their technical project management rather than an endless sequence of junior grunts