skysight-inc/HackerNews huggingface dataset:
You're a good writer. And I can relate to what you say: if we raise our children to be so practically-minded, where's the space for them to be imaginative and dream? If they don't believe magical things can happen, they're certainly not going to make them happen.

Are you going to tell Sophie or will you let her figure it out on her own?
And this is why we can't have nice things.

Web scraping, as fun as it is (and btw, this title again abuses "Fun and Profit"), is not a practice we should encourage. Yes, it's the sort of dirty practice many people do, at one point or another, but it shouldn't be glorified.
Yes, I have so for been working on getting the Base API right. Will be adding background process, stderr redirect etc. Thanks for the input.
40 years from now, people will have the same sentiment towards their bite-sized terahertz quantum computers costing a mere $1000 ($50 our time)
Seeing these prices reminds me how the C64 became so popular during this period - it was just a computer/keyboard which plugged into your TV - similar capabilities at a much lower price.
Very valid point. I am working on the powershell equivalent for this, and I would be happy to support only powershell commands and standard getopt commands.
The issue with web scraping is that it relies on the scraper to keep up with changes made to the site.

If a site owner changes the layout or implements a new feature, the programs depending on the scraper immediately fail. This is much less likely to happen when working with official APIs.
It's the only comedy which is slowly transforming into a documentary.
The project will never really be relevant. If you have a serious need to run OS X probably already have OSX and a Mac.

If you need to simultaneously run Linux you install it in a VM.

This project like WINE for the most part will always be a "look what I can do" type project. It's never going to be a rosetta type system designed to bridge ecosystems as the VirtualBox / VMWare bridge already works quite well.
One reason why consoles from MS and Sony are bad is their obsessive DRM and the notion of turning a general purpose computer into a locked up appliance.

But there is another architectural reason which is generic - consolized games degrade the quality of gaming interfaces by making them overly minimalistic (since consoles don't use keyboards), and even developers who release crossplatform games for PC and consoles often save time by not distinguishing these interfaces, and when you run the game on PC it feels crippled interface wise. So I'd say for gaming in general, consoles are a big disservice.

We'll have to see how this will develop.
If you like this essay, check out http://lesswrong.com/

They seem focused on what PG is proposing.
If this sort of a priori judgement occurs, it occurs because Hawking is a famously intelligent and thoughtful person.

-- The famous "reverse" Ad-Hominem attack? WAT

TLDR: Famously intelligent and thoughful people are frequently full of shit, evil, or amoral. Same rules apply. [1]

_________

[1] Its worth noting [a] ad-hominem is not a per-se fallacy; And [b] its useful to questions motives. But since many serial killers, terrorists, and supporters of the hollocaust could be considered "famously intelligent and thoughtful people", it goes without saying that arguments should be evaluated on the merits.
It's actually not that bad. This seems more like people reacting badly to change than anything else.
I was thinking the same. I wanted to support cmd.git(git_dir : blah).branch(a: true) kind of scenarios however. Think I would just not go with it if it means removing the extra (). Thanks
That's more satire. Parody can take the ball and run with it as far as it chooses; satire can only be relevant if its characterization has a degree of verisimilitude.
Why? I would gladly encourage web scraping.
In most cases, the site doesn't have an api... so we scrape and take the risk that the structure will change. One thing that helps is using tools which give you jquery-like selectors because they give a lot of freedom and are very easy to write/update.
I wasn't saying that it was impossible to run them. I'm just commenting on an issue that was immediately apparent to me.

If a program calls '/usr/bin/nc' and expect BSD options, it's not going to get them. "Replace GNU coreutils with BSD coreutils" is an option, but not one that everyone would be ready to jump at, because now things running on your Linux system that expect GNU options won't be getting them.
Read my comments section, if you are intelligent hacker will get your answer.
Checking for null values from malloc does not create serious overhead, it creates minor overhead for your branch predictor, unless for some reason malloc frequently returns null.

A single context switch back to the kernel generates far more load than the null checks could ever hope to accomplish.
Amazon Spyware results: skin care, shoes, day spa
vim+tmux+vimux+turbux screencast
My iPhone 5 is 1 in 214,000; my Galaxy S3 was unique.

Since the iPhone 5 makes up more than one in 200,000 of the Internet's traffic, even that "relatively low" uniqueness is worrisome.
What's wrong with this trend? High quality games require good performance. Find other alternatives to C++ which can offer the same performance, and you'll see more games developed using them (given that someone will write OpenGL interfaces for those languages).
Recall joost, ms web TV, and a few others.
Fair.

In addition, the target audience of the show are unlikely to know very much about the valley. I would expect most people believe the valley is "kids from Harvard who think they're hot shit, trying to get rich doing something useless".
Adventures in Depression (Cartoon)
Extract received positive reviews from 63% of reviewers on RottenTomatoes. I wouldn't consider that "critically panned."
Or an alternative API that uses the scraped data from Twitter to make requests... but that might be getting a bit ambitious (and legally dodgy)
If I recall correctly, as an independent company Feedburner charged a modest fee for CNAME feeds.

Once Google acquired Feedburner, Google made the feature free.

I'm pretty sure the consumer got a nice win here.
Halo 3, which I played a bit in college, certainly used one member as the host.
I agree that the valley isn't Seattle; good on Facebook for the warning. I used to park my multi-hundred dollar car out by the sports cars at what is now Facebook's campus; drag races out the main exit, over the railroad tracks, and on through EPA were not unheard of.

Here's my view of the Valley from 1997:
http://blog.ryjones.org/2006/10/21/welcome-to-the-bay-area/
Nice. A direct link to the 2011 10K with parent's figures is at:
http://files.shareholder.com/downloads/CMCSA/2210267075x0x56...
Meanwhile my alma mater recently changed its logo as well...

http://www.stanforddaily.com/2012/11/15/1073130/
I was actually thinking more of using ... and I shit you not... the really old school Aol model (you get 40 hours a month! Of the internet!) for more of the cheaper versions.

Everyone gets x hours a month, the system wide number of hours watched on them gets paid out to the channel providers as our users would have fairly watched that amount..
Hmm,

I'm not actually in favor of scraping. But I think it is possibility that needs to be considered on both ends. If a site has valuable info and doesn't provide a decent API, it naturally is going to encourage scraping.

And isn't a search engine a kind of scraper?
Yeah, I think skipping the extra () is worth giving that up. Plus, you could do this in most cases: cmd.git.branch(a: true, git_dir: blah)
Is this logo specific to UC-Berkeley or is it for the university system as a whole overall? Based on the video it looks like each school could customize the logo to use their colors.
Guess I'm not sure how/why this hit HN's front page? Scapy has been around for years. Nothing new to see here and, well, I'm not sure Scapy has a very vibrant development anymore... o_O
Aha. We're serving lots and lots of very large image file and PDFs. Thanks much.
When I first heard about the "new logo", as a UC student, I was mad, as were many of my colleagues. The reason for this is that UC has a storied, dignified seal, and replacing it with a watery mod-looking thing like this new logo would be a real mistake. But deeper investigation reveals that they are looking for a logotype mostly for marketing materials and that sort of thing, and are not planning to eliminate the seal from diplomas, official correspondences, or anything like that. I don't think it's a huge deal, but as a California taxpayer and fee-paying UC student, I am somewhat disappointed that it seems our money has been spent on such poor design.
when i need to scrap a webpage, i use phpQuery (http://code.google.com/p/phpquery/), it's dead simple if you have experience with jQuery and i get all the benefits of a server-side programming language.
Wow, this totally reminded me of hanging out in RS at the mall while my parents shopped and getting tips from other geeks hanging out around the demo computers. This is how I learned MS-DOS Shell (and gorillas & nibbles) :D
Good to know. I'll look into it and add it as a suggestion to remove it on my Ubuntu tips/tricks website. Thanks!
Scraping made easy
I have built a "record" and "replay" solution for scraping web content from any page. Simply download the Chrome extension and "record" your workflow on any site and then upload your task. The output can be obtained anytime via an HTTP request. Wanted to know if anyone wants to testdrive the solution. Is this something that developers will be willing to pay?
I leave my phone at home when I commit crimes
This. It would be better for the author to understand the legal requirements and the legal history of cable TV. The technical side is only a small part of the equation here. He would be dealing with regulations at local, state, national, and international levels.

http://www.fcc.gov/guides/regulation-cable-tv-rates
http://www.fcc.gov/encyclopedia/evolution-cable-television#s...

So I guess the real answer to his question 'Ask HN: I want to build a cable company. How would I get started?' is here.

http://www.fcc.gov/
Domain "hackathon.co" on sale
Totally agree, scraping is great.

I can see site operators being against the practice though, as it (usually):
- generates no ad revenue
- often enables someone else to use data that you struggled to put together, allowing others to profit with no gain for you
- hit's edge cases that were never optimized for (as it does not follow real user access)
This should be stressed - sites like Facebook do exactly this. Constant changes mean constantly updating your scraper. When it comes to A/B testing? Your scraper needs to intelligent find the data, which might not always be in the same place.

Sidenote: I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping. I guess this would be a caching nightmare, though.
Great read!

In the past, I have successfully used HtmlUnit to fulfill my admittedly limited scraping needs.

It runs headless, but it has a virtual head designed to pretend it's a user visting a web application to be be tested for QA purposes. You just program it to go through the motions of a human visting a site to be tested (or scraped). E.g., click here, get some response. For each whatever in the response, click and aggregate the results in your output (to whatever granularity).

Alas, it's in Java. But, if you use JRuby, you can avoid most of the nastiness that implies. (You do need to know Java, but at least you don't have to write Java.)

Hartley, what is your recommended toolkit?

I note you mentioned the problem of dynamically generated content. You develop your plan of attack using the browser plus Chrome Inspector or Firebug. So far, so good. But what if you want to be headless? Then you need something that will generate a DOM as if presenting a real user interface but instead simply returns a reference to the DOM tree that you are free to scan and react to.
Not every site. There is data I would really love to access on Facebook without having to gain specific authorization from the user. It's odd that for most user profiles the most you can extract via the graph API (with no access token) is their name and sex. Whereas I can visit their profile page in the browser, see all sorts of info and latest updates (and not even be friends with them)

Tried scraping Facebook. They have IP blocks and the like.
As others have said, you likely don't want to build a cable company, and probably couldn't anyway.

Google for terms like "unbundling" and "a la carte" in relation to cable. You'll find that some cable operators are in favor of it, but enough cable operators and content providers oppose it to prevent it from happening.
I reread the trilogy pretty recently. Still a great read!
If http://Aereo.com wins their fight with the broadcasters they could become the next generation cable company, as they will be able to attract a fairly large sized audience.

By accumulating a nice user/audience base Aereo could then begin to add other channels like small to mid size broadcasters like Al Jazzera and possibly even AMC or a live netflix channel and more.

I am hoping they win this fight with broadcasters as I want to watch my local antenna channels via the net!

Also and recently I saw an article of Bit Torrent live streaming app, which could be another way to disrupt and destroy the cable business.

Update: I'm reading other entries and wonder if the OP actually wants to build a cable company the old fashion way, i.e. running wires and stuff? Seems old fashion to me, as opposed to creating the new cable industry using the net.
Playing devils advocate here, Lua does indeed have some warts that make it less pleasant to use in a heterogenous environment (which pretty much all large code-bases are). Not so much due to deficiencies in Lua but due to the impedance-mismatch versus other languages.

The most obvious issue that everyone stumbles across is the "counting from 1". It seems like a minor thing, but the context-switch remains a drag when you're dealing with complex data-structures in two languages and only one of them is Lua.

The impedance-mismatch becomes even more apparent when the table-abstraction meets serialization. The lack of distinction between an "array" and a "hash" is awesome when you're in a pure Lua-environment, but it becomes a real problem when you need to exchange data with languages that do depend on this distinction (e.g. if you feed Lua an empty "array" it will later serialize it back to an empty "hash").

The final issue that I can't resist mentioning here is not a language but a community/mindset one. Up to this day Lua doesn't have an established package manager akin to RubyGems, Pip, Maven, Leiningen etc. (Luarocks exists but is... well, I've yet to see someone actually using it)

This is a deadly sin in terms of mainstream adoption. It makes deployment a serious pain in the ass.

GoLang shows how a modern language is supposed to handle this (importing/bundling packages directly from urls). I keep hoping someone will add something similar to lua-core, but I'm sadly not very optimistic about it.

I think many of the driving people behind Lua just don't care about it becoming a mainstream language or not. They care about it shining as an embedded language (and it does!) - it's just a little bitter for those of us who would love to use it on a broader scope.
Is that confirmed? Last time I checked those were only faint rumors, and switching to a proprietary architecture is not gonna help their bottom line, unless they want to get rid of the PC / Laptop division.
I am the founder of Function Space. I was really not expecting Function Space to get featured on Hacker News so prematurely! We're still working on community and apps. You can know more about it here - http://blog.functionspace.org/news/2012/11/28/know-more-abou...
We had our plans for Show HN, but this came way sooner than expected! But now that it's here, I would like to tell that we would be posting a big update about Function Space community in about 10 days on our blog. We're already in talks with some academicians for video lectures, interviews and useful articles. For any queries, mail me at aditgupta@functionspace.org
So you're not so hot on the whole search engine thing?

The article does slide into the sketchy side (I've always wanted an excuse to do that client side javascript trick too) but I found it more interesting because of that, these aren't secrets. Maybe if I put my "won't somebody please think of the children" hat on I agree that glorifying using trojan code to potentially ddos someones server to get around rate limits desired by the owners of the server is bad. Adults and especially adult self described hackers should be able to read this without mock outrage, it's interesting and it's happening all the time.

You can't condemn web scraping though, that's the backbone of the services we all depend on for most internet related things. That's the whole point of structured markup and the world wide web itself.
adventure time is MATH
I think going forward, you're going to need to handle scheduling decisions and power management as a coupled problem.
I've done a ton of scraping (mostly legal: on behalf of end users of an app on sites they have legit access to). This article misses something that affects several sites: JavaScript driven content. Faking headers and even setting cookies doesn't get around this. This is of course is easy to get around, using something like phantom.js or Selenium. Selenium is great because unlike all the whiz bang scraping techniques, you're driving a real browser and your requests look real (if you make 10000 requests to index.php and never pull down a single image, you might look a bit suspicious). There's a bit more overhead, but micro instances on EC2 can easily run 2 or 3 Selenium sessions at the same time, and at 0.3 cents per hour for spot instances, you can have 200-300 browsers going for 30-50 cents/hour.
Why not real IPTV over the Internet? Probably wouldn't want to without multicast, but wouldn't this be feasible when IPv6 adoption starts reaching decent percentages?
If you have multiple (modern) physical processors, it
is also good to set up appropriate interrupt affinity for your NIC and to selectively turn off NUMA interleaving where appropriate.
I love HTML scraping.
But Javascript???...The juiciest data sets these days are increasingly in JS.
For the love of me i can't get around scraping JS :(

I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share??
I would be eternally grateful.
There is one thing that I could do differently that's enormously significant: freely undertake teleportation and mind uploading, given I live to be able to do it; and it's in turn what's driving me just now into Philosophy (I'm learning German even). Surprisingly for me, Graham nails that, but may not quite explore the possibilities ensuing, when he talks about sectioning one's brain in half, putting the other in a clone body, and trying to figure out which person you are now.

My working line of reason is that consciousness is, as he puts it, merely an abstract concept we get attached to and have the illusion of having. There is no consciousness, only the awareness of a single moment--a point in time. What we take for consciousness is nothing more than a succession of these single moments, much as a computer's processor clocks at so many Gigahertz. With each moment of awareness creating new memory and storing that memory back in the brain, the subsequent moments are able to access it.

You coud think that the current moment of awareness is all that ever really "exists", because for all the other previous ones you only have the memory, you have no concrete evidence that it ever existed. There is no past awareness, there is only a current awareness of what we have registered as the past. With that decoupled thinking, you can start to realize that any given moment of awareness that I have is as different (and as similar) from any given moment you have as it is different from a moment I had 5 minutes ago, or ever.

So who I "am" now is as entirely different from who I was just a second ago, in essence, as it would be different from a clone that could be made of me. There is nothing in my flesh ensuring I am a continuous consciousness. There is only the memory. The reason I am closer to my 5-minute younger self than I am to another human being is that the former and I share all the memory up to 5 minutes ago. We have a base commit to our memories in common. But a clone of me, made now, would be exactly as close to my 5-minute younger self.

The bottom line is that I should not think that being teleported should kill me, as that would imply that I'm dying after every moment of awareness, birthing a new person who happens to share all my memories. Indeed I may well be, and we have the illusion that we have a continuous life since we have all these vivid memories of what just happened a second ago, even if it's not us who lived through it.

And though I'm finding this train of thought increasingly alluring, I know nothing of how philosophy ever brushed it, or it may well have tackled it head on and I wouldn't know it.
If a monothreaded eix-update was run immediately after a polythreaded eix, there's a high chance that a lot of the data from /usr/portage was in the disk cache. I'd like to see the results of running the test back-to-back multiple times (eix, cached eix, monothreaded eix, cached monothreaded eix), which might eliminate the caching delta.
You need the watch the Akira Kurosawa movie "the hidden fortress". Basically, the parts of the first Star Wars that weren't borrowed from WW2 movies were lifted from this movie. Many story elements, C3-PO/R2 and parts of the musical score are very similar.
Oh, the grammar...it hurts...
The world does not need another wine for any platform. This is what VMs are for. Waste of time
it can, yet, headers such as checksums will have to be handled as well.
This is shockingly bad
I haven't looked at it in a while but Redmine and its fork Cool Project might be worth a look.
that's a shitty excuse. HTML as a "protocol" mixes presentation with content. An API should not only be presentation agnostic, but presentation free.
Thanks, fixed where we could, except the actual quote.
Fluentd v0.10.30 is released
>The issue with web scraping is that it relies on the scraper to keep up with changes made to the site.

The OP addresses that point. His contention is, there's a lot more pressure on the typical enterprise to keep their public-facing website in tip-top shape than there is to make sure whatever API they've defined is continuing to deliver results properly.

Of course, part of the art of (and fun of) scraping is to see if you can make your scraper robust against mere cosmetic changes to the site.
Up until 1998 or so, PCs regularly yielded 20%+ margins. I basically paid for college hocking PCs at retail 20 hours a week. Corporate sales guys made very nice livings selling PCs.
See my other comment :-)

I've been doing this on a site that is 100% Javascript-driven for over a year, very successfully.

It's really no different than hitting a static site with Selenium. Figuring out the proper XPath to use is often the biggest challenge: Chrome Developer tools help immensely. Also, you need to watch for delays in JS rendering, so put a lot of pauses in your scripts.

It's of course slow, so if you want to distribute it across several machines, use Selenium Grid or a queue system (SQS, Resque, etc). Setup Xvfb to run on headless Linux instances.
Not to mention that many of the channels are run by the same network. It's unlikely they would license one channel to you without all of their channels unless you have massive financial clout to effectively shut out their other/useless channels.
Do you have any more details or links to articles about this that I could read?