skysight-inc/HackerNews huggingface dataset:
> Almost everyone I know makes SOMETHING and they normally like doing so and are proud of their creations.

In 2004, relatively few of the people I was listening to voted for George W. Bush.
Actually no, I repeated it several times in read only after a cold boot start on the 2 sysreccd kernel and altkernel.
The limiting factor is CPU bound.
Note: I have a quite fast SSD at around 500MB/s so the results may differ on other machines.
Your definition does not include that a system programming language must be respectful of the dlopen() and dlsym() functions. In all practical terms, it has to be; otherwise, it is not a valid system programming language.
This is something my friend brings up all the time and every time I have to explain to him why cable companies operate the way they do.
Twitter is one of the worst offenders. Anytime I visit their site I have to make sure to open it in a new tab.
Indeed, it looks like a very hairy problem.
What would the 201X version of this be?
http://en.wikipedia.org/wiki/TRS-80_Model_100

It ran for 20 hours on 4AA batteries. It was instant on and daylight readable and had a full sized keyboard. Maybe something like a largish netbook or 11" Macbook Air with a lowish resolution transflective display, the ability to sleep for a month or run for a week in a low power/console only mode. What I'm describing isn't a mass market item, of course. It would be more for field research. (Just a few years back, I'd heard that some Model 100's were still being used in this capacity.)
Another issue not covered: file downloads. Let's say you have a process that creates a dynamic image, or logs in and downloads dynamic PDFs. Even Selenium can't handle this (the download dialog is an OS-level feature). At one point I was able to get Chrome to auto-download in Selenium, but had zero control over filename and where it was saving. I ended up using iMacros (the pay version) to drive this (using Windows instances: their Linux version is very immature comparably).
A similar module for node.js: https://github.com/mape/node-scraper
Whoa awesome...My dream lists as a 5 year old...oh the endless hours i would have wasted pouring over these catalogues...

But then it reminds me of how thankful I should be for the times we live in now :D
It actually looks pretty nice here: http://i.imgur.com/pgqun.png (frame from http://vimeo.com/53530934). I wonder why they added the weird gradient.
Social Media Design Cheat Sheet 
Stay updated on the dimensions & requirements to properly customize your social media accounts with our Social Media Design Cheat Sheet.
I think you should look at this blog post:

http://www.bunniestudios.com/blog/?p=2407

It's a tour of the Arduino PCB manufacturing facility, with QA systems, and PCB manufacturing equipment.

He alludes to sending the stack of PCBs to the SMT facility, although he doesn't have a tour of that one on his site.
Craigslist, anyone?
Headless: Xvfb on Linux. (Virtual Framebuffer, let's you run apps that require a GUI) You can use one of the many options that include Webkit (like phantom.js, the capybara-webkit gem, or Selenium if you want a real browser like Firefox to do the work)
I'm curious why you didn't mention the $4999 computer ($11110 after inflation), which is the very first price you see. I'd guess that the reduced cost of hardware is at least one factor in why game prices haven't increased. Less because it makes the cost of producing a game lower, and more because it makes the market larger for selling the game into.
The biggest mistake this post makes is conflating the 'seal' of the uc system with its logo. And adding insult to injury, the article also completely ignores the other designed treatments for the logo.

How will the logo appear 90% of the time? Scrubbing to 1:44 seems to give the biggest clue, which indicates that the resultant 'system' is actually a series of font choices and a logo and a system for picking out colors or images which can be used within the confines of the logotype.

Scrubbing back to ~1:30 also indicates that this will also redesign http://www.universityofcalifornia.edu which will likely be a huge improvement over the existing website that appears to have been made at the height of the xhtml+prototype.js zeitgeist.

So no, Cal isn't being rebranded, the nebulous 'UC system' is, which means that the biggest consumers of this logo will be UC applicants & employees along with secondary school juniors and seniors who are receiving either fat packets or skinny envelopes with a UC logo strategically positioned in various parts. Will this logo appear on a diploma? almost certainly not. Will it appear on a paystub or employee badge? probably?

It's frustrating that most people project their fears and worries on a graphic mark but it's even more frustrating that the author ignored the actual identity's presentation.
...there goes my week :D
Perhaps they were afraid of legal issues ?

scraping for fun is okay, but if you would like to build a business most people would prefer to abide by the terms of use and still use the API. remember the story of Pad Mapper ?
This post reminds me of some really rad design work I saw recently on Dribbble:

http://dribbble.com/shots/822965-Onward-California

http://dribbble.com/shots/498949-Collie-For-Knee-Uh
It seems like it would be pretty hard to get investors to back a cable company. Cable television is dying and its for the same reasons you are mentioning. Most people don't watch half the crap they are buying when they pay for cable. That's why so many young people don't even have cable anymore, just Netflix, Hulu, or The Pirate Bay.
Having been disgusted by security holes in Preview and the lack of viable alternatives to Preview/Adobe Reader on OS X, I installed WINE and Foxit Reader on my Macbook Pro. (And used Automator to have it launched as an OS X app.) Google Chrome is ok for a PDF reader much of the time, but there are some really annoying bugs in its PDF functionality. (Which especially impact comics.)
+1. Thanks for the hints about Selenium.

My 2c about scraping - when you try to obtain data from large websites, always go for javascript content. Pages like Newegg or Amazon * may change html outline very often even without a single alteration to the front-user and even your smartest regex can have a brain fart. In contrast, even when site gets major overhaul, most likely old javascript will be left in place with all up to date variables, because engineers will be concerned of removing that code not to break some functionality .

* given you have rights to scrap.

not that there are no tools to debug the site; but I found websites like mentioned plus youtube, and bunch others just not fiddling too much with js.
eskimo.com in Seattle ran on a Model 16 for several years. It was definitely Microsoft Xenix. My brother wrote the multiuser chat system on there, nice piece of software.
You're right, Fox did do some bold things. Married with Children is probably another.
Thanks for sharing this! Albert Bartlett does a wonderful job explaining the consequences of sustained growth rates.

I find overpopulation to be one of the greatest concerns that we are not adequately dealing with. Albert's bacteria in a bottle analogy made me think of Elon Musk's attempt to bring human life to other planets.. will we not just be expanding our time of growth? Maybe we should start choosing a way to reach a population growth rate of 0% and see what countless problems are resolved.
My family usually puts addresses on calendar events, and it's always fun to have my phone beep at me 15 minutes before I need to leave the house, based on current traffic. Then you tap on the notification and you get a map with the option to start navigation.
Right, and this is why sites like Craigslist explicitly forbid scraping. If the site operators wanted, explicitly, to share their data with you, they would provide an API or give you permission to scrape.

The reality of scraping was really known many years ago. If you're doing if for above-board reasons like for research etc., you'll probably get a pass - if you're doing it in order to profit from someone else's work because you are too lazy to do it yourself, it's probably unethical and you won't get a pass --- these concepts have been around for at least a thousand years or more.

Full Disclosure: I have also scraped data - but only from government websites where the scraped data is explicitly public domain to begin with and APIs were not available.
The media report margin of error but not confidence interval, which is highly important. If you have a 95% confidence that the poll has a 3% margin, that's a very different statement than 80% confidence.
Freeware Network Security Tools
Meh, 58% if you click 'Top Critics' and filter out the blog-spam critics. A solid F+.

Plus all the big national papers: the Times, Post, Journal, and LA Times, plus all the trades: Variety, Hollywood Reporter, Entertainment Weekly all shat on it, and many of the small local paper reviews that are counted as positive are mixed at best:

"Although never boring and almost continually amusing, Extract doesn't work as a movie because you don't buy a minute of it, even as silly satire."

or

"Extract has some flavor, but the comedic kick is diluted by flat characters and a thin story."

http://www.metacritic.com/movie/extract/critic-reviews

It reminds me of something Tim Heidecker said about the horrible reviews for 'Tim and Eric's Billion Dollar Movie': "If you're a critic and you walk out of a theater wondering what the filmmakers could have possibly been thinking, maybe you missed something?"

You would think critics would give the benefit of the doubt to the guy who made Idiocracy and Office Space, especially considering the critical reappraisal each of them has gotten in the years since their release.
That is actually very simple, and you can even use a headless browser to execute javascript:

first install Xvfb and pyvirtualdisplay then try this snippet https://gist.github.com/4243582

selenium is great, it can even wait for ajax requests to finish (see WebDriverWait) ..
I absolutely loved my trash-80! I remember the book on basic that came with it and working through all of the programs in the back. The last one was like 200 lines of code and all it did was reproduce 4 basic geometrical shapes on the screen.

Sadly, that amount of typing (I was 10 or so and not very quick back then) for such little output stopped my interest in programming.

I'm 38 now and over the last year have renewed my interest in programming having self-learned html, css, js, python, and django. Really wish I had stuck with it back then.

Man though, does that catalog take me back.
(shameless plug) I can scrape asynchronously, anonymously, with JS wizardry, and feed it into your defined models in your MVC (e.g. Django). But! I need to get to a hacker conference on the other side of the world (29c3). Any other time of year, I'd just drop a tutorial. See profile if you'd like to help me with a consulting gig.

EDIT: Knowledge isn't zero-sum. Here's an overview of a kick-ass way to spider/scrape:

I use Scrapy to spider asynchronously. When I define the crawler bot as an object, if the site contains complicated stuff (stateful forms or javascript) I usually create methods that involve importing either Mechanize or QtWebKit. Xpath selectors are also useful for the ability to not have to specify the entire XML tree from trunk to leaf. I then import pre-existing Django models from a site I want the data to go into and write to the DB. At this point you usually have to convert some types.

I find Scrapy cleaner and more like a pipeline so it seems to produce less 'side effect kludge' than other scraping methods (if anybody has seen a complex Beautiful Soup + Mechanize scraper you know what I mean by 'side effect kludge'). It can also act as a server to return json.

Being asynchronous, you can do crazy req/s.

I will leave out how to do all this through Tor because I don't want the Tor network being abused but am happy to talk about it one on one if your interest is beyond spamming the web.

Through this + a couple of unmentioned tricks, it's possible to get insane data, so much so it crosses over into security research & could be used for pen-testing.
It's a grave injustice to the vast field of philosophy to judge it by one undergraduate's impression of Plato, Aristotle, Bertrand Russell, and Wittgenstein.

By his own admission, he didn't get much out of his experience. Should the entire field of computer science be judged by one unenthusiastic undergraduate's impressions of Java?
1. 0400-1000h, at home.

2. I like having a small amount of pre-defined hours (up to 4), but I hate being stuck in an office when I'm not getting anything done.

3. As long as interruptions first come to me in email or IM, vs. a phone call or someone in person, I'm fine. Having 5-15 minutes to wind things up before switching is important.

4. I sadly do more business stuff now, but was even more like this before. INTX, although not as strongly I vs. E as I once was, and less J vs. more X than I used to be, too.

5. I leave an IM client running sometimes, but not when I'm focusing. What I'd prefer is some kind of agent (like a receptionist) which could mediate IM and email for me.

6. Away, generally, although it helps to have people nearby or at least available if something depends on them.
Justice Markandey Katju claimed, "90% of Indians are 'idiots'."
I had the same thought, and my personal opinion is "not even close." Idiocracy was good, but very much "one note." I can think of other better films - MAS*H comes to mind, not to mention Catch-22 (though somewhat uneven, still better). Even Mel Brook's movies - Young Frankenstein, and The Producers, especially.

Nothing in Idiocracy matches "Springtime for Hitler."
Yeah thats what I am talking about. Game engines in any other thing than C++ is unimaginable.
Try this with perf stat on cpu 0, on cpu non-0, and without taskset at all.

If you have 2-threaded code, it's also fun to try taskset onto one core with hyperthreading.
I've done this successfully with Ruby Mechanize.
I bet you would only need to randomly shuffle between a few alternatives for all of them. You'd need a dedicated effort to work that one out and the cache implications could be managed. No getting around the trade-off of possible page alternatives vs cache nightmare-ness though, and doing that to json apis would get ugly fast.

At least it's easier to code these tricks than to patch a scraper to get around them.
The title makes it sound as if there is going to be some discussion of how the OP has made web scraping profitable, but this seems to have been left to the reader's imagination.

Otherwise, great article! I agree that BeautifulSoup is a great tool for this.
Enthuse.me – Start showcasing your expertise
If this is the case then why don't CPU schedulers try to keep single threads running on the CPU instead of context switching?

I've seen this for example encoding AAC audio. I have an 8-core system but the encoder is single-threaded but the Windows scheduler still spreads the process out over 8 CPUs. Wouldn't it be better to stick on 1 core causing cache hits to be higher?
Yes, Facebook used to do that. I had to scrap it once and was surprised by randomly changing classes around input fields.

but who cares, no one can beat Xpath :)
Just imagine the difficulty in designing and publishing the actual catalogue considering the power of the computers advertised.

It's full of big charts and text wrapped around deep etched images. Making me dizzy just thinking about it.
So what. If they decide to fire you before the effective date of your resignation, you're eligible for unemployment benefits and/or the severance package. I've known a few people who knew that layoffs were coming, but chose to stay anyway, simply because they wanted the severance payout and were confident they could find a new job relatively quickly.

Financially, you usually come out ahead, although you now have to admit that you have been fired in the past. (Interestingly, layoffs are legally distinct from firings.) But if you already have a new job lined up, I don't think it's that important.

Looking at the bigger picture, if they weren't willing to keep you on for your last two weeks, either they saw you as not that important or useful at the company, or the company is in serious financial trouble. Either way, you're probably better off forgetting about that company and enjoying your extra two weeks.
Friend of mine still writes actionscript by hand on occasion.
I have reread the series three times. The last time was about a year ago. It reads like history to me. As if this all really happened. I think that is one of the aspects that captured my imagination as kid. I could never fully articulate it. I look forward to listening to this while I work next week.
After we started letting women drive, this was only a matter of time.
"Go, Go, Go..."
Great Big War Game Developer: Windows App Store "Born To Fail"
Actually, I'd say that's more an in-valley appraisal. I suspect non-tech types would simply view SV as computer nerds finding ways to make millions. Now that Jobs is gone, the industry is dominated by nerd CEOs. Ballmer doesn't count.

We have, in the Valley, a preponderance of companies started and run by nerds. I suspect that reality has leaked into the public consciousness.
Haha that's what I was thinking when I read this. Hopefully they have more success than we had.
Gaymercon.org
What if there is no passage of time? You could be in a single state but remember the "past" and the perception of moving through time during the "past".
underlines my fear that universities are going to face a Quick, Painful Death...if money is going to be spent on matters such as these...
I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping.

In my spare time, I've been playing around with "scrapers" (I like to call them web browsers, personally) that don't even look at markup.

My first attempt used a short list of heuristics that proved to be eerily successful for what I was after. To the point I could throw random websites with similar content (discussion sites, like HN), but vastly dissimilar structures, at it and it would return what I expected about, I'd say, 70% of the time in my tests.

After that, I started introducing some machine learning in an attempt to replicate how I determine what blocks are meaningful. My quick prototype showed mix results, but worked well enough that I feel with some tweaking it could be quite powerful. Sadly, I've become busy with other things and haven't had time to revisit it.

With that, swapping variables and similar techniques to thwart crawlers seems like it would be easily circumvented.
Eclipse is a popular Java IDE (with plugins for other languages). Android programs are java with an XML file describing the UI. No need to re-invent the wheel here. If you have another IDE you want to use, see if there is an android plugin for it, or right the XML files yourself and use any java IDE, or use a plaintext editor and curse Google for making Android run Java.
> So you're not so hot on the whole search engine thing?

They scrape to generate links for users to go to the site. That's quite different than scraping for...any other purpose? So it seems. Would you (anyone) argue otherwise? (genuine curiosity).
Stanford only changed the font of the wordmark, not the S or the S-and-tree or the seal. I, for one, welcome our new wordmark. It's a pretty inoffensive change, compared to the lolfest that is the "perpetually loading" new UC logo.
Better than that is http://node.io/ Also, don't use jsdom (it is slow and strict), https://github.com/MatthewMueller/cheerio is much better.
Bird Buggy Keeps Noisy Parrot Quiet
I'm also an alum and worked as a graphic designer for about 5 years. I don't think it's that bad, and used properly is actually pretty effective.
Just out of curiosity, which security holes in Preview are you referring to?
I've been kicking around and researching this idea for literally years. There are a few huge roadblocks to success, some of which are mentioned in the replies here. I'd be interested in talking to you more if you'd like.
Less legacy, eh? Well, I guess there has been less emphasis on compatibility throughout the years, but NeXTSTEP is older than Windows NT...
"Darling also possesses some hope as the project is being worked on for a diploma thesis research project by a university student..."

I would draw the exact opposite conclusion.
1. That doesn't address search engines, which are doing it to profit from someone else's work. If you open the door for search engines then how many search engine like things do you give passes to?

2. What if I'm scraping it just for me, because I want a different interface? How many friends can I share that with? Can I open source the program?

3. What if I read a bunch of these sites to do research and write up a story on something about it? Not plagiarizing, just summarizing and providing analysis on craigslist rental prices? What if I do this every day? What if I automate that process? The data is transformed just as much as if I had read it myself and crunched the numbers myself, I made just as many requests to the site as my browser would have.

Concepts that have been around a thousand years or more are not fully applicable. Like the printing press, some things alter the scarcity equation for ideas and data distribution and ownership. Considering how little we've agreed on about print after 500 years I have some doubts that this is as closed an issue as you say.
Maybe so, but that doesn't mean they should not pre-install a free OS. The fact that they official support Ubuntu gives confidence that it will work with (almost) any linux distro (possibly with some work on your end).
Dereferncing NULL allowed on some architectures
You should put (2004) in the headline, or it'll look like Karma Fishing.
PhantomJS doesn't need Xvfb anymore it can run headless without this dependency.
Phantom.js and casper.js

If you can't get the data from the endpoints the javascript hits then write your scraper in javascript and have it run in a headless browser, and it's the webkit engine so most sites test their site against it heavily.

Either pull the data out of the javascript objects or trigger your extraction from the html by attaching to the events in the javascript.