skysight-inc/HackerNews huggingface dataset:
I don't mean a dictionary (also a good idea), I meant texts: articles, novels, blog posts, transcripts of conversations, etc.

As for dictionaries, there's wiktionary, but it's broken because it's based on words, not meanings, so you'd need 30*29 translations for each word.

Mmm, maybe I should do it...
"Including Rescuetime and 8aweek in our content for promotion of the program was our attempt to highlight successful YC innovations" - (A)

Hmm. The original sentence was

"We provide RescueTime (www.rescuetime.com) and 8aweek (www.8aweek.com) on your systems to keep an eye on what you are doing."
(B)

How are the applicants supposed to infer A from B ? Is this another attempt at "humor on the internet"?
If you really want to go bonkers on scraping, there are books on this.

http://nostarch.com/frameset.php?startat=webbots

http://www.oreilly.com/catalog/spiderhks/

That probably covers the topic of scraping pretty exhaustively.
This only works if you can guarantee that the changes are not intertwingled in the same code block. He is also taking a gamble that the result of his selection will actually parse/compile/pass unit tests (select as appropriate) - he has no way to be sure of this. He is pretty much going to have to do a "get" into another working copy to be able to check this out, so I don't think he is gaining quite as much as he believes.
That being said, it's important to have a broader knowledge so you can actually communicate with other team members intelligently and effectively. Simply specializing in a field, and mastering it, can sometimes lead to weird and uncomfortable "siloing" of tasks.
I'm sorry, but saying that when I use sites like Flickr for free I'm lending them free labour is like saying I'm an employee of NBC for watching TV.
By selling support and services. For example, take a look at Al Fresco.
awesome killscreen! :)
Young was just correlated with being cheap (I'm 27 with 2 kids and a mortgage. 5 years ago I was cheap but definitely not now). Also I remember you wrote that you wanted to test the lower bound of how old a startup founder could be. Your interest in undergrads was unusual when you started (maybe not so unusual now due to your success).

I chose acquirable as opposed to profitable because of the emphasis on creating great technology and great products, and worrying about monetizing and business plans later. Having a profitable business acquire you frees you from worrying about it at all. Being profitable doesn't hurt, but you've said over and over again that if you have a product that users love, you can find a way to make money.
You probably get a grace period after you leave your jobs that might be close to the 3 months required for YC. At least as a start.
They seriously need to improve this check. I've got a static DSL IP at home, and I get that screen way too often. I shouldn't have gotten it a single time, really.
An H1-B visa doesn't prevent you from switching jobs no?
You're missing the genius of it: Fake Steve Jobs is a fake troll. That's what makes him so funny.
I thought it was odd (plus I was a little jealous that I couldn't apply - married, grad school, lots of debt, etc). But the blogosphere and all the talking heads went crazy about pg forcing the children to drop out of school and move to Boston, giving them peanuts to take an unfair share of their equity, etc. People didn't get it so they attacked it. Looks like some things never change (Arc, caged lions, etc)
I used to get this when I used a proxy.
I think 'all failures' would be... kind of a downer, and wouldn't sell well either. I'd be very interested in reading interviews with people that succeeded after several failures, though.
It doesn't. But the company will have to be registered as a U.S. based entity first which may take some time and will only be done after YC starts.
Hm... you're right. Those are very insightful. Unfortunately, the two examples you came up with are 3 months apart, and fake steve posts new blog entries all the time. My experience has been that every time I see a link to one of his posts, and I read it hoping that this time he'll impress me, I keep being disappointed.

However, I notice that the two posts you point out are much more business related than technology related. Since the real fake steve is a business reporter, perhaps he shines when he's in his element and falls flat when he strays out of it.
Dang it, I fell in to the trap (again) of not distinguishing my opinions from a summary of the opinions I've read on the web. You'd have a hard time finding a bigger fan of the "wealth creation" view than me, but most people don't think that way.

I'm sure many (most?) investors and observers think there are a fixed number of potential successful startup founders and that if YC gets all the good ones, then there will be none left. pg takes the opposite approach, trying to grow the pool of founders through his writing and community efforts. His anti-job, anti-BigCo essays are eye openers to a lot of people (myself included) who had never thought there was another way.

pg is definitely not out to dominate the market - I pointed out that he purposely ignores a large part of the market to focus on the niche he's interested in.
True transcript history (not just the last 15 messages). Searchable transcripts and file uploads. Inline image preview. View Paste link generated when code snippets are posted. Invitation mechanism and permission settings. Better U/I.
http://nintendo8.com/game/629/super_mario_brothers/

http://nintendo8.com/toplist/more/
For me it's a mess.

If I start to fix bug B on top of a half implemented feature A, I end up with a mess.

I don't understand why you would work that way, and don't understand what git solves that svn doesn't. You still have to untangle the mess yourself.
I understand that, and appreciate it if it's done well, but it doesn't stand on its own. Trolls are like lies, the best ones have a hint of truth to them. Most of Fake Steve's posts that I have read miss that crucial element.
I think millionaires are made. Billionaires (or hundred-millionaires) need luck.
I come from a Perl background so I've been using HTML::TreeBuilder and XML::TreeBuilder to do my parsing. It will basically load an HTML/XML file into it's own tree structure and give you an easy way to go through it. By knowing how each site names their divs/classes I am able to scrape.

I took a quick glimpse at beautiful soup and it seems to be doing something similar - someone let me know if this is correct.
2007 called...
"Unfortunately, the two examples you came up with are 3 months apart, and fake steve posts new blog entries all the time."

You don't like most of his posts, but it's also the case that most of his posts are not on the HN front page. And thus, we have the value proposition for Hacker News: instead of filtering based on the author, filter based on the votes here.
How Often do you Calculate your Startup's CPC ROI?
On a desktop? Mouse for sure. Right now I'm on my laptop and using the trackpad though.

I hate the trackballs.
But then how do you explain the popularity of this post? It was completely vacuous, but is still at 13 points as I write this.
I would be amazed if it was just the basic game. But music, too?

It's strange that we don't see more simple/casual games in JavaScript. Is the popularity of Flash the reason why?
The most powerful, general level scraping stuff I've come across lately has been ScRUBYt : http://scrubyt.org/ .. although I admit I don't have much to do to use it often.

It lets you specify which items on an initial / prototype page you want to scrape, and then it builds up a set of rules than then work on future similar instances of that page. Good for scraping eBay, Google, stuff like that.
Screw.Unit, a new JS testing framework, version 0.1
As his extensive suite of benchmarks and product comparisons shows, this elegantly overcomes the problem of diminishing returns that come with adding more read-only slaves in a classic master-slave RDBMS replication setup without resorting to extensive use of non-relational solutions such as memcached.

Just kidding, he did it cuz it's fun to write cutesy JavaScript hacks that leak memory like hell and turn any machine slower than a Core 2 Duo into a paperweight.
Interview with Adrian Shaughnessy, Author of 'How To be A Graphic Designer Without Losing Your Soul'
"Also AJAX certainly has changed the way a lot of screen scraping is done."

I'd be interested in how you tackle this one. I've always used something like Perl/Curl/wget etc for scraping, but (like you say) JavaScript messes that up. I've had moderate success using GreaseMonkey and regexps in JavaScript code, but it's a bit fragile. I'm thinking of using GreaseMonkey + jQuery, since that should allow me to select DOM elements very easily. But if you have a better way, please share :)
From the article: "What strikes me as particularly entertaining is that, if their product/service offerings ARE comparable or better than Big Business offerings, perhaps if they turned their passion outwards instead of just ranting and gushing to each other and at you, more of the world might know about it and they might get more market traction and be greater catalysts for competition and change within their industries."

This is exactly why open source software isn't more popular than it is.

Maybe it is time to get some open source marketers on board?
I've recently finished reading his book and I highly recommend it. As a web developer much of the information still applies.
http://germany.usembassy.gov/germany/visa/vwp.html

There's a large list of countries for which a stay of up to 90 days for business or tourism does not require a visa.
I think that's pretty cool. Not sure about the "massively scalable" part, though
Google Launches Solutions Marketplace
Additionally you could sell add-ons and upgrades.
Thinking in URLs
Why no JSON?
He certainly sounds authoritative to me.
Longest Common Subsequences are quite useful as well.
nod
I've thought about this a fair amount, too. You can do a lot to, say, figure out which pages contain recipes, even identify the structured information like ingredient lists (they're just lists full of foodstuffs and quantities). But IME it all falls apart when you need to find a block of text - like the descriptive part of the recipe. That's rarely marked up very clearly, and tends to blend into the rest of the text. So you either miss parts of the recipe, or pick up chunks of junk from the rest of the page.

That said, it's likely do-able, as long as you don't need perfect results. There are plenty of sites around that seem to be doing things along these lines - but AFAIK none of them have open-sourced their code.

Meanwhile, I've been a coward and stuck to beautiful soup for my scraping projects. In the short term, it works out faster than trying to be too clever.
Yes it is common for offices to work in Saturday morning in India. But that is not the point here what I am trying to make. Many people commented on by blog and I am pretty clear now that it is not necessary for people to have same kind of motivation as I have because ultimately we are different. Working on saturday morning is not bad. Most businessman in America works 7 days a week and that makes that country looks so different.
It's all statistics. My suggestion is that HN is supposed to be a better filter than just ignoring bloggers who infrequently post good things.

And this off to a long digression about the opportunity cost of reading the bad articles vs. the benefit of reading the good ones and on and on...

But yeah, it's not perfect. If anything, if there were no articles you disliked on the front page, then HN would probably be too conservative a filter.
"If something goes wrong with Microsoft, I can phone Microsoft up and have it fixed. With Open Source, I have to rely on the community."

That's always sounded better than it is. My experience of contacting technical support with any proprietary vendor is the usual tale of being put on hold, messed about for days or weeks, waiting ages for a response to my emails. I started and finished a job with a company last year; it lasted 6 months. The first call I opened was with Sun, and it was still open when I left.

Anyway, the 90s just called, and they want their tired old debate back. Companies do make money off open source, it is used in all kinds of businesses, you can get support in a variety of ways and save a ton of cash in the process. The question in the OP is valid, but it was answered a long, long time ago. The best answers came from companies like Google, Red Hat, MySQL, etc, etc.
If you're working in the country already it's unlikely you can use that. I tried to switch to a tourist visa after leaving my job (was on a TN). Denied - they said that I failed to reestablish residency in the foreign country.
When you go beyond the consumer web and desktop markets, software actually sells for a lot of money. There are tons of industry-specific and line of business software markets that are very lucrative, but they aren't immediately obvious to an outsider.
You've seriously never looked at a file and said to yourself, "These changes belong in two separate commits"?
id say fake steve is sort of a blog version of the daily show / colbert report for tech news. its ridiculous and cynical, but sometimes very true.

however, this particular story is just silly fluff. thought news.yc might get a kick out of it though
Yes, in the posters case it's probably not useful, however it might be for other readers.
Can't you work on it as long as you don't get paid?
I didn't know there were wannabe quants on YC. I thought they were all on nuclearphynance. ( Which you should check out if you are interested in seeing how the real quants do things. Opentick was on np about 2 years ago. )
A temporary business visit under visa waiver or one of the B categories does not, as far as I'm aware, allow you to "work" except under certain circumstances.

Meetings, sales calls, repairs, market research, and similar activities are not considered "work" and are allowed. It would need to be investigated whether R&D (which, effectively, is what you're doing when working on a YC project) is allowed.
Mozilla out-sourcing their website usage data capturing
I use Ruby, with its nice regex support and libraries (open-uri, REXML) and the hpricot and mechanize rubygems.

Yahoo Pipes is also fun to play with; and Firebug is the scraper's best friend.

Right now I'm working on scraping public LinkedIn data. In the past I've done Craigslist and Twitter. I haven't done anything really hard, though -- mostly things that can be read as XML.

Here's a few cool links if you're interested in scraping with Ruby: http://del.icio.us/jeremyraines/scraping
I really shouldn't be doing this...

> Why is "metacrap" a problem for the semantic web, but not for data-Wikipedia?

Because Wikipedia is centralized, and the SemWeb isn't.

> The Shirky article is a well-known strawman.

DH3. Contradiction
Hmm, I'm surprised at the negative reaction to this comment. Anyone care to expand on why they voted it down?
I never said 37 Signal's idea was original or defended them on that point. I also stated that before. When you take the functionality and design from one specific site without adding innovation then that is a rip off. I think too many people are caught up on whether or not they like Ruby on Rails or 37 Signals vs. seeing what Google did and calling foul. Good for Google to own up and pull the site though.
That's the most insightful thing I've read about app engine yet.
We're using a general-use multi-threaded crawler to get the pages and then using Beautiful Soup and a bit of regex to parse them. Though we are scraping multiple sites, they are all in the same "category" so to speak, so there are a lot of generic parsing methods that are simply overridden when necessary. PyParsing was played with for a while, but since data comes in so many slightly varied forms I was ending up with rules that were miles and miles long just to find a simple price or date/time on a page that would work for the largest number of sites possible.
Misquotes? Looks verbatim to me...
There's a pretty good list here of what's allowed and not allowed:

http://vkvisalaw.wordpress.com/2008/02/23/what-“business”-ca...
It involves Paul Graham.
How is this Hacker News-worthy? When I saw this on the recent posts page I didn't expect it to end up #3 on the front page. Why so many points?
Even if they did copy Campfire, I don't really see the problem. It just goes to show, that if your software is really easy to make, you will probably see some competition. In my own startup, the business idea is to make localized clones of cool web apps.
Issue 29: googleappengine
Disclaimer: I use Rails daily, and have no hard feelings for DHH or 37Signals.

That out of the way, you are still saying that Google took the functionality and design from Campfire. Where as what I am saying is that Google took the functionality and design from almost every web chat program that came before it and there was a large number before Campfire itself existed.

To say they took it from campfire is to ignore all the existing implementations that existed before it. It is to rewrite history with the starting point for web chat on the launch date of Campfire.

I doubt Google pulled it because they thought they ripped off Campfire, my money would be on them pulling it because it wasn't important enough to risk bad PR over.
I would have to disagree with you. And the reason being that my boss got fired 2 weeks ago!
Mice are bad news for RSI -- you grip them constantly, and make individual finger movements, all while pronating. Trackballs and trackpads are less precise and convenient, but much better for my wrists.
sigh

All together now: Open source is a software development model, not a business model.
And finance by a tax, obviously. So you still get a positive net outflow from better off people.
I think it is the company as a whole and the benefits that they provide make ppl very comfortable with where they are. Plus, a lot of them don't won't to take chances like us!
This is a discussion that always happens, in just about every context. It also depends massively on perspective.
Take somebody who codes java and haskell, does database admin and web design, and hacks the linux Kernel: I'd call her a generalist. But from a wider perspective she might be a specialist: concentrating on hacking rather than literature, or business, or physics. It's all relative.

[Incidentally, I'll mention that Isaiah Berlin's essay 'the hedgehog and the fox' is a classic of the 'generalists vs. specialists' genre, applying the division to writers and philosophers. I'm not a massive fan of it (dividing people up like that gets a bit superficial), but looking up papers citing it will get you into the recent literature on the subject]
Our guys were just on Tourist Visas the whole time and are now going for O1 and J1 respectively. Not sure how it works differently if you have an H1-B first though.
Do you mean for adding a little resilience to your rigid model, or something funkier?
Someones got to pay. May be other revenue streams subsidizing the open source work. But its naive to believe that support & services alone can sustain a software producer.

Imagine would Red Hat be able to sell its version of Linux for the cost that it does if it were to produce the whole thing? Remember the guy who contributes to the overall effort cannot live on air and water. Could be large corporations(IBM, Sun) or the State(any country with strong social security, free education etc), putting the money to support the people working on open source.
Project Caroline - Sun's stealth mode cloud
Do you think Loopt would ignore the iPhone?
What about SEC Info(http://www.secinfo.com/)?

Ubiquitous availability of information is the first step. Then comes consistent data formats. And then the tools to slice, dice and weave shimmering insights. And then maybe, just maybe, act on that insight.

Humans are a very hopeful race..
There is no grace period for H1B, not a single day. If you leave your H1B job, your H1B status is lots immediately.

It might be possible to apply a new H1B based on your YC company, or obtain other work authorization like EAD to be able to work in the US.
Read some of your posts. You seem to be a good for nothing migrant who can't appreciate a novel attempt to promote startups in India. More so, you must be one of those guys who couldn't make it to Google, couldn't make it to IIMs and now massages your ego criticizing everything good that happens in your motherland. Ridiculous !!

We are a bunch of IIT CS grads all set to apply to this program at IIMA and take our ideas to fruition. Look forward to your more of your criticism once we are successful.
I'm surprised nobody has mentioned dapper yet (www.dapper.net) it's a really nice approach at turning web-sites into structured content.
You get a grace period if you leave your job right? I just don't see this as a showstopper...you could've applied and dealt with problems as they came up.
um yeah... i misread the article, didnt think he correctly linked to you, just thought he called you a "non-famous programmer dude". sorry, should have been more thorough (but i never imaged this post would get ranked so high -- who knew)

just an announcement, fakepaul.blogspot.com is open
the key advantage that a generalist has over a specialist is being able to
adapt to constantly changing conditions quickly and more easily... from the view of self-interest,
given parallels in the animal kingdom (racoons have survived even with their
natural habitats destroyed, ...) - it's the way to go
Switching jobs will require a new H1B, normally called 'H1B transfer'.
If you want Ruby instead: http://heroku.com
Hmmm, maybe. But it does provide we students a similar platform for launching our innovative ideas into the market.
For a startup, the real value of YC is in the ‘network effect’ (as a business grad friend of mine calls it) and the ‘know how’ (the whole advisory package) of building a business around an idea; that is hard to beat even with time, particularly with time.
Getting just the recipe would be the hardest part, but it's still doable. Once you figure out that you're currently parsing a recipe (via keywords, close matching, whatever) you could fan out and look for common start/end tags like <p>, <div>, etc. If you use something like Beautiful Soup you could do this pre-parse instead of post-parse and eliminate a lot of extra stuff (no recipes in the <head> tag, etc.)

After that it just becomes an issue of removing the cruft around the recipe. I would start with common stuff: splitting things up by <br> or inner <p> since if someone is gonna have something before / after their recipe (say, on a forum) it'll be split up with blank lines somehow (well, usually). This will be another time to use things like close matching and teaching the algorithm what it gets right/wrong so it can weigh things as recipe/not better in the future.

If you do all this and add more specific edge cases as time goes on, I think you'd be able to maintain a 50% correctness rate pretty easily.

Edit: And it'd be much cheaper than a neural network ;)
> Most of those value based companies sold to community companies.

Exactly. Large acquirers typically already have a community, and are looking for additional products and services to offer that community. Meanwhile, community sites are typically more difficult to integrate. Witness Yahoo's struggles with integrating Flickr and del.icio.us.
Read some of your posts Mr. ABCD. When was the last that you visited India? Ever tried starting up here? You aren't probably the best person to comment.

You seem to be a good for nothing migrant who can't appreciate a novel attempt to promote startups in India. More so, you must be one of those guys who couldn't make it to Google, couldn't make it to IIMs and now massages your ego criticizing everything good that happens in your motherland. Ridiculous !!

We are a bunch of IIT CS grads all set to apply to this program at IIMA and take our ideas to fruition. Look forward to more of your criticism anyways. That's what you are good at?.
10 lessons from a web startup founder