skysight-inc/HackerNews huggingface dataset:
I used Mechanize and Hpricot on a project recently to create a sort of poor-man's API. My client is a performing arts organization that wanted a new website but they already had a (dreadful) internally-hosted site for selling tickets.

In order to keep website users from having 2 accounts I created an interface that scrapes the sign in, sign up, lost password, change password, and couple other screens of the internal system. So when users come to the website and "login" they're actually logging in to the internal system and I just record their session from the internal system so I can masquerade as them as they go about their business.

It's not going to support 100s of connections per second but it gets the job done for their traffic levels (36,000 views the first day of launch).
Blog reading becomes a habit, study says
I cautiously agree, but I have two big concerns.

One is that Google's DB is both unique and closed-source. Worse, it isn't even a product like Oracle or SQL Server that I can license and then use the way I want. Even if I wanted to buy the hardware and maintain the software, I can't move my app off of Google. I'm a sharecropper. They get my traffic data for free, and if they change their pricing structure or the way their DB works I have to scramble to keep up. And if they decide to screw with me (like, ahem, becoming my competitor) I will have to port my app to some other scalable platform while serving all my accumulated traffic and simultaneously fending off a deep-pocketed competitor.

I'm also suspicious of the idea that BigTable is some sort of magic wand that solves any and all scalability problems. I'm very sure that BigTable scales for the classes of problems that it was designed for. And I'm sure that a wizard like Steve Yegge or Peter Norvig can adapt it to many other classes of problem. But, without actually knowing anything about BigTable, I'm prepared to bet that using it to scale your web app will require (a) knowing a fair bit about how the tool works and (b) customizing your app's data storage scheme to compliment the tool, after which (c) there will still be some corner cases that don't work very well and require clever hacks and compromises.

In other words: I predict that in three years there will be Craigslist job postings for "BigTable DBA with five years of experience".
Juggernaut self-contained habitat for plants
Playing with Signals in Ruby
I'm inspired by his other works too

http://philross.org/ross/index.php?kind0

http://philross.org/ross/index.php?kind040
He makes an interesting point. It is quite a process to find the right data.
What Would NFS.net Do?
Should this be joining the entry at http://news.ycombinator.com/item?id=158998 ?
Yes. You can even regex search through the tree.
Weeeeee!

BeautifulSoup is nothing unique, but it can handle malformed data that saves you a ton of hassle.
I don't want to turn this into a huge debate either, but those articles (and uncritical readings of them) have set the web back years.

> Because Wikipedia is centralized, and the SemWeb isn't.

If data-Wikipedia and a television station are both publishing data about when your favorite show is on that station, who are you more likely to believe?

Obviously you need to be careful about where your data comes from, but a single centralized source is not necessarily more trustworthy than many carefully selected sources.

Blind crawling isn't (and will probably not be) the norm for data collection on the semantic web.

> DH3. Contradiction

Heh, got me there.

Shirky's thesis is based on the idea that making inferences from data is the ultimate purpose of the semantic web.

But linked, machine-readable data--that is, the semantic web--is useful even if inferencing is useless. I don't think this is a claim that needs evidence, it should be fairly obvious.

Shirky's article's portrayal of the semantic web has little to do with the real thing. Here's a much broader debunking of it: http://www.poorbuthappy.com/ease/semantic/
I guess Joyent wasn't kidding about the free hosting
Pretty neat, but how useful can that be, surely not scalable; How about Insert and Update statements?
Calcanis on entrepreneurship and death by blogging
The Email Habits That Make People Hate You
Proves my point. Failed to crack CAT uh? Trying another attempt at inference problems - alas! For someone who doesn't know what rescuetime is - he would have Googled and figured out that its a product from YC. I didn't know about YC till I came across this program in India.
It's only mostly irrelevant: if you're not already using Microsoft tools (and you're probably not, if you're on news.yc), this is surely irrelevant to you. This surely isn't going to make anyone switch to Microsoft tools.

If you are a Microsoft developer, and you expect to need a cloud infrastructure, this means that if you can afford to wait and see, maybe you don't have to throw away your existing investment in Microsoft tools. If you have a sizable investment in Microsoft tools this is likely very important to you, because migrating from .Net to either AWS or Google App Engine would be enormously complicated.
Get full Gmail-like keyboard shortcuts for News.YC (cursor, scrolling, j/k, navigation, etc.)
Very cool - this is pretty similar to microformats.org's design principles. I could see this kind of HTML markup used as an alternative to maintaining a separate REST based API.
I've done my share of screen scraping, gathering all different kinds of data. Movies, sports, finance, you name it. Here are three things I can tell you:

1. Take the time to get very familiar with regular expressions. If you think you know your regex pretty well, go to the docs or get a book and find three things you don't understand and understand them fully. Then find three more.

2. The data doesn't have to be perfect. In most cases you can clean it after you've stored it. It's generally better to get more than you think you might need (in terms of data or html/formatting around the data) and then go back and clean it later

3. Generally, my most successful data mining algorithms involve a lot of hacks. There are very few clean formulas...usually I have to play with the data for awhile and fix a lot of one offs and special cases and then it ends up coming out ok
Ask YC: Which notebook/laptop do you use (+ suggestions)?
I currently have a MacBook Pro and love OS X and its architecture, but I was looking to purchase another notebook to run Linux (yes, I know I can do that with my MBP) and wanted to see if any of you had any recommendations.

I'm looking for something cheap (i.e., around $1,000) that looks decent, such as the Lenovo R, T, or X series. Obviously it'll have to be able to run Linux, too (probably Ubuntu).

Suggestions?
Hey,

would you please take your sophomoric personal attacks elsewhere. Good luck with the IIMA program.
XSS (Cross Site Scripting) Cheat Sheet
Hot startups for Founders at Work v2
The only book I've read cover to cover and I want MORE! If there were a follow up on this, which successful startups would you pick??
29 months is a long time to start something and launch. I think finding a local co-founder would always be helpful!
This reminds me of Brendan Barnicle's MyLackey E-Mail
http://www.fuckedcompany.com/extras/mylackey_letter.cfm
With 50 people you have to start measuring results not focus on effort or face time in the office. I would be interested in what the employees feel would be the best way to improve quality and productivity. They might opt for other changes than working longer hours.
Good luck at your experiment. I think it is great that you are trying something.

You ask an interesting question though - MBA + Techies > Techies. I am not sure. Most of us look at the sloanies as leeches although they did help me a little in defining the market for my project.
Windows collapsing under its own weight; Radical change needed
Taking on the World
I had a Thinkpad X series before I got my MBP and I loved it. It was light but powerful enough and installing Linux was a breeze.

That said given the choice I'd stick with my MBP and just tripleboot(already have vista installed for testing)
It's really spot on. It shows that what they're trying to do is build extra functionality on top of Google services. It is the closest step towards a Google desktop yet. It's like issuing an SDK for your OS.
Microsoft’s Mundie: New programming models needed
I was just talking about this with a YC applicant yesterday. Scaling up HUGE is a really interesting topic, and a lot of nerds love to talk about it...but the number of sites that actually require extreme scaling is very low. When we first started selling Virtualmin we had a lot of early adopters asking about database and web app replication, load balancing, etc. Not because the folks asking actually had sites that needed that kind of performance, but because they're cool technologies to play with. A thousand paying customers later and the demand for those features has dropped to background noise (the same early adopter folks who mostly just want to play with it rather than have the traffic to justify it). Far more of our users are asking for the ability to run more (more sites, users, mailboxes, applications, etc. and not generally more reqs/second) on a single server rather than the ability to spread load across many servers.

The performance of hardware has managed to keep pace with the needs of the vast majority of websites over the years, to the point where very few sites (like the top 500 or so) actually ever need more than just the basic scaling ideas that are easy for just about any sysadmin to implement (split mail, web, DNS, and database onto independent boxes, use memcached, maybe a web load balancer like Squid or pen, etc.).
By we, do you mean CIIE / IIM ?
In any case, Best of luck !

P.S. I hope you keep updating the blog.
With respect, many startups simply persist, some profitably, some at breakeven. Many founders would rather be in control than wealthy (or working for someone else). Doubtless you screen out folks who favor control over wealth so you don't see them. It's taken me a long time to understand that while every founder "wants to grow" many don't want to grow beyond where they can still be in charge.
I use a MacBook Pro with the highdef screen as my only machine. For multiple OSes I run VMWare currently with Windows 2000, Ubuntu 7.04 and the OLPC operating system. I wouldn't waste my time with another machine, since the MBP has two cores you can just use one for you Linux installation and have plenty of horsepower left for the Mac.
Thanks for the links and information. It seems like there aren't too many options. The other visa classes seem too restrictive.
Malicious software overtakes friendly apps
pg forced them to do this by putting an application form on the web with so many rounded corners they couldn't resist filling it out, or what? :)
i know sun and microsystem do promote a lot of startups in india...does google do a similar program?
Facebook, Youtube
I have a piece of paper stuck to the wall behind my desk that reads: "Research is blundering into an alley to see if it's a dead end". I think this falls into the same category. It's obviously not directly useful, but it might spark a thought in the creator or someone else that does lead somewhere.

And if not, well he probably had a good time coding it.
http://www.amazon.com/When-Giants-Stumble-Robert-Sobel/dp/07...

Not startups (although some of them weren't in business very long), but this is definitely a good overview.
Next time you're in the same TZ as me, buy me a beer and I'll tell you about:

1. Going public

2. Doing a fire sale where almost everyone lost their jobs

3. Being acquired and hating the acquiring company

4. Starting a company that's just barely breaking even but still alive
While I haven't used either (or BigTable for that matter), I was under the impression that both HBase (the Hadoop database) and HyperTable ( http://www.hypertable.org/ ) were Open Source competitors to BigTable. But there does seem to be some work happening there.

It's pretty rarefied air up in the high scalability world...my previous company built website acceleration tools, and the customer base there is pretty small (they have plenty of money, but it's not a high volume business). As much as us nerds like to think about HUGE problems like this, it's a problem that just doesn't come up that much.
Yahoo to outsource ad management to Google?
This is a great post, but mis-titled. "Does success of Y combinator point to existence of many alternate investing and incubation models" (in addition to traditional VC, Angel, and office space based incubators).

There are several very thought provoking ideas in his list of alternatives, in particular: A FPGA centric fund (certainly Xilinx, Altera, Actel might sponsor), revenue generating startups with a dividend focus (or venture buyout/buyback, especially if YC2 puts last investment dollar in and doesn't have to worry about dilution), mixed business and technical teams, more experienced professionals.
yes for sure....a lot of techies do end up doing their MBAs but rarely follow the tech-path post that...interestingly thou i've been reading about iim grads doing tech start-ups (not to mention the biggest successes of indian web 1.0 ventures - rediff, naukri n makemytrip were iim grad ventures and not iitians)....incubation is beginning to happen in India even at engineering colleges...i am from an NIT (comparable to some of the better IITs) and they've taken a leaf from IIMs to implement the placement holiday...i dont have to take the risk of not finding a good job in case my start-up fails...my college lets me sit in the placements over next 3 years
Is your MBP old? Why can't you just save that money and put a bit of it towards VMWare Fusion? The extra savings could come in handy for something else, especially if you're in the startup game :)
My Lenovo T42 runs Ubuntu great, but the 40s often have motherboard issues, so only get one of these if you are also getting warranty service. I hear the 60s are better, but don't know what current prices are on that series.
Free Music Encourages Sales According to Last.fm
I work in the same office as an affiliate marketing company, so it's their job to be checking Google all day for their adverts, but over the last few days this has been happening a lot. You'd think that having 20 people on the same IP address passing the check that they'd stop doing it, but no, we're still getting this. You would have thought that Google would have sorted this out pretty quickly.
Maybe Sun will dust off Clustra from whatever closet they left it in. Or maybe H-Store will come out soon and support a large enough subset of SQL to satisfy people.
quite risk averse believe me...getting 25k from my parents for a start-up is going to be tough....here i'd get some sustainance stipend plus system and server etc hopefully :)....i will at least try for a while now...helps me run for longer if i get selected for this program...how did you finally convince your mom..need some tips
$1000 for a laptop is not cheap. $400 for a laptop is cheap. At that price, you could literally buy a macbook and just install ubuntu on it if you wish.

I personally have a macbook pro, too, but just couldn't be bothered to add ubuntu to it, so I bought an old compaq off of craigslist for $350. Duo Core, 2gb ram, and 80 harddrive.

Now I have a laptop just for os x programming and cross-platform programming, and linux/(in the future)xp testing, minor programming as well.
I use the Lenovo X tablet for Ubuntu 7.10 and it works great - installation was a breeze and the wireless worked right off the bat.

Not sure why I got the table functionality though.
control dude...lets not make it personal...plinkplonk does have a point but he cannot understand our requirements...it seems iima does....see you at iima hopefully...:)
Tablet, when I'm doing graphics.
This is a pretty weak list -- make sure your emails have a point, don't use bad grammar, etc.
Ask YC: When can we expect to know our YC application status?
I know it's supposed to be today but the waiting is starting to drive me crazy. Any ideas?
U.S. Not Falling Behind in Internet Infrastructure Afterall
I have a MBP running Linux, following a PowerBook running Linux, and would now recommend against Mac hardware.

Apple have a nasty habit of changing out parts in the middle of a production run. That is fine if you are running OS X, but very bad if you end up with a part with no Linux drivers. I ended up with a DVD drive that can't even read data from out of region disks and a wireless card revision with flaky support when other people who bought the same model at the same time have no problems.

The M1330 laptops from Dell look pretty nice, are compact (13" screen) and are very good value. I tried one out a few weeks ago and liked the screen. Plus they come with vendor support for Ubuntu and it's not a bad idea to vote with your wallet to encourage that kind of thing :)
Personally I wouldn't risk announcing anything that's not a joke within 24 hours of April 1st...
Scraping EU public procurement contracts from the "Tenders Europa Daily" database (http://ted.europa.eu/). There's more than a million documents with each document requiring up to two requests. Been at it for several weeks with a multithreaded scraper and we're almost through. Using Solvent (simile.mit.edu/solvent/) to generate xpath expressions and HtmlAgilityPack (www.codeplex.com/htmlagilitypack) to run the xpath on the downloaded html with regexps as the topping. They're a match made in heaven (http://www.itu.dk/~friism/blog/?p=40).

The login procedure is gothic and took a lot of wiresharking to figure out. .Net has pretty good scraping-support in the WebClient and HttpWebRequest classes found in the System.Net namespace.

Will publish results soon... :-)
True comment. It had one or two good points, but not all 10.

However, I've received enough emails that don't seem to follow ANY of these suggestions that maybe someone can benefit from it :)
You can start work on your new job while that is in process, right?
I'm confused...Every IRC client I've used since the Amiga days had pretty much everything you've described (except image preview and maybe the "View Paste link generated" thing that I can't parse into anything sensible in my head). And all of them had/have a very similar UI to Campfire.

I'm not saying Campfire isn't useful (I don't know, I've never used it), but it sounds like you're basing your IRC comparison on a client that has long been lost to history. Modern (as in less than 15 years old) IRC clients have great transcript support, file transfers, and easy to use permissions (you have to have channel moderator privileges to use them, but they are definitely easy).
Unless I'm missing something, they currently aren't letting people create new accounts. So whatever free data they have (and they charge for most things, apparently) is unavailable for now anyway.
Even though it's actually a testing tool, you might have some luck with Canoo Webtest + Groovy (http://webtest.canoo.com). Webtest uses HtmlUnit which has pretty good Javascript support, and means you don't have to mess with regexps to get around the document structure, and Groovy lets you use an actual programming language rather than the awkward Ant-based syntax of Webtest. It takes some getting used to, and I haven't used it for web scraping, but it's a pretty powerful combination.
who is pg?....will be gud to have a yc in india
Merb and DataMapper Book (Part 1)
Anybody know a source for free (legally distributable) stock pricing?
A watched kettle never boils.
so, that fear some people had that all the software jobs were going to india? i guess we can all rest easy now.
Right, but titles never have been completely about accuracy, have they? I save my precise, rational arguments for after my title has them fired up enough to click on the link :)
Thinkpad T42 and R51 both with OpenSuSE
Fire and Motion - Leadership Strategies - Joel on Software
http://en.wikipedia.org/wiki/Paul_Graham
Actually, no. You don't get a single grace day as somebody else has pointed out below. Plus, even if these guys applied and incorporated immediately, they can't (i.e., the INS will not approve it w.h.p.) sponsor H-1s from their own company (even an H-1 transfer) unless the company receives significant investment. All-in-all, it's certainly a big show-stopper.
I couldn't agree more with this. You could also look into VirtualBox as a virtualization solution.

You might not want to choose this route if you want to use X windows because personally, switching back and forth between desktops is just plain annoying (and last I checked, Seamless mode only works with Windows). Also, if you're going to be testing hardware on linux, you'll probably be better off getting a new machine. But for all other cases, virtualization is the way to go.

My current setup is a Macbook with an Ubuntu 7.10 server VM for development (nothing beats apt).
Engelbart's Dilemma: Design for Mass Adoption and You Get a Kazoo Not a Violin
patience is a virtue and a necessity in our trade.
It's so true. Apparently 'a refreshed inbox never changes' also holds true.
For Canadians, the quickest method is the TN Visa (http://en.wikipedia.org/wiki/TN_status).
Meh, call me when there's heads in a jar...
well reporting it to the person hosting the spoof would help. usually they'll take it down.
Still equating OSS to anarchism & altruism, I see. The lessons of capitalism vs communism must have been very hard because lots of otherwise intelligent people can't frame OSS any other way.
I'm interested in this myself, if only for coolness as SwellJoe mentioned. The comments on there mentioned CouchDB, does anyone know if it's worth using yet?
Firms that compiled the data should sell it at whatever price the market is willing to pay. Just because the value of information tends toward zero, does not mean that the price should.
To make the waiting easier, I assume it's going to be after midnight. That way, insanity is held at bay.
Interesting article. I really want to try one of the 5 finger keyboards.
This thing is oooooold, but useful ;)
Lenovo Thinkpad T61 widescreen.

Pros: biometric scanner is cool. Good performance. Not too heavy for being a large, wide machine (certainly no Air). Good for mobile coding and design work when coupled with a widescreen monitor as a secondary display.

Cons: was unable to get mine with non-integrated graphics = suck. Vista device support was good, but had to update driver on wireless which was totally non-operational as shipped. Vista itself might be considered a con.

I installed Wubi at one point and everything worked with Ubuntu from the get go.
Don't needed? Even the book of Founders at Work is full of stories where the problem was scaling (delicious, paypal, blogger, ...). Every startup that's going to work well is going to have this kind of scalability problem (and it is most of the time related to DB and not HTTP front-end side).
So how did you guys do it?
Late tonight probably.
You left out Awesomenist
I'm partial to the X-series. I have an old IBM (pre-Lenovo) Thinkpad x31 and it has served me well for several years. I can't speak to the quality of Lenovos but I've been hearing good things about the X-series.

That said, I don't see why you would need a new laptop just for Linux.
My recommendation: go out and get a cool drink from some hippy juice bar, relax, and then go to bed before midnight. Read a book or watch a movie instead of checking until tomorrow morning. (I'm not being sarcastic; that's what I'm planning on doing. Blueberry-orange-pineapple smoothie ahoy!)

Finding out about the results sooner won't alter them, and the anticipation/worry is almost always worse than the reveal of whatever you're waiting for. If you resolve not to check until tomorrow morning, you'll have a more pleasant night, and be well rested to find out what's up in the morning.
Firebug can be helpful for finding elements you want to regex on!
Blogging your way into graduate school at MIT.