skysight-inc/HackerNews huggingface dataset:
I have to disagree. I think Arrington's got the direction right - whether this is the killer app or not is left to be seen.

Its also not about the iPhone but about what the iPhone represents - a whole new way of making the web and networking personal and portable. Right now the iPhone is the platform that is currently deployed that really gets it, but Google's Android platform and the Nokia Maemo platforms are right behind. In fact Android may not be as pretty but its direct to the middleware access may actually may be a platform thats more accessible for developers.

of course I'm biased my YC application is directly in this space ;-) but I wouldn't be doing it if I didn't think its the real Web 3.0
"Problem is I don't have a blog, I don't have friends who are bloggers"

I think the first thing you would need to do is start a blog. It's the whole 'eat your own dogfood' thing. If you had a blog you could do linking agreements with other bloggers to spread the word, and you would be able to better test your service.
Lets not stretch our imaginations too much. We are trying to provide help to hackers with no selfish motives !

With all due respect, our aim is not to belittle the hackers - we respect them and the whole spirit behind the program is to actually encourage them to start-up. Our interaction with a lot of tech start-ups (by tech i don't just mean IT) suggests that often the failure is rarely in development of product - it's often in the ability to understand the market and to penetrate it. We might be mistaken, but we feel that IIMA grads can do a good job on that. A good techie+MBA make a better team than an IIMA grad alone or a techie alone does.

Equity: We are a not-for-profit entity and don't plan to take equity at this stage - if we decide to provide additional funding to these start-ups we might.

Stipend: Yes, India is cheap. However our target audience is students and young IT professionals - the stipend is a sweetener. You get to work on your idea without having to think about how your expenses get paid. (Indian students typically don't take up RA/TAship in their universities and depend heavily on parental income for their education. So we are trying to provide them with a tension-free environment! Ahh, some would still believe it is a jail uh!!)

YC Model: We can't be a YC - if we are trying to be we are probably mistaken. Our aim is to be a group which is trying to make a difference to say the least. There have been no Google or Skype coming out of India - but there have been the Naukri.com and the Makemytrip.coms which are some of the more successful names in the Indian dot-com scenario. If you think that the founders of these can add some value, yes mentors would be there. We'll have regular mentoring and weekly networking dinners where you should expect a few successful entrepreneurs/VCs/investors to be around. We are beginners in this and would love to learn on how we could do this better.

IIM/Gujarat - the right place: Its at least a start to make the YCombinator culture more popular in India. Hopefully someone else - in the right city and right environment would do a better job. We'd love to see that happen. Not too sure if IIM is the best place - what else is? Are they ready to do it? We've got the resources to help the budding entrepreneurs - so we are trying. Gujarat, may be quite entrepreneurial, but there are very few hackers around. So we don't limit ourselves to Gujarat.
Digg had 5 million plus monthly uniques 4 months after launch. And probably double by the Series A, 9 months in. As I recall anyway.
Upmod * 10 if I could

If you don't have a blog, or know any bloggers, how do you know this app solves a need?

Dogfood it!
> It does seem like a stretch to call it open if they charge money and only provide an API.

I agree. It doesn't look like much of the data is available for free:

http://www.opentick.com/index.php?app=content&event=mark...
This article supposes a competition. Why is it a competition if the essence of creating a startup is creating wealth? If more companies get off the ground, then more total wealth will be created. The more startups that are created, then the more total wealth will be created. Startups are the antithesis of a zero sum game.

And, if YCombinator is starting startups, then it is a wealth creation machine. How will that be "beat" or competed against?

Also, I don't get the impression from PG that YC is out to dominate the startup market. I do get the impression that they started YC because it was an irresistible hack. That is, they set out to hack the economics of startup creation. And, if another entity comes out and has a greater success ratio, or greater volume of startups created, then I daresay that it is an ultimate success of PG and crew. Why?

Because the hack worked.
totally! - impressive pr pick up of this story though
Doom And Gloom Hits Silicon Valley
"For example, it returned false for Montaigne, who was arguably the inventor of the essay."
The essay was invented by the Stoic philosopher Seneca.
This can be very useful. I use pyparsing with custom python code for scraping.
The flamming naked girls are the point.
Yum, declarative!

I can't say I'm crazy about the syntax, but I'll give this a try when I get home.
Google's App Engine: Aiming At Facebook, Not Amazon
web scrapping with beautiful soup (is this old school already?) - parsing the Sydney Future Exchange for data back in 2003 (still running)
Check these out:
hypertable.org
cleversafe.org
FOSS Factory: community collaboration for FOSS (funding/design/etc)
That sounds like an interesting idea, but I'm thinking that the line between success and failure is composed of luck and interpersonal relationships.

Most of the stories in FaW involve meeting a key person, an important external event or being in the right place at the right time. It's very hard for someone to say "Well if we had talked more with this one person, I think that we would have stumbled into a great situation a few years later"

Likewise I've seen a lot of situations where personal relationships make the difference. Two founders disagree and one of them just has to be "right", even if there is evidence that their plan isn't best for the company.

So maybe you can work out the characteristics of failed startups by looking at the inverse of the characteristics of successful startups. I'm thinking that they would be: don't be committed to the product, don't explore every relationship and don't have an open mind about new opportunities.
Well, we run into a similar consideration with our feedback widget we recently released on featurelist.org:
http://featurelist.org/info/features

It won't always remain so, but at the moment, the average spambot isn't processing javascript or iframe content to populate spam forms. For now it seems the spammers are content to submit to open/typical HTML forms.

To compare some of the spam levels we get on our site bug.gd:

* We have open email addresses on our site. These are spammed insanely.

* We have a feedback widget (powered by featurelist). This has never been spammed and is getting us great user data.

* We have basic HTML search forms, comment forms, and solution forms. These are spammed 10 or more times a day.

* Our blog comments receive a good bit of spam each week.

All in all, widgets are pretty resistant at the moment. (Please no one here start a company to do this. Of course someone will.)
Getting visas to attend programs like YC
Hi all,
A group of us were interested to apply for the YC program in the summer but were detered because of visa issues.

All three members in the group currently have full-time jobs and two of us are on H1-B visas. The third member of the team is a U.S. citizen. Quitting our jobs to attend YC fulltime is therefore difficult as two of us will lose our H1-B status and will not be able to stay in the country.

I read on the YC website that people from other countries have been able to participate in YC before. I was wondering if anyone knows anything about this? What visa status were they on? Is there a temporary business visa available for this sort of situation?

Also, if you are facing similar problems, feel free to comment.

Thanks
I always do custom stuff in beautiful soup, but this looks somewhat cool.

Maybe have it so you can edit the sample text and language and see the results all on a web page?
I use BeautifulSoup when needed for simple scraping.

My biggest frustrations, right now, are really around getting data from lots of different websites in subtly varied forms. This is a tough problem to automate. I certainly haven't found any tools that make it simple.

I'd be happy with a 50% correctness rate, looking for very loose patterns. I just haven't found a tool and, while I have some ideas for how to do it, it's a major project in itself to produce something that can do this.

For example, imagine writing a scraper that would parse out every food recipe online. Whether it be in forums, blogs, etc, etc. That's the sort of scraping I'm looking for and the best I'd have is putting together a neural network or other system that I can train against human-provided data. Unfortunately getting such a system to partition the text to just the recipe would be difficult.
This is classic FUD: "we're working on it", "it'll be quite a bit better". It's only relevant when it's released. In the meantime, let's work with something that actually exists.
Thats an honest question. And I won't deny that I fall into that category. I guess since I read a ton of blogs, I made from the point of view of a reader who might like to interact with bloggers in a fashion different them email thats more open. Aswell I thought in the fashion of opennes seeing other people tipping off their blogger to stories and getting responses might encourage me to share tips as well.
This sounds redundant already, but I scrape using beautiful soup. Right now I'm scraping a lot of news sites and feeds for a project I am working on.
I think that's a great viewpoint to take however if you're going to now sell bloggers on this idea you need to also share their viewpoint on the product.
Except that pg was talking about the real Steve Jobs, not the fake one.
and what i'm saying is how much traffic and growth matters is almost totally dependent upon your own business model.

my latest startup only gets about 100 unique visitors a month. its an expensive, private, pay-for-use application directed towards a very niche market. i'm doing well. will your business be doing well at 100 uniques? probably not. i've had almost consistent 0% monthly growth. thats partly because we're still in beta and am not advertising it at all, but i'm still "successful" already.

success is defined by your business plan and how you monetize, not by traffic alone.
I guess I live without these all the time on my flavor of app framework, so I guess I find these as "nice-to-haves".
It's hard to see how, but I'm clearly not his target audience.
So along the lines of Skribit and Outbrain?
There's an open source project attempting to recreate GWT in python: http://code.google.com/p/pyjamas/
I remember seeing this company that organized data... oh yeah, they also wrote most of my textbook and aquired Reuters:

http://en.wikipedia.org/wiki/The_Thomson_Corporation

Could they really end up being the Encarta Encyclopedia of data?
Google Says Sorry! You Search Like A Bot
Yeah, like socialpicks there are many others but my idea is not about stocks. It is another type of investment.
Coding: Generalists vs. specialists
Historical data should be available for free. The money you pay for real-time data is the exchange fee- ie it is paid to the NASDAQ, NYSE, or whoever. The exchanges do not allow real-time data to be offered for free.
Ahhhh, thanks for a final word of appreciation. I was all of a sudden beginning to think that we were doing a grave mistake by trying to be innovative about promoting innovation. :)
Fake Steve is a troll, and a bad one at that. His posts are childish, but that's not the real problem with them. They consistently lack any real insight. Any exposure he receives is just due to his immature writing.

Can we now accept that he only got people's attention using classic trolling techniques and go back to ignoring him?
", especially your point about Bangalore being an expensive city."

I think I should have been clearer. As far as I know Bangalore is either the most expensive or second most expensive city In India. (reliable statistics are hard to come by).

To clarify.

Bangalore is expensive in terms of real estate / housing. I've been looking to buy a house and I was quoted close to 2000 $ per square foot for a good apartment ! Insane!

Hotel rooms are booked months in advance and very expensive.

Food /clothing etc is cheaper than in the USA , (though more expensive than almost anywhere else in India (excluding Bombay, perhaps). Traffic is a total snarl up at peak hours and power and water supplies are creaking (but holding up, barely, so far). The more urban parts of Bangalore have good internet connectivity (about35 $ a month for an always on , unlimited download connection).

All that said, I wouldn't live anywhere else (in India). Bangalore is no Silicon Valley , but what genuine technical talent exists in India is right here (though submerged in the crowd of those who barely know how to write "hello world" , but get "Senior Architect" type jobs in the large outsourcing firms!!).
My favorite Gary Vaynerchuk quote: “Most people in the wine business are douche bags.”
Dupe: http://news.ycombinator.com/item?id=157785
I used to get this pretty regularly when trying to access a cached version of a page.
Google should just release the source of the chat app.

Zed, dude, will you ever chill out?.
"YC sounded pretty ridiculous in 2005 when pg announced it"

That's very honest of him! I guess that just shows that you can't accurately judge a new idea a priori.
Where to Find Open Data on the Web
Including Rescuetime and 8aweek in our content for promotion of the program was our attempt to highlight successful YC innovations. Believe me in India very few people (including VCs) know about YC - we'd hope more people get to know about YC. Not the clones, but the original YC.
I digest a lot of inspirational/motivational stuff on a regular basis and most of it leaves an empty feeling - like the person writing it is inauthentic because no one is superman but it feels like Gary Vaynerchuk is talking to me. In terms of News.YC, he's telling us to build something that will make a niche group of users happy - don't try to go after too big a market - make a small group of users happy and you will excel...

(Felt the need to transcribe one of his 3 min video blogs in entirety just now - apologies for the length of the comment)

Not Lying to Yourself

"not lying to yourself is really the key to any social media, life, business, family, anything

i think what's going on right now is so many people are lying to themselves

they're saying well i'm going to make the next Digg, or I'm going to make the next Facebook, or I'm going to be the next Oprah or I want to be the next Ze Frank

what you really need to do is to embrace who you are. it's a DNA game. you need to know who you are. and uh really that's why some of the things are working for me

i stopped doing what i was doing and I said i need to do something else for my heart and soul. I stopped lying to myself (not that I've ever been a big liar to myself) but I think a lot of people are lying - they think they can do things they can't or more importantly they think they can't do things they can.

and i think going forward, in this age of personal brand, where the barrier of entry to have a personal brand in the world is now zero - uh the people who are going to be able to stop lying to themselves and embracing what they're about and not being worried if the niche is too small are going to be the people that are going to excel and win and win in heart -

for example, i think too many people are trying to keep apps that are so huge or trying to develop brands that are so huge or trying to say they are experts in such a global thing. narrow it down to the thing that you are most core about. if you are the biggest expert in basketball, fine do a video blog on basketball if that's what you want. but more importantly if you're a bigger Clippers expert maybe that's what you should talk about. even if there's 45 fans that you have in the world. those 45 fans will bring you the happiness that someone else might have if there a million. not everyone is going to be Oprah, not everyone's going to be LeBron James. but trust me there's a lot of ways to be very very happy if you're just yourself. yourself is the victory in this social media game and in this world and with no privacy and personal brand becoming bigger and bigger every day

the people that stop lying to themselves are going to be the people that the world embraces"
not quite the same, but I can see some similarities with skribit
We're in Canada and we're wondering the same thing. People here we've talked to don't know much about it, but we're assuming there must be something available since YC has funded Canadians before I'm pretty sure. I figure they'll let everyone know when they're accepted what forms need to be filled out.
There is also Swivel: http://www.swivel.com/
Sharecropping in the Orchard (2004)
* I dont imagine the more complex functionality google will introduce will align with the current standards either*

Probably not, but again, you don't necessarily have to use the bits that lock you in :-)

Nothing new under the sun here...
I'm working on something similar. Turns out scrapping a small part of the problem. I don't use beautifulsoup. Turns out you can transform html of a page into a list, which can easily be scrapped.

Now that I have used it to extract data out of many different types of pages. I'm looking to turn it into a dsl. So that the code looks natural. Currently it's just functions which search for tags in html. You can then easily filter some or others. here is an example

(extract-all page [(and (tagp _ :a) (classp _ "jdtd4"))])
http://shareit.com consolidates submissions to social bookmarking.
http://mashable.com is a popular web 2.0 news site.

Develop a "press release" and submit it to all the valley news sites you can... including here.
he uses some troll-like techniques, to be sure, but i still enjoy the writing, most of the time. he's been able to get at the truth from an oblique angle that nobody else has tried before. fake steve has real admiration for the real steve, which pokes through sometimes.
I summarize Y Combinator’s selection criteria as: “Young, Smart, Cheap, Determined, Acquirable”.

Smart, cheap, and determined, yes. Young doesn't matter except insofar as it helps cheap (and sometimes determined).

The point about being "acquirable" suggests a misunderstanding of how startups work. Startups almost invariably take one of 3 routes: (a) failure, (b) acquisition, or (c) IPO. Every one that takes (c) gets acquisition offers along the way, but turns them down. So "acquirable" simply reduces to "succesful." Which of course investors want.

And once you have that in the list, the entire list simplifies to "successful."
Actually, MLB data is fairly close to being open-sourced. Historical data IS open-sourced (though not by MLB itself):

http://baseball1.com/content/view/57/82/
http://retrosheet.org/

Current major and minor league data is available as well, though MLB will crack down on anyone who is trying to make money off of derivative products. Here's where you'll find it, as XML:

http://gdx.mlb.com/components/game/

One can do pretty cool stuff with all of it, and many people have, despite the fact that we can't make money off of it:

http://minorleaguesplits.com/ (my site)

http://baseball.bornbybits.com/2008/pitchers.html (analysis based on detailed pitch speed / break information that MLB started collecting last year.)
How does a software company make money, if all software is free?
I've been using C# with the HTMLAgilityPack. Probably not as fast as it could be, but C# is what I know best.
The author doesn't quite get, or fails to mention, that many companies that use Linux or make money from it, like Google, IBM, Motorola, Red Hat, Trolltech, etc., invest money and programmer time back into the open source projects they get software from - and this is not exactly altruistic. Still, not a bad article.
Absolutely, my first thought on reading this was "Why won't these data companies understand they should throw away their business so us coders can make some money?"
Play with this girl in Macromedia Flash
That's not what I mean. I don't anticipate so I can skip. Contrasting what I expected with what the author actually says helps me understand and retain what I am reading better.
I don't think FSJ "consistently lacks any real insight":

http://fakesteve.blogspot.com/2008/02/ballmer-im-completely-...

http://fakesteve.blogspot.com/2007/11/this-writers-strike-cr...

Are both quite insightful, I think. Most of the time he's just going for a laugh, but there are often quite observant commentaries on the industry.
I would postulate that a funding or acquisition event depends on meeting a key person or having a key external event happen. Making something people want, however, is not dependant on any external event.

It's really not just luck. If you get a lot of traction, the chance of those fortuitous things occurring simply increases.
I share the same public IP as a search marketing firm and am ALL too familiar with this screen.

The best are the inter-office emails that get sent off from IT asking if anyone happens to be using programs that automate search research...
The realization that they just want to float through and maximize for happiness, rather than chase after things that they don't feel that they can achieve in any case?

Not for me, but I think I can see the point of view.
Either Beautiful Soup, or Yahoo Pipes... I have a website that parses RSS feeds, and some sites don't have feeds yet! Or if they do, they aren't usable. So I use Pipes to scrape a page and turn it into a feed using their regexp operator, then my site uses that feed.
Scraping HN for fnid's to auto-post xkcd comics :-P. It was a hack so I just did string searches.
From whose perspective is that a mess? From the perspective of a source control system which cannot deal with how I choose to work?

Honestly, no offense meant, but I'm starting to wonder if Subversion is the Blub of source control systems.
I do all my screen scraping with PHP, curl, and some regex. Previously I used plain PHP.

I use it to scrape television listing data (http://ktyp.com/rss/tv/ was my old site, and http://code.google.com/p/listocracy/) and more recently to scrape resume data from job posting websites for a (YC-rejected :P ) side project I'm working on.

The hardest part I've encountered with scraping is odd login and form setups. For example Monster.com uses an outside script to attempt to fool scraping. A couple other sites use bizarre redirecting across pages. Also AJAX certainly has changed the way a lot of screen scraping is done.

Finally, the most useful tool I've used is LiveHTTPHeaders (http://livehttpheaders.mozdev.org/) which is great for following how a site operates.

Edit: For PHP, another interesting tool for scraping is htmlSQL (http://www.jonasjohn.de/lab/htmlsql.htm) which allows HTML to be searched using SQL like syntax.
The M&A graph had the same shape 5 quarters ago.
Something like this is really what we need, and the thing that would be really revolutionary about the web. Otherwise, even though people talk about the information singularity and such, there isn't a really high useful information signal. Too much useless or mediocre information is worse than useless because it makes everyone more stupid.

But, this is also not only a technology solution. People have to make a choice themselves to filter and promote good information.
This article brings up the quote: "Jack of all trades, master of none", which seems ok to me.
Thanks hendler, I'll try that
They concentrate on algorithms. Good for them. That is how it should be taught.