skysight-inc/HackerNews huggingface dataset:
The "Oh Crap" Moment: How do you know how much you know? 
with apologies to Kathy Sierra...

http://joshwand.com/files/ohcrap2.png

I've had this experience with several of my startup ideas now... I learn more and more about a given problem space/market, and eventually I start feeling like I have a decent grasp of the situation. The new learning tapers off, until one day, I discover a whole new dimension I've never thought of, and I realize just how little I know about the problem.

I've come to expect a few of these "oh crap" moments early on, but I wonder what conclusions I should draw from these moments later on in the process? They tend to be rather discouraging, as plans need to be modified/rethought, and sometimes the viability of the entire project comes into question, as when you discover that there are a whole slew of other startups trying to solve the same problem as you, all of them further along than you are!

I wonder if sometimes it's worth ignoring the extra information entirely, and solving a simpler version of the problem instead? The extra information might not even be relevant if the project doesn't expand in that particular direction. EDIT: this doesn't apply as easily to the situation where you discover your competition...

Does anyone else experience these moments, and what have you learned from them?

UPDATE: added axes/legend to graph
Maybe. It also borrows alot from "Teaching Software Engineering" (linked to in the article). However, I think it is very useful and has legitimacy because it's backed by experience. It's not the typical spew from 90% of the blogs out there.
Stuff like this makes my brain hurt. Ok, so you wrote a post that essentially yells at people, good job. What value does this add? what problem have you solved here? Let's keep hacker news to submissions that are relatively content rich and engaging.
I think the replacement for the high school system should be based on similar principles as the chess clubs. I don't know if there is such a thing in the US so let me tell you a few words about how chess clubs worked in Poland (and I assume in all countries of the soviet block).

First of all, they were open to any child. I was never any good at chess, but I had this friend who was quite good so I wanted to be like him so I decided to enroll. I showed up and declared that I wanted to become a chess player. They told me to play a game against my friend to asses my level (I remember at the same time there was another kid who came without a friend, so they made him play against a computer; the chess computer was big, easy to beat and manufactured in East Germany). Then they assigned me to a group. The number of children in my group was varying between three and five during my time there. Each group had a coach. Our coach was an international master and former champion of Poland--it's a bit like assigning university professors to teach you fractions; but that's how it was. We met twice a week and our coach would teach us all the important stuff, openings, endgames, combinations, strategy. He would analyze our games with us and give us homework. He would tell us what to concentrate on and gave specific tips on the effective practicing. The club had a nice library of chess books. It also organized tournaments. And, the clubs were competing against one another.

It was fun. Not all kids were good players, but we were passionate about it, we would play blitz tournaments, discuss various topics and argue about our favorite openings (which felt a lot like arguing about programming languages, now that I think about it). The fact that we got to meet other kids who played chess made us more excited about it. The whole experience was wonderful on many levels and, while not all of us made much progress, some did.

I imagine how great it would be if instead of compulsory schooling there was a system of math clubs, hacking clubs, physics clubs, chemistry clubs and so on. I don't think it would cost more than the current system.

Now, you may object that since this system would not involve coercion, many children wouldn't enroll to any clubs and they would remain terribly uneducated. I don't think in reality that would cause much difference. Let's do a thought experiment. Imagine you approach random 100 people in your city and ask them to complete two tests: one on chemistry and one on chess. Both tests would cover very basic questions as well as more advanced (of course, the decisions which questions represent the same level of difficulty is arbitrary, but let's skip over it and say we are using some common sense approach). What do you think the results would be? Would more people know the answer to "How does a Knight move?" then to "What is the valency of carbon?"? How about, say, King's Indian Defense vs. nucleophilic substitution? Personally, I think that on average people would score better on the chess test. But I have never really performed such experiment, so I may be very wrong here.
> To maintain intellectual honesty and consistency, MIT should announce that it would henceforth stop requiring formal credentials in evaluating candidates for this and other similar jobs. In other words, future candidates like her, who feel confident in their ability to perform the job, shouldn't feel the need to invent degrees on their resumes. Come on, you may say, how are they supposed to find out who is a good candidate and who is bad. Well, they hired her based on an invented degree, didn't they? Didn't she work out OK for 28 years? Then why pretend that the degree was actually needed in order for her to perform her job?

Exactly. I do have an answer for that, however: If they didn't require degrees for the job, they would undermine their own degree-selling business.

A majority of college students are in college because a degree is supposed to help them get some unspecified job, sometime later, out of a job pool which purports to pay better on average than the jobs that let you in based on mere ability.

In effect, colleges have taken on some of the characteristics of shady employment and modelling agencies, the ones that tell candidates to pay a fee and they'll help them get placed. We always hear how the legitimate ones don't charge fees in advance.

If college was only about teaching people what they desired to learn, enrollment might drop by an order of magnitude or more. Like any business, it wants to grow, and so they find ways to market themselves ("you need a degree to get a good job!") and arrange guaranteed contracts for themselves (encouraging you to take on debt) to further their own interests, not yours. Hence they tack on mandatory fees for all sorts of things you may not want and they establish mandatory requirements in quantity of "units" (as if 3 "units" of English or Computer Science makes any sense). Required classes are essentially a jobs program for professors of the less-popular subjects, guaranteeing enough sign-ups to keep the money rolling in for people who would otherwise be unemployable in that capacity.

You can learn most academic subjects out of the book. Having someone read the book at you and write a few phrases from the book on a chalkboard doesn't add information. All it takes is a tiny bit of initiative on your part to learn it on your own without someone else telling you to do it.

If you don't have that initiative, it's probably because you spent K through 12 always waiting to be told what to do by someone else. The usual classroom method is generally inefficient and it purposely stops you from developing self-direction. Why? Because a good employee needs to do what s/he's told and take direction from a "superior". They don't want you to be self-directed -- how's a megacorp going to exist if there're a million little start-ups running around?
The most interesting article I read from PG the past year.

Holding as much of the program in your head gives you the power to identify what will give you the best optimization.
Also this may be the short explanation why good software is written by teams of max 2-3 people!
Yes, I get these too. Pretty much every time I talk to someone who takes an interest in my startup, they raise some issue that I either hadn't thought of at all or hadn't thought about in the same way. That doesn't even cover the hacking-related 'oh crap' moments, but most of mine have been conceptual so far. It's to be expected: there are tens of startups in my space, I assume that some are technically better and many are worse.

As you point out, the hardest thing may not be the 'oh crap' moment but rather what conclusions you should draw from it, if any. In most cases I have elected to go simpler when in doubt. Other times I've decided to re-think various aspects of my project, but these are in the minority.
I honestly hope I will never have to worry about stuff like this... Ugh.
An Independent Journalist solves a 2400-year-old Mystery
I. F. Stone was an amazing guy.
"I wonder if sometimes it's worth ignoring the extra information entirely, and solving a simpler version of the problem instead?"

Yes. Always.
You may well be on to something there - I think this would do very well as a facebook app.
Do you guys have any ideas on how to market / promote something like this?
I've generally found that the best solution to a problem is also the simplest. If the complexity of your solution starts to grow more than linearly as you work on it, if it starts to look like the only way to finish it is through brute-force labour, then I've found that backtracking to right before things start to get complicated usually yields an "aha!" moment. A simple, elegant solution will be easier to implement and maintain, more fun and satisfying to work on, and will most likely even solve the problem better.

You don't always have that luxury, of course - some problems don't necessarily have a simple solution. But for me, when things get complicated I take it as a sign that I've taken a wrong turn somewhere.

I'm not sure if this actually answers your question. ;)
Obviously Greenspun's a webapp veteran. That doesn't mean his class doesn't need advertisement. As I alluded to in the previous post, it's not particularly popular at MIT. And to my knowledge, it's not a class like SICP that's been taken up at universities around the world.

I think at best Greenspun's inclusion of those factoids was meant to imply some causation (that his class helps people get jobs). Maybe he was just name checking for the fun of it. Either way, I suspect most of the alums who took his class and then went on to get jobs at GOOG and MSFT would have been able to without it.
very 1960's when business was uptight, formal. Guess they needed to loosen up a bit.
>+++++++++[<+++++++++>-]<+++.>+++++[<++++++>-]<.+++.>++++[<---->-]<.>
++++++++[<-------->-]<-----.>++++++++[<++++++++>-]<++++.---.>++++[<++
++>-]<+++.>+++++++++[<--------->-]<---.>+++[<++++>-]<+..>+++[<---->-]
<-.>++++++[<++++++>-]<.>+++[<+++>-]<++.>++[<+++>-]<.>+++[<++++>-]<+.>
++++[<----->-]<--.>+++++[<+++++>-]<.>++++++++[<-------->-]<-----.>+++
++++++[<+++++++++>-]<+++.>+++++[<------>-]<----.>+++++[<+++++++>-]<.>
++++++[<-------->-]<.>++++++[<------>-]<.>++++[<----->-]<---.
The kind that turns lead into gold. You should really study these things.
Oh snap. You sneaky sneaky. We might just have to launch tonight instead of tomorrow. Well played!

I really like how you've kept it drop-dead simple. Our site offers a little more social-wise, so I'm sure there will be plenty of people who prefer your application. Great design!
From footnote 3 here:
http://www.gigamonkeys.com/book/introduction-why-lisp.html

Psychologists have identified a state of mind called flow in which we're capable of incredible concentration and productivity. The importance of flow to programming has been recognized for nearly two decades since it was discussed in the classic book about human factors in programming Peopleware: Productive Projects and Teams by Tom DeMarco and Timothy Lister (Dorset House, 1987). The two key facts about flow are that it takes around 15 minutes to get into a state of flow and that even brief interruptions can break you right out of it, requiring another 15-minute immersion to reenter. DeMarco and Lister, like most subsequent authors, concerned themselves mostly with flow-destroying interruptions such as ringing telephones and inopportune visits from the boss. Less frequently considered but probably just as important to programmers are the interruptions caused by our tools. Languages that require, for instance, a lengthy compilation before you can try your latest code can be just as inimical to flow as a noisy phone or a nosy boss. So, one way to look at Lisp is as a language designed to keep you in a state of flow.
It's working now. Be warned, tho, I did it a few hours ago, and it took a ridiculously long time to load between questions (sometimes up to nearly a minute).
I've gotten completely lost following the competition who has been trying to solve the same problem. I've realized their take and solutions will be different from mine. What I do now is write down every day the problem in one simple sentence .. e.g. "I hate traveling and every wifi is password protected!" I keep working the solution, knowing it will grow in complexity because i have to deconstruct it in order to work toward simplifying it. I then create a solution using a three step function for myself and other users. This has been helping me, especially when I get overwhelmed in thinking about more features and functions.
Youtube's adoption of the video overlay will be good for its competitors
Youtube/Google has the volume of video streams and the relationships with advertisers to help push for an ad unit standard for online video advertising that will benefit all players
Pretty nifty, I must have missed the initial post. Now I have something to motivate me to workout daily. And something to haunt me when I consider marking my alcohol intake (in this case, you don't want to start a chain).
Facebook has extensive dossiers that its users unwisely volunteered and I predict it is all going to end in tears.

John Connor is not a Facebook user.
Maybe including names for the axes will made a better graph
I guess Katty Sierra is happy when people use her style.

This may be a nice Creating Passionate Users post.
;)
I like this idea as well, especially as a facebook app. Use the Facebook api to scrape together an initial group of events pertinent to an individual user and have them mold it from there.

Problem:
Right now when I want to know when something happens (new cd/movie release) i go to a reliable source and I know with some degree of certianty that that event will happen when they say it will. I give them some reasonable leeway since its still a future event with numerous opportunities to miss the date(later ship date, cancellation, etc.). However, when you tell someone an event just happened or is happening now, you better be 90% correct or the tool becomes an annoyance instead of useful.
better?

http://joshwand.com/files/ohcrap2.png
That makes an interesting point: the poster would maybe get more a varied response if he asked "how do you get ideas" instead.

One heuristic that may help find different ideas is to look for challengeable assumptions in your current ones.

If you don't believe in brainstorming, what works best for you?

[Edit: See section 2 on brainstorming here; very interesting:

http://www.gamasutra.com/features/20051026/gabler_pfv.htm ]
Manager Tools (a business podcast) has a very good podcast about this for those interested:

http://www.manager-tools.com/2006/09/meal-etiquette-part-1-o...

I know it may appear boring, but they're actually really good and I listen to them even though I'm a student.
Yes, I've noticed the same thing. My current idea is similar to several other start-ups. At times it seems like some companies could be competitors if only they changed their product offering a bit.

I think this is where the business/marketing side of the founders needs to kick in and be determined to differentiate and be better than what's out there.

It's all about providing value for a customer. You either need to provide a better value or have different customers. IMHO, a simpler version can be both good and bad.

Good, if you can start getting revenue and grow faster. Bad if it doesn't generate much revenue but slows you down with customer support and alerts your better funded competitors to your existence and plans.
I like the iGoogle widget, this is something I could actually use.
cool. did you still lose the karma after you deleted the posts? I can give you six points on other posts
Imagine the WTO lets Antigua disregard (it's not a violation if it's legal) intellectual property rights. That might be a nail in the coffin for the movie industry.
Agreed. You bring up an interesting point about social pressures b/c they are the driving force behind this whole thing. I'd like to run w/ this for a sec...

It is NOT socially acceptable to not attend college. It IS socially acceptable to not get funding for your startup. So what happens when someone wants to go to college but they don't have the money? They get loans, they work two jobs, they do what they have to do to make it happen b/c they will be socially rejected if they fail. The same is not the case for funding. If you are committed to your idea, treat funding like a college loan: Not optional. Once you get in this mindset you'll figure it out. You may not get a full ride at MIT (YC) but in state tuition + a TA job at U Mich isn't so bad either.

As of now nobody has commented on a sucessful YC reject. I think that is b/c YC applicants tend to be straight A students (blind generalization based on instinct.) Straight A students have other options that involve less social risk. I know of a number of successful entrepreneurs who were thrown out of / barely made it through college. Since they didn't have other options, they HAD to make it work. And since they had been failing their whole lives they've realized that failure is just someone else's or some organizations opionion about them. For good or for bad, they tuned those opionions out.

People need to realize that the currency of "failure" is just a bit of social rejection. Running your startup into the ground will not give you cancer. And a big fat 'F' on life's report card is just a sign that at least once, you weren't afraid to push the limits...
I think this is the class that inspired the infamous "Java is the SUV of programming languages" entry that got slashdotted a few years ago. Phil said that the students who picked J2EE all had to drop the class - the workload was too great for them to keep up.

At first, I was surprised to hear that MIT students were having trouble completing a program in Java, until I saw what was required and the time they'd have to complete it. I wasn't sure if I could do it in the time alloted, even without carrying a full courseload, and I knew java reasonably well at the time.

This post, along with Paul Graham's comments about Lisp, really changed the way I think about programming. Kind of like the JFK statement about seeing things as they never were, and asking yourself "why not?"

Why the hell should it take so long to go from an idea to an implementation. I was just accepting the achingly boring typing that Java had inflicted on my life as part of the cost of writing a program. Time to realize that I can't implement a chat server in half a day, and ask myself "why not?"

Anyway, I think that even the students who have to drop this course learn a valuable lesson, as long as they struggle with it for a while.
Ah, datablogging. I did something similar back when I was jobhunting, to track how many resumes went out (and how many responses received):

http://joshwand.com/php/jobsearch2.php
Time to buy an IR laser to really fool all but the most observant.
I suppose aiming might be hard, but getting a camera that sees low res IR shouldn't be too hard.

Some IR lasers
http://www.nvginc.com/irlasmod.htm
Yes, keeping a journal is key. The habit of acting upon your ideas- even if it's something as simple as writing them down-, stimulates your brain to give you more. Also, you can stimulate your creativity and get new ideas by trying to combine existing ones at random. It's harder to escape the block if you are in front of a blank page.
Speaking of Wolfram...

He gave a great talk on simplicity in nature's algorithms at MIT. I could see a lot of application to programming with his take on simplicity vs. complexity.

http://mitworld.mit.edu/play/147/

Also, he spoke at Startup School in 2005 - really excellent talk you should definitely check out if you get a chance.

http://feeds.feedburner.com/~r/Ycombinator-StartupSchool/~3/...

(For anyone new to the site, you can get all the available Startup School talk mp3's at this podcast: http://feeds.feedburner.com/ycombinator-startupschool )
Nice.

Feature request: could you offer more than one chain? Chains that aren't daily (e.g. if you're in sales, it wouldn't make much sense to expect the same quota for Wednesday as for Sunday)?
I signed an NDA so I can't talk about his project, but we nixed the non-compete clause (it was so broad it was unbelievable) so if he fails and closes up shop, I might look into running with the idea.
>the old code was essentially two years of hacks and cludges.

That basically describes every piece of software ever written. It's not always a justifiable excuse to re-write the codebase though.
LOVE the simplicity! Great job! And extra kudos for the iGoogle widget!
The programmers below, are white American - so what does that say?

see http://news.ycombinator.com/item?id=36900
LikeBetter Use Case:  Design a T-Shirt (Xobni)
Actually, the fact that they're working on a new system supports the idea that they aren't "making money". You don't fix what isn't broken.

Of course, it seems logical that if they do build this system, they may make a good deal from it.
we'll be more judicious about what to submit considering the changing focus of the site.
>If they didn't require degrees for the job, they would undermine their own degree-selling business.

Ding Ding Ding!

When we all realize that in a large way, higher education is a sham, we'll all be better off.
Facebook Integrates Book Swap Feature into Marketplace
Interview With Facebook's Joe Hewitt, iPhone God
I'm coasting along now with a 2.8 GPA and a willingness to let it slip to a 2.0. I'm an industrial engineering student and you may remember us as the efficiency experts from movies such as Office Space and American Beauty. The salary for an Industrial Engineer in Pittsburgh is almost constant for anyone with a QPA over 2.0. Those with better grades just get to work for a more nationally known firm instead of a local manufacturer.

College is so aversive to me because all it does is prepare me to work for someone else, at a guaranteed salary of 55k per year (sixty by the time I graduate?). True failure for me will be taking a job as an Industrial Engineer. I can't think of anything I want to do less. But if I haven't become successful by the time I graduate from college and no one is funding me, I won't have much of a choice.

I'm running my current startup like a business. We're not seeking funding and we're growing very slowly. We have no costs so everything we make is profit. We haven't made any profit because we haven't actually sold anything but we'll get there eventually.
IMHO, poor thinking. (sorry)

how will you prioritise and track them - open up all the files?
Hmm, that could be the motto: "EventKnower- It's better than Truemors!" :-)
This is quite nice, i like the look that it has on iGoogle.

P.S does anyone have a link to info on creating widgets for igoogle.
A 92 mpg car that does zero to sixty in 5 sec