skysight-inc/HackerNews huggingface dataset:
If your entire database fits in memory, then any decent DBMS will cache it all there, write changes back to disk at such time as to have a minimal impact on performance, all the while ensuring atomicity if someone kicks out the plug while it's in the middle of flushing the buffer. If you're not using a DBMS and you can still say all that about your own solution, you've probably put a lot of work into reinventing the wheel.
Anyone have any info on how Facebook is setup (Server infrastructure wise)?
"Passion knows no logic."
The hits:

Buchheit

Levchin

Partovi brothers

Graham

McAdoo

Recent founders panel (just the end bit where Alexis beautifully zings Paul, but you need to see the slides...I dunno if they're available)

The misses (misses because I expected a lot more from these guys, not necessarily because these were worse than the rest of the non-hits--both really smart guys with a lot of success under their belt):

Kapor (should have been half an hour, like everyone else)

Zuckerberg (age discrimination isn't really something useful to learn, as we all probably have enough of that built right in)

Though, I might say that if you do watch one of the misses, you need to watch the other. They're like Olbermann and O'Reilly squaring off on issues like hiring, firing, employee selection, etc. By the time you suffer through both talks you might have a balanced view.

From the 06 batch I vividly remember Kraus, Graham, and Fletcher being fantastic. Kraus and Fletcher are always worth listening to.
So, how do write-intensive sites like LiveJournal handle their updates? I was under the impression that LJ was all memcached + sharding + master/master pairs for the shards.

How do you handle features that rely on frequent UPDATE statements, like say a hitcounter? Is UPDATE LOW PRIORITY (MySQL) enough, or should you work out some application-specific caching/batching mechanism and perform all your updates at once?
Ensuring atomicity is actually very easy.

Databases SHOULD do these things, but I haven't seen any evidence that the popular ones do. Feel free to post actual measurements for updates per second from your db.
http://www.facebook.com/jobs.php

Looking at this you can see MySQL and PHP so I'm guessing at the highest level, a LAMP architecture. Don't know how they go about web farming, scaling though.
Most investors do not invest in competing or significantly overlapping companies. There are a few exceptions (Ron Conway), but it's rare enough that they feel like making that clear on every possible occasion.
Actually - more information here:

http://blog.facebook.com/blog.php?post=2223862130

From the blog post:

"Almost all our servers are running open-source software. Our Web servers use Linux and Apache and PHP. Our database servers run MySQL. We use memcached to help keep the site snappy. Some of our behind-the-scenes software is written in Python and Perl and Java, and we use gcc and Boost for the parts that aren't. Our developers use Subversion and git to keep track of their work. The list goes onlike many Web sites, we use it from top to bottom."
Chance favors the prepared mind - Louis Pasteur
How are you handling atomicity? Are you doing full-out undo/redo logging, or just something involving writing to a seperate file and then (atomically) retargeting a symlink?

I'm mostly an academic, so I can tell you a lot more about how things are supposed to work than about how they actually do. The only high-traffic DBMS with real users to which I can easily get access is MS-SQL, which doesn't inspire me to any leaps of faith. But once I finish building my new desktop system (with spiffy RAID array) I'll rig up some benchmarks on MySQL and PostgreSQL.
Totally understandable. I'll do my best to re-remember all the random ideas we threw down.
that's good stuff, Thanks!
Prolly a little bit of Thrift for client/server distribution.

http://blog.facebook.com/blog.php?post=2261927130
"I only have one hour of pure brilliance per week. Unfortunately I have to work 80 per week to find the one."

- Wil Schroter
Just write each transaction as a single record to your transaction log file, ensure that only whole records are replayed (like with a checksum), and you're done. Since there are no b-trees or other complicated structures on disk, the problem gets a lot easier.
Do much.
Say little.
Write nothing. (Except software)

JP Morgan
(I added the bit about software.)
Growth, innovation, scaling & pace of life in cities
The purpose of business is to create and keep a customer.
Peter Drucker
The difference between great people and everyone else is that great people create their lives actively, while everyone else is created by their lives, passively waiting to see where life takes them next. The difference between the two is the difference between living fully and just existing.
Michael Gerber
It's on my account page.
Great Startup Schools?
In preparation for a career as a startup founder, Im looking at grad schools right now for several reasons:

1)	Sharpen my skills
2)	Meet like-minded people
3)	Learn cool things

Obviously, number 1 is Stanford, the home of Google. And number 2 is MIT. Does anybody have good recommendations for numbers 3, 4, and 5?
Cheap, fast, or accurate.

Pick two.
Ok, isn't Web 2.0 a similar marketing fluff? I understand with all similarities it's not the same. I'm sure it is much better now, but add one more new factor, like the failure of Windows Vista for example, and you may have that critical mass for a new collapse.
Hmm, this is an interesting take on graduate school. I'm not convinced that graduate school is the only or best way to accomplish your goals. Why not move out to CA and join a startup?

Assuming you mean a PhD, I'll give you the following advice: If you really don't love what you're doing, graduate school hurts. Personally, it would have been hard for me to come into a program, thinking that I was going it to start a startup and not do research for the rest of my life.

A masters will certainly help your career within bigger companies, but do startups care? Does one or two more years of courses really teach you anything? I'm inclined to say no.

Finally, you realize your question is kind of absurd? If you mean a PhD, you should think about what excites you, figure out what labs are doing what you are interested in, and go work for those labs. The school is secondary. I'm sure you'll find good labs at stanford, mit, cmu, caltech and berkeley.
After all, the guy coined a new term - Bubble 2.0, and I liked it very much.
It sounds like you're describing very-nearly a textbook implementation of redo logging. I think what's making things easy for you is not the fact that you're using flat files, but rather that the entire contents of a transaction fit in memory. Most of the complexity involved in a commercial DBMS's log implementation stems from dealing with how to handle a transaction that modifies 20GB of data when you have 8GB of memory.

Edit: and yes, the irony is noted that I'm telling a Googler that he's not working with enough data.
15 Startup Commandments
Rule number 1 - Start - Experience is the best teacher
Shh... don't tell all the people applying to YC!
But why weren't you able to introduce them to prospective buyers once they'd decided to sell?

Isn't that where a well-connected investment firm adds value?
Video Podcasting Comes of Age with ON Networks
Going to YC SFP: Big Head Labs
"... http://www.stonetemple.com/articles/interview-bryan-eisenberg.shtml ...'

Update:

Oops. Got confirmation from a reliable source 'this is particular link is incorrect'.
"add one more new factor, like the failure of Windows Vista for example, and you may have that critical mass for a new collapse"

Forgive me, but I don't see the connection. How would the failure of Windows Vista do anything but help applications that run on the web? I can imagine some large things that could hurt this boom (economic collapse, maybe), but nothing small.

Web 2.0 has a lot of hype right now, but that doesn't make it marketing fluff. There is real substance here (and real profits, real innovation, and real users).

In contrast: here is a quick story about a company that I interviewed with in 1998. I don't remember the name; they were located South of Market in a really nice loft with lots of brick, glass, and chrome. Free lunch every day, free beverages, lots of perks. The business model: we are going to get your website's customers to fill out surveys. You are going to pay us for this. We will get the customers to fill out surveys by rewarding them with frequent-flier miles. I remember three things about this company, years later: how nice their loft was, how stupid and convoluted their business model was, and how arrogant they seemed to be about it. They acted like they were already a success.

In contrast, btw, the startup I did join was operating out of a cluttered office in Palo Alto that they'd got a really good deal on. Junk was everywhere, and the Founder/CEO asked me if I minded. I told him, "hey, it looks like a real startup." That company is still in business today. Frequent-Flier-Miles-for-Surveys, Inc: not so much.
... and this describes a bubble, how? This describes a boom.
Growth companies do not pay dividends, as a rule. The fact that these companies were having trouble getting anywhere near profits -- that was indeed a real problem. (Also, the fact that they were public at all was a real problem for them, since public investors want profits sooner than later.)
Probably UT Austin, U of Washington, and UMich Ann Arbor would be the next ones (in no particular order).

Good luck to you. I considered grad school but decided that since my ultimate goal was to do a startup, that I would just put the money from grad school into starting a business. It's been the most educational few months of my life. There's no grad school that will be as educational as joining a good startup. Be sure to read pg's articles - they are invaluable for avoiding very common mistakes.
Causes of some effect shouldn't necessarily be related, I was talking only about a critical mass, when causes add up.

I understand what you are saying otherwise.
"Make no little plans. They have no magic to stir men's blood and probably themselves will not be realized. Make big plans; aim high in hope and work, remembering that a noble, logical diagram once recorded will never die, but long after we are gone will be a living thing, asserting itself with ever-growing insistency. Remember that our sons and grandsons are going to do things that would stagger us. Let your watchword be order and your beacon beauty. Think big."

-Daniel Burnham, Chicago architect. (1864-1912)
From the Scottish mountain climber W. H. Murray:

"Concerning all acts of initiative (and creation), there is one elementary truth the ignorance of which kills countless ideas and splendid plans: that the moment one definitely commits oneself, then providence moves too. A whole stream of events issues from the decision, raising in one's favor all manner of unforeseen incidents, meetings and material assistance, which no man could have dreamt would have come his way."
Theodore Roosevelt:

"It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood, who strives valiantly; who errs and comes short again and again; because there is not effort without error and shortcomings; but who does actually strive to do the deed; who knows the great enthusiasm, the great devotion, who spends himself in a worthy cause, who at the best knows in the end the triumph of high achievement and who at the worst, if he fails, at least he fails while daring greatly. So that his place shall never be with those cold and timid souls who know neither victory nor defeat."
Much to do about "ster!"
"Keep away from people who try to belittle your ambitions. Small people always do that, but the really great make you feel that you, too, can become great."

- Mark Twain

(Good case for moving to a startup hub)
Open Source Search (nutch)
Dojo Toolkit Now Works Offline
I didn't realize the game was over. :-)
OpenDNS adds short-cut service (very cool...)
If you haven't tried OpenDNS, it's worth checking out. They takes an old, boring technology and make it faster & more useable.

David Ulevitch, the guy behind it, also runs everydns.net. That's worth checking out too, if you run a server.
Blood, Bullets, Bombs, and Bandwidth
I keep a file of these. Here are some of my favorites:

"In the long run, we only hit what we aim at."
- Henry David Thorou

"The more you sweat in peace, the less you bleed in war."
- Asian proverb

"No battle plan survives contact with the enemy."
- Helmuth von Moltke

"No wind favors he who has no destined port."
- Montaigne

"Never doubt that a small group of thoughtful, committed people can change the world. Indeed, it is the only thing that ever has."
- Margaret Mead

"Real artists ship."
- Steve Jobs
This is the unabridged version of a piece that appeared in Wired a while ago.
S3 performance from EC2 instances
I'm pumped for Disqus. Use the time in Cambridge to focus on building a technology war-chest. You'll never find the time to focus on code the way you will during SFP.
I don't see it there...
There's a difference between competing and cloning their service to the detail as soon as you lose a bidding war. I'm by all means a capitalist, but this somehow doesn't sit right with me.
This is a very good point. Some types of applications will find this scheme a bit difficult, though I'm in the midst of hacking through some POC work around it. I've got a database in the 100GB range potentially so it's a bit of a different domain.

I really think this concept works though for 8/10 applications which are using databases because, well, you've always used databases. Memory, save changes to flat files, read it all in during startup, it's really an elegant way to go about it.
I'm a Georgia Tech guy and we have a great CS program. But I'd go with the consensus here--get out there and join a startup. Maybe do that for a year, see how you like it. If you've already got startup fever in your blood going to grad school might not feel so good.
Link not working?
How about some symbol macro snippets?

http://people.csail.mit.edu/gregs/ll1-discuss-archive-html/msg02226.html

http://www.economicexpert.com/a/RTML.html
How long can it take for a blog to make some money?
Good info. Basically 10 MB/sec either way for a single transfer and 20 MB max combined for multiple.
Very useful article for anyone planning to make money from a blog..
Best language for interface development?
Of course there's Java, Flash and javascript. What other options are there? Examples please.
Yes, exactly, they are solving a number of much harder problems. However, these problems are irrelevant to 99% of web sites. How often does Twitter need to perform a 20GB transaction?
Max Levchin is like the Energizer Bunny. See show #2 on NerdTV: http://www.pbs.org/cringely/nerdtv/shows/

He said when he's not doing a start-up he lays on the couch morose, whereas he's happy and enjoying life when he IS doing a start-up.

I infer that the secret of success is to link pleasure to working on your start-up and pain to NOT doing it. If your start-up feels like a grind then it's doomed. Fortunately, you can re-program your emotional responses via various techniques.

You pretty much want to do that regardless so that if you do strike it rich, you don't feel useless and indolent for the rest of your life afterward. The continued success stories are the people who were REALLY into what they did. For example, Bill Gates could have retired long ago; Michael Jordan didn't win one championship and decide that was all he wanted; Steve Jobs kept on going while Steve Wozniak checked out, relaxed, and got fat. (I love Woz, I just think if you're possibly the world's best at what you do, abruptly stopping can't be very fulfilling, even though I understand he did many other worthwhile things like teaching kids.)
PHP: Choice of how many m/billionaires?

ColdFusion: MySpace

PHP: Facebook, Flickr

Python: YouTube, reddit

Lisp: Viaweb

Ruby: ?
Glad some of you are finding the Website Grader tool somewhat useful. It started out as a simple experimental project (basically I wanted something to meet my own needs).

Since then, it's taken on a bit of a life of its own.

The algorithm is basically trying to to simulate the same kinds of things I look for when trying to assess a site (a combination of traffic, SEO, site structure, popularity and other things). I'm making another big update to the underlying software within the next week or so.

If you have ideas for improvement, please leave a comment (either here or preferably on the websitegrader.com site itself).
"Raise your hand if you can do long division on paper, right now. Hands? Anyone? I didn't think so."

Is he serious? I knew how to do that in 3rd grade and a different version of it in 4th grade and on, and have never forgotten. How can somebody not know how to do long division? Please tell me I'm not the only one?
I can barely remember. But I just did a simple problem on paper (123 / 4) and I've still got it! :P it's 30r3 or 30.75...I hope.
Justin.TV has them :) But it's very spotty. Paul Graham's talk is available on Video I believe, on Google. Maybe I'm thinking of his last year's talk.
If you wrote this to give back to the community, then I applaud you. Sorry to criticize.
Yahoo sued for informing China on dissidents
Opps. My bad. I put it in the 'email' field thinking it'll be displayed. Now I know better. Added it to my account now. waleed_ka ((AT)) hotmail.com.
I think there'd be some period of 6-18 months where I'd truly do nothing but relax. Learn a new programming language really well. Travel and visit friends/family scattered across the globe. Then I think I'd end up doing some kind of startup hackery like Joe Kraus or PG.
I'm pissed that I'm forgetting discrete math. I loved it, but lost my textbook at some point.

I can still do long division but dividing 2651300296 by 46532 by hand is not on the list of Things I Want To Spend Any More Of My Life Doing.

Anyway Steve Yegge is my hero. He rips into Java and C++ better than anyone. I think my favorite is when he cited "Dr. Gary Larson's" definitive paper on static type systems.

http://steve-yegge.blogspot.com/2006/10/egomania-itself.html
I think you mean multi-millionaires - basically all the billionaires based their companies on C and C++.
Really great writeup Paul. I've been preaching "memory is magic" for the last couple years now -- since the capacity:price ratio got so good. Now I have a nice non-fluffy hacker post to send people to.

If there's any silver bullet to scaling most web apps it's properly utilizing gobs of memory. Pushing bytes from memory to your NIC is a pretty damn efficient operation.
he didn't sell the company. he sued the company and they settled out of court. he made his money via legal action, not business or programming acumen. the sale was for pennies and was only done so that the VCs wouldn't look completely retarded for investing in the first place.

he didn't really have any choice but to retire. no investor will give him money and no programmer with any sense would work for him.
Is it? I think the documentation on the actual site is more complete. It will be nice when they just get one set of complete documentation.
I don't know. Back in 1999 ACS was considered pretty ok. And his writings were the Joel/Paul back then.
Are you sure it's not due to how your email headers and content look? I had to remove my X-Mailer "Mail-Sender" and supply a proper address with a name before Hotmail would stop putting the emails into spam.
Maybe they were really good granola bars. It was basically a joke. I can't speak for her, really, but looked to me like an ice-breaker comment to get everyone relaxed.

It might make more sense if you understand that no one in the room would care about taking granola bars.
Moore's Law and a Free Market in Childbirth
nm - seems to be working now.
Venture Capital compared to the music business
"If you go to bat enough times you may never hit a home run, but sooner or later you're bound to be hit by the ball and walked."
Greenspun had other companies before Ars Digita. He took ICAD/Concentra public and also co-founded ConSolve and Isosonics.

It may be true that no programmer with any sense would work for him - I don't know him personally. But given how cheaply you can start a company nowadays, he certainly doesn't need investors. And as I understand it, he's not entirely retired: doesn't he teach a course at MIT?
Kleiner Perkins to Expand With Two Offices in China
Congrats to Daniel & Jason, that's great news.

From how you described the idea at Startup School, it sounds like you guys have a lot of potential.

Remember, this isn't a 3 month summer program, it's the start of a journey. Never relent, never give up :)
I remember you guys (and the rough idea) from the yc reception. Congrats!
Tee Shirt/Collared Shirt logo printing?
Does anyone have any recommendations for getting tee shirts or collared shirts which have their company's logo printed/embroidered? We're going to an 'industry trade show' and want to look sharp.
Doxa Print does some amazing print work on very fine American Apparel t-shirts. I don't think they embroider, but that looks dumb anyways :P. Just get a slick design made, and have them do it. It's very high quality.

http://doxaprint.com/

They're in southern California.
I have used this store in a previous project (about 2 years ago): http://www.casamariaint.com/

A bit expensive, but the shirt quality is great, and the logo came out perfect. FYI, I ordered the model #8527 for the guys and #8526 for the girls (under Golf Shirts, page 2)

UPDATE: I forgot to mention -- The reason we ended up choosing this particular store was because they did not have a minimum quantity requirement (that's why they're more expensive)
Jobs/Idea site for Ycomb folks (in development, suggestions welcome!)
Coda: one-window web development for Mac
I did this yesterday and deployed today, it's a standard rails app. User Authentication is done by acts_as_authenticated

Feel free to add items to the site for 'jobs' you'd like me to do. I envision it growing, and I'm working on making the body text wikified right now.

All apologies to reddit and news.YC for ripping off the votable-items format :)
Also, source is available at http://www.jaggederest.com/jobs/svn/

Commit access if you ask nice? Or something?

Note that is the development version, the production version is deployed in a different fashion (config mostly). So don't go trying to 'hack' that user on the server itself.
Reminds me of Sherlock holmes and his bouts of lethargy and cocaine-use between cases.
u mean like... a business?

Trying to provide a service that is of value?
Nice job.

Something much needed and wanted by people having trouble looking for co-founders, etc.

Hopefully this will bloom into a reliable resource and network.

Do others a favor and vote this up so they're aware of this potential tool.
Exactly. And it's mighty nice to have a declarative language (SQL) available to write your queries, and perform ad-hoc queries in a terminal, plus referential integrity, triggers, stored procedures etc.
8 GB of flash memory cost about $80

It does? Where can I get that?
We get our logo printed on American Apparel T-shirts by QRST's in Somerville, MA.
The record label as venture capital firm
Back in 1997 when I started web programming perl was free, then around 1998 mysql and PHP started being included in cheap hosting accounts.
I wonder what would happen if you use ZFS on top of a flash memory drive and replace/add a SQL layer next to the ZPL layer. In theory, you'd get transactions and checksums for free and possibly your sequential writes as well.
GTDwiki - Cute use of offline Javascript
+1 on QRST's. Quality company. And if you're in the area, they also deliver to local colleges.
Though this graphic joke wasn't planned, we did notice early on that YC was VC with, ah, more masculine characteristics.