skysight-inc/HackerNews huggingface dataset:
Here are my answers / comments to some of the recent posts (more replies to come). Thanks again to everyone for your help and please keep your suggestions coming:

to swombat:
The clients will have javascript so short commands rather than entire pages can be passed. Thanks for this suggestion, implementing this alone may totally eliminate my web server issues.

to jwilliams:
You said even if my page is tiny that's still 1Gbps sustained, so by implementing swombat's javascript solution I can (hopefully) pass only data, thereby avoiding the potential bandwidth issue.

to swombat:
The data cannot be stale, the response MUST include the results of the database search that is performed immediately after the visitor's last post is appended to the database.

to andr and swombat:
I don't know what 'sharding' or 'microsharding' is but if the additional details I've recently posted is any help, maybe one or both of you can tell me if you think it is something that's possible, practical and/or appropriate to my needs?

to mseebach:
Yes, every single write will affect every single subsequent read in a non-trivial manner. You said the hardware will cost a lot and I'm sure that's true, but I expect the money will become available if the system I've conceived can be built to function as I'm trying to describe it here.

to EVERYONE:

Given the information you all have provided prior to the time I wrote this post, I have further refined my approach to this problem as follows. Please tell me if this is the best approach in your opinions:

1- I will use JavaScript to eliminate page loads and pass only the required 5-10 characters of data from the visitor to the server each time. This should reduce or eliminate the need for multiple web servers and dramatically reduce bandwidth requirements.

2- I will find someone who knows C to write an app that receives the data from each browser, pass it to the database with an append command, pass another command to search/sort the database table, retrieve the results, and pass them back to the JavaScript in the visitor's browser.

If this appears to be the best approach so far, I still need to solve the database issue. If you have any new comments or suggestions that might help me find the database hardware or software I need, please post them here, thanks.
In http://www.stopie6.org/spread-t he-word ,
it would be better if you change src for img tags to absolute urls and not relative ones... that will make it easier for banners to appear on my/(anyone's else's) pages/blogs.

i.e. change ###img src="120_60.gif" alt="Stop IE6 campaign logo" width="120" height="60"###

to ###img src="http://www.stopie6.org /img/badges/120_60.gif" alt="Stop IE6 campaign logo" width="120" height="60"###

:)
Location is significant, period. It may be over-hyped. But so was the web on several occasions. Sometimes something gets over-hyped because it has tremendous potential to begin with.
I hate IE too. Its way too slow
I understand your problem and I empathize with you.

But you are at the very, very low end of the experience curve. It makes sense for me, as a designer, to cater to the largest number of people possible, and to provide them the best experience I can. If I continue to cater to the outliers, like yourself, or like a friend of mine who runs a 3560x2056 screen, then I am doing a disservice to the largest segment of my market-- the average user.

If IE 6 were to go extinct, I would be able to provide a better experience for a larger number of people. If no one used 800x600, I would be able to provide a better experience to a larger number of people. Therefore, I will try as hard as I can to make both of those things true. Even if it makes your experience slightly less enjoyable; there will be collateral damage.
IMO, a hacker's manifesto wouldn't need a lot of screen space. Just a line: "follow your heart"
DeepDyve: New browser aimed at uncovering the Deep Web (developed by HGP scientists)
Agreed. I rarely exit() out of an error. I was responding to the parent's want for "a big, flashy, loud, total failure".
Major Source of Online Scams and Spams Knocked Offline
http://www.nap.edu/catalog.php?record_id=10129

The report was issued by the NRC, not the NSF, but the NSF apparently agrees because they paying for an overhaul of the AP curriculum based on the report.

Also, if you do a Google search for:

NSF criticism of AP curriculum

You see an overview of this as the second result. Also, I couldn't find this in that Alfie Kohn book, so I must have stumbled upon it in something I read around the same time.
that's pretty neat. how about a recursive manifesto ?
Wow! I didn't know about RedHat. All this while I was under the notion that it was just a pun of words.

The biggest in the list should have read "Google - spelling mistake" (although its SAAS)
implementing that support for every new site design

If that's what you're doing, then you do have a problem.

How can you deliver content to IE6 users, irregardless of site, without "implementing that support for every new site design"?

Find an elegant answer to this question and change the way you do business. (You do know what a tradeoff override is, right?)
Coda 1.6 release notes
cater to the outliers

Yes, catering to outliers sounds like a compromise most wouldn't want to make.

All I'm suggesting is that we use our smarts and cleverness to find a way to deliver to outliers without catering to them and without breaking the bank. Now, that would be a real paradigm shift. (Oooh, I can't believe I said that.)
Wapple.net solves problems of mobile internet
No scaling guru here, but did this quick calculation for fun (gotta love C-x-e):

(/ (* 150000 ; users

3.5 ; times per min

(+ 5 ; cookie length

7.5 ; chars per post

)

)

(expt 10 6) ; bytes per MB

)

Evals to ~6.6. Double that (for max capacity estimate) = 13.2. Unit is MB/min, which is ~0.22 KB/sec. One hour is just short of 800 MB of plain text data. A gig of DB seems reasonable (I don't know what extra overhead is associated with, say, a CHAR(15) field).

Like Rectic said, a RAM based database seems better.

PS sorry, I don't know how to format code here.
Fusion - new Deck-like ad network
You can see the dropoff in my corporate spam reception at http://jim.studt.net/spamdrop.png . At about 3pm central standard time about 40% of my spam stopped arriving (red line and zinc chromate green line). Not bad for a single action.

(The vertical axis is "messages per hour". My filters are lined up as bogofilter,dcc,spamassassin,clamav. A positive for spam stops evaluation of the later, more expensive filters. Likewise, most viruses are caught in the spam filters and never get to clamav. The robots line is automated status reports from remotely deployed code.)
When in Google Earth, view as the Ancient Romans viewed
Do you have a universal drop-in solution that you'd like to share? If so, then it renders the IE6 issue moot.

If, on the other hand, you're expecting every developer to discover this solution themselves, then you're only proving my point.
An actual poll from a few weeks ago:
http://news.ycombinator.com/item?id=328901
T-Mobile G1 Consumer Review: First Impressions, Best Free Apps
And/or you inherited the curiosity trait.
After reading through all the other replies, I'm really interested in seeing suggestions on the search and sorting operation that needs to be done with every write.

Realizing that if the query returns a huge number of records, sorting and feteching those records will be THE bottleneck.
I'm not entirely sure about them, but I use S3 and EC2 with SQS so that the user can upload the file, it will be encoded while waiting in a queue, then place itself back into storage. That way nothing ever touches my production server.
Thanks, that's about what I was expecting. I was wondering where you got the data, does it include inflation, and how does it include dividends?

PS: I keep wondering what all those baby boomers taking money out of 401k's to retire with is going to do to the market.
hmmm... lemme think... lemme think...
HMH: Hacker's Manifesto is not HMH
The question is whether users are using IE6 voluntarily or just accidentially or because they are forced to do so. Maybe they need a reason to upgrade and giving them that reason may actually help them in terms of security and productivity.

Also, the cost of developing for IE6 is being passed on to users, be it in the form of fewer features or loads annoying adverts. They may not know that and are therefore unable to make a deliberate decision.
I agree. My interest went from 100 mph to near zero [1] after realizing this was the case.

[1] quoting somebody, ha, ha.
From your description you don't seem to really use the db, so can't you roll your own specialized data structure?
If each record is 20 byte, 1/2 million per minute for one hour is about 1/2 gig, so you can keep everything in memory.
Ah, now I agree totally with that.

And I've found that you can really easily make something that sort of works OK in IE6 most of the time if you follow all of these suggestions: http://anthonyshort.com.au/blog/comments/how-to-get-cross-br...
That paradigm shift is already in full swing. It's called Flash & Silverlight. Do you like it?
It's significant if you want to find something. That need is already pretty much fulfilled.

You can't just tack "location aware" on to existing products and make them out to be revolutionary, unless they really are revolutionary and satisfy as specific need.

It's like people have been given a new toy - location, and are tacking it onto everything. Like a web-designer who has just found the <blink> tag.
No bad at all! OLPC RAM = 256 MB.
Nicholas Carr on 'the big switch' to cloud computing
Wow, I can't wait to redeem the skymiles on this.
Can depend on what the code is embedded in.

C has been gaining traction in embedded avionics software, which is purposefully very simple, understandable, maintainable code, and tested & verified beyond nearly any other type of software.

C is also used on mobile phones, where it's more hit & miss, as I'm sure many of us have experienced first-hand...
the end of the article makes this sound more like a politically inspired statement than an economic one.
It's probably worth pointing out that the article's title is "Data Model != Database" as in not equal to.
what types of students would you consider better suited to each type of environment?
[edit] Also, what manner of 6 classes every semester courseloads are we talking about? Because some 6 course combos are easily manageable whereas I've seen other combos only a person who can score a perfect on the putnam exam while drunk could manage comfortably.
It's not a bug in MzScheme. The fundamental problem is that the server is a 32 bit machine with 4 GB of memory, only about 2.5 GB of which we can use. Really we should just upgrade to a 64 bit server.
Don’t Try To Force Fit Your Model Into A VC’s Thesis
Are AmEx's clients disproportionately going bust?

Visa isn't seeking help, does that make sense?

And if AmEx does go bust what's the damage?

AIG and Lehman had all kinds of counter party risk, but who exactly would be hurt by AmEx going belly up?
Well, first, what is aid? The big problem is that everyone is using the word "bailout". That makes one think of free money that doesn't have to be paid back. That isn't the case. Yes, if the firms still fail after receiving the government assistance, they won't pay back what amounts more to a loan than a bailout. So, yes, taxpayers can loose, but it isn't a bailout so much as an emergency loan program.

Both Circuit City and Mervyns have been dying for years. Mervyns has been shuttering stores for several years and went through a failed turn-around attempt. Circuit City simply couldn't compete with Best Buy. They didn't have sound businesses.

The big difference between a place like Circuit City and American Express is the amount that would be needed to overcome a blip. Circuit City, if it had a sound business model, might need a loan of a hundred million or so to keep paying salaries and such through a downturn. American Express needs to keep lending money - and all financial firms lend more money than their company is worth. They're intermediaries. So, Circuit City can go to a bank (or several) and get the cash they need if it would save their business. American Express can't. They need cash on a scale that Circuit City doesn't.

I understand why so many are outraged. It looks like a firesale of your tax dollars to a few firms. That isn't the case. If these firms make it (and based on their business models they will), the government will actually make money. Circuit City and Mervyns were just failing firms. Many in the financial sector have sound business models that need some cash for a year or two (in amounts that they can't raise from sources other than the government) because there was a perfect storm of credit conditions.
see also:

http://whitehouse2.org

http://fixthisbarack.com
Pictured: The robot that can pull faces just like a human being
90 petabytes: the "Deep Web" in 2002 - includes private databases, controlled-access sites and so on.

6 years ago, oh my.
The on-line nature would only make sense if it were for a highly dynamic behavior-based ad-serving thingie.

So, are you serving ads based on where people go?
Having worked for a bioinformatics company, I can describe my work as handling huge volumes of data.

Modern instruments are ever more computerized and spit out gigs and gigs of 1s and 0s with every use.

And then you have to take all that data and try to turn into something humans can understand.

And every time our instruments get better the amount of data we collect goes up.

Systems biology is basically a data management problem.
This is easily the most interesting thing I've read all week.

I was talking to some of our systems engineers about the size of the Hadoops clusters at Yahoo. I was pretty impressed with having access to thousands of machines with 10s of petabytes of storage (think kid in a candy shop).

It's nice to be put gently back into my place, by a simple raindrop.
Must've taken 4 hours for the DNS to switch over.
I love being a software engineer and I can't upvote timr's comment enough.

That's the hard truth folks, don't get into engineering if you don't love it.
It seems you would be better off using some custom in-memory data structure rather than classical ready-made database solution.

You do not want to sort the whole dataset after each new piece of data comes. You want to insert each new entry right into its place (probably via some hashing).

If, of course, it is possible. Too little details about what you want to do with the aggregate data.
yep. thats kind of what i meant when i said wonky. also, i'd show you mine, but it isn't even working correctly for me.
> You can travel back in time to see Rome, Italy as it was in 320AD

Which other Rome might we be talking about here? Really.
That probably depends on your definition of dying. Usage will drop asymptotically I suppose.
Erlang Web
This guy needs to read GEB. One could describe a raindrop with even more bits (subatomic particles anybody?) or less (size, volume, shape of raindrop).
Your doubling the salary for teachers due to tenure is questionable. The reason tenured professors don't quit is only partly due to tenure, also due to the fact that they find being a professor more interesting than the other work. Regular (grade-school to high-school) teachers don't exactly have tenure (though for those in most unions it is close), and they probably don't have the same employment options outside the profession.

I'd give a multiple of 1.2.
I don't know that they had a better grasp, they just paid the highest.
Unfortunately, actions like these do not knock off spam for long. iirc, the last time a major spammer was knocked offline, the result was a sharp drop for a day, then a return to pre-action levels.

We're better off making spam a less profitable venture, and thereby letting the market reduce the number of people turning to spam as a source of income.
You could cache or replicate the database for reads. When a post comes in, insert it into the master database, then sleep for a preset amount of time, then have it start checking the local database copy to wait for it to show up, then run the query. Just use timestamp <= posters_timestamp to get the result set you want.

Adjust the sleep time based how frequent the cache updates are.
> If these firms make it (and based on their business models they will),

The fact that Amex can't get private investment says that the smart money thinks that they're a dumb bet. (There tons of money around. Buffet isn't the only one snapping up actual bargains.)

It may be that the Amex biz model works but Amex the company can't do it. In that case, the loan is a dumb idea. Let someone else pay to take a shot at it.

Feel free to invest your money, to get together with others if need be. If you're right, you make out. If you're wrong, I don't lose.
MyLifeInACube no longer in a cube
Yeah, that's what I was getting at in my post. My point is that Google is ramming this chat stuff down our throats and nobody seems to complain. If it was true "widget architecture", I would be able to remove "Chat" and "Invite" completely, not just collapse it.
We just got a contract that will fund several months of development for our startup - maybe close to a year. The client uses IE6. You think I'm going to tell them they have to upgrade?
ObamaCTO, fixthisbarack, whitehouse2 -- what to do with this meme?
Full disclosure, I started fixthisbarack.com. The site got probably over 7000 hits the first day after the environment redditors decided to put it on the front page.

Thursday: 6148 (started tracking after reddit decided to put it on the front page, and the server became unresponsive at times)
Friday: 685
Saturday: 195
Sunday: 286
Monday: 261
Tuesday: 622

What is the natural history of these things? Do we know? Are there well understood branch points that have already been studied? A simple logical conclusion might be that something like this belongs on change.gov, but what would it accomplish? Does it carry the weight of the people? How could that be measured?
I am not saying you should buy GM but if a 3rd party where to get interested and do a leveraged buyout they could make a lot of money or if the government does a big bailout you could make a lot of money etc. The real problem with GM is they are not making money today so they might not continue to be.

PS: Options on GM stock could be a good investment of you are willing to take a lot of risk.
Yep, I use this model, too. Using this model is great, too, because multiple disparate servers can be dealing with the media coming from/heading to S3. You don't get stuck with media files on one box.

It's like a giant disconnected filesystem.

Oh, and use whatever language you are most comfortable using.
It's a sign of a shortage of graduates that companies trust to be of a certain caliber. Either you have done something to distinguish yourself, or Olin has developed a track record that companies trust, or both.
Thanks for posting - this is hilarious
Sniff around the Amazon S3/EC2 documentation and you'll find pipelines demonstrating what you want to do, such as:

http://developer.amazonwebservices.com/connect/entry.jspa?ex...
He makes quite a few valid points and suggests using a business rules engine called "coda server" http://codaserver.com/, does anyone have any experience/impressions with this?
"It's not the numbers that count but the quality."

For the short term economy (1-5 yrs), the quantity counts much more than quality. If Bank of America needs to hire 500 interchangeable Java programmers, they probably can't use or manage 10 excellent programmers instead. But for the longer term, a couple Sergeys and Zuckerbergs (or for that matter, Grahams and Morrises) are much more valuable.
It isn't really an unmitigated good, though. By increasing the incentive for a foreigner to get a US engineering degree, it would increase the competition for those slots, and result in fewer US students getting degrees. Of course, if the foreign degree holders stay in the US, that might be OK, but things must be thought through carefully. For example, presumably at least some people not interested in engineering, but interested in a green card would get degrees.

Mixing up incentives is a dangerous business. I would prefer a more mundane approach, like increasing the number of H1B visas.
article in the nytimes: http://query.nytimes.com/gst/fullpage.html?res=9401E4DB1431F...
Unfortunately you can graduate with a degree in finance and not know shit about accounting.
When Customer Service Steps Up | TechJaws
Did anyone who supported the bailout actually expect nothing like this to happen? The government is pumping SEVEN-HUNDRED-BILLION DOLLARS into the economy and every major company is going to try and get a piece. Companies will (and should) always try and do what is in their best interest in order to stay competitive and make money. No one is to blame except the senators and congressman who passed this atrocious legislation. Meddle with the market and this will always happen.
One of Buffet's largest holdings is Amex.
Linux boot loader visualized
Cloud Computing: What's stopping service-oriented clouds?
What is the value, or barrier to entry of a competitive site? Save for the mildly amusing domain name, I don't think you have anything of value, until you manage to build up an audience and following with some significant measurable stats.

No offense, but the "social news with voting" thing is pretty much a templated rollout these days (as your slinkset site would indicate). You're standing on the shoulders of a midget who is standing on the shoulders of a moderately large man.

The history of these things, IMO/IME, is that with a fair amount of work, promotion and luck you can turn it into something of moderate value (maybe mid 6 figures if you're lucky). A couple of mentions on reddit for a meme-site does not equal much in the short term.

Just my thoughts/opinions, etc. Somebody else might have a different perspective.
AirGlass: Today’s little bit of wow
Holy shit this is awesome for Uservoice. Congrats Rich and Co!
Django Caching with Backups
Pagination exists because it's relatively straightforward to program on the server-side.

To do client-side pagination, you need a hacker who is willing to take all the rendering out of his preferred language and using Javascript and Ajax instead.

And then you end up horribly breaking the back button.
yet another site that doesn't correctly forward urls when surfing via iPhone
you really, really don't want to be using a relational database here. hitting disks a million times per minute would be disastrous, if not for performance then for blocking IO, even with only row-level locking.

I'd look into memcached and write state to disk every once in a while to back up, kind of a reverse from what's usually done. Or, if thats really not enough, I'd look up Hadoop riding on GlusterFS. You're going to have synchrony and blocking problems no matter what, though, without really creative keying (bigtable style?). Both Facebook and Google have run up against these kinds of issues and have written and/or released source about it. Facebook has OSS'd Cassandra, their distributed database, and Google hasn't released much code but has papers out on their implementations of MapReduce (open source port->Hadoop, as suggested above) and BigTable (which the Hadoop team are in the process of cloning with HBase).
Restricted Stock vs Options When We Are "Under Water"
http://news.ycombinator.com/item?id=356187