skysight-inc/HackerNews huggingface dataset:
Wow, this is awesome. Unfortunately it will probably be a good wait before this is built and released to the public.

In the meantime, I've been developing a framework using EC2, S3, S3DFS (DFS = Distributed File System) along with SQLite to try and solve the database problem on AWS.

S3DFS allows you to mount an S3 bucket to one or multiple EC2 instances as if it were a local filesystem. It works at the block level and has read/write caching - so is very fast. (Note: It's free to use for development, but otherwise requires a paid commercial license.)

Since SQLite uses flat files and has no database server or client, it is ideal for use on S3. SQLite only scales well to a point, so to get around its limitations I am separating each user into their own database (each database is a separate flat file). This can sometimes be a bad idea, but SQLite allows you to attach multiple databases together to essentially create a temporary master database where you can run queries across all the user databases at once. This will be handy for search indexing, site-wide stats, etc. In order to change the user databases I will have utilities to allow global scheme alterations.

So, by adding EC2 instances that are all attached to the same S3 bucket, and using round-robin dns to distribute traffic among them, you can theoretically scale a web service infinitely. Just launch a new EC2 instance and add it to the round-robin dns pool and you're done! Even that could be automated to respond dynamically to traffic.

Imagine the long-term cost-savings from only having to build your app and architecture once and not having to go through a more complex scaling process like this: http://www.scribd.com/doc/21793/Scaling-PHPMySQL-Presentation-from-Flickr

Certainly you should only worry about scaling after your app is up and running, but what if you didn't have to ever worry about it? That's the goal of this framework.

This is pre-alpha of course and I'm still in the development and experimentation stage, but I wanted to share with the community and get some feedback!

http://www.openfount.com/blog/s3dfs-for-ec2

http://www.sqlite.org/

http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-06-26/

http://docs.amazonwebservices.com/AmazonS3/2006-03-01/gsg/

Update: Since this was so lengthy, I also added it as a post here: http://blog.nanobeepers.com/2007/04/07/infinitely-scalable-framework-with-aws/

Update2: Please leave any feedback here: http://news.ycombinator.com/comments?id=10001
Micrsoft is very much alive and though Paul makes some great observations about the challenges that lie ahead this discussion seems to be centred on people who want Microsoft to be dead and why Microsoft is such a threat. That's the real challenge...changing years of negative sentiment. That's what I care about
joe gregorio wrote more about this trend ~
http://bitworking.org/news/158/ETech-07-Summary-Part-2-MegaData, posted ~ bigdata ~ http://news.ycombinator.com/comments?id=9483
Nice article! Quick question: by Nero, do you mean the Roman emperor, or the CD/DVD burning software?
IIRC their main issues were threading instability in CMUCL on BSD or mac, which I believe has since been fixed in SBCL. In general, SBCL has come a long way, partly due to companies that use CL (e.g. ITA) funding its development. CL is in many ways much more production-ready than it used to be for that reason. It still has rough spots, but my experience is vastly improved over what it was 1.5 years ago.
The Sidebar Syndrome
Boston, 3(number of founders:)), 30, 27, 25
Not sure I follow your concern. Are you worried that people could submit articles/comments which are untrue? If so then the voting and wisdom of crowds solves this.
Obvious, but necessary. The most obvious of ideas are often the hardest to articulate.

Google seems destined to become the world's most valuable corporation[1]. Who wants to be Pepsi?

To compete with Google, copy Google's game plan. Focus on search. Evolve to AI.[2] (An AI is something that can give me a useful answer to my question.) Organize the world's information. Everything else Google does is cover fire.[3]

What I'm really asking is, why can't I name a single pure play search company other than Google?

[1] Besides oil.

[2] http://www.edge.org/3rd_culture/dyson05/dyson05_index.html

[3] http://www.joelonsoftware.com/articles/fog0000000339.html
I think the article does a good job of conveying what Lisp is to someone who just can't get it. I agree, however, that it is a bit on the verbose side. I think that at least a third of the text could have been removed without taking out any content. Heck, some of the content could have probably been removed without taking away from the point the author is trying to convey.

To make the analogy even more succinct, I will first ask you a different question. Have you switched to Firefox? It wasn't long ago that I went from using IE6 to using Firefox. Before too long, between tabs, quick searches, customizable toolbars, incremental page search, and more, I found myself saying, "Wow, this is what browsing should be. How did I ever use IE?" Lisp is the same way.

I disagree with the article's conclusion, "Learning Lisp is an uphill battle." If you have a good reference, it is not hard to start programming in Lisp. You could learn Lisp by writing trivial code in about the same amount of time it would take to learn C or Java. In fact, your code would probably be structured in much the same way. Once you've learned enough Lisp, however, and once you get to the chapter on macros (of whatever Lisp reference you're using), something will click. Then you will realize that you have been programming all wrong all along, and you will instantly rewrite everything you've ever done in one red-eyed night. The next morning, you will be reborn.

But seriously, if you really want to learn Lisp, drop the $40 (or whatever it costs at the moment) on ANSI Common Lisp; it's a great read. Once you plow through that and are hungry for more, look for On Lisp or Practical Common Lisp; both of them are available online.

If it wasn't for my classes with their requirements to program in C++ or Java, I never would have looked back after learning Lisp. I think any good programmer who gives Lisp an honest try will feel the same way.
Infinitely Scalable Framework with Amazon Web Services?
Added this in the hopes of getting some good feedback from the community on this framework I'm developing.

The post is derived from this comment: http://news.ycombinator.com/comments?id=9991
Kind of like Slashdot's moderation system, where you only have 5 points to use per day. (At least that's how it was 3-4 years ago when I stopped reading /.)
Affordable biometric digital ID systems to enable people in the developing world to get loans and other access to capital.

No patents.

I already more-or-less solved the first round of problems in doing affordable housing for the developing word:

http://Appropedia.org/Hexayurt_Project - and placed all that work into the public domain. So now it's time to hunt bigger fish.

Reality is much, much clearer if you think about the needs of the very, very poor, and the interaction between the shocking arrival of the cellphone and solar panels or the national grid and the basal economic needs of a family.

Got the contract from the Federal Govt. to write the papers and prepare a demo of the technology coming next week I'm told, and it'll be going into the public domain shortly thereafter.
This is pretty interesting. I'm not sure I understand what you mean though about separating databases "by user". Could you elaborate on that? How would my application's database change if I were a user of this service?
There have been multiple instances when the same grand declarations have been made and Microsoft has only survived and remain dominant.This is just the next cycle of competition.
Microsoft is paranoid enough and this article makes the assumption that they have blinders on and while the move to web etc is happening people at Microsoft are twiddling their thumbs and will say please come and take our business away. Maybe they are in a good position to take advantage of these new paradigms. Of course there is a fight out there but declaring death is premature, significantly challenged, yes.
From what I've read of evaluations of S3, the latency of S3 requests seems significant. Significant enough that using S3 as a backend database doesn't seem feasible. Is this true or am I mistaken?
I forgot to mention that I think this is fantastic idea. I'm just not sure if it will be able to replace a local mysql cluster.
I think you need to be passionate about your idea at the interview...if your not, who is going to be?
Timestamps and total page views.
So what's next? Shall I make the same joke on you using Common Lisp and the the user named pg can then pull up Arc...?
Excellent post. Not only did you nail the growing feeling that Microsoft is gradually becoming irrelevant, but you gave the best explanation as to why that I've seen anywhere.

I think another reason is that younger people are more computer literate and more open to new OS's and software. Mom and Dad were intimidated by computers and just wanted the simplest, most common software (microsoft/IBM), which they'd learned to use at work.
He basically said that all they have is the desktop...but that even that market is getting chewed away too.

microsoft can be getting hit by web apps and competing desktops simultaneously...no contradiction there. He just chose to choose one as a bigger factor than the other, that's all.
Here is a piece I wrote:
Java has harmed Unix but helped Windows win
http://juwo.blogspot.com/2006/11/flatface-musings-java-has-harmed-unix.html
It's true...what bright, iconoclastic young thinker applies for a job at microsoft these days? It's a stuffy drone corporate gig, and those are the people that'll gravitate to it.
I think Windows Vista (tm) is going to go down in history as the best thing that ever happened to Linux. MS has done the equivalent of pointing the pistol straight at their shoes.

"...How about the ability to install any of 20,000 programs from a convenient package/portage/rpm directory for free in seconds?..."

Well, I tell you, it's just not that simple. Linux distributions may or may not work. Just yesterday (Friday) I installed Ubuntu Drake (you know, the STABLE version) only to find it boots to ... a command line. No graphical desktop. Command 'startx' unrecognized. Try to install xserver-xorg. It appears to succeed but startx complains about font 'fixed' not found. I Google some forums, they say font directories have moved, but by golly I can't find the new directories the forums mention. I'm not alone, but no further advice is to be found. Just for fun I try to install ubuntu-desktop in the early afternoon only to find the installation grinding on for a couple hours then dying with my network partner disconnecting the logical link. The better part of a day wasted trying what should be a simple installation. Right.

If you find Linux installs fine and upgrades painlessly, then I'm happy for you. But don't claim Linux makes MY life any easier. In return, I won't demand support from providers of free software.

Maybe MS has the bad interfaces, but they work where others fail. You've got to give them some credit for that. Of course, this can all change with Vista and maybe even XP eventually. And eventually, when XP is bloated with DRM, maybe even Ubuntu will boot my system to a graphical desktop.
Yeah, that's a valid concern. Fortunately the read/write caching that S3DFS provides largely negates that problem. The caching is on the EC2 instance and so should provide great responsiveness to the app.
kind of. look at it from the competitor's end. they can't afford to sell at $5, they'd go under. It's like wal-mart putting mom and pop record stores out of business by selling CDs at cost.

So MS keeps their software standard at colleges, which influences students' buying habits (god forbid they grow accustomed to Mac or Linux and lose their fear of them).
Thanks, AWS is amazing and has been super fun to play with. This solution will of course only work for a certain subset of apps. Apps that require more database complexity will not be a good fit for this framework. My apps are typically designed so simply that they could just use a flat-file storage system instead of a db. SQLite is awesome because it provides the flat-file storage, but in a format that allows you to query it like a normal database. The simplicity of that model is amazing and hard for me to resist. I've used mysql clusters in most of the projects while working for companies, but I often found that the added complexity did little for us. My last company hired two database guys with fat salaries just to manage it. My goal with this framework is to really reduce the cost of maintenance, backups, etc. S3 is distributed and so really doesn't need much in the way of backups - but backing up this system is a simple as tar gziping the directory of flat-file databases.

This was interesting from pg's Viaweb FAQ:

http://www.paulgraham.com/vwfaq.html

"What database did you use?

We didn't use one. We just stored everything in files. The Unix file system is pretty good at not losing your data, especially if you put the files on a Netapp.

It is a common mistake to think of Web-based apps as interfaces to databases. Desktop apps aren't just interfaces to databases; why should Web-based apps be any different? The hard part is not where you store the data, but what the software does.

While we were doing Viaweb, we took a good deal of heat from pseudo-technical people like VCs and industry analysts for not using a database-- and for using cheap Intel boxes running FreeBSD as servers. But when we were getting bought by Yahoo, we found that they also just stored everything in files-- and all their servers were also cheap Intel boxes running FreeBSD.

(During the Bubble, Oracle used to run ads saying that Yahoo ran on Oracle software. I found this hard to believe, so I asked around. It turned out the Yahoo accounting department used Oracle.)"
I agree with you: Microsoft is dead, long ago. However you fail to note that another famous victim of Google, Javascript and broadband is Apple itself.
In fact desktop software in general is dead. Word processing, spreadsheets, e-mail, image editing, project management and many other traditional desktop killer-apps are now offered for free on the web. Let's face it: only professional graphic designers and architects really need the power of Adobe Creative Suite or Autodesk Autocad and, as a consequence, a Windows or Apple machine.

How people are coping with this new reality? There are, in my view, two different attitudes. People more sensitive to hype are buying expensive Apple laptops even when they are simply overkill for their real needs. Programmers buying Apple laptops just to run Textmate are similar to fashion victims buying the latest Gucci handbag. Other people are more rational and choose cheap but sound Linux boxes (especially the Ubuntu flavor).

I think 90% of computer users can live with Ubuntu without any problem. Of course the desktop software libraries for Linux are very meagre compared to Microsoft and Apple's universes but we've just stated that desktop software is dead: web apps' user experience is the same on every platform. The operating system really doesn't matter. Don't you have other needs to satisfie instead of buying an expensive and useless Apple fashion laptop? Buy a cheap laptop, install Linux and enjoy your life!

And what about the business side of the moon? What will happen when companies will suddenly discover they don't longer need desktop software? Companies are more cost-conscious and I assume they will migrate to Linux en masse.
what are some examples of that?
This really is a personal choice. We were lucky with our startup because we got desk space at the Obvious corp building in South Park (Obvious are the guys behind Twitter, Evan Williams et al). It's a fantastic working environment with lots of smart people around and great productive atmosphere.

However there were certain days when I actually found myself getting into the zone of productivity just sitting on the couch and working my laptop. If you find yourself in one those zones is it worth breaking it just because you've programmed yourself to think you're more productive in an office/library/wherever? I don't think so.

While there are just a couple of you working on your startup, I think you should just work wherever you think you'll be most productive that day. The majority of the time I work at Obvious, sometimes I work on my couch and sometimes in a random coffee shop or bench somewhere nice. I can only do this for so long before we start hiring and then we all need to be in the same place so I may as well make the most of it now.
Sure. Basically each user would have their own database file generated on account creation. Then in your app, instead of doing something like:

connectDB("/path/to/db")

you would do:

connectDB("/path/to/$user/db")

Also, if you needed to query all of your users data in aggregate, you would use the 'ATTACH' capability of SQLite to essentially create a master database. Note that you will need to design your scheme appropriately (thoughtful table/column naming, etc) for this to work well. See: http://www.sqlite.org/lang_attach.html
I think the death of the desktop is much exaggerated.

AJAX is better than the last generation of web-apps, but the browser still basically sucks as a platform. Latency is a bitch. :-)
Wow. Great article. I was going to put a few of my favorite points here, but there are too many :P
My problem with this essay is that it could just as well have been written in 1985, 1990, 1995 or 2000. Microsoft has always been slow in responding to new threats.

1985: Windows 1.0 was a totally insufficient response to the Macintosh. There were plenty of other Window Managers too, all superior (oops, not Topview, now that was an inferior product). Lotus and Borland is going to kill Microsoft with superior applications.
1990: Microsoft put out Windows 3.0. Still DOS inside. ACE consortium wants to kill Microsoft with Motif.
1995: Netscape is the latest presumptive Microsoft killer, IE is an inferior product. Sun unveils Java, purported Microsoft killer. Windows 95 looks nice, still very much DOS at heart.
Constant refrain during the latter 90's: Desktop Linux is going to kill Microsoft...
2000- The Web/Sony is going to kill Microsoft.

And it's not like the cool kids ran Microsoft products in 1990 or 1995 either. The tastes of an elite doesn't necessarily reflect the future of the broader market.As long as Apple won't let you run OSX on non-Apple hardware it can't be anything but a niche product.

Sure, Microsoft is vulnerable, but so is Google or Apple or any other company in this business. I don't know why Microsoft are underestimated so often, it might be that we root so much for the underdog that we fail to see what the big meanies strengths are.
Bay Area (SF,PA), 2, [31, 27]
justin.tv as a way to interview people for jobs
Would you broadcast your life for a chance at getting a job?

From an employer's perspective, this would be an excellent opportunity to really know employee candidates. The employer will see what they are like on a day to day basis -- both on professional and personal levels.

Moreover, the broadcast could include what's on the candidate's computer screen. This would be a good way to evaluate their coding ability and see how easily they get distracted by email, social apps, etc. for example.

Such a life broadcast might be an open-ended job interview with no particular employer in mind.
There's a guy working on a MySQL storage engine that uses S3:

http://fallenpegasus.livejournal.com/tag/s3

It's still at a very early stage in terms of implemented features, but he seems to be moving forward rapidly.

I have no idea what kind of performance he's getting -- I'm planning to check out the MySQL source sometime next week and do a test build. Extremely cool idea. :-)
Columbus, GA area; 2; 33, 27
This is a take off on a talk I heard years ago about networking
San Diego, CA; 2; 23, 23
Part II: http://www.informationarchitects.jp/webdesign-is-95-typography-partii
I also recommend the rest of the articles in their "IA Notebook" on design questions of all kinds: http://www.informationarchitects.jp/ia-notebook
People would keep using their browser.

That sort of trick used to work, once, when we all got our software off CDs on the cover of magazines. Firefox is a small download now. They won't try it twice.
For 80% of us, that 20% will do.
Good to Great: The Hedgehog Concept
That's a great looking font --- elegant, but still LEGIBLE.

I found the download page here:

http://www.levien.com/type/myfonts/inconsolata.html
I was thinking more of the latency involved in making the roundtrip request from the app server to S3, which is probably on the order of tenths of a second. It would force you to rely heavily on a local memcached to achieve acceptable performance (which is probably a good thing anyway).

I'm reminded of a conversation with Facebook's CTO at Startup School, and he mentioned that they experimented with S3 for their photo hosting, but the latency of loading images from S3 made it unacceptable by their performance standards.
Karl Sims' Evolved Virtual Creatures (old but still very cool!)
I wouldn't count them out just yet. It's true that they can't kill a web 2.0 businesses like youtube the way they could kill desktop-based competitors in the past, but don't forget that the browser itself is a desktop-based app.

If I controlled MS, I would try to develop a better platform for web apps than Firefox (which is definitely possible). I'd use patent-encumbered APIs, which I would share while trying to build market share, and then force my competitors to stop using once they were entrenched. Maybe somebody smarter than me could come up with a better idea, but the point is that it's probably still possible for them to leverage their OS near-monopoly into a web near-monopoly.

It would have been far easier for them if they had started earlier. Firefox would have been easy to kill 3 or 4 years ago. But MSFT has so much money and such a large installed base that they can get away with a lot of screw ups before they really cease to be a threat.

I agree that, for the time being, entrepreneurs don't really have to worry about MS. But they're not really dead. They're just comatose.
You can see how this was done here:

http://web.genarts.com/karl/evolved-virtual-creatures.html

Karl has done lots of other cool stuff btw:

http://web.genarts.com/karl/
Good to Great: Level 5 Leadership (the kind of people who were turn good companies into great ones)
Aspect-Oriented Programming: Radical Research in Modularity
Intentional Programming demo (did intentional programming inspire Eclipse?)
Part 2 is here:

http://www.youtube.com/watch?v=ZZDwB4-DPXE
In 1925, the British Empire controlled 25% of the earth's surface, but it was dead nonetheless.
Photoshop itself will soon have an online version:
http://news.com.com/2100-7345_3-6163015.html
Aspect-Oriented Design Pattern Implementations

http://www.cs.ubc.ca/~jan/AODPs/

Aspect-Oriented Refactoring (check out the wormhole refactoring in part 2)

http://www.theserverside.com/tt/articles/article.tss?l=AspectOrientedRefactoringPart1

http://www.theserverside.com/tt/articles/article.tss?l=AspectOrientedRefactoringPart2
Can we apply this idea to other domains such as web search?
A Venture Capitalist passes away peacefully, and...
I rarely write anything with a specific audience in mind, but I confess I was looking through my "logs" and noticed how much traffic I have been getting lately from news.ycombinator.com.

This is my way of saying, "thanks!" I hope you enjoy it.
It is a cultural shift that will bury Microsoft. Much like the Walmart distribution deal killed Levi's as a brand.
{location:'nyc', teamsize:2, ages:(25,33)}
That's exactly what made the Graham essays (pre-YC) so compelling: this wasn't someone with a vested interest, just an astute observer making comments few others were.

Blurring that line is a mistake.