skysight-inc/HackerNews huggingface dataset:
When coming from ASP PHP was just amazing.

Not that it doesn't suck, it just sucks so much less than ASP :p

And there are still ASP developers around, I just recently worked with a company that still writes all their stuff (websites for paying customers!) in ASP
A new approach to optical computing
So shard more, or get more disks, or get flash or whatever. (Yes, I don't know what I'm talking about.)
Getting Started with the Google App Engine
"In the first two AI box experiments, Eliezer Yudkowsky managed to convince two people (adamant that they will not let the AI out) that they should let the AI out."

Sounds like Eliezer is the AI.
There's a quote I like that I think came from Marvin Minsky, "the US needs a Department of Homeland Arithmetic." We're protecting ourselves from all the wrong risks.
Paying attention to half-baked advice like this is half the reason Twitter is fucked.
Nobody Likes a Liar
From a concept persepective, this type of site gets stickier the more upset people get. Which is a good thing, it can be a great place to explore differing opinions and think through your own beliefs, but you have to get people fired up... and Red Sox vs. Yankees is not going to do it. People disagree over which one is better, but there is nothing to talk about. Politics, religion, race, philosophy, Ruby vs. Python, basically everything that ought to make you uncomfortable as a webmaster... and it should, because if you're successful the debates will get ugly in a hurry. But you'll also get that cartoon where the guy doesn't want to go to bed because "someone is wrong on the internet". No one is going to stay up using your site because they've discovered someone prefers the Red Sox.

From a business perspective, in my opinion it's tight. Your CPM is going to be pretty low because you aren't that kind of niche. Probably a few cents. My gut says there isn't much opportunity for acquisition unless it truly takes off. RWW did a roundup I'm sure you saw (http://www.readwriteweb.com/archives/online_debate_sites.php) - you have direct competition from several sides. In my opinion you're competing with more than just those guys... the big poll sites are also indirect competetion, especially for your edge use case settling arguments. Poll Daddy may be your largest, least obvious competition. Data Portability is another... a lot of those types of conversations are happening in Friendfeed and Disqus now, and that trend will only gain momentum.

Routes to success... you have a built in attention base with bloggers - controversy is a major eyeball driver to their sites, and the "blog 2.0" trend - the concept that the blogger doesn't "own" the debate, they just contribute to it, could be leveraged - they will still be inclined to move the discussion over to wordpress, but less snarky about it. If you can find a way to give them what they're looking for (material to write about) you'll get some traction. Otherwise you can try to convince Harrison Ford and Steven Spielberg to debate pink vs. white grapefruit on your site. Good luck :)
I dont get it, what is it for?
If I had an intuitive grasp of statistics, I may not have a start-up!
I just joined chatterous and can't find the getting laid feature anywhere.
Yahoo Claims Record With Petabyte Database
Forbes: How To Lower Your Legal Bills when starting a company
My guess: "If you do let me out, I'll Paypal you $20!"
The released code is fairly old (I think the auto scribiding came after the last release of Arc).
The effective question is Why did PHP win over Perl for LAMP hosting? Java and .Net might as well have been on another planet.

The answer is that mod_php was a lot easier for a hosting provider to set up for shared hosting than mod_perl. Still is. The target user for scalable Perl and Python frameworks still seems to be someone with root access.
Huh?

"Django doesn't have good support for testing. Rails does."

You use the same standard doctest & unittest framework that any Python app would. Django even provides built-in support for dedicated test databases (autogenerated, and auto-cleaned-up after a test run) and a simulated web browser. Starting a Django project creates a dedicated 'test' directory that the Django testrunner looks in for test discovery. What more are you looking for?

http://www.djangoproject.com/documentation/testing/

"Django doesn't allow clean separation between business logic and presentation logic, forcing both into its "views". Rails does."

If you're doing it right, you should never be putting business logic into views. It should go in your model classes, as instance methods (for more global functionality, you can define a custom Manager and put your methods there). Django models are more than just dumb containers for database fields: you can add whatever intelligence you want to them.

"Django doesn't have migration and deployment system. Rails does."

I'll grant you that. There've been a bunch of 3rd-party efforts and some interest within the core Django team, but it's still nowhere near as mature as Rails Migrations & Capistrano.

http://code.djangoproject.com/wiki/SchemaEvolution

"Rails is stable 2.0. With Django you have to use SVN+QuerySetRefactor+NewFormsAdmin+YourOwnHacks branch to get anything done."

QuerySetRefactor was merged on Apr 26.

http://code.djangoproject.com/wiki/QuerysetRefactorBranch

I've heard NewformsAdmin is supposed to be merged soon (before 1.0) though I don't know of a target date.

This is unfortunately one of the problems with using pre-1.0 software, and if it's important that your software doesn't change, Django may not be for you. I've had very good luck simply avoiding the functionality that's under heavy development though, and the Django devs are very good at specifying what that is. I don't need the customizability of NewFormsAdmin, and QuerySetRefactor was largely an extension to existing methods.
So, This Is How Entrepreneurship Starts…? My Life In One Suitcase.
Inside 2008's hottest tech startups
This is Outlook's fault for encouraging poor email habits, like using "reply-all" by default instead of being conscious of the recipients of an email.
Because commercial LISP organizations spent too much time feuding amongst themselves and not enough time competing in the market, is the simple answer. Many commercial Unix organizations went the same way.
What exactly is original thinking, and what kind of "creatively worthwhile thinking" is not original? I was going to disagree with you, but what you said is too vague for me to even know whether I disagree or agree.
Its fascinating funny how you are so smart you can't even deign to share your wisdom on this. If I am wrong, tell me where. Given that I have spent the last several years studying/developing distributed databases if I just don't know what I am doing I'd certainly like someone as smart as you to tell me.
Hmm...is it just me or does is that comment fairly confusing?

My point was that the overall growing on their market is definitely valuable to Apple, and the fact that they are in the market of selling expensive computers doesn't influence that value. However, it does reduce the value of using the a larger dollar value as a criteria for determining market share. If the overall market for computers greater than $1000 is shrinking, then selling more in that market is not as valuable as it was may have been.

My tendency to use "it" in bad places really gets on my nerves sometimes.
Ask YC: Venture backed Startup or Bootstrapped/Consulting route?
I am curious to know the route founders/ potential founders would take, and more importantly Why?:

1.) Raise money from VC's?

2.) Bootstrap/ Consult you way through?
Uh... yes it is. All scaling problems are database problems. ALL of them. They can be solved by different strategies such as caching, distributed design, etc. But at the end of the day the problem is always that you cant get to the data fast enough. That is a database problem.
"Your examples are ridiculous. Stopping violence is not violence."

If you don't like the previous (perfectly valid) example, how about one that doesn't involve violence on the part of the perpetrator? Perhaps theft, or public nudity.

The point is that everything a state does is ultimately based on force. The whole point of preventing tyranny of the majority is to ensure that the only actions are taken are taken for the good of everyone, rather than a specific sect. It has nothing to do with denying the legitimacy of taxation.

It seems to me you don't understand the basic concept of a constitutional state. Reread On Liberty and Federalist #10.

"Who exactly do you think you're talking to if not someone who disagrees with that sentiment?"

I was pointing out exactly how far outside the terms of legitimate debate you actually are.

Now the thread is over.
Agreed. Any time you type a comment longer than a paragraph or two, you get that message.

This seems like a fundamental shortcoming of the continuation-based request approach. POST URLs should never expire.
Poll: petition for HN improvements
There are several annoying things about HN that people have been complaining about for a while. Vote for the ones you'd like to see fixed (or add suggestions in the comments and I'll update the choices)
longer timeouts for "unknown or expired link"
stop converting PDFs to Scribd (or make it optional)
downmod (or flag) option for posts
search (integrated into the site)
I guess that depends how much you can infer from such a comment. I would infer that the aversion exists because it's confusing and many times not helpful to cherry-pick criteria. Of course, I'm inferring that because it's my opinion on the subject and it may not be what anyone else actually agrees with.
You can do anything.
Uh, wouldn't this actually be called something like "the US Department of Education"?

It's been a rousing success, too - it's allowed our Ministry of Truth to claim things like "comrades! Our production of students that pass the standardized test is up 85% this quarter!".

It's all about families, man. Good families teach people what's really important - not federal departments of anonymoustaxeatingbureaucrat.
I created a poll http://news.ycombinator.com/item?id=196196 with some of this issues, I'd like to see how many people care about each one and maybe we can convince PG to start fixing some of them (or release updated source code and accept patches)
> though it was kind of fun learning about Greasemonkey

Pretty much the only reason I did this too. ;)
With all due respect andrew why dont you share your technical analysis of what exactly I am saying that is wrong here. Btw, have you ever actually worked on any database scaling problems? Just checking.
I don't use Twitter, but as I understand it, it's a massively many-to-many messaging service, where it's not unusual to follow hundreds of people's tweets and have thousands of people follow yours. Can you imagine the performance if every message required hundreds of disk writes as the "received" copies get spread across a hundred shards?
I don't know how much I trust the $75 figure. Three years before its launch, the XO-1 was predicted to cost $100, but the manufacturing cost is currently upwards of $175. I have my doubts that the XO-1 will even be as cheap as $75 by 2008.
A lot of Python projects seem reluctant to announce a 1.0 release. For example Pylons and Django are both in 0.9 range, SQLAlchemy is 0.4, Genshi is 0.4 (just some of the projects I've used in a commercial setting).
Here in Ottawa, we have express busses that service the exact same route that the regular busses do in some areas. I found it worthwhile to get an express pass ($20 more / month) simply because you'd be guaranteed a seat on the express bus and guaranteed to stand on the regular bus.
The define-input syntax makes form handling almost as easy as PHP with register_globals on, while at the same time being as secure as PHP with register_globals off -- the best of both worlds. It's arguably more secure because of how conveniently one can use validator functions.

The sql-repeat syntax with group-beginning? and group-ending? makes sophisticated use of result sets much more accessible than looking behind/ahead a row manually.

Hoisting a function definition you quickly made within a page out to a sitewide definition is fast and easy, so doing something quick-and-dirty doesn't lock you into dirt forever.

And don't underestimate those square brackets. They totally optimize the human "inner loop" of web development. They beat the tar out of <%, %>, and the mess of quotes, backslashes and dots that constantly trip up newbies.
> So shard more, or get more disks, or get flash

Your intuition is excellent. These are all valid techniques to alleviate disk seek limitations. They are likely to be successful. Unfortunately, they're expensive options.

You can partially overcome disk seek limitations by performing database batch inserts. So, rather than inserting messages indiviually and updating indexes for each message, you insert 100 messages and perform less disk seeks updating the indexes for each batch. Unfortunately, it can be hard to reliably merge data into suitable batches.
http://news.ycombinator.com/item?id=175378
I think she set up an email list. The article says that all recipients received replies and I'm hoping that dozens of VC's know better than to hit the reply all button.
I really doubt that touch typing on a touchscreen is feasible, especially for young children who have limited motor abilities. The existing XO keyboard is hard enough to type on as it is.
> Be careful when you use this thing that you don't unwind the loop so much that you overflow your machine's instruction cache.

I considered this case shortly after posting. Specifically, my rule of thumb fails miserably on the 68020 which has a very small instruction cache.
I considered this, but given that the messages are small and the primary issue is scaling not storage this, it does not seem to me is a good solution. The entirety of everything ever written on twitter is probably not big enough to worry about storage size. The issue is performance. By the way, this architecture is essentially how email works. All your inbound messages are stored in one place separate from outbound messages from the sender. This allows for infinite horizontal scaling.
Scalability is important, but it seems the raw speed of websites to any particular user gets less attention than it needs. There is quite a bit of data that slow sites are much less attractive to users, even if they don't consciously realize it.

So if your response time is bad, you may not even need the scalability ;)

If you use bigtable, the speed is mostly determined by the number of sequential data accesses that you have. You should strive to have just 2 or less. The other advice in the article still applies.
Let's say that 127 million people don't like the Mona Lisa. What does that tell you about the quality of the painting? Nothing. That's why downmodding is a bad idea.
the idea that ruby is a blub language while python is a foo language is laughable. Python has it's issues as well. New style classes vs old style classes. These types are treated differently than those types in terms of default behaviour. Decorators are not the easiest thing in the world to read. Metaclasses are another example.

In my opinion ruby's syntax is simpler from a users point of view and equally as powerful. This may be my bias, but a fair assessment is that they are fairly equal
I opted for the simple get rid of scribd or make it optional since the overwhelming majority in your poll wanted some option of getting the original PDF.
You don't store it twice - you store it hundreds of times, once for each recipient. Twitter isn't person-to-person messaging, it's broadcast.
Including systems that don't use databases, or are you defining a database loosely as "anything that stores data"?
That's why I didn't vote for that option myself. People keep asking about it though. One option would be to let users flag content as not appropriate for HN, and then have a per user option of filtering away content the community doesn't find appropriate.
Yeah, but when your partners are using inferior tools, you spend a lot of time waiting. OurDoings integrates with 9 other sites, so this means something to me.
Unless it's a problem with a bad algorithm slowing down the processing of the data.

So not all scaling problems are database problems.
"All scaling problems are database problems."

This is an argument in semantics.

You suggest storing all users physical addresses in memory across multiple machines. You also advocate "shard splitting" a la B-tree indexes which split when the node's population reaches 2n. OK, so when a split occurs, how do you maintain that massive table in memory. Do you lock it? Do you examine your existing queue of requests (which may soon point to the wrong node)? How do you handle new queue requests during "shard split maintenance"?

Calling these "database problems" is waving your hands. Why not just call them "computer problems"?
I agree with all of his points, but it was a very slim article-- I was kind of hoping for a more profound analysis from someone like Doctorow.
Actually, this is the beauty of the design: All of the writes happen in parallel. So there if you write to 2000 recipients, but all of the writes happen in parallel you have no bottleneck. And any time a given machine's load gets too heavy, you do a shard split. You really dont care how much writing you do as long as on every machine you write to is not being hit too hard. So if a given server has lets say 10,000 users on it. And each one of those users receives 10 updates an hour. Then it would mean handling, lets say 100,000 writes in an hour, or 27 writes a second. If this was the threshold for this machine (which would be low) you then split the shard so that it then is only handling 13 or 14 writes per second.
"Given that I have spent the last several years studying/developing distributed databases..."

That doesn't make your opinion of what's wrong with Twitter any more valid than the crap TC keeps printing unless you happen to also have lots of internal details of the Twitter architecture.
http://www.nicovideo.jp/
helloTxt.com is useful.
Yes, I am defining database as "anything that stores data" It does not have to be SQL for me to consider it a database problem. And in fact my solution is well above the sql level.
there are many search options available, which could be at least linked too... but if you don't know them you'll have to search for them :)
You should check out Mako, a templating system for Python. Embedding Python (or Python-like) code in HTML is not as bad as one might think.
Most scaling problems are database problems. Not all.

For example, I load tested my last rails app (load testing is my background-I have consulted for major pharma, tax, and other companies) and on a wimpy inherited server, processor maxed out first for the rails processes. I popped in a second CPU, then was able to get to the point where we could support 200 concurrent users, which was all we needed at the time.

I have seen limitations on bandwidth, firewall problems, load balancer misconfiguration, poorly written queries, memory usage,processor usage, thread deadlocks, and licensing limits (max database connections on a trial license) all be bottlenecks. At the larger scale, yes, database is more often the problem, but it's not clear twitter should be using a database in the way they are, anyway. Your statement goes too far.
So, it's scalable, but the total number of writes/sec necessary is O(messages sent/sec * average number of recipients per message). I could see this working as long as the average number of recipients per message remains constant as the service grows, though it's still a lot of hardware. How much funding does Twitter have again?

If the average number of recipients grows along with the number of users, then there's a problem, because average number of messages should be growing linearly with userbase, so the total hardware required is quadratic. No way that would work out economically...
It also looks like it's at version "works for me," so you're probably out of luck if it doesn't work for you.

There's actually a fair number of these popping up as of late, webkit2png (http://www.paulhammond.org/webkit2png/) and khtml2png (http://khtml2png.sourceforge.net/) are the other ones I've seen recently...
There's gnome-web-photo that uses Gecko.
Ask YC: Code Digesting
In a 1986 Bill Gates interview linked to by an earlier YC post (http://news.ycombinator.com/item?id=194844) was the quote:

"I still think that one of the finest tests of programming ability is to hand the programmer about 30 pages of code and see how quickly he can read through and understand it."

Do you agree this was true in 1986? Is it still true now? If your two answers disagree, whence the difference?
Well ok. I dont have to call them database problems, but I object to someone saying they are not database problems. I agree it is a bit of semantics and I would not have brought it up if he hadnt. Although I have to say I consider this work (and the good questions you ask) to be in the database domain.

Regarding when shard split how you maintain the table, the easy answer is Terracotta. It is amazing for this and handles all that kind of stuff totally in the background. If you didnt have terracotta it would be a harder problem indeed. It handles locking and all of that magic for you. I highly recommend it. Regarding how you handle queue requests while the split is going on, what happens is that while the split is happening, the original shard continues to respond, but all new updates are sent to the old shard and the new one that is being created in the split, so during a split there is addition messaging going on. Once its done, the original shards range of users is cut back, but actually it is still storing all the old data. At that point you can delete the data that has been moved to the new shard - though this is not necessary.
These design techniques could very well make a nice impact on performance. Then again, they might not.

Why? They are proposals to solve a problem that has not yet been defined.

Fact is, you don't know what the problem is.

So proposing a database solution to what could just as easily be a memory management or throughput problem is premature.
Poll: Social messaging - which one...
Choose only one please. Also, if you can say why.
Suggest others and I will add them to the list.
1. twitter.com
2. jaiku.com
3. hellotxt.com
4. pownce.com
Smartphone users obsessed with Craigslist, eBay, MySpace, Facebook
5. socialThing.com
I don't underestimate square brackets, in fact I liked them, but i think the main problem is that BRL is Scheme.

If Scheme is not popular among web programmers, there must be a reason behind it, more than one reason, maybe a hundred reasons. But it's a fact. So I think if you are going to bring Scheme/Lisp into the web development market, you (I mean, not you personally, but in general) have to bring 2-3 new killer features along with HTML embedding and the language itself.

I'll try to understand BRL better, but from what you say above, those things don't seem to be really killer features. Even square brackets thing isn't a killer, I think.
6. brightKite.com
I don't understand why there should be any timeout for #1.
true, but it makes me curious what brilliance someone bring to the table when they have nothing specific to say other than you are wrong. Also given that much of what I am talking about is running code, I at least know that it solves a major class of problems. If it doesnt apply to twitter it should be pretty easy to explain why.
I think that's a bit true but I also have to believe that from a syntax perspective PHP has always been easier to use. That and all the libraries are pretty much built into php. you never had to deal with CPAN - it was either built into PHP's core or you lived without it (generally).
Democracy sucks :/
That happened when I answered pg once. I always thought he just erased my comment because he thought I was a redundant cheesehead.