skysight-inc/HackerNews huggingface dataset:
"and all of that magic"

Right. Just because we didn't write it doesn't mean it's magic.

There is still a lot of work to be done. It can get very complicated for high velocity apps. And it takes time.

So you're right. Bigger minds than the two of us are probably struggling with this (and lots of other stuff) as we write.
7. friendFeed.com
no duplicates on RSS feeds.
This is true. Though I can say that if one were building twitter and you didnt either have a design like the one I suggested, or a design with some kind of distributed caching then you are going to have exactly the kinds of scaling problems they have had. In other words if you use tradition techniques for something like twitter you are screwed. However you are absolutely right that (as I said in my piece) they may already, for example have thought of all of this and either implemented or dismissed it because they have a better design, and I have no way of knowing.
8. iminta.com
"Here's how that works: imagine that you've got a disease that strikes one in a million people, and a test for the disease that's 99% accurate. You administer the test to a million people, and it will be positive for around 10,000 of them – because for every hundred people, it will be wrong once (that's what 99% accurate means). Yet, statistically, we know that there's only one infected person in the entire sample. That means that your "99% accurate" test is wrong 9,999 times out of 10,000!"

No, it means that the "99% accurate" test is wrong 9,999 times out of 1,000,000. It would be clear to anyone when stated that way. What's counterintuitive is the author's statement of the result, not the result itself.
Yeah, exactly. The two projects have a completely different dynamic. Like Python, Django development is cautious, minimal and relatively slow-moving compared to Ruby and Rails.

Rails burst onto the scene, brought everyone on board with a 1.0, published books, and already decided enough design elements needed to change to warrant a 2.0 release. Django, despite having the BDFL's semi-official endorsement, has crept along, nitpicking at its own API, cranking out beautiful versioned documentation and tutorials, and writing and editing the official Django Book in a wiki. And only now are they feeling confident enough to consider the 1.0 label and even publish their precious gem of a book.

Same timeframe, completely different personalities. This played out somewhat similarly with Python and Perl a decade ago, which leads me to believe that Django's time in the spotlight is still ahead of it.
I'm all for flagging the content. It'd be especially nice if it'd throw up a warning in front of the summary if enough people flag it.

Digg does pretty well in this regard. If enough people bury the story as inaccurate, you see a big "Warning! The contents of this article may be inaccurate!". That plus scanning a few comments are helpful for deciding not to waste my time on an article.

Suggested flags include: 'inaccurate', 'nsfw', 'not suitable for HN', etc.
The infographic that saved a million lives
How about a dating service that is.. well.. twitterlike. Just launched, so there's not much there yet. http://www.TalkAboutADate.com. Can't promise it'll get you laid. Anyone know of an api that'll help you do that? :)
Is #2 ("Unknown or expired link") a bug or a feature? It sure feels to me like a bug.
I dont understand. I am saying that Terracotta, which is an open source product, handles the in memory table and synching stuff I was talking about and that you asked about. Its like you dont have to write a C compiler and you dont need to write linux because someone has done the heavy lifting for us. So as far as that specific issue my point was just that the solution is available in an excellent well tested product that lots of folks are using. By the way, I am actually working on exactly these sets of issues -- not to say there are not bigger minds than mine working on high velocity apps, but it is what I spend most of my time on.
To do the writes requires a lookup. To do a read requires a lookup. That's how sharding works. If not a lookup, then a hash.

This doesn't even get into the complexity of scaling your model from 1,000 or 10,000 users where your brute-force methods seem like they will work and applying it to 10,000,000 users where your model will die a horrible death, kinda like Twitter is today.
9. tumblr.com
Right. Also, people are much more inclined to try mold the front page into their idea of what the site should be using down arrows than they would any given comment page.
Where does he get that $30 amount? Every single technical book I've bought has cost me at least $50. Except for the Learn C++ in 21 Days book, which was on sale at Wal-Mart for $20.

I admit that the price is worth it in some cases, but $30?? Where's he find those books?!
pg is using this site as his test bed for arc. I don't think he wants it to be community driven and improved (yet).

I know arc itself has two versions one developed by the arc community (anarki) and then a separate one developed by pg.
JSR 292: Supporting Dynamically Typed Languages on the Java Platform - Early Draft Review
The problem is that some pdfs on scribd are rendered in a way that makes the text fuzzy and almost unreadable. The only way to read the text in those cases is to load the raw pdf.
Do people say things like "srsly u shd try ruby on rails sumtime. ttttyl8ter" ? ;-)
Drinking The Erlang Kool-Aid
Is that your site?

If more people contribute this would be great. (and maybe some design work)
The choice seems to be, do you plan on building a leveragable product or not (http://blog.pmarca.com/2007/06/the_truth_about.html). If the answer is no, then you bootstrap, if the answer is yes, then leverage the hell out of it -- and so you need $$ from VCs.

My personal choice is to leverage. Riskier, but more fun.
Let's say that 127 million people like the Mona Lisa. What does that tell you about the quality of the painting?
At the linked page he says he doesn't want to explain how he did it. (With a terrible reason about learning to respect not knowing stuff. No thanks. I want to know.)

So I don't see how just asking him is going to change his mind.
"Progressivism is a form of social insurance. It is an acknowledgment that perhaps, through no fault of my own, I might have been born poor, or with skills not suited to the modern economy. It is the understanding that in the richest nation on earth, no one should want for food, shelter, or healthcare, no matter what their ability to contribute."

No, progressivism is nothing more than epicureanism, transported into the present. You propose building a world where everyone is free from pain and struggle, without realizing that this builds a society of child-men, unable and unwilling to do the hard work necessary to advance civilization, each clamoring for more and more handouts from the ever-bountiful teat of Government.
A science? I'm dubious. Too many factors to treat it very scientifically, or be able to repeat 'experiments'. Maybe there's room for some of that approach, but... it seems way too full of squishy human factors to get all that scientific about it.

> Good business books are very rare.

Any titles come to mind?
I have. I have managed systems that handle millions of clicks a day in databases and billions of dynamic (PHP) pageviews on a daily basis.

I am currently managing a system that serves over 100,000 DNS queries per second, each of which is dynamically evaluated as to how we respond to it. And each of which is logged, amounting to over 7,000,000,000 log lines a day, all which get processed, parsed and in many cases handed over to a database store. Yep, we're dealing with BILLIONS of rows of MySQL per day.
Original thinking being defined by novelty: new ideas and synthesis. Creatively worthwhile meaning good, usable, aesthetically sound, having value beyond novelty. The latter is a subset of the former. Or: not all new ideas are good ideas.
Heh, I picked up Competitive Strategy and am not sure what to think yet. It's fairly slow going in some ways.
42 Awesome Business Card Designs
10. beeMood.com
ability to change our votes
http://en.wikipedia.org/wiki/Bayes'_theorem#Example_.232:_Dr...

Basically, an extremely low rate of incidence (in Cory's example, 1 in a million) will render a "highly accurate" test meaningless.
If windows sucks, why are there so many successful...?

If Chinese writing is so hard to learn, then why are there so many...?

If religious explanations for the origin of everything sucks, why do so many...?
Riding on the subway earlier today, a guy got on an starting preaching how "the lord saves" and "you must embrace Jesus." I wish math could create equivalent evangelism. I'd love to live in a world where a guy gets on the subway and starts preaching the Pythagorean theorem?
iPaper: It's like paper...jabbed into your eye.
Bill Gates live at Advance08 (Advertising Leadership Forum)
Umm, hash functions ARE NOT used for crypto, one way functions are, and they are not hash functions.
a more basic problem is just plain ignorance.

many if not most citizens of the USA do not understand the basic organization and functions of their government. most of them cannot name a single sitting supreme court justice, have never read the constitution, do not know how a bill becomes a law.

and these people vote.
Will you bet on VenCorps? - A new 'wisdom of crowds' take on YCom level funding?
I think your math is off here.

Basically you end up with this equation

cost = ((NumOfUsers * avgMsgsRevceivedPerUser)/maxWritesPerCPU) * costOfCPU

There is nothing quadratic here. Your are able to scale linearly with the number of messages received in the system. There is a fixed cost per message received which is exactly what you want. By the way, this is exactly how the internet mail system works and it is why it is so scalable.
That one's not the problem. Perhaps Bayes' theorem?
Well, my mom has been trying to make me start a company since I was 16 and told her I wanted that someday...
uhh... yes it requires a lookup. There is no problem with that since the lookup is not centralized but local. How... read about terracotta. No I dont work for the company just use their open source product.
If you want a profound analysis, he wrote a (fiction) book recently on this subject, and put it out on the internet under a CC license. It's an enjoyable read; he pretty much distilled a few bits from the book to make that article.

Here tis: http://craphound.com/littlebrother/Cory_Doctorow_-_Little_Br...
Thanks. I read the page and I still don't get it. You might use self.my_function() or self.my_variable. When would you see selfSelfself? Or is Yegge being a poet here?
Neither am I -- but I am from a culture where tipping is a social convention.
That's not the biggest issue for me.

Often the PDFs that get posted are interesting enough for me to want to archive and read more in depth on the bus, when I don't have an internet connection. Scribd makes this a royal pain in the arse.

Also, I strongly prefer the native feel of a decent PDF reader over a slow Flash UI running in the browser. (no, Adobe doesn't count as decent.)
Here is my guess: because it's Arc with closures and web closures have a timeout. It can be long or short, but hardly infinite.
"Ask HN" as its own section for giving discussion/connection between members a place with more focus :)
Nevertheless you believe that there is no way to handle a distributed in-memory lookup system. It just goes to show you cant know everything.
It really does depend on the book. Looking at Amazon, the Django book, JavaScript: The Definitive Guide, The Pickaxe, CSS Mastery and The Ruby Programming Language are all <$30 each and are all well regarded. I'm sure there's plenty more.
That sounds understandable technically, but if there has to be a timeout, perhaps continuations aren't a great idea for this sort of thing.
Bug in Moody's Program Issues Erroneous AAAs
On "SEARCH!":

All HN posts show up in Google unbelievably, even suspiciously quickly. Not sure if it's a hack on Google's side or it's popularity, but a link to Google with "site:news.ycombinator.com" in the request would do perfectly.

(BTW: http://www.google.ie/search?q=%22Poll%3A+petition+for+HN+imp...)

Edit: Notice what Google's cache says: "Poll: petition for HN improvements. 4 points by cstejerean 32 minutes ago"
If that's the case, closures don't seem like a good idea in this sort of context.
I personally would choose scribd over PDF, but the firewall my employer has in place blocks scribd. I still want to read the linked document so PDF becomes a necessity unless I use a proxy which could get me fired. Maybe that'll jump start my startup. :D
Hank, your design is so high level as to be almost useless to anyone who'd actually benefit from a proposed design. It's not clear what issues you feel Twitter has, or how exactly this architecture would solve them. Usually your blog posts are more cooked than this one. I'm a regular reader, and at first I honestly thought your post was satirical.

As I read your design, one of the biggest problems is that to build up the infrastructure I think (and infer based of my experience with designing these kinds of systems) you are describing is that it's a massive engineering effort in and of itself. You basically end up rewriting a realtime-optimized BigTable or HBase, and that's much to much to expect of Twitter's small engineering team.

People seem to forget that scalability is a tradeoff, like any other aspect of a service. You devote an amount of resources to the problem that makes sense to your business. Despite twitter's downtimes, people don't seem to be leaving the service, despite several competitors. The service is down, but it isn't down so much that it's unusable.

For a site that's grown at such a rapid pace and has such a small engineering team, I think Twitter's scaling plan is probably very reasonable. They're growing their capacity at a rate they can afford, and a rate that they know will retain most of their userbase.
Let's say that 127 million people LIKE the Mono Lisa. What does that tell you about the quality of the painting?

For your argument to be sound, it has to apply equally to upmodding as to downmodding. So might as well get rid of modding entirely and just have all the crap come in unfiltered?

It is true that if a lot of people downmod something, this does not necessarily mean that thing sucks.

But it is also true that A LOT OF THE TIME when a lot of people downmods something, that thing does in fact suck.

Hows about we add downmodding and also tags and a way for each user to specify his/her own set of filter parameters?
But you are not going to write a comment infinitely, are you? Wouldn't 12 hours be OK, for example?

It's about memory allocated for the hash table. If they add some RAM and change the timeout to 12 hours you won't even notice what's going on under the hood.

And closures are nice from programming perspective. Should I say it's almost definitely the future of server-side development... ;)
The Perl community was too unfocused on the web. Perl was, and is, a general-purpose language built as a replacement for sed, awk, and shell scripts and later retrofitted for the web. PHP was built for one task, which gives it a certain coherency that is a major strength, especially for newbs.

You learn to build a web page with embedded PHP on page three of most tutorials. I don't remember the Camel book covering that at all. If it does I didn't notice -- too busy slogging through the regexps and the hash syntax and the insane special characters and the bizarre function-declaration syntax. And I fear that, in Perl, there is no standard way to do it. There are probably 75 standards, which is helpful, but not very helpful.

If I visit any random web forum about PHP I am 97% likely to find a discussion of website development; if I visit a random Perl forum I'm just as likely to find sysadmins debating how to automate database backups, or a linguistics debate, or a game of Perl golf.
Text entry on mobile devices is not convenient. Navigating large and/or complex documents is not convenient either. Mobile-specific services can exploit these inconveniences to make money.
How to get current events Search Engine traffic
The phrase "hash functions" refers to a general category that includes "cryptographic hash functions" (which are hopefully one-way) as a special case.
Do you believe this questions share the same answer?
Do you not understand that dragging a person to jail, when he wishes to stay in his home, is violent? That physical force exerted on his body, against his will, is violent? Do you not understand that it hurts people? Do you not understand that demands, backed by guns, are violent?

Never mind whether it's for the best in the grand scheme of things, or whether a superior alternative is known. Even if this is best it is still violent.
good case against data mining... and the "if it saved one life, would it be worth it?" argument doesn't change my mind.
Why do you think of parentheses as something about LISP that needs to be fixed? What exactly is the problem?
Well perhaps its a matter of staring at something so long it becomes simple. But I really don't think what I am suggesting is that hard. The main thing that requires some thought is shard splitting. That is not a big piece of code, but it does admittedly take some thought to get right. Aside from that everything else is provided out of the box by someone else like amazon web services or terracotta. I am not sure how I could have been more precise without source code.

It is certainly reasonable to suggest that Twitter is good enough as it is because people still use it, though the grumbling in the market suggests that that may not be something they can bank on forever.
http://www.lrde.epita.fr/~didier/research/verna.06.imecs.pdf

There is an easy work around for this. Submit the pdf link. Then comment with the pdf link. That adds one level of indirection, until pg gets around to fixing it.

This is hacker news, So I'm assuming people can use their browser and copy and paste.

1) copy link location.
2) paste link location in address bar.
2) remove scribd.com crap, leaving the original pdf link.
3) load original pdf link.
ADD some or all of a post (not just the title) on the RSS feed
tinyurl is a great idea too.
Nope. I'm just demonstrating how easy it is to come up with questions that have this structure, but which do not necessarily have the answer, "Because, really, it's actually awesome!"
"And there are still ASP developers around, I just recently worked with a company that still writes all their stuff (websites for paying customers!) in ASP"

The company I work for has about 4000 customers on an application using ASP / SQL Server 2000. If it works and it does what people want, they really don't care or know even what language it is in.
I was looking for something like this just last week.

This is a web service that will produce web page images, but it's useless to me because they don't specify which browser's rendering they're capturing: http://webshots.velocitysc.com/
My co-founder tried out some of the techniques in the paper on our system: "I was skeptical, probably because I've never done much performance tuning like this. But I changed (triples) to create a 2-d array of fixnums instead of the default list of lists from the select. I added the declarations. It's about 100x faster. 100!" Very nice :)
Google does a good job at indexing HN quickly. There are several problems however since Google's ranking algorithms aren't necessarily the best at ranking content in HN. SearchYC addresses some of these problems and I like their solution, but I would prefer not having to leave the site to search. Given that search dominates the top of the list in "Feature Requests" I'm assuming there's a good number of people that feel the same way.
i'm pretty sure pg considers the no downvote thing to be a feature, not a bug. it was designed that way on purpose.
Statistician Peter Donnelly on the same subject:

http://youtube.com/watch?v=kLmzxmRcUTo
A similar previous post:

http://news.ycombinator.com/item?id=166880
100x Lisp Speed Improvement (Using Lisp faster than C paper techniques)
It's incredible to see your post get so much karma given that your argument is so wrong.

You really should learn some statistics yourself - total revenue for the government happens to be a function of tax rates. Using examples such as "software piracy" just destroys your credibility.

Journalists in general are terrible with statistics as is the general public because the concepts of statistical significance and correlation versus causation are lost upon them.

In this case however, the article isn't making a statistical fallacy. Given the tremendous portion of government revenues collected from those in the Top 1% of income, it makes sense to analyze whether there is any sensitivity between the highest marginal rate and total revenue.

Furthermore, it makes sense to ask whether things like marginal tax rates and capital gains tax rates have an impact on GDP growth which in turn effects revenue growth for the government.
I think the scribd auto-link is awesome. I hate load times and absurd browser pauses (thanks, Adobe), and scribd cuts that down significantly.
As someone who deals in diagnostic tests all day, nobody's diagnosed on one test. There's a similar problem I worked out on the odds of getting into medical school, I'm sure this is elementary stats and has someone's name on it, but I'll call it the acceptance problem: how many medical schools do you have to apply to in order to have 90% chance of getting into medical school?
For any school there is a acceptance quotient:
Q = (acceptances sent out)/(number of applications received)

For any given student applying to some schools 1 thru n, the goal is getting at least once acceptance, and applying to more schools, mathematically, can't possibly hurt in the closed case (neglecting social engineering, time spent on applications, etc), so the chance of acceptance, Ca, approaches 1 with every new application in the following fashion:

Ca = (1 - (1 - Q1)(1 - Q2)(1 - Q3)...(1 - Qn))100

There's a visual and some worked out examples here:
http://nielsolson.us/MedSchool/

Similarly, if the sensitivity of a test is 90%, that means the test identifies 9 of every 10 people with the diagnosis. If I administer n different tests each with a sensitivity of S, then the chances of accurately diagnosing the disease, Cd, goes up* with each additional positive but never gets to 1.

Cd = (1 - (1 - S1)(1 - S2)(1 - S3) . . . (1 - Sn))100%

So lets say you are doing, say, genetic testing, and any one gene is 1% sensitive for the disease. If you tested 300 genes you could be no more than 95% certain of the diagnosis.

(1 - (1 - .99)^300)100 = 95.09591...%

Now, if your genetic tests were 5% accurate, you're panel could be no more than 95% accurate with 59 tests.

If your test was 50% accurate, you're panel could consist of 6 tests and be no more than 95% accurate.

Of course, if some of the tests are negative, things get more complicated. One of the problems with these data sets is that we have no idea how predictive they are. You can't even calculate the predictive power of the database. There simply haven't been enough events. Then we get into surrogate measures (how many were positive on tests 1 - n and were found to have razor blades in their homes, etc).

The claim that these databases can't be effective isn't true. They could be. P might also equal NP. Whether the hypothesis is strictly true or not, the vague but real set of 'practical concerns' suggest that the truth of the hypothesis is sufficiently difficult to test as to render the null hypothesis the de facto assumption until proven otherwise.
How did the Commerce Clause protect me from being regulated by the Federal government?
1. They are doing it with logic.
2. Do you consider the New York Times, or Washington Post, or any other major daily to be the newspaper of the poor? Give me a break.