skysight-inc/HackerNews huggingface dataset:
Before you start writing code and making architecture decisions, you need to do some sanity testing. Can you even run 15,000 SQL inserts per second of the format you want on a single machine? If not, you may need to abandon using a database entirely. You can test this with a simple script and the results might invalidate your whole plan.

Also, I'm betting you don't actually need to support this kind of load to do what you want to do. In particular you can probably batch up data, since you are going to have latency issues anyways. You need a specific target for the latency you need, and then do some measurement figuring out how much batching you can do.

Why don't you explain your idea in more detail? The community can probably be more helpful then.
That's a language issue and not a people issue. C also allows you to assign variable in if clauses, which can cause hard to find errors.
Bustup, not Bailout
Sufficiently advanced incompetence is indistinguishable from corruption.
I'm a programmer writing accounting software... not always exciting work but there's no shortage of work to be done.
I actually checked yesterday if that was still available.

I need to go outside more :p
Photovoltaic Cells Vs Photosynthesis Process
Photovoltaic Cells Vs Photosynthesis Process

Do they have anything in common?
I dunno. I've given Seth an honest chance in the past, only to be let down by trite, obvious ideas and platitudes.

Is Tribes somehow different? Anyone read it?
If my university only had a 5% drop out rate, I would have studied at the wrong university.
If everybody get's a degree, everybody is equal and your degree is worthless.
VMware's Biggest Threat isn't Microsoft
So, what you want to do is stream the file to disk. The problem that most people will face is that someone is uploading a 100MB video file and your code is trying to hold it in memory. Bad! Get it on disk, then deal with it by opening the file.

In terms of a daemon, you don't need one. PHP and other languages can execute other processes. So, you want to convert that AVI to Flash? Save it to disk, then execute another process to convert it.
I think he has worked with badly designed and maintained enterprise systems. If only one system is talking to the database then the Data Model should map directly to the Database, it's only when you have 20 systems each with their own Data Model that all use the same Database that you have the types of problems he is talking about.

Edit: That or you have stupid people building the Database. Remember just because SQL is easy does not make the scema unimportant.
1) Click through, read, upvote post.

2) Look at comments to see what HN denizens have said. See post is a duplicate. Upvote comment noting that.

3) Think to myself, "I should've checked comments before upvoting post."

4) Think to myself, "Why is it incumbent upon me to do this? The site should do it for me."

5) Think to myself, "There is an interesting analog between a social news site and ebay. At any given moment, there is some market for a given story or item (respectively). The size of this market varies around some mean, through time. The traction that a particular story/auction receives is a function of the size of this market."

I have a friend who makes his living off ebay, and he likes the store a lot better. He says he's getting triple what he would for small-market, rarely purchased items than he would if he'd posted an auction for the same thing.

A store averages the size of the market through time. An auction samples the size of the market for a much shorter time, and so experiences more noise in market size.

A social news site has some algorithm to let stories with traction decay from the front page as a function of time. The faster this decay, the more approximately instantaneous the sampling is of the size of the market.

So went a semi-random, diverging-from-topic chain of thoughts during lunch break. Maybe others will be interested.
Regardless of whether Amex needs the money, if their competitors are getting it, they've basically no choice but to follow suit or find themselves at a disadvantage.
A great graph showing the dropoff:

http://www.spamcop.net/spamgraph.shtml?spamweek
SONY ERICSSON W760I BLACK UNLOCKED
Sony Ericsson W760i Intended to deliver advance 3G features and crystal clear image capturing with 3.15 mega pixel camera. Equipped with a 3.15 mega pixel camera ensure that you’re always ready to shoot and share the best possible images.
Smart money seems to be in great scarcity these days. Those so called smart Wall St firms are the first to line up for tax dollar handouts. Those firms that should be in chapter 11 right now for their stupidity are the ones you and I are now involuntarily working for by proxy of our government.
We use the same internet but other than that no :)

We're more focused specifically on dialogue between company and customers (or employees) whereas SlinkSet is more generic and has more tools for aggregating 3rd party content instead of user ideas.
Also, what processing is actually going on with the data? Just updating the data in storage seem like less than half the problem. You're going to need an efficient system for parallelizing whatever processing jobs need to be run.
no .gov extension is the first clue thats its not real
I had a pretty poignant moment eating food in China. It was a chicken dish with some bones in. Lots of vegetables, flavorful, exceptionally fresh and delicious, all that. Suddenly I realized that the bone I was biting into was beak. It tasted like chicken.
This doesn't really make sense. If you have 100,000+ members online, each posting at least every 30 seconds, you are going to have way more user-provided data than you can possibly show to everyone. So why do you need to know the exact relative position of the visitor's new database record?
This is off topic, but this seems to be something PG would be interested in. I would like to see his take on the idea.

www.overcomingbias.com/2008/11/fund-ubertool.html
Yes, definitely. I don't want to use Paypal for anything.
Twitter: One Billion Tweets. Wow.
Origins of Online Blackjack
Boiled water freezes faster because the impurities have been burned out
You are mistaken and it's not true. If your site is php, let php handle the uploads.
Pivotal Tracker: The iPod of project management software
New Google Search results pages for iPhone
Multiple applications talking to one database is the common case.
Why not? They aren't trying to make a living off it. Some people are happy just to see their work used in the real world, and that's okay.
Forget Red vs. Blue -- It's the Educated vs. People Easily Fooled by Propaganda
I used to hang out with the guy who made this site. I think it Slashdot'd soon after it was created. That brought down the school servers :). This was done for a technical communication course that all Tech engineers have to take.
Rideshare website ruled illegal
How about saying something marginally intelligent. ;-)
This is fixed, by the way
FYI this was written by Chris Hedges, the same guy who wrote the book War Is a Force that Gives Us Meaning. I only mention it because it's a really good book.

That said, his literacy statistics come from Gatto's analysis for the 1993 NALS, even though there was a new one released just a couple years ago. And also, his other literacy stats come from here:

http://bookstatistics.com/sites/para/resources/statistics.cf...

While these numbers are probably roughly correct, and I have cited them myself in the past, they really do need to be verified.
IE 6 for Windows Mobile... what an unfortunate name
One really easy way that works extremely well is to use an old school CGI script to handle the upload. It will die as soon as the upload process is finished, which keeps things very self contained and clean.

The most important thing (as others have noted) is that you do processing asynchronously. Get the file on the server, queue it (however simply), and then process the uploads in however big of batches your machine(s) can handle optimally. 99% of the time you're going to want ffmpeg for videos and ImageMagick for images.
Completely agree. My mother is happy with Ubuntu too.
Garbage article.

It's a slow down, but it's hardly a collapse. Have some perspective please.

And the US debt level is not the big when you compare to other countries.

And recalling troops isn't going to help much. The money you give to soldiers gets put back into the economy anyway.

If the US really was worried about it's debt load, it would just inflate it into nothing. No need for bankruptcy.

That's an interesting form of tax: instead of direct tax, just borrow money and inflate it. You can keep doing that forever, and no one ever has to pay tax.
Plus the Death Star was full of really dense stuff that was fuel or some ship
Ratings are relative to the rest of the world, currently, while the US is doing really really bad, everyone is doing much worse.
No to what? I guess my point is that the best background to have is one that's interdisciplinary. Philip did a lot of stuff at aD that wasn't engineering, but really sort of like sales - evangelism, recruiting, etc.

The fact that I have a B.A., B. Sc., MBA and ABD have nothing to do with my contention. Of course, it's also better not to be old ;-)
"new record is appended, a search is performed and the post field is sorted, then the relative position of the visitor's new database record is identified in the results (example 49,501 records from the top)."

It might be a good idea to try to twist things so that no sort is necessary - instead of appending and then sorting you should try to insert the data right at the spot it should be. If your constraints allow you to not use a database, you could consider a data structure that inserts data in such a way that it is always sorted.

Also, as an improvement to your point 1, you should put most of the JavaScript itself as well as images, etc somewhere else - with this your bandwidth will be used only for the dynamic data (which will be more than enough).

Finally, the best tool I know for scalability are queues. Try to reorganize what you are doing to arrive at an architecture where you put things on queues that multiple consumers will process.
I'm not showing them all that user-provided data, all I'm showing them is the position of their last post relative to the position of 'another post' in the database.

The 'other post' changes from time to time based on an algorithm that takes into account all the currently existing posts. This means I must determine the position of the 'other post' each time, and then relate it to the position of the member's new post. Once I know their relative positions I can return the correct positioning data to the member.
Statistical Analysis and A/B Testing
Even in the case of a database that is only used by a single application, there are plenty of circumstances when it's better to interface with an abstract data model rather than the bare database schema.

One reason could be a desire to keep your database (somewhat or totally) normalized, while also being able to make queries against more natural (for the programmer) abstractions. It's rarely the case that all of a given user's information is stored in a single table (and for good reason... read up on normalization (http://en.wikipedia.org/wiki/Database_normalization) if you're not sure why), but it can sure save a lot of programmer time if your data model acts like it's all in a single table.

The case of multiple applications or systems talking to a single database isn't really all that different from a single application talking to a database, unless you only have one database call in your entire application. As soon as you're talking to the database in more than one spot in your code, unless you're careful, you start running into the brittleness/dependency issues described by the author.

Even if you're the only one working on a given application, in three months you'll have little chance of remembering every dependency created by every database interaction in your application. Abstracting things with a data model won't fix all of the issues addressed by the author of the article, but it sure does make it easier to address them.
Ask HN: Financial Products API?
Hey, we all know there are plenty of great general shopping APIs (amazon, cj, etc) but are there any that either offer or specialize in financial products (credit cards, brokerage accounts, bank accounts, mortgages, loans, etc)?
In my last long post I said Retric was wrong about providing data that's accurate only to +/- .2 seconds, but maybe I'm the one who is wrong. It is possible that I completely misinterpreted his meaning. I became aware of this possibility when I read lacker's post that said I can probably batch up data since I'm going to have latency issues anyways. He's right, I never thought of it this way.

The issue I'm imagining with batching is that I cannot send any data back to visitors until the next batch is processed -- because if I do they won't see how their last post has changed their positions in the hierarchy, and they will interpret this as a system error. In other words, every time a member receives a response from the server, that response must indicate his last post's position in the hierarchy.

Theoretically I could run a batch every second, and although this means holding up all the responses to requests that come in during the previous second, one second is also short enough that they will not interpret it as a problem with the system. On the other hand, 5 seconds in between batches would certainly be seen as a server problem especially by folks on high speed connections.
Here are some interesting stats in how much data 4chan handles on any given day while the upstream might not quite compare to something of the likes of Flickr or Youtube, but from what I can tell while software is certainly important it is your hardware that will make or break whether or not your site can handle such volume.

4chan is currently powered by seven servers (five content, two database). We are colocated on a full 500mbps Global Crossing connection, allowing us to push over 5TB (5,000GB) of data per day [Image: http://content.4chan.org/img/traffic.feb5-12.png]
the perfect tool for developers and PMs!
Sounds more like game or game-like application. Maybe live betting during some sport event?
Yes, eliminating the sorting is a better way to approach the problem I believe. I've always just 'sorted' to determine a new record's position in the list, but it seems there may be other (more efficient) ways to determine the record's position without sorting ...
As a person who spent few years doing C programming, and then got trapped into the Ruby realm, I must say that the main difference is focus. If you have to focus about memory management, strings, you think much more about the code you write.

Theoretically, high level programming should enable you to focus on the abstractions much more, but somehow it doesn't work this way.
Why mod_php is bad for performance?
Visa Card Features Buttons and Screen to Generate CCV Dynamically
Ajax Experience 2008 - Intro to jQuery with John Resig (Video)
Interesting that you thought of describing a raindrop... I thought of storing information in it ;)

Just imagine how much bigger our storage mechanisms will get...
It's certainly questionable, I didn't mean to imply it was more than a rough guess. It's true, professor jobs at research universities have an "interestingness" premium, so part of the multiple comes from that.

On the other hand, lack of external employment options makes tenure MORE valuable, since it raises the risk premium the teacher would demand to quit the tenured job.

Since my financial math is rusty, I won't try to compute the actuarial values of these revenue streams, however.
This kind of makes sense: if ever there was an industry that needed regulation, it's taxi/car service. PickupPal is basically the Napster of taxi dispatchers. I'm not saying I think it should be outlawed; I'm just saying, it's not crazy that others do think that.
We use this heavily at Kongregate - it's a great tool and never gets in the way.
A is for... A darkly amusing meltdown dictionary
what kind of data do you want? real-time? or end of day? If you just want end of day data, it is simple. Just write a script that collects it in the middle of the night and stores it in your database. I don't think they mind at all if you just do it once a day for all of the stocks - I know people who have been doing it for years. They use this:

http://www.gummy-stuff.org/Yahoo-data.htm

If you want real-time data, good luck. It will cost you.
Dude, the author of the article is a freelancer. He just doesn't like the connotations of the term "freelancer".
Twitter Shuns Venture-Capital Money as Startup Values Plunge
Ask HN: Solid MIME parser library in C or C++?
I'm looking for a solid MIME (email) parser for a C or C++ projects. Can anyone recommend a library that is not too intrusive and that is well maintained?
Definitely in the uncanny valley.
Budgeting for new team members
True. But I doubt people vote on everything posted here so it seems to me that the non voting is the default behavior (to me at least).

It seems the decision is to first decide whether to vote or not, and then if you decided to vote to choose whether to vote up or down. This probably happens instantaneously so they feel like one decision.
Press Release Tips For Startups - Pulp Facts
List of tips that help startups generate revenue from their press releases.

Good, pragmatic examples.
Except that until 8 days ago, the opposite was true.
We've used Pivotal Tracker over the past year and a half to manage a dozen software development projects. The tool gets out of the way and allows us to focus on what matters most: building great software for our clients. The Pivotal team has worked hard on creating a tool that balances simplicity and power. I would not choose another tool to manage my software projects.

Ben Curren
Founder of esomnie.com and gobootstrap.com
The sad thing is, I see $3.5Bn, and that's good news.
Yes... getting dangerously close to two years' worth.
Optimizing debugging php
Biology is worthless as a major; as long as you take the pre-med requirements, they don't care what your major is.

Granted, Biology requires a lot of the same classes, so it might be easier to be a bio major if you're already pre-med, but the biology degree itself isn't worth anything.
One Billion Tweets Later
Silverlite
My wife and I use it to keep track of household projects, errands, etc

I also use it for all of my coding projects :p
Ask HN: What's your Twitter rank?
Yahoo Pop Launches To Show Hot Searches
A friend of mine who is an engineer at Google wrote this up. It uses Real Math(tm), so it calculates your TwitterRank by taking into account how you interact with other Twitter users.

Mine is 17.2, which surprises me.
It sounds awesome. Almost everything we do is bound up under serious NDA. I don't see any story here about security.

What would rule would be if someone could build a service like this but use something like Slicehost as the deployment environment, so that each customer got their own machine. I wouldn't even need root, or even a shell login. I'd feel instantly 10x better.

This is going to come across as paranoia from a security geek, and that's a little true, but its my clients concern that's controlling here, and they're not paranoid security geeks; they're Fortune 500 IT/dev shops.
Venture capital advice - questions to ask VCs
Why does it need a password? twitter.grader.com does essentially the same thing, but doesn't require a password. I'm not sure what additional data the author gets by using the Twitter API vs. just looking at the public info (I can't think of any part of my Twitter stats that aren't publicly available)
If they're pre-funding, in the middle of an angel round, why are they hiring employees already from general public? First off, why aren't they doing themselves? Is the team technically incompetent (by the virtue of too many biz-dev people)?

It looks like they really need another person on their core team - not just a regular employee. Somebody who's willing to be in the garage with you ( why do you even have an office at this point, by the way? ).

You don't recruit that person by posting on the Internet, you use your social network ( and I don't mean Facebook ) to do that. Do they not have any acquaintances, former coworkers?

This may well not be the case - and in fact, I'd imagine it isn't - but the situation screams "these people don't have a technical clue themselves and don't have anyone in their social network who believes strongly enough in them to take the risk and go into a garage with them".
Materials science is so awesome. Now we just need inventors and engineers to come up with good applications. Get to it!
for small teams (like 2-3 persons), agile development tools might actually get in the way of agile development, as a significant amt of time is spent communicating what can be easily accomplished via Google docs. For larger teams, I can understand the utility as communication costs are much higher in a larger group.
When the debt burden becomes onerous I expect the government to default rather than inflate. High inflation would create far more problems in the domestic economy.

There is precedent for a country defaulting on debt denominated in it's own currency. Russia did that in 1998 (although it was also accompanied by high inflation). They managed to recover reasonably well afterward. Eventually the USA will probably force its debtors to accept a cram down, with bonds still retaining their face value but a drastic decrease in interest payments and an extension in the repayment period.
Really? Why? What makes driving a stranger around in your car so much more dangerous than driving your wife and kids around?

It's absolutely crazy. The bus company asking the government to protect it from delivering the service that their customers want and need.
My favorite part: some guys from a hedge fund that's betting against subprime mortgages, at a subprime industry conference in late 2007:

Their first stop was a speech given by the C.E.O. of Option One, the mortgage originator owned by H&R Block. When the guy got to the part of his speech about Option One’s subprime-loan portfolio, he claimed to be expecting a modest default rate of 5 percent. Eisman raised his hand. Moses and Daniel sank into their chairs. “It wasn’t a Q&A,” says Moses. “The guy was giving a speech. He sees Steve’s hand and says, ‘Yes?’”

“Would you say that 5 percent is a probability or a possibility?” Eisman asked.

A probability, said the C.E.O., and he continued his speech.

Eisman had his hand up in the air again, waving it around. Oh, no, Moses thought. “The one thing Steve always says,” Daniel explains, “is you must assume they are lying to you. They will always lie to you.” Moses and Daniel both knew what Eisman thought of these subprime lenders but didn’t see the need for him to express it here in this manner. For Eisman wasn’t raising his hand to ask a question. He had his thumb and index finger in a big circle. He was using his fingers to speak on his behalf. Zero! they said.

“Yes?” the C.E.O. said, obviously irritated. “Is that another question?”

“No,” said Eisman. “It’s a zero. There is zero probability that your default rate will be 5 percent.” The losses on subprime loans would be much, much greater. Before the guy could reply, Eisman’s cell phone rang. Instead of shutting it off, Eisman reached into his pocket and answered it. “Excuse me,” he said, standing up. “But I need to take this call.” And with that, he walked out.
Everything You Need to Know About Writing Successfully - in Ten Minutes
Well, there's Romes in Georgia, Illinois, Indiana, Iowa, Maine, Maryland, Michigan, Minnesota, New York, Oregon, Pennsylvania and Wisconsin, but I doubt any of 'em would have looked quite so interesting in 320 AD.

PS. Ain't wikipedia disambiguation pages grand?
check out ActiveCollab: http://www.activecollab.com/

it's open source and you are free to host yourself or pay for a hosted version.