skysight-inc/HackerNews huggingface dataset:
This is similar to one of the problem I used to deal with in tracking huge volume of billing events. Since the data (checkins) are immutable, it's fairly easy to build summary running totals daily. The total count then becomes the running-total plus the current day's count. The summary job can be run offline in the background. Just make sure to mark the current day's events as processed and update the running total at the same transaction.

In this way, the amount of current day's events are pretty small and you don't need to keep all historic events in memory.
Tool for Building Full Android Apps with Javascript and Rhino
The poor, stupid Waddell & Reed sell algorithm [...]

Ouch.
Please don't call things "-gate." Also, "no."
I doubt every last bit of their data needs to be hot (i.e. in memory) at any given time, but without specialized paging along the lines of what antirez has discussed for redis, enough of their data probably needs to be hot that, from a paging perspective, all of the vm pages of their data need to be hot.
Probably Arch. You can build really small images with Gentoo, Slackware, etc. but Arch is probably the simplest for you to manage.

TinyCore looks cool but not like something I would want to maintain as a server.
Moving an online webserver using public transport
Just wanted to chime in that puppet is an awesome tool if you're trying to manage a few or more servers. (Hell, even if you manage a single one but want re-deployment and roll-out to be consistent)

I also had puppet configs at one point to help get new OS X desktop and laptop settings sane after purchase.
Acunote ( http://www.acunote.com ) - project management, wiki, issue tracking, scrum software

Tarsnap ( http://www.tarsnap.com ) - offsite backup

DnsMadeEasy ( http://www.dnsmadeeasy.com ) - DNS

TrustCommerce ( http://www.trustcommerce.com ) - CC gateway

Google Apps ( http://www.google.com/apps ) - email

AuthorityLabs ( http://authoritylabs.com ) - SEO rank monitoring

GitHub ( http://github.com ) - OSS projects vss
Could be worse .... Apple could buy them, making an incremental Photoshop update 6 gig + an hour to install.
The iPad juggernaut is gaining momentum
Great article, except for the end:

"'Realistically, if you want a one-to- one conversation with an AI-driven character, that’s at least five years away,' Andreas says. 'But thinking about where that tech is going,' Burton adds, 'if it’s cloud-based, passing the Turing test becomes much easier. You just need a huge internet-like database, then work out how to data mine that.'"

Uh, yeah, easier said than done!

Solving global warming is easy: you just need a huge atmosphere filled with CO2, then work out how to remove that.
From the horse's mouth--do you see an organic access pattern in the resolved URLs, as a shadow of the popularity? Or are most of the 200K API requests batch processes?
Is it this one?

https://www.cloudkick.com/
I remember Heroku had something like this a few years ago. It was pretty easy to get a taste of Rails.
Plannr Shuts Down (Because It’s Been Bought By Google)
Only use VMs or sandboxes on production-like machines.

Local development doesn't scale.
Not sure that I buy into the "technical" and "nontechnical" argument. I'm middle aged and technical and while I -respect- Zuck's accomplishments, I can definitely say there's no hero worship involved.

The focus on his personal failures is to be expected, and it has little to do with age or technical awareness. Zuck is at the top of his field, so everyone loves a bit of schadenfreude when it presents itself.

I remember that people used to get their hate on for Marc Andreesen once upon a time.
I find it almost 100% more interesting that some of the people the design "community" exalts on a frequent basis, and even then some of the community itself didn't present any serious submissions when the call came. They all made fun of the idea that GAP was crowdsourcing a new logo.

That's slacktivism on a level Facebook breast cancer awareness can't even touch.

Participate or stfu is my opinion, but I'm no designer.
Why are you glad?

No one can win at the game they are playing, it was always mutually assured destruction, when it came to the big players suing the big players. Apple made a move on HTC thinking that they where not a big enough fish to start a meltdown. Their analysis was wrong, it did. Now with the big fish not focusing on their core competency it gives others an opportunity in the market. The mobile patent wars has grown to big for any of them to control, now the only way they will be able to draw it to an end will be to lobby for patent reform. There is no doubt in my mind that these wars are close to boiling over into the desktop market. Until someone mistepped and started the unwinable war, it was a game stack against the little guy and a tool used exclusively on them. Now the weapon has been turned on the wielders, which was the logical first step that had to happen before any weight would be put behind reform.
Are you sure? Availability in the CAP theorem is a state, as are (P)artition and (C)onsistency. Your system can't be simultaneously consistent and available in the presence of a network partition. The A in CAP doesn't mean always available. It just means the system can, at best, be any two of the three at a time.
New Report Decries "Failure to Teach K-12 Computer Science in a Digital Age"
LOC - The Linked Open Commerce Dataspace
Speaking as someone who worked multiple cf projects over multiple years, no, he's not.
Good design for indie online game account creation and login
You mean like Apple praised Google for Android? or Microsoft for Windows? I am sorry but Apple is one of the worst offenders when it comes to litigating copycats. I think we need to get rid of software patents all together but historically Apple has never praised copy cats of their original ideas why should this guy be any different? Because it is Apple?
California Incorporation 
Incorporate California with California Incorporation Attorneys! Form a California S corporation, California LLC, CA Professional Medical Corporation, Corporate Minutes or Register a Trademark. Start a California Small Business Online Now. LawInc.com
What do people think of this - it feels like this is sort of siphoning off legitimate conversation for HN to a somewhat-rival community.

(I'm sure many of us use both Quora and HN, but there is definitely overlap like this)
Josh, check out Cloudomatic.com . We catalog most of the SaaS apps there. Not organized by what companies are using, but should give you a good idea of all the tools out there (way more than you think).
Well I just tryed writing a provider for puppet and I found quite a few bad design choices such as :ensure allowed to be a version, 'present' and 'absent' double purpose variables are bad and this shows in a few providers having to implement checks on a few dual purposed variables. Chef from what I've done so far doesn't have these bad design choices.
Have you thought about how much extra overhead there is in storing the checkins using MongoDB's BSON structures? Considering how often you do this, you might consider packing a specialized version of a user's history (at least for this purpose -- sort of a denormalization) into a array of 64-bit placeIDs? This means a checkin would only occupy 8 bytes of space and there would be no overhead. 1 billion checkins = 8GB of data. You could even keep these in memcached and use a sort of write-through strategy.

EDIT: I thought about this a bit more, and the checkins are probably time-sensitive, so another 32-bit timestamp (Foursquare Epoch) would need to be stored for each checkin.
umjames - ColdFusion is built upon Java, of course it runs on the Mac as well as 'nix. If you go to a gathering of CF developers eighty percent of them are running Mac laptops.
Actually, according to your own blog post on the subject, Gilbert and Lynch define a partition tolerance as:

“The network will be allowed to lose arbitrarily many messages sent from one node to another”

There's a huge world of difference between the network losing arbitrarily many messages, and the definition you use elsewhere in this thread: namely, any subset of packets dropped, no matter how small, counts as a network partition.
Has this ever worked for you in the longer term? I mean from experience have you ever been paid out for taking an equity trade as a consultant?

I'd really like to hear from someone who's made this work to their benefit
"Sharepoint, SQLServer, Windows 7, Windows Server, Windows Virtual Server, CRM, Exchange Server, XBox, Xbox live, Bing, Zune, Microsoft Automotive platform, Win 7 phone."

Has it actually been determined that Microsoft makes any profit at all from any of those products? Entertainment division (where XBox and Live and Zune lives) is about 12% of their revenue and is still not profitable, while search and online (which is Bing and MSN) loses $700 Million every quarter or thereabouts (at least it did last fiscal quarter).

Their bread and butter has always been Windows and Office. That other stuff are loss leaders, in a way.
Le BigRep: World’s Most Gigantic Rep Strap Build Underway
I hated Windows Mobile 6.x and its predecessors.

Windows 7 Phone actually seems nice, despite the stupid name. It might be too little too late, but I think it has a UI that is both novel and distinct from the iPhone. The only other phone OS currently on the market that I can also say that about is WebOS.

I'm probably in the minority, but I think the Achilles heel of the Windows 7 Phone is that Microsoft is farming out the hardware to multiple manufacturers.
actually, this was only meant as a humorous replica to the OP's opinion of Zuckerberg being a genius. never mind ... :-)
Herein lies the big problem with software patents: patents are supposed to cover inventions, but with software they cover ideas. A completely different implementation in code could still infringe a sufficiently vague software patent.
How to Achieve Instant Success as a Lifestyle Designer (Satire)
do 'em
I wonder if Alec Baldwin and the other members of the Film Actors Guild feel slighted at being passed over?
Depends on the protocol, in general.

Paxos is, fundamentally, a quorum-based system that deals with reordering of messages. It sacrifices liveness for correctness - if the proposer does not hear back from a majority of nodes (in the case of, e.g. a partition), the protocol will not complete (availability is sacrificed).

My point is not that there is a 'vital packet' in every protocol, the omission of which will cause either a lack of availability or consistency (although I can certainly design protocols that way!) - it's that for every protocol there is a network partition which causes it to be either unavailable or inconsistent. That network partition might be dropping ten messages, or just one. Retransmitting would make sense, but in real life message failures are often highly temporally correlated :(

The proof of this, by the way, is in a very famous paper by Fischer, Lynch and Patterson called "The Impossibility of Distributed Consensus With One Faulty Process". One take away is that one slow-running process can take down any protocol. It may take a few missed messages, but only a single node...
I am talking about the database looking in memory for what it needs to answer a request; if it doesn't find what it needs, it reads from disk. Since it has been allowed limited memory, when it put something new in memory, it has to kick something old out. It uses a simple algorithm to identify what to keep and what to kick out. One of the benefits of this (obvious) design is that when too many users are active at the same time to keep all of them in memory, a few of them experience a small lag (as opposed to all of them experiencing a complete crash). If you have a few terabytes of disk then you have to really be asleep at the wheel (like for a decade) to totally crash.
upvoted, definitely would like to see this.

I think it would be interesting to go one further and (if it's appropriate for HN) to have one thread per vertical with discussion on different vendors and solution providers.

An example would be email campaign management providers come to mind (Mail Chimp vs Aweber vs Constant Contact vs others). I'm not really interested in a blog post from a single person but what the HN crowd think having used these tools in anger out there.
I wonder how much money there is in that. For organizations that want a vanity shortener but have no in-house IT resources it might have some appeal, but it seems like it would be a very cheap, simple thing to do in house in all other cases.
well duh :)

FWIW, the other day I read about a research (and this matches my personal anedoctes) that younger brothers (i.e. the ones who usually get bullied by the older brothers) tend to show more empathy towards other people than older ones.

In contrast, psychopaths almost always have a history of being a victim of child abuse by family members. So, in my mind, values coming from the authority figures in a family are really on a whole different level than simply having maturity.

>> enjoy having defended yourself

Uh. I think the term we should be using here is "relief".
I am not trying to start a debate of sql vs nosql, but i have an honest question. What if foursquare was using a standard RDBMS like Oracle? Would they have run into the same problem?
`uncertainties` module lets you make calculations while keeping track of errors
Isn't Quora more about asking a specific question with a more specific obtainable answer?

Ie the open question here is "what tools are people using?" where as on Quora it would have to be "what are the best tools to use"... not sure they're exactly the same.
I cracked up at this. Well played.
Sure they did, The iPhone was a eerily similar concept to a device patented in the late seventies or early eighties. I wish I could dig up the link, but man it looked like a wholesale copy of this patent. Multi-touch and everything. Someone at Apple is rifling through old patents for ideas.
I agree that replacing Illustrator and Photoshop are nontrivial. I doubt that a typical startup could do it.

On the other hand, it would be a natural fit for someone like Autodesk given their resources and markets. Retail box graphics applications could make sense in Microsoft's product line. Even Corel would be in a position to capitalize on Adobe leaving the Windows market.
There is no better time than during an outage to add these features. In my experience, band-aids are better way to resolve an outage than fixing root-causes. Make the problem go away by commenting out code, returning empty result-sets, (or whatever) - and re-balance your partitioned database at some later date.
If CS became Windows-only, it would mean a great resurgence for Quark, along with those guys in Texas who got Freehand back from Macromedia.
If the world worked this way no one would innovate because the bigger players would start their copy machines every time someone with courage and a few great ideas sought to change the world.

If someone has a plan to rid us of the patent system and protect innovation I have yet to hear it...
Unfortunately we have some badges that require more than just the venueid + timestamp, but some sort of "compressed" checkin format is something we'll probably end up looking at.
"it makes perfect sense for both of them..."

"I, for one, welcome..."

Uh, yeah... I agree it makes perfect sense for both of them. For anyone who isn't Microsoft or Adobe, the prospect seems less appealing. Two companies with considerable monopolistic sway in their respective markets come together to improve their profit margins?

Merging two huge globs that each can extract rent on their various properties into a single similar vast glob. What could possibly go wrong...
The Great Reset
This is just what .net needs... but the fact that it requires windows and visual studio just makes it impossible for me to use this.
Again why? Apple did not praise Google for Android or Microsoft for windows, no they sued them, well in the case of Android they went after the handset makers because Google's portfolio would have done more damage to Apple than Apple's to Google. Under the rules of the current game why should this guy be any different? Because he is small, Why are the little guys to praise the big guys while the Big guys sue the little guys if they do the same.
Seconding Tarsnap. Can't recommend it enough.
Twitter Exploit Warning: How Anyone Can Easily Snatch Your Direct Messages
It's pretty hard to NOT be the leader when you're pretty much the only game in town.

Yes, there are other devices around, but not many of them are providing the same level of performance at $500 as the base iPad, nor do they have the luxury of being sold in high traffic Apple stores and Best Buy.

As much as I love my iPad, there absolutely needs to be real competition in the marketplace to keep Apple honest.

I had high hopes for the Galaxy Tab, but I'm not convinced Samsung's marketing team knows how to market it. The Playbook is really impressive, but it seems a little vapour-y right now. It would have been nice if the press had a chance to physically use one at the announcement.
It seems that if you know your page size and you know your record size, you can compute the optimal size into which you should break your records into chunks and link the chunks with pointers. This causes more memory accesses, but surely that's worth keeping everything in memory? More chunks per record means more pointers and more accesses, but it should still be way faster in many cases.

How to compare this to compaction, in the general case, isn't immediately obvious to me.
But a user should not be forced into using any of those things.

imo, if I have an Apple monitor used by a large number of potential site visitors set at maximum resolution, then the page should at least center the div if you are going to do a fixed width layout, so 1/2 of the right side of the screen isn't a plain otherwise empty blue background. That way at least only 1/4 of the page is empty on both sides, which is less distracting.

Fluid width with adjusting fluid left/right % columns (divs, or table if you must) also works.
Today I have attended a talk by Charles Zhang at MIT and I am really amazed. I have not verified his claims but my impression is that he did not exaggerate anything. He is the founder of sohu.com one of the most popular sites in China. It is something like the yahoo of China. Having the opportunity to listen to one of the pioneers of the Chinese internet in person there are a few things that I find very interesting for people that are into start ups in the States.
- Every time that I read or listen about a pioneer in his field I realize that independently of the time that they were living. Their means seem primitive and poor. In his case 3-4 computers (i486), no PR experience, no MBA, no internet infrastructure, etc. If you do not feel that you do not have the means to continue, when you find your methods primitive do not worry it happened to one after the other of the pioneers.
- His presentation was not polished at all- unlike so many masters of PR- but at the same time he conveyed his message. In an unusual but capturing and engaging manner.

I do not want to go into every detail of his presentation.

If you ever have the chance to attend any of his talks do not miss the opportunity
Patents check in, innovation checks out.
The real world does not work that way, there is a lot of time, energy and risk put into success. The big players tend to opt for purchasing the little guy because the time, energy and risk have already been accounted for. Even the big guys fail at endeavors so many times it makes more sense to acquire the small guys who has mixed the formula correctly.
Death has the exact same empirical consequences--people you observe to die are never seen again alive. The subjective consequences of death are impossible to know whether or not reality is simulated.
Thoughts on incumbents from a startup's perspective
No, it does mean always available - honestly :)

If there is some time period during which requests are not responded to within a time bound, the system is not available then, and further is not a 'highly' or 100% available system. That is what the CAP theorem is talking about.

Consistency, similarly, is not a state but a property that holds across all responses. Either you return a consistent response to all your requests, or you don't. In the context of CAP, there is no middle ground.
Upvote! I had the same moment of panic, like: "aaaahhhhh shiii...wait a second. LOL"
COPYCAT ALERT: Maldives President Installs Solar Panels
This is more of a hack than coding, but I owed the school library money for a lost book in grade school. I didn't have the money and didn't want to to tell my parents about it, so I studied the library system.

It was a self-check in, check out system with a scanning wand. You take your book, log your student number, scan the barcode, and it was added to your account inventory. Returning the book was the same way; but if you didn't have the book, you obviously couldn't scan it back in. I figured out you could look up the book's ISBN through your account inventory, write it down, and manually enter it into the return form.

BAM! Lost book was magically returned :D. I told a few friends about it, I'm sure the school lost out on a few hundred dollars worth of books that year.
so, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of those machines, ran out of memory on them, and suffered a large amount of downtime as a result. mongodb had little to do with the problem, other than maybe it took longer than expected to migrate data to a third server.

i'm baffled at how there could be no monitoring or reporting in place to catch that days or weeks ahead of time, let alone just 12 hours needed according to the mongodb developer, to fix the problem without downtime. it's such a fundamental thing to keep track of for a system designed entirely around storing a bunch of data in the memory of those 2(!) machines. i have more servers than foursquare and none of them do even a small fraction of the amount of processing that theirs do, and yet i have real-time bandwidth, memory, cpu, and other stats being collected, logged, and displayed, as well as nightly jobs that email me various pieces of information. nobody at foursquare ever even logged into those systems and periodically checked the memory usage manually?

worse still, during all of this, the initial outage reports were blaming mongodb or saying the problem was unknown. even at that point nobody at foursquare realized that the servers were just out of memory?

how did the developers come up with 66 gigabytes of ram to use for these instances in the first place? was there some kind of capacity planning to come up with that number or is it just a hard limit of EC2 and the foursquare developers maxed out the configuration?
Or your datastore can take care of doing that for you, c.f. BigTable. This is one of my main gripes with most of the current set of NoSQL offerings — they leave too many decisions in the hands of developers.

Whilst it would definitely be advantageous for all developers to understand the intricacies of various CPU and OS scheduling algorithms, it's not an issue that most developers have to deal with directly. The App Engine datastore, in particular, proves that it is possible to create NoSQL datastores which don't force developers to think about issues like load balancing.
Not exactly patented, but the concepts behind tablets, smartphones, and the PC itself can be traced directly to -> http://en.wikipedia.org/wiki/Dynabook
While I understand this is foremost a monitoring and architecture (mongo and 4sq side...) issue, but I'm curious how dangerous it is to run such IO dependent systems on EC2. Would the prolonged downtime (due to shard migration, etc) been as severe if they were running on hardware? What if these two servers were on SSD RAID 0?
Writer for iPad: First Reactions