skysight-inc/HackerNews huggingface dataset:
Having been bullied myself, social skills often don't make a difference. All you have to do is cross a bully's path on the wrong day and from then on your life's a misery. Once you're tagged low down in the pecking order, most kids will go along with the bully, so nobody will talk to you or give you a chance to practice your social skills, much less a dozen of them.

I had a friend in Uni who had to move schools twice - once to get away from the bullies, a second time when the kids at his new school found out that he'd moved due to being bullied. He was fine up until that point, but as soon as he'd been pegged as a victim, the bullying started up again.

So it's not sad that you're being called a troll. You don't seem to have a particularly sound grasp of what some kids go through, and yet you're quite happy to take up an inflammatory position (kids who get bullied are asking for it by not having social skills, and should just shape up). That's pretty much a textbook definition.
Async HTTP Client 1.2.0
And now I have a vision of the word "Gap" with fingers pulling at the G and the p.

Yes. Blame goatse on me. But once you see it, it never goes away.
We've Already Surpassed Last Year's Patent Totals
Forget about Apple, it would have been nice if BP followed this example.
If you're lacking storage space, keep in mind that many mainstream linux distros offer jeos (just enough operating system) releases for these purposes. They're usually pretty minimal - Ubuntu Jeos takes up around 300MB of disk space after the install.
Why they have only two database servers running (with their database in memory, no less) with 200 million check-ins, is completely beyond me.
It sounds like even if the check-ins had grown evenly across the two shards, that would have only saved them for about another month. Without close monitoring of the growth in memory usage, foursquare was still due for an outage.
Skype's Android shackles are broken, 3G calling hacked into reality
The term strategy is misused in this context. Customer development is not a strategy, it is a process. A process can also be called a tactic.

Strategy is about the unique position that a company will be able to achieve by executing on their process.

There are two general types of advantage: competitive advantage and comparative advantage.

Delivering more value at a given cost is a comparative advantage. Equivalent value at a lower cost is not a comparative advantage.

Strategy is about competitive advantage. A competitive advantage is a unique position that a competitor can never attain.

In concrete terms, for startups, you will always get the "What if Google did the same thing" question or the "What if Microsoft did the same thing" question.

The answers are "Google will never successfully compete in this space because it will have a negative effect on their advertising revenue" or "Microsoft will never successfully compete in this space because it will have a negative effect on their Windows or Office revenue".

So the stack looks like this:

Strategy (Competitive Advantage or unique position)

Process/Tactic (Customer Development and Product Development)

Action (TODO list)
In general you want to have memcached implemented for stuff like this. Have most used stuff closer/faster with memcached.

I don't understand why they are running the whole nosql db on those monster machines - it just defeats the purpose. And sharding architecture that they are mentioning is quite error prone. I would go with read/write seperate channels for things of their nature - it appears they don't have that either.
Indeed, there is solid evidence that stuxnet is quite unusual compared to other worms and likely was developed using different processes and with greater resources than previous worms. Beyond that pretty much everything else is rampant speculation.
Microsoft is no slouch (especially compared to Google), but these charts seem to imply that Apple still has the more diverse revenue stream:

http://www.businessinsider.com/chart-of-the-day-microsoft-op...

http://www.businessinsider.com/chart-of-the-day-in-case-you-...
I wonder if burglaries also went down during the outage, since the bad guys couldn't tell if their targets were at home or busy being the Mayor at the local bowling alley.
Bungie was spun off successfully a couple years ago.
I run un-shortening service called http://unshort.me which is getting about 200K API requests/day.

Sample of 1,000,000 requests from the service:

bit.ly 612,170
tinyurl.com 55,091
goo.gl 41,812
ow.ly 35,531
is.gd 29,533
t.co 26,043
j.mp 21,172
other 178,648
In essence, although we had moved 5% of the data from shard0 to the new third shard, the data files, in their fragmented state, still needed the same amount of RAM. This can be explained by the fact that Foursquare check-in documents are small (around 300 bytes each), so many of them can fit on a 4KB page. Removing 5% of these just made
each page a little more sparse, rather than removing pages
altogether.

Interestingly, this is one of the reasons antirez gives as to why redis will not be using the built-in OS paging system, but instead will use one custom-written for redis' needs.
Alternately, crowdsourcing involves many people getting the chance to try their hand at something that, normally, only one person would do.

I don't think it's surprising that that one person dislikes the idea.
I completely agree, and just wanted to reply to touch on this point a little bit.

You are correct that the article (rightfully) provides a forumula to create a popular app. Sure, there are lots of technical details that weren't touched upon in this article, and certainly we are not going to enumerate each of them and criticize the author for not creating an end-all be-all make-the-best-app-ever article.

However, I do believe that security should be a primary concern from the very start. Think about security early, think about security often. Once your app is already popular with 500k active users is not a good time to realize your oversight in security, leading to a possibly fatal breach.
With 500M users, they can do an end-run around the goddamned Federal Reserve.
Start scripting your environment, the more repeatable your deploy procedure is the better your deploys will be. The easier your deploys are the more frequently you do them, the more frequently you do them, the better your deploys will be. Rinse wash repeat.

Once you have your env scripted with EC2 you can setup / teardown environments easily for testing.
He is probably talking about the handbook of human factors and ergonomics. It has been updated since I purchased it, and is more information systems focused now. It is like the bible of usability but it is an expensive text, that is why I did not recommend it.

I guess I should have and put a disclaimer on it. When I bought it, it was around $275. If you have to buy one book and you can spare the change, this is the one. But some of the others will get you going faster.

It depends on what you want to do, This book is deeply focused and considered an academic text, many teach from it. Once the subject matter in this book is mastered you will be a HF professional for sure, but some of the other books can get you up and running faster.

At some point you should consider this book for your bookshelf but it is not a beginners book that is another reason I was hesitant to recommend it.
This talk gives a good high level overview of the Facebook infrastructure:

http://www.infoq.com/presentations/Scale-at-Facebook

In short they use sharded mysql instances (sans joins) as a key-value store with memcache on top of that.
Wow, downvotes? Reddit must have found Hacker News...
I could use Windows but I'll never buy Windows.
> Is it acceptable/preferred to store your entire db in RAM?

This is actually one of the big long term challenges we're going to have to deal with @ foursquare. Right now we calculate whether you should be awarded a badge when you check in by examining your entire checkin history (which means it needs to be in ram so we can load it fast). While this works now, as we continue to grow it will become more and more of a problem so we'll have to switch to another method of calculating how badges are awarded. Several different options here, each with pluses and minuses.
Ask HN: Tools of the trade, 2010 edition
When I first started delicious, we had to host most of the services ourselves. CVS, mail, mailing lists, etc etc etc.

These days, lots of that stuff is available as SaaS. What are the tools and services people use instead of hosting their own?

(I'm not talking about actual production services like EC2 and Heroku and whatnot. We can go over this in another thread.)
I haven't had the pleasure of dealing with border agents, but I would assume that they'd leave you alone as long as everything looked normal at a casual glance.

A couple example suggestions for that:

1) TrueCrypt let's you change the bootup text, and can be set not to visibly show keystrokes when entering your password. This means that you can make it boot into what appears to be a standard "No operating system found" type error message, which will not react at all until you type your correct password and hit enter. Unless they're dedicated enough to examine the boot sector they wouldn't be able to tell that it's not just a corrupted drive. Toss an OS install disc in your bag for added realism.

2) TrueCrypt will boot from CD into an encrypted drive. Using this to your advantage, install a small Linux partition and set your computer up to boot into that unless the boot CD is inserted. Nothing suspicious at all remains on the drive except for a parition of apparently random data, which they'd have to be pretty suspicious already to even bother looking for. You could copy the TrueCrypt bootloader back onto the drive to boot without the CD for convenience if you're not going to be crossing borders for a while.

...And there are probably other ways. The TrueCrypt documention offers a number of suggestions on how to look minimally suspicious to suit whatever level of paranoia you may have.
Using Open Data and Mapping to Fight Human Trafficking
Facebook runs primarily out of ram via memcached. The last numbers I'm aware of were that they had about 200TB in memcache capacity [1]. They use a variety of data stores, but primarily sharded mysql. I don't have recent numbers there, but they were above 1000 master-master pairs as of 2008.

While buying that much ram sound costly, that's only looking at capacity. Assuming typical 1u servers and common pricing at the moment, you're paying roughly $1k in capital for each 10GB of ram capacity. However, each of these servers gives you a couple hundred thousand random reads per second. To duplicate that with spinning hard drives would take several hundred spindles at least. SSD's are better, but still, it ends up being a lot of devices to duplicate that io capacity.

This is why virtually everyone at stupendous scale (google, facebook, etc) ends up with very ram centric architectures.

There's a simple way to decide how much of what storage you need [2]. Look at the distribution of access times. With current technology, in very rough terms, if an item is accessed more than once a day, it's more cost effective to store it on SSD. If it's accessed more than once in an hour, it's more cost effective to store it in ram.

[1]: http://perspectives.mvdirona.com/2010/07/01/Velocity2010.asp...

[2]: http://www.cs.cmu.edu/~damon2007/pdf/graefe07fiveminrule.pdf
Good point. There are also downsides. Still, in general, I prefer to let my body fight illness as best it knows how, unless it is obviously mis-functioning.
They are dealing with web scale sharded NoSQL realtime geo scala. Old rules don't apply when 80% of the words describing your company didn't exist two years ago.
Why does this remind me of Steve Martin's "You can be a millionaire and never pay taxes! First, get a million dollars..."

Is there any new information here? Any fresh perspective? Or just an overly large promise combined with a couple of platitudes, some clip art, white-on-black Helvetica and a Steve Jobs reference?
I believe Aperture was developed in-house.
to be honest, as a developer i wouldn´t use it beause it´s so inefficient to edit text on touchscreens.
if i wanna work on code or think about it, i mostly take a sheet of paper and write stuff down when i´m not at a computer.
nevertheless i wish you guys all the best :)
Beware of the plugins blending in WordPress
I know, I'm equally amused by the valuations.

It's a bit like old school advertising (billboards, TV) versus Google pay-per-click ads. As soon as you can measure something accurately, it's value decreases.
Can someone vouch for this by showing us the results of Zuck's IQ test? Is he in Mensa?

Otherwise I would just say that Zuckerberg is very smart and a shrewd businessperson.
I think they mongodb guys are correct - it is an issue with app architecture and app monitoring failure.
It's true that this is elementary in and of itself, but looking at things with a bit of a wider lens shows the complexity. We're a small engineering team (10 people) working on a product that is growing extremely fast both in terms of usage and feature set. Meanwhile we're also pretty much constantly re-architecting things to keep up with growth and also doing the immense work of growing the company up from 3 people to 33 and beyond (this has turned out to be WAY HARDER than I would have guessed going in).

Further, there are lots and lots of different things that we need to be monitoring at any different time to make sure that everything is going ok and we aren't about to run into a wall. Automated tools can help a lot with this, but these tools still need to be properly set up and maintained.

I'm not saying we didn't screw up. We had 17 hours of downtime over two days. We screwed up bad, and we feel horrible about it, and are doing a lot to make sure that we don't screw up the same way again.

But it's not because we're morons that never thought about the fact that we should be monitoring memory usage. We just got overwhelmed with the complexity of all that we're doing at once.

-harryh, foursquare eng lead
I couldn't help but think of that exact issue. I suppose once compacting the data online is built in, this particular issue won't come up again. At the same time, when a machine is overloaded, often times you have even bigger problems. For example, if you are out of memory, you may not be able to create another SSH process to get at the box.
I wish they would provide links to these open source tools and data that are making a difference. Typically the providers would love the publicity and increase PageRank, and in turn, can provide even more service.
"It will be interesting to see if this type of personal socialism will continue in the future. "

The free exchange of one's property for a value you find acceptable sounds more capitalist than socialist.
I did too. And then I got to thinking that a post-it note with "Gap" written on it might make a pretty good logo.
These images do a good job illustrating the similarties between coverflow and scopeware:

http://www2.freedownloadscenter.com/ReviewImages/s1025.jpg

http://www.hardwarecentral.com/graphics/screenshots/10503519...

And a review of the original patents via Apple Insider 2008:

http://www.appleinsider.com/articles/08/03/17/lawsuit_target...

An interesting 5 year old Google answers thread about the fate of Scopeware ->

http://answers.google.com/answers/threadview/id/603242.html
It's not an error, it's a feature! In all seriousness.
Http://www.myspace.com/m1chaelmyers/music/playlists/russell-s-playlist-575284
What's the difference between freelancing and "consulting"?
I like a timestamp so that I know:

(1) How long the last command took (approx, as there's time between the prompt showing and me putting in a command)

(2) How long ago I ran the last command in a given shell window
Just a minor point, but I've never seen a 75,000 lot rip through ES in one order.
The situation antirez had in mind can be remedied by occasionally using a tool like vmtouch to steer what's in the OS cache.
I've had a setup like this going for years. I'm a big fan.

It's unlikely that your server is the same os as your desktop. So you waste time setting up versions of libraries and services for e.g. windows that don't always work quite the same as in production. Plus you can also take advantage of snapshots to mess around with your local vm, confident that you can always roll back to a known version. You can do simplified load testing without actually crashing your whole machine.

It also means your devs can use whatever OS they want - the hard work of setting up a dev environment is already done.
Why not keep the algorithm and process in batches? You don't need to have EVERY users checkin history in RAM at any moment. Throw the checkin on a queue, have a few processes that query a db for the checkin history and you're fine. Heck, if you delay the awards it's also a good excuse to throw the user an alert to come back to the site/app.
Building sharded systems isn't as simple as throwing consistent hashing into the code and calling it a day. You have to think carefully about what happens when nodes exceed capacity. There is good work in academia on distribution algorithms that gracefully handle reaching capacity (along with data center structure such as rack awareness) [1]. Alternately, if your algorithm doesn't handle a shard reaching capacity you need to have the monitoring and processes in place to ensure you always add more capacity and rebalance before running out.

Also, this validates Redis's VM position about 4KB pages being to large to properly manage data swapping in web application storage.

[1]: http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf
That is definitely one of the options we are considering. It (obviously) involves a change to the product, which we have to think about carefully, but it certainly could help from a technical standpoint.
It's easier to give a new developer a VM image than have them set up Apache, 2 Apache modules, PHP, 3 PHP extensions, MySQL, memcached, ImageMagick, mplayer, a queueing system, a custom database daemon for statistics, and a Lucene-based search daemon. And all with the same options as production.
> Is it acceptable/preferred to store your entire db in RAM?

If you can fit your db into RAM that's the ideal, but of course you also have to have a reliable, persistent backup. The best current compromise is probably PCI based SSD drives.

John Ousterhout (of Tcl fame) and his group at Stanford are working on a project called RAMCloud that's exploring the feasibility of storing data in RAM at all times, using disk only as backup. See his projects page http://www.stanford.edu/~ouster/cgi-bin/projects.php for more info. IMO, given the continuous improvements in RAM prices and network latencies, this type of setup for permanent storage will be the norm in data centers in a few years.
That was not "not enough voltage", that was not enough current.
Tiny Core is also a nice small, and therefore fast to download, payload for a network-booted system with gPXE:

http://www.etherboot.org/wiki/appnotes/tinycore
I have done a little PhotoShop. My GR has done a lot of CS4. I cant imaging how MS could make it less intuitive.
Microsoft isn't stupid. They won't make Flash Windows only and they won't make Flash Windows Phone 7 only.
I honestly cannot tell whether or not that comment was serious.
Presumably they're hoping to monetize the data they're harvesting, with things like the mentioned social news site.

How realistic it is that that will be a home run for them, I'm not sure.
for those who don't want to login to see the post:

http://gist.github.com/616108
Problem is lawsuits. Admitting guilt can and is used against you in court. Hence, US businesses don't apologize for anything. This is a refreshing exception.
You could set up a user-place relationship and simply increment the number of check-ins for that place, then periodically correct the number, subtracting old ones which no longer count. Or look at the entire history only if there is a chance of getting a badge based on this number and otherwise just increment it. For badges for the total number of check-ins this is even easier, since no recalculation is necessary.

I suppose if your CPU load is not too high you could even run background processes to monitor "near-badge" status for all users with high enough frequency. This you could really get wild with: people with most number of recent check-ins should get scanned first and of course this list should be updated as a separate entity so it stays small and always in memory.
When you're a small company growing fast you have an endless list of things you really should have done a long time ago. That's no excuse for not preventing a predictable and painful failure, but do try to remember it's a lot easier to criticize as an observer with no time commitments.
“So now it should work, right?”

-“I’m getting an error”

“Which error?”

-“Unnamed method in [...]”

Then, a mountain of pain and torture later.

"But now it works, ok?”

-“Ok.”

---

Ever had to fix someone else’s system via Skype or phone? I personally believe writing a rails clone in brainfuck is more fun.
You are speaking out of ignorance.
Why you don't like changes to your design - The Oatmeal
This is 100% true. We were always planning on moving to more shards, we just thought we had more time to do it than we did.
Aikido is fantastic once you've had five or ten years of serious practice to master it, and some of it's techniques border on magic. So yeah, not likely to be good for high school aged kids.

btw, Aikido is not a nice but ineffective hippy martial art, despite its reputation and philosophy. There are lots of really nasty joint locks. See http://en.wikipedia.org/wiki/Wristlock#Supinating_wristlock for one example - "If performed correctly this technique will break the opponents wrist, elbow and dislocate the shoulder."

The throws can also be pretty brutal ie. throw people face/head first into the ground. Some of the comments on http://www.aikidodemonstration.com/aikido-questions-and-answ... might be worth a read.
There are many businesses for whom vague hand-waving about data mining is supposedly a monetization strategy.

Most of them don't work, because people forget that not all data is valuable.
Mr.doob is amazing, everything he turns out leaves me a little wow-ed
I want to believe that, but I have a hard time proving it to myself. Growing up, I loved to read the paper. Without cable or Internet, it was the best information source I had available.

However, when I visit home & look at the local paper now, I find it pointless. The "news" is outdated & rarely as in-depth as what I can find online. Classifieds? I'm more likely to find what I want online, if not with Craigslist, with Ebay & Amazon. Opinion & perspective? Nothing really insightful. In fact, I don't think we'd lose anything if the local reporters simply became online bloggers. Likewise, local stories & the rest. I can get a richer set simply looking at local bloggers. They don't really do investigative reporting. So that leaves local sports. But does my high school football team really need a newspaper solely because of it? In the end, I realized that my local paper of old was an unconfigurable RSS reader & not a very good one as that.
PaintbrushJS
pmjordon has a great reply, but I'd like to add that I'm a little surprised by your mention of jQuery... jQuery is just a framework built on top of JavaScript; it's not like it's a different language or even a language extension. It's just a standard piece of JavaScript code, an object which you can use.
The only way that would happen is if Adobe went crying to the FTC... which they would, which is why Apple offers first-class PDF support in the OS and Microsoft doesn't.

An Adobe acquisition would render this a moot point. Full .PDF support would show up in Windows very quickly, which wouldn't be a bad thing IMO.