skysight-inc/HackerNews huggingface dataset:
I'm wondering if the decision to shard based on users was taken with data (ie, at that point did each user have a roughly similar number of check-ins); if not, hashing by that seems kind of fail for that kind of app.
Yeah, that's sort of what actually does happen.
Speed Geeks
They have thought about - MS tried to buy Yahoo. Didn't work - not too many other alternatives - maybe they'll try again. Don't forget also that MS owns a chunk of Facebook. MS is starting to go big on the cloud too, so they are getting into position slowly but surely for the future...
Surely if they moved more than 5% across, this would have freed up more memory, despite the fragmentation.

Maybe they would be better identifying regular users, these would require on the fly compacting, be hosted on one or two machines, with a third smaller server for the non-frequent users.
They say that productivity used to be measured by the number of patents filed...

Why do I feel like my taxes just went up?
Amazon Amps Up Apps Rivalry--Amazon App Store for Android Confirmed by WSJ
This is gonna sound a little snarky, but isn't that the benefit of OS code. You can do add your own features if you really want them.

Edit: Clickification:

http://documentcloud.github.com/underscore/

http://jsbin.com/

http://github.com/remy/jsbin/issues#issue/32
Useful Collection of Cheat-Sheet Desktop Wallpaper for Web Designers
OK, good point -- by all means enjoy having defended yourself. Just don't enjoy inflicting suffering on others (lest you become a bully yourself).
Here's a list of books I've put together for classes:
http://hci4.me/static/16-475-references.pdf

MIT's OCW class on UI - http://ocw.mit.edu/courses/electrical-engineering-and-comput...

Also, look at RISD, Stanford, CMU, Berkeley, Cornell (in no particular order).

If you can, anything by Doug Engelbart is amazingly motivational (inventor of the mouse) and insightful.

If you contact me, I'd be happy to send you my slides from lectures I've done on interaction and answer any questions.
Garmin developing Windows Phone 7 and Android smartphone
This is the most critical point in the thread, it seems to me!
I am interested to see when a broader coalition against the existing patent system is forming. or even if ...

The world of patents has become perverted in recent years: patents are seen as valuable things in themselves – the more the merrier – irrespective of whether they do, truly, promote innovation. Worse: in the world of software, they are actually brakes on that innovation, particularly as they begin to interact and form impenetrable patent thickets.

http://www.h-online.com/open/features/Is-Microsoft-running-o...
Ahh, not so good memories. I remember trying to make some sort of picture or pattern with the windows before the screen cleared in a few seconds.
That's a good solution. Thanks.
Wow, Longhorn is cited as putting them out of business. If only they could've seen the future...
Study shows fat men and thin women are paid more
David Gelernter is an interesting guy; I remember hearing him on NPR's OnPoint talking about his book Judaism: A Way of Being. Link if anyone's interested: http://www.onpointradio.org/2009/12/david-gelernter
I didn't realize you only had 10 guys behind the scenes, that's pretty impressive in and of itself. In all actuality this is a good problem to have and one I wouldn't mind having. It means you're growing and fast.
Monitor Trending Topics, Favourite Blogs and More with Regator
Transform your Ubuntu Box into a Karaoke Machine
Haven't used governor - the rules engine is embedded in an ecommerce platform for doing coupons/promotions with its own custom interface for defining the coupons/promos. Then those get translated into drools rules.
Movie or no, "They trust me. Dumb fucks." is the core of Facebook.
Nagios is pretty nice. It's dead easy to write custom monitors and clients are everywhere, there's even a Firefox extension. It requires a bit of learning to get going with but it's not so bad and the pay-off is big.

That said I'm looking at monit too. I hear it's quite nice and has less of a learning curve.
US anti-P2P law firms sue more in 2010 than RIAA ever did
It's more jealousy than anything. People hate the fact that someone their age or younger is living the dream -- Ivy League education, explosive traction, rapid growth, investors at your feet, multimillionaire, all during your 20's.
Overall the tips and explanations are quite direct and succinct, unlike so many other poorly written tutorials. Its connection with C# is absolutely missing though.
If there was anything Foursquare could be sued over here, we'd be seeing a different story.
Farming games actually have a pretty long history: http://www.chinasocialgames.com/?p=400

I think this type of game has occurred to many people. Back when I was doing coding for a MUD I was considering adding farming mechanics, because I thought it would be fun to come back later to the game to gather and sell the plants. More fun than the mining mechanic they had, which consisted of basically moving around the map and just typing "mine" over and over again.

But then I figured, who would care about farming when you can already kill monsters in battle. Instead I created a virtual sauna.
I don't think those are the same problem -- could you provide a link?

The problem isn't paging, per se -- the paging system is doing exactly what it should be doing, paging in blocks off of the disk and into memory as they become hot, which for this use case is always.

The problem is that you get fragmentation in your pages. If you allocate three records in a row that are 300 bytes, and then need to rewrite the first one to make it 400 bytes, or delete it altogether, you end up creating a hole there.

The typical strategy for dealing with those holes it to maintain a list of free blocks which can be used and hope that the distribution of incoming allocations neatly fits into unused chunks.

However, as they note, if you have a fully compacted 64 GB active data set and remove 5% of it you just end up with address space that looks like swiss cheese; it's not set up to elegantly shrink, but to recycle space as it grows.

There are a couple of things that I find a bit odd here though: they should have seen this coming before migrating data; it's pretty obvious from the architecture. Second is that they mention the solution being auto-compacting, which wouldn't have actually helped them.

Auto-compaction is in fact useful, but all that it does is, well, compact stuff. It means they would have hit the limits later, but once the threshold was crossed, they'd have the exact same problem. Auto-compaction is either an offline process that runs in the background or a side-effect of smarter allocation algorithms. Both of those things need time once you remove data from an instance to reclaim the holes in the address / memory / disk space ... which is exactly what they did manually.

The only really sane ways to handle something like this are notifications at appropriate levels, or block-aware data removal -- e.g. "give me stuff from the end of the file". I don't know if mongo uses continuation records and stuff like that enough to know how difficult that would be for them.

(Note: Directed Edge's graph database uses a similar IO scheme, so I'm doing some projecting of our architecture onto theirs, but I assume that the problems are very similar.)
Mirror Worlds patents that are probably the issue here:

http://www.google.com/patents/about?id=XJ4SAAAAEBAJ

http://www.google.com/patents/about?id=pcASAAAAEBAJ

Personally, I find it outrageous for an academic to demand credit for "inventing" displaying files side by side, searching files by content, and archiving files by time. The Sumerians probably already did this.
Try Ubuntu Server in the Cloud on our Dime
Edit: now that I think about it, this is a perfect use-case for Quora:

http://www.quora.com/What-SaaS-products-are-on-the-startups-...
GreenPois0n Jailbreak for iOS 4.1 coming on the 10th at 10
Pivotal Tracker ( http://www.pivotaltracker.com/ ) -> Tracking feature backlog, release planning.
What are you talking about? How can you see how the users see foursquare after this? Sure, in the hacker community this detail is appreciated, but how do you know how many foursquare users have left because of these outages? How many people are upset with the company, and are now taking facebook locations or the other companies more seriously?
Mono Success Story
Although I love Scala, I preferr Groovy/Grails over Scala/Lift for my web development needs. I would love to give Scalatra a try too.
Would it be too difficult to keep a state for every badge that you update when a new checkin happens? You'd need some specialized logic for every badge, but not much more complex than you already have. Depending on how much space you save it could be worth it.

E.g. for "you've seen foo 4 times today" you'd keep a list of the foos visited. When a new foo comes in you first remove the foos older than 1 day from the list, and insert the new foo. If the list now contains 4 foos you award the badge.

You can use a bitvector to record the badges that have any active state at all, so for badges that have no state associated with them yet (e.g. no foos seen yet) you only pay 1 bit. Or if very few badges have active state on average you could use a list of badges that have state instead of a bitvector, so that you only pay for badges that actually have active state. So total storage is 1 bit per badge or less + a couple of bytes per badge with active state?
Is either company particularly litigious? Microsoft has filed on the order of 5 patent lawsuits in the last ever.
Man Who Somehow Failed To Delete Facebook Account Complains About New Feature
Hasn't contnent avalanche been out for years? Www.contentavalanche.org has a 2007 copyright and I know I've read about it on some "guru" sites.
Also child labor laws, which are not a change in the economy per se.
For the Python world, virtualenvs and pip requirement files make deploying a local copy of a project take about 30 seconds, and it's completely independent of your overall system and all other projects you happen to be working on.
Great Article. As QA Analyst and Engineer, I know first hand about the pains of replicating production environment. Thanks for submission.
No, dropped packets are partitions. They really are. A partitionable network is modelled as one which may fail to deliver any subset of sent messages between nodes. The Gilbert and Lynch paper makes this explicit.

The consistency guarantee requires that RW histories are compatible with some sequentially consistent history on a non-concurrent RW register. Defining a total order on operations is sufficient, I believe, but not necessary (does it matter what order two consecutive reads happened in?).
I see it as a very American characteristic, from the people I have worked with before. Many Americans will never admit fault, never admit they do not know something, and always looking for someone else to blame. I think this comes from the culture of large companies. I have not seen this at smaller US companies, and other companies I have worked at.

Of course, not all Americans, but I was surprised when I first saw this.
Disclaimer: I work at Cloudkick.

We can help you all with these problems.

Here's how fast/easy it is:

1. Create an account (~30 sec)

2. Add your cloud credentials (~45 sec)

3. Install the monitoring agent (~90 sec/node)

4. Create a CPU/Memory/Disk monitor, query-targeted all your servers (~60 sec) (example: "provider:EC2")

5. You get an email whenever the monitors you created reach the thresholds you set

There are a lot of other cool features - check 'em out on the site: www.cloudkick.com

All our plans are free for 30 days.
The "designer" gets a little bit of scratch and exposure they never would have gotten otherwise. The "client" gets a design that meets their budget, and which is likely to be much better than what a professional designer would do for the same price. Who's losing out here?
Interesting approach to constructing an expertise keiretsu, similar to the kinds of experts that KP, Sequoia, and YC each have in a local orbit.
Android has a patent fee. It's not like Android's free. You do have to license patents. HTC's signed a license with us and you're going to see license fees clearly for Android as well as for Windows.

seems to be the only good thing happening in the mobile world at the moment for microsoft.
As one of the two (former) owners of vb.ly that got pulled, I'm sure bit.ly is concerned (and should be).

Given he was having his investors sign contracts on a $10m Series B investment the same day, I'm not surprised John Borthwick said 'it was just noise' -- I'd have done the exact thing myself. It's what you do as an entrepreneur to get shit done.

Doesn't mean it is just noise for them.

The peanut gallery have gone to great lengths to tell us Libya is in its right to set whatever rules it likes about domains (and even content of sites), that it is reasonable for Libya to change those rules at any time and that it is even ok for Libya to interpret those laws differently on a case-by-case basis. (wow).

So with that in mind, I'd be fairly concerned.

Given the way things seem to be going over at NIC.ly I'd be interested to see what kind of arrangement NIC.ly propositions to Bit.ly if/when they decide they no longer like bit.ly linking to depictions of the prophet Mohammed (PBUH) or hardcore pornography. I'm sure they won't simply can the domain but come to some other arrangement (ie financial) to resolve this.

And if I was bit.ly, with everything I have in play now, I'd accept rather than have my domain taken away from me and a significant part of my business shut down. It's just business (Internet = Serious Business)
Compulsory schooling is the top of my list, as any society which values freedom must value the freedom of one's own mind above all.
Google Code-in: Encourage kids to code
While I am doing the same exact thing, I think you're running sideways of the point being made here.

The point isn't that minimalism can't help you achieve a particular set of goals, but rather that it isn't superior to anything else.

The implied alternative to minimalism is not egregious prodigality but rather measured enjoyment of life that doesn't strip you bare but doesn't attempt to max out your financial high score.

I'm a minimalist, but I try very very hard not to come off as the aforementioned "sanctimonious anti-consumerist" type. I'm a minimalist because I really really care about my work, and I don't want much.

Well that's just dandy for me, but my values are not other peoples' values, and I can see how other people would prefer to make use of their time in a manner that makes them more immediately happy.

Don't mistake the apparent fiscal conservatism and "safety" of minimalism as somehow being superior or safer than enjoying and consuming things as they are now.

I would in fact argue, that as a minimalism I am taking a very great risk with my life compared to someone who gets out more and has more fun.

I am taking a risk with my time and my life, because if I get hit by a bus tomorrow before I've had the time to amass the money->time necessary to work on the projects I'd like to pursue, I'm going to die screaming in anger at my self.

Someone who's taken a less concentrated approach to being happy with their lives probably isn't going to be nearly as upset as they lay bleeding under that bus that ran them over.

You and I are risk takers of a different kind.

Our values are not their values, and those values are not inferior nor superior to anyones'. It's all case specific.

If you value money more than most other things in life, then of course minimalism will be a "superior" means to the end you seek, but it's not somehow morally superior.

The man that puts aside his career/continued financial growth to spend time with his children, is in my eyes, the greater person than the one who manages to become a CTO by 40 at the expense of said family.

But those are my values and perspective, and they're still not the choices I would make at this point in my life.

I'm sad that a hackerne.ws user would be so myopically acquisitive without understanding the greater implications in terms of risk and how you spend your time.

I'll probably make a proper blog post on this at some point.
I don't think those are the same problem -- could you provide a link?

You are correct, the actual problem is paging out LRU keys as opposed to memory holes. The issue is related but not the same.

From http://antirez.com/post/what-is-wrong-with-2006-programming....

Multiply this for all the keys you have in memory and try visualizing it in your mind: These are a lot of small objects. What happens is simple to explain, every single page of 4k will have a mix of many different values. For a page to be swapped on disk by the OS it requires that all contained objects should belong to rarely used keys. In practical terms the OS will not be able to swap a single page at all even if just 10% of the dataset is used.
For a distributed system to be continuously available, every request received by a non-failing node in the system must result in a response.

For a CA system, any node which is unable to assure global consistency reports itself as failed. It will neither return bogus results, nor hang indefinitely.
Any specific book recommendations?
IMF chief's warning of currency war 'real threat'
Also this is an example of where it is reasonable to build out a service that is itself a feature but a valuable one for a potential acquirer.

If AOL, Microsoft, etc wanted to set up a url shortener, it might be worth their while simply obtaining the technical infrastructure, domain knowledge of the staff, business deals and of course all the existing links + data then to just start their own from scratch.

(Mind you having taken $14m-$15m that does now make it an expensive aquisition. I'd have thought it a no brainer for the AOL/Microsoft/Yahoo's to pick up bit.ly for $20-$40m)
I haven't sat through a team meeting or buzzword filled death by powerpoint for 5 years. I haven't had to attend corporate vibe sessions or listened to compliance officers drone on about anything. I also, 5 years ago, looked at a map and said 'where do I really want to live', then moved there. My whole life philosophy was to create the lifestyle I wanted, and then work out how to pay for it. It's a powerful motivator. Because most people first find a job, which then locates them on the map, and they then try and fill in the rest of their lifestyle based around those two parameters.

The thing that keeps me going is the thought : well, I can always go and grovel for a permanent job somewhere. But I think I'm pretty much unemployable now, in the traditional 9-5 sense. I get 2-3 requests for work per day at the moment, but pass on most because, like the blog post says, people aren't serious or are expecting low prices.
I'm afraid you're not quite correct. CAP says that, in the face of potentially arbitrary network partitions (which are precisely modelled by a set of dropped messages) you can be 100% consistent, or 100% available, but you can't be both.

If the network is allowed to drop packets there are times where either you must not respond to requests (as doing so would violate sequential consistency) or you must respond incorrectly, potentially with stale information.

The network partition that forces you to drop one of these guarantees might be quite dramatic - but note that, for example, a quorum system is only available on the majority side of its partition. Therefore if a single node is unable to deliver messages (due to a network partition event), it will not be able to correctly respond to requests and must either not respond to its clients or respond incorrectly.
arguments = Array.prototype.slice.call(arguments);
I have mixed feelings about this.

I think Gerlernter is an important computer science visionary. There are still good ideas for today's interfaces in his 1993 book predating the web 'Mirror Worlds'.

But I also don't think he deserves $625 million for having patented some of his ideas/predictions about computer interfaces.

In his own writings, the tone is often prophetic: not "here's my nifty invention" but "this is the inevitable path we're on". Seeing what everyone else will see a few years early is a valuable skill; it's not so valuable it needs to be rewarded with a 20-year patent monopoly.
Github.

Mailchimp.
Steven Johnson on Innovation & Ideas
RichCopy, an Advanced Alternative to RoboCopy
Is your time so lacking in value that you'll expend a substantial amount of time rolling and maintaining a custom distro of Linux that'll save maybe 50-150 megabytes of RAM? Are you in the web hosting business or something?
Microsoft announces NuPack, a new package manager for open source .NET libraries
Apple might have lost this time. But the only winners are the big companies when it comes to patents. MS and others have a huge portfolios that crushes everyone who does anything like what they have a patent for. Why are you glad? Do you think patent laws will be overturned by this in the future? They're all just going to continue playing the same game.
Dup (same text, different website) of http://news.ycombinator.com/item?id=1755831
I'll roll up all the answers into a spreadsheet or table soon.
In asynchronous networks it is surprisingly hard to detect failures, even of yourself.

Reporting an error condition counts as an availability violation.
Not into Quora? As several people answer it opens up a community-editable answer summary to bring them all together.
Yeah, I think it's common, if unstated, knowledge in the US that's the reason for it. Fear of getting sued by some opportunistic lawyer looking for a quick buck.

When we're growing up our parents teach us to take responsibility for our actions, and if we screw up or wrong someone, admit it and make it right. Then we get out in the real world and it's the exact opposite, and the higher you go the worse it seems to get.
No, it's exactly the same. Availability is a guarantee that all requests are eventually responded to within some time bound, whatever that is. During the partition, availability is violated.

Therefore it's not a CA system, but a C system.
Shrug. Harder to organize. Maybe when it's done?
This blog post that I wrote a few months ago also explains the same issue, and may be of interest for those looking for a separate explanation:

http://www.cloudera.com/blog/2010/04/cap-confusion-problems-...
Perelman might not be the right example, for him choosing to stay poor by denying to accept the Fields Metal.

just by the way ...
Can you design the system such that you can turn off ("darkmode") features in an emergency? For example, in an emergency you can turn off badge-awarding as it is known to be RAM intensive (and degrade by seeking to disk).

To the extent that you do this across the board, you'll have yet another tool to defend against being over capacity.
While I usually frown on such blatant self promotion, I couldnt help but upvote this for being such a good advertisement, I want to use you now.
What level of traffic have you received so far?
http://ringojs.org/wiki/web_framework/
Exposure where?

The rhetoric involved is an extension of the line "I can't pay you for this, but you could use it in your portfolio".

The whole process reeks of exploitation. It's bullshit, imo.
not snarky at all. in fact it was what i thought just after hitting submit. but it's php and i was hoping to avoid learning it...

I am in the middle of CoffeeScript, Scheme, AWK, ZSH, and git.
or, as Balzac wrote in his novel Le Père Goriot:

behind every great fortune lies a great crime
you can analyze the statistics of when/where specific people are most probable to check-in and pre-load (or even pre-calculate) the data in advance.

For example, starting at ~9pm the data about check-ins into library may be safely unloaded to disk (the probability of needing this data is low and reading from disk for whose rare cases would do just fine) and be replaced in memory with data about check-ins to clubs, etc...