skysight-inc/HackerNews huggingface dataset:
Sure, they did it. But they didn't do it on a computer. Then on the internet. Then on a mobile device. They are all completely different issues, worthy of an extra patent or two.

I am just waiting for the next big thing, so I can patent everything all over again. Do you think iPads count as computers or a completely new device?
Other threads I want to do:

- web-includable services that integrate into a web page

- production services (aws heroku etc.)
The only logical limit is conception.

If you accept conception as a logical limit, then you'll have to explain the relevant moral distinction between a human embryo and a fish embryo, besides that one can eventually become a person, given the right external conditions (an ovum can too).

I propose sentience as a logical limit, acquired I suspect late in pregnancy.
Pretty much all software patents boil down to "method of organizing information".
Suggestion:

In additions to hiring scalability experts (many will claim to be experts but most aren't), talk to your investors to find outside technical advisors who have worked on and are still working on similar problems.

Find advisors from Google, Facebook, and other places who have dealt with these issues. In my experience, you have as much to learn from people who have made mistakes as from people who have succeeded.

Also, in my judgement, you guys are a little too risk-tolerant for your significance (first major Scala 2.8 upgrade, largest MongoDB deploy...).
Acquisitions: Pulling Back the Covers
I'm really into Postmark as an SMTP replacement for transactional emails - http://www.postmarkapp.com

And MailChimp for mailing lists.
We have a framework for doing this sort of thing but it's only implemented around certain features so it doesn't provide us a ton of benefit at the moment. Obviously doing work to give us more control like this could be a big win, so we'll probably be doing some of that.
The lack of sets as a data type at the language level is a serious problem with JS. Most languages which implement sets as a library make some assumption about the elements: they're ordered, for example, or they all have a unique string serialization. If you want to make a set data type in JavaScript, the language doesn't give you any of those assumptions. You might end up just using linear search of unordered arrays. (This is actually very fast for small sets, but still.)

I wrote some more about this here:

http://finger-tree.blogspot.com/2010/09/sets-in-javascript.h...

And here is a library that offers sets in JS:

http://github.com/PeterScott/simplesets-nodejs

If you're using Node.js, you can get it with npm. It could be adapted to the browser pretty easily. This is what I use, and it makes my life so much easier.
You've just touched on why the 'release early, release often' mantra, in some cases, makes sense.

If he were in fact going to be beaten by Longhorn (who knows), it's likely that Apple would have gotten him anyway, but this proves that giving up on the rumor that Microsoft is swinging into your space is naive, pussy-footed thinking at its finest.

If, in fact, Longhorn WAS his only threat, he would have had what, 6 years in which to prosper, and possibly even dominate as a 'preferred' alternative to what Microsoft was offering.

He did just fine on his own, of course, so the conversation is moot.
Not entirely. People change their mind.

So the question will be (and mainly already is): when is too late to change your mind?
Precisely. Real capital market value, based on demand vs VC paper games.
I don't use NoSQL (yet) but I'm curious if there would be a better key to shard by (instead of user_id)?
Any comment with the words "web scale" should be considered a joke until proven otherwise. :)
I seem to run into everyone that hasn't heard of robocopy (or at least doesn't use it when they should be) and think, how can you not known about that... but I hadn't heard of richcopy, so I guess I better stop doing that.
On Acquisitions: Focus on the personal decision of the entrepreneur. +1
I dont understand this, you seem to think that Microsoft has ignored the web when they are in fact one of the largest web companies on the planet. There is almost no company they could buy who would not overlap in concerns. You might argue that some companies are better at certain things or have larger customer bases but this idea that they should buy a web company and quit holding on to the desktop is just goofy.

All that said, one important web asset MSFT doesnt have a strong play in is web analytics.
But he followed up by forming a company, developing, and shipping actual software based on these ideas. This doesn't look like mere patent trolling.
I like coffeescript, but it is definitely a very young language. When I installed it, I had to modify the emacs editing mode because the language syntax had changed recently.

I still use it, though, because it's just so much nicer than JavaScript in every other way.
That number is ten times higher than many (I think most) other estimates. Is there a reason you prefer their methodology?
They actually said 4. Two shards with slaves for redundancy. It was apparently working fine until they lost track of what was going on. :)
This is probably feasible for some badges, but would be hard to do for all of them.

This approach also increases the complexity of adding new badges, which is undesirable for product and business reasons.

It could certainly help in some case though, and it's something we're considering.
Yeah an A
What antirez is getting at there is actually a much harder (and more interesting) problem that could be generalized as something like "efficient data locality for mixed latency access".

However, assuming that all data must actually be in memory (as stated in the posted email), you don't actually solve the Mongo problem with more efficient organization of the data set, though compacting could be considered a sub-problem of the one that antirez describes.

But as I noted in my earlier comment, compacting wouldn't have actually solved their problems, it just would have delayed them. It's reasonable to ask if all of their data truly needs to be hot, but even there, you'd eventually hit diminishing returns as you approached the threshold where your active set couldn't fit in memory, and there smarter data organization wouldn't actually fix things once you started pulling chunks out for sharding; you'd still need to recompact.
I agree with all of the above.

And also, for the love of all that is holy, when you offer a sandbox and test data, give me a way to deliberately trigger error responses.

It's all well and good to give me an is_test parameter on your payment API, but if using it causes you to blindly approve all payments, I can't test all of the code paths dealing with all of the possible failure modes(true story).
I just don't see the synergies of a full takeover beyond owning the rights to the PDF format. A full takeover would result in Microsoft paying a high premium for a set of desktop design applications (CS suite) that are not part of its DNA. Microsoft excels at building excellent tools for developers, but never seems to do a good job with design tools. Adobe is the opposite.

What's more possible is some sort of partnership that enables stronger PDF support in Office, improved Flash support in Windows Phone and better integration between Office and CS applications.

Microsoft's primary business is selling Windows and Office desktop licenses. Since many Adobe CS users also own Windows and Office licenses, better integration between the two suites would be beneficial for both companies. Introducing a new mobile phone OS based on Flash/Flex/Silverlight or some other combination would not be worth the acquisition cost.
I would have only taken 1 million to tell them it's a bad investment. This whole URL shortening business is so pathetic. OK, so twitter decided to be SMS compatible and limit messages to 140 characters, so what? It's an arbitrary limit attached to a standard that has already reached it's peak and can only go down from here. With 3G and smartphones becoming ubiquitous less and less people are going to be using sms for their messaging and the need to limit messages to 140 characters will diminish. It's a waining market not a startup opportunity.
Spotify crashes into Apple on way to U.S.
State of un-change
I've found that developing against a VM is a great tool for a dev. For OpenStack swift, we develop all of our code against a VM. Since we deploy on debian and I develop on a Mac, I can use OS X tools to develop (TextMate, etc) and code against an environment similar to our production. After code works on a VM, we can move to a lab or a staging environment to do more scale testing before deploying to production.

Developing against a VM is not something I did before working at Rackspace, and I find the practice so useful that I will do what I can to continue doing it no matter where I work in the future.
yes that's certainly true. if there isn't a sensible option for a merge policy and non technical users have to resolve it manually you're fighting a losing battle in an AP system.
Prior art: Zoomracks. See http://en.wikipedia.org/wiki/Zoomracks
I think HTML Xtreme would do fine.
Dell offers an opportunity to these employees which, while lousy by Western standards, is better, in their judgment, than the alternatives available to them. That's why they have voluntarily accepted the position. Dell has made their lives better, not worse.

Slaves didn't agree to be slaves.
Reviewing The Social Network - Constructing Grand Narrative
Airplanes are far safer, per mile, than buses or trains (about 10x less likely to die), never mind cars (60x).

Edit: corrected numbers.
Alex Payne — Settling Down Without Settling
I've used GitHub and Unfuddle for source code. Both are great.

SendGrid for email delivery from my apps.

Google Apps for email on our domains.

S3 to host images/videos (we deal with a lot of them).

I've tried just about every mailing list service there is. I haven't "fell in love" with any of them but MailChimp and Campaign Monitor are my favorites.

Of course, there are now several recurring billing services. I've tried a few and have mixed feelings. Real slow response on support requests from a couple–which IMO is bad for a company that touches your money.

For customer support tickets, we had a home-built rails app but are currently transitioning to ZenDesk. Tender Support is another option.
From the email thread, it sounds like the decision to shard on UID was made mostly to increase locality of data, so that you didn't have to query more than one node to get a single user's data.

There's no silver bullet here. Hashing on insertion order would basically guarantee that writes would favor one node over another, which random hashes would force you to aggregate results from all available nodes for each query.
The linked in engineering team just deployed scalatra as part of their big new faceting search engine.

http://sna-projects.com/blog/2010/10/linkedin-signal-a-look-...
I came here to say the same thing : C# makes it intentionally very difficult to create a global variable, and the first example is just a case of sloppy coding, whatever the language. You cannot develop a habit of creating global (or undeclared) variables if you're coming from a C# background.

Similarly, the global namespacing - a C# developer is going to look for ways to declare namespace.definition style, and is not going to spontaneously come up with namespace_definition, as this is an unseen pattern in C# examples and general practice.

Perhaps it would be a good primer for a VB6 -> Javascript developer, but I don't imagine there are too many of those around.
A hacker's Blackberry
Who are North Korea’s 13 Twitter friends?
Who are the 13 Twitter users that North Korea has chosen to follow using their official uriminzok Twitter account?
Crowd sourcing doesn't mean "no pay", you should probably check some quality crowd sourcing sites, like TopCoder.
Conveniently scheduled on a day-job vacation day, and conveniently located at CIC!
Why on earth would you want to keep 236 million check-in documents (66 gig / 300 bytes) in memory ?

Here is an idea: write an algorithm that keeps the most recently used 100 million check-in documents in memory. That'll save 38 gig of RAM.

Or how about this idea from the 1970s: write an algorithm that keeps the most recently used 4 gig of check-in documents in memory. That will save 62 gig of RAM, and the most recently active 14 million users (4 gig / 300 bytes) will still enjoy instantaneous response.

Can someone help me out with the math here?
I have a Rails app I want to distribute as a VM to each member of my team. What's the best option that's just 50 megs bigger than Tiny Core?
Writing your interface in the native language/toolkit and then interfacing it to a cross-platform backend is often a better solution than trying to shoe-horn everything into one GUI that won't fit to any OS properly. Which is especially true for Macs.

I remember that the imeem guys did something similar with their chat/social app (defunct now).
Old Overholdt is perfectly palatable. I don't think it's worth knocking.
The fail here isn't just that you should have set up a shard 12 hours before or whatever. The other fail is that it seems (from the groups post) that you were relying on your data set only fitting in RAM but the underlying I/O system wasn't quite up to scratch in terms of being able to page in/out that additional 1GB of data.

It's not like your 66GB is hot all the time (or even big by any sort of measure), so that makes no sense to me, and that part of things needs to be further explained by 10gen and/or 4sq.
Err...I meant "Equivalent value at a lower cost is not a competitive advantage.". It's a comparative advantage because you are comparing 2 numbers or features.
I am unfamiliar with the details of JavaScript's semantics, so I won't be able to tell at a glance what is part of the language and what is part of a library.
I'm especially unconvinced by "cargo cults" and "w3c standards" being put next to each other as equivalents.

But I'm also generally unconvinced by the larger point; "the right way" might have a longer learning curve, but once you know it there's no reason it should be any slower than the "wrong" way.
I was speaking in RAM, not the disk-space of a VM. I didn't know the context of what you were looking for.

Is there some reason you can't accomplish this using RVM and a dev database? Any specific software that makes you think it isn't practical to do so?

Virtualization seems a little heavy duty given the options for sandboxing python and ruby apps these days.

Edit: You really really don't want to cope with the raw horror and pain of a minimalist kernel that doesn't provide any of the things you usually take for granted.
Reminds me of grabbing a random .dll, renaming it to .doc, putting it on a floppy disk and handing it in, all to gain myself a few extra days to finish my assignment.
> save maybe 50-150 megabytes

I think my typical server setup idles at 30~40mb of ram before I add services.

At the moment with nginx (with passenger, no active rails processes), cloudkick-agent, a fat irssi and some php-fastcgi workers I'm only using 122mb of ram. I don't know how much ram you really have to shave off.
Isn't that what the on-demanding disk cache (and paging) from the OS give you? Only the data being accessed and used are in memory.

Unless they're constantly doing data churning over the whole dataset, there is no need to keep everything in memory.
More or less my point. the ~150 additional megabytes was if they were deploying with Ubuntu + some kind of java server management software (worst case scenario for a base deployment template?)
I suspect most of check-ins are happening in big cities such as NYC, SF etc. So some sort of "geo-based sharding" would be more helpful.

Also I recommend giving @naveen a separate shard. The guy just keeps roaming around in the city. Thank you.
> HN has its own very limited markup syntax

Just what we need, another subset of Markdown :(
I'm glad I'm not the only one. I was worried for a minute, and then I thought "wait, I'm on Ubuntu. How did IE crash?"
He may have tried, but his execution wasn't anywhere near Apple's. Instead of suing Apple for money, he should have praise Apple for making his vision a reality
I'll also chime in and support Dynamics. It's an amazing market with a ton of actual growth potential.
This is a large company / management culture thing, not a US thing. Also, it's a techie versus non-techie thing.

People in Canada and the US are more prone to admit mistakes than in more prestige / respect-oriented cultures. I.e. apparent lack of respect has its benefits.
"Steven Jobs may be no inventor, but to his credit he recognized elegance when he saw it and understood its significance." -- David Gelernter, in his 1998 book Machine Beauty.

About ten pages after the above quote, Gelernter describes pretty much everything this suit was about in some detail, e.g. a use of the "time travel" metaphor very much like Time Machine's Finder interface, wherein the whole interface reverts to a previous state. Even if you don't agree with the verdict (or the damages, yikes) it's unthinkable after reexamining this book that folks at Apple didn't get some ideas from Gelernter's work, perhaps without even realizing it.

Incidentally, it is a great book that I highly recommend.
Google Apps is the big one for us - especially mail and calendars. I run Redmine off Ubuntu Server inside VirtualBox...can't think of anything else we might be using. It is tempting to switch to Chargify but we're still on Auth.net for the time being.
Or continue to use the last available version for Mac until a viable replacement is available for their preferred platform.
Gotcha. With git and a Phing build script, I can do the same in PHP :)
As Pewpewarrows said, you can use pip with a virtualenv to deploy with one command. If your settings.py is any good, you can load your data dump in an sqlite database in one more command, and you're ready to go.

We used to develop historious like this, until we switched to redis/celery. Then it became too much hassle to install these in every machine, plus we wanted to test on the db we used (postgres) rather than sqlite, so we made a VM and now everything's dandy.
What I don't understand is why Skype would block 3G calling in the first place. What's the incentive for them? I understand why carriers wouldn't like it, but what power do they have over the Skype app?
I've had good experience with DNSMadeEasy.com. We actually kept DNS/bind configs in version control and used XFR to let DNSMadeEasy slurp them up.

Jim
verbing nouns does indeed weird language, but that's not the point here.

The point that analysis of HTML5 as a standard is not the point of the current effort to improve the web, and improve computing in general.

It does not matter where HTML5 and Flash happen to stand at the moment in aggregate usage across companies. Who cares. There are many reasons for adoption of each technology, dependent on the circumstance of each company and their business models.

That's why HTML5 as a noun is a bad banner, avatar, symbol, whatever. Web2.0, nebulous as it may have been, could not be confused with an actual object.

HTML5 suffers the same problem that XML did. People thought anything that touched XML was going to be the wave of the future, and improve everyone's lives, and be both machine and human readable, and lead to a revolution in how metadata was transported.

It is true that XML lead to cool new capabilities on the web. It also was not a panacea, nor did it instantly make any project that used it awesome (XML config files are one of the most heinous digital inventions ever).

People should not confuse the movement to create better web technologies, with the web technologies themselves, or we will all miss the point.
You do need a VM if you use memcached/redis/some other thing and don't want to install it on your machine. Other than that (for 90% of cases), virtualenv has you covered.
Unlogo: Web service that eliminates logos from videos
I find it interesting that with all this talk of scaling, and the specialized tools that go along with it, 4sq's service was essentially running on 2 servers.

Maybe I only need 1 server to be profitable. Hmm...
Anything that is both latency sensitive (web apps are) and requires high throughput is going to be RAM-centric. Even services operating at much smaller scales with "old school" single DB boxes need to keep the vast majority of active data in RAM and use things like battery-backed write caches to get acceptable write throughput. Even with the sophisticated tricks mature RDBMS software uses to squeeze every ounce of performance out of spindles, most "old school" DBAs will tell you that to get high throughput for a database you basically need to have RAM = database size.
Paging doesn't happen in 300 byte chunks, though. It's usually 4 KB, so if a record or one of twelve neighbors has been touched, it stays in. The granularity isn't right.
The verdict would suggest, and it's a common misunderstanding to think that, the person who comes up with the idea first is more important for innovation than the person who executes it. This is encouraging talkers and discouraging doers.

Yes Gelernter may be the first person that came up with the "mirror world" idea (not to mention he has already made money from the books he sold), but instead of suing Apple for money, he should have praise Apple for making his vision a reality.
and out comes mongo's dirty little secret - you have to have enough ram in your boxes to hold not just all the data in ram, but all the indexes too, or it completely shits the bed.

putting hundreds of gigs of ram in a box isn't cheap.

are the foursquare folks considering rewriting with a traditional datastore like postgres and some memcached in front of it?
Freeing memory would require that there be an entirely empty page. Each page holds 13-14 objects (4k/300). The chances that 13 consecutive objects were in the 5% (assuming independent, random distribution) is 1 in 20^13, putting the expected number of empty pages well below 1. You have to migrate much more data before you can hope to see page-size holes.
I'm summarizing things here:

http://spreadsheets.google.com/pub?key=tmxXdwODQTCsdQzatHS6D...
Nope, me too. I can see maybe using a prompt like this temporarily, or putting this info in an XTerm's title, but I spend most of my time on the command line and really don't want crap like this distracting me.

My prompt on localhost is always "$ " or "# ", but I add user and host on remote machines to avoid making any big mistakes on the servers I manage.
Super optimized JavaScript framework for mobile based on the Dojo Toolkit
My Indyarocks: Amazing cigarette Magic
..except that he didn't. What he actually wrote was (one translation):

"Behind a great success for which there is no obvious source is a crime that has never been discovered, because it was properly executed."

Bear in mind this is a translation from 19th century french to modern english, so there's a lot of wiggle room for getting different meanings.

The common, shorter version that is bandied about on the internet drops the important qualifier "success which there is no obvious source for" - in other words, there's only a crime behind a success that has no apparent reason.

Also keep in context that this was written ~7 years before the French revolution, about French society. People should keep that in mind when applying that to modern situations.

In Facebook's example, I'm sure most will agree the reason for the success is pretty apparent, and thus no 'great crime' was committed.
How do you explain Paxos, then? How does a dropped packet prevent the system from responding to queries? How about if I broadcast every response 10 times to everyone I know? How many packets must be dropped for the system to be considered unavailable?
Ad-hoc Wireless Mesh Networking for Android