For many apps even 64G is small. The problem is less about the height of the ceiling but about the fact that a ceiling exists - and that redis effectively stops working when it's reached. I agree that there are applications where redis is a perfect fit, but for many others this limit is a serious problem. Also note that many projects, especially those just starting out, simply can't afford to start with 64G servers. 4-8G is a more realistic scale to assume and that will fill up faster than anyone likes.

Yes I understand this concerns, and this is why I'm trying to address this with virtual memory and redis-cluster (a proxy that takes care of handling fault tolerant consistent hashing). But my point is that there is also a cultural barrier about this issues. with 8 GB of RAM, especially if you are starting up and if you take care of selecting a data layout that is cheap, there is a lot of data you can put in memory. Another interesting alternative is to put only "hot" data (metadata) on Redis, and use another on-disk DB for the rest.

FWIW, the EC2 high-memory instance (64G) you cite in your FAQ goes for a swift $1728 USD/month.

High performance DBs and EC2 IMHO are not a great fit. Not only RAM is expensive, but memory bandwidth is not optimal.

EC2 is just expansive from every angle you see it, it's not a problem just with Redis, but also with MySQL performances as memory is crucial to make MySQL working well.

Sharding is always worthwhile for scaling but it's a fairly delicate subject (esp. the rebalancing after add/remove of a shard and queries spanning multiple shards) and I haven't seen a shrinkwrapped solution for redis, yet.

Application level partitioning is a good option I think. It's easy to manage, for instance you take your users in an instance, their blog posts in another, and comments in another one. Are you little and low traffic in the start and can't buy three hosts? Just use three Redis instances in the same box, and move them in different hosts as you grow.

So, while an option, most people will probably rather use a competing product (e.g. MongoDB) before opening that can of worms.

MongoDB and Redis are very different products. If MongoDB is a good fit for your application, use it, it's great. But if you need Redis, MongoDB is not a drop in replacement in any way IMHO.

Yup, that would be me. Once the RAM-limitation is gone redis will suddenly become very interesting to me.

This will only work well if data access pattern is biased btw. And the virtual memory will not remove completely the dataset size limitation. For instance if you have 2 GB of RAM it will make sense to setup VM to have up to 32 GB of data for instance, given a biased enough access pattern, but it's not like it will work well with 1 TB of data.

So I in short I think: most in-memory barrier is cultural. There are solutions to distribute among different servers that are not hard to implement and maintain. In every kind of DB the memory should be proportional to the dataset for it to scale, and with some kind of evenly distributed data access pattern you need to take everything in memory anyway.

Also the fact that in Redis writes are as cheap as reads is not something to forgot. There are many applications where it will be much more viable to have more RAM that scaling concerns with writes.

Redis is not for everything, but I think there is a domain of applications where it is a very good fit.