Heuristics is something that AV companies tac on when they mean they don't update their DB too often ;P

In general heuristics work very poorly for binary detection some of them might look for various patterns e.g. block the creation of registry entries from non-installer based malware (which is a large amount of false positives) or look for various interactions like hooking into certain applications or functions but in general I haven't seen a good heuristics engine as of yet.

There are some very interesting machine learning tools but they detect large scale anomalies that could indicate a breach, there are also some machine learning binary analysis tools but they work on a different level and they perform something more akin to reverse engineering.

Signature based detection has it's uses and that's to detect common crap that is here to say. If you look at current trends then 70-90* of malware is unique for each organization (the large gap is due to different verticals) most of that malware when detected won't receive a public signature their AV vendor will release it for that organization only and maybe distribute it among that vertical or to similar organizations. And while you might think well that's bull you don't really want all those signatures anyhow for the AV DB to be effective it needs to be fast so vendors have to limit the amount of signatures they carry, to do that they also tend to remove malware which all of it's exploit have been patched or outdated malware which is no longer relevant (e.g. very small amount of infections). The signatures vary from time to time and if there will be an outbreak of old malware that has been removed the vendors will introduce those signatures again into the DB and they became quite good at predicting outbreaks of common malware.

Now if AV's only detect 10-30% of stuff what detects the rest? Sandboxing doesn't really work, Windows isn't really built for that and to implement a security solution that actually provides true sandboxing for all applications is a nightmare, so you are left pretty much with detecting anomalies across the network. The malware it self is usually then discovered once a breach or an incident has been detected, the most common point of detection is when a breach is ongoing and you discover the egress, sometimes if you have really good monitoring across the board than your FIM(file integrity monitoring) or your user activity monitoring will start yelling very quickly.

Now AV's still have their role they are quick and dirty an they usually work for most (home) users for stuff like thumb drives and simple mail attachment crap, and they are still one of the only tools that can provide real time on-excute/write binary detection most of the new heavy weights are based on detecting the malware post exploit.

Some security solutions now claim to be "anti-exploit" the claim part is because their success rate varies they usually are much more focused for example will only protect browsers, office suits, and common PDF reader and what they do is a couple of things: 1 they know how an existing known exploit will look like when it's being triggered in the application and they will terminate it before the payload is executed (hopefully ;)), 2 they have the ability to detect various exploit oriented artifacts like ROP detection, 3 they do some behavior analysts as they usually only protect 10 or less applications they can application specific heuristics rather than attempt at creating general ones.

Like AV's they usually excel at 1, 2 is tricky especially on 64bit applications because the virtual memory now is so large that it's hard (well impossible blindly) to look through it so ironically enough some "anti-exploits" apps actually require you to disable ASLR (a security feature randomized the memory address of an application across the entire/most of 64bit virtual address space to make it harder to exploit B/O's), and some of them are becoming quite good at 3.

If you are an individual keeping basic opsec, using free AV for generic malware, and not running plugins like Java/Flash automatically should be good enough. If you are an organization then you need to have solutions in place across all tiers to actually detect malware, or at least it's activity since if you are infected you are most likely have been targeted so even if it's not a zero day it will be something unique for you.