[Sony Xperia S, Acro S, Ion] RaymanFX:
Well, the rabbit hole goes deep..

Let me try to explain :)

At first, you need to use the caf display driver, not the 'pure' mako one.

build.prop must contain the following defines to enable the 'layer issue effect' (and generate *incredible* smoothness - not a joke, really):

debug.composition.type=dyn
persist.hwc.mdpcomp.enable=true

Where especially the second line is causing the issue.

From what I understand, it forces MDP composition as the default composition being used by the hwcomposer.

Commenting that line in build.prop takes it all back to normal, but also returns the lag on notification shade pulldown.

Moreover, as to my understanding, it should be an issue with layers being underrun by MDP since hwc.dpcomp.cpp from display-caf says this:

sEnabled = false;
if((property_get("persist.hwc.mdpcomp.enable", property, NULL) > 0) &&
(!strncmp(property, "1", PROPERTY_VALUE_MAX ) ||
(!strncasecmp(property,"true", PROPERTY_VALUE_MAX )))) {
sEnabled = true;
}

...

bool MDPComp::prepare(hwc_context_t *ctx,
hwc_display_contents_1_t* list) {
if(!isEnabled()) {
ALOGE_IF(isDebug(),"%s: MDP Comp. not enabled.", __FUNCTION__);
return false;
}

overlay::eek:verlay& ov = *ctx->mOverlay;
bool isMDPCompUsed = true;
[B][I]********* it goes on after here, but if 'isEnabled' is false, it will return 'false' for MDP compostition immediately[/I][/B]

...

bool MDPCompHighRes::draw(hwc_context_t *ctx, hwc_display_contents_1_t* list) {

if(!isEnabled() || !isUsed()) {
ALOGD_IF(isDebug(),"%s: MDP Comp not configured", __FUNCTION__);
return true;
}

if(!ctx || !list) {
ALOGE("%s: invalid contxt or list",__FUNCTION__);
return false;
}

/* reset Invalidator */
if(idleInvalidator)
idleInvalidator->markForSleep();

const int dpy = HWC_DISPLAY_PRIMARY;
overlay::eek:verlay& ov = *ctx->mOverlay;
LayerProp *layerProp = ctx->layerProp[dpy];

int numHwLayers = ctx->listStats[dpy].numAppLayers;
for(int i = 0; i < numHwLayers; i++ )
{
hwc_layer_1_t *layer = &list->hwLayers[i];
private_handle_t *hnd = (private_handle_t *)layer->handle;
if(!hnd) {
ALOGE("%s handle null", __FUNCTION__);
return false;
}

[B][I]************ still goes on here, but I guess you see what I mean. If the build.prop property is set, it draws the layers using MDP, which somehow fails big time on our device, since it wants to use more layers than actually available.[/I][/B]

...

bool MDPComp::isDoable(hwc_context_t *ctx,
hwc_display_contents_1_t* list) {
//Number of layers
const int dpy = HWC_DISPLAY_PRIMARY;
int numAppLayers = ctx->listStats[dpy].numAppLayers;
int numDMAPipes = qdutils::MDPVersion::getInstance().getDMAPipes();

overlay::eek:verlay& ov = *ctx->mOverlay;
[B][I]*********** this is where I attempted to limit the layer usage using another build.prop define.. (or atleast make it stop using MDP composition, but the issue is still present...[/I][/B]
int availablePipes = 0;
char property[PROPERTY_VALUE_MAX];
if(property_get("debug.mdpcomp.maxlayer", property, NULL) > 0) {
if(atoi(property) != 0)
availablePipes = atoi(property);
}

if(ctx->mNeedsRotator)
availablePipes -= numDMAPipes;

if(numAppLayers < 1 || numAppLayers > MAX_PIPES_PER_MIXER ||
pipesNeeded(ctx, list) > availablePipes) {
ALOGD_IF(isDebug(), "%s: Unsupported number of layers",__FUNCTION__);
return false;
}

I will keep on trying and see if I can get something. The CM Samsung msm8660 devices seem to be using this define (MDP composition enforced for hwc) without major issues, being still on linux 3.0. They based their vide/msm drivers upon jb_mr1.1_chocolate over at CAF, AFAIK.