[Nexus 5 Original Android Development] _zagto:
I started with transplanting files from z3's lastest build to mine an see if it the lag goes away. First I tried boot.img which did not change anything. Second guess was libsurfaceflinger.so because that is the component responsible "Faking VSYNC ..." messages in logcat. And there was no more lag.

Now it's time to compare the files what is actually different. To disassemble a .so file I use the following commnd: (this requires you to have GNU binutils for ARM installed)

arm-none-eabi-objdump -d --no-addresses --no-show-raw-insn -Mforce-thumb libsurfaceflinger.so | sed 's/0x\S*/HEX/g' > libsurfaceflinger.S

The flags --no-addresses --no-show-raw-insn and the regex are there to make it not print the hex addresses for everything, because the binary layout changes for different code, meaning different addresses everywhere, even where the code logic is the same. This is not fully effective but makes the assembly files somewhat comparable.

Now you can use your favorite diff tool (I use Meld) to compare the assembly files. There will be a lot of differences that don't mean anything like the compiler choosing a different register to store something or places where the disassembly is broken and tries to interpret addesses as instructions(?).

Around line 81200 you can see the density of differences becoming much higher (blue scrollbar in Meld). Around that area I looked at the individual changes and found one that stood out:

at line 81428 (my version) the z3 version hat additional code not corresponding to anything in mine:

ldr.w	r0, [r9, #324]	; HEX
cbnz	r0, <__emutls_t._ZN7android3pdx3rpc17ThreadLocalBufferIhNS1_30DefaultInitializationAllocatorIhNSt3__19allocatorIhEEEELj4096ENS1_20ThreadLocalIndexSlotILj1EEEE7buffer_E@@Base+HEX
ldr.w	r0, [r9, #332]	; HEX
cmp	r0, #9
add.w	r1, r0, #1
str.w	r1, [r9, #332]	; HEX
blt.n	<__emutls_t._ZN7android3pdx3rpc17ThreadLocalBufferIhNS1_30DefaultInitializationAllocatorIhNSt3__19allocatorIhEEEELj4096ENS1_20ThreadLocalIndexSlotILj1EEEE7buffer_E@@Base+HEX
movs	r4, #0
str.w	r4, [r9, #332]	; HEX
b.n	<__emutls_t._ZN7android3pdx3rpc17ThreadLocalBufferIhNS1_30DefaultInitializationAllocatorIhNSt3__19allocatorIhEEEELj4096ENS1_20ThreadLocalIndexSlotILj1EEEE7buffer_E@@Base+HEXa

It introduces new comparisons and jumps (b.n, blt.n) and uses two variables in memory that the code around does not use (offsets 324 and 332) from pointer in register r9. This means this is probably the code doing the magic but we need to translate it to C++ to integrate with new builds.

To get some context where exactly the code is inserted and what it does I analyzed the files with radare2. First re-enable address printing in objdump to find the corresponding addresses. For example line 81428 of disassembly is address 0x7d898 in my file.

To make analyzing easier, for my version I don't use the file extracted from the device. Instead in the directory where you compile the ROM from source, in out/target/product/hammerhead/symbols/system/lib there will be a version that includes debugging symbols that are removed before packaging the ROM.

[zagto@zagtostation SurfaceFlinger-diff]$ radare2 symbols.so
-- In r2land usability is treated as a bug
[0x00048000]> aa
[x] Analyze all flags starting with sym. and entry0 (aa)
[0x00048000]> s 0x7d898
[0x0007d898]> pd
│ ╎╎╎ ; CODE XREF from android::impl::DispSync::addPresentFence(std::__1::shared_ptr<android::FenceTime> const&) @ 0x7d8cc
│ ┌────> 0x0007d898 97f83c00 ldrb.w r0, [r7, 0x3c] ; DispSync.cpp:541
│ ╎╎╎╎ 0x0007d89c 0028 cmp r0, 0
│ ╎╎╎└─< 0x0007d89e 3ff424af beq.w 0x7d6ea
│ ╎╎╎ 0x0007d8a2 d7e90c01 ldrd r0, r1, [r7, 0x30]

radare2 will print the code with nice comments what is from what line in C++ and what is called from where. Use s and pd (seek and print disassembly) commands to look around in the file.

So I found out the following:

we are in DispSync::addPresentFence method in line 541

r7 (r9 in z3 version) is the this pointer of C++, r4 holds the return value (0=false)

DispSync class is defined in DisplaySync.h file which can be used to calculate to what variable the offsets are corresponding to. If my calculations are right 324 is mNumResyncSamples. 332 is mPresentFences which makes no sense because that is of type shared_ptr, but the code is incrementing it by one, which you would not do with a shared_ptr. This means probably a new variable was introduced here.