[Samsung Galaxy Nexus] metalspring:
[MENTION=4608636]bsmitty83[/MENTION], I have some new cflags for you to try out :P

try using these instead of what you have now

For graphite optimizations-

-fgraphite-identity -floop-parallelize-all -ftree-loop-linear -floop-interchange -floop-strip-mine -floop-block -ftree-loop-distribution -ftree-parallelize-loops=2

For link time optimization-

-flto -fno-toplevel-reorder -flto-compression-level=5 -fuse-linker-plugin

Also you may want to add this in-

ifdef CONFIG_CC_GRAPHITE_OPTIMIZATION
CFLAGS_KERNEL	+= -fgraphite-identity -floop-parallelize-all -ftree-loop-linear -floop-interchange -floop-strip-mine -floop-block -ftree-loop-distribution -ftree-parallelize-loops=2
endif
ifdef CONFIG_CC_LINK_TIME_OPTIMIZATION
CFLAGS_KERNEL	+= -flto -fno-toplevel-reorder -flto-compression-level=5 -fuse-linker-plugin
endif

Right now you only have graphite and LTO cflags set for KBUILD_CFLAGS, but not the CFLAGS_KERNEL

Not sure if it makes a difference but its similar to what liquidsmooth did here

Also if you want to know more about those different cflags I chose-

Graphite cflags-

-ftree-loop-distribution

Perform loop distribution. This flag can improve cache performance on big loop bodies and allow further loop optimizations, like parallelization or vectorization, to take place

ftree-parallelize-loops=n

Parallelize loops, i.e., split their iteration space to run in n threads. This is only possible for loops whose iterations are independent and can be arbitrarily reordered. The optimization is only profitable on multiprocessor machines, for loops that are CPU-intensive, rather than constrained e.g. by memory bandwidth. This option implies -pthread, and thus is only supported on targets that have support for -pthread.

ftree-loop-linear (may not be needed but probably cant hurt)

Perform loop interchange transformations on tree. Same as -floop-interchange. To use this code transformation, GCC has to be configured with --with-ppl and --with-cloog to enable the Graphite loop transformation infrastructure

LTO cflags

-flto-compression-level=n

This option specifies the level of compression used for intermediate language written to LTO object files, and is only meaningful in conjunction with LTO mode (-flto). Valid values are 0 (no compression) to 9 (maximum compression). Values outside this range are clamped to either 0 or 9. If the option is not given, a default balanced compression setting is used.

(compression level 5 was taken from this- http://vaniraosp.goo.im/#/c/11189/4/core/combo/TARGET_linux-arm.mk)

-fuse-linker-plugin

Enables the use of a linker plugin during link-time optimization. This option relies on plugin support in the linker, which is available in gold or in GNU ld 2.21 or newer.

This option enables the extraction of object files with GIMPLE bytecode out of library archives. This improves the quality of optimization by exposing more code to the link-time optimizer. This information specifies what symbols can be accessed externally (by non-LTO object or during dynamic linking). Resulting code quality improvements on binaries (and shared libraries that use hidden visibility) are similar to -fwhole-program. See -flto for a description of the effect of this flag and how to use it.

This option is enabled by default when LTO support in GCC is enabled and GCC was configured for use with a linker supporting plugins (GNU ld 2.21 or newer or gold).