yashsingh1/xdaforums huggingface dataset:
[Xperia Z3 Original Android Development] :
I have it on my build box already. I'll push to git when i get home. I'll have a cm14 and a n 7.0 aosp branch

---------- Post added at 08:27 PM ---------- Previous post was at 08:19 PM ----------

# build new leo kernel (zImage-dtb)

make clean

export ARCH=arm

export SUBARCH=arm

export CROSS_COMPILE=../../../prebuilts/gcc/linux-x86/arm/arm-eabi-4.8/bin/arm-eabi-

make shinano_leo_defconfig

make -j16

Make sure you point your cross compiler to the right directory.
[Xperia Z3 Original Android Development] nailyk:
This article describe a better than me why it will be hard to get a fully functional device on N.

full cm-13.0 with the concept kernel from [MENTION=6163210]TheAvengingTITAN[/MENTION]:

[COLOR="Green"]#### make completed successfully (39:55 (mm:ss)) ####[/COLOR]

but the system crash caused by

FileUtils: Could not read from file /sys/devices/virtual/timed_output/vibrator/vtg_level

I will try to fix it tomorrow.

Thanks TheAvengingTITAN!

Edit:

[MENTION=3771760]aclegg2011[/MENTION] All my repo are up-to-date. But this kernel seems really different from the other i test. Among other things actual cm implementation of thermd is not working anymore (also a path changed in /sys) so the device overheat (i reach 72°C with the bootloop) and my test device is open in back and front :s The drivers in this kernel sounds promising and implement a lot of new think (fm, bluetooth, wifi, a lot of v4l-stuffs).
[Xperia Z3 Original Android Development] DjowBr:
Dont know if will be posible but, is there any chance to get these builds working on z3c too?
[Xperia Z3 Original Android Development] :
being this is more of an aosp kernel, Im sure you're going to have to add back the cm stuff into it. It may take sometime, but you should find everything you need by referencing other 8974 kernels too.
[Xperia Z3 Original Android Development] :
Ive got the z3 mm kernel patched up and ready for N testing...

https://github.com/aosp-8x74/android_kernel_sony_z3

next is device tree and vendor files.
[Xperia Z3 Original Android Development] Swidilator:
Waking up to see actual technical development happeing is such a sight for sore eyes:crying:
[Xperia Z3 Original Android Development] :
The only problem is it's using the glitchy screen 3.10 unified sony kernel and the camera is messed up. Im going to try and see if I can get that concept mm kernel to boot instead.
[Xperia Z3 Original Android Development] Swidilator:
Forgive my ignorance, but when you say glitchy, what does it do? I've heard it's glitchy, but I've never seen what it does.
[Xperia Z3 Original Android Development] :
the screen randonmly flickers every couple of minutes.
[Xperia Z3 Original Android Development] TheAvengingTITAN:
Great work man!

Quick question, when you say that the camera is messed up, is it the same fish eye effect or something else?

Once again, thank you so much for your work on the Z3 and managing to get Nougat to boot on our device
[Xperia Z3 Original Android Development] :
The actual viewfinder screen was messed up. I tried using the mm concept kernel, but it wouldn't boot. It could just need some patches though. Worst case, we run the unified 3.10 kernel and try to patch up the adreno/msm drivers to work with the z3 correctly.

Update: [MENTION=6979267]nailyk[/MENTION] is on vactaion, but before he left he was taking a look at some kernels. He thinks that something broke in the drivers/gpu/msm adreno code, causing our issues. I took a look at the 3.4 cm android_kernel_sony_msm8974 kernel and it looks there's just a couple of files changed.

Update1: upon further investigation and what clues [MENTION=6979267]nailyk[/MENTION] had from dmesg logs we found that inside of the adreno.c where the HAL is made on 3.4 it has the function .wait_timeout = 0, where as it's not present in the 3.10 kernel.

Update2: I checked the lg v10 kernel source (they run 3.10.x) and it also has the .wait_timeout = 0 in drivers/gpu/msm adreno.c and adreno.h
[Xperia Z3 Original Android Development] :
I have a new android N build ready, but the camera dont work. The screen is semi fixed. It's a lot better now.

Im force setting the kernel to permissive mode for now to see if it fixed the camera. If it does, I'll have to fix selinux stuff.
[Xperia Z3 Original Android Development] some_shyguy:
That sounds extremely tempting. I always use my camera but that itch to flash is starting to act up (for the first time ever with my Z3).
[Xperia Z3 Original Android Development] yecomixer:
hello [MENTION=3771760]aclegg2011[/MENTION] when you can share the rom to test, greetings is a great job :good:
[Xperia Z3 Original Android Development] :
the camera doesn't work. the screen is half green and half black. :( And the screen glitch is still hanging around every once in a while.

Before I release, I want to try one more thing.
[Xperia Z3 Original Android Development] :
Expect a release later today!
[Xperia Z3 Original Android Development] Swidilator:
A bit of a high hope, but will gapps work on the future build, or is it purelN at the moment?
[Xperia Z3 Original Android Development] :
I'm not sure why, but Gapps kept causing fcs.

---------- Post added at 08:06 PM ---------- Previous post was at 08:00 PM ----------

I might be able to figure out a fix for it. Working on Roms isn't my speciality, working on devices is.
[Xperia Z3 Original Android Development] TheAvengingTITAN:
I tested AOSP 7.0 on the Xperia SP and it had the same issue, basically what we did was flash the gapps, delete setup wizard folder in privapp via TWRP file manager in system partition and after booting up we enabled all the permissions for Google play store and Google play services and everything stopped FCing :)

Hope this helps!
[Xperia Z3 Original Android Development] :
Hrm, so the google gapps setup wizard is giving the issues?
[Xperia Z3 Original Android Development] TheAvengingTITAN:
Well setup wizard was crashing constantly for us while we were trying to set up, saying that Google play services kept crashing.

Think the problem is with the permissions for the play store and play services
[Xperia Z3 Original Android Development] :
Yea, the new Gapps fixes the setup wizard issues. Screen glitch is still present, something in the kernel causes that. Camera don't work either.
[Xperia Z3 Original Android Development] Swidilator:
What makes the Sony kernel so different from an AOSP kernel? Why is it made to be so different in any case?
[Xperia Z3 Original Android Development] TheAvengingTITAN:
By the way quick request [MENTION=3771760]aclegg2011[/MENTION]

Before releasing the first test build, I think it would be nice to have a thread exclusively for Android N because after all, this is a marshmallow cm thread.

Would help users see that there is work being done on nougat for the Z3.
[Xperia Z3 Original Android Development] nailyk:
[MENTION=6163210]TheAvengingTITAN[/MENTION] No problem discussing nougat here. The idea is proof-of-concept for the future of our z3.

Btw: on #cyanogen-mod dev, a few days ago some persons says cm-14.0 is buildable for msm8974 :)

Why we don't have custom rom based on M or N for now?

As [MENTION=5075064]optimumpro[/MENTION] said earlier, blobs from sony since lollipop are not usable. It break some functionality as the camera. Lollipop based custom rom works because they use kitkat blobs. Another point is screen glitches, i will detail in the next part.

For now, i'm on vacation. Progress will be really slow but i will try to read a lot of doc.

Before i leave i made a lot of experiments. I test almost every kernel for msm8974 i found (only for display glitches atm).

Obvisouly, the 3.10 unified kernel. Cm one and xperiadev one. (Some promising info here): screen glitches, camera should be working but in which state? Data, bluetooth, wifi, etc working as this kernel is maintained.

Old 3.4 cm kernel: you know the result, the build in the op is based on this one: no more screen glitches, don't know about the camera, everything else is working, absolutely weak as no more updated.

Aosp msm unified 3.10 kernel: screen glitches, i throw it before the rom finish booting.

Hammerhead kernel cm 3.4 : can't remember... I should give another try.

htc msm8974 cm kernel: i don't build it. As usual, before building, i diff the driver/gpu/msm part. A quick reading of registers in htc and sony reveal they are not the same.

cm 3.10 unified sony kernel, hal qcom variant: same glitches.

From where those glitches are from? Why the other msm8974 based devices doesn't have this problem? (hammerhead, htc, lg, etc...)

So i'm wondering: can the _same_ soc be different? Why the adreno 330 chipset into the msm8974 htc and the adreno 330 in the msm8974 sony doesn't have the same addresses offset? Can other part of the soc be different? If yes, can the glitch bug came from the blob? (btw i test with hammerhead blob with cm3.10 unified sony kernel and same glitches).

Another important point is: the 3.4 kernel for this device is no more updated (last commit april 2015) and the work is now done on the 3.10 kernel.

I will not ask the same question but hope we get constructive answers.

Someone ask for z3c (iirc): yes basically all this work can be ported to all the others sony msm8974 based devices. Just i will not do it for now as 1. I don't own the device, 2. it is a proof-of-concept, until a decision is taken about the kernel, i will not have time to build for another device.

Thanks for reading!
[Xperia Z3 Original Android Development] :
Another thing to remember is going from a 3.4 - to a 3.10 kernel porting is a big step. Anything from that port could be broken. Even on the g3 most kernel ports stayed on the 3.4 kernel and patched up to 3.4.1xx. the cm g3 devs took the last CAF release of 3.4 kernel and merged all the lg drivers into it. So either we find the fix for the 3.10 kernel, or try and update the 3.4 kernel. If anyone can remember when the 3.10 screen glitch started, we could probably narrow down the commit that broke it. Which mm Roms don't have a glitchy screen? If we can figure that out, we can narrow down the bug.

After looking at some of the Roms this is what I found:

All cm11 and cm12 based Roms used the 3.4x kernel.

All mm Roms used the unified 3.10 kernel and had issues with screen flickering. I've seen some posts talking about the flickering due to the adreno driver.

I've built 6.0 aosp and 7.0 aosp using the Sonydev/kernel and they both have screen flicker.

So honestly, I don't think the 3.10 kernel has ever worked right. Which is weird, because it's supposed to be the most updated kernel for our device.
[Xperia Z3 Original Android Development] Swidilator:
I'm not sure if anyone had posted an issue about the flickering on the 3.10 kernel github, so I went ahead and posted one, let's see what happens...
[Xperia Z3 Original Android Development] :
They'll probably want a catlog
[Xperia Z3 Original Android Development] Swidilator:
That'll have to wait for the weekend, I don't have the time to switch ROMs at the moment :P

---------- Post added at 04:08 PM ---------- Previous post was at 03:26 PM ----------

@aclegg2011 they want to know if you built it according to their guide? I presume you did?

Also, I posted on the Unified XDA thread, and someone suggested a catlog/dmesg like you did. Would it be possible to provide one from your device in the meanwhile, or is that not possible at this point? Thanks for your help in any case, looking forward to when this is fixed!
[Xperia Z3 Original Android Development] optimumpro:
Guys. Could you please take your discussions about Nougat and 3.10 kernel to your own threads. This thread is about building MM and possibly N with 3.4 kernel. This is spamming...

If there is a chance to have stable N, it will not be with Sony unified kernel, which really unifies bugs and failures. As a matter of fact, nothing that has ever come out of Sony AOSP is stable: not on JB, not on Kitkat, not on L, M or N. So, they are pretty consistent. And also they have made sure that no stock blobs from any release after JB could be used with custom roms.

Again, if we can return to CM trees with JB stock camera blobs, camera wrapper and 3.4 kernel, you will have bugless roms...
[Xperia Z3 Original Android Development] nailyk:
I don't read enough doc for now. :confused: What do you suggest?
[Xperia Z3 Original Android Development] optimumpro:
I thought you have built CM13 based on CM tree? Haven't you? You should try to build it with 5.1.1 camera files, which are located in 4 or 5 spots: /hardware/; /frameworks/av; /system/core and somewhere else., /frameworks/base/ I think. Replace those with LP directories and files. Then you should use 5.1.1 camera blobs (camera only) and it should work with camera + camerawrapper+ JB stock camera blobs (which are in /github/themuppets CM12.1)..
[Xperia Z3 Original Android Development] :
did camera work on cm12.1?
[Xperia Z3 Original Android Development] nailyk:
Except fisheye bug and 120fps, yes pretty well.

I want to give a try with the blobs provided by sony (not extracted from a running stock rom). Maybe i can test on 3.10 kernel to go faster. (kernel-headers problems with the 3.4 kernel)

If, as i suspect, it doesn't give anything, i will do what optimumpro suggest: downgrade the whole camera stuff, but I can't do that while on vacation, not on my computer, not enough time, etc... The first time i tried it (a few month ago) i ran into a lot of build error and give away, so i know it will be hard (for me).
[Xperia Z3 Original Android Development] optimumpro:
It did. As [MENTION=6314247]Nayli[/MENTION]k said, LP on Z3 was using Z1 stock camera blobs from JB plus camera wrapper.

Doing what I mentioned in the previous post wouldn't be easy. There will be errors to resolve and on Nougat, you would probably need to disable ninja build... Maybe not...
[Xperia Z3 Original Android Development] :
I disable ninja anyways. if camera works on cm13 though, it should work on cm14 just fine with the same commits. Mostly selinux changes are needed to the device tree and kernel. The fisheye thing. Is it when you look through the viewfinder or is it seen in the saved picture too?
[Xperia Z3 Original Android Development] blackhawk_LA:
It appears in the saved picture. What a shame Sony makes things so hard for the developers!
[Xperia Z3 Original Android Development] optimumpro:
Camera has never worked properly on CM13, because everyone has switched to Sony AOSP trees and their crappy Open Camera. So, Sony's horrible 3.10 kernel + no less horrible Open Camera give you (them) the double fail. So, the point is to return to 3.4 kernel + CM device tree + camera wrapper with JB stock camera blobs...
[Xperia Z3 Original Android Development] nailyk:
2016-09-08 release

*New release*

Nothing really new. I start to regress the camera (q&d) but blobs are not changed. Feel free to experiment and report.

Old blobs are available here and other subfolders (shinano/msm8974/etc...)
[Xperia Z3 Original Android Development] ILikeBubbles:
It sounds like with the update from 09/08 this could almost be a daily driver... I'm running 12.1 right now but with nougat out I've been itching for marshmallow :)
[Xperia Z3 Original Android Development] Swidilator:
Somebody on the Z3C forums posted a good theory about the fisheye bug:
[Xperia Z3 Original Android Development] nailyk:
I really don't know if it can be used as a daily driver. In my case, this rom, when installed, is installed on a broken device [1] [2].

The only thing i am sure, is, now the rom is running for ~2 days and the device doesn't seems to present problems.

Your post about fisheye bug is interesting. Have you the link of the original post? My theory is little different, but i don"t use stock rom from a while. I totally agree with the fact the fisheye bug is, at the beginning, an hardware issue. On z3 you have a wide-angle sensor. Try to put a random compact camera side-by-side to the z3 and you will see the z3 captor had a wider field of vision. They use a special lens on a classic sensor (i still look at the name and datasheet btw ;) )

Here come the pain:

On stock rom, if i remember correctly the reverse fisheye bug depend of the used resolution. In auto mode, you are on 8Mpx (3264*2448) which use only a part of the cmos sensor, and don't know why, a small part of this view area.

The idea is, your sensors capture a rectangle (21Mpx) and your display/record is a smaller rectangle of the first one, with the same center.

Now imagine your reverse fisheye bug is, too much correction but, only for the view area between 8Mpx rectangle and the edge: you will only get the 'reverse-fisheye-bug' for the full size sensor capture. (on stock rom, setup-up manual mode, go to advanced, and switch to 21Mpx and the 'too much correction' is present on the picture).

For the software part: I'm still unsure if it is a dedicated DSP chipset or if it is the cpu which correct the lens but i am sure it is a matlab implementation.

Am-i the only one who read the camera blobs sources? (hint: i guess the fisheye correction implementation is in the module/isp/rolloff folder).

Anyway, i'm not worried about fisheye bug, i have some idea to correct that _but_ we need a working camera before everything! And getting the camera working will not be an easy job...

What about opening a dedicated dev-thread about camera and custom roms? If it is the last non working peripheral on our devices family maybe we can find a solution together?
[Xperia Z3 Original Android Development] nonperson:
Thanks very much for your work on this. Tried 8 Sep build on a D6653.

--> No screen glitches at all, in a couple hours of use anyway. That problem looks solved.

--> However, wi-fi hotspot no longer working whereas it did in earlier builds. Turns on, appears to be working normally, but doesn't broadcast anything.

--> Like all earlier builds -- of this and just about any CM 13 or AOSP rom for Z3 -- NFC not working. Won't turn on, or constantly turns on and off.
[Xperia Z3 Original Android Development] okij:
Here you go: http://xdaforums.com/showpost.php?p=68624944&postcount=144 (I posted this yesterday on the Z3c Open Devices Project forum.)

Thanks, now there's already two of us. :)

I'm not a dev myself, just a user with some knowledge of physics / optics, so unfortunately I can't help with coding or reverse-engineering to fix this issue.

But hopefully some of the other devs (e.g. [MENTION=3989065]serajr[/MENTION]?) can have a look at this too.

No, that's not the case. It doesn't appear at all, only when using 4K recording.

No, that's not the case either. Also the 8Mpx mode uses the whole sensor area, the picture is then just scaled down from 21Mpx to 8Mpx. Otherwise you would have a tele-lens effect (it would appear as a 40 mm lens instead of a 25 mm lens), which is not the case.

Your theory about a dedicated DSP chipset is interesting, but I don't think there is one. On the one hand we would have read about it in the specs or at least on one of the hardware teardown sites which have photos of the motherboard, on the other hand I think that the Snapdragon 801 is perfectly able to handle this calculation in all camera modes, only in 4K video mode it reaches its limitations.

I hope that there are other devs too so that you can establish a task force. :fingers-crossed:

IMHO that would be a great idea! It's probably the only way to finally get fully working AOSP and CM roms on the Z3 and Z3c. :)
[Xperia Z3 Original Android Development] xxlsm:
There are some open source algorithms available which support re-distortion. One algorithm would be this one by graphcismagick. You just need to experiment a bit with the parameters of the tool. This was discussed in this thread either http://xdaforums.com/z3/help/cm12-camera-distortion-t3060606/page2. Unfortunately, for re-distorting the pictures you have to type in a specific command for every new picture. Maybe this algorithm could be implented in the camera driver so that on the fly re-transcoding would be possible. But at first the camera has to work, after all.

http://www.graphicsmagick.org/
[Xperia Z3 Original Android Development] ILikeBubbles:
You've recommended upgrading my bootloader before I flash.. do I run a risk of bricking if I don't? I read your notes on it.. and read the post on Emma but still don't 100% understand what it's for.
[Xperia Z3 Original Android Development] nailyk:
For the wifi i still not take time to check the right path of the blobs. As a temporary workaround, i cp the blobs in the both location:

- /system/etc/firmware/wlan/bcmdhd/

- /system/vendor/firmware/

and on the 8 sept build i forgot the '_apsta' blob which is used for the the hotspot. I it is a required functionality, you can find the blobs here

adb shell, su, remount -o remount,rw /system, then cp the blobs in the both locations.

Sorry for that.

For the nfc, i don't work on it for now. The idea, at the moment, is to check the viability of newer roms (nougat, cm-13, cm-14) with the old 3.4 kernel.

If the others peripherals of the devices are working, it is great news for this old kernel!

Btw is the D6533 the dualsim variant? Is the ril working? If not working you can also try to switch those blobs.

Thanks for feedback.
[Xperia Z3 Original Android Development] nailyk:
The bootloader, on normal android phone, load kernels.

On non poorly developed devices (aka sony), bootloader can boot two kernels: 1. system kernel (boot partition) 2. recovery kernel (recovery partition or FOTAKernel in our case).

Our xperia bootloaders on first version only handle one partition. On this thread, we are experimenting, especially on kernel. If, for some reason, i post a rom with a broken kernel and you flash it, you won't be able to boot anymore. Nevertheless fastboot mode and flash mode should remain available. You can find detailed information here (not updated since a while).

(with stable rom like cm-12.1, it sounds less risky, but in every case i guess you could reflash the full device with emma, and the safiest solution still remain to upgrade bootloader: the new one had this 'dual kernel' support)

Good luck :)