yashsingh1/xdaforums huggingface dataset:
[Windows Mobile Software Development] rsenden:
Possible improvement?

I've been having an idea that could possible improve performance and battery usage of IMAP IDLE solutions like these.

Basically, one would implement some kind of IMAP proxy. This proxy would consist of a socket listener (localhost only) and an IMAP IDLE client. Pocket Outlook is then configured to point to this proxy instead of the actual IMAP server.

The IMAP IDLE client would basically remain the same. However, whenever Pocket Outlook connects to the proxy and issues a LOGON command, IMAP IDLE service is suspended. All remaining Pocket Outlook commands are simply passed through over the same connection that was previously used for the IDLE command. Once a LOGOFF command comes in and Outlook closes the connection to the proxy, the IMAP IDLE client resumes IDLE operation (again, over the same connection). Of course, LOGON, LOGOFF and similar commands issues by Pocket Outlook will not be passed on to the actual IMAP server; the connection will be managed by the IMAP IDLE client.

This way, you prevent Pocket Outlook from setting up its own connection to, and authorizing with, the remote IMAP server for each incoming email. Instead it will re-use an existing authorized connection.

I'm not sure whether it makes that much of a difference, but it might be worth the effort.

Kind regards,

Ruud.
[Windows Mobile Software Development] Fixup:
I've done embedded programming for 15 years, so you can trust my code.

Many programs, such as PortSip and EmailScheduler, are not event-driven, they keep polling which sucks battery, runs the phone hot and not reliable. XinConnManager is a true event-driven service, has no such problems.

When I have time, I'll re-write the entire ImapPusher from VB to native C++. Anything simply sucks when touchs .Net and VB.

Actually my current push solution is direct http push, very reliable and no any impact to battery life. Here is how to do it:

1) Run Depeche, a http server for PPC. Modify the SimplTex ISAPI to tell Outlook to sync mail when it gets a http request.

2) Run Pocket DynDNS to update "mypocketIP.homeip.net". This program works, but it uses Sleep(60 seconds) to update DynDNS. You know, Sleep() does not work as soon as the phone goes into suspend mode. So I made "XinServerManager", a service similar to "XinConnManager", to keep the connection and meanwhile call PDD to update dyndns immediately after a new (re)connection. Then kill the PDD.

3) Run PPC emulator on a XP machine and run a slightly modified xImapPusher on top of it. When email arrives, it sends a http request to my phone and the phone then will do a sync to get the new mail.

This is by far my push solution. It does not need any open connection between the phone and the server. True and pure push. This solves the porblem of 5-min timeout with many 3G providers such as Verizon. Such 5-min timeout kills battery. Before, 4 hours of push running would drain 10% battery, now with my direct http push, 10% will run the phone for 24 hours, like it is no push running at all!
[Windows Mobile Software Development] bsinclair:
Battery will always be worst using IDLE, this is one thing Microsoft did right. If you wanna know a little bit more, see here:

http://gudensoft.com/?p=140

Instead of using this, I moved to active sync as now Google supports it.
[Windows Mobile Software Development] Fixup:
That article is totally wrong, see my comments at the bottom. Both require an open connection, just inactive, no difference here between these two.

IMAP idle is actually better on this regard: it'll detect the 5-min timeout and make a reconnection, no push lost.

With ActiveSync, it won't detect the timeout and any mail arrived after 5 min until the next beatheart will not be pushed in time.

That's why AS makes you feel a little better on battery life. The default heartbeat is 8 min. So you do get slightly better battery life, but you pay for that with push delay. You may set "MinHeartBeat" to 300 (5 min) so you won't suffer from delay, but then the battery life is no any better than IMAP idle.

The only reason MS does not add IMAP support to WM is because they want to sell Exchange servers and the ActiveSync Direct Push service. Both Apple and Google went to this stupid route.

To address this 5-min timeout issue, I set xImapPusher to re-send the heartbeat without raising a mail sync.

BTW, MS Live push is based on IMAP idle, not ActiveSync. You know, years ago MS bought Hotmail which was Linux based, seems still Linux. Live is far more reliable than AS. Like rsenden, I too was using Live because it does not have all kinds of troubles with Gmail's AS push. However, Verizon applied the stupid 5-min timeout sometime this year or so, so I had to figure out my direct http push.
[Windows Mobile Software Development] SOHKis:
this sounds like a nearly perfect solution, now if it could only be done w/2-3 accounts.

i'll have to give this method a try someday when i get some freetime on my hands. thanks man!
[Windows Mobile Software Development] Fixup:
The phone keeps connections and sockets in suspend mode, it is the service provider (mine is Verizon) that closes inactive sockets (not the connection) after a timeout (5 min in case of Verizon). You have no control about that.

However, the connection does get disconnected sometime, a random thing, that's why your program needs to wait for the "NOTIFICATION_EVENT_NET_DISCONNECT" event. Note that when you unplug your phone from our computer, it also generates this event, so your program needs to deal with that too.

How long you can use Sleep() is defined by the power settings. For example, if you set your phone to suspend in 60 seconds, you cannot use Sleep(61000). Anyway, you should not use Sleep() at all, because it'll either prevent your phone from suspending or hang your program till the phone wake up.

This is how I do wait when needed:

SYSTEMTIME GetTimeAfterSeconds(DWORD nTimeInSeconds)
{
SYSTEMTIME sTime;
FILETIME fTime;
GetLocalTime (&sTime);
SystemTimeToFileTime(&sTime, &fTime);
ULARGE_INTEGER uTime;
uTime.HighPart = fTime.dwHighDateTime;
uTime.LowPart = fTime.dwLowDateTime;
uTime.LowPart += nTimeInSeconds*1000*1000*10;
fTime.dwHighDateTime = uTime.HighPart;
fTime.dwLowDateTime = uTime.LowPart;
FileTimeToSystemTime(&fTime, &sTime);

return sTime;
}

void Wait(HANDLE hEventTimer, DWORD dwTime)
{
CeRunAppAtTime (_T("\\\\.\\Notifications\\NamedEvents\\EventTimer"),&GetTimeAfterSeconds(dwTime));
WaitForSingleObject(hEventTimer, INFINITE);
}

main()
{
HANDLE hEventTimer = CreateEvent (NULL, false, false, _T("EventTimer"));
......
Wait(hEventTimer,60);
......
CloseHandle(hEventTimer);
}

Note that WaitForSingleObject(hObject, 60000) won't work, like Sleep(60000).
[Windows Mobile Software Development] Fixup:
Multiple accounts sync should be easy, just tell which account to sync in the http request.
[Windows Mobile Software Development] rsenden:
It's not that I don't trust your code; as a programmer (non-embedded Java and some C#) I'm just interested in what components like these exactly do. Anyway, I had some time left yesterday so I installed xImapPusher. So far it's working good, too early to compare battery usage with EmailScheduler yet though.

This indeed looks like a very good solution. Unfortunately it currently requires a Windows machine running 24/7 (which I don't have) and is too difficult to set up for the average user. For me, a good alternative would be to run the 'IMAP IDLE to HTTP request' component on my CH3SNAS.

Of course, the ultimate solution would be to have a hosted solution on which users can register their IMAP accounts, together with a program on their PPC that both listens for push notifications send by the hosted solution, and sends IP updates to the hosted solution.
[Windows Mobile Software Development] rsenden:
Basically, I got the idea by reading somewhere that Pocket Outlook IMAP solutions are always less efficient than, for example, ActiveSync. This is because every time Outlook needs to retrieve mail, it needs to set up a TCP/IP connection, set up SSL if applicable and do a LOGON (and afterwards do a LOGOFF, close the connection).

With an IMAP IDLE solution like xImapPusher you already have a secured and authenticated connection set up, so why not have Pocket Outlook re-use that same connection? However, I don't know how much data and processing power is involved in setting up and closing the connection, and thus how much impact this could have on battery life and time taken to retrieve new mail.

So basically, xImapPusher would not only function as an IMAP client, but also as an IMAP server. The xImapPusher server component handles incoming connections, together with LOGON and LOGOFF requests. All commands between LOGON and LOGOFF are simply passed on to the real IMAP server via the connection that was previously set up by the IMAP IDLE client component (which has temporarily suspended IDLE operation).
[Windows Mobile Software Development] SOHKis:
sorry, the multple accounts was in reference to the re-writing of imappusher to c++

still gonna give the http method a shot when i get some time. thanks again man
[Windows Mobile Software Development] Fixup:
Compared to the excessive battery drain from the 5-min timeouts, nothing else matters.

Don't believe too easily what you see on the net. Highschool bloggers dont know what they talk about. If IMAP idle is not good, why MS Live uses it?

I've done extensive tests between imap idle and ActiveSync. I also tested z-push (AS clone) and many other things in searching for the best push solution.

The real problem with imap idle is there is only few mobile clients support it and none of them work. That's why I spent time on this project.
[Windows Mobile Software Development] rsenden:
I don't believe everything I see ;-). I was just trying to find an explanation why for example EmailScheduler drained the battery so much faster than Windows Live. After trying out xImapSync for the last few days, I now know that it is just caused by the way EmailScheduler is implemented.

With xImapPusher battery drain is about the same as Windows Live I think, and it's also more stable than EmailScheduler (e.g. I don't have to force a reconnect every now and then). So thank you for your contribution to this project!
[Windows Mobile Software Development] ironhead65:
I might be thick...but does install order matter?

I installed the multi-software. Then setup an account. AFTER this, I realized (readme.txt was helpful!) I had to create an account in POutlook that matched the name I put in the xImapPusher program.

I configured my account in POutlook with only manual updates. Sent an e-mail to the gmail account I was using with imap. Nothing. Waited a while. Nothing. So, I went into POutlook and hit manual refresh, I got the test e-mail.

So, I finally set it for update once. Sent another test e-mail. Waited, nothing.

Am I missing something? I am on Sprint with a TP2 (apparently a closing socket is not my issue). Do I need to setup a poll in PocketOutlook?

In the mean time, I will try de-installing this and removing the account. Reset, create the account, then install the software.

Thanks!

--Ironhead
[Windows Mobile Software Development] ironhead65:
I admit, I'm thick! I forgot, when you add a Gmail account to WinMo, you have to put in an INVALID address for the first screen, otherwise you do not get to choose "use SSL" for incoming and outgoing mail. I could Receive, but not Send from the account. That's when I realized my mistake.

Now, it JUST WORKS!

AWESOME!

--Ironhead
[Windows Mobile Software Development] iivisionii:
Has anyone tried this and gotten it to work with yahoo mail? I have a touch pro 2 and can't get it to work with yahoo.
[Windows Mobile Software Development] chalid:
Hi,

I was using IMAPIdle 0.60 for quite some time now, but it stopped working with the HD2 in sleepmode. So I tried your version and it is working perfectly fine! When internet-connection is lost, its reconnecting a few seconds later, and of course the IMAP-idle service is working too.

I have one question though: Is it possible to unload the Xin Conn Manager? At the moment its reconnecting and restarting the service, after closing imappusherservice and the internet connection. I just don't need the service the whole time (e.g. in the night ... :)) and soft resetting to end the service is a litte unpractical.

Other than that its perfect. Thank you!

bye
[Windows Mobile Software Development] Fixup:
XinConnManager is a service, you need a good task manager such as FdcSoft to stop/start/restart a service.

I'll modify the LoadXCM.exe, so the second time you run it, it'll stop the service.
[Windows Mobile Software Development] 1sae:
Great work! Just what I was looking for. You would think that it would be a built in feature in pocket outlook.
[Windows Mobile Software Development] chalid:
I've tried Schaps Advanced Taskmanager, but the Servicetab won't load. FdcSoft Taskmanager will freeze the device occasionally. :-/

A modification of LoadXCM.exe to end the service without the need of third-party-software would of course be the best solution ... :-)
[Windows Mobile Software Development] Fixup:
This is the latest version:

1) LoadXCM now will also stop the service.

2) When XCM service is stopped, ImapPushService is unloaded too.

3) No longer prevent task manager such as FdcSoft from loading.

4) Even more reliable.

Few notes:

1) Before you use LoadXCM to start the service, make sure ImapPushService is closed, because the XCM service will automatically load it.

2) When you use LoadXCM to stop the service, wait for several seconds after you click on the "Yes" button, until you see the confirmation box. Then click "OK", otherwise LoadXCM itself will not be closed.

[update 12/11/2009]Found a terrible null pointer bug, please re-download. A new version kills the annoying waiting cursor after 15 seconds: http://xdaforums.com/showthread.php?p=5127985#post5127985
[Windows Mobile Software Development] chalid:
wow, that was fast. will try when im home. ty
[Windows Mobile Software Development] SOHKis:
you're the man fixup, thanks!
[Windows Mobile Software Development] Fixup:
Found a terrible null pointer, please download again, sorry.
[Windows Mobile Software Development] chalid:
So, just got home a few minutes ago ... First: thank you for the fast update and fix! :)

I had your first version running the whole day and I have to say it is by far the most stable IMAPIdle-solution out there!

The new version is perfect for my purpose:

- very light

- very stable

- recovers reliably from disconnects

very nice job.

two minor flaws, which aren't a big deal:

- Starting IMAPPushService.exe causes the WM waiting cursor to stick in the front, although nothing is loading. Going into the start menu (or any other newly drawn screen) solves that.

- Closing the still active internet connection after ending the service, will result service.exe to not close properly. WM wants to send an error report. By switching error-reportings off alltogether in the system settings, solves that.
[Windows Mobile Software Development] Fixup:
Thank you for your feedback. The busy cursor is from the original ImapPusher-0.60, as the source code is written in VB, I have a hard time to figure out where to fix that. As you've found out, it is no harm, a refresh of the today screen makes it disappear. So I might add such a refresh after loading the service.

I have not noticed the second issue, because I always turns off the error send. Did you wait for the confirmation message box after click on "Yes"? If so, I think it is because we have to close the service in force - the Run thread listens to disconnect forever. If we give it a timeout, then we do not need to force terminating, but battery life suffers.
[Windows Mobile Software Development] Fixup:
This version kills the annoying waiting cursor after 15 seconds.

Camro, you need to create a named event for NOTIFICATION_EVENT_NET_DISCONNECT, very similar to my Wait().

[update 12/12/2009]Now the waiting cursor never appear.
[Windows Mobile Software Development] chalid:
You really should pack this into a CAB and post it in the application-forums. This IMAP Idle solution is now absolutly perfect. Thank you very much!
[Windows Mobile Software Development] bakasani:
@FIXUP

Hi

I keep gettng this repeating error in the log, any ideas?

Error setting up socket: could not connect the socket. 0

socket errored out.

System.Exception: could not connect the socket. 0

at ImapPusher.ImapClient.SetUpSocket()

at ImapPusher.ImapClient.Connect(String pUserName, String pPassword)

at ImapPusherService.Module1.Main()

error disconnecting. 10093

thanks
[Windows Mobile Software Development] Fixup:
Because the stupid carriers close inactive sockets after several minutes, e.g. Verizon closes sockets after only 5 minutes.

You know, push, both ActiveSync (Exchange) and Imap idle, rely on inactive sockets to eliminate impact on battery. Such a 5-min timeout reduces battery life by 5x!!!

This is one thing I did to the original ImapPusher. Before, it simply quited if it saw this error 3 times. Now it ignores such errors and keeps going with a reconnection. I also make it does not do a mail sync after such an error to minimize impact on battery life.

As Chalid said, finally we have an imap idle push solution that works.

Honestly, I'm not a WM programmer and never made a cab. This is the first WM program I ever made and it is a service, amazing!

[update]This is the latest with a new readme file and a cab for easy install/uninstall.
[Windows Mobile Software Development] chalid:
Got a small problem with the latest version. After some time no mails are pushed anymore. That didn't happen with the other versions.

processing ok
Sending: IMAP0003 IDLE

read: + idling

read: * 2038 EXISTS

exist arrived
new mailbox count: 2038
getting new messages
Normal refresh has expired.
resetting idle
refreshing idle
Sending: DONE

read: IMAP0003 OK IDLE terminated (Success)

processing ok
Sending: IMAP0005 IDLE

read: + idling

read: * 2039 EXISTS

exist arrived
new mailbox count: 2039
getting new messages
read: * 2040 EXISTS

exist arrived
new mailbox count: 2040
getting new messages
read: * 2041 EXISTS

exist arrived
new mailbox count: 2041
getting new messages
read: * 2042 EXISTS

exist arrived
new mailbox count: 2042
getting new messages
read: * 2042 EXISTS

exist arrived
new mailbox count: 2042
Normal refresh has expired.
resetting idle
refreshing idle
Sending: DONE

read: IMAP0005 OK IDLE terminated (Success)

processing ok
Sending: IMAP0007 IDLE

read: + idling

read: * 2042 EXISTS

exist arrived
new mailbox count: 2042
Normal refresh has expired.
resetting idle
refreshing idle
Sending: DONE

read: IMAP0007 OK IDLE terminated (Success)

processing ok
Sending: IMAP0009 IDLE

read: + idling

read: * 2042 EXISTS

exist arrived
new mailbox count: 2042
Normal refresh has expired.
resetting idle
refreshing idle
Sending: DONE

read: IMAP0009 OK IDLE terminated (Success)

processing ok
Sending: IMAP0011 IDLE

read: + idling

read: * 2042 EXISTS

exist arrived
new mailbox count: 2042
Normal refresh has expired.
resetting idle
refreshing idle
Sending: DONE

read: IMAP0011 OK IDLE terminated (Success)

processing ok
Sending: IMAP0013 IDLE

read: + idling

read: * 2042 EXISTS

exist arrived
new mailbox count: 2042
Normal refresh has expired.
resetting idle
refreshing idle
Sending: DONE

read: IMAP0013 OK IDLE terminated (Success)

processing ok
Sending: IMAP0015 IDLE

read: + idling

Seems that its just not restarting the service although normal refresh has expired. I set it to 4 minutes after I got the error the first time. So 5 refreshs after 4 minutes = 20 minutes after the service fails to work. I did wait 35 minutes to check if it was working - it wasn't. :-/

Maybe its related to the waitingcursor-fix, as the last version was working fine for two days.
[Windows Mobile Software Development] Fixup:
Did you download the latest one that I uploaded last night? I made a small change on hiding the wait cursor. If it is the ImpaPusher (not the ConnManager) that cannot restrat after the refresh period, then the 0.60 needs some more treatments. I have not made any changes to the 0.60 after the first post here. You may also try to use the latest ImapPusher 0.61 I posted on the other thread.

I'll look into the 0.60 code to see why it does not restart after the refresh. XinConnManager does not handle that, it only deals with the data connection.

You may use a long refresh such 25 minute. This way it will never hit the refresh bug.
[Windows Mobile Software Development] chalid:
yes, its your latest version from this thread. will give the longer refresh-time a try and report back tomorrow evening.

the strange thing is, that it was working fine for the last days, since I discovered this thread and now I can reproduce the problem just by waiting long enough without getting an e-mail.
[Windows Mobile Software Development] Fixup:
I just repeated your test: set refresh to 4 min, waited for 35 min, sent an email, and no problem. I did notice that gmail stopped push sometimes, usually during midnights and weekends. When that happened, I did not do anything and after a while push would continue. So, it could be a mail server issue.
[Windows Mobile Software Development] chalid:
I fear its another problem. The first time I noticed the problem was midday today. Service was running since yesterday and no new mail was pushed to the phone today. When I checked manually, there were 2 mails in the inbox from at least 3 hours earlier.

Maybe check your logs. As you can see with mine, it just stops refreshing the Inboxcount. The last line of the codebox posted above is the last line in the log-file, when the pushing stops.

But I will watch closely and report back. Maybe it has something todo with the weekend and/or daytime, since it was working flawlessly the last few days.
[Windows Mobile Software Development] bakasani:
Sorry what I meant was that is all there is in my log, it does not fetch email or work, it just repeats those same exact lines over and over with cpu usage upto around 25 - 30% constant - any ideas what I could check?

thanks
[Windows Mobile Software Development] Fixup:
*Install all files to any directory you like, no longer has to go to \Windows.

*Service now can be stopped immediately and gracefully.

*No more error when close connection after exit.

*New icon.

*Sync after reconnect, more reliable push.

*Debugging log file now has time stamps, so you know better what happened and exactly when.

*Wait cursor still appeared sometimes, now fixed.

So far I have not seen any issue with this version yet.
[Windows Mobile Software Development] bakasani:
Hi

I still receive only this same error as before with high cpu usage on phone while running:

##

Error setting up socket: could not connect the socket. 0

socket errored out.

System.Exception: could not connect the socket. 0

at ImapPusher.ImapClient.SetUpSocket()

at ImapPusher.ImapClient.Connect(String pUserName, String pPassword)

at ImapPusherService.Module1.Main()

error disconnecting. 10093

##

Also looking at mail server (courier) logs I also get this repeating fast every 1 second, I have also tried port 993 and 143 with same result :

##

Dec 14 16:54:35 imapd-ssl: Connection, ip=[::ffff:]

Dec 14 16:54:35 imapd-ssl: couriertls: read: Connection reset by peer

Dec 14 16:54:35 imapd-ssl: Disconnected, ip=[::ffff:], time=0, starttls=1

Dec 14 16:54:36 imapd-ssl: Connection, ip=[::ffff:]

Dec 14 16:54:36 imapd-ssl: couriertls: read: Connection reset by peer

Dec 14 16:54:36 imapd-ssl: Disconnected, ip=[::ffff:], time=0, starttls=1

Dec 14 16:54:37 imapd-ssl: Connection, ip=[::ffff:]

Dec 14 16:54:37 imapd-ssl: couriertls: read: Connection reset by peer

Dec 14 16:54:37 imapd-ssl: Disconnected, ip=[::ffff:], time=0, starttls=1

Dec 14 16:54:38 imapd-ssl: Connection, ip=[::ffff:]

Dec 14 16:54:38 imapd-ssl: couriertls: read: Connection reset by peer

Dec 14 16:54:38 imapd-ssl: Disconnected, ip=[::ffff:], time=0, starttls=1

##

does this help indicate anything I might try?

thanks
[Windows Mobile Software Development] Fixup:
Maybe you try Gmail first? Mail servers do not implement imap idle the same way, ImapPusher is mainly aimed at Gmail.