IDLE support?

Topics: Feature Requests
Apr 19, 2013 at 2:22 AM
Hello,

I may have overlooked this but I can't find any documentation on it- is there already IMAP IDLE support in ImapX or is this a planned feature? I spent quite a bit of time trying to google this to see If I could find an answer but go nowhere.

Thanks!
Coordinator
Apr 19, 2013 at 9:54 AM
Good day,

you're not mistaken, there is no IDLE Support at this time.
Nevertheless this feature is planned for the next release, so currently I'm working on implementing it.


Best regards,

Pavel Azanov
Apr 17, 2014 at 2:03 PM
Hello Pavel,

i would like to ask you about the status of the IDLE support, as the only place it was mentioned in the sourcecode is the Capabilities class.

With regards,

Alexander Powolozki
Coordinator
Apr 18, 2014 at 10:00 AM
Hi Alexander,

the IDLE support is in process, I just didn't push the code parts to the repository as still having a bit of trouble with it.

At the moment it's possible to initiate and stop the idle state for any folder, and get a notification about new messages have arrived. A few minor issues still persist, but I will update the code soon.

Best regards,

Pavel
Apr 28, 2014 at 8:57 AM
Edited Apr 28, 2014 at 8:58 AM
Any news on this matter?

Best regards

Martin
Coordinator
Apr 28, 2014 at 12:59 PM
Hi Martin,

I pushed the code to the repository, IDLE is working, however it's making some problems when IDLE is on and you try to select another folder. I will make another update tomorrow and release the library in new version.

Greets,

Pavel
Apr 29, 2014 at 6:01 AM
Edited Apr 29, 2014 at 6:02 AM
Sounds great, I love your component.

This IDLE is the only thing missing to kompleate my project.

Best regards

Martin
Apr 30, 2014 at 5:59 AM
I can try out your new changes in some way? (sorry if this is a stupid question)
Coordinator
Apr 30, 2014 at 9:19 PM
Hi mve,

you can always try the latest change by downloading and compiling the latest source code.

Greets,

Pavel
May 1, 2014 at 6:39 AM
Hi Pavel

will there be a new update on NuGet any time soon?

Best regards

Martin
Coordinator
May 1, 2014 at 7:50 AM
Hi Martin,

I will release version 2.0.0.16 today, including the nuget packages, so everyone can test the new features. Another release is planned for the weekend, as I need to fix some more bugs.

Greets,

Pavel
May 1, 2014 at 7:53 AM
Nice good work.

Looking forward to trying IDLE event out

Best regards

Martin
Coordinator
May 1, 2014 at 12:46 PM
Hi Martin,

the Nuget package is online.

Best regards,

Pavel
May 1, 2014 at 1:28 PM
Hi Pavel

Sounds great can you point me in the direction of an example?

Thanks

Martin
Coordinator
May 1, 2014 at 1:32 PM
Edited May 1, 2014 at 2:10 PM
Hi Martin,

you can take a look at the code of the sample application (C# only), it contains a sample on how to use IDLE. The documentation has also been updated.

-- Pavel
May 7, 2014 at 11:35 AM
Edited May 7, 2014 at 11:36 AM
Hi Pavel

I am struggling to use ImapX in a windows service.

The service must use IDLE to wait for incoming messages and save their attachments to disk.

It works periodically but mostly not, maby you can see what I'm doing wrong?


using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Diagnostics;
using System.Linq;
using System.ServiceProcess;
using System.Text;
using System.IO;
using ImapX;
using ImapX.Exceptions;
using System.Net.Sockets;

namespace IMAP.Service
{
public partial class Service : ServiceBase
{
    private static string pathRoot = @"C:\attachments\";
    private static string host = "xxxxxxxxxx";
    private static string lockin = "xxxx@xxxxx.com";
    private static string password = "xxxxx";

    public Service()
    {
        InitializeComponent();
    }

    protected override void OnStart(string[] args)
    {
        try
        {
            ImapClient client = new ImapClient(host, useSsl: true, validateServerCertificate: false);
            if (client.Connect())
            {
                if (client.Login(lockin, password))
                {
                    LogEvent("Imap Client Loged in", EventLogEntryType.Information);

                    // Attach handler to message arrived event
                    client.OnNewMessagesArrived += ImapClient_OnNewMessagesArrived;

                    // Start idling for inbox messages
                    client.Folders.Inbox.StartIdling();
                    LogEvent("IMAP Service idling started", EventLogEntryType.Information);
                }
                else LogEvent("User or password invalid", EventLogEntryType.Error);
            }
            else LogEvent("Connection could not be established", EventLogEntryType.Error);
        }
        catch (Exception e)
        {
            LogEvent("General exception.\n" + e.Message, EventLogEntryType.Error);
        }
    }

    static void ImapClient_OnNewMessagesArrived(object sender, IdleEventArgs e)
    {
        LogEvent("Message received", EventLogEntryType.Information);

        foreach (Message massage in e.Messages)
        {
            foreach (Attachment file in massage.Attachments)
            {
                file.Download();
                file.Save(pathRoot);
            }
        }

        LogEvent("Message processed", EventLogEntryType.Information);
    }

    private static void LogEvent(string message, EventLogEntryType type)
    {
        EventLog EventLog = new EventLog();
        EventLog.Source = System.Diagnostics.Process.GetCurrentProcess().ProcessName;
        EventLog.WriteEntry(message, type);
    }

    protected override void OnStop()
    {
        LogEvent("IMAP Service stoped", EventLogEntryType.Information);
    }
}
}
Coordinator
May 11, 2014 at 9:30 AM
Hi mve,

sorry for the delay, been busy these days.
In order to see what the problem is, I need a log file of a session where you test idle. To create one, do the following:
// Add this before you create the client
Debug.Listeners.Add(new TextWriterTraceListener(@"C:\users\public\test.txt"));
Debug.AutoFlush = true;

// After you created the client, set:
client.IsDebug = true;
In this way all requests and responses will be logged and you can send me the log file to info@imapx.org. Please make sure to remove the username & password from it.

Best regards,

Pavel
Coordinator
May 11, 2014 at 10:10 PM
Hi mve,

I made some more tests and found there was a bug in the idle support. Take a look at the latest code in the repository, it should be working fine now.

Greets,

Pavel
May 12, 2014 at 8:01 AM
Edited May 12, 2014 at 8:35 AM
Hi Pavel

It seems that it works if you send continuously, but if you take a break and start sending again, the service stopped working.

I have send you a log file.

Best regards

Martin
Coordinator
May 12, 2014 at 2:08 PM
Hi Martin,

I think the problem will be solved if the client will send a NOOP command from time to time to keep the connection. I will make another fix and let you know once it's available!

Greets,

Pavel
Coordinator
May 12, 2014 at 7:08 PM
Hi Martin,

you can check the latest source code in the repository. Now, once you start idle, the client will keep track of the time when last request to the server was made. Should it reach 14 minutes (client.Behavior.NoopIssueTimeout) since last activity, the client will send a NOOP command to the server in order to keep the connection alive.

Now everything should be working fine for your service!

Greets,

Pavel
May 16, 2014 at 9:00 AM
Hi Pavel

Thanks a lot for your support.

I downloaded the new version but I found some issues. The new thread has an active wait (it is in an loop) and I think that a Thread.Sleep should be better, isn't?
On the other hand, there is no return condition on such method and the join will never finish.

Best
Agustin
Coordinator
May 16, 2014 at 11:32 AM
Hi Augustin,

thank you! I checked the code, this problem only exists for the MaintainIdleConnection method. The other two methods (ProcessIdleServerEvents, WaitForIdleServerEvents) will stop once idle is paused/stopped.

I will update the code later tonight to fix the MaintainIdleConnection method.

Greets,

Pavel
Jul 6, 2014 at 10:49 PM
Hey Pavel,

I had some time in the last days and played around with the IDLE support running under Mono (multiple versions of Mono). I found a few things that should be considered to be changed in the future:

1) ProcessIdleServerEvents, WaitForIdleServerEvents and MaintainIdleConnection run in a tight loop each. The only loop that blocks if there is nothing to do is the one of WaitForIdleServerEvents. The other ones just burn the CPU like there is no tomorrow (I got 100%+ CPU usage when using IDLE, the profiler told me to look into your code and so I found the spots). I made some changes and the result is that using IDLE is harmless.
  • WaitForIdleServerEvents does not need any change. The IO is already blocking and the thread won't use much resources while waiting for new events
  • ProcessIdleServerEvents can be rewritten using a AutoResetEvent that signals the thread if new items were queued. It mustn't spin while there is nothing to do. Of course that needs a small change in WaitForIdleServerEvents as well (to set the signal). Alternatively one should use a blocking collectionfor that purpose. I don't know if there is a blocking collection in Windows Phone so that is why I used the signaling construct.
  • MaintainIdleConnection should be replaced completely by using a timer. There is System.Timers.Timer and something similar should be available for WinPhone if google got me right. In the current implementation there is a loop spinning continiously for Behavior.NoopIssueTimeout seconds without doing anything but using CPU. A timer that kicks in every Behavior.NoopIssueTimeout seconds to send a NOOP would not hurt anyone. One could also restart the timer on every communication to only send NOOPs if no communication was done for the last Behavior.NoopIssueTimeout seconds.
Like I stated I already made these changes and the lib keeps working (for my purpose, on Mono/.NET, not Phone). These are no big changes but they do have a great effect. Let me know what you think about it!

Little 2) The internal field that keeps track of "idle - YES/NO" is accessed from different threads concurrently without using a lock. That is no real problem until now because as far as I saw the code it won't really break anything in a race condition, but it would be cleaner to access it in a safe way. Just my 2 cents ;-)
Coordinator
Jul 7, 2014 at 8:01 AM
Hi DomeB,

thank you for doing such a great work!

I think the ideas are good, especially for the ProcessIdleServerEvents method. As for MaintainIdleConnection, I already replaced it with a timer for the coming version.

If you like, you can submit your changes as a patch, and I will take care to provide support for the new code on Windows Phone.

Thank you,

Pavel
Jul 7, 2014 at 10:21 PM
Edited Jul 8, 2014 at 6:08 AM
Hey Pavel,

I tried my best to get the patch as nice as possible. Had some trouble with Tab / Spaces ;-)
It is uploaded, hopefully you can apply it without problems!

Cheers,
Dome

Edit: I missed an important point: you need to protect the queue with a lock. Otherwise you may end up reading and writing to it at the very same time. If you do not need support for .NET < 4.0 you may also consider using a BlockingCollection<T>. It is available for Windows Phone as well. In that case you can also dismiss my patch with the AutoResetEvent ;-)
Jul 9, 2014 at 12:47 PM
Edited Jul 9, 2014 at 12:48 PM
Hey All,

I am new with Impax and I am trying to implement idle support in my project. But unfortunately when I tried to add global events for example OnNewMessagesArrived it is undefined when I called it. Here is my code. Please help me regarding that

Thanks!
Imports ImapX
Public Class MailHandler
    Public Sub StartIdleSupport()
          Dim client As ImapClient = New ImapClient(Config.MailHost, Config.MailPort)

          If client.Connect() Then
            If client.Login(Config.MailUser, Config.MailPass) Then
              
                 client.OnNewMessagesArrived 'unable to find it why?

            end if
          end if
      End Sub
End Class
Coordinator
Jul 9, 2014 at 2:52 PM
Hi alam99,

in order to handle new messages, you need to add an event handler, and later start idle for a specific folder. Consider that IDLE can only be started for one folder at a time. Once you start it for another folder, the first one will no longer be watched.

I changed your code a bit to make it work, here you are:
Imports ImapX
Public Class MailHandler

    Public client As ImapClient

    Public Sub StartIdleSupport()

        client = New ImapClient(Config.MailHost, Config.MailPort, True)

        If client.Connect() Then

            If client.Login(Config.MailUser, Config.MailPass) Then

                AddHandler client.OnNewMessagesArrived, AddressOf client_OnNewMessagesArrived
                client.Folders.Inbox.StartIdling()

            End If

        End If

    End Sub

    Public Sub client_OnNewMessagesArrived(sender As Object, eventArgs As IdleEventArgs)

        ' use eventArgs.Messages to access newly arrived messages
        ' all new messages are also automatically added to the folder
        ' use eventArgs.Folder to access it

    End Sub

End Class
Hope this helps,

Greets,

Pavel
Coordinator
Jul 9, 2014 at 2:53 PM
Hi DomeB,

thank you! I will include the patch in the main code this weekend!

Greets,

Pavel
Jul 9, 2014 at 3:09 PM
Hi Pavel,

Yes now I am unable to add events. Thank you very much for your help and I really apprecaite it!.
Jul 10, 2014 at 12:51 PM
Edited Jul 10, 2014 at 12:55 PM
Hi Pavel again,

I have a question regarding Idle support. I have made a window service and install it on my computer. Service is running correctly and calling Mail.MailHandler.StartIdleSupport() method but when I am sending a new email to my gmail account which I wanted to read. So my question is shouldn't be fired event OnNewMessagesArrived ?. Is something else I need to do for event fire? one more thing when I am checking client.Capabilities.Idle it thrown a null exception thats why I commented it in my my code see below.

I am sorry if I am asking very basic question :( I am not aware how idle support actually work. I would really appreciate your help

Thanks!
Public Class MailHandler

      Public Shared Sub StartIdleSupport()

            LogEvent("Started Idle Support", NSL.Config.EventLogEntryType.Information)
            Dim client As ImapClient

            Try
                Debug.Listeners.Add(New TextWriterTraceListener(NSL.Config.LogFile))
                Debug.AutoFlush = True

                client = New ImapClient(NSL.Config.MailHost, NSL.Config.MailPort, True)

                client.IsDebug = True

                'If client.Capabilities.Idle Then
                '    LogEvent("Server supports idle", NSL.Config.EventLogEntryType.Information)
                'End If

                'Do not examine folders, it will gain us performance
                client.Behavior.ExamineFolders = False

                'Download only the headers we need
                client.Behavior.MessageFetchMode = Enums.MessageFetchMode.Body Or Enums.MessageFetchMode.Attachments Or Enums.MessageFetchMode.Flags Or Enums.MessageFetchMode.Headers

                'Download body, attachments, flags and headers
                client.Behavior.RequestedHeaders = {MessageHeader.ContentType, MessageHeader.Date, MessageHeader.From, MessageHeader.MessageId}

                client.Behavior.AutoPopulateFolderMessages = True

                If client.Connect() Then

                    If client.Login(NSL.Config.MailUser, NSL.Config.MailPass) Then

                        LogEvent("Imap Client Loged in", NSL.Config.EventLogEntryType.Information)

                        AddHandler client.OnNewMessagesArrived, AddressOf client_OnNewMessagesArrived

                        client.Folders.Inbox.StartIdling()

                    End If

                End If
            Catch ex As Exception
                LogEvent(ex.ToString(), NSL.Config.EventLogEntryType.ErrorMessage)
            End Try

        End Sub

        Protected Shared Sub client_OnNewMessagesArrived(sender As Object, eventArgs As IdleEventArgs)

            ' use eventArgs.Messages to access newly arrived messages
            ' all new messages are also automatically added to the folder
            ' use eventArgs.Folder to access it

            LogEvent("Total new arrivded messages are: " & eventArgs.Messages.Length.ToString(), NSL.Config.EventLogEntryType.Information)

            Dim success As Boolean = True

            For Each msg In eventArgs.Messages

                Dim plainTextBody As String = If(msg.Body.HasText, msg.Body.Text, "")
                Dim htmlBody As String = If(msg.Body.HasHtml, msg.Body.Html, "")

                Try

                    For Each attachment In msg.Attachments
                        attachment.Download()
                        attachment.Save(NSL.Config.AttachmentsFolder, Nothing)
                    Next

                Catch ex As Exception
                    LogEvent(ex.ToString(), NSL.Config.EventLogEntryType.ErrorMessage)
                    success = False
                End Try

            Next

            If success Then
                LogEvent("Read all new messages successfully.", NSL.Config.EventLogEntryType.Success)
            End If

        End Sub

        Private Shared Sub LogEvent(message As String, eventtype As Integer)

            Select Case eventtype

                Case NSL.Config.EventLogEntryType.Information
                    NSL.Logger.LOGFILE("MailHandler log event type information", message)
                Case NSL.Config.EventLogEntryType.ErrorMessage
                    NSL.Logger.LOGFILE("MailHandler log event type error message", message)
                Case NSL.Config.EventLogEntryType.Success
                    NSL.Logger.LOGFILE("MailHandler log event type error success", message)
                Case Else
                    NSL.Logger.LOGFILE("MailHandler log event type unknown", message)
            End Select

        End Sub

    End Class
Jul 10, 2014 at 7:11 PM
Hey alam99,

the code provided doesn't show an important part: do you ensure that your program keeps running after you call "StartIdleSupport()"?

Part of my code that ensures that (for Linux):
StartIdleSupport();

// singals to wait for
UnixSignal[] signals = new UnixSignal[] { 
    new UnixSignal(Signum.SIGINT), 
    new UnixSignal(Signum.SIGTERM), 
};

for(bool exit = false; !exit;)
{
    // this call blocks until a signal is received
    int id = UnixSignal.WaitAny(signals);
    if(id >= 0 && id < signals.Length)
    {
        if(signals[id].IsSet)
        {
            exit = true;
        }
    }
}

// do cleanup stuff here
Another possibility is setting a signal on your own (untested):
StartIdleSupport();

ManualResetEvent evt = new ManualResetEvent(false);

evt.WaitOne();

// do cleanup stuff here
In this example you should make sure to call "Set()" on "evt" whenever the application should quit.
Jul 11, 2014 at 8:34 AM
Edited Jul 11, 2014 at 11:02 AM
Hi DomeB,

Thankns for your reply. Yes you are right application should running but i was running it as window service so it shouldn't be end. Just to test I made console application and it seems that it is working but I noticed that before new message arrived event occured firstly all old emails written in TextWriterTraceListener log file and in the end OnNewMessagesArrived function called. Is this correct functionality? or we can reduce the time ? just read new emails instead here is my new code according your explaination
Public Class WinService       
    

        Public Shared Sub Main()

            Dim _thread As Thread
            _thread = New Thread(AddressOf Mail.MailHandler.StartIdleSupport)
            _thread.Name = "My Worker Thread"
            _thread.IsBackground = True
            _thread.Start()

            Dim _shutdownEvent As ManualResetEvent = New ManualResetEvent(False)

            _shutdownEvent.WaitOne()
            _shutdownEvent.Set()
            _thread.Abort()

        End Sub

   End Class
Coordinator
Jul 11, 2014 at 2:53 PM
Edited Jul 11, 2014 at 2:54 PM
Hi alam99,

the old messages are downloaded because you have set client.Behavior.AutoPopulateFolderMessages to True. Which makes the client download all the messages once you access the Folder.Messages property. Simply remove that line from your code to stop the client downloading old messages.

To your previous post, asking about client.Capabilities.Idle throwing a NullReferenceException: You're accessing client.Capabilities before you call the Connect method - at this point it is not yet initialized.

Greets,

Pavel
Jul 11, 2014 at 3:05 PM
Hi Pavel,

Thanks your reply. I will test it later however, I really appreciate your comments.

Thanks & Have a nice Weekend!

BR/Usman
Coordinator
Jul 13, 2014 at 1:59 PM
Hi DomeB,

I applied the patch you provided, made some minor changes and added support for Windows Phone. I will also see to implement a thread-safe queue to use with .Net < 4.0 a little later.

Thank you really much!

Greets,

Pavel
Jul 13, 2014 at 6:21 PM
Hey Pavel,

no problem, I feel lucky to switch back to the "official" version of your library and dismiss my custom build.
Lucky you, there are only three times where "_idleEvents" is accessed in the code (ImapBase) so it will be easy to protect all three occurrences with a new locl object :-)
Jan 16, 2015 at 10:13 PM
Hey Pavel,

I have been using the ImapX library to connect to a gmail account get information from an email process that information and then use the
client.Folders.Inbox.OnNewMessagesArrived += Folder_OnNewMessagesArrived;
            client.Folders.Inbox.StartIdling();
to Idle and wait for a new email. Well at first I used the ImapX 2.0.0.16 that worked fine but maxed my cpu usage out. Then I found this discussion and decided to compile the development code and use it hoping you had fixed my issue in some of the newer patches. Good news is that fixed my cpu usage problem but now It will idle and then once it receives a new email to process it just ends? I thought about recalling the idle command after it process the email but am worried about not properly closing the idle command.