Together better

Coordinator
Jan 10, 2014 at 9:05 AM
Hi kiquenet,

can you specify what kind of collaboration you suggest?

Greets,

Pavel
Coordinator
Jan 10, 2014 at 4:30 PM
Hi kiquenet,

you're right, and there are many projects in this field. Considering your words and also what Jeffrey Stedfast said:

I was looking for a good way to implement the MIME parsing part, and some work has been done already. However, MimeKit seems a pretty good alternative, I will consider using it. This will allow to put more focus on providing easy to use APIs to deal with the IMAP protocol and communication with the server in general, and worry less about parsing the messages on the client.

The approach about working together is good, however as Jeffrey Stedfast said, he doesn't want to be involved in other projects, so will less of other authors do.

So from my side I can integrate MimeKit in ImapX to make it a better library, but I cannot provide support for other libraries.


Best regards,

Pavel
Jan 12, 2014 at 4:46 PM
Hi Pavel,

There are a few problems I see in the ImapX code.
  1. It doesn't appear to me that ImapX handles parsing of LIST responses that use literals for the mailbox name (unless the regex logic somehow handles it, but I didn't know where to look for the regex pattern).
  2. In Folder.cs, in the AppendMessage() method, your code assumes that the email.Length is the same as the octet length, but they are not. The String.Length property is the unicode character count, not the octet count. You need to convert to a byte[] and then use that length.
If you decide to switch to using MimeKit.MimeMessages instead of strings in your API, you could do the following:
var options = MimeKit.FormatOptions.Default.Clone ();
options.NewLineFormat = MimeKit.NewLineFormat.Dos;

long octets;
using (var measure = new MimeKit.IO.MeasureStream ()) {
    message.WriteTo (options, measure);
    octets = measure.Length;
}
A measure stream is just there to measure the number of bytes written to it but doesn't actually store any data, so it's really useful for this sort of thing.

It's a bit worrisome is that I can't find where you handle literal strings in responses from the IMAP server. It seems to me that you use a StreamReader to read line-by-line, and simply assume that the text after the first line is the message body until you reach a line beginning with a ")". This, however, is incorrect for at least 2 reasons:
  1. What if a line in the message begins with ")"? I have some LISP emails that have lines beginning with a ")" that your ImapX client library would likely break on if I tried to fetch them over an ImapXClient connection.
  2. The message data itself isn't required to end with a CRLF which means that the last line of the literal may end mid-line which means that the ")" would also be mid-line and not at the beginning of a line like your code seems to depend on.
You MUST parse the literal octet count and use that as your guide to determine how many bytes the message is. That length value is not there for show, it's there because it is absolutely required in order to properly parse the response.

Hopefully I am just misunderstanding your code.
Coordinator
Jan 15, 2014 at 11:00 AM
Hi Jeffrey,

nice to see you here!

Thank you really much for the code review. I agree with all points you mentioned. ImapX was originally written by another author, so some concepts are inherited, and by far not the best. With the latest version I started to change the approach how messages and other commands are executed and server responses are parsed. However, it takes time to completely go away from the string based processing, especially the line-by-line reading of server responses. First step here was adding of realtime processing of the server responses. Next is, as you have pointed out, base the message parsing on literal octet count and not dependent on any characters, such as ')'.

Currently I'm reading the docs of MimeKit to see how I can best use it in order to improve the performance of ImapX and also eliminate the existing bugs.

Best regards,

Pavel
Jan 15, 2014 at 12:02 PM
Hi Pavel,

It is refreshing to hear that you seem to be on top of things with regards to the use of IMAP literals.

Email is a pet peeve of mine and I can often be quite critical of mail library implementations that don't do things correctly (and is, unfortunately, very often the case). I'm glad that you do not seem to have taken my criticisms personally (as that was not the intention).

If you have any questions about MimeKit (or if you spot anything MimeKit is doing wrong), shoot me an email (should be listed on my github page) and/or file a bug on my github project and I'll try to answer it or fix it, as the case may be.

I haven't released a v1.0 of MimeKit yet mostly because I want to get more feedback on the API (I'd also like for a crypto expert to look over my S/MIME and PGP code to make sure I'm understanding things correctly). When I started working on MailKit around late November or early December and implemented SMTP and POP3, I discovered that it'd be really nice if MimeKit's parser (MimeMessage.Load, MimeEntity.Load, MimeParser.Parse*) and serializer (MimeMessage.WriteTo, MimeEntity.WriteTo, etc) had a way to cancel the operation because that would allow me to write messages directly to the socket (SMTP) or parse them directly from the socket (POP3). Without having written an SmtpClient or Pop3Client, I never would have thought of having such an API in MimeKit.

I'm not sure if using MimeKit in an IMAP client situation will reveal more API improvements that could be made in MimeKit or not, so if you do start using MimeKit and think of features that would really help, let me know and perhaps I can design an API to offer you whatever feature you might need.

Oh, also, since you are reading over the documentation... that's another thing holding me back from releasing a v1.0. I'm sure there's docs that can be improved there, so if there's any particular areas that you feel the docs are lacking in, don't hesitate to let me know so I can try and fix that (by either expanding on the remarks and/or by providing examples if needed).

Good luck to you on ImapX,

Jeff
Jan 19, 2014 at 7:31 PM
Just a bit of a status update on my end... I was curious what it would take to implement my own ImapClient (holy crap a lot of IMAP extensions have been published since I last worked on an IMAP client!), so the past 36 hours or so I've had a bit of a hacking marathon.

The IMAP protocol is really designed to use a tokenizer, so I wrote an ImapStream class that tokenizers server responses and, when it runs into a literal token, toggles itself into "literal mode", allowing you to read from it like a normal Stream, cutting you off when you've finished reading the literal data (IOW: it returns 0 once you've finished reading data).

On top of that, I have an ImapEngine and ImapCommand which is more-or-less a command pipeline engine for queueing and dispatching commands to the server and then reading the responses and dispaching untagged events to custom handlers (or to the engine itself for stuff that it needs to keep state).

Sitting on top of that, I have ImapClient and ImapFolder which drive the engine.

ImapFolder, at this point, is just a stub, but you can get a list of all of the personal, shared, and other namespaces from the ImapClient (along with all of the special folders if the server supports the SPECIAL-USE extension). You can also authenticate via a handful of SASL mechanisms.

This is all part of my MailKit project, if you are interested in taking a look. I promise you won't be disappointed.
Jan 21, 2014 at 11:06 AM
I've been on a roll and ImapFolder is now almost fully implemented as well. At this rate I should have a fully functional IMAP client by the end of the week.
Coordinator
Jan 23, 2014 at 9:02 AM
Sorry, couldn't answer faster, a lot of daily work these days. I took a look at your code, and I think using a tokenizer is a good choice. A couple of weeks ago I looked over the Antlr3 docs, to see if I can use it to create a parser for the server responses. However, the question if it can deal with literal tokens well stayed open.

I will take a closer look at your code this weekend.

Greets,

Pavel
Jan 23, 2014 at 8:19 PM
Using an actual tool to generate a parser may be a better option than hand-rolling one like I did, but eh... mine seems to work well enough :-)

I've written up some unit tests for ENVELOPE, BODY, and BODYSTRUCTURE list-expressions and uncovered a few buglets in my parsers (none in the tokenizer yet, though!)... so I'm a weeee bit closer to having a working client. I was hoping to not have to write a custom INTERNALDATE parser, but sadly .NET's DateTimeOffset.TryParse*() methods don't have a format specifier for a timezone that includes hours and minutes w/o a : separator - who uses a : when writing a timezone offset? boggle

I have uncovered one area that may be useful to try and provide helpers for in MimeKit, though, which is the decoding of Content-Type and Content-Disposition parameters in BODY and BODYSTRUCTURE responses. My current ImapClient parser doesn't handle that yet.

One of the last things I need to do to get a working ImapClient (that I know of, may be other things) is to have the engine update its selection state when a SELECT, EXAMINE, CLOSE, or UNSELECT command is processed. Not sure if I want to make the engine watch for those commands to pass through itself, or have the ImapFolder objects update the engine's state since they already have that info...
Jan 23, 2014 at 10:53 PM
n/m, apparently the "zzz" specifier will parse timezones like those used in the INTERNALDATE string. Yay!
Jan 24, 2014 at 12:40 AM
Blah, found a bug in my tokenizer tonight while getting the last of the pieces together to fetch messages/summaries/etc. On the plus side, I've now got a "fully working" working ImapClient that can fetch summary info for messages, the messages themselves, get folders, list them, etc.

I still have to implement CREATE, DELETE, and RENAME... but first I need to figure out the public API I want for that (implementing the commands themselves is trivial).
Jan 25, 2014 at 3:26 PM
Just implemented SEARCH support which was the last of the things that needed to done to call it "complete".

Wrote CREATE, DELETE and RENAME yesterday.

Now I just need to write docs and more unit tests... i.e. the fun part :-)
Jan 25, 2014 at 9:16 PM
Okay... docs written and then implemented some GMail IMAP extensions: XLIST, X-GM-MSGID, and X-GM-THRID
Coordinator
Jan 26, 2014 at 8:44 AM
Hi,

nice to hear you have implemented a working ImapClient! I will give it a test run today.
I've written a grammar for ANTLR for parsing the server responses, now have to generate a proper parser and see if can already include it in the coming update.

Guess I will first make a bug fix release to cover the reported issues, and then focus on integrating MimeKit and the ANTLR based parser.
Jan 26, 2014 at 11:22 AM
Sounds good! Let me know if you have any questions (especially about MimeKit, I know nothing about ANTLR except what it is).