Together better

Jan 9, 2014 at 10:27 PM
Jan 10, 2014 at 10:05 AM
Hi kiquenet,

can you specify what kind of collaboration you suggest?


Jan 10, 2014 at 12:02 PM
Edited Jan 10, 2014 at 4:20 PM
I'm newbie, but I have seen several open source components-libraries about IMAP. I'm considerer better for community that gurus and experts work together, IMHO. Greetings.

Jeffrey Stedfast says:

higlab: Seems like a pretty awful implementation, so it'd be a complete waste of my time even trying to integrate with their product. I'd be more productive for me to rewrite their library from scratch... so not gonna bother.

mailsystem: Wow, this is pure crap. It uses String.IndexOf('}') to parse literal strings returned by the IMAP server? Really??? Not worth my time.

imapx: Extremely sloppy parsing code, so again, not worth my time.

aenetmail: Another pretty sloppy MIME, POP3, IMAP, and SMTP implementation.

If they want to use MimeKit to replace their shitty MIME parsers, that's great, but I'm not going to get involved in these projects. It'd be time better spent if I just write a mail library on my own... which is what MailKit is all about.

The only piece that MailKit is missing is an ImapClient at this point, and since all of the other libraries you listed have horrendous implementations of everything, it'd be easier to just write an ImapClient for MailKit than it would be to fix any of those other libraries.

I'd like to implement IMAP support in MailKit... just not sure how high/low-level to make it. IMAP is pretty complex and any mail client dealing with IMAP is going to want to cache information for offline use. Should MailKit do that? Or should that be left to the application using MailKit? The IMAP protocol also supports partial downloads of messages (you can download individual parts or specify the number of bytes and a byte offset into the message to download... Should ImapClient expose this functionality? If so, how? These are just a few examples of the type of questions I need to think about. POP3 is dead simple because the purpose is simply to copy the mail to the local machine. IMAP on the other hand is meant to keep the mail on the server.
Jan 10, 2014 at 5:30 PM
Hi kiquenet,

you're right, and there are many projects in this field. Considering your words and also what Jeffrey Stedfast said:

I was looking for a good way to implement the MIME parsing part, and some work has been done already. However, MimeKit seems a pretty good alternative, I will consider using it. This will allow to put more focus on providing easy to use APIs to deal with the IMAP protocol and communication with the server in general, and worry less about parsing the messages on the client.

The approach about working together is good, however as Jeffrey Stedfast said, he doesn't want to be involved in other projects, so will less of other authors do.

So from my side I can integrate MimeKit in ImapX to make it a better library, but I cannot provide support for other libraries.

Best regards,

Jan 11, 2014 at 9:46 AM
Thanks for your comments. Really good.

About working together is good if anyone wants it. There are several projects about SMTP, POP3, IMAP, MIME...maybe anyones working together for improvement better libraries, all efforts to Community. However, every developer is free to do it or not. Like I'm newbie, if only suggestions for all good developers like you and others in those projects.

Christtian Nilsson say

Why I started to use and still use S22.Imap instead of any other lib:
S22.Imap is a IMAP Client, nothing else just a IMAP client.
When I was trying to find a IMAP Client I found and looked on AENetMail however that is not a IMAP Client, it is a full email client library.

Any other library that is not only a IMAP Client is bloated for my purposes (sorry for the choice of word)

Something that might be interesting however is
If I can find the time I would like to check it out and see if it could be reused.
But MailKit is of no use to me since (as with many of the others) it contains a POP3 client as well.
And then we have Comments like "Wow, this is pure crap" who wants to work with someone with a attitude like that?

One of the biggest issues with the .Net implementation of mail is the standard Net.Mail namespace, but most of what is needed for other client libraries are Internal in the base framework, and thus must be recreated with extra code.

Jeffrey Stedfast says:

MimeKit can be reused (that was the goal of the project). In fact, I wasn't really planning to write MailKit since I have no real interest in writing a mail client. The plan was for other people to write POP3, SMTP, and IMAP libraries built on top of MimeKit.

After getting a bunch of emails asking me how to send MimeKit messages via SMTP and/or asking me to plug MimeKit into this or that POP3 library, I gave up and implemented my own SmtpClient and Pop3Client. Since these clients are vastly more robust than any of the other Smtp and Pop3 clients I've seen, I can just point people at using those. I'm not super enthusiastic about writing an ImapClient because the last C implementation I wrote was ~10,000 lines of code to support all of the features and be usable while the network is offline, handling reconnecting when the server sends back untagged BYE responses that weren't the result of a LOGOUT command, etc. and that didn't even include IDLE support (which these days is a pretty MUST HAVE feature).

You'll notice that unlike every other POP3 .NET library, MailKit's Pop3Client does not use a lame line-reader approach (see Pop3Stream), nor does it duplicate the retrieved messages into a memory stream before parsing, it parses directly from the socket doing on-the-fly dot-decoding and is completely cancelable (another advantage of MimeKit).

I called that other IMAP client implementation crap because, well, it's crap. I mean, you can't just read a hunk of data from the socket, convert it into unicode (without first knowing what charset the mime message is in) and then use IndexOf(')') to figure out where the end of the message is with the assumption that that character does not appear in the message itself (haha, good luck with that).

Most people who implement POP3 and IMAP clients assume that the protocols are string-based, but they really aren't. At least not when you are using a programming language where strings are multi-byte (such as UTF-16). Sure, they are text-based, but when using Java or C#, you can't really use strings because you need to know the charset first and you can't know the charset before you parse the email message (or, in the case of IMAP, envelope data as well), and even then, each part might be in a different charset, so you can't use a single charset for conversion into unicode. The only parts of the response that you can read as strings are the actual protocol tokens (because they are ASCII tokens), which means you CANNOT use a TextReader. You have to read raw bytes (you can convert segments of the byte[] into strings as you go, of course, but you can't just gulp up the entire response as a string).

Then, of course, there's the fact that when reading a literal string response from the IMAP server, most IMAP implementations I've seen make the incorrect assumption that the octet count provided in the "{xxx}" is the same as the String.Length, which of course is wrong.

Now that I've pointed out these fundamental problems that plague every Pop3 and IMAP client I've seen for .NET, I'm sure you better understand why I call such clients crap.

Just an example of what I'm talking about: andyedinborough/aenetmail#150

AENetMail doesn't properly decode the dot-encoding that is used in the RETR command, nor does it even use the correct approach to figuring out the end of the response which is what causes any further commands to fail.

Some bugs are just bugs, but others are fundamental design problems.

Anyway, I don't care one way or another if S22.Imap decides to use MimeKit, although I'm more than willing to answer any questions the developers have about using it and fixing any bugs or adding any features they need (within reason, of course).

Most of the time, plugging in an alternative XYZ library is not trivial and requires a lot of rewriting of significant chunks of code, and generally developers of libraries such as this aren't going to bother rewriting their library if the itch they set out to scratch has already been solved well enough for their needs.
Jan 12, 2014 at 5:46 PM
Hi Pavel,

There are a few problems I see in the ImapX code.
  1. It doesn't appear to me that ImapX handles parsing of LIST responses that use literals for the mailbox name (unless the regex logic somehow handles it, but I didn't know where to look for the regex pattern).
  2. In Folder.cs, in the AppendMessage() method, your code assumes that the email.Length is the same as the octet length, but they are not. The String.Length property is the unicode character count, not the octet count. You need to convert to a byte[] and then use that length.
If you decide to switch to using MimeKit.MimeMessages instead of strings in your API, you could do the following:
var options = MimeKit.FormatOptions.Default.Clone ();
options.NewLineFormat = MimeKit.NewLineFormat.Dos;

long octets;
using (var measure = new MimeKit.IO.MeasureStream ()) {
    message.WriteTo (options, measure);
    octets = measure.Length;
A measure stream is just there to measure the number of bytes written to it but doesn't actually store any data, so it's really useful for this sort of thing.

It's a bit worrisome is that I can't find where you handle literal strings in responses from the IMAP server. It seems to me that you use a StreamReader to read line-by-line, and simply assume that the text after the first line is the message body until you reach a line beginning with a ")". This, however, is incorrect for at least 2 reasons:
  1. What if a line in the message begins with ")"? I have some LISP emails that have lines beginning with a ")" that your ImapX client library would likely break on if I tried to fetch them over an ImapXClient connection.
  2. The message data itself isn't required to end with a CRLF which means that the last line of the literal may end mid-line which means that the ")" would also be mid-line and not at the beginning of a line like your code seems to depend on.
You MUST parse the literal octet count and use that as your guide to determine how many bytes the message is. That length value is not there for show, it's there because it is absolutely required in order to properly parse the response.

Hopefully I am just misunderstanding your code.
Jan 15, 2014 at 12:00 PM
Hi Jeffrey,

nice to see you here!

Thank you really much for the code review. I agree with all points you mentioned. ImapX was originally written by another author, so some concepts are inherited, and by far not the best. With the latest version I started to change the approach how messages and other commands are executed and server responses are parsed. However, it takes time to completely go away from the string based processing, especially the line-by-line reading of server responses. First step here was adding of realtime processing of the server responses. Next is, as you have pointed out, base the message parsing on literal octet count and not dependent on any characters, such as ')'.

Currently I'm reading the docs of MimeKit to see how I can best use it in order to improve the performance of ImapX and also eliminate the existing bugs.

Best regards,

Jan 15, 2014 at 1:02 PM
Hi Pavel,

It is refreshing to hear that you seem to be on top of things with regards to the use of IMAP literals.

Email is a pet peeve of mine and I can often be quite critical of mail library implementations that don't do things correctly (and is, unfortunately, very often the case). I'm glad that you do not seem to have taken my criticisms personally (as that was not the intention).

If you have any questions about MimeKit (or if you spot anything MimeKit is doing wrong), shoot me an email (should be listed on my github page) and/or file a bug on my github project and I'll try to answer it or fix it, as the case may be.

I haven't released a v1.0 of MimeKit yet mostly because I want to get more feedback on the API (I'd also like for a crypto expert to look over my S/MIME and PGP code to make sure I'm understanding things correctly). When I started working on MailKit around late November or early December and implemented SMTP and POP3, I discovered that it'd be really nice if MimeKit's parser (MimeMessage.Load, MimeEntity.Load, MimeParser.Parse*) and serializer (MimeMessage.WriteTo, MimeEntity.WriteTo, etc) had a way to cancel the operation because that would allow me to write messages directly to the socket (SMTP) or parse them directly from the socket (POP3). Without having written an SmtpClient or Pop3Client, I never would have thought of having such an API in MimeKit.

I'm not sure if using MimeKit in an IMAP client situation will reveal more API improvements that could be made in MimeKit or not, so if you do start using MimeKit and think of features that would really help, let me know and perhaps I can design an API to offer you whatever feature you might need.

Oh, also, since you are reading over the documentation... that's another thing holding me back from releasing a v1.0. I'm sure there's docs that can be improved there, so if there's any particular areas that you feel the docs are lacking in, don't hesitate to let me know so I can try and fix that (by either expanding on the remarks and/or by providing examples if needed).

Good luck to you on ImapX,

Jan 15, 2014 at 10:37 PM
Very interesting sharing all this information, very useful
Jan 19, 2014 at 8:31 PM
Just a bit of a status update on my end... I was curious what it would take to implement my own ImapClient (holy crap a lot of IMAP extensions have been published since I last worked on an IMAP client!), so the past 36 hours or so I've had a bit of a hacking marathon.

The IMAP protocol is really designed to use a tokenizer, so I wrote an ImapStream class that tokenizers server responses and, when it runs into a literal token, toggles itself into "literal mode", allowing you to read from it like a normal Stream, cutting you off when you've finished reading the literal data (IOW: it returns 0 once you've finished reading data).

On top of that, I have an ImapEngine and ImapCommand which is more-or-less a command pipeline engine for queueing and dispatching commands to the server and then reading the responses and dispaching untagged events to custom handlers (or to the engine itself for stuff that it needs to keep state).

Sitting on top of that, I have ImapClient and ImapFolder which drive the engine.

ImapFolder, at this point, is just a stub, but you can get a list of all of the personal, shared, and other namespaces from the ImapClient (along with all of the special folders if the server supports the SPECIAL-USE extension). You can also authenticate via a handful of SASL mechanisms.

This is all part of my MailKit project, if you are interested in taking a look. I promise you won't be disappointed.
Jan 21, 2014 at 12:06 PM
I've been on a roll and ImapFolder is now almost fully implemented as well. At this rate I should have a fully functional IMAP client by the end of the week.
Jan 23, 2014 at 10:02 AM
Sorry, couldn't answer faster, a lot of daily work these days. I took a look at your code, and I think using a tokenizer is a good choice. A couple of weeks ago I looked over the Antlr3 docs, to see if I can use it to create a parser for the server responses. However, the question if it can deal with literal tokens well stayed open.

I will take a closer look at your code this weekend.


Jan 23, 2014 at 9:19 PM
Using an actual tool to generate a parser may be a better option than hand-rolling one like I did, but eh... mine seems to work well enough :-)

I've written up some unit tests for ENVELOPE, BODY, and BODYSTRUCTURE list-expressions and uncovered a few buglets in my parsers (none in the tokenizer yet, though!)... so I'm a weeee bit closer to having a working client. I was hoping to not have to write a custom INTERNALDATE parser, but sadly .NET's DateTimeOffset.TryParse*() methods don't have a format specifier for a timezone that includes hours and minutes w/o a : separator - who uses a : when writing a timezone offset? boggle

I have uncovered one area that may be useful to try and provide helpers for in MimeKit, though, which is the decoding of Content-Type and Content-Disposition parameters in BODY and BODYSTRUCTURE responses. My current ImapClient parser doesn't handle that yet.

One of the last things I need to do to get a working ImapClient (that I know of, may be other things) is to have the engine update its selection state when a SELECT, EXAMINE, CLOSE, or UNSELECT command is processed. Not sure if I want to make the engine watch for those commands to pass through itself, or have the ImapFolder objects update the engine's state since they already have that info...
Jan 23, 2014 at 11:53 PM
n/m, apparently the "zzz" specifier will parse timezones like those used in the INTERNALDATE string. Yay!
Jan 24, 2014 at 1:40 AM
Blah, found a bug in my tokenizer tonight while getting the last of the pieces together to fetch messages/summaries/etc. On the plus side, I've now got a "fully working" working ImapClient that can fetch summary info for messages, the messages themselves, get folders, list them, etc.

I still have to implement CREATE, DELETE, and RENAME... but first I need to figure out the public API I want for that (implementing the commands themselves is trivial).
Jan 25, 2014 at 4:26 PM
Just implemented SEARCH support which was the last of the things that needed to done to call it "complete".

Wrote CREATE, DELETE and RENAME yesterday.

Now I just need to write docs and more unit tests... i.e. the fun part :-)
Jan 25, 2014 at 10:16 PM
Okay... docs written and then implemented some GMail IMAP extensions: XLIST, X-GM-MSGID, and X-GM-THRID
Jan 25, 2014 at 11:26 PM
Fantastic, great mr. Stedfast !! Congratulations to Pavel using MimeKit and news features.
Jan 26, 2014 at 9:44 AM

nice to hear you have implemented a working ImapClient! I will give it a test run today.
I've written a grammar for ANTLR for parsing the server responses, now have to generate a proper parser and see if can already include it in the coming update.

Guess I will first make a bug fix release to cover the reported issues, and then focus on integrating MimeKit and the ANTLR based parser.
Jan 26, 2014 at 12:22 PM
Sounds good! Let me know if you have any questions (especially about MimeKit, I know nothing about ANTLR except what it is).
Jan 26, 2014 at 3:12 PM
Sound good working together using MimeKit, MailKit and ANTLR. Thanks.