Cyrus Import Mail Invalid Header

This is a common issue people are having when importing mail messages from other services or desktop clients. The message normally shown in the console when importing the messages is:

Message contains invalid header

This is because Cyrus IMAPD is strict and requires the message header to be within a certain bound.

Finding problematic messages

The easiest way to find the problematic messages is to purge the ones that imports fine and hash out the remaining messages. I did this in Thunderbird by using the Remove Duplicate Messages Alternate add-ons. Also the ImportExportTools add-ons comes in handy.

1. Prepare the folder you want to move over to the Cyrus IMAP server
2. Let it copy and wait for the transfer to end with the Invalid Header error
3. Make a copy of the original messages folder
4. Copy back all the messages on the Cyrus server back to the new copied folder
5. In the Duplicate Messages Alternate add-on options, change the filtering to "To", "From", "Message-Id" and "Subject"
6. Right-click on the newly copied folder with all the Cyrus messages also included
7. A list of lots duplicates will be displayed, click No Copies, and click Ok to purge the ones that did copy to the server
8. You're left with a folder with mostly messages that did not copy to the Cyrus server
9. Try to copy a few of the messages until you get the Invalid Header error
10. View source of the message and inspect the header and fine what is different from a working message

Headers containing '>>From header'

In my case, it turned out that the messages with headers starting with:

">>From "

and

">>>From "

Was detected to be invalid and thus report an "Invalid Header" error.

Fixing mbox

By simply exporting all the messages to a mbox-file, parsing all the lines in the mbox and imorting the mbox again fixed the problem.

Install the ImportExportTools add-on.

1. Right-click on the folder you want to fix and select Export as mbox file
2. Open a terminal window (Mac or Linux) and run the following command to see which lines return
   1. cat INBOX.mbox | grep ">>From "
   2. Result should be lots of line similar to: >>From vbei@cox.net Fri Feb 20 16:10:42 2004
3. Now, to change all the instances of ">>From " to ">From" (which is fine by Cyrus)
   1. cat INBOX.mbox | sed -e 's/^>>From \(.*\)\([0-9]\) 20\(.*\)/>From \1\2 20\3/g' > INBOX-fixed.mbox
4. The new file should be ready to import, some useful commands to check and verify
   1. cat INBOX.mbox | grep ">>From "|wc -l
   2. tail -100 INBOX-fixed.mbox
5. Import the fixed mbox in Thunderbird by right-clicking the Local folders and Import mbox file
6. Finally, drag the newly imported folder over to the Cyrus IMAP folder where you want the messages to be stored