[VM] Info node "External Messages"

[Alan Wehmann]

I tried out some of what is described in Info node "External Messages"
yesterday. I chose to limit message size to 50K bytes and I used the
IMAP server at work. Since I am used to how Thunderbird implements
its size limitations I was surprised to find myself being asked yes or
no for each over-sized message. I answered no in each case (there
were 533 messages and 30-odd were oversized); I was then asked whether
or not to delete them from the "maildrop". I got confused at this
point and chose to delete them (a bad choice, it turned out). My
memory of Thunderbird's behavior is that a choice of deleting a
message means that the message is moved to some sort of trash folder
(or is marked for deletion). I found out that in the case of VM, the
message on the IMAP server is deleted and is completely gone.

My memory of how Thunderbird behaves may be colored by the fact that
work used to have a different IMAP server & I have had more experience
using that one--as compared to the current one.

Based on this experience I would recommend that the Info node
"External Messages" have wording added to it, so that the user is
warned in some manner that the treatment of oversized (and deleted)
messages differs from what he might have experienced with Thunderbird.

I was using the trunk version of VM (at revision 1390).

One lesson for me is that I need to clean out my IMAP inbox folder at
work (by deleting messages that I don't want and moving others to a
different folder). I used to do this more diligently (using e.g. the
server web interface). In those times I moved messages with
attachments that I wished to keep, to a different folder.

I did note that VM seemed "slow" when it went through my ~500 messages
for a second time and remarked that all of them had already been
downloaded already.

[Uday Reddy]

Post by Alan Wehmann
I tried out some of what is described in Info node "External Messages"
yesterday.

Hi Alan, I can't tell what you tried from this info node, but the rest of
your message indicates to me that you were using Local Folders. External
messages are only available for IMAP Folders. So, I would expect that
whatever you tried from the info node would not have had any effect.

Since you say that you chose to delete the downloaded messages from the
server, the only copies you have now are in the Local Folder. So, please be
very careful with what you do with them. They are gone from the server.

My impression is that Thunderbird doesn't really have Local Folders for IMAP
servers. They are a bad idea. If I had a choice, I would get rid of them.
But Kyle Jones implemented them and a lot of people use them. So, it
appears that my hands are tied.

The variable `vm-imap-max-message-size' is also used for downloading
messages into Local Folders but I notice that it wasn't documented in the
info node. I will fix that.

Cheers,
Uday

[Alan Wehmann]

Subject: Re: [VM] Info node "External Messages"
Date: Thu, 5 Apr 2012 07:07:58 +0100

Post by Alan Wehmann
I tried out some of what is described in Info node "External Messages"
yesterday.

Hi Alan, I can't tell what you tried from this info node, but the rest of
your message indicates to me that you were using Local Folders. External
messages are only available for IMAP Folders. So, I would expect that
whatever you tried from the info node would not have had any effect.
Since you say that you chose to delete the downloaded messages from the
server, the only copies you have now are in the Local Folder. So, please be
very careful with what you do with them. They are gone from the server.
My impression is that Thunderbird doesn't really have Local Folders for IMAP
servers. They are a bad idea. If I had a choice, I would get rid of them.
But Kyle Jones implemented them and a lot of people use them. So, it
appears that my hands are tied.
The variable `vm-imap-max-message-size' is also used for downloading
messages into Local Folders but I notice that it wasn't documented in the
info node. I will fix that.
Cheers,
Uday
2nd Subject: Re: [VM] Info node "External Messages"
2nd Date: Thu, 5 Apr 2012 08:54:49 +0100
2nd
2nd
2nd > IMAP server at work. Since I am used to how Thunderbird implements
2nd > its size limitations I was surprised to find myself being asked yes or
2nd > no for each over-sized message.
2nd
2nd The doc-string for `vm-imap-max-message-size' says that the yes/no question
2nd is asked only when `vm-get-new-mail' is invoked interactively. It is not
2nd supposed to be asked if you did `vm-visit-folder' and had
2nd `vm-auto-get-new-mail' set to t.
2nd
2nd Did that work correctly?
2nd
2nd Cheers,
2nd Uday
2nd

I'll try to describe the settings that I used (the ones that are
relevant to this discussion):

vm-auto-get-new-mail t
vm-fetched-message-limit 10
vm-enable-external-messages '(imap)
vm-imap-max-message-size 50000
vm-spool-files '("imap:127.0.0.1:143:inbox:login:*:*")

I had stunnel running as a service (on my Windows XP Home laptop).

In the stunnel configuration file I had

[imaps]
accept = localhost:143
connect = email.fnal.gov:993

I used the command "vm" to initiate the downloading of messages from
the IMAP server.

There were 533 messages in the IMAP inbox. Roughly 35 of these were
greater than 50000 in size. For each of these "large" messages I was
asked if I wished to download them. For each I responded no. I was
then asked if I wanted to delete them from the "maildrop". In a
confused state, I answered yes (not a good answer). As a result they
were completely gone.

I attach a couple of configuration windows from Thunderbird, for the
IMAP server. I have chosen not to have Thunderbird download message
copies to my computer, but it is possible to do so, and impose a size
limit. One of the windows illustrates what happens to messages that
are marked for deletion via Thunderbird.

Once you add the documentation that describes the use of
'vm-imap-max-message-size' for downloading messages into Local
Folders, then the source of my confusion might go away.

For me there is a mode of operation where I like to download selected
IMAP messages into Local Folders & have a local copy. As I indicated
in one of my messages, I need to use the web interface to the IMAP
server to clean out the IMAP server inbox. I would use the web
interface for this, since then the limited DSL upload speed is not a
factor when moving messages from one folder to another on the server.

[Uday Reddy]

Post by Alan Wehmann
vm-spool-files '("imap:127.0.0.1:143:inbox:login:*:*")
I used the command "vm" to initiate the downloading of messages from
the IMAP server.

Right, you have used an IMAP spool file to download messages into a local
folder, the vm-primary-inbox. So, the external message stuff doesn't
apply.

On the other hand, the vm-get-new-mail was invoked automatically. So, VM
should not have asked you yes/no questions. I will check why that is
happening.

Post by Alan Wehmann
There were 533 messages in the IMAP inbox. Roughly 35 of these were
greater than 50000 in size. For each of these "large" messages I was
asked if I wished to download them. For each I responded no. I was
then asked if I wanted to delete them from the "maildrop". In a
confused state, I answered yes (not a good answer). As a result they
were completely gone.

Were only the downloaded messages gone, or did the skipped messages go away
too? That would be quite a serious bug if the skipped messages got deleted.
But nobody reported anything like it all these years!

Cheers,
Uday

[Alan Wehmann]

Post by Uday Reddy

Post by Alan Wehmann
vm-spool-files '("imap:127.0.0.1:143:inbox:login:*:*")
I used the command "vm" to initiate the downloading of messages from
the IMAP server.

Right, you have used an IMAP spool file to download messages into a local
folder, the vm-primary-inbox. So, the external message stuff doesn't
apply.
On the other hand, the vm-get-new-mail was invoked automatically. So, VM
should not have asked you yes/no questions. I will check why that is
happening.

Post by Alan Wehmann
There were 533 messages in the IMAP inbox. Roughly 35 of these were
greater than 50000 in size. For each of these "large" messages I was
asked if I wished to download them. For each I responded no. I was
then asked if I wanted to delete them from the "maildrop". In a
confused state, I answered yes (not a good answer). As a result they
were completely gone.

Were only the downloaded messages gone, or did the skipped messages go away
too? That would be quite a serious bug if the skipped messages got deleted.
But nobody reported anything like it all these years!
Cheers,
Uday

The messages that were under 50000 bytes in size were downloaded to my
local file "~/INBOX" (and are still present on the IMAP server). For
the ones that were greater than 50000 bytes in size, I was asked (1) did
I want them downloaded (I answered no), (2) did I want them deleted from
the "maildrop" (I answered yes--a bad answer).

In summary, the messages <50K are still on the IMAP server; the ones

Post by Uday Reddy
50K are not. The messages <50K are also in my file "~/INBOX"; the

messages >50K are not.

[Uday Reddy]

Post by Alan Wehmann
For me there is a mode of operation where I like to download selected
IMAP messages into Local Folders & have a local copy. As I indicated
in one of my messages, I need to use the web interface to the IMAP
server to clean out the IMAP server inbox. I would use the web
interface for this, since then the limited DSL upload speed is not a
factor when moving messages from one folder to another on the server.

If you are really sure that you wan to use Local Folders (I still don't see
why), then please be aware that you are using functionality that Thunderbird
doesn't provide. So, you can't expect VM to behave like Thunderbird. As I
said previously, I would prefer to get rid of Local Folders for IMAP because
they are confusing and don't work well.

Post by Alan Wehmann
On the other hand, the vm-get-new-mail was invoked automatically. So, VM
should not have asked you yes/no questions. I will check why that is
happening.

It looks like this is happening because we have changed the automatic
`vm-get-new-mail' to be regarded as an "interactive" use so that people can
type in passwords when they do `M-x vm'. So, it now asks yes/no questions
during `M-x vm' as well.

I guess I will need to add another flag to say that it is ok to ask for
password, but not ok to ask yes/no questions.

Cheers,
Uday

[Alan Wehmann]

I reduced the number of messages in my IMAP server inbox to six, using
the server's web interface. I then started up Thunderbird and checked
the Options setting that synchronizes such messages to a file on my
computer. The result is a file on my local computer that looks as
follows (from a Dired buffer):

c:/Documents and Settings/Alan/Application Data/Thunderbird/Profiles/qn5qd3de.default/ImapMail/email.fnal.gov:
-rw-rw-rw- 1 Alan None 43765 04-07 13:43 INBOX

From
Received: from gateway01.fnal.gov (131.225.104.18) by MAIL05V-CAS02.fnal.gov
(131.225.199.7) with Microsoft SMTP Server id 14.1.355.2; Wed, 7 Mar 2012
10:04:17 -0600
Received: from localhost (localhost.localdomain [127.0.0.1]) by
gateway01.fnal.gov (Postfix) with ESMTP id 5BAE5D80F06; Wed, 7 Mar 2012
10:04:17 -0600 (CST)
X-Virus-Scanned: amavisd-new at fnal.gov
Received: from gateway01.fnal.gov ([127.0.0.1]) by localhost
(gateway01.fnal.gov [127.0.0.1]) (amavisd-new, port 10024) with LMTP id
8NhI+x9lwUmj; Wed, 7 Mar 2012 10:04:17 -0600 (CST)
X-Mailgw-Auth: no
Received: from [127.0.0.1] (adhq107679-lt.dhcp.fnal.gov [131.225.169.179]) by
gateway01.fnal.gov (Postfix) with ESMTP id 786C6D80C93; Wed, 7 Mar 2012
10:04:16 -0600 (CST)
Date: Wed, 7 Mar 2012 10:04:14 -0600

One reason to show how it starts off is to illustrate that the first
line is not recognized by the value of
"vm-leading-message-separator-regexp-From_" inside the function
"vm-skip-past-folder-header", so this file cannot be parsed by
function "vm-build-message-list" (I had hoped to show a Summary buffer
for this file).

I suppose that such files are also built when Thunderbird is made to
go "offline", but that is somewhat off-topic.

In the place of a Summary buffer, I can show the following (from the

5 matches for "^Subject:" in buffer: INBOX
57:Subject: Next quarterly lunch Wednesday, April 11 [retired scientist list]
127:Subject: LIBRARY - You borrowed this book
208:Subject: Next quarterly lunch Wednesday, April 11 [retired scientist list]
365:Subject: Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26)
551:Subject: Type 3X Faster with Your Voice - 50% Off Dragon

(for a later comparison with the Summary buffer from
"~/mail_rmail/imap-cache-d0925bcceba88ad7703984efcad1f482")

I had a size limit of 50K imposed & one of the six messages in my IMAP
inbox was not therefore not displayed when I went "offline" in Thunderbird.

In VM I made/have the settings illustrated below (copied from the *scratch*

(format "%s\n%s\n %s\n%s\n%s\n"
vm-primary-inbox
vm-imap-account-alist
vm-enable-external-messages
vm-imap-max-message-size
vm-imap-sync-on-get)
"imap:localhost:143:inbox:login:wehmann:*
((imap:localhost:143:*:login:wehmann:* fermi_email))
(imap)
50000
nil
"

so that I can use VM to access the IMAP server in the manner that you
favor.

I used the command "vm-visit-imap-folder" and answered
"fermi_email:inbox". I also used "vm-get-new-mail", as needed
(e.g. to get asked the password). The Summary buffer that resulted I

-> 1 U Dick Carrigan Mar 7 37/5605 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
3 Dick Carrigan Apr 6 91/7806 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
4 Mark Messier Apr 7 137/10825 "Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26)"
5 U WinZip Computing Apr 5 293/17469 "Type 3X Faster with Your Voice - 50% Off Dragon"
6 Fermilab Today Apr 6 0/113632 "Fermilab Today - Friday, April 6"

Message #6 is the one that is greater than 50K (the summary buffer
indicates that its size is 113632 bytes).

For message #6 there is no indication that it was left on the server

Retrieving message 1 (of 1) from fermi_email:inbox, 69%...
inbox: Decoding MIME message...
inbox: Inlining text/html by emacs-w3m...
WARNING: Could not load feature w3m.
WARNING: Related functions may not work correctly!
inbox: Inline text/html by emacs-w3m display failed: Symbol's value as variable is void: w3m-cid-retrieve-function-alist
inbox: Recreating summary... done
inbox: Decoding MIME message... done

(Clearly something is wrong with my setup of w3m.)

I was not asked any question about whether or not I wished to download
message #6, nor was I asked any question about whether or not I wished
to delete it from the "maildrop".

Cheers,
Alan

Subject: Re: [VM] Info node "External Messages"
Date: Fri, 6 Apr 2012 23:52:55 +0100

Post by Alan Wehmann
For me there is a mode of operation where I like to download selected
IMAP messages into Local Folders & have a local copy. As I indicated
in one of my messages, I need to use the web interface to the IMAP
server to clean out the IMAP server inbox. I would use the web
interface for this, since then the limited DSL upload speed is not a
factor when moving messages from one folder to another on the server.

If you are really sure that you wan to use Local Folders (I still don't see
why), then please be aware that you are using functionality that Thunderbird
doesn't provide. So, you can't expect VM to behave like Thunderbird. As I
said previously, I would prefer to get rid of Local Folders for IMAP because
they are confusing and don't work well.

Post by Alan Wehmann
On the other hand, the vm-get-new-mail was invoked automatically. So, VM
should not have asked you yes/no questions. I will check why that is
happening.

It looks like this is happening because we have changed the automatic
`vm-get-new-mail' to be regarded as an "interactive" use so that people can
type in passwords when they do `M-x vm'. So, it now asks yes/no questions
during `M-x vm' as well.
I guess I will need to add another flag to say that it is ok to ask for
password, but not ok to ask yes/no questions.
Cheers,
Uday

[Uday Reddy]

Post by Alan Wehmann
One reason to show how it starts off is to illustrate that the first
line is not recognized by the value of
"vm-leading-message-separator-regexp-From_"

You can't read Thunderbird's IMAP folders in VM, only the Local Folders.
(In fact, you can see from the size of the file that it is not a full folder
anyway.)

Post by Alan Wehmann

6 Fermilab Today Apr 6 0/113632 "Fermilab Today - Friday, April 6"

For message #6 there is no indication that it was left on the server
because of being over-sized.

The "0" in the size indicator (presumably 0 lines?) tells you that the
message wasn't downloaded. What happens if you try to read the message 6?

Post by Alan Wehmann
I was not asked any question about whether or not I wished to download
message #6, nor was I asked any question about whether or not I wished
to delete it from the "maildrop".

Indeed, that is how it is supposed to work. It is designed to be painless.

If you delete message 6, expunge and save the folder, then message 6 will
get deleted on the server.

Cheers,
Uday

[Alan Wehmann]

I'll continue in evidence-based mode and take a slightly different
tack. In Thunderbird I removed the size limitation of 50K for
synchronization. Then the Thunderbird synchronization file has a

-rw-rw-rw- 1 Alan None 162119 04-07 20:56 INBOX

It has grown in size to 162119. Prior to doing this, my previous
email (Date: Sat, 7 Apr 2012 14:54:11 -0500) had reported for that

-rw-rw-rw- 1 Alan None 43765 04-07 13:43 INBOX

I can make VM parse the "INBOX" file (the Thunderbird synchronization
file) by doing two things:

1) Evaluate '(setq vm-leading-message-separator-regexp-From_ "^From[^:]")'

2) Make sure that the "From" line that starts each message is preceded by a
blank line

-> 1 N Dick Carrigan Mar 7 36/1573 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
3 N Dick Carrigan Apr 6 91/3729 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
4 N Mark Messier Apr 7 137/6457 "Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26)"
5 N WinZip Computing Apr 5 293/15111 "Type 3X Faster with Your Voice - 50% Off Dragon"
6 N Fermilab Today Apr 6 1811/113630 "Fermilab Today - Friday, April 6"

The entry in the buffer generated by the command "buffer-menu" for

%* INBOX 162922 VM c:/Documents and Settings/Alan/Application Data/Thunderbird/Profiles/qn5qd3de.default/ImapMail/email.fnal.gov/INBOX
1 U Dick Carrigan Mar 7 37/5605 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
3 Dick Carrigan Apr 6 91/7806 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
-> 4 Mark Messier Apr 7 137/10825 "Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26)"
5 U WinZip Computing Apr 5 293/17469 "Type 3X Faster with Your Voice - 50% Off Dragon"
6 Fermilab Today Apr 6 0/113632 "Fermilab Today - Friday, April 6"
% inbox 49639 VM ~/mail_rmail/imap-cache-d0925bcceba88ad7703984efcad1f482

I can do "compare-windows" (with a prefix, to ignore whitespace
differences) and compare the presentation buffers in each case. They
compare equal for the first four messages. For the fifth message the

Subject: Type 3X Faster with Your Voice - 50% Off Dragon
Date: Thu, 5 Apr 2012 12:23:07 -0400

in each case. In Thunderbird, for that message, much more is shown.

INBOX: Decoding MIME message...
INBOX: Inlining text/html by emacs-w3m...
WARNING: Could not load feature w3m.
WARNING: Related functions may not work correctly!
INBOX: Inline text/html by emacs-w3m display failed: Symbol's value as variable is void: w3m-cid-retrieve-function-alist
INBOX: Decoding MIME message... done

For the sixth message, for "inbox", I was asked for the password for

Warning: No user-definable flags available for the IMAP mailbox
Retrieving message 1 (of 1) from fermi_email:inbox, 38%...
Retrieving message 1 (of 1) from fermi_email:inbox, 41%...
Retrieving message 1 (of 1) from fermi_email:inbox, 65%...
inbox: Decoding MIME message...
inbox: Inlining text/html by emacs-w3m...
WARNING: Could not load feature w3m.
WARNING: Related functions may not work correctly!
inbox: Inline text/html by emacs-w3m display failed: Symbol's value as variable is void: w3m-cid-retrieve-function-alist
inbox: Recreating summary... done
inbox: Decoding MIME message... done
-> 6 Fermilab Today Apr 6 1812/113632 "Fermilab Today - Friday, April 6"

This time around the relevant VM variables are (again, from the

(format "%s\n%s\n %s\n%s\n%s\n"
vm-primary-inbox
vm-imap-account-alist
vm-enable-external-messages
vm-imap-max-message-size
vm-imap-sync-on-get)
"~/INBOX
((imap:localhost:143:*:login:wehmann:* fermi_email))
(imap)
50000
nil
"

One can note that this time "vm-primary-inbox" is different from what
I reported in my email earlier today, (Date: Sat, 7 Apr 2012 14:54:11
-0500). At that time I had customized "vm-primary-inbox" for just
that VM session and now it is back to the saved customization value.
In this case "~/INBOX" should not be confused with "INBOX" above, since
they are in different directories. File "~/INBOX" is what I have been
using with a Yahoo POP server. It was also used in a VM session on
April 4, when I was using the IMAP inbox as a spool file.

As for the size numbers being different on the Summary buffers for
"INBOX" and "inbox", that I assume is because "inbox" has attributes
saved in it and "INBOX" does not.

As a further test, I added another large message to the IMAP inbox

-rw-rw-rw- 1 Alan None 387242 04-07 22:32 INBOX
-> 1 N Dick Carrigan Mar 7 36/1573 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
3 N Dick Carrigan Apr 6 91/3729 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
4 N Mark Messier Apr 7 137/6457 "Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26)"
5 N WinZip Computing Apr 5 293/15111 "Type 3X Faster with Your Voice - 50% Off Dragon"
6 N Fermilab Today Apr 6 1811/113631 "Fermilab Today - Friday, April 6"
7 Fermilab Today Apr 5 3123/218957 "Fermilab Today - Thursday, April 5"

(after making sure that there was a blank line ahead of each "From"
line--as before).

In the "inbox" buffer I did the command "vm-get-new-mail". After a

-> 1 U Dick Carrigan Mar 7 37/5605 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
3 Dick Carrigan Apr 6 91/7806 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
4 Mark Messier Apr 7 137/10825 "Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26)"
5 WinZip Computing Apr 5 293/17469 "Type 3X Faster with Your Voice - 50% Off Dragon"
6 Fermilab Today Apr 6 1812/113632 "Fermilab Today - Friday, April 6"
7 Fermilab Today Apr 5 0/225037 "Fermilab Today - Thursday, April 5"

when I stepped to message 7 in the Summary buffer, more was fetched

1 U Dick Carrigan Mar 7 37/5605 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
3 Dick Carrigan Apr 6 91/7806 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
4 Mark Messier Apr 7 137/10825 "Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26)"
5 WinZip Computing Apr 5 293/17469 "Type 3X Faster with Your Voice - 50% Off Dragon"
6 Fermilab Today Apr 6 1812/113632 "Fermilab Today - Friday, April 6"
-> 7 Fermilab Today Apr 5 3124/218959 "Fermilab Today - Thursday, April 5"

I can emphasize the following conclusions from this evidence:

1) Thunderbird is synchronizing a file on my computer's hard drive,
matching the contents of the IMAP inbox folder.

2) The setting of 50000 for "vm-imap-max-message-size" is working.
Messages in excess of that size are having only their headers
downloaded, when "vm-get-new-mail" is used. The full message is
downloaded if I choose it in the Summary buffer.

3) It is possible to have VM parse the Thunderbird synchronization
file (local on my hard drive), but it involves a bit of trickery.

My experience on April 4 (when I was using the IMAP inbox folder as a
spool file), of being asked a) if I wanted to download each large file
(answer no), and then being asked b) if I wanted to delete it from the
"maildrop" (answer yes . . . bad choice, bad choice, bad choice) has
me wondering about the design of those questions. It seems to me
rather odd to ask me if I want to delete a large message if I haven't
even looked at it (with VM).

Cheers,
Alan

Subject: Re: [VM] Info node "External Messages"
Date: Sat, 7 Apr 2012 21:26:22 +0100

Post by Alan Wehmann
One reason to show how it starts off is to illustrate that the first
line is not recognized by the value of
"vm-leading-message-separator-regexp-From_"

You can't read Thunderbird's IMAP folders in VM, only the Local Folders.
(In fact, you can see from the size of the file that it is not a full folder
anyway.)

Post by Alan Wehmann

6 Fermilab Today Apr 6 0/113632 "Fermilab Today - Friday, April 6"

For message #6 there is no indication that it was left on the server
because of being over-sized.

The "0" in the size indicator (presumably 0 lines?) tells you that the
message wasn't downloaded. What happens if you try to read the message 6?

Post by Alan Wehmann
I was not asked any question about whether or not I wished to download
message #6, nor was I asked any question about whether or not I wished
to delete it from the "maildrop".

Indeed, that is how it is supposed to work. It is designed to be painless.
If you delete message 6, expunge and save the folder, then message 6 will
get deleted on the server.
Cheers,
Uday

[Uday Reddy]

Post by Alan Wehmann
1) Thunderbird is synchronizing a file on my computer's hard drive,
matching the contents of the IMAP inbox folder.

Yes, indeed, it does! VM's IMAP folders work the same way too. But what
shape the cache folders are in, before you force a full synchronization in
some way, are internal decisions for Thunderbird as well as VM. They are
not meant to be portable across different mail tools.

(Note that Thunderbird also keeps essential information about the folder in
.msf files. The format of those files is again internal to Thunderbird. If
you make any changes to a Thunderbird folder inside VM, VM will delete the
.msf file. For Local Folders, that is perfectly fine. Thunderbird can
reconstruct the .msf file from the folder itself. For IMAP cache folders,
deleting the .msf file can result in loss of information for Thunderbird.
So messing with Thunderbird's cache folders inside VM is not recommended.)

Post by Alan Wehmann
2) The setting of 50000 for "vm-imap-max-message-size" is working.
Messages in excess of that size are having only their headers
downloaded, when "vm-get-new-mail" is used. The full message is
downloaded if I choose it in the Summary buffer.

That is good.

Post by Alan Wehmann
3) It is possible to have VM parse the Thunderbird synchronization
file (local on my hard drive), but it involves a bit of trickery.

Covered in my response to (1).

Post by Alan Wehmann
My experience on April 4 (when I was using the IMAP inbox folder as a
spool file), of being asked a) if I wanted to download each large file
(answer no), and then being asked b) if I wanted to delete it from the
"maildrop" (answer yes . . . bad choice, bad choice, bad choice) has
me wondering about the design of those questions. It seems to me
rather odd to ask me if I want to delete a large message if I haven't
even looked at it (with VM).

That was Kyle Jones's design. I guess the idea must have been that, if you
don't want to download a message from the server due to its size, then it
would be sitting on the server for ever and ever, filling up disk space.
So, there must be a way to get rid of it. Looking through the CHANGES file,
this has been there probably since VM 6.14 (February 1997), first for POP
then adapted to IMAP (July 1998). In those days, mail servers were meant
for delivering email, and you were expected to download the email as quickly
as possible and delete it from the server. So, it made sense.

POP folders were implemented in December 2001. "External messages" (headers
downloading) was planned but never done. Here is a thread on vm.info
archives which indicates the plan:

https://groups.google.com/forum/?fromgroups#!searchin/gnu.emacs.vm.info/pop$20max$20message$20size/gnu.emacs.vm.info/17qDHuWoIqc/HiuVa_FoIfgJ

(Look for Terry W.'s message on 2/21/2002 and Kyle's responses.)

Now that we have external messages implemented, that is the preferred way of
reading IMAP mail. We should regard the idea of IMAP spool files as a
historical relic that is out-of-date and to be avoided at all costs.

Cheers,
Uday

[Alan Wehmann]

What follows are some observations that I've made with the one IMAP
folder that I have been using of late, and speculations on how things
might have been done differently.

First off, I show the Summary buffer for the folder

-> 1 U Dick Carrigan Mar 7 37/5605 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
3 Dick Carrigan Apr 6 91/7806 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
4 D Mark Messier Apr 7 137/10825 "Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26)"
5 WinZip Computing Apr 5 293/17469 "Type 3X Faster with Your Voice - 50% Off Dragon"
6 Fermilab Today Apr 6 0/113632 "Fermilab Today - Friday, April 6"
7 Fermilab Today Apr 5 0/218959 "Fermilab Today - Thursday, April 5"
8 Fermilab Today Apr 4 0/133659 "Fermilab Today - Wednesday, April 4"
9 Fermilab Service Apr 8 124/9300 "Please take this survey related to Incident INC000000231795"

As a reminder, I've got "vm-imap-max-message-size" set to 50,000.

I then use function "vm-goto-message" to go to message #6 and the

1 U Dick Carrigan Mar 7 37/5605 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
3 Dick Carrigan Apr 6 91/7806 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
4 D Mark Messier Apr 7 137/10825 "Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26)"
5 WinZip Computing Apr 5 293/17469 "Type 3X Faster with Your Voice - 50% Off Dragon"
-> 6 Fermilab Today Apr 6 1812/113632 "Fermilab Today - Friday, April 6"
7 Fermilab Today Apr 5 0/218959 "Fermilab Today - Thursday, April 5"
8 Fermilab Today Apr 4 0/133659 "Fermilab Today - Wednesday, April 4"
9 Fermilab Service Apr 8 124/9300 "Please take this survey related to Incident INC000000231795"

I then do "vm-quit" and follow that with "vm-visit-imap-folder" and
re-visit "fermi_email:inbox". The Summary buffer now indicates that

-> 1 U Dick Carrigan Mar 7 37/5605 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
3 Dick Carrigan Apr 6 91/7806 "Next quarterly lunch Wednesday, April 11 [retired scientist list]"
4 D Mark Messier Apr 7 137/10825 "Fwd: LBNE Reconfiguration: webpage and workshop (April 25-26)"
5 WinZip Computing Apr 5 293/17469 "Type 3X Faster with Your Voice - 50% Off Dragon"
6 Fermilab Today Apr 6 0/113632 "Fermilab Today - Friday, April 6"
7 Fermilab Today Apr 5 0/218959 "Fermilab Today - Thursday, April 5"
8 Fermilab Today Apr 4 0/133659 "Fermilab Today - Wednesday, April 4"
9 Fermilab Service Apr 8 124/9300 "Please take this survey related to Incident INC000000231795"

It isn't hard to examine the "fermi_email:inbox" folder and see that
the headers for message #6 are present, but that the body of the
message isn't there.

This leads me to ask the speculative question of why wasn't the design
decision made to have the Presentation buffer show the header
information for a large message--without fetching the body, and then
letting the user decide whether or not to fetch the body, based upon
the header contents? Any decision the user wishes to make about
deleting the message--based upon the header contents--could be made at
this point also, rather than the interactive question that asks him
about deleting the message even if he hasn't seen the header
information (with IMAP folders, it is not clear to me the
circumstances under which he would be asked the interactive questions
about downloading a large message and if he wishes to delete it--as I
experienced when using the IMAP inbox folder as a spool file). It
seems to me that if the headers only could be presented (for a large
message), there wouldn't be any circumstances under which to ask the
interactive questions about loading the full message or about deleting
it. Does the design choice that was made have something to do with
whether or not the server has the capability to provide only the
headers of the message?

Cheers,
Alan

Subject: Re: [VM] Info node "External Messages"
Date: Sun, 8 Apr 2012 10:44:46 +0100

Post by Alan Wehmann
1) Thunderbird is synchronizing a file on my computer's hard drive,
matching the contents of the IMAP inbox folder.

Yes, indeed, it does! VM's IMAP folders work the same way too. But what
shape the cache folders are in, before you force a full synchronization in
some way, are internal decisions for Thunderbird as well as VM. They are
not meant to be portable across different mail tools.
(Note that Thunderbird also keeps essential information about the folder in
.msf files. The format of those files is again internal to Thunderbird. If
you make any changes to a Thunderbird folder inside VM, VM will delete the
.msf file. For Local Folders, that is perfectly fine. Thunderbird can
reconstruct the .msf file from the folder itself. For IMAP cache folders,
deleting the .msf file can result in loss of information for Thunderbird.
So messing with Thunderbird's cache folders inside VM is not recommended.)

Post by Alan Wehmann
2) The setting of 50000 for "vm-imap-max-message-size" is working.
Messages in excess of that size are having only their headers
downloaded, when "vm-get-new-mail" is used. The full message is
downloaded if I choose it in the Summary buffer.

That is good.

Post by Alan Wehmann
3) It is possible to have VM parse the Thunderbird synchronization
file (local on my hard drive), but it involves a bit of trickery.

Covered in my response to (1).

Post by Alan Wehmann
My experience on April 4 (when I was using the IMAP inbox folder as a
spool file), of being asked a) if I wanted to download each large file
(answer no), and then being asked b) if I wanted to delete it from the
"maildrop" (answer yes . . . bad choice, bad choice, bad choice) has
me wondering about the design of those questions. It seems to me
rather odd to ask me if I want to delete a large message if I haven't
even looked at it (with VM).

That was Kyle Jones's design. I guess the idea must have been that, if you
don't want to download a message from the server due to its size, then it
would be sitting on the server for ever and ever, filling up disk space.
So, there must be a way to get rid of it. Looking through the CHANGES file,
this has been there probably since VM 6.14 (February 1997), first for POP
then adapted to IMAP (July 1998). In those days, mail servers were meant
for delivering email, and you were expected to download the email as quickly
as possible and delete it from the server. So, it made sense.
POP folders were implemented in December 2001. "External messages" (headers
downloading) was planned but never done. Here is a thread on vm.info
https://groups.google.com/forum/?fromgroups#!searchin/gnu.emacs.vm.info/pop$20max$20message$20size/gnu.emacs.vm.info/17qDHuWoIqc/HiuVa_FoIfgJ
(Look for Terry W.'s message on 2/21/2002 and Kyle's responses.)
Now that we have external messages implemented, that is the preferred way of
reading IMAP mail. We should regard the idea of IMAP spool files as a
historical relic that is out-of-date and to be avoided at all costs.
Cheers,
Uday
.

[Uday Reddy]

Post by Alan Wehmann
This leads me to ask the speculative question of why wasn't the design
decision made to have the Presentation buffer show the header
information for a large message--without fetching the body, and then
letting the user decide whether or not to fetch the body, based upon
the header contents?

The original motivation for "external messages" mechanism (also called
"headers-only messages" during their inception) was to allow VM folders to
be small. Some people get megabytes to gigabytes of email on a monthly
basis, and keeping all of it in an Emacs buffer is impractical. Some people
also use gmail with practically infinite mail folders and downloading all of
it into an Emacs buffer is also impractical.

Reducing the *initial* download-time was also a motivation, because Emacs is
basically locked up while it is downloading messages. (Note that
Thunderbird doesn't have the second problem, because it is written in a
language with concurrent threads. We will have to wait for Emacs Lisp to
develop concurrent threads to implement something similar for VM.)

I knew that Rob F had this on his To-Do list, and it now appears that Kyle
Jones had it on his To-Do list too. Neither of them got around to doing it.
It was a major task to implement them, because it required a careful
re-engineering of almost all of VM.

Avoiding the downloading of *individual messages* wasn't particularly on my
mind. The wait-time there is not significant, at least in my experience,
and I also think it is quite rare for people to want to delete a message
without even looking at it. But it is on the To-Do list and I might spend
some time on it depending on how important it turns out to be. (Note that
Thunderbird doesn't have such a feature. It downloads a message as soon as
you want to view it.)

It could be more important to avoid downloading attachments. That way, you
can download the basic text of the message and decide to delete it. This is
also on the To-Do list, but it requires significant amount of new coding to
query and parse the MIME structure of messages from the IMAP server. So, it
will only get done when I get a large block of free time.

Post by Alan Wehmann
Any decision the user wishes to make about
deleting the message--based upon the header contents--could be made at
this point also, rather than the interactive question that asks him
about deleting the message even if he hasn't seen the header
information (with IMAP folders, it is not clear to me the
circumstances under which he would be asked the interactive questions
about downloading a large message and if he wishes to delete it--as I
experienced when using the IMAP inbox folder as a spool file).

There are no interactive questions with IMAP folders.

It is possible to delete the message without viewing it, but it requires
some planning on the part of the user. In typical usage, the users will
tend to view the message before deleting it, and that requires downloading
the message. (The same answer as to the previous question.)

Post by Alan Wehmann
it. Does the design choice that was made have something to do with
whether or not the server has the capability to provide only the
headers of the message?

No, the server capability is not an issue. All IMAP and POP servers have
the capability to provide the headers if we ask for them. The issue is more
of how to organize the internals of VM so that it can deal with a message
without the body in the first instance, then download the body, and
recalculate everything. (There are some dark corners in VM here in terms of
how VM interacts with the Emacs display engine. I don't fully understand
how it works and doing it wrong can cause "jumping cursor" problems. Rob
F's 8.1.x branch had many such problems, and I applied some "vodoo" here and
there to make them go away without really understanding why they were
happening. See revisions 565, 544.1.22, 544.1.31, 647, 969.1.2 in the
trunk. So, some things which appear to be simple on the surface actually
turn out to be hard.)

Cheers,
Uday

[Uday Reddy]

Post by Alan Wehmann
I'll continue in evidence-based mode and take a slightly different
tack. In Thunderbird I removed the size limitation of 50K for
synchronization. Then the Thunderbird synchronization file has a

I guess you are a "natural scientist" of some form. Unfortunately, software
is not part of nature, it is man-made. Decisions in software should be made
based on what is defined and documented, not how things appear. So,
"evidence" is practically useless!

Cheers,
Uday

[m***@googlemail.com]

Post by Uday Reddy
I guess you are a "natural scientist" of some form. Unfortunately, software
is not part of nature, it is man-made. Decisions in software should be made
based on what is defined and documented, not how things appear.

Sorry to intervene, but even though that is correct

Post by Uday Reddy
So,
"evidence" is practically useless!

this is not a universal conclusion. If it were, then reverse engineering
closed source software would be impossible, because there you do exaxtly that:
You try to look for evidence (either by sniffing, disassembling, etc.) that
the software does a certain thing.

Even for well documented software the conclusion only applies if what
you want to to with it is part of what is documented, which is not always
the case.

Granted, in the context of this discussion (viewmail/thunderbird) the conclusion
does hold but only because both are well documented and what he wants to do
is covered in it.

Bye,

Moritz

[Uday Reddy]

Post by m***@googlemail.com

Post by Uday Reddy
So,
"evidence" is practically useless!

this is not a universal conclusion. If it were, then reverse engineering
You try to look for evidence (either by sniffing, disassembling, etc.) that
the software does a certain thing.

We are going off-topic, so I won't belabour this.

Reverse-engineering works only as long as the original developers stick to a
particular behaviour. If the behaviour is not documented and published,
essentially being made into a contract, the developers can change the
behaviour at any time and break interoperability. That was the foundation
of the celebrated EU vs. Microsoft case. Even though Microsoft made its
source code available, EU said that it wasn't enough. The specifications
needed to be published. (One wonders whether Microsoft had any
specifications at all in order to divulge them. They learnt the hard way
that they had to write specifications, if only to help their own
competition!)

Coming to the issue at hand, I doubt if Thunderbird developers are any
better at writing specifications than the Microsoft developers. The best
you will get is stuff like this https://developer.mozilla.org/En/Folders.
The reason we are able to inter-operate with Thunderbird is not that
Thunderbird is well-documented, but rather that they have chosen to base
their folder format on a standard RFC, the same one we use. (RFC 4155).
Nothing stops Thunderbird from deviating from the RFC, but in all
likelihood, they won't.

On the other hand, for IMAP cache folders, the very first line is a
violation of the RFC. So, in a way, they are warning us that this is not an
RFC-client folder and we shouldn't mess with it.

Cheers,
Uday

Post by m***@googlemail.com
Even for well documented software the conclusion only applies if what
you want to to with it is part of what is documented, which is not always
the case.
Granted, in the context of this discussion (viewmail/thunderbird) the conclusion
does hold but only because both are well documented and what he wants to do
is covered in it.
Bye,
Moritz

[Uday Reddy]

Post by Alan Wehmann
IMAP server at work. Since I am used to how Thunderbird implements
its size limitations I was surprised to find myself being asked yes or
no for each over-sized message.

The doc-string for `vm-imap-max-message-size' says that the yes/no question
is asked only when `vm-get-new-mail' is invoked interactively. It is not
supposed to be asked if you did `vm-visit-folder' and had
`vm-auto-get-new-mail' set to t.

Did that work correctly?

Cheers,
Uday