[VM] VM writes r/o folders

[Julian Bradfield]

VM is behaving badly with read-only folders.

If I visit a folder read-only, I don't expect it to be changed and
written back to disk. However, this happens:

(a) If I visit a folder created by my MTA, VM says "upgrading
attributes", and then the folder is modified, and VM writes it
back to disk when I quit. This of course causes corruption if the
MTA is writing new mail at the same time, since they use different
locking mechanisms.
(b) If I visit a virtual read-only folder, and then I change the
attributes of a message (by, say, filing it), the underlying
folder has its attributes changed and gets written when I quit the
virtual folder. I don't think this is right either.

My understanding of a read-only folder is that nothing I do to it
should cause any modification of the disk file.

[Uday Reddy]

Post by Julian Bradfield
If I visit a folder read-only, I don't expect it to be changed and
(a) If I visit a folder created by my MTA, VM says "upgrading
attributes", and then the folder is modified, and VM writes it
back to disk when I quit. This of course causes corruption if the
MTA is writing new mail at the same time, since they use different
locking mechanisms.

I don't think that that kind of semantics has ever been promised for
`vm-folder-read-only'. Actually, I don't find it documented in the manual
at all, which means to me that no promises were ever made.

I haven't taken out any read-only checks that were present in VM. The
behaviour should be the same as it always was. (Of course, some bugs here
and there are always possible, but there was no deliberate attempt to change
the behaviour here.)

If you do not want a folder changed, your best bet is to use
`vm-quit-no-change' when you are done.

We can discuss assigning a strong semantics to read-only, but I suspect that
it may be difficult to implement. "Upgrading attributes", for example,
needs to be done when an old folder is read because the new code base will
only work with the upgraded version of data. We can perhaps refrain from
writing the folder back to the disk at the end if the read-only flag is set.

Personally, I don't think you should ever be using VM on folders that an MTA
is writing to. Failsafe concurrency is only possible if you use a proper
mail server. Concurrency is the main reason I myself switched to IMAP. I
regard what I used to do in the old days as "stone age"!

Post by Julian Bradfield
(b) If I visit a virtual read-only folder, and then I change the
attributes of a message (by, say, filing it), the underlying
folder has its attributes changed and gets written when I quit the
virtual folder. I don't think this is right either.

Indeed, this wishlist item has been recorded.

https://bugs.launchpad.net/vm/+bug/907731

Please feel free to contribute a patch if you are able to.

Cheers,
Uday

[John Hein]

Post by Uday Reddy

Post by Julian Bradfield
If I visit a folder read-only, I don't expect it to be changed and
(a) If I visit a folder created by my MTA, VM says "upgrading
attributes", and then the folder is modified, and VM writes it
back to disk when I quit. This of course causes corruption if the
MTA is writing new mail at the same time, since they use different
locking mechanisms.

I don't think that that kind of semantics has ever been promised for
`vm-folder-read-only'. Actually, I don't find it documented in the manual
at all, which means to me that no promises were ever made.
I haven't taken out any read-only checks that were present in VM. The
behaviour should be the same as it always was. (Of course, some bugs here
and there are always possible, but there was no deliberate attempt to change
the behaviour here.)
If you do not want a folder changed, your best bet is to use
`vm-quit-no-change' when you are done.
We can discuss assigning a strong semantics to read-only, but I suspect that
it may be difficult to implement. "Upgrading attributes", for example,
needs to be done when an old folder is read because the new code base will
only work with the upgraded version of data. We can perhaps refrain from
writing the folder back to the disk at the end if the read-only flag is set.
Personally, I don't think you should ever be using VM on folders that an MTA
is writing to. Failsafe concurrency is only possible if you use a proper
mail server. Concurrency is the main reason I myself switched to IMAP. I
regard what I used to do in the old days as "stone age"!

I've used local mail folders that the mta writes to for a while now
(procmail to spool file read by vm). movemail seems to do a good job
- I have not experienced corruption/locking problems in this area when
using vm over the years.

Post by Uday Reddy

Post by Julian Bradfield
(b) If I visit a virtual read-only folder, and then I change the
attributes of a message (by, say, filing it), the underlying
folder has its attributes changed and gets written when I quit the
virtual folder. I don't think this is right either.

Indeed, this wishlist item has been recorded.
https://bugs.launchpad.net/vm/+bug/907731
Please feel free to contribute a patch if you are able to.

Compared with 8.1.1...

(a) automatic upgrading of attributes happen in 8.1.1, so there's
nothing to compare

(b) 8.1.1 gets this right. It doesn't add the 'filed' mark when you
file a message from within a r/o folder. And if you use
vm-set-message-attributes, it whines that the folder is r/o.
folder). So it's just a "simple matter" of finding the diff from
8.1.1 to 8.2.0 that causes the regression.


For (a), I think that updating attributes for a read-only folder could
just be kept in memory.

In fact, I think I would prefer that for read-write folders that have
old attributes. That is, update the attributes in memory and don't
mark the folder changed for just folder attribute changes. Then if
you don't make any other changes and then do vm-quit or
vm-quit-no-change, vm just quits the folder without saving the updated
attributes (and without asking you to save the [internal] folder
attributes change). If you do make "real" changes, then vm saves the
updated attributes along with those changes.

That's one thing I noticed with 8.2.0 - just visiting my old folders
immediately marks it changed and prompts me when I use
vm-quit-no-change. Now that I've gotten used to this behavior and
pondered it for a while, I would prefer the behavior I just described
(don't consider attributes changes only actual changes) if it's not
too much of a pain to implement. I can live with the current behavior
(and have been), but it's not the best option from a user experience,
IMO.

[Uday Reddy]

Post by John Hein
I've used local mail folders that the mta writes to for a while now
(procmail to spool file read by vm). movemail seems to do a good job
- I have not experienced corruption/locking problems in this area when
using vm over the years.

Oh, movemail is a different matter. It is a separate program and knows how
to inter-operate with MTA's.

Spelling it out, the MTA delivers mail to what we call "spool files".
movemail moves the mail from spool files to VM folders. Then, we operate on
VM folders.

VM wasn't designed to operate on spool files directly, which is what Julian
seems to be trying.

Post by John Hein
(b) 8.1.1 gets this right. It doesn't add the 'filed' mark when you
file a message from within a r/o folder. And if you use
vm-set-message-attributes, it whines that the folder is r/o.
folder). So it's just a "simple matter" of finding the diff from
8.1.1 to 8.2.0 that causes the regression.

I will double check. But I think the current behaviour is still the same.

Post by John Hein
In fact, I think I would prefer that for read-write folders that have
old attributes. That is, update the attributes in memory and don't
mark the folder changed for just folder attribute changes. Then if
you don't make any other changes and then do vm-quit or
vm-quit-no-change, vm just quits the folder without saving the updated
attributes (and without asking you to save the [internal] folder
attributes change). If you do make "real" changes, then vm saves the
updated attributes along with those changes.

Yes, this behaviour has changed from 8.1.1 to 8.2.0. The old code was not
marking the buffer as modified. I will reset it back to the old way.

However, note that Kyle Jones's comment in the code says this:

;; tink the message stuff flag so that if
;; the user saves we get rid of the old
;; short vector. otherwise we could be
;; dealing with these things for all
;; eternity.

Personally, I can't see what is wrong with keeping old formats "for all
eternity", but I haven't thought it through fully. I just think it is safer
to upgrade old formats to new ones in general.

Cheers,
Uday

[John Hein]

Post by Uday Reddy

Post by John Hein
(b) 8.1.1 gets this right. It doesn't add the 'filed' mark when you
file a message from within a r/o folder. And if you use
vm-set-message-attributes, it whines that the folder is r/o.
folder). So it's just a "simple matter" of finding the diff from
8.1.1 to 8.2.0 that causes the regression.

I will double check. But I think the current behaviour is still the same.

Hmm..
I tried to reproduce it to find where after 8.1.1 the behavior changed,
and now I can't produce the problem (in a new emacs/vm session). So
it was either some odd state vm got into or something I fouled up when
I tried to see the issue Julian reported (probably the latter).

Post by Uday Reddy

Post by John Hein
In fact, I think I would prefer that for read-write folders that have
old attributes. That is, update the attributes in memory and don't
mark the folder changed for just folder attribute changes. Then if
you don't make any other changes and then do vm-quit or
vm-quit-no-change, vm just quits the folder without saving the updated
attributes (and without asking you to save the [internal] folder
attributes change). If you do make "real" changes, then vm saves the
updated attributes along with those changes.

Yes, this behaviour has changed from 8.1.1 to 8.2.0. The old code
was not marking the buffer as modified. I will reset it back to
the old way.

Okay.

Post by Uday Reddy
;; tink the message stuff flag so that if
;; the user saves we get rid of the old
;; short vector. otherwise we could be
;; dealing with these things for all
;; eternity.
Personally, I can't see what is wrong with keeping old formats "for
all eternity", but I haven't thought it through fully. I just
think it is safer to upgrade old formats to new ones in general.

Nothing wrong I can think of either, except a bit more work for vm
each time the old folder is loaded or missing features.

One disadvantage to upgrading old formats to new is when you try to
load new formats in older versions of vm. But that's a separate
issue.

[Uday Reddy]

Post by John Hein
One disadvantage to upgrading old formats to new is when you try to
load new formats in older versions of vm. But that's a separate
issue.

VM version 5 onwards were designed to allow such "forward compatibility".
That is why the header line is called "VM-v5-Data". In principle, the old
versions of VM will work ok with the new formats. But, in practice, things
seem to be breaking in some light way:

https://bugs.launchpad.net/vm/+bug/916695

I intend to look at it before the 8.2.0 release goes out.

I also intend to backport the changes to 8.1.x branch to create an 8.1.2
release for those people that might need to keep with the 8.1.x versions for
some reason or other.

Cheers,
Uday

[Julian Bradfield]

Post by Uday Reddy
VM wasn't designed to operate on spool files directly, which is what Julian
seems to be trying.

I'm specifically NOT trying to "operate on" spool files. I just want
to *read* them. (The MTA makes a backup copy of incoming mail, which I
sometimes need to look at if my overenthusiastic pre-processing has
lost something, or if I accidentally delete a message.)

Post by Uday Reddy

Post by John Hein
In fact, I think I would prefer that for read-write folders that have
old attributes. That is, update the attributes in memory and don't

Yes, this behaviour has changed from 8.1.1 to 8.2.0. The old code was not
marking the buffer as modified. I will reset it back to the old way.

Good.

Post by Uday Reddy
Personally, I can't see what is wrong with keeping old formats "for all
eternity", but I haven't thought it through fully. I just think it is safer
to upgrade old formats to new ones in general.

When suitable, yes. But one shouldn't tinker with archive copies,
other than on the explicit instructions of the user.