[VM] Non-ASCII ISO-8859-1 characters not shown correctly

[Yoshiharu Kohayakawa]

I'd be very interested in learning how to fix the following problem,
related to iso-8859-1 characters not showing correctly in messages
read with VM.

When I receive a message with non-ascii characters and read it with
VM, the non-ascii characters don't show properly, and I see their
octal code. A screen shot is available at

http://www.ime.usp.br/~yoshi/TMP/VM

The problem happens in presentation mode with text/plain messages.

The example message of the screen shot, filed in a folder on its
own, is also at the URL above.

I'm on a Mac OS X machine, but I believe this is a problem in
GNU/Linux also.

I have this problem with VM 8.1.2 running on both

GNU Emacs 24.2.1 (x86_64-apple-darwin, NS apple-appkit-1038.36) of
2012-08-27 on bob.porkrind.org

and

GNU Emacs 23.3.50.1 (i386-apple-darwin9.8.0, NS apple-appkit-949.54)
of 2011-10-25 on braeburn.aquamacs.org - Aquamacs Distribution 2.4

I don't have this problem with VM 7.19 running on

GNU Emacs 21.2.2 (i686-pc-linux-gnu, X toolkit) of 2006-05-09 on
galena.ime.usp.br

If I cut and paste the message into a new plain text file and save
it with the latin-1 coding system, then when I visit the file I can
see the characters correctly (such a file is at the URL above also).

One more thing: non-ascii characters are also shown in octal in my
summary buffers, when they are in the subject line of the messages.

I'd really love to know how to solve this.

Many thank in advance!

Yoshi

--
Yoshiharu Kohayakawa

[Yoshiharu Kohayakawa]

One addition: when I do a

vm-reply-include-text

the octal code stay in the included message.

Yoshi

Post by Yoshiharu Kohayakawa
I'd be very interested in learning how to fix the following problem,
related to iso-8859-1 characters not showing correctly in messages
read with VM.
When I receive a message with non-ascii characters and read it with
VM, the non-ascii characters don't show properly, and I see their
octal code. A screen shot is available at
http://www.ime.usp.br/~yoshi/TMP/VM
The problem happens in presentation mode with text/plain messages.
[...]

[Uday Reddy]

Post by Yoshiharu Kohayakawa
When I receive a message with non-ascii characters and read it with
VM, the non-ascii characters don't show properly, and I see their
octal code. A screen shot is available at
http://www.ime.usp.br/~yoshi/TMP/VM
The problem happens in presentation mode with text/plain messages.

The first thing to check would be what happens with the default settings.
Rename your .vm file to something else, restart Emacs/VM and visit the
folder. If it works, then you need to narrow down which setting in your .vm
file is problematic. If you are just upgrading from 7.19, then there might
have been quite a few changes since then. Please read through the NEWS file
for the list of all the changes and new features.

Post by Yoshiharu Kohayakawa
One more thing: non-ascii characters are also shown in octal in my
summary buffers, when they are in the subject line of the messages.

This sounds more serious. Please check the value of
`enable-multibyte-characters' in the Summary buffer. Its value should be
`t'.

You might need to worry about bad settings in your .emacs file as well.

Cheers,
Uday

[Yoshiharu Kohayakawa]

Post by Uday Reddy

Post by Yoshiharu Kohayakawa
When I receive a message with non-ascii characters and read it with
VM, the non-ascii characters don't show properly, and I see their
octal code. A screen shot is available at
http://www.ime.usp.br/~yoshi/TMP/VM
The problem happens in presentation mode with text/plain messages.

The first thing to check would be what happens with the default settings.
Rename your .vm file to something else, restart Emacs/VM and visit the
folder.

Dear Uday, this is embarrassing---this is something I didn't try.

Post by Uday Reddy
If it works,

... it does work.

Post by Uday Reddy
then you need to narrow down which setting in your .vm
file is problematic. If you are just upgrading from 7.19, then there might
have been quite a few changes since then. Please read through the NEWS file
for the list of all the changes and new features.

I'll do that---many thanks.

Post by Uday Reddy

Post by Yoshiharu Kohayakawa
One more thing: non-ascii characters are also shown in octal in my
summary buffers, when they are in the subject line of the messages.

This sounds more serious. Please check the value of
`enable-multibyte-characters' in the Summary buffer. Its value should be
`t'.

I get the following:

====
enable-multibyte-characters is a variable defined in `C source code'.
Its value is t
Local in buffer INBOX Summary; global value is t

Automatically becomes buffer-local when set in any fashion.
====

I've also posted a comment on what happens if I do

vm-reply-include-text,

namely, that the octal codes show up. This was also some .vm setting
problem, i.e., it's fixed if I don't have any .vm.

Many thanks and best wishes for 2013!

Yoshi

Post by Uday Reddy
You might need to worry about bad settings in your .emacs file as well.
Cheers,
Uday

[Yoshiharu Kohayakawa]

Post by Yoshiharu Kohayakawa
[...]

Post by Uday Reddy
The first thing to check would be what happens with the default settings.
Rename your .vm file to something else, restart Emacs/VM and visit the
folder.

Dear Uday, this is embarrassing---this is something I didn't try.

Post by Uday Reddy
If it works,

... it does work.

Post by Uday Reddy
then you need to narrow down which setting in your .vm
file is problematic.

Because of the problem of MS email clients with windows1252
charsets, the variable vm-fsfemacs-mule-p was set to nil in my .vm.

Removing this fixed the problem.

Post by Yoshiharu Kohayakawa

Post by Uday Reddy

Post by Yoshiharu Kohayakawa
One more thing: non-ascii characters are also shown in octal in my
summary buffers, when they are in the subject line of the messages.

This sounds more serious. Please check the value of
`enable-multibyte-characters' in the Summary buffer. Its value should be
`t'.

====
enable-multibyte-characters is a variable defined in `C source code'.
Its value is t
Local in buffer INBOX Summary; global value is t
Automatically becomes buffer-local when set in any fashion.
====

This subject line problem in Summary buffer is also fixed now.

However, this fix for the Summary buffer only works for plain text
messages. For a message sent in rich text mode, the subject line
shows the octal codes.

Post by Yoshiharu Kohayakawa
I've also posted a comment on what happens if I do
vm-reply-include-text,
namely, that the octal codes show up. This was also some .vm setting
problem, i.e., it's fixed if I don't have any .vm.

Leaving vm-fsfemacs-mule-p as t, I don't have this problem any
longer.

Thanks and best, Yoshi

[Uday Reddy]

Post by Yoshiharu Kohayakawa
Because of the problem of MS email clients with windows1252
charsets, the variable vm-fsfemacs-mule-p was set to nil in my .vm.

Glad that you have found the problem!

vm-fsfemacs-mule-p is an internal variable of VM. Please don't change it.

The only problem with MS email clients was that they used to incorrectly
label windows-1252 as iso-8859-1. But I haven't seen this problem for a
while. So it is likely that they fixed it. If so, nothing special is
needed for handling windows-1252 any more.

Post by Yoshiharu Kohayakawa
This subject line problem in Summary buffer is also fixed now.
However, this fix for the Summary buffer only works for plain text
messages. For a message sent in rich text mode, the subject line
shows the octal codes.

It might be a problem with cached-data. If so, typing `j'
(`vm-discard-cached-data') should fix the problem. There are still some
outstanding problems with the cached-data handling, which will be fixed in
8.2.0 release.

If it is not a cached-data problem, please file a bug report with a sample
message.

Cheers,
Uday

[Yoshiharu Kohayakawa]

Post by Uday Reddy

Post by Yoshiharu Kohayakawa
Because of the problem of MS email clients with windows1252
charsets, the variable vm-fsfemacs-mule-p was set to nil in my .vm.

Glad that you have found the problem!

Many many thanks for the help!

Post by Uday Reddy
vm-fsfemacs-mule-p is an internal variable of VM. Please don't change it.

OK.

Post by Uday Reddy
The only problem with MS email clients was that they used to incorrectly
label windows-1252 as iso-8859-1. But I haven't seen this problem for a
while. So it is likely that they fixed it. If so, nothing special is
needed for handling windows-1252 any more.

Very good.

Post by Uday Reddy

Post by Yoshiharu Kohayakawa
This subject line problem in Summary buffer is also fixed now.
However, this fix for the Summary buffer only works for plain text
messages. For a message sent in rich text mode, the subject line
shows the octal codes.

It might be a problem with cached-data. If so, typing `j'
(`vm-discard-cached-data') should fix the problem.

Excellent---it did solve the problem (but see below).

Post by Uday Reddy
There are still some
outstanding problems with the cached-data handling, which will be fixed in
8.2.0 release.
If it is not a cached-data problem, please file a bug report with a sample
message.

Discarding the cache fixed all subject lines but one. This is a message in
utf-8, which I put in the file octal_subject_message at

http://www.ime.usp.br/~yoshi/TMP/VM/

I've tried looking at this message with Emacs/VM with default settings, with
basically the same result. The behaviour in my two emacses is a little
different, though: one of them shows octal code and the other shows the
corresponding iso-8895-1 characters.

Thanks and best, Yoshi

Post by Uday Reddy
Cheers,
Uday

[Uday Reddy]

Post by Yoshiharu Kohayakawa
Discarding the cache fixed all subject lines but one. This is a message in
utf-8, which I put in the file octal_subject_message at
http://www.ime.usp.br/~yoshi/TMP/VM/

This message is not in correct MIME format. Any non-ASCIi characters in the
message headers are required to be MIME-encoded. Here, they are not. VM's
treatment of it is therefore correct and the best possible under the
circumstances.

Cheers,
Uday

[Yoshiharu Kohayakawa]

Post by Uday Reddy

Post by Yoshiharu Kohayakawa
Discarding the cache fixed all subject lines but one. This is a message in
utf-8, which I put in the file octal_subject_message at
http://www.ime.usp.br/~yoshi/TMP/VM/

This message is not in correct MIME format. Any non-ASCIi characters in the
message headers are required to be MIME-encoded. Here, they are not. VM's
treatment of it is therefore correct and the best possible under the
circumstances.

Very good. So I have no bug to report! Best wishes, Yoshi

Post by Uday Reddy
Cheers,
Uday