> On 4 Oct, 15:42, Ben Fritz wrote:
>>
>> You can also set fileencoding manually after a file read, so that you
>> can convert it to a different encoding when writing the file. You
>> will probably want this new encoding in your fileencodings option so
>> it can be detected,
>
> If I set fileencoding manually, I see no changes on the screen. What
> does this option exactly controls?
It controls the encoding used when writing the file.
> According to the help: "Sets the character encoding for the file of
> this buffer." But honestly, I don't get it. What does the statement
> means? I have a number of fundamental questions about the subject of
> (vim and) encoding:
>
> 1) As far as I know, there is no information stored with a text file
> about in what encoding the series of bytes makes sense as a text. An
> editor makes a guess on trying to open and display the file based on
> fist N bytes, on certain patterns, etc, but in the end is it always a
> guess, and sometimes the editor get it wrong. Is this right?
If you tell the editor to only ever consider certain encodings, you can
improve its "guess". Also, various Unicode formats support a Byte-Order
Mark (BOM). This is common with UTF-16, and discouraged with UTF-8[1].
The BOM prevents the need for guessing, but so does explicitly
specifying what character sets you want to use.
[1] http://www.unicode.org/faq/utf_bom.html#bom4
> 2) When a file is loaded from disk into vim, what does exactly happen
> with the bytes? Is there any option in vim that influences this
> process? My guess is that the editor interprets the original sequence
> of bytes (as on disk) according to the rules of some character
> encoding; for vim, this would be the value of the 'encoding' option.
> Is this correct?
Vim uses the 'fencodings' option to choose (unless explicitly given a
++enc= argument). See :help 'fencodings' for the sequence of what Vim
tries. See :help ++opt for how to use ++enc=
> 3) Based on these rules, the editor knows when to take one or two or
> more bytes to build a single *character*, and if more that one, in
> which order. From that, the editor has decided which *characters*
> (not bytes) the text contains. So for example, the sequence 1A 2B F3
> E5 66 could be interpreted as
> (1A 2B) (F3) (E5 66) according to encoding 1
> (2B 1A) (E5 F3 66) according to encoding 2
> where each () group represents a 'character' in the respective
> encoding. Thus, according to encoding 1 one would have for example:
> "small a", "capital z" and "digit 8", whereas according to encoding 2
> one would have "question mark" and "small u umlaut". Is this
> description correct?
Yes, that's roughly it.
> 4) What decides how the bytes are displayed in the screen? My
> understanding is that the font comes now into play; to each
> *character*, a glyph is provided by the font, and this is what is
> displayed on the screen. Is this description correct?
Oversimplified, but yes. In some encodings (e.g. Unicode), there are
also "combining characters"[2]. Languages that are written
right-to-left need to be laid out. For scripts that have letters whose
shapes differ depending on their context, there is also "shaping" (e.g.
Arabic[3], or Urdu[4]). Other characters might have different glyphs
depending on the locale (e.g. Simplified or Traditional Chinese
characters[5]).
Depending on whether you're using Vim or Gvim, this might be handled by
Gvim or the underlying terminal (in Vim). Most of these things aren't
well-supported by Vim, particularly in Vim proper, as Vim frequently
relies on the assumption that characters can be arranged in a grid (many
discussions on this list if interested).
[2] http://en.wikipedia.org/wiki/Combining_character
[3] Arabic: http://www.w3.org/International/tests/tests-html-css/tests-webfonts/generate?test=5
[4] Urdu: http://www.w3.org/International/tests/tests-html-css/tests-webfonts/generate?test=6
[5] http://en.wikipedia.org/wiki/Help:Multilingual_support_(East_Asian)
[*] (generally interesting tests) http://www.w3.org/International/tests/tests-html-css/list-fonts
> If yes, how can I in vim change the way vim interprets the sequence of
> bytes according to a different encoding? Is it necessary to reload the
> file?
Yes, it's necessary to reload. Once fully loaded in a buffer, the
characters are characters (not bytes).
> If I use 'set fileeconding=blah', no change is visible, whereas if
> when I use ':e ++enc=blah', the displayed glyphs do change. This is
> probably due to the fact that ':e ++enc' effectively reloads the
> sequence from disk (or rereads the original sequence of bytes from
> memory), and in doing so it resolves the bytes into characters
> according to the newly specified character encoding. On the other
> hand, 'set fileencoding=blah' does not seems to reload/reread
> anything. What is the effect of this option?
(as above: the encoding that will be written to disk)
> I have a couple of ideas, but I first like to know the answer to the
> following question.
>
> 5) What happens when I type something on the keyboard? This is a
> similar situation a reading from the disk; in the end, it about a
> sequence of bytes being inserted at some place in the file; there is
> also the need to interpret them as characters and look for glyphs on
> some font to represent them (in case the file is being displayed or
> printed). Also in this case I would expect some option in vim to
> control how the bytes sent by my keyboard are to be interpreted. Which
> are these options?
If in Gvim, it uses the underlying library functionality. If in a
terminal emulator, see:
:help mbyte-terminal
> Is it the current value of 'encoding'? Or of 'fileencoding'? or of
> 'termencoding'? And when? Only on terminals, or also in GUI? and does
> makes a difference whether I am on Win32 or on *nix? Or if I use GTK
> or not? or if I use Cygwin or not?
To oversimplify: Basically the only option that is significantly
different between OS'es (Win32 vs. *nix vs. Cygwin) is 'ff'/'ffs' (the
end-of-line format), which isn't really even discussed in your email.
The defaults for the rest ('tenc','fenc','fencs','enc') generally depend
on the locale (external to Vim -- can affect 'enc', and by association
'fencs') or being in Gvim vs. Vim (Gvim defaults 'tenc' to utf-8).
Each option's help text describes its defaults.
> 6) What happens when the file is written to disk (:w)? My guess is:
> after reading the bytes, resolved then into characters and having
> found a glyph for each character and displayed on the screen, the
> editor works exclusively 'on characters, not on bytes'. According to
> this, when writing back to disk, the editor would then
> reverse-engineer the characters into bytes according to the rules of
> some encoding option. What would be this option, 'encoding',
> 'fileencoding', something derived from 'fileencodings', what?
'fileencoding' if set, otherwise 'encoding'.
> As you see, too many basic question that cannot be answered with
> 'fileencoding: Sets the character encoding for the file of this
> buffer'.
But that's just the summary: If it's set, fileencoding does exactly
that: it sets the character encoding for the file (on disk) of this
buffer.
The text after that (in :help 'fileencoding') explains what happens if
you don't choose something explicitly. Vim tries to pick a reasonable
default. If you use 'encoding=utf-8', that default is usually what you
want. If you don't, Vim has to fall back on more heuristic approaches
(it "guesses").
--
Best,
Ben
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php
No comments:
Post a Comment