Re: Limitations of vim-python interface with respect to character encodings

On Sun, Jul 18, 2010 at 3:08 AM, Ted <cecinemapasderange@gmail.com> wrote:

Thanks for that guidance, that does fix the problems, also for eg
`vim.current.range.append(u'\u2026'.encode('utf-8')`.  It's sort of
inconvenient to have to do this though, is there no way to set the
default encoding that will be used when sending text from Python to
vim?  If there is no way to do this with the Python-Vim interface,
then perhaps that is something that should be worked into a higher-
level vim module for Python.  Does such a thing exist?

Until now, i have not seen such things, maybe the vim python interface should be given some improvement. :-)
 

I'm also wondering why there is a need to set the encoding in a Vim
variable as you've done.  As I understand it (perhaps not so well...),
Vim always works with utf-8 natively, even under Windows.

I am not very clear about the relationship between vim internal encoding and python unicode string.
When i use the unicode directly in vim, something incorrect always comes out. So i found an alternate way as i said.
And it is OK now without any problem although it is not very convenient.

The vim internal encoding is set by "encoding" global variable as i know.
If you not set that, gvim "maybe" use the "utf8" as default, but i am very sure about this.
And if you set the encoding to "utf16", the vim should use the "utf16" as internal encoding when it read in a file, as i understand.
 

Cheers
-Ted

On Jul 16, 1:36 am, winterTTr <winterTTr....@gmail.com> wrote:

> On Fri, Jul 16, 2010 at 11:45 AM, Ted <cecinemapasdera...@gmail.com> wrote:
> > I've recently discovered that there are some limitations with respect
> > to using the python interface to bring non-ascii characters back into
> > vim.
>
> > For example, this command works as expected:
>
> >    :py vim.command(u'echo "\u0061"')
> >    a
>
> > However this one gives the following error message:
>
> >    :py vim.command(u'echo "\u2026"')
> >    Traceback (most recent call last):
> >      File "<string>", line 1, in <module>
> >    UnicodeEncodeError: 'ascii' codec can't encode characters in
> > position 10-11: ordinal not in range(128)
>
> > Other functions seem to be even more restricted:
>
> >    :py vim.current.range.append(u'\u0061')
> >    Traceback (most recent call last):
> >      File "<string>", line 1, in <module>
> >    TypeError: bad argument type for built-in operation
>
> > Even output from python is restricted:
>
> > :py print u'echo "\u0061"'
> > echo "a"
> > :py print u'echo "\u2026"'
> > Traceback (most recent call last):
> >  File "<string>", line 1, in <module>
> > UnicodeEncodeError: 'ascii' codec can't encode character u'\u2026' in
> > position 6: ordinal not in range(128)
>
> > I've discovered that this last issue can be remedied via
>
> >    import sys, codecs
> >    sys.stdout, _held = (codecs.getwriter('utf-8')(sys.stdout),
> > sys.stdout)
>
> > Setting python's default encoding from within /etc/python2.6/
> > sitecustomize.py has a similar effect.
>
> > These last two workarounds only fix issues with printing; issues with
> > appending to buffers are not affected by either of these changes.
>
> > Is it possible to fix this by reconfiguring something within Python,
> > or is this a limitation of the vim-python interface?
>
> Actually, for my script, if i want to use the unicode string into vim
> environment.
> # my setting " set encoding=utf8"
> I will always use the encode && decode to transfer the string into vim
> internal encoding, and then pass it to the vim function.
> for example:
>
> python << EOS
> uniStr = u"\u2026"
> str = uniStr.encode( vim.eval("&encoding") )
> print str
> EOS
>
> OR, as regards to your example, it would be :
> :py print u"\u2026".encode( vim.eval("&encoding") )
>
> and for another example of yours:
> :py vim.command(u'echo "\u2026"')
> you should not use the "u" to modify the string pass to vim, you'd better to
> use the vim internal encoding string instead of the unicode string directly,
> it will be OK if you run it like this ( just remove the "u" decorator )
> :py vim.command( 'echo "\u2026"')
>
> OR use it like i suggested:
> py vim.command(u'echo "\u2026"'.encode( vim.eval("&encoding") ) )
>
>
>
> > --
> > You received this message from the "vim_use" maillist.
> > Do not top-post! Type your reply below the text you are replying to.

> > For more information, visithttp://www.vim.org/maillist.php

--

You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php