Re: Float printf, floor and float2nr differ

Am 12.07.2012 22:10, schrieb richard emberson:
> let p = 0.705882352941176
> let pf = p * 255
> echo pf
> echo printf("%.10f",pf)
> echo float2nr(pf)
> echo floor(pf)
>
> 180.0
> 180.0000000000
> 179
> 179.0
>
> I have a Vim script app that converts between RGB numbers
> and HSV floats. One would expect that going from RGB to
> HSV and back to RGB would result in the original RGB values
> but both floor and float2nr produce rounding errors.
> If I add 0.000000000001 to values in the HSV converter
> just prior to applying float2nr, I get no rounding errors.
>
> Of course, using printf to convert the float to a string,
> removing the decimal point character and all characters after
> it in the string, and then converting that string into a
> number also works. But, is rather heavy.

You can also try
:echo float2nr(round(pf))

There are scripts that (probably) do color conversion, e.g.
pscolor vimscript #760
csapprox vimscript #2390
Erik Falor's scripts (ColorSchemeMenuMaker vimscript #2004)
etc. ("vimscript" => www.vim.org)

--
Andy

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php