Re: Opening large files?

Hi Bram,

2014/5/10 Sat 20:23:47 UTC+9 Bram Moolenaar wrote:
> Ken Takata wrote:
>
> > 2014/4/29 Tue 22:04:23 UTC+9 Bram Moolenaar wrote:
> > > Ken Takata wrote:
> > >
> > > > 2014/4/17 Thu 0:58:01 UTC+9 Adrian wrote:
> > > > > 'm helping another user open a large, 3Gb, file. The standard windows
> > > > > editors balk, so I recommended VIM. Unfortunately, even vim crashes
> > > > > after scrolling some amount. For instance, he can't go straight to
> > > > > the end of file.
> > > > >
> > > > > The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
> > > > > any settings to modify to make vim more stable with large files, or is
> > > > > there some Windows performance limitation and just out of luck?
> > > >
> > > > There is a related item in the todo.txt:
> > > >
> > > > | Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
> > > > | pointer in long and seek offset in 64 bit var.
> > > >
> > > > I wrote some patches to fix this, but they seem to be still unstable.
> > > >
> > > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/support-largefiles-on-windows.patch?at=default
> > > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/use-stat_T.patch?at=default
> > >
> > > Did you make progress on this?
> >
> > I wrote "still unstable", but it seems that it was my mistake.
> > Now I think that the patches are OK.
> >
> > Sometimes Vim (without the patches) freezes when I open a very big file
> > (about 2 GB) and scroll up and down using scroll bar. After applying the
> > patches, Vim sometimes takes very long time for scrolling up and down, but
> > it wasn't a freeze. (I misunderstood that.)
> >
> > BTW, I found that 32-bit Vim couldn't handle a very big file properly when
> > ":set noswapfile". In my understanding, this is an expected(?) behavior
> > because Vim tries to load the whole file into the memory when 'swapfile' is
> > off, and a 32-bit program can't allocate larger than 2-GiB memory.
> > (Actually, a 32-bit program can get 3-GiB user space if /LARGEADDRESSAWARE
> > option is specified for 'link'.)
> >
> >
> > > Can we also add some tests to verify the fix?
> >
> > I'm thinking what is the best way to test this.
> > Something like this?
> >
> > " Make sure that a line break is 1 byte.
> > :set ff=unix
> > :set undolevels=-1
> > " Input 99 'A's. The line becomes 100 bytes including a line break.
> > 99iA<Esc>
> > yy
> > " Put 19,999,999 times. The file becomes 2,000,000,000 bytes.
> > 19999999p
> > " Moving around in the file randomly.
> > G
> > 10%
> > 90%
> > 50%
> > gg
> > ...
> > " Edit some lines.
> > ...
> > " Extract some lines and write them to test.out.
> > ...
>
> Although it would be good to test this way, I think it should not be
> part of "make test", since it will probably fail on smaller systems.
> Perhaps we should make a "make bigtest" target, for more testing.

I agree.


> Generally, I think we need to test that the patch doesn't break anything
> for normal editing. But perhaps the tests we already have are
> sufficient for that. If you look at your patch, are there any commands
> that would not be used by the existing tests?

I think that the existing tests cover almost all part, but maybe they
doesn't cover the following functions, commands and features:
* getfsize(), getfperm(), getftime(), getftype()
* :checktime
* +cscope, +netbeans_intg

BTW, I have updated the patch:
https://bitbucket.org/k_takata/vim-ktakata-mq/src/1ded06658094d26a5eef3ffe42e5edc94d93f59c/support-largefiles-on-windows.patch?at=default (same as before)
https://bitbucket.org/k_takata/vim-ktakata-mq/src/1ded06658094d26a5eef3ffe42e5edc94d93f59c/use-stat_T.patch?at=default
* Define HAVE_STAT64 in vim.h and use it in os_mswin.c.
* There was no need to use stat_T in os_unix.c. stat_T should be used with
mch_stat().

Regards,
Ken Takata

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.