Re: Opening large files?

Ken Takata wrote:

> Hi Bram,
>
> 2014/5/10 Sat 20:23:47 UTC+9 Bram Moolenaar wrote:
> > Ken Takata wrote:
> >
> > > 2014/4/29 Tue 22:04:23 UTC+9 Bram Moolenaar wrote:
> > > > Ken Takata wrote:
> > > >
> > > > > 2014/4/17 Thu 0:58:01 UTC+9 Adrian wrote:
> > > > > > 'm helping another user open a large, 3Gb, file. The standard windows
> > > > > > editors balk, so I recommended VIM. Unfortunately, even vim crashes
> > > > > > after scrolling some amount. For instance, he can't go straight to
> > > > > > the end of file.
> > > > > >
> > > > > > The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
> > > > > > any settings to modify to make vim more stable with large files, or is
> > > > > > there some Windows performance limitation and just out of luck?
> > > > >
> > > > > There is a related item in the todo.txt:
> > > > >
> > > > > | Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
> > > > > | pointer in long and seek offset in 64 bit var.
> > > > >
> > > > > I wrote some patches to fix this, but they seem to be still unstable.
> > > > >
> > > > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/support-largefiles-on-windows.patch?at=default
> > > > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/use-stat_T.patch?at=default
> > > >
> > > > Did you make progress on this?
> > >
> > > I wrote "still unstable", but it seems that it was my mistake.
> > > Now I think that the patches are OK.
> > >
> > > Sometimes Vim (without the patches) freezes when I open a very big file
> > > (about 2 GB) and scroll up and down using scroll bar. After applying the
> > > patches, Vim sometimes takes very long time for scrolling up and down, but
> > > it wasn't a freeze. (I misunderstood that.)
> > >
> > > BTW, I found that 32-bit Vim couldn't handle a very big file properly when
> > > ":set noswapfile". In my understanding, this is an expected(?) behavior
> > > because Vim tries to load the whole file into the memory when 'swapfile' is
> > > off, and a 32-bit program can't allocate larger than 2-GiB memory.
> > > (Actually, a 32-bit program can get 3-GiB user space if /LARGEADDRESSAWARE
> > > option is specified for 'link'.)
> > >
> > >
> > > > Can we also add some tests to verify the fix?
> > >
> > > I'm thinking what is the best way to test this.
> > > Something like this?
> > >
> > > " Make sure that a line break is 1 byte.
> > > :set ff=unix
> > > :set undolevels=-1
> > > " Input 99 'A's. The line becomes 100 bytes including a line break.
> > > 99iA<Esc>
> > > yy
> > > " Put 19,999,999 times. The file becomes 2,000,000,000 bytes.
> > > 19999999p
> > > " Moving around in the file randomly.
> > > G
> > > 10%
> > > 90%
> > > 50%
> > > gg
> > > ...
> > > " Edit some lines.
> > > ...
> > > " Extract some lines and write them to test.out.
> > > ...
> >
> > Although it would be good to test this way, I think it should not be
> > part of "make test", since it will probably fail on smaller systems.
> > Perhaps we should make a "make bigtest" target, for more testing.
>
> I agree.
>
>
> > Generally, I think we need to test that the patch doesn't break anything
> > for normal editing. But perhaps the tests we already have are
> > sufficient for that. If you look at your patch, are there any commands
> > that would not be used by the existing tests?
>
> I think that the existing tests cover almost all part, but maybe they
> doesn't cover the following functions, commands and features:
> * getfsize(), getfperm(), getftime(), getftype()
> * :checktime
> * +cscope, +netbeans_intg
>
> BTW, I have updated the patch:
> https://bitbucket.org/k_takata/vim-ktakata-mq/src/1ded06658094d26a5eef3ffe42e5edc94d93f59c/support-largefiles-on-windows.patch?at=default (same as before)
> https://bitbucket.org/k_takata/vim-ktakata-mq/src/1ded06658094d26a5eef3ffe42e5edc94d93f59c/use-stat_T.patch?at=default
> * Define HAVE_STAT64 in vim.h and use it in os_mswin.c.
> * There was no need to use stat_T in os_unix.c. stat_T should be used with
> mch_stat().

So is the patch now ready to be included, or did you still have a
problem to fix?

--
hundred-and-one symptoms of being an internet addict:
142. You dream about creating the world's greatest web site.

/// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.