Re: Opening large files?

Ken Takata wrote:

> 2014/4/29 Tue 22:04:23 UTC+9 Bram Moolenaar wrote:
> > Ken Takata wrote:
> >
> > > 2014/4/17 Thu 0:58:01 UTC+9 Adrian wrote:
> > > > 'm helping another user open a large, 3Gb, file. The standard windows
> > > > editors balk, so I recommended VIM. Unfortunately, even vim crashes
> > > > after scrolling some amount. For instance, he can't go straight to
> > > > the end of file.
> > > >
> > > > The work station is Windows 7, 64 bit, with 32Gb of RAM. Are there
> > > > any settings to modify to make vim more stable with large files, or is
> > > > there some Windows performance limitation and just out of luck?
> > >
> > > There is a related item in the todo.txt:
> > >
> > > | Win64: Seek error in swap file for a very big file (3 Gbyte). Check storing
> > > | pointer in long and seek offset in 64 bit var.
> > >
> > > I wrote some patches to fix this, but they seem to be still unstable.
> > >
> > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/support-largefiles-on-windows.patch?at=default
> > > https://bitbucket.org/k_takata/vim-ktakata-mq/src/192069dac4356c186b89e0451a254599713d2309/use-stat_T.patch?at=default
> >
> > Did you make progress on this?
>
> I wrote "still unstable", but it seems that it was my mistake.
> Now I think that the patches are OK.
>
> Sometimes Vim (without the patches) freezes when I open a very big file
> (about 2 GB) and scroll up and down using scroll bar. After applying the
> patches, Vim sometimes takes very long time for scrolling up and down, but
> it wasn't a freeze. (I misunderstood that.)
>
> BTW, I found that 32-bit Vim couldn't handle a very big file properly when
> ":set noswapfile". In my understanding, this is an expected(?) behavior
> because Vim tries to load the whole file into the memory when 'swapfile' is
> off, and a 32-bit program can't allocate larger than 2-GiB memory.
> (Actually, a 32-bit program can get 3-GiB user space if /LARGEADDRESSAWARE
> option is specified for 'link'.)
>
>
> > Can we also add some tests to verify the fix?
>
> I'm thinking what is the best way to test this.
> Something like this?
>
> " Make sure that a line break is 1 byte.
> :set ff=unix
> :set undolevels=-1
> " Input 99 'A's. The line becomes 100 bytes including a line break.
> 99iA<Esc>
> yy
> " Put 19,999,999 times. The file becomes 2,000,000,000 bytes.
> 19999999p
> " Moving around in the file randomly.
> G
> 10%
> 90%
> 50%
> gg
> ...
> " Edit some lines.
> ...
> " Extract some lines and write them to test.out.
> ...

Although it would be good to test this way, I think it should not be
part of "make test", since it will probably fail on smaller systems.
Perhaps we should make a "make bigtest" target, for more testing.

Generally, I think we need to test that the patch doesn't break anything
for normal editing. But perhaps the tests we already have are
sufficient for that. If you look at your patch, are there any commands
that would not be used by the existing tests?

--
"The sun oozed over the horizon, shoved aside darkness, crept along the
greensward, and, with sickly fingers, pushed through the castle window,
revealing the pillaged princess, hand at throat, crown asunder, gaping
in frenzied horror at the sated, sodden amphibian lying beside her,
disbelieving the magnitude of the frog's deception, screaming madly,
"You lied!"
- Winner of the Bulwer-Lytton contest (San Jose State University),
wherein one writes only the first line of a bad novel

/// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.