Re: ex large file

On Monday, July 30, 2012 2:05:33 PM UTC-5, Dominique Pelle wrote:
> Dominique Pellé wrote:
>
>
>
> > Enda wrote:
>
> >
>
> >> 'echo "" > filetowipe' would be a new file, I want to edit the same file.
>
> >>
>
> >> I don't want to crash my desktop again. I think the fact that ex creates
>
> >> .swp and .swo files might contribute to the problem.
>
> >>
>
> >> - Enda
>
> >
>
> >
>
> > Somehow %d (same 1,$d) is using a lot of memory
>
> > with big files.
>
> >
>
> > This is what I did:
>
> >
>
> > 1) Create a 150 Mb file (10 M lines) with:
>
> > $ yes 'this is a test' | sed 10000000q > test.txt
>
> >
>
> > 2) Start vim with: vim -u NONE test.txt
>
> > On my PC, this is using 202 Mb of virtual memory
>
> >
>
> > 3) :%d
>
> > Virtual memory usage jumps to 729 Mb
>
> >
>
> > My suspicion was that undo needs lots of memory.
>
> > Doing the same steps with :set ul=1 | %d
>
> > hence disabling undo saves a lot of memory but
>
> > memory usage still jumps from 202 Mb to 462Mb
>
> > when replace step 3) with:
>
> >
>
> > :set ul=-1 | %d
>
> >
>
> > I wonder what uses so much memory when undo
>
> > is disabled.
>
> >
>
> > I also profiled memory usage with...
>
> >
>
> > $ valgrind --tool=massif ./vim -u NONE
>
> >
>
> > ... and it gives the following usage at the peak of memory use
>
> > (using a 75 Mb test.txt file):
>
> >
>
> > --------------------------------------------------------------------------------
>
> > n time(i) total(B) useful-heap(B) extra-heap(B) stacks(B)
>
> >
>
> > 21 6,382,232,109 237,533,520 192,136,937 45,396,583 0
>
> > 80.89% (192,136,937B) (heap allocation functions) malloc/new/new[],
>
> > --alloc-fns, etc.
>
> > ->80.79% (191,902,967B) 0x810FE7B: lalloc (misc2.c:929)
>
> > | ->72.35% (171,861,290B) 0x810FD98: alloc (misc2.c:828)
>
> > | | ->40.45% (96,088,064B) 0x81F6471: mf_alloc_bhdr (memfile.c:951)
>
> > | | | ->40.45% (96,088,064B) 0x81F5B1B: mf_new (memfile.c:392)
>
> > | | | ->40.29% (95,703,040B) 0x80F1F14: ml_new_data (memline.c:3503)
>
> > | | | | ->40.29% (95,698,944B) 0x80F08C6: ml_append_int (memline.c:2751)
>
> > | | | | | ->40.29% (95,698,944B) 0x80F03E6: ml_append (memline.c:2525)
>
> > | | | | | ->40.29% (95,698,944B) 0x80C4A2D: readfile (fileio.c:2219)
>
> > | | | | | ->40.29% (95,698,944B) 0x804E0D8: open_buffer (buffer.c:140)
>
> > | | | | | ->40.29% (95,698,944B) 0x81EF6DB: create_windows (main.c:2632)
>
> > | | | | | ->40.29% (95,698,944B) 0x81ED22D: main (main.c:833)
>
> > | | | | |
>
> > | | | | ->00.00% (4,096B) in 1+ places, all below ms_print's
>
> > threshold (01.00%)
>
> > | | | |
>
> > | | | ->00.16% (385,024B) in 1+ places, all below ms_print's
>
> > threshold (01.00%)
>
> > | | |
>
> > | | ->31.58% (75,001,587B) 0x8110278: vim_strsave (misc2.c:1253)
>
> > | | | ->31.57% (74,998,440B) 0x8132825: op_yank (ops.c:2985)
>
> > | | | | ->31.57% (74,998,440B) 0x8130043: op_delete (ops.c:1728)
>
> > | | | | ->31.57% (74,998,440B) 0x80AF4F0: ex_operators (ex_docmd.c:8500)
>
> > | | | | ->31.57% (74,998,440B) 0x80A68CC: do_one_cmd (ex_docmd.c:2668)
>
> > | | | | ->31.57% (74,998,440B) 0x80A4194: do_cmdline (ex_docmd.c:1122)
>
> > | | | | ->31.57% (74,998,440B) 0x812710A: nv_colon (normal.c:5412)
>
> > | | | | ->31.57% (74,998,440B) 0x812070B: normal_cmd (normal.c:1193)
>
> > | | | | ->31.57% (74,998,440B) 0x81EDBA3: main_loop (main.c:1294)
>
> > | | | | ->31.57% (74,998,440B) 0x81ED5C0: main (main.c:998)
>
> >
>
> > I'll try to investigate more later.
>
> >
>
> > -- Dominique
>
>
>
>
>
> Ah of course, when doing %d Vim is storing the deleted text
>
> into the unnamed register so that's using lots of memory.
>
>
>
> I can delete the entire file without memory increase by
>
> disabling undo and using the black hole register "_ :
>
>
>
> :set ul=-1
>
> gg"_dG
>
>
>

So I suppose the OP should try (if they're on a 64 bit system):

ex -s largefile -c '%d _' -c wq

If they're not on a 64-bit system, Vim can't open the file so they should instead use (from Tony):

ex -s -c 'saveas! largefile' -c q

...which will be faster and use less memory anyway, even on a 64-bit system.

Or they could just use any of the several non-Vim solutions given.

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php