Re: ex large file

Dominique Pellé wrote:

> Enda wrote:
>
>> 'echo "" > filetowipe' would be a new file, I want to edit the same file.
>>
>> I don't want to crash my desktop again. I think the fact that ex creates
>> .swp and .swo files might contribute to the problem.
>>
>> - Enda
>
>
> Somehow %d (same 1,$d) is using a lot of memory
> with big files.
>
> This is what I did:
>
> 1) Create a 150 Mb file (10 M lines) with:
> $ yes 'this is a test' | sed 10000000q > test.txt
>
> 2) Start vim with: vim -u NONE test.txt
> On my PC, this is using 202 Mb of virtual memory
>
> 3) :%d
> Virtual memory usage jumps to 729 Mb
>
> My suspicion was that undo needs lots of memory.
> Doing the same steps with :set ul=1 | %d
> hence disabling undo saves a lot of memory but
> memory usage still jumps from 202 Mb to 462Mb
> when replace step 3) with:
>
> :set ul=-1 | %d
>
> I wonder what uses so much memory when undo
> is disabled.
>
> I also profiled memory usage with...
>
> $ valgrind --tool=massif ./vim -u NONE
>
> ... and it gives the following usage at the peak of memory use
> (using a 75 Mb test.txt file):
>
> --------------------------------------------------------------------------------
> n time(i) total(B) useful-heap(B) extra-heap(B) stacks(B)
>
> 21 6,382,232,109 237,533,520 192,136,937 45,396,583 0
> 80.89% (192,136,937B) (heap allocation functions) malloc/new/new[],
> --alloc-fns, etc.
> ->80.79% (191,902,967B) 0x810FE7B: lalloc (misc2.c:929)
> | ->72.35% (171,861,290B) 0x810FD98: alloc (misc2.c:828)
> | | ->40.45% (96,088,064B) 0x81F6471: mf_alloc_bhdr (memfile.c:951)
> | | | ->40.45% (96,088,064B) 0x81F5B1B: mf_new (memfile.c:392)
> | | | ->40.29% (95,703,040B) 0x80F1F14: ml_new_data (memline.c:3503)
> | | | | ->40.29% (95,698,944B) 0x80F08C6: ml_append_int (memline.c:2751)
> | | | | | ->40.29% (95,698,944B) 0x80F03E6: ml_append (memline.c:2525)
> | | | | | ->40.29% (95,698,944B) 0x80C4A2D: readfile (fileio.c:2219)
> | | | | | ->40.29% (95,698,944B) 0x804E0D8: open_buffer (buffer.c:140)
> | | | | | ->40.29% (95,698,944B) 0x81EF6DB: create_windows (main.c:2632)
> | | | | | ->40.29% (95,698,944B) 0x81ED22D: main (main.c:833)
> | | | | |
> | | | | ->00.00% (4,096B) in 1+ places, all below ms_print's
> threshold (01.00%)
> | | | |
> | | | ->00.16% (385,024B) in 1+ places, all below ms_print's
> threshold (01.00%)
> | | |
> | | ->31.58% (75,001,587B) 0x8110278: vim_strsave (misc2.c:1253)
> | | | ->31.57% (74,998,440B) 0x8132825: op_yank (ops.c:2985)
> | | | | ->31.57% (74,998,440B) 0x8130043: op_delete (ops.c:1728)
> | | | | ->31.57% (74,998,440B) 0x80AF4F0: ex_operators (ex_docmd.c:8500)
> | | | | ->31.57% (74,998,440B) 0x80A68CC: do_one_cmd (ex_docmd.c:2668)
> | | | | ->31.57% (74,998,440B) 0x80A4194: do_cmdline (ex_docmd.c:1122)
> | | | | ->31.57% (74,998,440B) 0x812710A: nv_colon (normal.c:5412)
> | | | | ->31.57% (74,998,440B) 0x812070B: normal_cmd (normal.c:1193)
> | | | | ->31.57% (74,998,440B) 0x81EDBA3: main_loop (main.c:1294)
> | | | | ->31.57% (74,998,440B) 0x81ED5C0: main (main.c:998)
>
> I'll try to investigate more later.
>
> -- Dominique


Ah of course, when doing %d Vim is storing the deleted text
into the unnamed register so that's using lots of memory.

I can delete the entire file without memory increase by
disabling undo and using the black hole register "_ :

:set ul=-1
gg"_dG

-- Dominique

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php