Re: 'maxfuncdepth' ignored

On Monday, January 2, 2017 at 11:40:32 AM UTC-6, ZyX wrote:
> 2017-01-01 0:40 GMT+03:00 Brett Stahlman <brettstahlman@gmail.com>:
> > On Saturday, December 31, 2016 at 12:23:10 PM UTC-6, ZyX wrote:
> >> 2016-12-30 20:09 GMT+03:00 Brett Stahlman <brettstahlman@gmail.com>:
> >> > Consider the following recursive user function...
> >> >
> >> > fu! Fun(count)
> >> > if a:count > 0
> >> > call Fun(a:count - 1)
> >> > endif
> >> > endfu
> >> >
> >> > :h 'maxfuncdepth' describes the option's purpose as follows:
> >> >
> >> > Maximum depth of function calls for user functions. This normally
> >> > catches endless recursion. When using a recursive function with
> >> > more depth, set 'maxfuncdepth' to a bigger number.
> >> >
> >> > So I would expect to be able to make the following recursive call with no error:
> >> > :set maxfuncdepth=1000
> >> > :call Fun(500)
> >> >
> >> > But I get the following error after slightly less than 200 recursive calls:
> >> > E169: Command too recursive
> >> >
> >> > The documentation for E169 states the following:
> >> >
> >> > This happens when an Ex command executes an Ex command that executes an Ex
> >> > command, etc. This is only allowed 200 times. When it's more there probably
> >> > is an endless loop. Probably a |:execute| or |:source| command is involved.
> >> >
> >> > It's as though the :call (Ex command) is triggering the error long
> >> > before the number of calls to the user function Fun() has reached
> >> > 'maxfuncdepth'. But if this is the way it's supposed to work, what's
> >> > the point of 'maxfuncdepth'? Don't all calls to user functions involve
> >> > an Ex command (since both `call' and `let' are Ex commands)? Is there
> >> > a way to permit more than 200 recursive calls to Fun() without
> >> > triggering the error?
> >>
> >> I tried lambdas, but they also catch this error due to the way they
> >> are implemented. Unlike (until you consider their internal
> >> implementation) lambdas regular functions are lists of Ex commands, so
> >> this is not surprising. Note that by default &maxfuncdepth is 100
> >> which is lesser then 200.
> >
> > Hmm... Perhaps Bram will weigh in on this, but effectively limiting 'maxfuncdepth' to 200 feels like an unintended consequence, rather than design intent - especially since the help on 'maxfuncdepth' makes no mention of the limit. The documentation on E169 suggests that the purpose of the 200 limit is to detect certain types of recursion involving :source and :execute commands. If it was meant to apply to function calls generally, why even have a separate option for function calls, especially if you can't increase its value to something that would permit meaningful recursion?
> >
> > Since there's no option governing the E169 limit, perhaps it could be changed to the maximum of 200 and 'maxfuncdepth'. Or perhaps it could take into account the type of Ex command (i.e., source/execute vs call/let). Or perhaps there could be a 'maxmemfunc' option (analogous to 'maxmempattern'), which would limit function call recursion by stack space consumed (or some rough approximation thereof) rather than # of calls.
>
> Limiting stack space consumed should be cleaner and more in line with
> the purpose of the limit (recursive nature of VimL executor and a
> number of different functions requires either imposing such limits or
> catching stack overflows). I am not sure though whether it is possible
> to get stack space consumption rate on any of the platforms Vim
> supports: after some searching I found only a number of dirty hacks
> like in http://stackoverflow.com/questions/53827/checking-available-stack-size-in-c.
> &maxmemfunc would be possible if VimL executor was not recursive and
> reimplemented stack based on malloc() like some other interpreted
> languages do, but it is recursive and uses system stack instead.

I can definitely see advantages to a limit based on memory usage, but I haven't looked at the VimL implementation to see what sort of added complexity that would entail. In addition to the overhead of the function calls themselves, there's also the memory allocated (dynamically) by each function invocation to consider. I'm guessing that's stored on some sort of Vim heap (not on the system stack).

Example:
fu! Func()
let big_data = Build_big_data_structure()
.
.
call Func()
endfu

Using 'maxfuncdepth' for E169 is probably the simplest approach, and the only downside is that if the user does something stupid, he could exhaust process memory. But he could already do that: consider that Build_big_data_structure in the example above could build an arbitrarily large structure on each recursive invocation, such that overflow could occur even within 200 calls.

Brett Stahlman.


>
> >
> > I noticed this because I'm running a tree processing algorithm that is inherently recursive. I had intended to compute 'maxfuncdepth' as a function of another option, but discovered that my choice was silently ignored for anything over 200. Although the depth of the trees can exceed 200 in extreme cases, the depth is bounded and known, so it made sense simply to boost 'maxfuncdepth' long enough to recurse the tree. If there's no way around the 200 maximum, I'll probably have to rewrite the algorithm to use breadth-first traversals, rather than the much more natural (and simple) tree recursion.
> >
> > Thanks,
> > Brett Stahlman
> >
> >>
> >> >
> >> > Thanks,
> >> > Brett Stahlman
> >> >
> >> > --
> >> > --
> >> > You received this message from the "vim_use" maillist.
> >> > Do not top-post! Type your reply below the text you are replying to.
> >> > For more information, visit http://www.vim.org/maillist.php
> >> >
> >> > ---
> >> > You received this message because you are subscribed to the Google Groups "vim_use" group.
> >> > To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@googlegroups.com.
> >> > For more options, visit https://groups.google.com/d/optout.
> >
> > --
> > --
> > You received this message from the "vim_use" maillist.
> > Do not top-post! Type your reply below the text you are replying to.
> > For more information, visit http://www.vim.org/maillist.php
> >
> > ---
> > You received this message because you are subscribed to the Google Groups "vim_use" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.