[slim-vim] design discussion: network callbacks & recursion

[Larry Clapp]

Hi, everyone,

While adding Win32 socket support to Vim, I've seen that Windows can
call the network IO handler callback before the previous callback has
finished, i.e. indirectly recursively.  The Netbeans interface, on
which Jim based the network_io interface, shows that this can happen
on X11/Athena, Motif, and gtk platforms, too.  This causes problems if
you don't account for it.

I'd like to hide that detail from Vim network users.

Currently, Vim calls the Lisp callback directly, which reads directly
from its associated stream.

I'd like to add some indirection.  I'd first add a Gray stream that
buffers data from the real stream.  Then I'd add a Lisp function (the
"network callback") that Vim would call instead of the "real"
callback.  The network callback would drain input from the real
stream, and then, if-and-only-if it's *not* a recursive call, it would
call the real callback.

The real callback would happily read from the intermediary Gray stream
and do whatever it does.  If Vim invokes the network callback before
the real callback finishes, the network callback would drain the real
stream into the Gray stream and return, without calling the real
callback again.  If the real callback finishes and there's still data
on the Gray stream, the network callback would call it again.

So from the point of view of the real callback, it gets called once
for any input, it can read as much or as little as it wants off the
stream (presumably it'd read "a single message", for whatever that
means in the context of that callback), and then it returns and waits
for another message.

Discussion:

On Unix/Linux/Cygwin, you probably don't need the Gray stream, you can
just do a blocking read on the stream and wait for more data.  But you
do need some sort of "recursion guard", and as I said, I'd like to
hide that from the user and not make every network client re-implement
their own recursion guard.

And to be honest, I'm not sure you need the Gray stream in Windows
either.  In the course of writing this email, I ended up doing some
more research into ECL socket code, and I found that there's a
function called (setf non-blocking-mode).  I haven't tried it, but if
it works, it might make the Gray stream unnecessary in Windows, too.

Or it might not.  I don't know enough about how Windows handles
network callbacks, and I don't know where to look to find out more.
In particular, what does Windows do if you register a listener for a
socket, and it calls your listener, and your listener returns with
data left on the socket?  Does Windows wait until it gets even more
data on the socket to call your listener again, or does it do like I'm
going to do and call it repeatedly until you drain the darn thing?  If
the latter, then obviously we need to drain the socket every time
Windows calls us.

Thoughts / answers / comments / criticism?

-- Larry