[slim-vim] More details on the socket crash bug

[Brad Beveridge]

On 6/12/06, Larry Clapp <larry at theclapp.org> wrote:
> On Mon, Jun 12, 2006 at 07:34:52AM -0400, Larry Clapp wrote:
> > On Sun, Jun 11, 2006 at 03:20:09PM -0700, Brad Beveridge wrote:
> > > A quick recap - the bug is this:  When you evaluate a form on
> > > Swank it send back a reply via a socket.  In Slim-vim we listen on
> > > that socket and print some data out to a Vim buffer.  Eventually
> > > this crashes and leaves you with a broken console (keystrokes
> > > don't appear).  I can blindly issue (cl-user::quit) and get to a
> > > bash prompt, I then need to "reset" the terminal.
> > >
> > > I spent today trying to reproduce the bug (see attached files for
> > > my attempt).  Serverstream just listens on a socket and prints
> > > data to a buffer, test.lisp sends data to serverstream.  I run
> > > these files in different Vims so that I can have 2 instances of
> > > ECL.
> > >
> > > I could not break this simple configuration, TEST can happily
> > > stream loads of data to SERVERSTREAM, which dutifully puts it out
> > > to a Vim buffer.  This implies to me writing to a Vim buffer from
> > > the Async handler is completely fine.
> > >
> > > So I had a bit of a brain storm and thought about what is
> > > different between my test and Swank.  Well, the test is kinda one
> > > way, Swank starts comms in Vim and responds to them in Vim.  So I
> > > made the SERVERSTREAM also echo back any socket data to the TEST
> > > client.  Whoa!  It breaks.
> >
> > I haven't tried your code yet, but I noticed this
> >
> >   (defmethod si::stream-write-string ((stream vim-buffer-output-stream) string &optional start end)
> >     (append-lines (subseq string start end) (buffer-of stream)))
> >
> > If START is NIL, subseq will throw an error.  Just a thought.
>
> Okay, I tried your code, and I observed the following behavior:
>
> In test.lisp,
>
>   (progn
>     (dotimes (x 10)
>       (format *stream* "hello ~D~%" x)))
>
> printed these lines in the serverstream lisp scratch buffer:
>
>   "hello 0"
>   "hello 1"
>   "hello 2"
>   "hello 3"
>   "hello 4"
>   "hello 5"
>   "hello 6"
>   "hello 7"
>   "hello 8"
>   "hello 9"
>
> but printed only
>
>   "\"hello 0\""
>   "\"hello 1\""
>
> (or up to 0, or 3, or 7, but rarely if ever the whole 10 lines) in the
> test.lisp lisp scratch buffer, until I pressed at least two keys.  I'd
> press one key (e.g. "h"), and the rest of the lines would show up.
> I'd press it again, and the cursor would show back up in the test.lisp
> buffer.  Funky.
Did you see the "oh my god the world has crashed" bug?  I needed to up
the amount of data to about 1000 format calls.  I also noticed the
keypress thing, it is like Vim is blocking elsewhere?  The keypress
might be important.  What I suspect may be happening (though I don't
know how) is that the ECL interpreter is being re-entered, and it is
not at all happy about it.

>
> I would suggest the following methodology, testing at each step, and
> saving snapshots at the end of each step: 1) get the client (test) /
> server (serverstream) code to work in ECL only (not run under Vim)
> just to make sure you have the plain networking code working.  2)
> Import the server code into Vim; have it talk to the ECL-only client.
> 3) Change the client code to write to a string.  4) Import the client
> code into Vim; have it talk to the ECL-only server.  5a) test the
> vim-buffer-output-stream apart from the client/server code; 5b) change
> the client code to append to a Vim buffer.  6) Change the client code
> to talk to the Vim server.
The code won't (I don't think) port easily to ECL, there is no
callback on data present feature that I am aware of.  Though the
server can certainly sit in a polling loop.
#2 - I've sort of done this.  If the server doesn't echo back to the
client, then all is well, the server can get messages all day long
from the TEST client.  Admittedly, TEST was running in ECL in Vim, but
without issues.

#3 - I've also altered the clientside callback to do nothing but pull
data from the socket during the callback - this eventually breaks.

#4 - Could be a good idea.  But remember that this bug happens during
Swank communications where effectively the echo server is Swank.

#5a - I've come to believe that vim-buffer-output-stream isn't
responsible for this particular bug.

>
> I may do some work on this, or I may adapt some of my own Vim
> networking code to work with sb-bsd-sockets, and try that.
Would you mind testing your networking code in my harness?  I'm
expecting similar results, but you never know :)

Cheers
Brad