Factor and Lisp, part two

I've been continuing my work on extra/lisp, and I've finally realized the solution to my current problem, that of macros. The solution, in a nutshell, was to replace the arrays I'd been using to represent lisp s-expressions with actual linked lists. After Slava pointed this out to me, I began work on a simple implementation of the lists. However, my work had not gone far, when the good doublec suggested that, as there was a great deal of overlap, I move my extra/lisp/conses to extra/lists and merge it with extra/lazy-lists (now known as extra/lists/lazy). This took a while, as I also ended up refactoring lazy-lists to use new style accessors and then had to go through all the vocabs that used lazy-lists (which actually wasn't that many) to figure out which needed to be USING: lists, lists.lazy or both.

And then, things get interesting!

After this, things began to get odd...A test starting failing in extra/lists/lazy. It seemed that >>nil? (which, if you're not familiar with factor is a "new-style" accessor: You'd use it like obj new-nil?-val >>nil?, and would leave the modified obj on the stack - elegant, n'es pas? (the converse of that would be nil?>>, which would work like obj nil?>>, and would leave obj's value of nil? on the stack)) was being called on the wrong type of object: That accessor is supposed to be called on a memoized-cons, but was being called on a lazy-filter. This seemed patently impossible to me, as the only place that the word was called was in a method of memoized-cons, meaning that the only possible value that could be on the stack was a memoized-cons.

n After puzzling this one over for a while, and vainly trying to use Factor's normally very usefully walker to determine what was happening, I brought the matter to the attention of the gurus of #concatenative. At the recommendation of doublec, I tried inserting print statements in the word, to figure out where the object was changing from a memoized-cons to a lazy-filter.

The results left me even more puzzled than before.

My descent into madness intensifies

Initially, I tried just printing out the object that >>nil? was dispatching on: As expected, I saw that, while the first dozen times it was called, it was receiving the proper memoized-cons object, it was indeed ending up with a lazy-filter. Okay...that shouldn't be happening, but I already know that it is, so that's at least somewhat expected. What happened next was, however, quite unexpected.

"Well," I thought. "I can see that it's a lazy-filter here...but it must be a memoized-cons when it enters...I know, I'll put print statements throughout the word, so I can see just where it changes!"

Now, below is the word sans debugging print statements:

M: memoized-cons nil? ( memoized-cons -- bool )
    dup nil?>> not-memoized? [
        dup original>> nil? [ >>nil? drop ] keep
    ] [
        nil?>>
    ] if ;

(if you don't understand this, go to the Factor home page, and start learning)

After adding various debugging print statements, it looked like this:

M: memoized-cons nil? ( memoized-cons -- bool )
    "Entering nil?: " write dup class . dup nil?>> not-memoized? [
        "True branch: " write dup class . dup original>> nil? [ "Now at >>nil?" write over class . >>nil? drop ] keep
    ] [
        "False branch: " write dup class . nil?>>
    ] if "\n" write;

So now, after reloading lists.lazy and running the test case, I get a bunch of output like this:

Entering nil?: memoized-cons
True branch: memoized-cons
Now at >>nil?: memoized-cons

Entering nil?: memoized-cons
False branch: memoized-cons

Entering nil?: memoized-cons
True branch: memoized-cons
Now at >>nil?: memoized-cons
  

So, ready for the weird stuff? Here're the last two calls...

Entering nil?: memoized-cons
False branch: memoized-cons

Now at >>nil?: lazy-filter
  

Um...what? The print statements at the begining of the word, and on the conditional branch that was taken aren't being called. It seems that it's somehow jumping into the middle of the word, without calling anything prior to this one section.

What Next?

Well, I have no idea why this happening. doublec hypothesizes that I've exposed a bug in method dispatch. I'm not sure what else I can do, if that is indeed the case...I guess I'll wait until Slava returns and see if he can figure it out.

In the meantime, I guess I'll keep working on extra/lisp, which was the point of this whole exercise, was it not?

Update:The next day, I went through the code some more and realized I was looking at the wrong place: The bug was actually in the code for lazy-filters, and had to do with the transititon to new-style accessors. The skip word needed a drop at the end, since the new >>foo has the stack effect ( obj newval -- obj ), while the old version it replaced, set-obj-fo had the effect ( newval obj -- ).