I'm an idiot. Spent half a day chasing a bug that isn't really there. I've been writing the code that lets you download new programs into the HullPixelbot over the network. It worked fine for tiny programs, and then failed for larger ones.
I can hear you thinking "Buffer overrun". Except that I'm not that kind of idiot, and my code is carefully protected against too much data arriving when it shouldn't. And the fault didn't appear at a consistent point in the conversation, which it would if the program was hitting a hard limit somewhere. The failure threshold moved about a bit. Sometimes the program failed with a ten line program, other times it failed with an eight line program.
Any ideas? Took me a little while to figure it out. And of course it was my own stupidity.
It turned out that my over enthusiasm for code instrumentation was my undoing. When you are writing embedded programs you need code instrumentation to stay sane. In simple terms code instrumentation is "putting in print statements to see what is happening". My little program was notifying me each time a data byte arrived so I could convince myself that all was well.
And therein lies the stupidity. It was sending out three bytes of diagnostic information for each incoming byte. Since the send and receive rates are the same, this meant that after a while the program became unable to deal with incoming data because it was taking too long to send out the debug information.
The Arduino will do some buffering of data in and out of the device, but at some point this will fill up, at which point bad things happen. It's not going to happen at a consistent point in the program because the serial timings will vary slightly as other events occur on the processor.
So, I took out the debug code and the program worked perfectly. Strange but true.
And I'm an idiot.