Yeah, had to put on a show this afternoon. On a Monday. At the client site, so it meant moving the test program from my workstation to my laptop, and packing up the test-bed hardware.
First oopsie: the control MCU enumerates correctly as /dev/ttyACM0, and I can open a PuTTY session connected to it, but pounding on the keyboard has no effect. WTF???
After much power-cycling of the testbed, cable plugging and unplugging, and so on, I realize that I'd been pounding on the wrong keyboard. Putting the laptop on my desk for the process was maybe not the best move; I kept getting mice and keyboards mixed up.
That having been sorted out, I proceeded to box up the hardware. Second oopsie! I left the mini-USB cable plugged into the hub board. It had enough leverage against the edge of the box to pop the little SMT connector right off the board.
Well, that's why we make the boards in threes. One for the client, one for me, and one for Murphy, right?
So I grab USB Hub Board S/N 2 and swap it in, then quickly power up the unit, plug it into my lab PC, and confirm that everything still enumerates correctly. Off to the client site!
Discussion, display of hardware, setting-up of laptop, connecting of cables! Send the first few commands to the control MCU: all appears well. Run the test program. Issue a few commands....
Third oopsie: the power LEDs on the remotes go out after a couple of commands. Everything that goes through the UARTs (comm or modem control) seems to have stopped working.
Much fussing and fiddling. Eventually discover that, shortly after I started doing things with the UARTS, there was a flurry of USB errors, and the UARTS disconnected and re-enumerated, which didn't exactly preserve my program's connections to them.
After much further fiddling, I dug a compact USB 2.0 High-Speed hub out of my laptop case - good thing I had one in there, and good thing it was of the High Speed persuasion! - and used that to connect everything.
Still strangeness, but eventually I managed to get it working well enough to demonstrate basic operations.
Now I gotta figure out what was going on with the USB: is there something wrong with the second USB hub board, is it something to do with the laptop, or was it just the built-in crisis detector at work?
It is actually rather important that the USB hub work properly, as that and the UARTS have to end up on the same grand unified board with the control MCU and the power circuitry, comes the time to turn this into a product.
Oh, well. The client got to see something working, anyway, and there's a pretty good chance I'll have the glitches fixed by the end of the week.
Update: It appears that the hub is fine. The UARTs may or may not have a problem, and the Linux driver for the FTDI UARTs may or may not have a problem. There's some issue with the modem-control signals, and some sorts of activity on those signals triggering error status, which doesn't particularly make sense.
Also, there may have been a mechanical Heisenbug in the data path to one of the UART chips: a poor solder joint may have made it sensitive to vibration.
It's now Friday morning, and I'm still bashing my head against this... though I now have seven out of eight ports in the test setup apparently working (port 0 goes to permanent error status if I turn on RTS, though it seems to be OK if I exercise the modem-control signals from GtkTerm).