I think I won't go the USB route.
Yes, it's convenient, and it works fine if there's a full-spec PC in the middle of it all.
But, with a little embedded SBC/SOM, with a lesser processor? Too much overhead.
Better to use a processor solution with an old-school system bus of some sort - do they make those anymore? - and multi-UARTs with a regular bus interface. Even if interrupt management does get Interesting.
Last time, many years ago, I used an octal a quad UART chip from, if memory serves, TI*. That seems to have gone obsolete. There are still quad UARTs from other chip vendors, in inconveniently bulky packages, starting at around $7 in modest quantities. I think this is part of why I went the USB route: anything with a regular system-bus connection was just going to take up too much space (in addition to requiring some fiddling with drivers, to handle large numbers of them, which I didn't expect to be an issue with USB).
The alternative, maybe, is to stuff multiple UARTs into a cheap small FPGA / large CPLD. Hey, I have a known-good UART cell in Verilog. That means writing a new device driver for Linux, if it's not made 8250-compatible (which would eat up an awful lot of logic, to no good purpose), but maybe that's not such a big deal.
The FPGA route looks like it comes out cheaper, too.
The USB approach looked like a good solution, and it did save a bunch of development time. If we throw enough processor power at it, it'll meet the performance requirements, and processor power is getting cheap. But for the next generation? Using UARTs-in-FPGAs might be best. Also, it opens interesting possibilities for embedding certain special features right at the edge of the system, where the CPU wouldn't have to be bothered all the time.
* Yup, TI. Part number was TLC16C554. Came in a 12mm square, 80-lead SQFP. Which wouldn't have worked for the current project anyway. I was misremembering it as octal, and getting 8 ports in that size package might have worked.