![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
5 years ago I was working with financial customers optimizing performance of their trading applications. While using what now is ancient hardware, one of the tasks was to find optimal balance between optimizing CPU budget we spent on processing networking tasks by processing many packets at once with maximizing response time for processing each packet.
One of the simplest tests was to measure the fastest small packet RTT, to use it as a base point for latency measurements of the full application. Surprisingly, the stock Linux s/w stack of that time running on latest h/w did well after minimal setup. Just disable interrupt moderation on both ends, and you get a ping of about 30us on 1G back to back connection. It could be improved when tweaking some other parameters but only by a little bit.
Recently working on some industrial project using embedded OS I observed a high RTT. I reproduced a setup described in the previous paragraph with modern h/w and Linux OS, and got a ping of 120us! I guess something important had changed in ethernet driver/ip stack in last 5 years. I was not following that, so I can't guess what was it. (Of course, no interrupt moderation on both ends, no power management/C states, idle system, PCIe cards connected to off-core PCIe controller, not through the chipset)
One of the simplest tests was to measure the fastest small packet RTT, to use it as a base point for latency measurements of the full application. Surprisingly, the stock Linux s/w stack of that time running on latest h/w did well after minimal setup. Just disable interrupt moderation on both ends, and you get a ping of about 30us on 1G back to back connection. It could be improved when tweaking some other parameters but only by a little bit.
Recently working on some industrial project using embedded OS I observed a high RTT. I reproduced a setup described in the previous paragraph with modern h/w and Linux OS, and got a ping of 120us! I guess something important had changed in ethernet driver/ip stack in last 5 years. I was not following that, so I can't guess what was it. (Of course, no interrupt moderation on both ends, no power management/C states, idle system, PCIe cards connected to off-core PCIe controller, not through the chipset)
