I've discovered a niggling little bug in the RTCD feature during testing, and was wondering if anyone else had run into this and if so, what resolution they found.
Basically, it seems that if the logging source IP address is different than the NCM polling address, the config change will not be detected inside NCM. The syslog message arrives, the helper program fires, but some internal logic in NCM discards it unless the source address matches the polling address. This can be problematic for a few reasons:
1. Sometimes you want a particular polling address for a device based on what you are trying to monitor, and that is not where you want your logging coming from
2. Some devices do not support all interface types for logging source-interface. For instance, specifying particular VLAN interfaces is not supported on some platforms (I ran into this on a Catalyst 6509)
3. This failure mode is entirely silent. You can test your RTCD setup on one device and it can work flawlessly. The only way to detect this behavior is to inspect syslog messages from each device and compare them against the NCM polling address. So you could think you have RTCD set up on your entire network, and never be the wiser unless you test each device individually.
The workaround seems to be to change the source-interface, or when that is undesireable to change the polling address for NCM. Both of these seem like suboptimal choices to me. It would be great if NCM could do a lookup against all monitored interfaces instead of just the polling interface, as it seems like this could be a fairly common scenario.