Too many time sources – Isilon

So as many Aussies know, we have 2 time shifts each year for daylight savings, being the first Sunday in April (clocks back 1 hour), and first Sunday in October (clocks forward 1 hour)

Sunday evening I received an alert from my Isilon cluster saying that the time had drifted by more than 4 minutes from the AD time and Auth would be affected. 

Sure enough, it had, any CIFS/SMB access was problematic, but NFS was unphased.  

After a discussion with EMC support it was determined that there were too many time sources on the cluster. Being that both NTP and SMBTime. SMBTime is a service that pulls time from an Active-Directory, and NTP being Network Time Protocol.

In our setup both eventually point to the same time source. But it became apparent the two services don’t have a precedence order between themselves, and just end up in a race condition. 

EMC support were excellent in assisting me to diagnose the issue and provide recommendations to remediate the system.

On EMC recommendation we disabled the SMBTime service and forced a re-sync so NTP would reset the clock. Once done it was all back to normal for data access and the alerts were cleared as a result. 

It has left me with questions as the system was setup by an Integrator/Partner, and left in a “unrecommended” setup

More to follow….