Monitoring agent update causes reboot loops on customer workstations

Summary

An update of the monitoring agent was pushed to customer workstations in the early morning. Within the hour the first customers called in: workstations restarted again and again before anyone could sign in.

Impact

Availability only. About one hundred and twenty workstations at nine customers were caught in a reboot loop, and the staff concerned could not work until their machine was fixed. No data was lost or exposed.

Response

The service desk linked the calls to the update within twenty minutes. The engineers stopped the rollout, withdrew the update from the deployment console, and published a fix for machines already affected. By the end of the day the previous version was back on every workstation, and the last stubborn machines were repaired by hand on site.

Notifications

Every affected customer got a call from their account owner and a written follow-up. A status message was shown on the service desk page during the incident.

Root cause

The new agent version conflicted with a disk encryption driver used on one model of workstation. The update went to all customers at once, without a trial group.

Lessons learned

Updates of the agent now go out in waves, starting with our own workstations and one friendly customer. A wave only continues when a day has passed without new tickets.