Making 8,500 screens across the country observable

Context

A national retail network ran more than eight thousand digital displays across stores around the country. Each showed scheduled content — promotions, pricing, campaign media — pushed from a central content management system.

The failure mode was that nobody knew when it broke. A display showing a blank screen, a stale campaign, or a frozen frame looked identical from headquarters to one working correctly. Problems surfaced when a store called, which meant the average issue ran for about two days, and campaign reporting was unreliable because nobody could confirm what had actually played.

Client name withheld by agreement.

The constraints

The fleet was heterogeneous. Hardware had been deployed over several years in several generations, on connections ranging from solid to genuinely bad.

Bandwidth was not free. Pushing large media files to thousands of endpoints on retail networks, often shared with point-of-sale traffic, had to be handled carefully.

No physical access. Fixing something in person meant dispatching a technician to a store. Anything requiring a truck roll was expensive, so remote diagnosis and recovery mattered enormously.

What we changed

Making devices report in

The core problem was an absence of information. We built a lightweight agent that reported device health on an interval: what content it believed it should be showing, what it was actually rendering, its storage and connectivity state, and when it last successfully synced.

That distinction — intended versus actual — is what made the system useful. A device that believed it was current but was rendering a stale frame had previously been invisible. Now it was a specific, actionable alert.

Rebuilding media delivery

The original delivery pushed full media files to every device on every change. That saturated links and failed unpredictably on weak connections, and a failed transfer usually meant a stuck screen.

We restructured delivery to pull rather than push, with content addressed by hash so devices fetched only what actually changed and could resume interrupted transfers rather than restarting. Devices staged content locally and switched over only after verifying a complete download, which removed the class of failure where a partial file left a display in a broken state.

That change alone eliminated most of the recurring outages, because they had not been hardware failures — they were interrupted transfers.

Alerting that a human could act on

Fleet-wide monitoring produces enormous noise. Eight thousand devices generate a steady background of transient connectivity blips that resolve on their own, and alerting on all of them trains everyone to ignore alerts.

We tuned alerting around sustained conditions and patterns rather than individual events: a device offline beyond a threshold, a cluster of failures in one region suggesting a network issue rather than a device issue, content that failed to reach a meaningful percentage of its target fleet. Operations staff got a dashboard showing fleet state at a glance and a drill-down for specific sites.

Detection time went from roughly two days to about fifteen minutes.

Architecture

Content is authored in the CMS and processed by a media pipeline into device-appropriate formats, each addressed by content hash. Devices poll for their assigned manifest, fetch only changed assets, verify, and stage locally before switching. Agents report health and playback state on an interval to a monitoring service, which evaluates alerting rules and drives an operations dashboard.

Results

More than 8,500 displays managed centrally with reliable visibility. Incident detection went from about two days to fifteen minutes. The recurring outages — overwhelmingly caused by interrupted media transfers rather than hardware — were eliminated.

Campaign reporting also became trustworthy for the first time, because there was finally a record of what had actually played.

What we would tell you before starting something similar

Report intended state and actual state separately. A device that thinks it is healthy is not the same as a device that is healthy, and the gap between them is where the real failures hide.

Pull beats push at fleet scale. Devices that fetch what they need on their own schedule degrade gracefully on bad connections. Push assumes a reliability that retail networks do not have.

Alert on conditions, not events. Noisy monitoring is functionally the same as no monitoring, because people stop reading it.