Research · Quant
Equinix SmartView: multi-rater evaluation
A calibrated-observer study combining self-rated workload, reliability, and mixed-effects modeling.
Evidence
Measured
−57.3%
NASA-TLX self-rating · prototype study
Measured
−49.9%
NASA-TLX self-rating · prototype study
Measured
−46.0%
NASA-TLX self-rating · prototype study
Measured
−39.6%
NASA-TLX self-rating · prototype study
Measured
0.96
ICC(2,K) · multi-rater reliability
Measured
0.92
Krippendorff α · multi-rater reliability
Stakes
Data-center technicians and site managers used SmartView to watch power draw and alerts. Navigation was inconsistent, alerts were not prioritized, and circuit-level data was missing, so people missed signals that mattered to customers and to billing.
The bet
If the dashboard made operational state legible, people doing real tasks would report lower workload, and independent observers would agree about what they saw.
Three moves
Structured the task. Twelve technicians and two site managers worked through a scripted task analysis in think-aloud sessions, so every session could be compared with the others. Why: a fixed script turns anecdotes into comparable observations.
Calibrated the observers. Six cross-functional observers coded every session in the second study against a shared coding matrix. Why: agreement between raters is evidence; one researcher’s reading is not.
Modeled the rater. A mixed-effects model with random intercepts for participant and observer separated the effect of the interface from the effect of who was watching. Why: without it, a harsh observer looks like a bad interface.
For the methodologist
Within-subjects, counterbalanced comparison of the current interface and a mobile-web prototype (n = 6: four technicians, two site managers). Workload was self-rated on NASA-TLX. Six calibrated observers; ICC(2,K) = 0.96, Krippendorff’s α = 0.92. Percentages are reductions from the current interface; with n = 6 they describe this study, not a population.
What I’d do next
Repeat with a larger operational sample, keep the full model-comparison record, and tie workload to operational outcomes such as time to acknowledge an alert rather than stopping at self-report.
Request the full case
The full case, with the working artifacts, is shared with people reviewing my work for a role or engagement. Confidential, participant-identifying or collaborator-owned material is never shared.