\"SecureSafety PPE detection in action — what you are testing for during a pilot acceptance test.\"
The vendor demonstration took place on a Tuesday afternoon in a purpose-built pilot bay. Good light from the skylights, clean concrete floor, workers in bright yellow hi-vis walking at a measured pace between two lines of tape marking the detection zone. The system performed beautifully: 98% detection rate, the presenter reported. Zero false positives across two hours.
Three months into the pilot at the actual site — a busy automotive parts warehouse with racking that cast long shadows in the afternoon, compressed video streams over an ageing network switch, and contract workers in faded orange vests that the model kept classifying as background — the numbers looked nothing like the demo.
This is not a story about a dishonest vendor. It is a story about what "acceptance testing" means when the thing being tested is a safety-critical system in an industrial environment. Vendor demonstrations validate that a system works in the best possible conditions. Acceptance testing determines whether it works in yours.
Why vendor demos in clean environments do not predict real-world performance
Computer vision models learn to detect by training against datasets. Training datasets almost invariably over-represent the ideal: adequate lighting, clear sightlines, unobstructed subjects, standard PPE items in standard colours, predictable movement patterns in open space. When the model encounters conditions outside that distribution in deployment — low-contrast PPE against a similar-coloured wall, directional shadows from industrial racking, partial occlusion by machinery, compression artefacts from a legacy NVR, workers at unusual angles — performance degrades, and in some cases degrades sharply.
The gap between demo performance and site performance is a consequence of distribution shift: the well-documented phenomenon in machine learning where a model's accuracy in deployment falls when the deployment environment differs meaningfully from its training environment. This is not a flaw unique to any particular vendor; it is an inherent characteristic of how computer vision models work. The responsible question to ask any safety AI vendor is not "what is your average detection accuracy?" but "what was the accuracy in environments that resemble mine, and how was it measured?"
The purpose of a structured acceptance test is to answer that question with your own data, from your own site, under your own conditions, before you sign off the pilot and commit to a full deployment. Skipping this step is equivalent to commissioning a pressure relief system based on the manufacturer's theoretical performance data without testing it at your operating pressure. For a safety-critical system, that is not a shortcut anyone should be comfortable taking.
The five dimensions of a rigorous acceptance test
A complete acceptance test evaluates five dimensions. Each must meet defined criteria before the pilot is accepted. A conditional pass in one dimension may be acceptable if the remediation condition is specific and achievable — a camera angle adjustment, a network upgrade, a firmware update — and is documented as a binding pre-deployment requirement. A fail in any dimension should halt deployment until the failure is resolved.
Any IP camera accessible via ONVIF or RTSP can be used. The acceptance test determines whether your installed cameras meet the quality thresholds the AI requires.
Dimension 1: Detection recall
Does the system detect the events it is designed to detect? Recall is the true positive rate: of all the target events that actually occurred during the test period, what proportion did the system flag? A high recall rate means the system is not missing events. Test this by staging controlled scenarios — workers entering a restricted zone, PPE removed in a monitored area, a vehicle proximity scenario — and recording whether every staged event generates a detection. Target: ≥ 90% recall on staged critical events; ≥ 95% on PPE non-compliance at controlled entry points.
Dimension 2: False positive rate Does the system fire alerts for events that are not actually hazardous? False positives are the practical enemy of operational trust. A system that fires alerts correctly 90% of the time but generates a constant background noise of incorrect alerts will erode operator confidence until people stop acting on them — which is a more dangerous state than not having the system at all. For tier-1 safety events, a false positive rate above 5–10% typically triggers alert fatigue within weeks. Target: ≤ 5% false positive rate for tier-1 events.
Dimension 3: Alert latency From the moment a target event occurs to the moment an alert appears in the control room, how much time elapses? This is total system latency, including edge detection processing, network transmission, and display rendering. For a vehicle–pedestrian proximity event, a 200ms latency provides meaningful response time. A two-second latency in the same scenario does not — the vehicle has moved six metres. Test this with time-stamped frame evidence, not estimates or vendor-reported averages. Target: ≤ 200ms for tier-1 events.
Dimension 4: Failure modes What happens when the system encounters conditions it cannot handle? Every AI system has failure modes; what matters is whether they fail safe or fail dangerous. A safe failure produces a false positive or a logged error — annoying, but manageable. A dangerous failure silently fails to detect a real event, or goes offline without notifying operators. Test this deliberately by introducing known difficult conditions. Target: all failure modes documented; no dangerous silent failures.
Dimension 5: Integration reliability Does the system connect reliably to your VMS, network, and notification infrastructure? Integration failures are the most common cause of post-deployment performance problems, and they are usually invisible during vendor demos that use test networks and pre-configured equipment. Test every integration point explicitly: trigger a test alert and trace its path to every endpoint — control room screen, supervisors' devices, audit log, VMS overlay. Do not assume integration is working because the vendor says so; verify it with a timed trace.
How to run a structured scenario test
A structured scenario test is a planned series of events staged in the real site environment during the acceptance test period. It is not a covert test: the vendor knows it is happening, and the site's EHS team designs the scenario set with reference to the site's critical risk register.
For each detection capability being assessed, design three scenario types:
Positive confirmation: the target event clearly occurs — a worker clearly inside a restricted zone, a piece of PPE clearly missing, a vehicle within the defined proximity threshold — and the system should detect it.
Marginal positive: the target event occurs in a condition approaching the system's detection limits — partial occlusion by a structural column, a worker at the far edge of the camera's field of view, a vehicle approaching at the minimum speed threshold — and the system should still detect it.
True negative: the target event does not occur — a worker approaches but stops outside the zone, a worker wearing full PPE enters a monitored area, a vehicle travels within speed limits — and the system should not fire an alert.
Run each scenario type at least five times, under different lighting conditions where relevant, and at different times of day to capture the full range of ambient conditions on the site. Document the result of each run against the pass/fail criteria for each dimension before moving to the next scenario.
Assign a named test lead from the customer side — not the vendor — to coordinate scenario runs and record results independently. The vendor may attend and assist with interpretation, but the test itself must be observed and documented by the customer's own team. The acceptance test documentation forms part of the procurement record and, in the event of a later incident investigation, may be relevant to demonstrating due diligence.
Precision and recall: the two numbers that matter
Two statistics drive most substantive discussion about detection system performance. Both are necessary; neither alone is sufficient.
Recall answers: "Of the real events that happened, did the system detect them?" It is calculated as: true positives ÷ (true positives + false negatives). A high recall system catches most real events. Low recall means the system is missing things — which, for a safety-critical application, is the more dangerous failure direction.
Precision answers: "Of the alerts the system fired, how many were genuine?" It is calculated as: true positives ÷ (true positives + false positives). A high precision system fires alerts that are almost always real events. Low precision means a high false positive rate, which drives alert fatigue.
A system with 99% recall and 10% precision catches almost everything, but 90% of its alerts are wrong — operators will stop responding within weeks. A system with 95% precision and 60% recall rarely alerts incorrectly, but misses 40% of real events — which for PPE detection or vehicle proximity monitoring is a dangerous gap.
For industrial safety AI, reasonable acceptance criteria prioritise recall over precision for tier-1 events, because the cost of a missed detection exceeds the cost of a false positive — provided the false positive rate stays below the level that triggers alert fatigue. The acceptance criteria table below reflects this hierarchy.
Conditions to stress-test during acceptance
The conditions most frequently responsible for unexpected detection degradation in UK industrial environments are:
- Direct backlight: cameras facing windows, skylights, or open yard entrances that create silhouettes and reduce foreground detail — test during the worst backlight hours
- Night-time IR mode: cameras switching to greyscale IR produce different image characteristics than daytime colour footage; colour-coded PPE loses its colour information — test detection performance during overnight operation
- Rain and lens condensation: common in UK outdoor and semi-indoor environments; a lens with water droplets or condensation film degrades detection accuracy significantly — test during or after rain
- Steam and atmospheric interference: common in food processing, chemical, and some manufacturing environments — stage tests during normal process operation, not during shutdowns
- Camera angle edge cases: PPE detection accuracy degrades significantly if the camera is mounted directly overhead; vehicle detection is more robust but still angle-dependent — ensure test scenarios replicate the actual camera geometries, not ideals
- Partial occlusion: workers partially hidden by structural columns, racking, or machinery; vehicles partially obstructed during approach — stage at least two scenarios per detection type where the detection target is partly obstructed
The acceptance criteria decision table
| Dimension | Pass | Conditional Pass | Fail |
|---|---|---|---|
| Detection recall (staged scenarios) | ≥ 90% | 80–89% with documented improvement plan and committed timeline | < 80% |
| False positive rate (tier-1 events) | ≤ 5% | 5–10% with specific threshold adjustment plan | > 10% |
| Alert latency (tier-1, measured) | ≤ 200ms | 201–500ms with documented network optimisation plan | > 500ms |
| Failure modes | All documented; no silent dangerous failures | Minor silent failures documented, notification fix confirmed and scheduled | Any critical detection silent failure |
| Integration testing | All endpoints confirmed functional | One endpoint requires configuration fix (documented and scheduled) | Core VMS or notification integration non-functional |
A conditional pass is not a pass. It is a written commitment to a specific remediation with a named responsible person and a fixed delivery date. If the condition is not met by the agreed date, the deployment pauses. Include this language explicitly in the pilot agreement.
10-point acceptance test checklist
- Site risk register reviewed to define the critical event types that receive highest test priority
- Named test lead assigned from the customer's EHS or operations team — independent from the vendor
- At least 5 positive, 5 marginal positive, and 5 true negative scenarios designed per detection capability being tested
- Tests conducted across at least two shifts (day and night) to capture the full range of lighting conditions
- Recall and false positive rates calculated independently from the customer's test log, not from vendor-reported data
- Alert latency measured using frame-by-frame timestamp evidence, not estimated from observation
- All five failure mode scenarios deliberately staged: lights off, camera obscured, network interruption, high-volume concurrent event, edge-of-frame target
- Every integration endpoint traced end-to-end to confirmation — control room screen, supervisors' devices, audit log, VMS
- DPIA completed or updated before the test period ends, in line with ICO guidance on automated processing of workers' personal data
- Acceptance criteria decision table completed in full, with named sign-off from the customer's EHS lead, before deployment proceeds
SecureSafety's acceptance testing methodology was built in the environments where the cost of a missed detection is measured in lives rather than audit findings. The offshore oil and gas platforms where the system first ran operate under no-compromise conditions: sub-zero temperatures, salt spray, compressed camera runs over 200 metres, and a response time to any incident measured in helicopter hours rather than ambulance minutes. The sub-0.05% error rate held across approximately 18,000 daily events was not asserted in a demonstration; it was measured under sustained operational load in the hardest environments the industry offers. When you run your acceptance test against the criteria in this article, you should expect those numbers — and if any dimension falls short, you should expect a documented explanation and a credible remediation path, not a renegotiated specification. Book a demo to discuss what acceptance testing looks like for your specific site and camera infrastructure.
