The Birmingham Symptom specific Obstetric Triage System (BSOTS) involves standardised initial triage assessment (within 15 min15 minutes of arrival) of women when unscheduled pregnancy-related attendances occur. This standardised assessment identifies the level of clinical urgency defined as red (see immediately), orange (see within 15 min), yellow (see within an hour) and green (see within four hours). This audit assessed inter-rater reliability.

Methods Thirty randomly chosen midwives (15 B and 6 and 15 B and 7), who had undergone BSOTS training, currently working in Triage, were asked to assign a triage level to eight fictional scenarios. Sample size was determined by power analysis, calculated for a minimally acceptable IRR of 0.66 with a moderate effect size (0.5) and a p-value of 0.05.

Results The overall Intraclass correlation coefficient (ICC) was 0.96 (95% CI = 0.91–0.99) with total accuracy of 90.8%. Accuracy for individual categories ranged from 68.3% - 100% with the highest [k = 0.99, 95% confidence interval (CI) 0.97–1.00] and lowest severity [k = 0.94, 95% CI 0.92–1.00] categories demonstrating the strongest inter-rater reliability. Category two (yellow) offered the least successful reliability [k = 0.71, 95% CI 0.67–0.86] and had significantly more incorrect assessments [χ²(2) = 27.91, p < 0.001]. There was no relationship between demographic variables (e.g., band level, triage experience), and the level of accuracy or inter-rater agreement.

Conclusion Results demonstrate that BSOTS has strong inter-rater reliability suggesting it offers a reliable method of triaging women. Variability between triage categories requires further investigation.