Author response: Long-term risk prediction after major lower limb amputation: 1-year results of the PERCEIVE study

BJS Open, https://doi.org/10.1093/bjsopen/zrad135, published 24 January 2024

Dear Editor

We welcome the comments by our colleagues Norvell et al., who are world-renowned experts in the area of risk prediction after amputation surgery, and are grateful for the opportunity to provide a response.

To address first our results concerning the AMPREDICT Mobility and Reamputation models, these included patients who died before 1 year after surgery¹. Additional sensitivity analyses including only those alive 1 year after surgery, not included in the article due to space constraints, resulted in marginally worse performance.

Regarding variable capture, we deliberately designed a study that did not need ethical approval for direct patient contact and data collection. This allowed us to undertake what we believe to be the largest international external validation of risk prediction tools for amputation outcomes at both 30 days and 1 year^{1, 2}. Thus, for AMPREDICT-mobility we did not capture two variables (highest education level and patient self-rated health) and rather imputed using ‘worst case’. All other variables, including functional independence and alcohol misuse, were captured and included in our analyses. While this would impact on performance evaluation, we made the pragmatic choice to include the model as there are very few similar models available, and the predicted impact on performance was thought to be low. Our study design also influenced how ambulation was defined and captured in PERCEIVE; post-amputation ambulation is most commonly evaluated using the Special Interest Group in Amputee Medicine (SIGAM) grades within the UK. These grades differ from the Locomotor Capability Index-5. However, we harmonized the two categorizations to the best of our ability and again predicted the impact on performance would be low.

Norvell et al. correctly highlight that validation study design choices that do not directly replicate development studies will influence the observed predictive performance of models. This statement draws attention to a key discussion topic concerning prediction model validation. Prediction models have generally been found accurate in predicting outcomes such as mortality and postoperative complications in their own populations³, but this is not assured, and performance is unpredictable. We argue that the validation of prediction models should not be limited to ‘narrow validation’ where patient cohorts and data capture methods are exactly replicated in the validation process. The ultimate purpose of a prediction model is to improve patient care in real-world practice, and ideally would be usable in diverse populations. An ideal model would provide useful data even when slightly different outcome definitions or categorizations are used. Determining whether a model is appropriate for use in clinical practice, especially if considered for use in other countries, requires multiple validations that extend beyond ‘narrow validation’. This approach is not only acknowledged by international guidance on prediction model studies but is encouraged⁴. We are grateful to Norvell et al. for their dedication to improving amputation research and practice and for engaging with us on this important topic.