New Rapid PTSD Screening Tools Can Diagnose Other Psychiatric Disorders

In part 2 of this video, Robert Gibbons, PhD, Blum-Riese Professor of Biostatistics, Departments of Medicine, Public Health Sciences, and Psychiatry; Director, Center for Health Statistics, University of Chicago, Chicago, Illinois, discusses new developments in screening tools for post-traumatic stress disorder (PTSD) using computerized adaptive testing and a machine-learning algorithm used to derive a diagnosis.

Watch part 1 here.

Read the Transcript:

The type of machine-learning algorithm we're using is the extremely randomized trees algorithm to derive a screening diagnosis of PTSD, which we confirm using the clinician-administered PTSD scale for the DSM-5, which is called the CAPS-5, which is the gold standard for PTSD clinician diagnoses.

You should note that the CAPS-5 takes about an hour to administer and requires a trained clinician. To calibrate and validate these instruments, we recruited 713 VA participants who took the entire 211-item bank and the PCL-5, which is a traditional PTSD assessment. Three hundred and four of the 713 participants also received a full CAPS-5 structured clinical interview.

In terms of the results, for diagnostic screening, the CAD PTSD had an AUC of 0.91, which is in the outstanding range for predictive accuracy. What is an AUC of 0.91? AUC is the area under the curve. The curve is the receiver operator characteristic curve, which describes the balance of true positives and false positives across the score range of the test, 0.91 is off the charts.

In essence, it exceeds what we would expect for the agreement between two trained clinicians. The amount of time that it takes on average to complete the CAD PTSD is 35 seconds in contrast to an hour for the CAPS-5 diagnostic interview. Also, there is no longer a need for trained clinicians. Our opportunity to screen entire populations is dramatically improved.

In terms of the CAT PTSD, an average of only 10 adaptively administered items was required to extract the information from the full 211-item bank, maintaining a correlation of 0.95 with the total 211-item bank score. Convergent validity with the PCL-5 was demonstrated with correlation of 0.88. It's quite strongly related.

Diagnostic validity was also demonstrated for the CAT PTSD with an AUC of 0.85 which, while not outstanding, is in the excellent range for the CAPS-5 diagnosis of PTSD.

Interestingly, all 211 items, if we score based on the entire bank, the AUC was slightly worse at 0.84, showing that we are maximizing the information in those items by adaptively administering them to the specific levels of severity for each individual respondent.

Despite the high correlation between the CAT PTSD and the PCL-5, the PCL-5 had much lower diagnostic accuracy with an AUC of only 0.75 relative to the 0.85 for the CAT PTSD, despite the fact that the CAT PTSD used only half the number of items, 10 items rather than 20 items for the PCL-5.

The CAD PTSD, the diagnostic screener, is even better as a diagnostic screener than the PCL-5 with an AUC of 0.91 versus 0.75 -- these are huge differences -- despite the fact that the CAD only requires 6 items when there are 20 items required for the PCL-5.

What we've done is we've achieved our goal of increasing the precision of measurement while at the same time decreasing the burden of measurements. Were there any kinds of outcomes of our research that were unexpected? Absolutely.

The ability to accurately reproduce an hour-long structured clinical interview diagnosis of PTSD in 35 seconds with an AUC of 0.91 is even better than we ever hoped or expected. Note that we no longer need a trained clinician to derive the screening diagnosis. It can be done anywhere on the planet.

We can administer these tests in and out of the clinic or behind enemy lines in active and remote areas in the active military. The finding also that the CAT PTSD dramatically outperformed the PCL-5, which is a widely used measure, using half the number of items was also unexpected.

Are there any practical applications of our work? I was trained at the University of Chicago to never do anything remotely useful. I hope that I'm a complete failure. Because I think there are a lot of practical applications of this work, and the answer is absolutely, yes. The practical applications are enormous.

We can now replace an hour-long structured diagnostic clinical interview with a 35-second machine-learning-based diagnostic screener and maintain a level of accuracy that is within the limits of agreement of 2 trained clinicians. In those that screen positive, we can then seamlessly measure the severity of their PTSD using the CAT PTSD in another minute.

This means that we can both screen for PTSD and measure its severity in and out of the clinic without the need for a trained clinician. We can also measure the response to treatment in individuals over time at any interval in time and even track them remotely over time to determining if they are experiencing a relapse of their illness.

Can these tools that we've been discussing for the measurement of PTSD be extended to other psychiatric disorders like major depression or generalized anxiety disorder? The answer there, of course, is most definitely.

We have already developed instruments based on this technology for adults to measure depression, anxiety, mania, hypomania, psychosis, adult ADHD, substance-use disorder, suicidality, and social determinants of health.

For children ages 7 through 17, we measure depression, anxiety, mania, hypomania, ADHD, conductive disorder, oppositional-defiant disorder, substance-use disorder, and suicidality. All of these measures are available for widespread use in clinics, schools, colleges, hospital systems, population studies. They're available in English, Spanish, and Chinese languages.

They all have been validated against structured clinical interviews and extant severity measures and can be integrated into the electronic health record systems such as Epic.

As a great example, SAMHSA and RTI are using these tools as part of their $30-million national prevalence study of mental health and substance use disorder, the MDPS study, as first-stage screeners to identify those individuals across the US that will receive a full SCID DSM-5 structured clinical interview.

Finally, what are the areas of further research and related studies that are needed? Now, we are involved in large-scale implementation rollouts of this technology for PTSD and suicide risk assessment in the VA, also with Dr. Lisa Brenner and her colleagues, and also implementation studies in many other areas.

A good example is the Cook County Jail, which, unfortunately, is our nation's largest mental health care provider. There's several other implementation studies that will use the new CAD and CAT PTSD tools. Areas for future research based on our current paper include the generation and validation of PTSD subdomain scores and studies of sensitivity to treatment-related changes.

For our other measures, we have already conducted such studies. Our existing measures have demonstrated that adaptive tests based on patient self-reports outperform traditional fixed-length tests based on either self-reports or trained clinician ratings.

Thank you so much for your attention.

Dr Robert Gibbons, PhD, is the Blum-Riese Professor of Biostatistics in addition to serving as a director in the departments of medicine, public health sciences, and psychiatry at the University of Chicago, Chicago, Illinois, as well as the director of the Center for Health Statistics. Dr Gibbons received his PhD in statistics and psychometrics from the University of Chicago.

Dr Gibbons is a fellow of the American Statistical Association, where he received a lifetime achievement award and created a section for mental health statistics, and several other national and globally recognized statistical societies.