Section 3 - Validation of PSPS

Construct validity is used to measure the extent to which a test or measure accurately assesses what it’s supposed to and how it relates to other measures. For the validation of the PSPS, correlational analyses have been performed to inspect the degree of overlap among mental defeat, pain, and moods, using the data obtained from participants with chronic pain.

Table 2: Construct Validity. Correlational analyses

*P<0.05; **P<0.01; ***P<0.001

The correlations between mental defeat and between the Short-Form McGill Pain Questionnaire (sensory, affective and VAS) and the Hospital Anxiety and Depression Scales (HADS) were within the medium range (r=0.40-0.65), suggesting a moderate association of mental defeat with pain and other indices of distress.

The purpose of examining internal consistency is to assess how well the items in a scale or questionnaire measure the same underlying construct. The internal consistency of the PSPS was evaluated in various participant groups, including those with chronic pain, acute pain, pain-free controls, and anxiety patient controls.

The findings revealed the following results for internal consistency and item-total correlations:

Participants with Chronic Pain:

The PSPS showed excellent internal consistency, with a Cronbach's alpha coefficient of 0.98.

Strong item-total correlations were observed, ranging from 0.72 to 0.88, with a median of 0.82. This indicates that all items consistently contributed to the total score of the PSPS in this group.

Participants with Acute Pain:

High internal consistency was observed, with a Cronbach's alpha coefficient of 0.95.

Item-total correlations ranged from 0.35 to 0.84, with a median of 0.72.

Pain-Free Controls:

The PSPS demonstrated high internal consistency, with a Cronbach's alpha coefficient of 0.91.

Item-total correlations ranged from 0.30 to 0.81, with a median of 0.63.

Anxiety Patients:

Excellent internal consistency was observed, with a Cronbach's alpha coefficient of 0.98.

Item-total correlations ranged from 0.68 to 0.93, with a median of 0.83.

The assessment of internal consistency ensures that the items in the PSPS are reliable and coherent in measuring the concept of pain self-perception across different participant groups. The strong internal consistency and significant item-total correlations indicate that the PSPS is a robust tool for evaluating pain self-perception in individuals with chronic pain, acute pain, as well as in pain-free and anxiety patient controls.

The purpose of test-retest reliability is to assess the consistency or stability of a measurement instrument over time. In this case, the test-retest reliability of the PSPS (Pain Self-Perception Scale) was examined by evaluating the response provided by separate samples of chronic pain patients and pain-free controls over a 2-day interval.

The test-retest reliability of the PSPS was examined by evaluating responses from separate samples of chronic pain patients and pain-free controls over a 2-day period. The findings showed satisfactory temporal stability, with a test-retest correlation of 0.92 (P < 0.001) for chronic pain patients and 0.94 (P < 0.001) for the entire sample This suggests a strong and significant association between initial responses and those obtained after the 2-day interval, indicating that the PSPS reliably measures pain self-perception over time for both chronic pain patients and the entire sample, including pain-free controls.

The PSPS has been translated into both Chinese and Spanish.

For the Chinese translation, the first author of the original PSPS team performed a forward translation of the PSPS from English to Chinese. A native Cantonese clinical psychologist, fluent in English, conducted the back translation from Chinese to English. The last author inspected both the original and back-translated questionnaires to identify any discrepancies. Any ambiguities were discussed and resolved between the translators. The resulting questionnaires underwent user-testing by a senior clinical psychologist in Hong Kong, who provided further suggestions to improve readability.

The Chinese version of the PSPS demonstrated excellent internal reliability among participants who reported pain, including patients and community volunteers. The Cronbach's alpha coefficients for internal reliability ranged from 0.97 to 0.99, indicating high consistency among the items of the PSPS. Additionally, the item-total correlations ranged from 0.31 to 0.92, suggesting that each item contributed significantly to the overall score of the PSPS.

Regarding the Spanish translation, permission to translate and validate the PSPS was obtained from the original authors. Two Spanish-speaking content experts, one psychiatrist and one psychologist, independently translated the PSPS into Spanish. The Spanish versions were then evaluated by the experts, and any discrepancies were discussed and rectified by consensus. An independent Spanish-English translator, with no knowledge of the original questionnaire, performed a back-translation of the combined Spanish questionnaire into English. The two content experts compared the original English version to the back-translated questionnaire.

The PSPS-Spanish version demonstrated high internal consistency, with a Cronbach's alpha coefficient of 0.90. The item-total correlation coefficients ranged between 0.68 and 0.86, indicating a strong relationship between individual items and the overall score. Principal components analysis revealed a one-factor structure that accounted for 61.4% of the variance.

The test-retest reliability, assessed using the intraclass correlation coefficient, yielded a correlation of 0.78 over a 1–2-week interval, suggesting reasonable stability of PSPS-Spanish scores over time.

Furthermore, the total PSPS score showed significant correlations with other questionnaires administered, including HADS, PVAS, PCS, and FIQ. This suggests that the PSPS-Spanish is related to various aspects of psychological well-being and pain experiences.

Face validity of the questionnaire was evaluated through pilot testing with two focus groups: one comprising healthy volunteers and the other consisting of chronic pain patients. Based on the feedback received, further modifications were made to enhance the ease of self-administration. Specifically, item rewording was conducted to reduce extreme language, such as changing "I felt completely defeated" to "I felt defeated." Additionally, reverse scored items were eliminated, for instance, changing "I feel able to deal with whatever life throws at me" to "I didn't feel able to deal with things that life threw at me." These adjustments aimed to improve the questionnaire's overall clarity and user-friendliness.

A ROC analysis was used to evaluate and compare the predictive accuracy of mental defeat alongside other significant multivariable predictors. It aimed to establish an optimal cut-off value for categorising individuals based on their SBQ-R scores, distinguishing between those with scores of ≥8 (comprising patients with chronic pain, both with and without opioid use disorder) and those with scores of ≥7 (encompassing chronic pain patients). The divergence in cut-off values was attributed to the diverse recruitment settings of participants, with the former group sourced from substance use disorder facilities and the latter consisting of chronic pain patients from the general population. This approach aligns with the original validation criteria of the SBQ-R questionnaire (Osman et al., 2001).

Findings in patients with and without opioid use disorder

Figure 2. ROC curve and AUC-PSPS. AUC= area under the curve; PSPS= Pain Self Perception Scale; ROC= receiver operating characteristic. From: Cheatle et al., 2023.

Subsequent ROC analyses indicated that the mental defeat measure performed comparably to pain catastrophizing for classifying patients presenting with SBQ-R scores of 8. A mental defeat cut-off score of 40 provided a sensitivity value of 0.69 and a specificity value of 0.70. This cut-off score demonstrated an acceptable ability to predict participant classification as SBQ-R scores 8. As seen in Figure 2, the area under the curve (AUC) was 0.77 (95% CI: 0.72, 0.82) and statistically significant (P<.001). A pain catastrophizing cut-off score of 16 provided a sensitivity value of .78 and a specificity value of 0.68. This cut-off score demonstrated an acceptable ability to predict participant classification as at high.

risk for suicide. For pain catastrophizing, the AUC was 0.75 (95% CI: 0.71, 0.81) and statistically significant (P<.001). However, the difference between the AUC between the two curves was not statistically significantly different (P=.582). This indicates that a mental defeat score of 40 and a pain catastrophizing score of 16 are comparable in estimating an SBQ-R score of 8.

Findings in chronic pain patients

Figure 3. Combined ROC curves to show relative predictive powers of the Pain Self Perception Scale (PSPS) assessing mental defeat, Perceived Stress Scale (PSS), and HADS-D (depression) for independently classifying Suicidal Behaviours Questionnaire-Revised (SBQ-R) scores ≥7. From: Themelis et al., 2023

A ROC curve was constructed to examine the performance of significant multivariable predictors of PSPS, depression (measured by HADS-D), and perceived stress (measured by PSS)) in classifying SBQ-R scores > 7. PSPS had the highest AUC compared to PSS and HADS-D. PSPS cut-off score of 36.5 provided a sensitivity value of .707 and a 1-specificity value of .307. As presented in Figure 3, this cut-off score demonstrated an acceptable ability to predict participant classification as SBQ-R score > 7. The AUC was = .740 (95% CI:.688–.792) and statistically significant (P < .001). Based on the optimal cut- off 36.5 and the final multivariable model from this analysis, an individual meeting this cut-off score is estimated to have an odds ratio of 73% higher than individuals with no symptoms of mental defeat (e.g., a score of 0 on the PSPS). Using the PSPS cut-off of 36.5, the PPV value was 56.12% and the NPV was 76.57%. A HADS-D score of 8 (rounded down from 8.5) provided a sensitivity value of .632 and a 1-specificity value of .345. As presented in Figure 3, this cut-off score demonstrated an acceptable ability to predict participant classification as SBQ-R score > 7. The AUC was = .699 (95% CI: .643–.765) and statistically significant (P < .001). The PPV was 54.07% and the NPV was 77.95%.

Distribution of PSPS scores

The distribution of PSPS total score (M=31.3, SD=28.0, Minimum=0, Maximum=94) for individuals with chronic pain does not differ significantly from a normal, symmetric distribution (skewness=0.70, SE=0.22; kurtosis= 0.81, SE=0.43), albeit being slightly positively skewed. PSPS scores can be used to calculate mean scores for different groups of people with chronic pain or for the same people at different time periods(7). Mean scores can be compared using standard deviations and 95% confidence intervals.

Due to its roughly normal distribution, the PSPS scores are anticipated to encompass the entire range of mental defeat without floor or ceiling effects. As a result, the PSPS is expected to suitable for monitoring trends over time and assessing the impact of programs or interventions aimed at reducing mental defeat, however the sensitivity of PSPS to change has not yet been demonstrated at the time of writing.

PSPS score variation across demographic and clinical characteristics

Table 3 . PSPS score variation across demographic and clinical characteristics.