Risk Adjustment

BackgroundThe performance of the Charlson and Elixhauser comorbidity measures in predicting patient outcomes have been well validated with ICD-9 data but not with ICD-10 data, especially in disease specific patient cohorts. The objective of this study was to assess the performance of these two comorbidity measures in the prediction of in-hospital and 1 year mortality among patients with congestive heart failure (CHF), diabetes, chronic renal failure (CRF), stroke and patients undergoing coronary artery bypass grafting (CABG).MethodsA Canadian provincial hospital discharge administrative database was used to define 17 Charlson comorbidities and 30 Elixhauser comorbidities. C-statistic values were calculated to evaluate the performance of two measures. One year mortality information was obtained from the provincial Vital Statistics Department.ResultsThe absolute difference between ICD-9 and ICD-10 data in C-statistics ranged from 0 to 0.04 across five cohorts for the Charlson and Elixhauser comorbidity measures predicting in-hospital or 1 year mortality. In the models predicting in-hospital mortality using ICD-10 data, the C-statistics ranged from 0.62 (for stroke) - 0.82 (for diabetes) for Charlson measure and 0.62 (for stroke) to 0.83 (for CABG) for Elixhauser measure.ConclusionThe change in coding algorithms did not influence the performance of either the Charlson or Elixhauser comorbidity measures in the prediction of outcome. Both comorbidity measures were still valid prognostic indicators in the ICD-10 data and had a similar performance in predicting short and long term mortality in the ICD-9 and ICD-10 data.

BackgroundThis manuscript describes a method for adjustment of nursing home quality indicators (QIs) defined using the Center for Medicaid & Medicare Services (CMS) nursing home resident assessment system, the Minimum Data Set (MDS). QIs are intended to characterize quality of care delivered in a facility. Threats to the validity of the measurement of presumed quality of care include baseline resident health and functional status, pattern of comorbidities, and facility case mix. The goal of obtaining a valid facility-level estimate of true quality of care should include adjustment for resident- and facility-level sources of variability.MethodsWe present a practical and efficient method to achieve risk adjustment using restriction and indirect and direct standardization. We present information on validity by comparing QIs estimated with the new algorithm to one currently used by CMS.ResultsMore than half of the new QIs achieved a "Moderate" validation level.ConclusionsGiven the comprehensive approach and the positive findings to date, research using the new quality indicators is warranted to provide further evidence of their validity and utility and to encourage their use in quality improvement activities.

IntroductionIn critical care observational studies, when clinicians administer different treatments to sicker patients, any treatment comparisons will be confounded by differences in severity of illness between patients. We sought to investigate the extent that observational studies assessing treatments are at risk of incorrectly concluding such treatments are ineffective or even harmful due to inadequate risk adjustment.MethodsWe performed Monte Carlo simulations of observational studies evaluating the effect of a hypothetical treatment on mortality in critically ill patients. We set the treatment to have either no association with mortality or to have a truly beneficial effect, but more often administered to sicker patients. We varied the strength of the treatment's true effect, strength of confounding, study size, patient population, and accuracy of the severity of illness risk-adjustment (area under the receiver operator characteristics curve, AUROC). We measured rates in which studies made inaccurate conclusions about the treatment's true effect due to confounding, and the measured odds ratios for mortality for such false associations.ResultsSimulated observational studies employing adequate risk-adjustment were generally able to measure a treatment's true effect. As risk-adjustment worsened, rates of studies incorrectly concluding the treatment provided no benefit or harm increased, especially when sample size was large (n = 10,000). Even in scenarios of only low confounding, studies using the lower accuracy risk-adjustors (AUROC ConclusionsLarge observational studies confounded by severity of illness have a high likelihood of obtaining incorrect results even after employing conventionally "acceptable" levels of risk-adjustment, with large effect sizes that may be construed as true associations. Reporting the AUROC of the risk-adjustment used in the analysis may facilitate an evaluation of a study's risk for confounding.

Nearly all empirical studies that estimate the coefficients of a risk equalization formula present the value of the statistical measure R2. The R2-value is often (implicitly) interpreted as a measure of the extent to which the risk equalization payments remove the regulation-induced predictable profits and losses on the insured, with a higher R2-value indicating a better performance. In many cases, however, we do not know whether a model with R2 = 0.30 reduces the predictable profits and losses more than a model with R2 = 0.20. In this paper we argue that in the context of risk equalization R2 is hard to interpret as a measure of selection incentives, can lead to wrong and misleading conclusions when used as a measure of selection incentives, and is therefore not useful for measuring selection incentives. The same is true for related statistical measures such as the Mean Absolute Prediction Error (MAPE), Cumming's Prediction Measure (CPM) and the Payment System Fit (PSF). There are some exceptions where the R2 can be useful. Our recommendation is to either present the R2 with a clear, valid, and relevant interpretation or not to present the R2. The same holds for the related statistical measures MAPE, CPM and PSF.