Stata Modules for Calculating Novel Predictive Performance Indices for Logistic Models

1. Background

Currently risk prediction is an appealing research area (1). In the last two decades, there has been an increasing trend in the discovery of new biomarkers in clinical medicine (2). It is by use of predictive models that people can use their risk factors profile for a certain medical condition to calculate their corresponding risk of developing that event in the future (1). In the view of the current shoot-up in the discovery and emergence of new risk markers, statisticians as well as clinicians will need to tackle the challenge of the assessment of predictive capacities of these biomarkers. Clinically speaking, many predictive models provide risk values (probability of developing a medical condition in the future) that fall into high, low, or intermediate range. While making medical decision on high-risk and low-risk individuals is somehow straight forward, dealing with intermediate range subclass will be cumbersome (1). As such, enhancement to the extant models has been sought to reclassify individuals more efficiently. Strong biomarkers have been added to relevant models in order to improve their predictive power (3).

Having been frequently making statements that the predictive performance of a model is superior to another, researchers are frequently challenged by statistical reviewers of scientific journals to provide rigorous statistical justification for their statements (4). How best to quantify the improvement in risk prediction offered by these new models? The answer to this question would play a pivotal role in adopting or rejecting a new risk marker into clinical decision making algorithms (5). Merely demonstrating a statistically significant association of a new biomarker with certain medical condition is not enough (6-9). The performance of prediction models can be assessed using a variety of methods and metrics. Traditionally, a model predictive performance has been assessed from two perspectives, first, discriminatory predictive power and second, calibration. Discriminatory predictive power of a logistic regression model is usually assessed by calculating the area under the receiver operating characteristic curve. The calibration of logistic models is usually tested by calculating Hosmer-Lemeshow χ².

Several new measures have recently been proposed. Among which the most commonly adopted and employed are reclassification tables, net reclassification improvement (NRI), and integrated discrimination improvement (IDI) for binary outcome (2, 5, 7-11).

Commonly-used, user-friendly statistical packages (e.g. SPSS) are yet to provide calculations for novel predictive performances statists. Many studies, thus, do not make any notice of the novel statistics. Open-source statistical packages can be used to calculate novel statics. They need, however, some knowledge of programming. This has rendered their usage limited.

The STATA could be counted among the softwares that provide users with an environment where new statistical analysis, which has not incorporated into the original software, can be performed by users who have some knowledge of programming. Furthermore, user-developed modules could be incorporated into the original software and utilized by researchers with no programming skills, provided that the module has been developed in the standard format of the STATA.

2. Objectives

We wrote, therefore, "adpred" command, in the STATA standard format, for calculating NRI and IDI for logistic regression models. The module is easy to run and can be used by researches with few knowledge of programming. The help file has also been written in the standard STATA format and can be found in the same as was other statistical modules. Examples, using real data, have been incorporated in the program and can be found in the standard STATA format.

3. Materials and Methods

3.1. Added Predictive Ability for Logistic Models

3.1.1. Integrated Discrimination Improvement

IDI has been defined by Pencina et al. (2) as:

Equation 1.

Where IS is integral sensitivity over all cutoff values and IP is integral “one minus specificity,” “new” refers to the prediction model incorporating the new biomarker and “old” refers to the prediction model that does not. The arguments in the first presentation assumed nested models. However, IDI is applicable to situation where aim is to compare any two models with possibly different predicators and different analytic techniques, as far as predicated probabilities are calibrated to the same incidence or prevalence rate (12). Pencina et al. (2) provided the following estimator for IDI:

Equation 2.

P̅ is the average estimated probability of an event. An average is taken for people in sample who experienced an event (“event”) and for those who did not (“nonevents”). Rearranging terms we obtain:

Equation 3.

Which is the difference in discrimination slope proposed by Caputo et al. (13). The magnitude of discrimination slopes and their difference is tends to be small (14), 15). This could be more conspicuous when the incidence or prevalence of the event of interest is relatively low (16). Considering the definition mentioned above, one could define relative IDI as the increase in discrimination slopes divided by the slope of the old model. As such, relative IDI could be estimated as follows:

Equation 4.

Hereafter, we refer to IDI as absolute IDI.

3.1. 2. Net Reclassification Improvement

To obtain NRI, predicted probability based on the basic (old or without new biomarker or new risk factor) and enhanced (new or with new biomarker or new risk factor) are classified into three categories (17); these two cross-classification are then cross-tabulated. The reclassification of people who develop and those who do not develop events is to be considered separately. Any ‘upwards’ movement across classes for those with the event (i.e. event group) implies improvement; whereas, any ‘downwards’ movement across classes indicates worse reclassification. The interpretation is opposite for those who do not develop event (the event nonevent group). The NRI will be a sum differences in proportion of individuals moved up minus proportion of those who moved down among event subjects, and the proportion of individuals moved down minus proportion of those who moved up among nonevent subjects. The NRI as such quantifies the improvement in reclassification. If assign 1 for each upwards movement and -1 for each downwards movement, and 0 for no movement in categories, the NRI can be estimated as:

Equation 5.

where v (i) is the above-defined movement indicator.

In general, it is not recommended to use more than three categories unless they are already established and there are rigorous justifications to do so. For situation where finer portioning is needed, Pencina et al. (17) have suggested the cutpoint-free NRI. The definition of cutpoint-free NRI remains consistent with cutpoint-based formula defined above with the only difference in the meaning of the upwards and downwards reclassification (17).

3.2. Commands

3.2.1. “Adpred” Command

3.2.1.1. Syntax

Adpred depvar oldrisk newrisk, cutpoint (numlist).

Depvar represents dependent binary outcome variable.

Oldrisk is the variable that represents the risk calculated based on the baseline model.

Newrisk is the variable that represents the risk calculated based on the enhanced model.

Numlist is the list of risk cutoff points for cutpoint-based NRI.

3.2.1.2. Description

“adpred” calculates absolute and relative IDI, as well as cutpoint-based and cutpoint-free NRI. For cutpoint-based NRI to be calculated cutoff-points of risk should be specified by users.

3.2.1.3. Option

“Cut (numlist)” gives the numbers that present cutpoints of risk based on the old model at which new model is to be evaluated.

4. Results

4.1. Example

4.1.1. Study Population

We used a real data set of the Tehran lipid and glucose study (TLGS) to predict incident cardiovascular diseases (CVD). Detailed descriptions of the TLGS have been reported elsewhere (18, 19). We used data on 4 052 women with complete data on covariates, contributing to a 42 659 person-year follow-up. The median follow-up time was 11.5 years, at the time of the current study.

4.1.2. Statistical Analysis

In the analysis of outcomes (CVD), CVD we used the logistic regression model. The baseline model was developed based on the age, smoking, systolic blood pressure, use of anti-hypertensive drugs, total and HDL cholesterol, smoking, and diabetes. Improved model was developed by adding family history of premature CVD to the baseline model to basic logistic model and a family history of premature CVD, waist circumference, and fasting plasma glucose to the basic logistic model.

We set the statistical significance level at a two-tailed type I error of 0.05 and used Stata version 12.0 (Stata Corp, College Station, Texas USA) for all statistical analyses.

4.1.3. Assessment of Model Performance

We generally use several criteria to compare the overall predictive values of alternative models. However, for the current paper to be succinct and more focused on the STATA module we have curtailed other measures and herein are presenting added predictive capacity.

4.1.4. Added Predictive Capacity

Absolute and relative IDI and cut-point-based and cut-point-free NRI were used as measures of predictive ability added to the baseline model by paraclinical parameters (2). Bootstrapping method was implemented in order to obtain 95% confidence intervals (95% CIs).

Improvement in the discriminative capacity of prediction models can be quantified in numerous ways. A natural approach takes the difference in discrimination metrics between the models with and without the new predictor. The ΔAUC (difference in the area under the receiver operating characteristics curves of the prediction model with and without new marker) is produced in this manner and so is the IDI, defined as a difference in discrimination slopes. The relative IDI can be calculated as the ratio of IDI over the discrimination slope of the model without the new predictor (generally referred to as the baseline model). Integrated discrimination improvement can be seen as continuous version of NRI with probability differences used instead of categories. Alternatively, it can be defined as a difference in discrimination slopes. Discrimination slope in the binary context is defined as difference of mean predicted probabilities of events and nonevent. The cutpoint-free NRI, is obtained when a study is focused on the relative increase in the predicted probabilities for individuals who have experienced events and the decrease for individuals who do not.

Tables 1 presents the novel analysis obtained from Stata (please see also appendix for further details). As is evident the table, adding the family history of premature CVD to the baseline model does not improve the prediction capacity of the baseline model in terms of IDI or NRI. Herein, Pvalues are pertinent to testing the null hypothesis that the magnitude of the increase in the predictive capacity of the baseline model conferred by a novel marker added to the baseline model is zero. All P-values are greater than 0.05 that is to say the improvement observed in the predictive capacity of the enhanced model based on the IDI and NRI is not statistically meaningful.

The Framingham’s “general CVD risk” algorithm incorporated age, systolic blood pressure, using blood pressure lowering drugs, total and high-density lipoprotein cholesterol, smoking, and diabetes. The enhanced model was developed by adding a family history of premature CVD, waist circumference, and fasting plasma glucose to the basic Framingham’s “general CVD risk” algorithm components.

5. Discussion

When examining the clinical relevance of a new risk biomarker or when examining if the predictive power of a currently available predictive model can be augmented by new biomarker (s), NRI and IDI can be very informative. We hope by using the packages provided herein novel analysis will be more extensively utilized in studies aimed at examining the predictive ability of prediction models or clinical usefulness of new biomarkers.