Abstract
Background:
Nowadays, outcome prediction models using logistic regression (LR) and artificial neural network (ANN) analysis have been developed in many areas of healthcare research.Objectives:
In this study, we have compared the performance of multivariable LR and ANN models, in prediction of psychological symptoms six months after mild traumatic brain injury.Methods:
In a prospective cohort study, information of 100 mild traumatic brain injury patients, during a six months period between 2014 and 2016 were included. Data were divided into two training (n = 50) and testing (n = 50) groups, randomly. 300 ANNs and LRs were studied in the first group and then the predicted values were compared in the second group using the two final models. The receiver operating characteristic (ROC) curve and accuracy rate were used to compare these models.Results:
The results showed that accuracy rate for the neural network model was 90.65%, while it was 75.96% for the LR model.Conclusions:
The ANN models appeared to be more powerful in predicting psychological symptoms versus the LR models.Keywords
Artificial Neural Network Logistic Regression Mental Disorder Mild Traumatic Brain Injury Prediction Principle Component Analysis Psychological Symptom
1. Background
The main goal of modeling is to determine relationships between variables as well as effective variables and predictions (1). According to Wyatt and Altman, a predictive model must be simple to calculate, have an apparent structure and be tested in independent data sets with evidence of generality. The most considered subject in medical and epidemiological researches is human health (2), and available methods of modeling and prediction in classic statistics are not practical enough due to their limitations; so, applying these sorts of models brings some problems and limitations (3). For example, in LR, the assumption of independence of errors and variables is essential (4). Under this condition, if the relevant data is complicated, the model’s assumptions may not be true anymore, thus providing methods that can predict with the lowest error and most confidence, and can also solve these kinds of problems seems to be beneficial and valuable. One way to solve such problems is using artificial neural model which has been used increasingly during last decades (5). These models are in use with different fields of medicine including cancer anticipation (6), prediction of death after gastric surgery (7), etc. Artificial neural networks (ANN) are adaptive models of analyzing data which are inspired by the functioning processes of the human brain (8). They are particularly suited for solving problems of the non- linear types to reconstruct the approximate rules that put a certain set of data-which describes the problem being considered - with a set of data which provides the solution (9).
In recent years, there has been significant improvement in head trauma surgery as well as in hospital services and trauma care and recovery. However, mild traumatic brain injury patients mostly show a collection of physical, psychological and nervous problems and also symptoms of psychology and weeks after the onset of injury (10-12). This problem can, in turn, lead to chronic disabilities; however, it doesn’t receive enough attention and appropriate treatment due to its biological, mental and social complexity. Normally, investigating traumatic brain injuries by an organized method is difficult because injury mechanisms can cause incongruous clinical manifestations due to distribution of brain injury (13). Thus, most of experiments have focused on identifying the nature and effect of consequent physical problems caused by mild traumatic brain injury. However, investigating the anticipation of psychological symptoms following mild traumatic brain injury is still in its infant stages and requires more growth and development. Thus, providing statistical models which enable us to model psychological symptoms is a matter of great importance.
2. Objectives
This study intends to model the psychological symptoms after mild traumatic brain injury using LR and neural network forecasting and offer comparison of the two models.
3. Materials and Methods
3.1. Patients and Study Design
This prospective cohort study was conducted on 100 mild traumatic brain injury patients (13 to 15 GCS), hospitalized in neurosurgery unit. Samples were taken from 15-to-65-year old individuals of both sexes via non-probability sampling method, and were studied after receiving the approval of ethic committee of the hospital and conscious agreement of patients. The exclusion criteria were as follows: patients with any evidence of spinal cord injury, or with communication problems, who use anti- epileptic drugs and have history of seizure and/or psychotic diseases as well as patients with mental retardation. In addition those who were not willing to contribute to the study were also excluded.
In this study, the variable such as having psychological symptoms (yes, no) as an output variable and 14 factors, including age, sex, marital status (married, unmarried), education level, financial status, smoking habits (smoker, nonsmoker) and having history of mental illness in the immediate family, hospitalization in neurosurgery unit, trauma, underlying disease, use of psychological drug, anesthesia, use of alcohol and drug substances , as predictor variables, were used to compare the performance of these two models.
In the first phase, hospitalized patients were examined by the neurosurgeon, and based on the defined criteria, if they were eligible for the study, their data, both demographic and clinical, were entered into the questionnaire. Then, after 6 months, they were asked to attend advising and neurological evaluation center to complete BSI questionnaire. In order to meet consistency in completion of the tests, questions were uttered one by one, then, their oral answers were recorded in the related section of the questionnaire by a psychology expert (MA).
In this study, Instrument that has been used was brief symptom inventory (BSI) questionnaire which is the short form of the SCI-90-R questionnaire. This questionnaire includes 53 questions which evaluate psychological symptoms, and it can be used to separate healthy individuals from the patients. In 1973, Hessel et al. (14) introduced this questionnaire which consists of 9 dimensions as following: somatization, obsessive-compulsive, interpersonal sensitivity, depression, anxiety, anger, anxiety phobia, paranoia, psychoticism. After completion of the questionnaires, the score for each questionnaire was calculated, and also, individuals were classified as having mental health problems with the T values of the global severity index (GSI) ≥ 63. The measure has been shown to be reliable and valid in mild traumatic brain injury patients. Internal consistency was determined by Cronbach’s alpha coefficient and estimated to be 0.86 (15) and in another study in Iran, reliability and internal consistency were 0.71 and 0.96 (16).
3.2. Statistical Analysis
The whole dataset was split into two parts, ANN and logistic models were developed using the first part training (n = 50) and they were validated (or testing) in the second part (n = 50) which the latter have been called the test set.
3.3. ANNs Modeling
The most popular ANN, a supervised multilayer perceptron comprised of an input, a hidden, and an output layer which was performed by MATLAB software. The number of nodes in the input and output layers is determined based on the data structure, but finding the optimum number of hidden node is a crucial step in the architecture of the neural network. The number of hidden nodes enhances the learning capability of the network and leads to over fitting of the training data. Moreover, increases nodes in the hidden layer may cause ANN to learn the training examples too correctly while they cannot be generalized to new cases. These methods need two distinct sets. The first part is training set that help to learn patterns presented in the data and the second part testing (or validation) set to evaluate over-fitting. Therefore, in the first part, the data was divided into two training (n = 50) and testing sets (n = 50), randomly. This procedure was repeated 300 times. In this study, the different networks trained by changing the number of nodes in the hidden layer and compared the performance of these networks by root mean square (RMS) in the testing set. We fitted 300 ANN models with 7 to 12 hidden nodes in our architecture. Finally, the network with minimum RMS was selected. The networks were trained by back-propagation algorithm with learning rate in the range of (0.1 - 0.5) and momentum value in the range of (0.1 - 0.3) on the training set.
3.4. LR Modeling
Procedure of splitting data was like ANN models. We fitted 300 LR models to our data. In this subsection, Stata software program (version.10.0) was used for modeling a binary response variable. Then, in the testing set, the ROC curve and Akaike information criterion (AIC) criteria were calculated to choose the best LR model. P values of 0.05 or less were considered significant.
4. Results
The modeling process in ANN was performed using training data set. Table 1 consists of architecture, RMS, area under ROC curve and percent of inaccurate prediction for six ANN models. The structure of six models was with architectures: (14/7/2), (14/8/2), (14/9/2), (14/10/2), (14/11/2) and (14/12/2) based on the 14 input variables, hidden layer (middle layer) that was chosen with 7 to 12 neurons and 2 neurons according to the output variable with two responses (having psychological symptom (yes, no)). The best model was selected based on minimum amount of RMS, maximum value of area under ROC curve and minimum percent of inaccurate prediction indices. By fitting different three-layer ANN models for six structures on the basis of 7 to 12 neurons in middle layer, the model with nine neurons in hidden layer, learning rate of 0.05, momentum of 0.9 and back-propagation learning algorithm was selected as the most suitable model for data prediction in which RMS was 0.1029, and accuracy rate of the model was 90.65 % (Table 1).
Choosing the Best Neural Network Model for Observations
| Architecture (Input/Middle/ Output) | RMS | Area Under ROC Curve | Percent of Inaccurate Prediction |
|---|---|---|---|
| (14/7/2) | 0.1151 | 0.802 | 11.16 |
| (14/8/2) | 0.1072 | 0.831 | 10.10 |
| (14/9/2)a | 0.1029 | 0.869 | 9.35 |
| (14/10/2) | 0.1054 | 0.853 | 9.78 |
| (14/11/2) | 0.1293 | 0.801 | 11.84 |
| (14/12/2) | 0.1312 | 0.795 | 11.97 |
Next, the LR model was fitted to the data and the appropriate model was selected. AIC criteria for the best model were 103.05. Stepwise LR model was used in order to select the significant variables. Only education (OR = 4.28), financial status (OR = 2.49) and age (OR = 1.07) were significant and have remained in the final model. However, based on the final neural network model, variables such as having previous history of trauma, drug use and level of education were the most affected (by the top 5% normal effect percent).
Situation of Selected Variables in LR and ANN Models
| Level of Significance of Variables in Logistic Regression Model in Descending Order | Level of Significance of Variables in Neural Network Model in Descending Order |
|---|---|
| Level of educationa | History of traumab |
| Financial statusa | History of using substanceb |
| Agea | Level of educationb |
| History of being hospitalized in neurosurgery unit | History of mental illness in the immediate familyb |
| Sex | History of using alcohol |
| Job | History of using psychological drug |
| Marital status | Marital status |
| History of mental illness in the immediate family | Sex |
| History of trauma | Age |
| History of underlying disease | Job |
| History of using psychological drug | History of anesthesia |
| History of anesthesia | Financial status |
| History of using alcohol | History of underlying disease |
| History of using drug substance | History of being hospitalized in neurosurgery unit |
One of the diagnosis criteria for comparing the models is the area under the ROC curve that for which values 0 to 0.5 show a random classification, and values 0.5 to 1 indicate the model’s total diagnosis capacity. According to table 3, the area under the ROC curve in the experimental set for logistic regression and neural network models were obtained as 69.5 % and 86.9 %, respectively.
Results of Comparing 300 Pairs of Logistic Regression (LR) and Neural Network Models (ANNs)
| Index | LR | ANN | P Value |
|---|---|---|---|
| Area under the ROC curve | 0.695 (0.571 - 0.820)a | 0.869 (0.785 - 0.952) | 0 |
| Accuracy rate | 75.96 (75.23 - 76.35) | 90.65 (90.31 - 91.01) | 0 |
Figure 1 illustrates the ROC curve for both models. Another index of fitness is the classification of accuracy rate that is the proportion of cases correctly classified in each group. Values 0.5 to 1 show the ability to recognize the models. The correct predicted proportion in Table 3 indicates the proper classification of the neural network model compared with LR. According to this table, the accuracy rate was 90.65 % for the final neural network model and 75.96 % for final LR model.
The ROC Curve According to the Final LR and Neural Network Model
5. Discussion
Studies which have been conducted on statistical modeling of post-traumatic psychological symptom (15, 17-20) mostly aimed to examine the effective factors on psychological symptoms by using the LR model. However, in the present study, we aimed to investigate the prediction of accuracy and precision of LR and neural network models for the first time. The results show ANN performs well compared with its conventional alternative.
ANN is not limited in its formulated function and can capture complicated patterns more flexibly than the LR (21, 22). ANN is capable of finding patterns despite the missing data.
ANNs are semi parametric classifiers, which are more flexible than parametric models. A very copacetic characteristic of these models is their learning procedure, i.e. learning by examples. When there is little information of the actual connection, this specific makes them more powerful in pattern recognition. Moreover, since ANN has no constraint regarding its formulated function, it is more pliable and has more stability in simulating complicated patterns than LR (23). Another favorable feature of ANN is their capability to find patterns despite of missing data. They are powerful networks, which are tolerant in searching incomplete noisy patterns.
There are some problems in practical application of ANN: we should firstly note that designing a network is not so easy and knowing the fundamental theory is necessary. Secondly, no formal techniques are available to test the relative relevance of the input variables and to make the variable selection process in non-linear methods (24). Thirdly, we may find no etiologic interpretation for the calculated weights in the network as compared in the conventional models (25). No mathematical relationship is determined between target and input variables. And finally, we should know that learning an ANN is computationally time consuming which requires sophisticated software.
Despite all problems, there are some situations where the use of old-style models is impossible and we need some other alternatives. Using the old-style models requires many assumptions that may not be true in some real requests (26). Contravention of these assumptions may generate error in hypothesis testing and prediction. Since the logistic models use linear combination of variables therefore, they are not suitable for modeling multifaceted relationships.
As a general result, ANN may be the best choice when the complex dependencies and interactions exist in the dataset. In contrast, when the most important goal of modeling is causal conclusion among variables and we want to identify the effect of each variable on the response variable, LR is useful.
As a conclusion, neural network is an appropriate approach for predicting psychological symptoms in trauma patients. Thus, resulted predictions of this method can be used in classifying trauma patients which should be followed by recognizing the required healthcare resources according to patients’ needs. In addition, considering that almost all predictive models use linear and logistic methods for analysis, applying discovered nonlinear relationships in nervous system can be helpful in planning more efficient programs for screening individuals susceptible to psychological symptoms.
References
-
1.
Bashiri M, Farshbaf Geranmayeh A. Tuning the parameters of an artificial neural network using central composite design and genetic algorithm. Scientia Iranica. 2011;18(6):1600-8. https://doi.org/10.1016/j.scient.2011.08.031.
-
2.
Wyatt JCAD. Prognostic models: clinically useful or quickly forgotten? Brit M J. 1995;311:1539-41. https://doi.org/10.1136/bmj.311.7019.1539.
-
3.
Buchner A, May M, Burger M, Bolenz C, Herrmann E, Fritsche HM, et al. Prediction of outcome in patients with urothelial carcinoma of the bladder following radical cystectomy using artificial neural networks. Eur J Surg Oncol. 2013;39(4):372-9. [PubMed ID: 23465180]. https://doi.org/10.1016/j.ejso.2013.02.009.
-
4.
Agatonovic-Kustrin S, Beresford R. Basic concepts of artificial neural network (ANN) modeling and its application in pharmaceutical research. J Pharm Biomed Anal. 2000;22(5):717-27. [PubMed ID: 10815714].
-
5.
Aladag CH. A new architecture selection method based on tabu search for artificial neural networks. Expert Systems with Applications. 2011;38(4):3287-93. https://doi.org/10.1016/j.eswa.2010.08.114.
-
6.
Lv DJ, Zhang Y, Wang XY, Guo XM, Wang CY. [Application of artificial neural network to diagnosis of prostate cancer]. Beijing Da Xue Xue Bao. 2009;41(4):469-73. [PubMed ID: 19727241].
-
7.
Amiri Z, Mohammad K, Mahmoudi M, Parsaeian M, Zeraati H. Assessing the effect of quantitative and qualitative predictors on gastric cancer individuals survival using hierarchical artificial neural network models. Iran Red Crescent Med J. 2013;15(1):42-8. [PubMed ID: 23486933]. https://doi.org/10.5812/ircmj.4122.
-
8.
Andersson B, Andersson R, Ohlsson M, Nilsson J. Prediction of severe acute pancreatitis at admission to hospital using artificial neural networks. Pancreatology. 2011;11(3):328-35. [PubMed ID: 21757970]. https://doi.org/10.1159/000327903.
-
9.
Adoko AC, Yu-Yong J, Li W, Hao W, Zi-Hao W. Predicting tunnel convergence using Multivariate Adaptive Regression Spline and Artificial Neural Network. Tunnelling and Underground Space Technol. 2013;38:368-76. https://doi.org/10.1016/j.tust.2013.07.023.
-
10.
Lin MR, Chiu WT, Chen YJ, Yu WY, Huang SJ, Tsai MD. Longitudinal changes in the health-related quality of life during the first year after traumatic brain injury. Arch Phys Med Rehabil. 2010;91(3):474-80. [PubMed ID: 20298842]. https://doi.org/10.1016/j.apmr.2009.10.031.
-
11.
De Silva MJ, Roberts I, Perel P, Edwards P, Kenward MG, Fernandes J, et al. Patient outcome after traumatic brain injury in high-, middle- and low-income countries: analysis of data on 8927 patients in 46 countries. Int J Epidemiol. 2009;38(2):452-8. [PubMed ID: 18782898]. https://doi.org/10.1093/ije/dyn189.
-
12.
Garber BG, Rusu C, Zamorski MA. Deployment-related mild traumatic brain injury, mental health problems, and post-concussive symptoms in Canadian Armed Forces personnel. BMC Psychiatry. 2014;14:325. [PubMed ID: 25410348]. https://doi.org/10.1186/s12888-014-0325-5.
-
13.
Draper K, Ponsford J, Schonberger M. Psychosocial and emotional outcomes 10 years following traumatic brain injury. J Head Trauma Rehabil. 2007;22(5):278-87. [PubMed ID: 17878769]. https://doi.org/10.1097/01.HTR.0000290972.63753.a7.
-
14.
Hessel ASJ, Geyer M, Brahler E. Symptom-Checkliste SCL-90-R: Testth - eoretische Überprüfung und Normierung an einer bevölkerungsrepräsentativen Stichprobe. Diagnostica. 2001;47:27-39. https://doi.org/10.1026//0012-1924.47.1.27.
-
15.
Fakharian E, Omidi A, Shafiei E, Nademi A. Mental health status of patients with mild traumatic brain injury admitted to shahid beheshti hospital of kashan, iran. Arch Trauma Res. 2015;4(1):17629. [PubMed ID: 25866741]. https://doi.org/10.5812/atr.17629.
-
16.
Mohammadkhani P. Psychometric properties of the Brief Symptom Inventory in a sample of recovered Iranian depressed patients. Int J Clin Health Psychol. 2010;10(3):541-51.
-
17.
Vassallo JL, Proctor-Weber Z, Lebowitz BK, Curtiss G, Vanderploeg RD. Psychiatric risk factors for traumatic brain injury. Brain Inj. 2007;21(6):567-73. [PubMed ID: 17577707]. https://doi.org/10.1080/02699050701426832.
-
18.
Chong SL, Liu N, Barbier S, Ong ME. Predictive modeling in pediatric traumatic brain injury using machine learning. BMC Med Res Methodol. 2015;15:22. [PubMed ID: 25886156]. https://doi.org/10.1186/s12874-015-0015-0.
-
19.
Booth-Kewley S, Schmied EA, Highfill-McRoy RM, Larson GE, Garland CF, Ziajko LA. Predictors of psychiatric disorders in combat veterans. BMC Psychiatry. 2013;13:130. [PubMed ID: 23651663]. https://doi.org/10.1186/1471-244X-13-130.
-
20.
Perron BE, Howard MO. Prevalence and correlates of traumatic brain injury among delinquent youths. Crim Behav Ment Health. 2008;18(4):243-55. [PubMed ID: 18803295]. https://doi.org/10.1002/cbm.702.
-
21.
Ahmadizar F, Soltanianb K, AkhlaghianTabb F, Tsoulos I. Artificial neural network development by means of a novel combination of grammatical evolution and genetic algorithm. Engineering Applications of Artificial Intelligence. 2015;39:1-13. https://doi.org/10.1016/j.engappai.2014.11.003.
-
22.
Zhou Q, Kwong J, Chen J, Qin W, Chen J, Dong L, et al. Use of artificial neural network to predict warfarin individualized dosage regime in Chinese patients receiving low-intensity anticoagulation after heart valve replacement. Int J Cardiol. 2014;176(3):1462-4. [PubMed ID: 25164187]. https://doi.org/10.1016/j.ijcard.2014.08.062.
-
23.
Song JH, Venkatesh SS, Conant EA, Arger PH, Sehgal CM. Comparative analysis of logistic regression and artificial neural network for computer-aided diagnosis of breast masses. Acad Radiol. 2005;12(4):487-95. [PubMed ID: 15831423]. https://doi.org/10.1016/j.acra.2004.12.016.
-
24.
Chokmani K. Comparison of ice-affected streamflow estimates computed using artificial neural networks and multiple regression techniques. J Hydrol. 2008;349(3-4):383-96. https://doi.org/10.1016/j.jhydrol.2007.11.024.
-
25.
Seixas JM, Faria J, Souza Filho JB, Vieira AF, Kritski A, Trajman A. Artificial neural network models to support the diagnosis of pleural tuberculosis in adult patients. Int J Tuberc Lung Dis. 2013;17(5):682-6. [PubMed ID: 23575336]. https://doi.org/10.5588/ijtld.12.0829.
-
26.
Shi HY, Lee KT, Lee HH, Ho WH, Sun DP, Wang JJ, et al. Comparison of artificial neural network and logistic regression models for predicting in-hospital mortality after primary liver cancer surgery. PLoS One. 2012;7(4):35781. [PubMed ID: 22563399]. https://doi.org/10.1371/journal.pone.0035781.