1. Background
Bone age assessment (BAA) is defined as a clinical procedure for evaluating the skeletal maturity of pediatric patients (1). Discrepancies between bone and chronological age indicate abnormal skeletal development. Hand-wrist radiographs, including images obtained using the Greulich-Pyle (GP) method (2) and Tanner-Whitehouse 3 (TW3) approach (3), are commonly used to determine skeletal maturity. Generally, GP is an atlas-based method, which compares a patient’s radiograph with an atlas of representative age, whereas TW3 is a scoring system based on the osseous stages and events at each level.
The deep learning technology based on artificial neural networks has rapidly evolved in recent years, particularly in the field of medical imaging (4, 5). Convolutional neural networks (CNNs) allow for a successful classification of thoracic abnormalities (6), neuroanatomic structure segmentation (7), detection and classification of breast cancer (8), and performance of BAA (9-12). A BAA is an ideal method for an automated image evaluation, as a single study contains few images, and the results are relatively standardized. Therefore, various automatic and deep learning methods have been proposed for BAA.
Most of the proposed methods used for BAA employ a deep learning approach on the whole hand (9-11, 13, 14) or an extracted region of interest (ROI) (12). Although these methods enhance the clinical efficacy and accuracy of BAA, their application in young children is insufficient, as some previous studies excluded patients under the age of 4 years (9, 11), and others lacked a carpal bone analysis (12), which is important for skeletal maturity assessment in infants and toddlers (3). Generally, the carpal bones play an important role in determining the bone age of infants and toddlers during pre-puberty before these structures begin to overlap (3). Besides, since the survival rates of infants in all preterm and birth weight categories have improved (15), a BAA is needed in young children.
2. Objectives
In this study, we proposed augmenting BAA by integrating a carpal bone analysis to improve its accuracy in young children and to achieve a similar BAA accuracy for all age groups.
3. Patients and Methods
This study was approved by the institutional review board and the ethics committee of our institution (IRB No. 2020AN0475). The need for informed consent was waived, because data were collected retrospectively and analyzed anonymously.
3.1. Datasets
Two public datasets were used for training in this study. The first dataset was obtained from the Radiological Society of North America (RSNA) pediatric bone age 2017 challenge and included 14,236 hand radiographs from Stanford University and Colorado University (16). The second dataset was retrieved from the digital hand atlas (17), which includes 1,375 hand radiographs from the University of Southern California. We used 10% of the datasets as the validation set and 200 images as the test set.
For external validation, a total of 453 participants were selected through stratified random sampling among patients who had visited the pediatric department and underwent left-hand radiography in our institution. Individuals with congenital anomalies (i.e., Down’s syndrome, Noonan syndrome, congenital adrenal hyperplasia, and methylmalonic acidemia) were excluded, along with individuals with poor-quality images. The reference standard for the external validation set was the average independent bone age estimated by three reviewers (reviewer 1, a pediatric endocrinologist with 29 years of clinical experience, and reviewers 2 and 3, musculoskeletal radiologists with 19 and 12 years of clinical experience, respectively). The three reviewers independently estimated the bone age based on the GP Atlas, with the first digit after the decimal point (y). The reviewers tried to appraise the bone age according to the GP standard if possible; nonetheless, they were allowed to use the calculated median age based on their level of experience. If there was a discrepancy of more than two years, the image was re-evaluated until consensus was reached.
3.2. Model Development
The proposed BAA model relied on hybrid TW3 and GP artificial intelligence (AI)-based automatic bone age measurements (18). The proposed BAA method is illustrated in Figure 1. First, multiple regions based on TW3 were automatically detected using the CNN algorithm. Next, each ROI and the holistic hand image were automatically classified based on the maturity level, using the CNN algorithm. To achieve greater accuracy and detailed evaluation than the nine stages of TW3, in our model, we applied 34 stages with six-month intervals of 1.5 to 18 years in age for maturity. For the maturity classification of each CNN model training, we used the physicians’ ratings as the reference standard. Finally, each ROI and all holistic hand features were integrated and classified to estimate the final bone age for the input image.
Figure 1.
An overview of the hybrid model of Greulich-Pyle (GP) and modified Tanner-Whitehouse 3 (TW3) methods for bone age assessment with two different regions of interest (ROIs)
To compare the accuracy of BAA, the model was applied to two separate models with different radiographic areas. Our proposed model (M1) used eight regions (radius, ulna, distal phalanges, middle phalanges, proximal phalanges, metacarpal of the third digit, metacarpal of the first digit, and carpal region), whereas the former model (M2) applied seven regions (radius, ulna, distal phalanges, middle phalanges, proximal phalanges, metacarpal of the third digit, and metacarpal of the first digit) without a carpal region (Figure 2).
Figure 2.
Comparison of the regions of interest (ROIs) of two bone age assessment (BAA) models. A, The model with a carpal bone analysis, including DP, MP, PPMC, MC1, radius, ulna and carpal bones; B, The model without a carpal bone analysis, including DP, MP, PPMC, MC1, radius, and ulna. DP, distal phalanges; MP, middle phalanges; PPMC, proximal phalanges/metacarpal; MC, metacarpal
The entire BAA procedure was fully automated. The model was implemented using an open-source machine-learning library (TensorFlow Version 0.9.0; Google, Mountain View, CA, USA).
3.3. Statistical Analysis
To validate the reference standard, the intraclass correlation coefficient (ICC) was measured for the assessment of interobserver agreement among the three reviewers by considering the following values and levels of agreement: 0 - 0.20, poor; 0.21 - 0.40, fair; 0.41 - 0.60, moderate; 0.61 - 0.80, substantial; and 0.80 - 1, almost perfect (19). The results of the two BAA models and the findings of the reviewers are summarized as mean, standard deviation (SD), median, minimum, and maximum values. To measure the predictive accuracy of the models, the root mean square error (RMSE) and mean absolute difference (MAD) of each model were measured as model performance metrics; the latter was calculated based on a 95% confidence interval (CI). A BAA was considered equivalent to the reference standard when the upper limit of the 95% CI of MAD was < 0.5 years. Overall, the smaller the MAD is, the closer the predictions are to the ground truth.
The absolute differences between the two models were compared via paired t-test, and Bland-Altman plots (20) were used to correlate the predictions to the ground truth. Additionally, the MAD of each model was compared between sex and age groups with respect to four major developmental stages of the skeletal system: Pre-puberty (males ≤ 9 years; females ≤ 7 years), early and mid-puberty (9 years < males ≤ 14 years; 7 years < females ≤ 13 years), late puberty (14 years < males ≤ 16 years; 13 years < females ≤ 15 years), and post-puberty (males > 16 years; females > 15 years) (9). P-values less than 0.05 were considered statistically significant. All other analyses were conducted in SPSS Version 20.0 for Windows (IBM SPSS Statistics for Windows Version 20.0, released in 2011; IBM Corp., Armonk, NY, USA) and SAS Version 9.4 (SAS Institute Inc., Cary, NC, USA).
4. Results
Figure 3 presents the age distribution of the participants. According to the developmental stage distribution, the largest group included 339 participants in early and mid-puberty (74.8%; 74 males and 265 females), while 53 patients (11.7%; 42 males and 11 females) in pre-puberty were included in this study. The ICC (95% CI) of reviewers 1 - 3 was 0.993 (95% CI: 0.990 - 0.995), which is sufficiently high, allowing for the use of the average BAA value as a reference standard.
Figure 3.
Age distribution of the external validation set with respect to four major developmental stages of the skeletal system: Pre-puberty (males ≤ 9 years; females ≤ 7 years), early and mid-puberty (9 years < males ≤ 14 years; 7 years < females ≤ 13 years), late puberty (14 years < males ≤ 16 years; 13 years < females ≤ 15 years), and post-puberty (males > 16 years; females > 15 years)
A performance summary of the two models on the test dataset is presented in Table 1. The mean bone age ± SD of the M1 model, M2 model, and reference standard was 11.12 ± 2.60, 11.14 ± 2.59, and 11.19 ± 2.63 years, respectively. According to the Bland-Altman plots, more than 95% of the predictions across both models were within one year of the ground-truth age (Figure 4). In the M1 model, there were 19 predictions with an absolute difference of more than one year (range, 1.01 - 1.99 years), whereas in the M2 model, there were 22 predictions (range, 1.01 - 1.95 years). The cumulative percentages of absolute differences within six months between the reference standard and the models were 72.4% and 70.6% for M1 and M2 model, respectively.
Table 1.The Results of Bone Age Assessment
| Bone age, years | Total (n = 453) | Males (n = 148) | Females (n = 305) |
|---|---|---|---|
| Reference standard bone age by the three reviewers | 11.19 ± 2.63 | 11.10 ± 3.83 | 11.23 ± 1.78 |
| Median | 11.5 | 12.0 | 11.5 |
| Range, min-max | 2.0 - 17.0 | 2.0 - 17.0 | 3.0 - 16.0 |
| Automatic BAA using the model with a carpal bone analysis (M1 model) | 11.12 ± 3.05 | 11.10 ± 3.79 | 11.13 ± 1.76 |
| Median | 11.41 | 12.11 | 11.34 |
| Range, min-max | 2.14 - 16.8 | 2.14 - 16.88 | 2.97 - 15.22 |
| Automatic BAA using the model without a carpal bone analysis (M2 model) | 11.14 ± 2.59 | 11.13 ± 3.76 | 11.15 ± 1.77 |
| Median | 11.46 | 12.11 | 11.31 |
| Range, min-max | 2.09 - 16.86 | 2.09 - 16.86 | 2.85 - 15.39 |
Abbreviations: BAA, bone age assessment; SD, standard deviation.
a Values are expressed as mean ± SD unless otherwise indicated.
Figure 4.
The Bland-Altman plots indicate agreement between each model estimate and the ground truth. A, M1 model with a carpal bone analysis; and B, M2 model without a carpal bone analysis. Black lines represent 1.96 standard deviation of the difference.
The MAD between the models and the reference standard was 0.366 year for the M1 model (95% CI: 0.337 - 0.395) and 0.388 year for the M2 model (95% CI: 0.358 - 0.418), both of which were below 0.5 year (Table 2). The RMSE value was 0.483 for the M1 model and 0.505 for the M2 model. The M1 model achieved a higher accuracy with a lower MAD compared to the M2 model for all age groups (P < 0.001). The M1 model yielded better predictions than the M2 model in all age groups, regardless of sex (P = 0.002 for males and P = 0.003 for females).
Table 2.Comparison of Absolute Differences of Automatic Bone Age Assessment with a Carpal Bone Analysis (M1) and Without a Carpal Bone Analysis (M2)
| M1 | M2 | P-value | |||||
|---|---|---|---|---|---|---|---|
| MAD | 95% CI | RMSE | MAD | 95% CI | RMSE | ||
| Males | 0.355 | 0.305 - 0.406 | 0.471 | 0.389 | 0.334 - 0.444 | 0.515 | 0.002 |
| Pre-puberty | 0.364 | 0.362 - 0.465 | 0.485 | 0.441 | 0.326 - 0.557 | 0.573 | 0.008 |
| Early and mid-puberty | 0.319 | 0.246 - 0.392 | 0.446 | 0.338 | 0.216 - 0.415 | 0.473 | 0.064 |
| Late puberty | 0.432 | 0.325 - 0.540 | 0.505 | 0.431 | 0.316 - 0.546 | 0.513 | 0.938 |
| Post-puberty | 0.415 | 0.072 - 0.759 | 0.512 | 0.459 | 0.069 - 0.848 | 0.570 | 0.333 |
| Females | 0.371 | 0.335 - 0.407 | 0.488 | 0.388 | 0.352 - 0.423 | 0.500 | 0.003 |
| Pre-puberty | 0.540 | 0.101 - 0.979 | 0.824 | 0.618 | 0.211 - 1.024 | 0.845 | 0.022 |
| Early and mid-puberty | 0.370 | 0.335 - 0.406 | 0.473 | 0.386 | 0.350 - 0.421 | 0.484 | 0.008 |
| Late puberty | 0.270 | 0.174 - 0.366 | 0.364 | 0.275 | 0.177 - 0.373 | 0.370 | 0.839 |
| Total | 0.366 | 0.337 - 0.395 | 0.483 | 0.388 | 0.358 - 0.418 | 0.505 | < 0.001 |
Abbreviations: MAD, mean absolute difference; CI, confidence interval; RMSE, root mean square error.
According to the developmental stage and sex distribution, the M1 model showed a lower MAD compared to the M2 model in pre-pubertal males, pre-pubertal females, and early and mid-pubertal females (P = 0.008, P = 0.022, and P = 0.008, respectively). Particularly, the M1 model reduced the 95% CI upper limit of MAD to < 0.5 for pre-pubertal males (MAD = 0.364; 95% CI: 0.362 - 0.465), which could be considered equivalent to the reference standard.
5. Discussion
The present study investigated the clinical significance of carpal bone analysis in improving the accuracy of BAAs. Additionally, a hybrid model was established by incorporating a carpal bone analysis using an open dataset. Both models demonstrated human expert-level accuracy, with an upper 95% CI of < 0.5 for the MAD between the BAA models and the reference standard. The MAD values for the M1 and M2 models were 0.366 and 0.388 years, respectively, which are similar to or lower than those reported in previous research (21-23). The current study showed that the model integrating a carpal bone analysis was more accurate than the model without a carpal bone analysis. Particularly, the model using a carpal bone analysis showed significantly improved accuracy in pre-pubertal patients.
The classical method of BAA in children is based on bone growth in the phalanges, carpal bones, and wrist joints. Carpal bone maturity varies widely, and the analysis of this variable does not provide accurate or significant information in patients > 7 years of age (24). In girls aged >6 years or boys > 8 years, a phalangeal feature analysis yields more reliable information than a carpal bone analysis; accordingly, most recent deep learning methods do not utilize the carpal bones for analysis (9-14). However, carpal bones are a vital and distinctive feature of skeletal maturity, which provide accurate and significant information for determining the bone age of young children (3). Combined with the existing phalangeal and wrist joint analyses, a carpal bone analysis significantly improves the accuracy of BAAs in young children.
Several studies have used carpal bone extraction to calculate the bone age (25-27). Most of these studies focused on classical image segmentation rather than bone assessments, without applying a deep learning algorithm. Meanwhile, the accuracy of BAAs was unacceptable. In this regard, Hao et al. (28) exploited carpal bones using a regression CNN to evaluate the bone age. Although they achieved an accuracy of 90.15% within six months from the ground truth for males, their study was only performed on individuals with a chronological age of 0 - 6 years, and a limited BAA was performed for children of all ages.
Additionally, Iglovikov et al. (29) trained several deep network architectures using different parts of images (whole hand, carpal bones, and metal carpals/proximal phalanges) to evaluate the contribution of various bones to the model performance. Moreover, the linear ensemble of these regional models outperformed all the aforementioned models, which is consistent with our results. However, our study reported a higher accuracy and a lower MAD compared to the study conducted by Iglovikov et al. (29) with the lowest MAD at 6.1 months.
Recently, the survival rates of infants in all preterm and birth weight categories have improved owing to advances in obstetric and perinatal care (15). As the survival rates of preterm and low-birth-weight infants increase, concerns regarding the developmental outcomes and growth of survivors also increase. Most infants exhibit an accelerated compensatory growth pattern (known as “catch-up growth”), which is usually completed by two years of age (30). Nevertheless, in the absence of compensatory growth, infants are unlikely to grow to the extent of their peers and reach their target height in adulthood. Consequently, the need for recombinant human growth hormone therapy has increased in these young children (31), and the need for BAA for these young children has also increased. Our model incorporating a carpal bone analysis yielded better predictions compared to the model without a similar analysis in prepubertal men, suggesting its improved accuracy in young children.
Our hybrid model overcame the limitations of GP and TW3 methods by focusing on regions that are highly related to changes in bone maturity and by applying finer-grained maturity stages compared to TW3, yielding a reliable and accurate bone age estimate (18). Similar to our approach, the BoneXpert (Visiana Aps, Holte, Denmark; http://www.boneexpert.com), an automated commercially available BAA system (recently released, version 3.0), is used to evaluate the bone age using both GP and alternative TW2 methods (32). However, unlike our model, the BoneXpert is based on a feature extraction technique, which reconstructs the boundaries of 15 bones (i.e., metacarpals, phalanges, distal radius, and ulna) (33). Overall, the present results showed that the accuracy of bone age predictions improved remarkably for males versus females, which is consistent with the results of a previous carpal bone analysis (25, 29), probably due to the fact that girls mature sooner than boys. This finding is evident in the dataset applied in the experiment, where the carpal bones of girls ossified at a much earlier age than those of boys.
The present study had several limitations. First, although previous studies reported ethnic differences in the growth patterns of certain age groups (34), our study was limited to patients of a single ethnicity. Therefore, a prospective multicenter study with a large sample size is required. Second, our model required radiographs of diagnostic quality. The model itself did not evaluate the radiographic quality, such as the rotation or incomplete filming of the left hand. Third, our model could not detect traumatic or congenital deformities. Finally, the reference standard for the external validation set was based on the GP Atlas, which is inherently limited to young children. There are also differences in the nutritional and ethnic characteristics of children today compared to those during the 1930’s and 1940’s, which were used to generate standards.
In conclusion, the present results showed the improved accuracy of the hybrid GP and modified TW model for BAA by integrating a carpal bone analysis. Particularly, the model utilizing a carpal bone analysis showed a significantly improved predictive ability in pre-pubertal patients; therefore, it can be sufficiently used in young children.
