Validity and Reliability of the Test of Everyday Attention for Children (TEACh) in Iranian 8-11 Year Old Normal Students

1. Background

Attention deficiency can affect all cognitive functions. Without attention, perception would be impaired, and the surrounding world will be meaningless and confusing (1).

According to Posner’s structural equation model, there are exclusive systems in the brain for attention, which can be separated from other cognitive and perception systems. Furthermore, in the attention system, there are separate and specific regions and networks that work separately from each other (2). In animal and human pathological studies, it has been concluded that the posterior parietal lobe region, the external posterior thalamus, prefrontal cortex and the upper colliculus play a key role in attention control; moreover, the anterior cingulate is especially active in selective attention, and the right hemisphere, particularly the pre-frontal cortex, is associated with the ability of sustained attention (3).

Since attention disorders can result from different brain system damages and initiate at different stages of development, there will be a variety of attention deficit patterns. This stresses the importance of assessing the attention skills for better description of weakness and strength patterns (4).

There are a variety of procedures adopted for screening and assessing attention problems involving variable neuropsychological tests. These tests, rather than focusing on attention abilities, assess multiple frontal abilities like response inhibition, flexibility, and planning/organization (4). For example, intelligence assessments focus on the average individual performance in cognitive areas (5) where children with attention deficit and hyperactivity disorder (ADHD) cannot be distinguished from normal children based on intelligence (6).

For exclusive assessment of adult attention problems, Robertson et al. designed the test of everyday attention in line with Peterson and Posner’s theory of attention network (7). In 1999, Manly et al. adapted the TEA assessment for children (8). This test is an objective measure with several advantages compared to other methods assessing attention (5).

When performing TEA-Ch, multiple sensory modalities such as vision, hearing, and motor were used, while most mental neurological tests assessing the sustained and selective attention only focus on presentation of visual stimulus (4). The TEA-Ch assesses the components of attention in play-like tasks; thus, the child will be more comfortable during the test administration and is motivated to cooperate with the examiner (6).

In Iran, there is no valid and reliable tool to exclusively assess all the components of attention. The behavioral rating scales and general cognitive performance assessments are used to identify children’s attention problems and screen for attention deficit disorder and/or hyperactivity. Selecting appropriate and standard tools to assess attention problems will facilitate early diagnosis and specialized treatment for attention disorders and other psychiatric and neurological disorders. Considering the benefits of the TEA-Ch, and the need to precise and early diagnosis of attention problems for children’s health promotion, in this study we aimed to determine the content validity, reliability and construct validity of TEA-Ch in the Iranian community of 8 - 11 year old normal children. The necessity of assessing the function of attention by specific tools such as TEA-Ch and the different nature of attention and intelligence was examined through discriminate validity (one subtype of construct validity).

2. Materials and Methods

2.1. Participants

In this methodological research, reliability, content validity and construct validity of TEA-Ch were investigated on normal 8 - 11 year old children in the local district municipality (No 13) in Tehran. Two schools for boys and girls were selected and 8 - 11 year old students were recruited from the enrollment lists. The selected children were divided into four age groups, including 8 - 9 year olds (with the average age of 8 years and 4 months), 9 - 10 year olds (with the average age of 9 years and 6 months), 10 - 11 year olds (10 years and 6 months), 11 - 12 year olds (11 years and 2 months). After performing the preliminary study, and administrating with α = 0/05 and power sure = 80%, the study size was concluded to be 96 (53 girls and 43 boys).

The inclusion criteria for the study were the age range of 8 - 11 years, lack of major behavioral and emotional disorders (through using child symptom inventory-4), lack of history of head injury and developmental delay (according to the parents report). Students who did not adequately participate in the test or failed to accomplish the test for any reason, or had vision or hearing problems were excluded from the study. All students and their parents signed the informed consent form.

2.2. Assessment Tools

Child symptom inventory-4 (CSI-4): the CSI-4 is a behavioral rating scale developed by Sprafkin and Gadow for screening behavioral and emotional disorders in 5 - 12-year-old children. The CSI-4 entails two forms of parents and teachers. The parents form has 97 items designed to screen 18 behavioral and emotional disorders. This questionnaire was normalized by Mohammad Ismail in 2007. The reliability coefficients for parents form were estimated through test-retest with a two- week interval from 0/29% in social phobia disorder up to 0.76% in conduct disorder, which were all significant at the level 0/01, with the exception of social phobia disorder. Meanwhile, the internal consistency of severity scores in the CSI suggested the correlation between the identical items and the underlying structure of the scale items. The high sensitivity and specificity of most disorders in the list, especially the parents list, indicated the criterion validity of the instrument, in which the majority of the disorders were characterized by specificity of over 0.9 and sensitivity of over 0.8 (9).

Wechsler’s intelligence scale for children-fourth edition (WISC-IV): the WISC-IV contains 15 subscales, in which the four scores are obtained for indices of verbal comprehension, perceptual reasoning, working memory, processing speed and overall IQ score. Sadeghi et al. examined the validity and reliability of this test in Iran. The overall validity coefficient of IQ was 0.91, while in the subscales the validity ranged from 0.88 (verbal comprehension) to 0.8 (processing speed intelligence) (10). In this study, the subtests of block design and picture assembly were used similar to other studies (2), while math and digit span subscales were selected by the study researchers according to Wechsler (IV) considering its relation to attention.

Wechsler introduces intelligence as a general concept that includes the ability to act purposefully, to think rationally, and to deal effectively with the environment. He chose and developed some subscales to highlight the cognitive aspects of intelligence. He believed that verbal comprehension, abstract reasoning, perceptual organization, quantitative reasoning, memory and processing speed should be measured to determine intelligence. All of these areas have been confirmed as important aspects of intelligence theories and assessments (10). Attention is not considered as a part of intelligence (11). Processing can occur independent of attention to some degree and attention is considered as an addendum for information processing. For example, during the pre-attending phase, which occurs below the level of awareness, and during the automatic processing there is no need for attention. Thus, attention is not the integral part of processing or intelligence (12).

Test of everyday attention (TEA-Ch): this test has been standardized and validated for children 6 - 16 years of age, and is particularly suitable for children with attention problems. It entails nine subtests, measuring the selective attention, sustained attention, and attention control separately. This test includes two matched versions (A and B) that can be used in parallel. The parallel version has been designed to examine test-retest reliability. Each attention factor can be screened briefly only through the first four subscales since the full assessment, using all subtests, will take approximately one hour. Depending on the child’s participation, more time may be required. Subtests 1 (sky search), 3 (creature counting) and 4 (sky search Dual Task) require more calculation (5). Table 1 demonstrates subtests in detail.

2.3. Data Analysis

To assess the qualitative content validity, we selected an expert panel, consisting of 10 occupational therapists with at least five years of work experience in the field of children with mental and cognitive problems. They were asked about the validity of the subtests individually, and a preliminary study was done to find equivalents to test items. To investigate the inter-rater reliability, two raters simultaneously scored the 35 students. To assess test-retest reliability, appropriate time interval for retesting is generally two weeks to one month after the first administration of the test (14). Eighteen students were selected to complete the TEA-Ch version B after two weeks from the administration of TEA-Ch version A. Version B is the parallel form of version A, which is designed to examine the test-retest reliability. The mean age of the participants was 9 years and 9 months. No significant differences were found between the groups in terms of age and academic achievement as compared to the total sample. ICC was calculated to determine the inter-rater reliability of the subtests. Discriminate validity was examined to determine the distinct nature of TEA-Ch and WISC-IV. All 96 children took the two tests in random order with enough break time between them, which generally lasted for two hours. As the most important point, children reported that TEA-Ch was an interesting game-like test, and not boring. Parents filled the CSI-4 questionnaire. The tests were conducted by trained experts in quiet rooms without too many distracters to help children to concentrate on tests. Due to the normal distribution of the data (Kolmograph-Simonov), the Pearson’s correlation analysis was used to determine the relationship between the TEA-Ch and the four subtests of the Wechsler’s Intelligence Scale. The study was approved by Ethics committee of Iran University of Medical Sciences.

3. Results

3.1. Validity

According to the experts’ opinion, all subtests were appropriate for use in the assessment process, except for “news text” in score dual task subtest. The “news text” was translated and adapted for the names of animals in Persian language. The adapted news text was pilot tested in six normal children, and then revised for some items according to experts’ opinion.

3.2. Reliability

The intraclass correlation coefficient (ICC) was calculated for test-retest reliability of the raw scores in each of the TEA-Ch subtests. For inter-rater reliability, the correlation coefficient of 0.4 or more was considered as acceptable (15). According to the data presented in the correlation between the test and retest of TEA-Ch in Table 2, the ICC was from 0.40 to 0.92, indicating acceptable test-retest reliability. To assess inter-rater reliability, 35 students were observed by two raters at the same time. The relationship between scores of the two raters is presented in Table 3, and it was 0.78 to 1, indicating excellent inter-rater reliability.

The Pearson’s correlation between the TEA-Ch and the subtests of the WISC-IV was calculated, and the results are presented in Table 4, showing a correlation range of -0.32 - 0.33 between the items.

4. Discussion

This study aimed to assess content validity, construct validity and reliability of TEA-Ch among normal 8 - 11 year- old students living in Tehran.

4.1. Content Validity

To check the relevancy of the subtests with Iranian culture and language, the expert panel checked all the subtests through qualitative content validity. According to expert opinions, the score dual task subtest of the news text was translated and changed in accordance with Iranian language and culture such as changing some animal names to those more familiar to Iranian children (for example, changing pigs to roasters).

4.2. TEA-Ch Reliability

The TEA-Ch was first adopted in 1999 by Manly along with the Wechsler’s intelligence scale-fourth version (WISC-IV), and was administered on 293 Australian children aged 6 - 16. For all the subtests, the ICC values reported to be 0/57 to 0/87. For score!, score dual task, and walk-don’t walk subtests, in which test ceiling led to unrealistic correlations, agreement percentage was obtained on a standard deviation for the first and second versions of the test, and it was 76/2%, 71/4%, and 71%, respectively (2). Moreover, Chan et al. (2008) implemented the TEA-Ch and the Wechsler’s intelligence scale-revised version (WISC-R) on 232 normal Chinese children aged 6 - 16. Thirty-two children from the sample group performed the retest after one month. The reliability of the five subtests was high (P = 0.01). The Spearman’s correlation for walk- don’t walk subtest was 0.433, which is close to 0.5, and considered as reliable for a small sample of 32 participants. Furthermore, the score!, map mission, and score dual task subtests achieved the correlation of 0.045, 0.386, and 0.268, respectively (6). Considering the influence of factors that changed the child’s performance (e.g., mood, noise, and other distractions), a stable instrument could be considered as clinically valuable (2). In this study, since the reliability of the test-retest was acceptable for the TEA-Ch, it was a clinically valuable instrument. However, the changes that occur in childhood limit the ability of TEA-Ch, despite the fact that this test is reliable. Test administration fatigue might have affected the performance of the students while taking the version A of the test, as because of the tasks’ novelty more time and energy was required to take the first version compared to the second version of the test. Although the raters made an effort to provide ideal conditions for administering the tests, it can be argued that daily mood and fatigue for taking a long test and children’s different levels of motivation to participate, affected the results. Another factor contributing to the test-retest reliability was the impact of learning. After implementing the version A, students were more confident and prepared to do the version B of the test. Similar to the study by Chan et al. the sky search dual task achieved the lowest amount of reliability compared to other subtests (6). The score obtained in this subtest is an outcome of calculating the child’s performance in three tasks (5), and if the child performs differently even in one of them, the final score will change, tremendously affecting the test-rest reliability. The inter-rater reliability was excellent because observation of children’s performance in the subtests was limited to calculation of the spent time, using digital stopwatch and counting the correct items. The high inter-rater reliability suggested that the two different raters that evaluated the child at the same time, had almost the same interpretation of the student’s performance (16) and they trusted each other assessments.

4.3. Construct Validity

In the studies that have been conducted to date, it has been confirmed that attention could predict intelligence at some levels. In 1969, Moray published a list including mental concentration, search, selective attention, divided attention, and vigilance as a variety of attention functions (11). In the studies that have been conducted so far, mental concentration, which became known as sustained attention later (17), has shown moderate correlation (r = 0.31) with intelligence (11). Search is the ability to screen the long lists to find the special numbers or letters where the distracters are also letters or numbers. In 2010, Schweizer found a low correlation between this type of attention and intelligence (r = -0.20) (11). Selective attention that has a close relation with focusing is the ability to discriminate stimuli from other stimulants. The discrimination time is related to intelligence, and research results have shown that this relation is moderate (r = -0.30) (18). Vigilance, also known as alertness, means attending to the rare stimulus in a long time while the rate of stimulus presentation is very low. In one study in 2005, this type of attention showed a low correlation (r = -0.15) with intelligence (19). Attention switching is the ability of quick transition between different demands of a task. The final destination is not specified, but it is marked with a clue. This type of attention reflects the cognitive flexibility. A low correlation (r = -0/18) was observed between this kind of attention and intelligence. Space attention is the ability of locating a target in an unexpected place compared to the expected place. The clue always guides the person to the expected places and the expected place is the correct answer in most of the trials. In the rest of them, the clue cause handicapped performance. Space attention has shown a low correlation (r = -0.20) with intelligence (20). The attention independency and autonomy can be explained by its limitation to predict intelligence. In a study conducted by schweizer et al. in 1994, the most part of intelligence was predicted by means of attention, visual search, long and short term memory and working memory (19). Schweizer in 2000 could estimate more than the half of the variance observed in IQ with other perceptual processing tools beside attention assessment tests (19). In a study by Schweizer et al. in 2005, they could estimate almost one third of observed variance of intelligence, using attention assessment tools (19). However, almost the majority of intelligence was estimated by other cognitive processing instruments. In other words, a person’s IQ is partly related to attention function, but many other factors play a role in predicting IQ. Thus, to assess these two related but different cognitive skills, we should use separate assessment instruments. Cognitive heterogeneity leads to significantly positive associations between different instruments for assessing cognitive functions in large samples. Manly et al. conducted a study and found a significant relationship between the 4 subtests of the Wechsler’s Intelligence Scale and the TEA-Ch (2). Moreover, Chan et al. conducted a study, in which a significant positive relationship was found between the 5 subtests of the Wechsler’s intelligence scale, and the TEA-Ch (6). Moreover, in this study, Block design (a subscale of WISC-IV), a test suitable for assessment of the attention performance, was related to the frontal lobe. It might be expected that the block design subtest be more correlated with the creature-counting subtest (to assess attention switching) due to its visio-spatial processing and promptitude requirements (5). Moreover, the memory span subtest of WISC has been desirably correlated with attention performance in IQ calculation tests, including the CPT (6).

The TEA-Ch entails sufficient reliability for children in Iran, and it has been designed exclusively to assess attention function, which cannot be examined through IQ tests. The therapists can employ the TEA-Ch to assess the effectiveness of cognitive interventions that are designed to improve attention in children. One limitation of this study was the students’ lack of motivation during the assessment. Moreover, due to spending almost two hours to take the tests, children tiredness might have affected the results. Therefore, to reduce this negative influence, we allowed the children to rest between the subtests.