Item Analysis of the Registered Nurse Licensure Exam Taken by Nurse Candidates from Vocational Nursing High Schools in Taiwan

Transcription

1 Proc. Natl. Sci. Counc. ROC(D) Vol. 9, No. 1, pp Item Analysis of the Registered Nurse Licensure Exam Taken by Nurse Candidates from Vocational Nursing High Schools in Taiwan LI-CHAN LIN*, HUI-MIN TSENG**, SHIAO-CHI WU*** *Institute of Clinical Nursing National Yang-Ming University Taipei, Taiwan, R.O.C. **Division of Item Banking Management Ministry of Examination Taipei, Taiwan, R.O.C. ***Institute of Public Health National Yang-Ming University Taipei, Taiwan, R.O.C. (Received December 23, 1998; Accepted April 16, 1999) Abstract The purposes of this study were: (1) to investigate the problem items on the RN licensure exam, and (2) to determine the consistency of classification of item difficulty between the items writers and test examinees responses. Data was sampled from 13,277 examinees responses to items on the RN licensure exam held in July, Research findings revealed that problem items found in basic nursing numbered four items (5%), seven problem items (8.75%) in medical-surgical nursing, nine items (11.25%) in comprehensive nursing and the highest number, 17 items (21.25%), in basic medical science. In determining the consistency of item difficulty classification between the items writers and test examinees responses, 30 items (37.97%) in basic nursing, 43 items (53.75%) in medical-surgical nursing, 34 items (42.50%) in comprehensive nursing, and 38 items (46.50%) in basic medical science were consistent, but not statistically significant (ps >.05). Study findings recommend pre-test reliability and validity analysis for newly developed items before every formal exam administration to eradicate problem items. Key Words: registered nurse, licensure exam, item difficulty, item discrimination, item analysis I. Introduction In order to assure the quality of entry level nurses in practice, the Ministry of Examination in Taiwan initiated in 1968 a national exam for newly graduated students from vocational nursing high schools to determine if they met a minimum registered nurse (RN) licensing requirement. Therefore, to pass or fail to pass the RN licensure exam is one of great concerns of administrators, faculty, students and their parents. To the health care profession, failure of students to pass the RN licensure exam might reduce the number of newly licensed personnel and create a shortage of registered nurses. In nursing education, success in the RN licensure exam is frequently used as a standard for the evaluation of nursing schools. If poor RN licensure exam performance persists, schools might face the possibility of reduction in student enrollment (Fowles, 1992; Schaal, 1990). If nurse candidates fail to pass the RN licensure exam, they might experience lowered self-esteem and the entry of individuals into nursing might be delayed (Ashley & O Neil, 1991). In a society, credentials offer an opportunity to judge the probable adequacy of professional performance, knowledge and self-regulation. The RN licensure exam sets the minimum requirement for professionals capable of delivering nursing care to meet public health needs (Felts, 1986; Lambert & Lambert, 1989). The Registered Nurses Act of Taiwan states that those who pass the RN licensure exam and possess registered nurses licenses are eligible to be clinical nurses. Thus, the licensure exam must be able 24

2 Item Analysis of the Licensure Exam to measure competencies required for safe and effective job performance, and distinguish between nurse candidates who do and do not possess these competencies (National Council of State Boards of Nursing, 1991). In other words, the essential requirement for the RN licensure exam is test sensitivity. If the RN licensure exam does not have the capability to distinguish among individuals with varying levels of ability, then not only will nurse candidates rights be affected, but clients security and safety will be as well. From the above statement, the RN licensure exam is the critical factor for getting individuals into nursing practice; educational requirements alone do not qualify one to legally practice as a RN. When it comes to the RN licensure exam in Taiwan, discussion of item construction, item analysis and item selection for the RN licensure exam is unavoidable. There are three main concerns with the Taiwan RN exam: (1) a loose item construction process, (2) a lack of quantitative item analysis before and after a formal RN licensure exam, and (3) a lack of objective information used to select items for the RN licensure exam. In terms of the item construction process, item writers for the RN exam in the U.S. are volunteer nurses who are currently employed in clinical nursing practice or responsible for teaching students in the clinical area. After item construction, panels review the test items and then put 72 tryout items together in a RN licensure examination to perform empirical item analysis (National Council of State Boards of Nursing, 1991). In Taiwan, although the Ministry of Examination is in charge of all national examinations like government employee exams, and professional and technician exams, there is no department currently overseeing the RN licensure exam. The Ministry of Examination organizes a temporary examination committee for each RN test, which consists of four nursing experts, to review and evaluate the items on the RN licensure exam. The chair of the committee selects six experts for each test to construct test items for the licensure exam. Every item writer needs to construct 100 items and denote the item difficulty for each item. After collecting 600 items for each test, one expert in measurement reviews the format of the items. Lack of quantitative item analysis before and after a formal RN licensure exam exists in Taiwan is important, since item analysis is regarded as an important step in determining the performance of items and provides support for retaining or deleting items (Davis, 1996). The analysis of item difficulty provides information regarding the difficulty level of each item. It is well known that the easiest way to estimate item difficulty is to calculate the proportion of test examinees who answer an item correctly (Brown, 1983). Currently however, after editing the format of the items, the temporary examination committees review and evaluate the content of these items, and determine how appropriately the items are classified by the item writers into categories of difficulty, which include difficult, moderate and easy. There is a lack of objective information for selecting items for the Taiwan RN licensure exam in the current procedure and this is significant because, as Crocker and Algina (1986) pointed out, item discrimination is a primary criterion for item selection. The item discrimination index indicates whether an item differentiates among test examinees having varying degrees of knowledge or ability. A good rule of thumb for the item discrimination index is 0.2 or higher (Brown, 1983; Crocker & Algina, 1986; Mehrens & Lehmann, 1984). The simplest discrimination index is the ordinary Product Moment item-total correlation between each item and the total test score. Item-total correlations above 0.3 are desired (Ferketich, 1991; Nunnally & Bernstein, 1994). If the items are scored dichotomously, a point-biserial correlation needs to be used (Nunnally & Bernstein, 1994). In Taiwan, once the temporary examination committee completes the item review process, the items are put into an item bank. When the date of the formal RN licensure exam nears, one of the exam committees in the Ministry of Examination uses proportional sampling based on the classification of item difficulty as determined by the item writers to select items for the RN licensure exam. The pass rate of the RN licensure exam in the U.S. has frequently been used as a criterion to measure nursing students minimum competency. The research findings have been used by nursing faculty in the U.S. to examine their curricula in order to identify which components most influenced their students ultimate success in the profession (Mills, Sampel, Pohlman, & Becker, 1992; Payne & Duffey, 1986; Woodham & Taube, 1986). However, since the RN licensure exam s inception, the exam has never been subjected to any form of item analysis to determine its suitability as a measuring instrument in Taiwan. Also, while there have been several research studies on nursing education in Taiwan, no research related to item analysis of the RN licensure exam has been done. Without item analysis on each test of the RN licensure exam before a formal exam is administered, item selection is based entirely on the subjective judgment of item difficulty made by the item writers. 25

3 L.C. Lin et al. Because the item writers are recruited from faculty in colleges, it was a question whether or not rating differences of item difficulty exist between item writers and test examinees. In addition, no information about item discrimination for each item is referenced in the process of selecting items capable of differentiating ability of test examinees. There is ample reason to doubt whether a sufficient level of reliability exists for the items on the exam. The purposes of this study were, thus, (1) to investigate the problem items in the RN licensure exam, and (2) to determine the consistency of classification of item difficulty between the item writers and test examinees responses. II. Methods 1. Source of Data The RN licensure exam in Taiwan is a paper-andpencil multiple-choice test consisting of separate 80- item sections covering each of the four areas of basic nursing, medical-surgical nursing, comprehensive nursing (including obstetric, pediatric, psychiatric and public health nursing), and basic medical science. Because the Ministry of Examination in Taiwan oversees this exam, this governmental body is responsible for storing the exams, as well as the examinees demographic data, their responses on the exam, and the results. For this study, we linked the data of item difficulty classified by item writers and items responses on four components of the RN licensure exam held in July, Criteria Used to Define Problem Items One of the principles followed in writing multiple-choice items is to have only one correct answer to every item. Therefore, if an item had more than one correct answer, it was categorized as a problem item. In addition, since the RN licensure exam is a content-referenced test, where examinees are expected to attain a certain level of performance, item difficulty was set at , which was easier than the rule of thumb (Brown 1983). Item writers might purposefully put easier items at the beginning of a test as warm-up questions to build confidence or to get test examinees used to the testing situation. They also may include more difficult items to ferret out the more capable examinees. To help neutralize these biases, a problem item was one which did not meet all of criteria (2), (3) and (4). In summary, problem items were those which did not meet the following criteria: A. Every item has only one correct answer. B. Item difficulty (the percentage of examinees who get the item correct) is between C. Item discrimination index is 0.2 or over. D. Point-biserial correlation is 0.3 or over. 3. Data Analysis The software package Iteman was used for analysis. The item responses for the RN licensure exam were analyzed according to item difficulty, the item discrimination index and point-biserial correlation. Each section of the exam, basic nursing, medical-surgical nursing, comprehensive nursing, and basic medical science, was examined for internal consistency. Kappa agreement was then used to determine the consistency of classification of item difficulty between the item writers and test examinees. III. Results In total, 13,277 test examinee responses for the RN licensure exam in July 1996 were used for analysis. 4. Item Description for the Four Test Sections A. Basic Nursing Since item #23 of the basic nursing section had two correct responses, it was excluded from internal consistency analysis, item difficulty and item discrimination, but retained in the problem item count. KR 20 for 79 items in the basic nursing component was The number of items answered correctly ranged from 0-76, with a mean of 58.3 (SD=12.53). Item difficulty ranged from , with a mean of Item discrimination indices ranged from , with a mean of Point-biserial correlation ranged from , with a mean of B. Medical-Surgical Nursing KR 20 for the 80 items in the medical-surgical nursing section was The number of items answered correctly ranged from 0-73 with a mean of (SD=12.09). Item difficulty ranged from with a mean of Item discrimination indices ranged from with a mean of Pointbiserial correlations ranged from with a mean of

4 Item Analysis of the Licensure Exam Table 1. Number of Problem Items Identified on four Components of The RN Licensure Exam Item Item Item Point-biserial Total of problem difficulty difficulty discrimination correlation items <.5 >.85 index <.2 <.3 (1) (2) (3) (4) Basic nursing a Med-surgical nursing Comprehensive nursing Basic medical sciences Note:total number of problem items: items which did not meet criteria (1), (3) and (4); or (2), (3) and (4); a : #23 on the basic nursing component of the exam was excluded from estimation of item difficulty, the item discrimination index and point biserial correlation, but counted as a problem item because it had two correct answers. C. Comprehensive Nursing KR 20 for the 80 items in the section on comprehensive nursing was The number of items answered correctly ranged from 0-72 with a mean of (SD=11.83). Item difficulty ranged from with a mean of Item discrimination indices ranged from with a mean of Point-biserial correlations ranged from with a mean of D. Basic Medical Science KR 20 for the 80 items in the section on basic medical science was The number of items answered correctly ranged from 0-68 with a mean of (SD=10.40). Item difficulty ranged from with a mean of Item discrimination indices ranged from with a mean of Pointbiserial correlations ranged from with a mean of Identification of Problem Items The total number of items with item difficulty below 0.5 and above 0.85, the number of items with item discrimination indexes below 0.2, the number of items with point-biserial correlation below 0.3 and the number of problem items were determined. As can be seen in Table 1, basic medical science had the highest number of items with item difficulty below 0.5 (47 items), an item discrimination index below 0.2 (23 items), and a point-biserial correlation below 0.3 (35 items). The results indicated that items in the section covering basic medical science were harder and had a lower level of discrimination than did those in the other three sections. The number of problem items found in basic nursing was four (5%). There were seven problem items (8.75%) in medical-surgical nursing, nine items (11.25%) in comprehensive nursing and the highest number, 17 items (21.25%), in basic medical science. 6. Consistency of classification of item difficulty between the item writers and test examinee responses A. Basic Nursing The classification of item difficulty from test examinee responses was below 0.3 for difficult, for moderate and over 0.7 for easy. Examining the consistency of item difficulty classification for basic nursing items between item writers and test examinee responses, we found that 30 items (37.97%) were consistent. Among them, one item was consistently classified as difficult, while 13 items were consistently classified as moderate, and 16 items were consistently classified as easy. In contrast, 49 items (62.03%) in basic nursing were inconsistent between item writers and test examinee responses. The coefficient of Kappa was 0.088, p = 0.171, which indicated that there was no significant consistency of classification of item difficulty between item writers and test examinee responses. B. Medical-Surgical Nursing As can be seen in Table 2, no item was consistently classified as difficult while 40 items were consistently classified as moderate, and three items were consistently classified as easy by item writers and based on test examinee responses. A total 43 items (53.75%) were consistent, but 37 items (46.25%) in medical-surgical nursing were inconsistent. Kappa agreement was used to determine the consistency of item difficulty classification between item writers and test examinee responses and yielded a result of , 27

5 L.C. Lin et al. Table 2. Consistency Analysis of Classification of Item Difficulty between Item Writers and Test Examinees Responses Item writers Total Kappa test dificult moderate easy items Basic Nursing test examinees a. difficult moderate easy Med-Surgical Nursing Test examinees a difficult moderate easy Comprehensive Nuring Test examinees a difficult moderate easy Basic Medical Science Test examinees a difficult moderate easy Note: a : difficult, below. 3; moderate,.3-.7; easy, above.7 p = C. Comprehensive Nursing Table 2 shows that one item was consistently classified as difficult, 30 items were consistently classified as moderate, and three items were consistently classified as easy by item writers and based on test examinee responses, with a total of 34 items (42.50%) showing consistency. Forty-six items (57.50%) in comprehensive nursing were found to be inconsistent, with a Kappa agreement of , p = D. Basic Medical Science Examining the consistency of item difficulty classification of basic medical science items between item writers and test examinee responses, we found that 38 items (47.50 %) were consistent. Among them, four items was consistently classified as difficult, 27 items were consistently classified as moderate, and seven items were consistently classified as easy. In contrast, 42 items (52.50%) in basic medical science were found to be inconsistent. The coefficient of Kappa was 0.107, p = 0.179, which also indicated that there was no significant consistency in item difficulty classification between the item writers and test examinee responses. IV. Discussion With respect to internal consistency, KR 20 for the four test categories of the RN licensure exam ranged from These values were modest and showed that items on the RN licensure exam were homogenous. Table 1 reveals that 11 items in the basic nursing section, 19 items in the medical-surgical nursing section, 33 items in the comprehensive nursing section and 47 items in the basic medical science section of the exam had item difficulty below 0.5. In contrast, 31 items in basic nursing, five items in medical-surgical nursing, eight items in comprehensive nursing and two items in the basic medical sciences section had item difficulty above The results indicated that the test section covering basic medical science was the most difficult section for the test examinees whereas the section on basic nursing was the easiest one. These results were not unexpected as the section on basic medical science consisted of anatomy, physiology, microbiology, and pharmacology, areas most examinees from vocational nursing high schools are poorly teaching in due to lab equipment limitations in these schools. In contrast, basic nursing forms the 28

6 Item Analysis of the Licensure Exam foundation of all levels nursing courses and clinical site teaching regardless of where students were enrolled. The mean item discrimination indices over all four sections of the RN licensure exam were all over 0.2 ( ), and the mean point-biserial correlations for all four sections were over 0.3 ( ). From the above results, we can say that overall, the items in the four sections of the RN licensure exam seem to have a modest level of discrimination. However, the ranges of the basic nursing item discrimination indices were and for point-biserial correlations, the med-surgical nursing item discrimination indices ranged from and for point-biserial correlations, the comprehensive nursing item discrimination indices ranged from and for pointbiserial correlations and the basic medical science item discrimination indices ranged from and for point-biserial correlation. The above results revealed that the item discrimination indices of the four sections of the RN licensure exam varied widely. Some item discrimination indices were 0, which could have resulted from items which were too easy, too hard or ambiguous. Furthermore, some item discrimination indices were negative, which indicated that poorer students more often answered the item correctly than did better students. Since 13,277 examinees took the RN licensure exam, negative item discrimination values are not likely to result from chance, but were more likely due to item ambiguity. Problem items were defined as those which did not satisfy the following criteria: (1) only one correct answer for each item, (2) the item difficulty from , (3) an item discrimination index of 0.2 or more, and (4) a point-biserial correlation of 0.3 or more. Under these loose criteria, among the four sections of the RN licensure exam, four items (5.00%) in basic nursing, seven items (8.75%) in medicalsurgical nursing, nine items (11.25%) in comprehensive nursing and 17 items (21.25%) in basic medical science were judged to be problem items. The results indicated that basic medical science had the most problem items. Further investigation showed that 50% (2 items) of the problem items in basic nursing, 43% (3 items) in medical-surgical nursing, 67% (6 items) in comprehensive nursing and 59% (10 items) in basic medical science had item difficulty below 0.3, and item discrimination below 0.2 and point biserial value below 0.2. This indicates that most of the problem items were too difficult to differentiate among the test examinees on the basis of ability and also implies that the item writers over-estimated the ability of the examinees. Ideally, a national competency exam should not have any problem items, but this study found a high proportion of problem items in this exam. It is felt that these results support the need for pretest analysis on the RN licensure exam before each formal exam is held. Analysis of the consistency of item difficulty classification between the item writers and test examinee responses reveal that, 30 items (37.97%) in basic nursing, 43 items (53.75%) in medical-surgical nursing, 34 items (42.50%) in comprehensive nursing and 38 items (46.5%) in basic medical science were consistent. The results indicated no significant consistency between the item writer s classifications and the test examinee responses (ps >.05). The most plausible explanation for this inconsistency may be that most of the writers of the items were faculty members in colleges while the test examinees were graduates of vocational nursing high schools. College level faculty might not perceive or be cognizant of the different levels of achievement, experience or knowledge of graduates from vocational nursing high schools compared those from colleges. The % rate of inconsistent estimation of item difficulty on the RN licensure exam between item writers and examinee responses further supports the need for pretest validation of the exam items. It also shows the need for overall reorganization and standardization of the exam and the various learning institution curricula which prepare graduates for it. For instance, clinical experience might be a source to affect nurse candidates perfomance on RN licensure exam. Although 29.8% of teaching hours in vocational nursing high schools are clinical hours, the appropriateness of clinical sites and quality of clinical teaching vary (Ministry of Education, 1990). Therefore, to evaluate clinical learning environment for vocational nursing high schools is needed. Implications In 1968, the Ministry of Examination began to administer the RN licensure exam. Since then, though much criticism has centered around the quality of the items in that exam, few individuals have questioned its fairness. Also, no request has been received by the Ministry of Examination in the form of published work or proposals for item, reliability or validity analysis of the RN licensure exam. Item analysis performed in this study revealed % of problem items on the RN licensure exam for July, The major concern from the findings is that problem items might undermine the fairness of the exam. Vale (1986) stated that a test item should consist 29

7 L.C. Lin et al. of substantive content and statistical characteristics. The determination of item quality should include assessment of the quantitative value and the qualitative content of item simultaneously. Therefore, the Ministry of Examination should invite experts in nursing to review and then revise problem items immediately after each RN licensure exam is compiled. This can ensure that quality items are put into the exam item bank, where they can accumulate and be drawn upon for future exams. Evaluation of student performance can provide faculty teaching feedback. This study found that no significant consistency of item difficulty classification existed between item writers and test examinee responses. We found a definite difference in the definition of item difficulty between exam writers and examinees. Overestimation or underestimation by item writers of the ability of examinees might result in an inappropriate proportion of item difficulty on the RN licensure exam. This brings up the question of the basic right that examinees have to be given a fair exam. To improve the above weakness, recruiting exam item writers from among faculty of vocational nursing high schools is recommended. However, the best way to eradicate the above weakness is to instigate adaptation and use in Taiwan of the U.S RN practice of pre-testing newly constructed items before they are included in a formal RN licensure exam. Acknowledgement This study was supported by a grant from the National Science Council, NSC S References Ashley, J. & O Neil, J. (1991). The effectiveness of an intervention to promote successful performance on NCLEX-RN for baccalaureate students at risk for failure. Journal of Nursing Education, 30(8), Brown, F.G. (1983). Principles of educational and psychological testing (3rd ed.). New York: Holt, Rinehart and Winston. Crocker, L. & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart and Winston. Davis, A.E. (1996). Instrument development: Getting started. The Journal of Neuroscience Nursing, 28(3), Felts, J. (1986). Performance predictors for nursing courses and NCLEX-RN. Journal of Nursing Education, 25(9), Ferketich, S. (1991). Aspects of item analysis. Research in Nursing & Health, 14, Fowles, E.R. (1992). Predictors of success on NCLEX-RN and within the nursing curriculum: implications for early intervention. Journal of Nursing Education, 31(20), Lambert, C. E. & Lambert, V. A. (1989). Perspectives in nursing: the impacts on the nurse, the consumer, and society. Englewood Cliffs: Appleton & Lange. Mehrens, W. A. & Lehmann, I. J. (1984). Measurement and evaluation in education and psychology (3 rd ed.). New York: Holt, Rinehart and Winston. Mills, A., Sampel, M.E., Pohlman, V. & Becker, A.M. (1992). The odds for success on NCLEX-RN by nurse candidates from a four-year baccalaureate nursing program. Journal of Nursing Education, 31(9), Ministry of Educaiton (1990). Standard of curriculum and equipment in vocational nursing high school. Taipei: Department of Technical and Vocation, Minsitry of Edcuation. National Council of State Boards of Nursing (1991). The NCLEX process. Chicago: National Council of State Boards of Nursing. Nunnally, J.C. & Bernstein, I.H. (1994). Psychometric theory (3 rd ed.). New York: McGraw-Hill. Payne, M.A. & Duffey, M.A. (1986). An investigation of the predictability of NCLEX scores of BSN graduates using academic predictors. Journal of Professional Nursing, 2, Schaal, M.J. (1990). Wanted: nurses with critical thinking skills. The Journal of Continuing Education in Nursing, 21(2), Vale, C.D. (1986). Linking item parameters onto a common scale. Applied Psychological Measurement, 10(4), Woodham, R. & Taube, K. (1986). Relationship of nursing program predictors and success on the NCLEX-RN examination for licensure in a selected associate degree program. Journal of Nursing Education, 25(3),

8 Item Analysis of the Licensure Exam 31