INPATIENT SURVEY PSYCHOMETRICS

Transcription

1 INPATIENT SURVEY PSYCHOMETRICS One of the hallmarks of Press Ganey s surveys is their scientific basis: our products incorporate the best characteristics of survey design. Our surveys are developed by conducting focus groups of patients, providers and/or administrators, reviewing surveys from health care facilities across the country, reviewing current professional and scientific publications on health care delivery, and utilising the latest research on survey statistics and design. Testing the Questionnaire Effective questionnaires have three important attributes: focus, brevity, and clarity. Questions should focus directly on a specific issue or topic, be as brief as possible while still conveying the intended meaning, and be expressed as simply and as clearly as possible. When we were confident that the set of questions developed for the Inpatient questionnaire met these criteria and that their face, content, and consensus validities had been established, we tested the questionnaire with the assistance of 167 test sites in Australia and New Zealand. Patient satisfaction questionnaires were mailed to patients four weeks after discharge from their Inpatient facility. A mail-out methodology was chosen over hand distribution to eliminate selection and acquiescence biases. A total of 29,331 surveys were mailed to patients. The test concluded with the receipt of 14,632 questionnaires, resulting in an average response rate of 50%. Perceptions of Inpatient care and service were measured by asking patients to rate 62 specific aspects of their visit. The 62 test items were divided into ten sections, designed to index care dimensions of the patient s experience during an Inpatient visit: Admission, Room, Meals, Nursing Care, Tests and Therapy, Visitors and Family, Doctor Care, Discharge, Personal Issues, and Overall Assessment of the Hospital. As with other Press Ganey questionnaires, a Likert-type response scale was used with the following categories: very poor, poor, fair, good, and very good. Because this scale is balanced and parallel unlike a poor to excellent scale responses can be quantified and used statistically without violating methodological assumptions. Also, variability in patients responses with this scale allows for the identification of opportunities to improve, unlike commonly used yes/no response scales. Questionnaire Psychometrics The accuracy of a questionnaire is assessed by measuring its validity and reliability. Validity is the degree to which a questionnaire measures what it was designed to measure. Reliability is the degree to which survey data are consistent and reproducible across respondents or across surveys. The Inpatient instrument was found to be psychometrically sound across a wide variety of tests of reliability and validity as described below. Response Patterns and Variance. Measures of central tendency (i.e., mean, median, and mode) and variability (standard deviation, standard error) were examined for all questions. Response frequencies and patient comments were evaluated for patterns of missing data and question-wording or ambiguity problems. These steps are important for ensuring the clarity of questions and the absence of instrument bias and error. Inpatient Survey 2010 Press Ganey Associates Page 1

2 Measure Redundancy. As discussed above, it also is important for a survey instrument to be concise. A correlation matrix of all test questions was examined to find questions that were too highly associated with one another. If two questions are highly intercorrelated, it suggests that they may be measuring the same issue or concept and are, therefore, redundant. Removing one of the two questions removes this redundancy and leaves a more parsimonious scale or questionnaire. Likelihood of recommending this hospital to others was found to be highly correlated to Likelihood of returning to this hospital if you had a choice. The question Likelihood of recommending was retained to promote consistency between the various setting-specific Press Ganey surveys. Doctor s concerns for your questions and worries was highly correlated to both Time the doctor spent with you and How well the doctor kept you informed and was, therefore, dropped as a standard item. Doctor s efforts to include you in decisions about your treatment was highly correlated with both Courtesy of the doctor and How well the doctor kept you informed and was consequently dropped as a standard item. Construct Validity. Factor analysis is a technique used to identify factors that statistically explain the variation among responses to a questionnaire. In other words, factor analysis helps to identify which questions belong together, confirming a questionnaire s construct validity or structure. A factor analysis was completed on the test data for the 51 remaining questions. Overall assessment questions were omitted from the factor analysis due to their predicted high intercorrelation with other items. Because it was likely that factors would not be independent of one another (i.e., a patient s response to one question may be correlated with his or her response to a question in another section), a principle component extraction was performed with Promax oblique rotation. Factor analysis is used to divide a multi-question survey into meaningful subscales. Questions that are highly correlated with one another typically represent a common dimension or concept. For example, Courtesy of the doctor and How well the doctor kept you informed are more likely to define a Doctor Care dimension than Pleasantness of room décor or Nurses attitude toward your request. The factor analysis identified nine dimensions of care and service in addition to patients overall assessment of their visit. However, one question Courtesy of patient support staff (e.g., patient assistants, orderlies, porters) loaded lowly on several factors and was, therefore, dropped as a standard question. A principle component factor analysis with Promax oblique rotation was re-run using the 49 questions and is displayed in Table 1. Factor loadings in Table 1 represent the correlations between factors and variables uncontaminated by factor overlap. Table 2 shows the correlations between the nine factors. Together the nine factors accounted for 61% of the variance in patient responses. Inpatient Survey 2010 Press Ganey Associates Page 2

3 Table 1. Item content and Primary Factor Loadings Factors and Items Admission Rating of pre-admission instructions (clear and understandable) If your stay was prearranged, extent to which the hospital was ready for you Courtesy of staff at the front admission desk Waiting time for admission to ward/unit Loading on Primary Factor Factors and Items Visitors and Family Courtesy of switchboard staff Staff attitude toward your visitors Information given to your family about your condition and treatment Comfort for visitors (e.g., chairs, toilets, waiting rooms) Loading on Primary Factor Room Pleasantness of room décor Room cleanliness Courtesy of the person who cleaned your room Room temperature Noise level in and around room How well things worked (call button, lights, TV) Doctor Care Time the doctor spent with you How well the doctor kept you informed Communication between doctors and nurses regarding your care Extent of explanation of diagnosis and treatment Courtesy of the doctor Meals Courtesy of the staff who served your food Temperature of the food (cold foods cold, hot foods hot) Quality of food Flavour of food Timeliness of delivery If you were placed on a special/restricted diet, how well was it explained Discharge Extent to which you felt ready to be discharged Waiting time after you were told you could go home Instructions given about how to care for yourself at home Help with arranging home care services (if required) How hospital staff handled billing Nursing Care Courtesy of the nurses Promptness in responding to the call button Nurses attitude toward your requests Amount of attention paid to your special or personal needs How well the nurses kept you informed Extent to which nurses placed things within your reach Extent to which medications were received in a timely manner Nurses efforts to include you in decisions about your care Personal Issues How well your pain was controlled Degree to which hospital staff addressed your emotional/spiritual needs Courtesy of religious/pastoral care staff Extent to which you were made aware of your rights and responsibilities Extent to which you were made aware of how to voice a complaint Tests and Therapy Courtesy of the x-ray staff Courtesy of the staff who took your blood Courtesy of the physiotherapy staff Explanations about what would happen during tests or therapy Concern shown for your comfort during tests or therapy Skill of the staff who took your blood (e.g., did it quickly, minimal pain) Inpatient Survey 2010 Press Ganey Associates Page 3

4 Table 2. Factor Correlation Matrix Admission Room Meal Nurses Tests Visitors Doctors Discharge Personal Overall Admission 1 Room.51 1 Meals Nurses Tests Visitors Doctors Discharge Personal Overall are significant at the p=.01 level (2-tailed). Convergent and Discriminant Validities. Although factor analysis is a popular method of establishing the construct validity of a questionnaire, other methods are available. For example, one of the assumptions of questionnaire and scale construction is that an individual item from a scale should be well correlated with the other items in that scale. Researchers suggest a minimum item-to-scale correlation of.30. Table 3 shows the average and range of correlations between each question and its parent section. These correlations are corrected in the sense that the item of interest is omitted from its section score when correlations between the two are calculated. Corrections are performed to avoid inflated or spuriously positive correlations. An item should be correlated with its own scale (convergent validity) as well as correlated more with its own scale than with other scales (discriminant validity). That is, item-to-scale correlations should be higher than item-to-non-scale correlations. For example, a Nursing Care question should be more highly correlated to its section than to the Doctor Care section. As Table 3 confirms, the Inpatient questionnaire demonstrates both convergent validity and discriminant validity. Because these are subclasses of construct validity, the questionnaire s effectiveness at measuring perceptions of Inpatient care is confirmed across multiple tests. Table 3. Item Analyses and Reliability Estimates Scale Alpha Average Corrected Item-Scale Range of Corrected Item-Scale Average Item-Non-Scale Range of Item-Non-Scale Admission Room Meals Nursing Care Tests and Therapy Visitors and Family Doctor Care Discharge Personal Issues Overall Assessment Inpatient Survey 2010 Press Ganey Associates Page 4

5 Criterion or Predictive Validity. Predictive validity is defined as the ability of an instrument to predict outcomes that theoretically should be tied to the construct measured by the instrument. In consumer satisfaction, the perception of a satisfying experience is expected to be linked to two types of behavioral outcomes in an individual: re-purchase behavior (coming back at a future date to purchase another product or service) and positive word of mouth (recommending a product to family and friends). Within the health care industry, limitations on patients choice of providers and health services make it difficult to assess the relationship between satisfaction and re-purchase behavior. An estimate of the relationship between satisfaction and positive word of mouth can be obtained by asking patients about their intent to recommend a provider or service. The predictive validity of a satisfaction instrument then can be estimated by the degree to which individual items on the instrument predict the patient s intentions to recommend. In this regard the questionnaire shows high levels of predictive validity. A series of simple regression analyses revealed that each of the individual items is a significant predictor (at the p=.001 level) of patients reported likelihood to recommend the facility (beta s ranged between.22 and.77). Additionally, multiple regression analysis revealed that, collectively, all items are significant predictors of patients reported likelihood to recommend the Inpatient facility to others, F(57, 14,574) = , p <.001, R 2 =.64 (adjusted R 2 =.64). In other words, the revised instrument explains approximately 64% of the variance in patients likelihood to recommend the Inpatient facility. According to Carey, 1 A good survey will account for 50% or more of the variance in global evaluations The Inpatient Survey has surpassed this high standard. Reliability. Reliability testing is a method of evaluating the internal consistency of a questionnaire. The traditional statistic used to illustrate the degree of consistency among the items of a scale or questionnaire is Cronbach s alpha. A set of questions with no internal consistency no reliability has an alpha of 0.0 indicating that the questions within the scale may not be measuring the same issues. A set of questions with perfect internal consistency has an alpha value of 1.0. All eight of the subscales exceeded the stringent.70 standard for reliable measures: Reliability estimates range from.80 to.94 (see Table 3). The Cronbach alpha for the entire questionnaire is.98, confirming the instrument s high internal consistency and reliability. Readability. According to the Flesch-Kincaid Index, which is based on the average number of syllables per word and words per question, the final questionnaire tests at a 6 th grade reading level. Descriptive Statistics. The means and standard deviations of all survey questions are listed in Table 4. The questions and sections appear in the order they appear on the survey and in the report. The means and standard deviations of the subscales are listed in Table 5. The overall database mean is (SD=12.88). 1 Carey, R. G. (1999). How to choose a patient survey system. The Joint Commission Journal of Quality Improvement, 25, Inpatient Survey 2010 Press Ganey Associates Page 5

6 Table 4. Question means and standard deviations Mean SD ADMISSION Rating of pre-admission instructions (clear and understandable) If your stay was prearranged, extent to which the hospital was ready for you Courtesy of staff at the front admission desk Waiting time for admission to ward/unit ROOM Pleasantness of room décor Room cleanliness Courtesy of the staff who cleaned your room Room temperature Noise level in and around room How well things worked (e.g., call button, lights, TV) MEALS Courtesy of the staff who served your food Temperature of the food (e.g., cold foods cold, hot foods, hot) Quality of food Flavour of food Timeliness of delivery If you were placed on a special/restricted diet, how well was it explained NURSING CARE Courtesy of the nurses Promptness in responding to the call button Nurses attitude toward your requests Amount of attention paid to your special or personal needs How well the nurses kept you informed Extent to which nurses placed things within your reach Extent to which medications were received in a timely manner Nurses efforts to include you in decisions about your care TESTS AND THERAPY Courtesy of the x-ray staff Courtesy of the staff who took your blood Courtesy of the physiotherapy staff Explanations about what would happen during tests or therapy Concern shown for your comfort during tests or therapy Skill of the staff who took your blood (e.g., did it quickly, minimal pain) VISITORS AND FAMILY Courtesy of switchboard staff Staff attitude toward your visitors Information given to your family about your condition and treatment Comfort for visitors (e.g., chairs, toilets, waiting rooms) DOCTOR CARE Mean SD Time the doctor spent with you How well the doctor kept you informed Communication between doctors and nurses regarding your care Extent of explanation of diagnosis and treatment Courtesy of the doctor Inpatient Survey 2010 Press Ganey Associates Page 6

7 Table 4. Question means and standard deviations cont d Mean SD DISCHARGE Extent to which you felt ready to be discharged Waiting time after your were told you could go home Instructions given about how to care for yourself at home Help with arranging home care services (if required) How hospital staff handled billing PERSONAL ISSUES How well your pain was controlled Degree to which hospital staff addressed your emotional/spiritual needs Courtesy of religious/pastoral staff Extent to which you were made aware of your rights and responsibilities Extent to which you were made aware of how to voice a complaint OVERALL ASSESSMENT OF THE HOSPITAL Overall cleanliness of the hospital Ease of finding your way around the hospital (e.g., signage, clear instructions) Staff concern for your privacy Extent to which you felt secure (e.g., personal safety & belongings) How well staff worked together to care for you Extent to which you now have a better understanding of your medical problem than when you entered the hospital Extent your condition has improved at least as much as expected Likelihood of you recommending this hospital to others Overall rating of care provide at the hospital Table 5. Subscale means and standard deviations Section Mean SD Admission Room Meals Nursing Care Tests and Therapy Visitors and Family Doctor Care Discharge Personal Issues Overall Assessment Inpatient Survey 2010 Press Ganey Associates Page 7