Development and psychometric testing of the Clinical Learning Organisational Culture Survey (CLOCS)

Transcription

1 Development and psychometric testing of the Clinical Learning Organisational Culture Survey (CLOCS) Author Henderson, Amanda, Creedy, Debra, Boorman, Rhonda, Cooke, Marie, Walker, Rachel Published 2010 Journal Title Nurse Education Today DOI Copyright Statement 2010 Elsevier. This is the author-manuscript version of this paper. Reproduced in accordance with the copyright policy of the publisher. Please refer to the journal's website for access to the definitive, published version. Downloaded from Griffith Research Online

2 Development and psychometric testing of the Clinical Learning Organisational Culture Survey (CLOCS) Abstract Aim: This paper describes the development and psychometric testing of the Clinical Learning Organisational Culture Survey (CLOCS) that measures prevailing beliefs and assumptions important for learning to occur in the workplace. Method: Items from a tool that measured motivation in workplace learning were adapted to the nursing practice context. The tool was tested in the clinical setting, and then further modified to enhance face and content validity. Participants: 329 registered nurses across three major Australian health facilities were surveyed between June 2007 and September Data analysis: An exploratory factor analysis identified five concepts - recognition, dissatisfaction, affiliation accomplishment, and influence. Validity and reliability: Internal consistency measures of reliability revealed that four concepts had good internal consistency: recognition (α =.914), dissatisfaction (α =.771), affiliation (α =.801), accomplishment (α =.664), but less so for influence (α =.529). Results: This tool effectively measures recognition, affiliation and accomplishment three concepts important for learning in practice situations, as well as dissatisfied staff across all these domains. Testing of additional influence items identify that this concept is difficult to delineate. Conclusion: The CLOCS can effectively inform leaders about concepts inherent in the culture important for maximising learning by staff. Words 199 Keywords: Instrument development, nurse education, organisational development, staff development Words: 4500

3 INTRODUCTION Learning within health practice environments needs to be optimized for quality care to be sustained (Henderson & Winch, 2008). This imperative is greater than ever before as the nursing workforce is highly mobile and susceptible to rapid turnover and attrition (International Council of Nurses, 2004; Productivity Commission, 2005). The successful transitioning of graduates and new staff into workplace practice relies on clinical contexts that optimize teaching and learning (Steinbinder & Scherer, 2006), foster the integration of new staff and ensure that standards of care are maintained. A valid and reliable tool that measures the assumptions inherent in the clinical learning organisational culture that directly influences learning and ultimately individuals performance in the workplace is a useful barometer of positive and negative elements within practice environments. The specific knowledge obtained from a measure of the clinical learning organisational culture can subsequently guide leaders as to the structures, processes, and practices that are effective or need to be developed to foster learning within their clinical contexts. A review of available instruments (Scott-Findlay & Estabrooks, 2006; Scott, Mannion, Davies, & Marshall, 2003), revealed few valid, freely available tools that measure workplace culture that facilitates staff learning. The development of a tool is potentially very useful it can be used to progressively gauge whether initiatives in clinical contexts impact on those factors important for learning. BACKGROUND Organisational culture is not consistently defined. There are multiple definitions in use (Scott-Findlay & Estabrooks, 2006; Scott et al., 2003) and 2

4 little agreement about how organisational culture should be observed or measured (Scott et al., 2003). The often cited framework developed by Schein (2004), identifies culture manifestation at three distinct and hierarchical levels: artefacts, espoused beliefs and values, and underlying assumptions; which are intrinsically linked to and influenced by leadership and vice versa. Schein s (2004) definition identifies the existence of assumptions in culture. Given the difficulties of challenging assumptions, clear articulation of these assumptions are essential to effectively guide leadership teams interested in shaping positive learning environments. A review of the literature identified a tool titled the theory of motivation of personal investment by Maehr and Braskamp (1986). This tool, derived from empirical studies within workplace environments, seemed relevant to use when exploring the norms inherent in clinical nursing practice contexts as it explored the motivation of individuals to learn and work within an organisation. The concepts from this existing tool formed the basis of a new tool, the Clinical Learning Organisational Culture Survey that recognised the generic concepts but modified to suit contemporary health care contexts. AIM This paper describes the development and psychometric testing of the Clinical Learning Organisational Culture Survey. This survey measures the existence and prevalence of assumptions (through sub-scales) that provides information 3

5 about staff attitudes within clinical contexts important for learning to occur in the workplace. Ethical Considerations All processes used to develop and test the tool were approved by the Human Research Ethics Committees of the university and participating hospitals. Participants were informed that their participation was voluntary. The information sheet also identified the purpose and expected benefits of the study, details about the research team, ethical considerations, and advised that the return of a completed or partially completed survey was accepted as their informed consent to participate. METHOD Development of the tool (CLOCS) involved the following sequential stages: a review of the literature identified a tool with the best fit to investigate clinical learning culture, namely, the theory of motivation of personal investment; exploration of the relevance of the theory of motivation of personal investment to contemporary nursing practice by the project steering team and an advisory panel (Maehr and Braskamp 1986); assessment of content and face validity by recognised experts in the field. The written feedback from the experts about the wording of items resulted in modification of existing items and generation of new items; a pilot study with 24 clinical nurses to verify that wording of items were meaningful to them and that these meanings were explicit and 4

6 consistent - minor revision of the tool was subsequently undertaken based on this feedback; a main survey and exploratory factor analysis on the final version to analyse the underlying structure; assessment of internal consistency of the sub-scales; two focus groups with six to eight registered nurses each were conducted to generate statements that represent to nurses that they feel comfortable to challenge and question practices (the core concept of the influence subscale); re-testing with the new items. Relevance of theory to contemporary nursing practice contexts The original tool by Maehr and Braskamp (1986) was modified in our study to gauge nurses perception of their clinical learning organisational culture. The applicability of these concepts to contemporary practice and specifically nursing are detailed in the following Table 1. INSERT TABLE 1 HERE Assessment of content and face validity by recognised experts Items congruent with contemporary meanings were modified from the original tool (Hoyle et al., 2002). The reported reliability of the original subscales that pertained to organisational culture were recognition 0.87; affiliation 0.85; accomplishment 0.80 and; influence 0.51 [based on data from 339 men and women] (Braskamp & Maehr 1985). The first version developed by our team comprised a total of 32 items that intended to describe the subscales of 5

7 recognition, affiliation, accomplishment and influence with a five-point Likert response scale (1 = strongly disagree, to 5 = strongly agree). The entire scale with an explanation of concepts that the items were based was then sent to six experts in academia and industry with a background in organisational culture within and outside of clinical nursing contexts to ascertain face validity. The experts rated each statement according to its relevance to the concept presented. This method of expert checking of concepts, was guided by Polit, Beck and Owen s (2007) approach to content validity of individual items (rather than the overall scale). This method of content validity checking concerns the degree to which a scale has an appropriate sample of items to represent the construct of interest. Based on the recommendations of the expert panel 8 of the 32 items were negatively re-worded to minimise response bias. Issues of face validity (content and readability of items) were also revisited. The draft survey was reviewed by members of the project Advisory Board (comprising nine representatives from the nursing profession and other health professionals with experience in learning in clinical contexts); only minor adjustments, such as the correction of grammatical errors were made to the format. Pilot study In 2007, 24 clinical nurses in an acute tertiary hospital completed the survey and provided feedback. Items were presented in random order so as not to identify the specific sub-scales thereby promoting an intuitive response from 6

8 respondents rather than one indirectly guiding or persuading their ratings. From this feedback, the questions I feel well supported during student clinical placements by the organisation and I don t have a great deal of influence over things that affect me in the job were deleted as meanings related to these statements were not consistent among the nursing staff. Exploratory factor analysis Main Survey The revised tool contained 30 items that aimed to measure clinical organisational culture dimensions of accomplishment, recognition, influence, and affiliation. Eight items were negatively worded. Procedure The survey was distributed to practising registered nurses in three major hospitals in South East Queensland during the months of June and August A brief explanation of the project accompanied the survey requiring minutes to complete. Time for completion was an important consideration given that ward staff are often time poor, thus arrangements were made with the nurse unit manager or unit educator to book in-service time between the morning and evening shifts. The best rates of survey return occurred when nurses were provided time to complete and return the form. Participants Registered Nurses working in wards/ units that hosted undergraduate nursing students from the participating tertiary education provider completed the 7

9 survey. A total of 329 surveys were returned from an approximate accessible population of 1192, thus represented a 28% response rate. The age of participants ranged from 18 to 64 (M = 34.74, SD = 10.18) and years of practice in a clinical setting ranged from less than one year to 34 years (M = 10.00, SD = 9.30). The majority of respondents were registered nurses with a degree, working in general medical/surgical areas. Place of work (hospital and division), professional group categories and education levels are reported in Table 2. INSERT TABLE 2 HERE Results Results were analysed using SPSS for Windows statistical package, Version An exploratory factor analysis (FA) was conducted on the 30 items to identify grouped variables. Initially, data were examined for missing values. Two cases contained non-random missing values and were deleted from the data set. Of the remaining 327 cases, 20 cases showed missing data spread across 16 items, and were assessed as randomly distributed. Missing data was replaced by either the variable mean (14 responses) or an estimate based on the participant s response to items strongly correlated with the missing value items (11 responses). The data was screened for outliers, and 38 of the 327 cases produced scores not satisfying the α <.001 criterion for Mahalanobis distance, that is, the responses did not indicate any similarity with the majority of surveys (Tabachnick & Fidell, 2007). One of these cases showed consistent 8

10 responses ranging from 1 to 3 across both positive and negative items, indicating a possible response bias, and was deleted. The other 37 outlier cases contained a higher than average number of extreme scores but were found to be randomly occurring across all variables. Thus these 37 were considered to be part of the population and were retained. Subsequent analysis was conducted using the remaining 326 cases. Variables were screened for normality by examining histograms. Twenty-two variables were negatively skewed, and square root transformations were performed to reduce skew. As there was no difference in the interpretation of Principal Component Analysis (PCA) between analysis of raw data and analysis of transformed data, the raw data was used in PCA. The PCA identified patterns in the questions, that is, the degree to which questions were related. Prior to extraction, the suitability of data for factor analysis was assessed. The Kaiser-Meyer-Oklin value was.92, exceeding the recommended value of.6, thus indicating there was sufficient in common across the questions to perform a PCA, and Bartlett s Test of Sphericity, the test for the minimum standard of correlation reached statistical significance ( 2 (435) = , p <.001). Given these overall indicators, factor analysis was deemed to be suitable with all 30 items (Tabachnick & Fidell, 2007). The initial PCA revealed the presence of six factors with eigenvalues exceeding 1, explaining 32.7%, 9.0%, 4.7%, 4.4%, 4.1%, and 3.4% of the variance respectively. Solutions for three, four, five and six factors were each examined using 9

11 varimax and oblimin rotations of the factor loading matrix to simplify the statistical findings. The five factor solution, which explained 55% of the variance, was preferred because: (a) the curve of the scree plot begins to tail off after two factors, but there is another drop after five factors before a plateau is reached; (b) of previous theoretical support; and (c) this solution retained as much variance as possible. When oblique rotation was requested, correlations between factors ranged from.07 to.43 indicating the factors were interrelated to some degree. Thus it was decided to use oblique rotation solution in the final solution. Two items ( There is respect for every member of staff at this health facility and I know how to get things done around here ) were eliminated because they failed to meet a minimum criteria of having a primary factor loading of.4, resulting in 28 items contributing to the final analysis. For the final stage, a PCA of the remaining 28 items was conducted using oblimin rotation, with five factors explaining 56% of the variance. All items in this analysis had primary loadings over.4. Factors from 1 to 5 explained 33%, 9%, 5%, 5%, and 4% respectively. Loadings of variables on components and communalities are shown in Table 3. Variables are ordered and grouped by size of loadings to facilitate interpretation. Interpretative labels for each component are in the footnote. INSERT TABLE 3 HERE 10

12 The five-factor model generally reflected the dimensions underlying the subscales, those concepts identified as important for learning environments. The first factor, which accounted for most of the variance, reflected the dimension of recognition. Items that correlated highest with the factor pertained to having a voice, recognition of value, and contribution to the organisation. The second factor appears to reflect the concept of dissatisfaction and represents overall discontent with the workplace. Factor three reflected the perception of affiliation, involving items related to teamwork, respect and support. The fourth factor comprised of items related to performance standards and pride in the work, and appeared to reflect the dimension of accomplishment. Factor five contained three items and appeared to reflect the dimension of influence. Internal Consistency In sum, five sub-scales became apparent: recognition (importance and effectiveness of reward/feedback systems operating within the organisation), dissatisfaction (overall discontentment with the workplace), affiliation (need and opportunities for interaction within the organisation), accomplishment (the degree of self-imposed and organisation-level performance standards), and influence (effects of power and competition within the organisation). Prior to further analysis, the 8 negative items were reverse scored. Internal consistency estimates, measured using Cronbach s α, was acceptable for the groups of items which formed recognition (α =.914), dissatisfaction (α =.771), and affiliation (α =.801), accomplishment (α =.664), and less so for influence (α =.529). 11

13 Composite variables for the five subscales were created by computing the mean across the associated items: recognition (items 12, 13, 19 to 25, 29, 30), dissatisfaction (4, 14, 15, 17, 18, 26), affiliation (items 1, 2, 6, 11), accomplishment (items 5, 7, 27, 28) and influence (items 8, 9, 10). The means, standard deviations and response range of the five sub-scales for 326 participants are reported in order of highest to lowest mean value in Table 4. INSERT TABLE 4 HERE Improving the internal validity of influence subscale Following initial testing and analysis, two focus groups of six to eight registered nurses each, directly involved in working with students and new staff members was undertaken to review the concept of influence and develop additional items to strengthen the influence sub-scale. Through this process, that involved approximately one hour, each group explored statements that described an environment responsive to questioning. The definition of influence focused on describing a clinical context receptive to open inquiry and debate. A set of five items were deemed to cover the revised definition of influence. The same scale with the addition of the new influence items was retested on a sample of 310 nurses. The new items failed to differentiate into a separate subscale but rather strengthened the existing subscales of recognition and affiliation. As re-testing with new items did not clearly differentiate a subscale of influence further work is needed to identify items that specifically communicate the meaning of staff feeling psychologically safe when sharing their ideas. 12

14 DISCUSSION This clinical learning organisational culture survey was tested in three acute care hospitals within the Australian health care system and therefore the items are relevant to this context. Items consistent with concepts of recognition and affiliation had strong internal reliability (cronbach alpha>0.7), less so for accomplishment. The influence concept, that is also important in clinical learning contexts, despite further testing of additional items these items were not specific enough to differentiate as a single concept. A further sub-scale of dissatisfaction was distinct, as the other four concepts are associated with motivated work environments. Dissatisfaction is a useful sub-scale in that it raises awareness for nurse leaders about staff who are dissatisfied regardless of the positive beliefs and assumptions that may exist within the workplace. Differentiation of these subscales may guide nurse leaders to better understand assumptions that impact on learning within clinical practice situations. The strength of the prevalence of these assumptions can inform leaders and management teams about structures, processes, and practices that are effective or need to be developed to foster learning within their clinical contexts. The tool may also assist nurse leaders to subsequently monitor the effectiveness of intervention strategies with their staff. CONCLUSION Learning environments are an increasingly important consideration in contemporary workplaces given the constant changes in the organisation of 13

15 work, such as, the skills and equipment needed to perform work, and knowledge underpinning practice. There is considerable investigation within work environments given the universal importance of generic concepts that impact on staff engagement within their workplaces. By drawing on existing research around motivation, interest, and learning in work, nurses and other health professionals can readily modify and tailor existing tools for their specific purposes. The value of such a tool is that it can be used within a practice context where diverse health professionals interact to inform and guide health care teams about their learning culture. ACKNOWLDEGEMENT This study was supported by a Leadership grant (LD614) from the Carrick Institute for Learning and Teaching in Higher Education. 14

16 References Benner, P. (2001). From novice to expert: excellence and power in clinical nursing practice (Commemorative ed.). (Upper Saddle River, NJ: Prentice Hall.) Bradburn, N. M. (1983). Response effects. In P. H. Rossi, J. D. Wright & A. B. Anderson (Eds.), Handbook of Survey Research (pp ). (Sydney: Academic Press, Inc.) Braskamp, L., & Maehr, M. (1985). Spectrum: An organizational development tool [Manual]. (Champaign, IL: Metritech, Inc.) Buckingham, A., & Saunders, P. (2004). The survey methods workbook. (Cambridge: Polity Press.) Duddle, M., & Boughton, M. (2007). Intraprofessional relations in nursing. Journal of Advanced Nursing, 59, Fowler, F. J. (1995). Improving survey questions: Design and evaluation. (London: Sage Publications.) Fox, R., Henderson, A., & Malko-Nyhan, K. (2006). A comparison of preceptor and preceptee's perceptions of how the preceptor's role is operationalized. Journal of Clinical Nursing, 15, Henderson, A., & Winch, S. (2008). Managing the clinical setting for best nursing practice: a brief overview of contemporary initiatives. Journal of Nursing Management, 16, Hoyle, R. H., Harris, M. J., & Judd, C. M. (2002). Research methods in social relations. Melbourne: Thomson Learning. International Council of Nurses. (2004). The Global Shortage of Registered Nurses: An Overview of issues and Actions. Geneva: International Council of Nurses. Lynn, M. (1986). Determination and quantification of content validity. Nursing Research, 35, Maehr, M., & Braskamp, L. (1986). The Motivation Factor: A Theory of Personal Investment. (Lexington, MA: Lexington Books.) National Institute of Clinical Studies (2003) Factors supporting high performance in health care organisations. Prepared by the Health Management Group at Latrobe University. Melbourne. Papp, I., Markkanen, M., & von Bonsdorff, M. (2003). Clinical environment as a learning environment: student nurses' perceptions concerning clinical learning experiences. Nurse Education Today, 23,

17 Pearcey, P. A., & Elliott, B. E. (2004). Student impressions of clinical nursing. Nurse Education Today, 24, Peterson, R. A. (2000). Constructing effective questionnaires. (Thousand Oaks: Sage Publications.) Polit, D., & Beck, C. (2006). The Content Validity Index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing & Health, 29, Polit, D., Beck, C., & Owen, S. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30, Productivity Commission (2005) Australia's Heath Workforce, Research Report, Canberra Saris, W. E., & Gallhofer, I. N. (2007). Design, evaluation, and analysis of questionnaires for survey research. (New Jersey: John Wiley & Sons.) Schein, E. H. (2004). Organizational Culture and Leadership (3rd ed.). (San Francisco, CA: Jossey-Bass.) Scott-Findlay, S., &, & Estabrooks, C. (2006). Mapping the organizational culture research in nursing: a literature review. Journal of Advanced Nursing, 56, Scott, T., Mannion, R., Davies, H., & Marshall, M. (2003). The Qualitative Measurement of Organizational Culture in Health Care: A Review of the Available Instruments. Health Services Research, 38(3), Senge, P. (2006). The Fifth Discipline. The Art and Practice of the Learning Organisation. (New York: Doubleday/Currency.) Steinbinder A. & Scherer E. (2006). Creating nursing system excellence through the forces of magnetism. (In Malloch K. and Porter-O'Grady T.(Eds) Evidence-Based practice in Nursing and Health Care (pp ). Boston MA: Jones and Bartlett.) Sudman, S., Bradburn, N., & Schwarz, N. (1995). Thinking about answers: The application of cognitive processes in survey methodology. (San Francisco, CA: Jossey-Bass Publishers.) Tabachnick, B. G. & Fidell, L. S. (2007). Using Multivariate Statistics, 5th ed. (New York: Allyn & Bacon.) 16

18 Table 1 Background and explanation of concepts in CLOCS MEANING OF CONCEPT degree of selfimposed and organisation-level performance standards importance and effectiveness of reward / feedback systems operating within the organisation effects of competition, influences and conflict present within the organisation need and opportunities for interaction within the organisation Term ascribed by Maehr and Braskamp (1986) Contemporary term (Senge 2006) Identified concept within the nursing literature Accomplishment Personal mastery Successive knowledge and skill acquisition: Benner (2001) Recognition Sharing a vision Learners need to feel acknowledged in the workplace: Papp, Markkanen, & von Bonsdorff, 2003; Pearcey & Elliott, 2004 Power/influence Reflection on practice Staff feel 'safe' to express their opinions and ideas (National Institute of Clinical Studies 2003) Affiliation Team building Positive relationships are recognised as fundamental to transitioning graduates of nursing into the workplace and the retention of staff: Duddle & Boughton, 2007; Fox, Henderson, & Malko-Nyhan,

19 Table 2 Demographic Characteristics of 329 Participants Characteristic Total No. % Hospital A B C Work Division Medical Surgical Medical/Surgical 8 2 Other Missing Data 27 8 Professional Group Register Nurse (RN) RN buddy 28 8 Clinical Nurse Facilitator or Preceptor 13 4 Manager Role 8 2 Enrolled Nurse 22 7 Missing Data Highest Education Qualifications Hospital Certificate 26 8 Graduate Certificate 19 6 Graduate Diploma Degree Masters 13 4 Missing Data

20 Table 3 Pattern and Structure Matrix, and Communalities (h 2 ), for Principal Components Extraction and Oblimin Rotation on Cultural Survey Items Item Pattern Coefficients Structure Coefficients h 2 F 1 a F 2 F 3 F 4 F 5 F 1 a F 2 F 3 F 4 F 5 25 My opinion is valued I have a say in what happens here I feel as if I'm listened to here I feel that I am important Nursing staff are well supported I am encouraged by my co-workers to do my best work 30 It is clear that my job is important to the success of the hospital 12 My contribution is recognised

21 20 Staff on this ward learn from each other In our ward we are encouraged to try new things 29 My co-workers are supportive of my professional development 4 It is difficult to get help when I require support and advice 26 There is little or no acknowledgement about the quality of my work 14 We are not rewarded when we do a good job 17 I receive little feedback about what I do Staff in the ward/unit are worried about making mistakes 18 Changing practice in this ward/unit is

22 difficult 2 We work as a team here Staff at my hospital strive for excellence Nursing staff help each other to get the job done 6 The quality of work is important here I am able to balance all of the requirements of my role 27 I really believe in the value of what I am doing 7 I am proud of my work I am clear about what is expected of me as a member of the nursing staff 10 People don't take advantage of their position in this hospital 21

23 9 You need to have legitimate power to have any influence around here 8 Nurses views are ignored at this health facility a Factor labels F 1 F 2 F 3 F 4 F 5 Recognition Dissatisfaction Affiliation Accomplishment Influence 22

24 Table 4 Mean, Standard Deviations and Ranges of Sub-Scales Sub-Scale M SD Theoretical Range Observed Range Affiliation Accomplishment Recognition Dissatisfaction Influence