Intergenerational Observation Scale Analysis

Abstract

Innovative intergenerational programs continue to grow in number and scope. The development of standardized evaluation instruments, however, lags behind, leaving many researchers and practitioners without tools to effectively assess their programs. Evaluation data often focus on outcomes without attention to the nature of the interactions between generations. Understanding the process of intergenerational contact is central to understanding its outcomes. We developed the Intergenerational Observation Scale to assess the social interactions and affect of young and old participants during intergenerational activities. Our 3-step observer training process demonstrated good scale reliability. We present the process of developing the scale, achieving observer reliability, and next steps to continue exploring the scale’s utility across intergenerational populations and settings.

Development of a standardized tool for intergenerational programming:

The Intergenerational Observation Scale

Providing opportunities for meaningful engagement to persons with dementia challenges family and professional caregivers alike. While some practitioners and researchers seek innovative means to support client well-being (e.g., Allen-Burge, Burgio, Bourgeois, Sims, & Nunnikhoven, 2001; Orsulic-Jeras, Judge, & Camp, 2000), many dementia care programs are characterized by extreme levels of inactivity (Ice, 2002), which may result from infantilization (Salari & Rich, 2001) and can lead to agitation and depression and detract from well-being (Teri et al., 2003; Voelkl, 1986). Kitwood and Bredin (1992) emphasized caregivers’ responsibility for supporting the personhood of individuals with diminished capacity to do so for themselves. Personhood, or the dignity and respect owed to each individual, encompasses an individual’s experiences, preferences and values. For many people, their social history involves significant time spent interacting with and caring for children. Recently, care professionals have turned to intergenerational programming (IGP) as one therapeutic method to support personhood and well-being of persons with dementia (e.g., Camp et al., 1997; Jarrott & Bruno, 2001).

Though contact between young and old generations remains an integral part of most families (Eggebeen & Davey, 1998), non-familial IGP remains a relatively new treatment milieu in elder care. IGP as a whole varies widely, depending on the populations involved, program objectives, and available resources. Interventions targeting children at risk for drug use (Taylor, LoSciuto, Fox, Hilbert, & Sonkowsky, 1999), older adults needing employment (Larkin & Newman, 2001), and foster care families needing community support (Eheart & Hopping, 2001) have utilized intergenerational strategies to achieve program goals. Programs have employed IGP to promote positive affect and engagement of older adults (Xaverius & Matthews, 2003), improved attitudes about aging among children (Middlecamp & Gross, 2002), and greater sense of community among staff (Jarrott, Morris, Kemp, & Stremmel, 2004). Despite the significant cognitive impairment of many elder care clients, practitioners have found elders’ experiences interacting with and caring for children so ingrained that they remain able to interact appropriately and positively with children until late in the progress of a dementing illness (Camp et al., 1997).

The means of assessing IGP vary as much as the programs themselves. Researchers have utilized interviews (Jarrott & Bruno, 2007), observational scales (Marx, Pannell, Papura-Gill, & Cohen-Mansfield, 2004; Xaverius & Matthews, 2003), attitudinal surveys (Jantz, Seefeldt, Galper, & Serlock, 1977; Kocarnik & Ponzetti, 1986), drawings (Lichtenstein, et al., 2001), and cognitive assessments (Newman, Karip, & Faux, 1995) to evaluate the impact of IGP on one or more groups of participants. While the range of scales utilized is not inherently problematic, the current state of intergenerational research tools requires significant improvement for several reasons. First, scales are often created for a single study without any report of psychometric properties (Kuehne, 2004). Consequently, researchers reinvent the wheel rather than use valid, reliable scales. Second, most scales assess the experiences of a single generation (e.g., Camp et al., 1997; Underwood & Dorfman, 2006) although IGP should, by definition (Newman & Smith, 1999), provide mutual benefit for young and old participants. Third, the developmental and disease characteristics of a large portion of IGP participants, pre-school age children and frail elders (e.g., Epstein & Boisvert, 2006; Middlecamp & Gross, 2002; Salari, 2002), limit the opportunity for valid self-report measures. Consequently, researchers often rely on proxy report and direct observation to tap participants’ experiences with IGP.

A critical limitation of much IGP research lies in the black box that conceals the process of bringing young and old together. That is, assessments targeting the impact of IGP often neglect what actually transpired during the IGP. Useful process data will vary from project to project; it might detail the level of activity, the type of interpersonal interactions, the physical environment, facilitators’ behaviors, or the activity’s age appropriateness. For example, Xaverius and Matthews (2003) assessed the impact of IGP involving fourth graders and senior center participants who met for six intergenerational activities. The authors described the theme and setting of activities where elderly participants’ engagement was coded. Data were not gathered regarding the nature of the activities or what happened when participants were engaged in the intergenerational activity (i.e., if they engaged with the activity materials, with an age peer, or with an intergenerational partner).

In contrast, Taylor and colleagues (1999) reported on a senior mentoring program targeting attitudes towards aging, drug use, and civic engagement of participating at-risk youth. The treatment group as a whole demonstrated improved attitudes towards school, civic engagement, aging, and resisting drug use compared to a control group. The authors also tapped into important process data by rating seniors’ intensity of involvement as a mentor. The researchers found a greater degree of attitudinal improvement among children whose senior mentors were more intensely involved with their student partners. Such studies exemplify the importance of capturing process as well as outcome data.

The variety of populations and settings that avail themselves of IGP supports the study of multiple paths leading to positive outcomes. Most would agree that a one-size-fits-all model of IGP is impossible and inappropriate; however, identification of practices and processes that optimize outcomes improves the overall quality of IGP and enhances understanding of how IGP uniquely meets individuals’ needs across the lifecourse. In turn, greater understanding of the processes by which positive IGP outcomes are achieved informs development of theory pertaining to intergenerational relationships. For example, while the contact theory (Allport, 1954) provides necessary conditions for achieving positive intergroup contact, Allport did not describe the processes by which these outcomes would be achieved (Pettigrew, 1998).

The limitations of IGP and related research stem from the relative infancy of IGP research. Researchers have been studying IGP for only the last 30 years (e.g., Jantz, et al., 1977), yet they are trying to raise the field to match those of child and adult development. Practitioners are anxious to know how IGP affects the physical, cognitive, and mental health of participants, yet the more basic question about whether and how children and elders interact with each other during proscribed IGP remains largely unanswered. Before we can reliably draw conclusions about the effects of IGP on children and elderly participants, we must determine the nature of their time spent together. Kuehne’s (2003) state of our art report on intergenerational research implored researchers to tap the experiences of young and old participants and to develop and disseminate standardized measures relevant to IGP. By addressing these points, researchers can better inform practitioners’ efforts and build a cohesive body of research. The scale described in the current paper addresses each of these recommendations.

We sought to address the challenge of measuring the experiences of young and old IGP participants, focusing on frail elders and pre-school age children because they constitute a large portion of participants involved in IGP (Goyer & Zuses, 1998). We conducted a three-phase study to develop an observational scale tapping the social behavior and affect of both young and old IGP participants. We turned to the child development literature, with its long history of observational research, for inspiration.

Parten’s (1933) categories of children’s play behaviors appealed to us; they encompassed categories reflecting a continuum of social behaviors ranging from non-engagement to cooperative engagement. Rubin (2001) developed the Play Observation Scale, drawing on Parten’s work and children’s cognitive development research. The broad social behaviors of unoccupied, watching, solitary, parallel, and cooperative captured by the Play Observation Scale reflect behaviors of interest to practitioners working to support meaningful engagement among elders and children. For example, a code for unoccupied behavior is salient given the high rates of inactivity found at elder care programs (Ice, 2002) and the goal of utilizing intergenerational strategies to promote positive social engagement. Furthermore, our experiences with IGP (e.g., Gigliotti, Morris, Smock, Jarrott, & Graham, 2005; Jarrott & Bruno, 2003; Jarrott & Bruno, 2007; Jarrott, Gigliotti, & Smock, 2006; Jarrott, Gladwell, Gigliotti, & Papero 2004; Jarrott et al., 2004; Weintraub & Killian, 2007) highlighted interaction as the central mechanism for achieving mutual benefit during IGP. Thus, a code for solitary behavior is relevant as it reflects engagement in a presented activity without social interaction. The first author used the original Play Observation Scale to observe elders during structured IGP (Gladwell & Jarrott, 2003), determining that older adults’ engagement was greater during IGP than non-IGP. However, Gladwell and Jarrott found the scale cumbersome as they gathered salient data as well as information of little contemporary significance to IGP. Furthermore, they violated some of the scale’s specifications by utilizing the Play Observation Scale in a structured activity setting.

We made several adaptations to Rubin’s Play Observation Scale for use with structured IGP (see Table 1 for descriptions of the scale categories).  First, Rubin’s social behavior categories included sub-categories indicative of cognitive development; however, given our emphasis on interactions irrespective of developmental abilities, we eliminated cognitive behaviors from our scale. Second, the Play Observation Scale was designed to be used during free play sessions where children self-initiate behaviors. However, intergenerational researchers have repeatedly emphasized the need for structure to optimize IGP (e.g., Camp et al., 1997; Jarrott, 2006; Xaverius & Matthews, 2003), and so we developed the scale with planned IGP in mind. Finally, we expanded the IOS to distinguish between social behaviors with age peers and intergenerational partners (i.e., interactive peer versus interactive intergenerational).

The first phase of the study involved qualitative observations of IGP conducted at a shared site intergenerational program serving frail elders and pre-school age children. The observations were then used to modify Rubin’s Play Observation Scale for use in a structured intergenerational setting. Phase two involved piloting the scale with two observers coding video of IGP and working with the second author to reach consensus and create a master coding scheme for the video sessions. In phase three, the scale was further modified and tested with a larger group of four observers coding video and live IGP. The current paper describes the three phases of the development and initial validation of the Intergenerational Observation Scale (IOS). While the IOS captures both behavior and affect of targeted child and elderly participants, the current paper focuses on social behaviors, which comprise the more complex sub-scale of the instrument.

Method

Participants

Virginia Tech’s Neighbors Growing Together, is a shared site intergenerational program designed to improve the lives of people across the lifespan through intergenerational collaboration involving teaching, research, and outreach. Neighbors Growing Together includes two co-located programs: Adult Day Services and the Child Development Center for Learning and Research. Adult Day Services provides activities, care, and supervision daily to approximately 15 adults (50+ years old) with cognitive and/or physical impairments. The Child Development Center provides year-round, full-day care for 41 children ages 15 months to 5 years. Through daily programming designed to nurture development, enhance competencies, and facilitate positive social interactions between the generations, Neighbors Growing Together provides high quality services to children, older adults, and their families. Children from each of three classrooms have one to two weekly opportunities to join their elderly “neighbors” for IGP, which typically involves three children and three older adults in a variety of activities, such as gardening, art, or sensory projects. Children and adults work together in a group facilitated by staff and students from the Child Development Center and Adult Day Services. Staff partners plan and implement activities that support an overarching goal of positive interactions. Activities further target developmental goals for both generations, such as fine motor skills or cooperation. Children and adults meet in a shared space adjoining the two programs. Child- and adult-sized chairs designed to put all participants at eye level and developmentally appropriate books and art materials are provided.

Procedures

Phase 1: Scale development

In fall 2005, four research students gathered qualitative observations of IGP involving Adult Day Services participants and Child Development Center children. Observers attended different intergenerational sessions. Each week, observers had a distinct focus, starting with holistic observation to orient the observers to the setting and proceeding to target the environment, the participants, and the facilitators. Following their weekly observations, the observers and the first and second authors discussed the observational data, focusing on the interactions between participating children and elders and factors that influenced those interactions. After observing 3-5 intergenerational sessions apiece, observers read and reviewed the IOS scale and codebook developed by Gladwell and Jarrott (2003) and closely mirroring the Play Observation Scale developed by Rubin (2001). They discussed how well the categories applied to the intergenerational context they observed and how to modify the scale to reflect the social behaviors critical to intergenerational interactions in planned activities. Through an iterative process, the first two authors used observers’ notes and conversations to modify the Play Observation Scale to capture data reflecting the interactive process of IGP.

Based on the observations from Phase 1, we further developed and refined the IOS (see Table 1). The scale builds on earlier observational research (Rubin, 2001) by coding participants’ behavior and affect and the affect of intergenerational partners with whom a target participant interacts.  The IOS was developed for live coding. It captures observations for the duration of an intergenerational activity, which tends to last 15-30 minutes. Each observer identifies 4-5 participants for observation and watches them for 1-2 minutes to become familiar with the participants’ behaviors before beginning to record data. He or she codes a participant for one 15-second interval, then codes the next participant for 15 seconds, followed by the third participant, etc. After the last participant is coded, the observer cycles back to begin observing participant one again. All observers in a session start coding when the facilitators begin the activity and end when the activity completes.

When coding, if a behavior occurs for the majority of the coding interval, it is coded as the predominant behavior. When multiple behaviors are observed for equal intervals during a 15-second coding episode, we use the following hierarchy to code the predominant behavior most indicative of intergenerational interactions: Interactive Intergenerational, Parallel Intergenerational, Interactive Peer, Parallel Peer, Staff, Watching, Solitary, and Unoccupied.

Phase 2: Establishment of reliability

Video coding of the observations was introduced during Phase 2 of scale development because learning the IOS through the use of video has several advantages. When first learning the scale, observers watched a 15-second interval repeatedly to better understand the IOS behavioral scoring. The video coding procedure allowed observers to review their coding with the second author, who is experienced in observational coding. Weekly meetings were held during which observers’ coding was reviewed and discussed until observers reached a consensus on what behaviors constituted the IOS categories. Weekly review helped observers achieve acceptable reliability in assigning predominant codes to participants’ social behaviors. The video procedure enabled us to refine the IOS manual.

The video coding process began with two observers who both had experience with live coding during IGP. After studying the manual, observers filmed three weekly sessions of planned IGP between the elders and each of the three classrooms of children (one session per classroom per week). The procedure for using video to establish acceptable reliability was completed in three steps. First, observers independently coded sessions in 15-second intervals and re-watched these as many times as necessary to determine the predominant social behavior. In the second step, observers more closely approximated live coding by watching the 15-second intervals on video only once to code the predominant social behavior.  In the third step, observers coded live sessions (scoring participant behaviors individually but observing the same participants at the same intervals) and filmed the activities, which allowed observers to review their coding at the weekly meetings and resolve discrepancies. Based on the consensus between the two coders and the expert coder, master coding sheets were developed indicating agreement on the predominant behavior for each 15-second interval of the coded sessions.

Before observers could proceed to the next step in the training process, they had to achieve acceptable inter-rater reliability measured by Cohen’s kappa. A kappa score of .60 or higher was considered acceptable (Cohen, 1960). The kappa scores for the two observers in Phase 2 were .67 for the first step, .85 for the second step, and .81 for the live coding in the third step.

Replication of reliability

In Phase 3, we used the IOS video coding procedure to train four new observers. In addition to establishing that the video coding procedure would work with observers new to the IOS, we wanted to determine the amount of training necessary to achieve acceptable reliability with our 3-step process. The observers started with an introductory period where they read the manual and watched live IGP to observe the range of behaviors common to IGP. Observers attended weekly meetings where they watched IGP videos and received instruction on using the IOS. The introductory period lasted approximately 3 weeks or until observers felt comfortable with the coding, which was an additional 2 weeks for one of the observers.

After the introductory period, the observers started on step 1 in the video coding procedure. Each step lasted approximately 4 weeks. At the weekly coding meetings, observers reviewed any discrepancies between their coding and the master codes determined in Phase 2. Once the observers achieved acceptable reliability in step 1, they moved on to step 2. Coding pairs were formed, and they coded live IGP during step 3, achieving acceptable kappas of .92, .69, and .75.

Discussion

In the current paper, we discuss the development of the IOS, a scale designed to measure young and old participants’ social interaction and affect during IGP.  In developing the IOS, we began with qualitative observations of IGP to refine a well-established child development observational scale (Rubin, 2001). We modified the scale for observations of elder and child participants in a typical range of intergenerational programs.  Based on two groups of observers, we present a system for training individuals to use the scale and establish inter-rater reliability.  Currently, the use of standardized measure within the field of IGP research is extremely limited, and our goal is to develop a scale to address this need.

The IOS addresses several of Kuehne’s (2003) recommendations for advancing  intergenerational research and evaluation.  First, the IOS is grounded in theory that shaped our view of positive IGP outcomes. Personhood theory (Kitwood & Bredin, 1992) leads us to intergenerational relationships as a once common source of positive social interaction for many elders that can continue to support their well-being in late life. Contact theory (Allport, 1954; Pettigrew & Tropp, 2000) elucidates necessary conditions for positive intergroup exchange, such as that between youth and elders. The conditions of cooperation and shared goals inform our expectation that the central mechanism of effective IGP is positive intergenerational interactions. The IOS captures the level of interactive behaviors between generations, thus reflecting the presence or absence of cooperation and shared goals of participants.

Second, we draw from standardized measures of social interaction, namely the work of Parten (1933) and Rubin (2001).  Working from their concepts, we qualitatively looked at the behaviors seen during IGP.  We then refined the social behaviors described within Rubin’s scale to fit an intergenerational population and to capture levels of social engagement or interaction during IGP.  Third, we steer away from the over-reliance on attitudinal measures, moving instead to behavioral and affective outcomes of IGP. Fourth, by taking steps to develop a standardized scale, we contribute to the research tools available to IGP researchers, which increases the capacity to compare outcomes across studies, thus advancing the collective understanding of IGP.

Our results establish the IOS as a replicable measure of social interactions during IGP.  The three-step process we used to achieve acceptable inter-rater reliability proved effective with two groups of observers. The process allows observers to develop a thorough understanding of the social behaviors in the IOS. Observers’ coding is checked against the master coding scheme to insure observers’ accuracy coding observed behaviors.  The master coding scheme allows for faster, more accurate training on the use of the IOS. Now that we have developed a successful process for training observers to use the IOS and achieve acceptable inter-rater, observers could be trained relatively quickly, most likely within four to eight weeks. A reliable, quick training method will allow data collection to begin at an earlier date and will allow more researchers to use the scale with confidence.

An important strength of the IOS is its utility with both generations of IGP participants. Most research on individual outcomes of IGP participation focuses on one generation or the other (e.g., Middlecamp & Gross 2002), neglecting the experiences of the other generation. When researchers do evaluate both age groups, they typically use non-parallel measures because parallel scales for disparate age groups rarely exist.  Not only will the IOS allow for standardized data collection across sites; it enables standardized data collection across generations.

We expect the IOS to yield valuable process and outcome data for practitioners and researchers. Considering first the IOS as a source of outcome data, a program introducing or modifying an IGP could, for example, use IOS data to determine the affect and social behavior of participants during IGP with a goal of achieving high levels of positive affect, increasing intergenerational interaction, and reducing inactivity. In regards to process data, the IOS can be used in conjunction with salient outcome measures (e.g., depression or attitudes towards aging) to help interpret the effects of IGP (another recommendation by Kuehne, 2003). In asserting that IGP positively affects (or does not affect) targeted outcomes, researchers’ analyses would be strengthened by including data on the level of intergenerational interaction and affect that transpired among participants. For example, Seefeldt (1987) reported that children who participated in regular intergenerational visits to a nursing home had worse attitudes about aging than children who did not.  These findings would have been more easily interpreted if they incorporated process data such as the affect and level of interaction between children and elders.  Because these data were not included in the analyses, readers are left to speculate whether all IGP involving nursing home residents would negatively affect children’s attitudes or if the nature of the IGP in that study contributed to the negative outcomes. Seefeldt described IGP that yielded little support for intergenerational interaction (i.e., children performing in the center of the room, surrounded by the elders). She also indicated that elders were largely non-responsive to the children, suggesting that the negative outcomes may have been due to the nature of the IGP. While IGP professionals advocate standards of IGP practice (Epstein & Boisvert, 2006; Rosebrook & Larkin, 2003), significant variability on dimensions of IGP likely affects targeted outcomes and should be assessed. By capturing data that reflect the process of connecting generations, the IOS promotes understanding of why IGP succeeds or fails.

As outlined above, the IOS provides many benefits for IGP research and evaluation. Our next step in establishing the scale’s value and utility to the intergenerational field entails establishing the reliability for coding occurrence of behaviors and predominance and occurrence of affect. Second, we plan to use a variety of means to establish validity, including video training to address substantive validity, expert panels to assess content validity, and statistical modeling of IOS data and related measures to determine convergent or divergent validity. Third, the scale was developed within the Neighbors Growing Together shared site intergenerational program, which involves structured programming between preschoolers and frail older adults. Generalizability is a critical indicator of validity and depends on the scale’s utility across intergenerational sites, populations, and programmatic approaches. Fourth, the behavior category codes are developmentally salient for pre-school age children and frail elders; we need to determine if the categories will be equally informative when applied to older children and well elders. Finally, the scale should be tested across cultures to support Kuehne’s (2003) recommendation to take a more global view of IGP innovation.

Initial indicators reveal that the IOS could become a useful tool to researchers and practitioners alike. As they seek to develop, improve, and sustain IGP while linking programming data to instrumental outcomes of interest to practitioners and funders, the availability of a standardized scale appropriate for use with young and old will prove invaluable. At a time characterized by tremendous creative energy and innovation in the intergenerational field, the IOS can capture the essence of IGP as practitioners support meaningful outcomes through intergenerational relationships.

References

Allen-Burge, R., Burgio, L. D., Bourgeois, M. S., Sims, R., & Nunnikhoven, J. (2001). Increasing communication among nursing home residents.  Journal of Clinical Geropsychology, 7, 213-230.

Allport, G. W. (1954). The nature of prejudice. Reading, MA: Addison-Wesley

Camp, C. J., Judge, K. S., Bye, C. A., Fox, K. M., Bowden, J., Bell, M., et al.. (1997). An intergenerational program for persons with dementia using Montessori methods. The Gerontologist, 37, 688-692.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.

Eggebeen, D. J., & Davey, A. (1998). Do safety nets work? The role of anticipated help in times of need. Journal of Marriage and the Family, 60, 939-950. 

Eheart, B. K., & Hopping, D. (2001). Generations of hope. Children and Youth Services Review, 23, 675-682

Epstein, A. S., & Boisvert, C. (2006). Let’s do something together: Identifying effective components of intergenerational programs. Journal of Intergenerational Relationships, 4(3), 87-109.

Gigliotti, C. M., Morris, M., Smock, S., Jarrott, S. E., & Graham, B. (2005). Supporting community through an intergenerational summer program involving persons with dementia and pre-school children. Educational Gerontology, 31, 425-441.

Gladwell, M. S., & Jarrott, S. E. (2003, November). An observational assessment of elders with dementia during intergenerational activities. Poster presented at the meetings of the Gerontological Society of America, San Diego, CA.

Goyer, A., & Zuses, R. (1998). Intergenerational Shared Site Project, A Study of Co-located Programs and Services for Children, Youth, and Older Adults: Final Report. Washington, DC: AARP.

Ice, G. H. (2002). Daily life in a nursing home: Has it changed in 25 years?  Journal of Aging Studies, 16, 345-359.

Jantz, R. K., Seefeldt, C., Galper, A., & Serlock, K. (1977). Children’s attitudes toward the elderly. Social Education, 41, 518-523.

Jarrott, S. E. (2006). Tried and true: A guide to successful intergenerational activities at shared site programs. Washington, DC: Generations United. [Electronic version available at www.gu.org]

Jarrott, S. E., & Bruno, K. A. (2003). Intergenerational Activities Involving Persons with Dementia: An Observational Assessment. American Journal of Alzheimer’s and Related Diseases, 18, 31-38.

Jarrott, S. E., & Bruno, K. (2007). Shared site intergenerational programs: A case study. Journal of Applied Gerontology, 26, 239-257.

Jarrott, S. E. Gigliotti, C. M., & Smock, S. A.,  (2006). Where do we stand? Testing the foundation of a shared site intergenerational program. Journal of Intergenerational Relationships, 4.

Jarrott, S. E., Gladwell, M. S., Gigliotti, C. M., & Papero, A. L. (2004). Fostering intergenerational community between child & adult care programs: A Results Management approach. Canadian Children, 29(2), 4-13.