Practitioner-informed improvements to early childhood intervention performance checklists and practice guides * 1

Results from four early childhood practitioner field-tests of performance checklists and early intervention practice guides are reported. Findings from the first field-test were used to make changes and improvements in the checklists and practice guides were evaluated in the second and third field-tests, and findings from the latter two field-tests were used to improve the checklist and practice guide evaluated in the fourth field-test. The results indicated that changes made in response to practitioners’ suggestions and feedback were associated with (1) progressive increases in the practitioners’ social validity judgments of the checklists, practice guides, and checklistpractice guide correspondence; and (2) progressive decreases in the number of practitioner suggestions and feedback for improving the early intervention materials. The field-test research demonstrates the importance of practitioner input, suggestions, and feedback for improving the usefulness of early childhood intervention practices.


Introduction
Early childhood intervention for infants, toddlers, and preschoolers with identified disabilities or developmental delays and their families, as well as for young children who are at-risk for poor developmental outcomes for biological or environmental reasons is now common practice throughout the world (e.g., Farrell, Kagan, & Tisdall, 2016;Groark, Eidelman, Maude, & Kaczmarek, 2011;Guralnick, 2005;Odom, Hanson, Blackman, & Kaul, 2003;Sukkar, Dunst, & Kirkby, 2017).Early childhood intervention includes the experiences and learning oppor-tunities afforded young children to promote acquisition of functional behavior (Bailey & Wolery, 1992;Dunst & Espe-Sherwindt, 2017) and the supports and resources provided to or procured by parents and other family members to strengthen family functioning (Cowan, Powell, & Cowan, 1998;Dunst, 2017b).
The field of early childhood intervention has a relatively short but rich history (Dunst, 1996;McLean, Sandall, & Smith, 2016;Meisels & Shonkoff, 2000).In the 50+ years since Hunt (1961) first noted that experiences early in a child's life could alter developmental outcomes, and later, that respon-sive caregiving was an important factor in shaping those outcomes (Hunt, 1987), considerable advances have been made in terms of understanding which experiences under which conditions have which kinds of outcomes and benefits (e.g., Britto, Engle, & Super, 2013;Farrell et al., 2016;Odom & Wolery, 2003;Reichow, Boyd, Barton, & Odom, 2016).Early childhood intervention practitioners now have many choices and options in terms of the intervention practices they can use in their work with young children and their families.
Many factors influence practitioners' adoption and use of different kinds of early childhood intervention practices, including, but not limited to, personal beliefs about practice-outcome relationships and one's ability to use a practice competently and confidently (Bruder, Dunst, & Mogro-Wilson, 2011;Trivette, Dunst, Hamby, & Meter, 2012).These beliefs include the social validity appraisals of early childhood intervention practices and also their intended outcomes (Kazdin, 2005).
The practical importance of social validity appraisals is that these types of judgments can help explain why a practitioner does or does not use an intervention practice.Subjective judgments of the importance and acceptability of intervention goals, practices, and outcomes likely influence practitioners' adoption and use of different kinds of intervention procedures (Foster & Mash, 1999).According to Strain et al. (2012), intervention practices are not likely to be used by practitioners (or parents) if the practices themselves are not viewed as socially valid and worth the time and also effort to use.Dunst, Raab, and Hamby (2016), for example, found parents' social validity judgments of interestbased child language learning practices were directly related to parents' fidelity of use of the practices and indirectly related to child language development mediated by fidelity of use of the practices.
The study described in this paper is part of a line of research and practice on (a) the development of evidence-informed early childhood intervention performance checklists and both practitioner and parent practice guides and (b) the influences of practitioner feedback and suggestions on the improvement of both sets of materials.The study involved four field-tests that solicited practitioner social validity judgments of selected checklists and practice guides as well as suggestions for the improvement of both products.Each field-test involved practitioner review and evaluation of a different performance checklist and a different practice guide where feedback and suggestions were used to inform improvements in both sets of products.The findings from the first field-test were used to make changes in the checklists and practice guides in the second and third field-tests, and findings from the latter two field-tests were used to inform improvements to the checklist and practice guide in the fourth field-test.Preliminary findings from this line of research and practice indicated that changes made to the checklists and practice guides in response to practitioner evaluations of the intervention materials were associated with stronger social validity appraisals of revised checklists and practice guides (Dunst, 2017a).

Evidence-Informed Checklists and Practice Guides
The performance checklists and practice guides that were the focus of field-test research were developed at the Early Childhood Technical Assistance Center at the University of North Carolina -Chapel Hill, United States (www.ectacenter.org).Checklists include lists of the key characteristics of a method or procedure that operationally defines the active ingredients of desired performance.Early childhood intervention checklists include the key characteristics of intervention practices that are used to produce observable changes or improvement in child or family functioning.Practice guides include descriptions of everyday intervention activities that can be used to affect changes in child or family functioning.

Performance Checklists
Early childhood intervention performance or procedural checklists provide concrete reminders for using intervention practices in a competent manner (Gawande, 2009;Wilson, 2013).The checklists were developed using a conceptualization-operationalizationmeasurement framework (Babbie, 2009;Dunst, 2017c;Dunst, Trivette, & Raab, 2015) where research findings from primary research syntheses and reviews (Dunst, in preparation) informed checklist indicator selection or development.Performance checklists differ from other types of checklists by specifying a "list of tasks or steps required to complete a procedure [intervention practice] successfully" (Wilson, 2013, p. 4).According to Gawande (2009), these kinds of checklist indicators provide practitioners concrete reminders for how to implement an intervention practice consistently, reliably, and competently.
Twenty-nine performance checklists were developed by first using the Division for Early Childhood Recommended Practices (Division for Early Childhood, 2014) to identify internally consistent sets of practice indicators for different types of intervention practices where the final selection of checklist practice indicators were informed by research evidence.The checklists were all formatted in the same way because "applying organizations to new learning causes learners to focus on the meaning" [intent] of the checklist indicators (Schwartz, 2014, p. 107).
Each checklist includes (1) a brief description of a checklist practice and how the checklist can be used, (2) a list of evidenceinformed practice indicators, (3) a rating scale for doing a self-evaluation or coach-facilitated evaluation of the use of the practice indicators, and (4) space for recording notes about a practitioners' experience using the checklist practices.Appendix A shows the performance checklist that was the focus of practitioner evaluation in the fourth field-test.The reader is referred to Dunst (2017) for a more detailed description of the procedures used to develop the checklists.

Practice Guides
Two sets of practice guides were developed using the checklist indicators as the sources of intervention activities: One set for parents and other primary caregivers and a second set for early childhood intervention practitioners.The practice guides were also all formatted in the same way.Each practice guide includes: (1) a description of a practice and its intended outcome, ( 2) examples of activities for using a practice, (3) videos of parents or practitioners using the practice, (4) a vignette of parents or practitioners implementing a practice, (5) functional outcome indicators for determining if the practice had expected benefits, and (6) a link to external resources for additional ideas (activities) for using a practice.Appendix B shows the practice guide for the Family Capacity-Building Practices Checklist used in the fourth field-test.The practice guides are modeled after the ones that have been extensively field-tested and evaluated by parents, practitioners, technical assistance providers in previous research and intervention studies (e.g., Dunst, Masiello, Meter, Swanson, & Gorman, 2010;Dunst, Trivette, Gorman, & Hamby, 2010;Trivette, Dunst, & Hamby, 2010).

Hypotheses
The analyses focused on two primary and two secondary hypotheses.The two primary hypotheses were: H 1: The social validity judgments of the performance checklists and practice guides will increase linearly as a result of changes and improvements made in response to practitioners' evaluations as evidenced by the sizes of effect for the linear increases and associated improvement indices.
H 2 : The number of practitioner suggestions for improving the checklists and practice guides will decrease linearly as a result of changes and improvements made in response to practitioners' evaluations as evidenced by the sizes of effect for the linear increases and associated improvement indices.The two secondary hypotheses were: H 3 : The sizes of effects and improvement indices for Field-Test 1 vs. Field-Test 4 will be larger than those for Field-Tests 2 + 3 vs.Field Test 4 as a result of the progressive changes and improvements made in response to practitioners' evaluations of the checklists and practice guides.Field-Test 4 will provide the best estimates of the cumulative changes made in response to practitioners' evaluations of the checklists and practice guides.Thus, the four hypotheses were tested by a priori linear and orthogonal contrasts for between-field-test comparisons in the analyses of the field-test research data.

Method
The participants consisted of 67 practitioners from an early head start program in one state and two early childhood intervention programs in other states.The three programs have a history of using innovative practices where the program practitioners are knowledgeable about contemporary evidenceinformed early childhood intervention practices.There were no between-group differences in the percentage of participants in the different field-test studies, χ 2 = 6.68, df = 6, p = .3516,and nor was there in the percentage of participants in the type of early childhood program in the field-tests, χ 2 = 2.77, df = 6, p = .8375.Note: a Speech and language pathologists, child and family specialists, early interventionists, and social workers/family workers.
The background characteristics of the participants are shown in Table 1.The majority of practitioners (75%) had either bachelor's or master's degrees.Most of the practitioners had degrees in early childhood education or early childhood special education/ special education.The participants' median years of experience ranged between 6 and 10 with 78% having from 6 to 20+ years of experience.Nearly two-thirds of the participants worked primarily with parents and their children (family-focused) and 37% worked primarily with children (child-focused).There were no between field-test differences for any of the participant background characteristics shown in Table 1, χ s = 0.17 to 10.73, df s = 3 to 15, p s = .1004to .9817.

Procedure
The performance checklists and practice guides that were the focus of the field-test evaluations are shown in Table 2.The four topic areas included child, parent-child, parent, and family-focused intervention practices.Both the checklists and practice guides included different kinds of interventions for (a) using everyday activities as sources of child learning opportunities and (b) parent sensitivity and responsiveness to child behavior in the activities as the primary caregiver practice to reinforce child competencies and sustain child engagement in the activities.
The checklist in the first field-test included practice indicators for strengthening caregiver and child relationships that focused on bidirectional, reciprocal interactions between interactive partners (Eshel, Daelmans, Cabral de Mello, & Martines, 2006).The practice guide for the checklist indicators included a number of different socially interactive games that caregivers could use to engage young children in your turn-my turn interactive episodes (e.g., Dunst, Pace, & Hamby, 2007).The checklist in the second field-test included indicators for identifying everyday activities that provide the most opportunities for child learning (Dunst, Bruder, Trivette, & Hamby, 2006).The practice guide included ideas and strategies for engaging a child in the activities (Dunst, Raab, & Trivette, 2013b).The checklist in the third field-test included indicators for using naturalistic teaching practices for reinforcing child behavior initiations and elaborations while engaged in everyday activities (Dunst, Raab, & Trivette, 2011).The practice guide included different kinds of intervention activities and strategies for using natural reinforcing consequences for reinforcing child behavior (e.g., Dunst, Raab, & Trivette, 2013a).
The checklist in the fourth field-test included methods for strengthening family capacity to provide a child everyday learning opportunities (Swanson, Raab, & Dunst, 2011).The practice guide included a set of step-bystep instructions for practitioners to use to encourage and support parent-mediated everyday child learning (e.g., Raab, Dunst, & Trivette, 2013).
The checklist and practice guide in the first field-test had not been subjected to prior review and feedback, and practitioner evaluations of both products were used as the baseline for evaluation of subsequent revisions and improvements to the checklists and practice guides in the second and third field-tests.The changes to the checklists in response to practitioner feedback and suggestions included clarifying the purpose of the checklist instructions and intended users (practitioners), rewording the checklist indicators to improve meaning and intent, clarifying how to use the checklist indicators to plan intervention sessions with parents, and the way how to use the rating scale to do a self-evaluation of how many and how well the checklist indicators were used with a child or parent.The changes to the practice guides included adding captions to the videos of parents or practitioners using the practices, adding additional intervention activities to the practice guides, including suggestions for making adaptations to the practice guide activities (where appropriate), and clarifying how to use the outcome indicators for evaluating the benefits of the practice guide activities.
Feedback and suggestions on the second and third field-tests were used to make additional changes to the checklist and practice guide in the fourth field-test.The changes to the checklist included clarifying the difference between using the checklist indicators for a priori intervention planning and doing a post hoc self-evaluation of the use of the indicators and clarifying the instructions for how to use the checklist indicators for completing a self-evaluation.The changes to the practice guide included additional specificity in terms of the focus and intent of the practice guide as well as the practice guide activities.

Field-Test Survey
The survey included four sections: (1) practitioner social validity judgments of the checklists, practice guides, and correspondence between practice guides and checklist indicators; (2) open-ended questions asking for suggestions to improve the checklists and practice guides; (3) levels of experience needed for a practitioner to understand and use checklists and practice guides; (4) background information about the field-test participants (Table 1).Each field-test involved an emailed invitation sent to the directors of each program that included instructions for participation in the field-tests, PDFs of the checklists and practice guides, and a URL link to the survey.The program directors were asked to forward the emailed invitation to their staff.Participation in the field-tests was voluntary, Carl J. Dunst, Deborah W. Hamby, Linda L. Wilson, Marilyn Espe-Sherwindt, & Donna E. Nelson and the field-test research was considered exempt from human subjects review because practitioners were asked only to evaluate materials designed for routine early childhood intervention.The surveys were completed online using Qualtrics Survey Soft-ware.
The social validity items for the performance checklists, practice guides, and checklist-practice guide correspondence (four per each section) were developed by using Foster and Mash's (1999) framework for developing indicators for measuring the importance and acceptability of intervention practices and outcomes.In addition, the social importance of the checklists and practice guides was measured in terms of the subjective value attributed to the intervention materials (e.g., The checklist items are easy to understand and follow; The practice guide activities would be engaging to most children).The social acceptability of the checklists and practice guides was measured in terms of judgments about the fit of the practices to everyday life (e.g., The checklist indicators would be easy to use with a parent or child; The practice guide would be worth my time and effort to use).The social validity items were each rated on a 5-point scale ranging from Do Not Agree At All (with the survey items) to Agree a Great Deal (with the survey items).The items were adopted from the ones used in field-tests of other intervention practices (e.g., Dunst et al., 2007;Dunst, Trivette, et al., 2010).
The principal component factor analysis of the three sets of items in each field-test with orthogonal rotation each produced a single-factor solution indicating that summated scores were warranted as measures of social validity judgments (Spector, 1992).The average coefficient alpha for the checklist indicators was .89(Range = .81to .97), the average alpha for the practice guide indicators was .85(Range = .77to .91), and the average alpha for the correspondence between the checklists and practice guides was .92(Range = .85to .95).The alpha's in all 12 factor analyses reached acceptable levels of internal reliability (Nunnally & Bernstein, 1994).
The open-ended questions for improving the checklists asked for suggestions about the (1) checklist instructions, (2) checklist indicators, (3) self-evaluation scale, and ( 4) any other suggestions for improvement.The open-ended questions for improving the practice guides asked for suggestions about the (1) practice guide format, (2) practice guide activities, (3) videos of the practices, (4) child outcomes, and (5) any other suggestions to improve the practice guides.

Methods of Analysis
The 3 Between Field Test ANOVAs with preplanned linear and between group contrasts were used to evaluate the effects of changes to the checklists and practice guides on participants' social validity judgments.The independent variable was the different fieldtests (Field-Test 1 vs. Field-Tests 2 + 3 vs.Field-Test 4).The linear contrasts and between-field-test comparisons permitted tests of the four study hypotheses.The dependent measures in three ANOVAs were the summated social validity scores for the performance checklists, practice guides, and correspondence between the checklists and practice guides.
The primary metrics for testing the study hypotheses were Cohen's d effect sizes and associated improvement indices (What Works Clearinghouse, 2014).Effect sizes rather than statistical significance testing is the preferred metric for substantive interpretation because effect sizes and not p-values are the best estimates of the magnitude of the differences between two groups or contrasts (Coe, 2002).As a general rule, effect sizes between .20 and .49are considered small, those between .50 and .79 are considered medium, those between .80 and 1.19 are considered large, and effect sizes equal to or greater than 1.20 are considered very large.Improvement indices are measures of the practical importance of the changes made to the checklists and practice guides (Durlak, 2009).The indices convert effect sizes into a percentile change (gain) score by a target group.These indices vary from -50 to 50 where a positive difference between later and earlier field-tests provides a measure of the amount of improvement that occurred as a result of changes made to the checklists and practice guides.ZCalc was used to evaluate the improvement indices (Neill, 2006).
Primary analyses of the practitioners' social validity judgments were supplemented by computing the percent of indicators rated a 4 or 5 on the 5-point scale to ascertain the overall levels of agreement with the indicators.As found in consumer sciences research, the larger the percent of indicators rated a 4 or 5 on a 5-point scale, the stronger the endorsement of a product, practice, or service (Mackiewicz & Yeats, 2014;Reichheld, 2003).The Mantel-Haenszel test for linear trends was used to determine if there were progressive increases in the percent of practitioners rating the social validity items a 4 or 5 from the first to fourth field-tests (SPSS Inc., 2005).
The effects of the changes made to the checklists and practice guides in response to practitioner suggestions were tested by both 3 Between Field-Test ANOVAs for the total number of practitioner suggestions and by 3 Between Field-Test Chi-Square analyses for dichotomous responses for each open-ended section.The same linear contrasts for the social validity appraisals were made for evaluating changes in the practitioner suggestions.

Social Validity Judgments
Figure 1 shows the mean social validity scores for four field-tests for each set of importance and acceptability judgments.The 3 between field-test ANOVAs produced between group differences for the practitioner social validity judgments of the performance checklists, F(2, 64) = 3.49, p = .0364,and checklist/practice guide correspondence, F(2, 64) = 4.94, p = .0101,but not for the practice guides, F(2, 64) = 0.42, p = .6562.
The results for linear contrasts and between-field-test comparisons are presented in Table 3.There were small linear increases to the practice guides to medium linear increases to both performance checklists and checklist/ practice guide correspondence increases from the first to fourth field-tests as evidenced by the sizes of effect for the linear trends.The effect sizes were associated with improvement indices of 9, 22, and 27 percent, respectively, in response to the progressive changes in the practice guides, checklists, and correspondence comparisons.The findings are consistent with Hypothesis 1 that changes made in response to the practitioners' feedback and suggestions would be related to improvements in the social validity judgments of the checklists and practice guides.There were small (practice guides) to large (checklists and checklist/practice guide correspondence) effect sizes for the differences between the Field-Test 1 vs. Field-Test 4 social validity judgments (Table 3).These were associated with improvement indices of 11, 32, and 36 percent, respectively, for the practice guides, checklists, and correspondence judgments.The effect sizes for the Field-Tests 2 + 3 vs.Field-Test 4 for the betweenfield-test comparisons were small for both the performance checklists and checklists/practice guides correspondence.The betweenfield-test comparisons were associated with improvement indices of 9% for the checklist differences and 13% for the checklist/practice guide correspondence differences.The comparisons of the two sets of results in Table 3 shows, as hypothesized, that the sizes of effect and associated improvement indices for Field-Tests 1 vs. 4 are considerably larger than those for Field-Tests 2 + 3 vs. 4.
The cumulative effects of the progressive changes made in response to the practitioner evaluations are evidenced from the Field-Tests 1 + 2 + 3 vs.Field-Test 4 comparisons.There were small (practice guides) to medium (checklists and checklists/practice guides correspondence) effect sizes for these between-field-test comparisons.The effect sizes were associated with improvement indices of 8, 25, and 28 percent, respectively.The results are consistent with the hypothesized relationships between changes made in response to practitioner feedback and suggestions and improvements in the social validity judgments of the intervention practices.
The percent of social validity items rated a 4 or 5 on each section of the survey for the different field-tests are shown in Figure 2.There were linear increases in the percent of indicators rated a 4 or 5 for the performance checklists, χ = 9.04, df = 1, p = .003,d = .79,practice guides, χ 2 = 5.88, df = 1, p = .015,d = .62,and checklist/practice guide correspondence, χ 2 = 10.98,df = 1, p = .001,d = .97.The effect sizes for the linear trends were medium to large and associated with improvement indices of 29, 23, and 33 percent, respectively.The smaller effect size for the linear increase in the social validity ratings of the practice guides was not unexpected given the fact that practitioner judgments of the practice guides were higher than those for the checklist on the first three field-tests.As shown in Figure 2, 98% to 99% of the social validity items received the highest two ratings in the fourth field-test which are noticeably higher than that in the other three field-tests.Figure 3 shows the percent of practitioners who made suggestions for improving the checklists and practice guides in the different field-tests.The 3 between-field-test ANOVAs for the total number of practitioner suggestions produced between field-test differences for both the performance checklists, F(2, 64) = 7.11, p = .0016,and practice guides, F(2, 64) = 10.51,p = .0001.There were linear decreases in the number of suggestions for the checklists, F(1, 64) = 11.41,p = .0012,d = .85,and practice guides, F(1, 64) = 18.31, p = .0001,d = 1.07.Both sizes of effects were large for the linear decreases in the number of practitioner suggestions.The effect sizes were associated with the improvement indices of 30% and 36% respectively.The patterns of results are consistent with the hypothesis that the practitioners would suggest fewer changes as a function of improvements made in response to their feedback and suggestions.
Further examination of the suggestions for improving the checklists found linear decreases in the percent of practitioners who made suggestions for changes to the checklist instructions, χ 2 = 6.43, df = 1, p = .011,d = .77,the checklist indicators, χ 2 = 6.96, df = 1, p = .008,d = .83,and the self-evaluating rating scale, χ 2 = 2054, df = 1, p = .0555,d = .45.The sizes of effects were medium, large, and small, respectively, and associated with improvement indices between 17% and 30%.There were also linear decreases in the percent of practitioners making suggestions to improve the practice guide format, χ 2 =12.43, df = 1, p = .0000,d = 1.15, practice guide activities, χ 2 = 9.49, df = 1, p = .001,d = .87,practice guide outcome statements, χ 2 = 2.06, df = 1, p = .051,d = .38,and videos of parents or practitioners using the practices, χ 2 = 17.17, df = 1, p = .0000,d = 1.36.The effect sizes were small to very large and associated with improvement indices between 15% and 41%.These findings, taken together, further support hypothesized relationships between the changes made in response to the practitioners' evaluations of the checklists and practice guides and fewer suggestions for improving the intervention practices.

Discussion
The findings provide support for the two primary hypotheses that changes made to the performance checklists and practice guides in response to early childhood practitioners feedback and suggestions would be related to the study outcomes.The results showed that practitioners' social validity ratings increased as a function of the improve-ments to both the performance checklists and practice guides and also to the checklist/practice guide correspondence.The results also showed that there were fewer suggestions for making changes to the checklists and practice guides as a function of using practitioner feedback to improve both sets of products.
The patterns of results also provide support for the two secondary hypotheses.The sizes of effect for the first vs.fourth field-tests were larger than those for the second and third vs. fourth field-tests (Table 3).These results were expected because fewer suggestions for changes to the checklists and practice guides were made on the second and third field-tests compared to the first field-test (Figure 3).The cumulative effects for the changes made in response to practitioners' suggestions were evidenced by the sizes of effect for the first three field-tests vs. the fourth field-test.Both the effect sizes for these comparisons and improvement indices (Table 3) indicated that the progressive sets of changes made in response to practitioners' suggestions were associated with the highest social validity rating (Figure 1) and fewest suggestions for change (Figure 3) on the fourth field-test.
The fact that the effect sizes and improvement indices for the practice guides were smaller than those for the performance checklists and checklist-practice guide correspondence deserves comment in order to place the results in empirical context.The practice guides were modeled after the ones that had previously been field-tested with parents and practitioners where the results were used to improve the intervention materials (e.g., Dunst, Trivette, et al., 2010;Trivette, Dunst, Masiello, Gorman, & Hamby, 2009).It was therefore not unexpected that the majority of social validity indicators for the practice guides on the first three field-tests were higher than those for the checklists and checklistpractice guide correspondence (Figure 3).This was the case because the practice guide format and content were informed by lessons learned in previous research and practice.
The focus on the social validity of the performance checklists and practice guides was based on research indicating that subjective judgments of the importance and accept-ability of intervention practices and outcomes are related to both adoption and fidelity of use of the practices (e.g., Dunst et al., 2016;Strain et al., 2012;Trivette, Raab, & Dunst, 2014;Wainer & Ingersoll, 2013;Wehby, Maggin, Moore Partin, & Robertson, 2011).As noted by Strain et al. (2012), these "likingimplementation with fidelity relationships" (p.197) are important because they help explain at least the likelihood of early childhood intervention practices being used as intended.

Conclusion and Suggestions
The study described in this paper has both strengths and limitations.One strength is the fact that the procedures used to inform changes in the checklists and practice guides illustrates how the consumer level input can be used to improve social validity appraisals of the intervention materials constituting the focus of evaluation.Another strength is establishing the inverse relationship between increases in social validity ratings and concomitant decreases in practitioner suggestions for changes.In another set of analyses in this line of research and practice, practitioners' cognitive judgments of the performance checklists and practice guides were the only variable accounting for variations in the social validity ratings of the intervention materials (Dunst & Hamby, 2017).
One limitation of the study is that the field-tests were conducted in only three early childhood intervention programs.Therefore, it is not known if practitioners in other early childhood intervention programs would judge the checklists and practice guides in the same or different ways.Another limitation is the fact that only 4 out of 29 performance checklists and only 4 out of 67 practice guides were evaluated in the field-tests.Whether other checklists and practice guides would be judged similarly is therefore not known.
Advances in our understanding of the role social validity judgments play in practitioners' and parents' use of different kinds of early childhood intervention has broadened our knowledge of the antecedents for and conditions under which intervention practices are used with fidelity (Leko, 2014;Strain et al., 2012).One simple way of assessing practi-tioners' and parents' social validity appraisals is to ask the question "Was using XYZ practice worth your time and effort or was it more trouble than it was worth?"If a practitioner or parent responds that it was not worth the trouble to use, it is unlikely that the practice will be used with fidelity or used at all.
Although the field-test process was used to inform improvements in early childhood intervention performance checklists and practice guides, the process itself could easily be used in other fields for achieving performance excellence.This is especially the case in professions where there needs to be practitioner buy-in to ensure actual performance mirrors desired performance.

Figure 2 .Figure 3 .
Figure 2. Percent of social validity items judged as acceptable and important by the field-test participants Practitioner Suggestions

Table 1 .
Background characteristics of the field-test participants

Table 2 .
Performance checklists and practice guides that were the focus of practitioner social validity judgments and feedback