Interprofessional Multi-Source Feedback Tool

BACKGROUND

Multisource or 3600 assessment has become increasingly used in graduate medical education. The process involves assessment of an individual by numerous individuals with whom the learner has interacted in the workplace setting over an extended period of time. Attending physicians, residents, nurses, other health care providers, students, and patients may be involved. The focus is generally formative, intended to provide “continuous, longitudinal data and feedback to all healthcare providers,” with a focus on the development of learners who are capable of co-creating their own competencies and “self-directed in seeking ongoing assessment and feedback and capable of continuous reflection, evaluation of, and modifications of his or her emerging roles” (Holmboe & Batalden, 2015, p.1219). Multisource assessment has generally been difficult to use with undergraduate health professions students since clinical workplace assignments are generally brief as the students rotate through a variety of specialties. With the increased use of longitudinal integrated clerkship models in medical education, multisource feedback can be used as it has been in residency education. Our multisource survey uses the Interprofessional Collaborator Assessment Rubric (ICAR) developed and validated previously by Curran et al (2011).

IMPLEMENTATION

  1. Tool Name: Multisource Feedback Tool for Interprofessional Collaborative Practice

  2. Participants: The Multisource feedback survey can be used with any level of health professions student, resident, or provider. Numerous studies that demonstrated that 5 to 11 physician assessors are needed to provide acceptable reliability for multisource surveys with larger numbers needed for non-physician assessors or patients. However, given use as a formative instrument, we selected 5 to 6 assessors as a reasonable sample, noting that assessors need not agree since they are each viewing the encounter with the students from different perspectives, all of which can be useful.

  3. Setting: Clinical workplace.

  4. Requirements: We have chosen to deliver the survey via Research Electronic Data Capture (REDCap), free, web-based software developed by Vanderbilt University and widely used by the REDCap Consortium schools. Individuals from health professions schools that have adopted REDCap should contact their local research support services for information. All others can learn more from http://www.project-redcap.org/

  5. IPEC Competencies: Communication, teamwork

  6. Administration: We believe that the utility of the Multisource Feedback survey is higher in those workplace assignments, in which students have spent four or more weeks. Longer placements can provide both multiple opportunities for use of the Multisource survey and/or more continuity with teachers, other providers, and peers. To administer The Multisource Feedback survey, we developed an electronic communication tool allowing information flow between an observee, an observer, and the REDCap. The process begins with an email to students asking them to provide the names and email addresses of those whom they would like to complete the multisource feedback survey. The email also has a link to the fillable observer identification form. The observer identification form contains fields for student information (first name, last name, and email) as well as names and emails of 6 observers and a “Send to Observers” button. The observer identification form states: “Please select a broad range of individuals with whom you have worked. Since the attending physicians will be completing a final evaluation form for your work in a clerkship, we suggest that you select other team members for this exercise.” http://apps.medsch.ucla.edu/medyear3/observer_signup/form.php


  7. [PICTURE]

    Students are reminded that this is not an evaluation but a way to learn more about how they are perceived as a team member for purposes of improvement. Once students have submitted the observer names, the system generates a request to the observers to complete the survey and a link to the survey automatically prefilled with an observee’s name.

    [PICTURE]

    Although REDCap produces reports in several formats, we created a report template that could be easily populated with a single download of results from REDCap. Copies of the multisource survey, the observer identification form, the observer request email follow this guide along with instructions on setting up REDCap to handle all of these tasks.
  8. Recommended uses: As an assessment tool: The Multisource survey is intended for feedback purposes only. It is important to implement the process by the mid-point of a longitudinal workplace assignment so that time remains for improvement based on the feedback. If used for summative assessment, then, to reach an acceptable level of reliability (.70), the number of observers needed will vary based on the number of different occasions, the competencies being assessed, and the type of observer. Estimates across multiple studies suggest that when used on a single occasion, you will need 5 to 11 physicians, 10 to 20 non-physicians, and 50 patients. As an instructional tool: The Multisource survey can also be used for classroom teams and as part of simulation activities with debriefing for both individual and group performance. As a program evaluation tool: Results can be aggregated across students, occasions, and settings as a needs assessment.

  9. Strategies for providing results to students: Although REDCap provides several report templates, we developed a different reporting format in Excel to allow for distribution of responses rather than descriptive statistics. The report shows how each individual assessor responded on each item so that the pattern of responses is the focus of the feedback. Gingerich et al (2014) have suggested that variability among assessors may be more realistic and helpful to the person being assessed when feedback is the purpose of the exercise. In this respect, the authors stated:

    rather than reflecting suboptimal judgments, inconsistencies among assessors’ interpretations may very well reflect the complexity of the performance and the inherently ‘subjective’ interpretation of that performance filtered through the assessor’s understanding. If differences in assessment judgments were to come from differences in the way the trainee’s performance can be perceived and experienced by others, then the inconsistencies among assessors’ interpretations might be complementary and equally valid. (Gingerich et al. 2014, p.1060)

    Students receive an individual report for personal feedback purposes only, including a guide to interpretation. An example is attached to this guide. We chose not to share results with teachers but encouraged students to share their results with mentors and advisors for purposes of discussion.

  10. Challenges: There is a debate in the literature about whether students/residents should identify their own observers. In one study, ratings by self-selected individuals versus those of assigned supervisors were significantly higher. (Brown, Fillingham, Murphy et al, 2014)

    Unless formative assessments are required, we found that students were reticent to participate. Weaker students may be more reticent than others to seek out feedback opportunities so close monitoring and timely completion is necessary.

    Students may not know the full name of the many potential assessors with whom they are collaborating in the workplace and email addresses may be even more difficult for the student to identify.

    Multisource surveys may only be useful in assessing a very limited range of competencies given the use of assessors who represent many different types of interaction with the student. Interpersonal communication, professional behaviors, and teamwork competencies are more readily observed across types of assessors.

    The logistics of conducting a multisource assessment are complex and available software tools programmed for this purpose are expensive. Gathering enough assessors also poses a problem given brief workplace assignments and an ever shifting team composition.