This project to assess change in undergraduate life education was piloted over the last year by the Partnership for Life Science Undergraduate Education (PULSE). The project seeks to recognize life science programs that have progressed in implementing the recommendations of the 2011 Vision and Change Report: A Call to Action. This report details the consensus among life science educators about how to teach and motivate students learning, with many recommendations based on understanding of how learning works from cognitive, neuroscience and learning science/education research. The Vision and Change (V&C) recommendations place more emphasis on scientific reasoning and the ability to think critically rather than the mastery of facts only. Laboratory work and hypothesis generation and testing is given a new prominence and the focus shifts from teacher-directed exercises to student-centered ones.
The recognition project began with a group of PULSE fellows who were interested in rewarding and recognizing those life science programs that had made strides in implementation of the V&C recommendations, reasoning that such recognition would be motivating for faculty and departments and provide objective standards by which to judge progress. The group initially researched a large number of existing certification and accreditation models and communicated with other groups that are developing similar models for certification in the higher education arena. The group also looked at rubrics developed by other organizations, such as the AAC&U Value rubrics and the Project Kaleidoscope (PKAL) departmental thermometer, among others. From this research, the certification team decided on a ‘certification’ rather than an ‘accreditation’ model, one that is built along the lines of the tiered LEED (Leadership in Energy and Environmental Design) certification program (http://www.usgbc.org/leed). Since the inception of the original project, the name has been changed to PULSE Recognition.
A key first step was solicitation of the broad life sciences education community to provide input on the characteristics of the ideal department, as envisioned by the Vision and Change report. From this input and the V&C recommendations/report, the team developed a set of rubrics to measure the degree to which life science programs had achieved the goals of V&C and progressed toward full implementation. Major categories aligned with the overall recommendations were developed and specific criteria within each category were set out to measure quantitatively where programs lie along a continuum of implementation of those recommendations. The rubrics were designed to be applicable to all types of higher education institutions and are meant to provide a relatively objective measure of progress. These rubrics are designed to form the basis for ‘peer review’ of life sciences programs alignment with V&C.
The Vision and Change Rubrics
There are five areas covered by the rubrics.
During the development of the rubric criteria, input was sought from the life sciences community via the PULSE community website, at workshops at various education and scientific meetings, and from a number of biology programs that were familiar with V&C recommendations, including many of the Fellows departments. The PULSE Vision and Change Rubrics (version 1.0) were released to the community via the pulsecommunity.org website in July 2013 and further publicized as a “Letter to the Editor” in CBE Life Sciences Education, December 2013.
The Pilot Program
The pilot program was undertaken with the support of an NSF grant (Award # 1350120) that proposed to recruit eight institutions to participate in an initial evaluation process. In November 2013, a call (via the pulsecommunity.org website) was put out for life science programs to apply to be one of the eight schools in the PULSE certification pilot program. The application included questions that asked about efforts and current status of alignment with Vision and Change recommendations, as well as a letter of support from the Dean or Provost. The team was surprised and encouraged when more than 70 applications were received. In January 2014, two schools from each institution type, including one minority serving institution were selected; each had strong letters of administrative support and provided evidence of engagement with the process of departmental transformation.
The eight pilot schools provided information including: scores on the PULSE Vision and Change Rubrics, written justification for rubric scores, CVs and a statement of pedagogical philosophy from department faculty, information about courses and curriculum, course syllabi, sample exams, department governance structure, and descriptions of key staff positions. The materials were evaluated to obtain a complete picture of the state of each program relative to V&C implementation.
To ‘ground truth’ the information and determine if the rubrics effectively evaluated progression towards the ‘ideal department’ as intended, site visits took place during 2014, with teams consisting of two members of the Certification group. Each site visit team included at least one member who is from an institution of the same type as that visited. The self-reported scores were quite accurate, with only minor adjustments in a few scores made by the visiting team. Most aspects of the program’s progress were well represented in their rubric scores, but there were a few important aspects that were not captured well. To make sure the aspects that were not captured were included in the scoring, discretionary points were added as appropriate. For example, some programs had an exceptionally cohesive and supportive faculty culture or a unique educational ‘gem’ that had an important impact on the student experience. Most programs earned discretionary points. In the future, the Certification group plans to make revisions to the Rubrics to attempt to capture these aspects.
Another factor that helped with interpretation of pilot programs rubric scores is additional rubric data that were entered via an online rubric portal, which was made available to the community in April 2014. Complete data from 18 programs, including the 8 pilot schools, were analyzed. While the rubric data entered via the portal is self-reported, with no justifications or ‘ground truthing’ via site visits of the scores for programs other than those in the pilot, these data help provide a broader view of the spectrum of transformation. Having data from these programs also allowed some statistical analysis to ensure that the rubrics did not disadvantage or advantage any institution type. The analysis supports the idea that there is no systematic bias that would disadvantage one type of institution or prevent one institution type from being able to achieve the highest level.
The project has also collected partial rubric data from more than 30 additional programs. Analysis of this larger data set is underway.
Data is still being collected via the electronic portal, with the aim of benchmarking the state of life sciences education programs at the present time and to document change in programs over time. The project seeks data from any life science program along the spectrum of V&C implementation as part of this initial effort and plans to continue to collect data over time and to ask programs to update their scores after a few years to document any change.
The Certification team hopes to move forward with an ongoing certification program, after making some revisions to the rubrics and supplemental information gathering strategy and content.
Outcomes from this pilot were released June 1, 2015, with all eight programs being recognized as having progressed along the continuum of change. Five levels of achievement were defined. Earning any score greater than zero indicates significant progress in V&C implementation. The data set available is limited at this time and the programs that voluntarily submitted data are likely to be already engaged in V&C motivated reform efforts. Thus, the team believes that the data so far collected does not likely represent the full spectrum of V&C implementation.
From the 18 complete data sets, the progression level of each program was estimated. These programs included 4 research universities, 3 regional comprehensive/master’s granting schools, 6 liberal arts colleges and 5 two year schools. Overall, among the 18 schools, 4 are at level one (“beginning”); 11 at level two (“developing”); and 2 at level 3 (“accomplished”). [One program that was not part of the pilot was very close to the level 2-3 border and due to the additional information used in evaluating the pilot schools and setting the level cutoffs, it is not possible to be certain whether this program would have fallen into level 2 or 3.] It is clear from this small sample that a wide range of institutions are making good progress, but there is still a long way to go to realize the promise of the V&C report recommendations to catalyze changes in undergraduate life sciences programs that will result in better educating our nation’s future life scientists and citizens.
Comments from the administrators and faculty at the schools that participated in the pilot have been very positive and indicates that their engagement with the process has been rewarding and important.
Charles L. Robbins, Vice Provost Undergraduate Education and Dean of the Undergraduate Colleges at Stony Brook stated,
“Stony Brook University is very pleased to have been selected to participate in the PULSE certification process. It is very gratifying to be recognized for the positive changes that we have made in teaching our introductory Biology Laboratories. Having participated in the National Vision & Change Conference as well as the [PULSE] campus site visit, I am acutely aware of how rigorous and comprehensive this process is. As Vice Provost for Undergraduate Education, I am also very aware of how important these efforts are. We are pleased to share that this review has already served as a catalyst for a new program [for Graduate TA training] to be implemented in the Fall 2015 semester. We look forward to our continued collaboration with our PULSE colleagues.”
Barbara Lom, Chair of Biology, Davidson College, stated,
“Davidson College is greatly honored that the PULSE Certification Pilot Program has recognized our Biology Department for its “Accomplished” work in undergraduate biology teaching. Long before an official “Vision and Change” nationwide effort, Davidson’s Biology Department intentionally and thoughtfully developed a culture of student-centered pedagogy that focuses on integrating research, trans-disciplinary problem-solving, active classroom learning, and scientific responsibility throughout the curriculum. The department proactively gathers a variety of data in order to evaluate the impact of new pedagogical approaches on student learning. All of this takes hard work, dedication, creativity, and a joy for teaching, and my colleagues in Biology have these qualities in abundance. We at Davidson are proud of our Biology colleagues’ commitment to engaging students in doing science, and we are pleased and grateful that the PULSE Pilot Certification Process has both encouraged and acknowledged their accomplishments.”
Wendy Raymond, Vice President for Academic Affairs and Dean of the Faculty, Davidson College, was enthusiastic about Davidson’s participation and the potential for the Certification program stating,
“Thank you for your helpful analysis and report of the educational program offered by Biology at Davidson. We are grateful to have been part of the PULSE Certification Pilot Program. Our Biology faculty worked hard in advance of your visit, and I appreciate that you note well that their creative and effective engagement with undergraduate students results in ‘an extremely strong program with a large core of engaged faculty that is committed to student success.’
Certification could attract the attention of many college and university programs who would thereby be motivated to invest in active-learning strategies centered on doing science. Certification could lead deans like me to invest in those departments' efforts to increase active classroom learning approaches. All told, those are big outcomes for a "small" certificate. In other words, I think certification could leverage institutional transformation locally and widely, which is no small feat. But perhaps of equal importance for a dean is the validation such external certification would give to the scientist-professors who have invested their intellects, hearts, and souls into this set of best practices for our students' educations. While it's important that I as dean acknowledge this excellence, I think that the outcomes of a respected peer-review process such as this PULSE pilot program can do wonders for a program's self-esteem, continued motivation, and sustained investment in undergraduate education.”
Dr. Cleo Hughes Darden, Chair of the Biology Department at Morgan State University, stated
“As an original participant in the Vision and Change movement, this pilot will better align our ongoing vision and activities with the National Vision and reveal unique roles for our institution in its fulfilment. We are committed to excellence in our role as one of the top ten producers of African American PhD recipients in the Life Sciences (http://www.nsf.gov/statistics/infbrief/nsf13323/), and we look forward to actively working with PULSE to improve Life Sciences education at Morgan State University and across the nation.”
Mark Lyford, Director, Life Sciences Program, University of Wyoming stated that
“The University of Wyoming’s Life Sciences Program has benefitted greatly from being involved in the PULSE Pilot Certification Program. Having an external evaluation of our program has done two key things. First, it has provided an important validation that our program is doing many things really well and is already well-aligned with Vision & Change. Second, the process has helped us identify areas where we can improve to meet the call of Vision and Change, which has helped us leverage resources on our campus to do so.”
Malcolm Campbell, Biology faculty member at Davidson College and author of a introductory biology textbook based on the recommendations of Vision and Change report, stated,
“It is common for college departments to be evaluated by outside visitors, but the PULSE evaluation was an order of magnitude more rigorous. The PULSE evaluators measured the Department’s success in moving towards the goals of V&C. ……… [The faculty in Biology] are very proud of their PULSE evaluation and they look forward to additional improvements to the biology curriculum at Davidson. The entire department is responsible for the high rating and together, they will continue to adjust what they do to help students learn and retain as much as possible.…….An area that Davidson needs to improve upon is assessment and responding to the collected data. As a result of the self-evaluation, Davidson had already discovered this shortcoming and had begun to address it before they learned of the PULSE evaluation results. This type of self-reflection and response is exactly what the PULSE Fellows wanted to see happen.”