Impact on Completion

We expect that this active-learning, student-centered, reform model will increase the pass rate in the course. More importantly, we expect that this reform will have significant impact on the success of women, underrepresented, and first-generation students in organic chemistry courses. As a result, we expected that more students would continue as STEM majors and graduate with STEM degrees in a timely fashion.

Potential lessons to be learned

The implementation and assessment of the flipped-classroom-PLTL reform model in organic chemistry will give faculty an understanding of how this model works in large, diverse undergraduate classrooms. Additionally, it will give us insight into how to effectively support and address the needs of diverse student groups in STEM courses. This model can be made available and adapted to other institutions.

Concept Description

Area of Need. Across the United States, Organic Chemistry is one of the most challenging science courses. Many STEM majors require this course to pursue their undergraduate degree and to continue in STEM pathways beyond the undergraduate degree. For many students this course has become a barrier to successfully pursuing STEM pathways (Barr, Gonzalez, & Wanat, 2008; Lovecchio & Dundes, 2002). Nationally, the failure and withdrawal rates for Organic Chemistry have been known to be as much as 50% (Horowitz, Rabin, & Brodale, 2013). At Georgia State University (GSU) the failure/withdrawal rate (C- or lower and withdrawals) average 30% for the first- and second-semester Organic Chemistry courses (Table 1 in Appendix). High failure rates lead to more students repeating the course, thereby delaying graduation and increasing their chances of leaving the STEM pathway. Approximately 1600 students enroll in Organic Chemistry courses each year at GSU. As a result, some 480 students each year are either repeating the course or considering a non-STEM major.

 Our baseline data also indicate that women and underrepresented students earn disproportionally lower grades in these courses (Figure 1 in Appendix).  For instance, the average grade for female students in the first-semester Organic Chemistry course is 2.70, while that of male students is 2.89. Also, the average grade in the course for Black or African American students is 2.51, while the average of other student-groups range from 2.80 to 3.03. Another indicator of student success in Organic Chemistry is the American Chemical Society (ACS) National Examinations. The ACS National Examination is a standardized exam that assesses students’ conceptual understanding of course material. This exam can be used to compare our students to a national sample. The average score on the ACS Exam for Organic Chemistry courses at GSU is in the 40^th percentile. Our goal is to raise this average to at least the 50^th percentile.

Chemistry faculty at GSU have been investigating the use of high impact, evidence-based practices with the goals of increasing student success and retention in our high enrollment undergraduate courses. To this end, Drs. Suazette Mooring and Joan Mutanyatta-Comar have recently piloted Peer-led Team Learning (PLTL) to improve student performance in Organic Chemistry. PLTL is a nationally recognized practice that provides an environment in which students can engage in problem solving and discussion among themselves (D. K. J. Gosser, 2012; Varma-Nelson, Cracolice, & Gosser, 2004). PLTL utilizes undergraduate students who have successfully completed the course to facilitate group problem-solving sessions. The problems used in the PLTL workshops are designed to be sufficiently challenging to encourage group discussion and build conceptual understanding of topics. There are strong indications that PLTL helps elevate exam scores and final grades and increase student retention in chemistry courses (Akinyele, 2010; Gafney & Varma-Nelson, 2007; D. K. Gosser & Roth, 1998; Hockings, DeAngelis, & Frey, 2008; Horwitz et al., 2009; Lewis, 2011; Mitchell, Ippolito, & Lewis, 2012; Wamser, 2006). For example, in a study involving an Organic Chemistry course, students in PLTL had an 85% pass rate compared to 69% for students who did not attend the optional PLTL (Wamser, 2006). Also, students participating in the PLTL workshops outperformed those who did not take workshops on course GPA (2.90 versus 2.51), and on the standardized American Chemical Society (ACS) Examination scores (77th versus 69th percentile).

Addressing the Need. The preliminary results of our PLTL implementation were encouraging. Students in the Spring 2014 (PLTL section) showed a statistically significant difference in ACS Scores compared with the Spring 2013 (non-PLTL section) with the same instructor. That is, the 2013 section had a raw score of 31.7 (29^th percentile), while the section has a raw score of 34.2 (37th percentile). In addition, the percentage of students earning a grade of C or better for students in the reform course was significantly higher (72.3%) than in previous year taught by the same instructor (66%). This pass rate was also higher than the departmental average for this course of 63%.

Although our initial results showed some improvements in student outcomes, further improvements are still needed. Our continuing plans are to use the lessons learned from our pilot implementation of PLTL to further improve these outcomes. We propose to combine PLTL with a flipped classroom model to improve the outcomes for all students, particularly for women, underrepresented minorities, and first-generation students.  We are combining these two pedagogies because they both support elements of active-learning in a social context. Active learning refers to pedagogies that are student-centered and includes activities that engage students in the learning process. These strategies have been shown to increase student engagement, motivation, and learning (Freeman et al., 2014; Hake, 1998; Knight & Wood, 2005). A recent meta-analysis that included 225 studies investigated the effect of active learning strategies and traditional lecture on student learning and performance (Freeman et al., 2014). The study revealed that, on average, student performance on examination and concept inventories were higher (0.47 standard deviations) and failure rates were lower for students in active-learning environments compared to traditional lecture (Freeman et al., 2014). Moreover, other studies have shown that although all students benefit from social, active-learning environments, there is a larger improvement in the grades of women, underrepresented students and first-generation students (Eddy & Hogan, 2014; Preszler, 2009).

The flipped classroom is based on inverting instruction, such that, traditional classroom activities become homework and what is traditionally homework is done in the classroom (Lage, Platt, & Treglia, 2000). According to Bergman, one of the pioneers of the flipping method, the flipped classroom is designed to allow students to take responsibility for their learning, encourage student-centered learning, and to continually engage students in the learning process (Bergmann, 2012). For a flipped course, the online materials are typically video lectures or readings that students do outside of class, while active problem solving activities are implemented in the classroom setting. We will use the video lectures outside of class and activities such as group problem-solving and peer-instruction (polling and discussion with fellow students) in class.

Potential Impact on Student Success and College Completion in STEM. We hypothesize that the use of this reform model that combines PLTL with a flipped classroom will lead to further improvement in the performance of all students in Organic Chemistry and especially so for women, underrepresented minorities and first-generation students. We also hypothesize that students will also have a more positive attitude towards Organic Chemistry. As a result, more of these students will be encouraged to stay as STEM majors and graduate in a timely fashion.

Lesson Learned and Impact. The proposed study involves undergraduate students from a large, research institution in the University System of Georgia (USG) that has a significant number of underrepresented minorities, first-generation students and women. As a result, this study provides an opportunity to disaggregate our findings by these variables and will therefore provide additional information about how this reform will affect these populations. We will also add to the best practices for integrating active-learning efforts for large undergraduate STEM courses. We will disseminate this work among other STEM faculty at our STEM lunch and learns or workshops, thereby providing opportunities for faculty development. We will also present our results and lesson learned at a State or regional STEM conference. We believe that the components of this reform and the results that the assessment will be transferable to other USG institutions.

Project Plan

This goal of this proposal is to implement and assess the impact of flipped classroom-PLTL reform course on student performance in Organic Chemistry courses.  The following three objectives will be addressed:

Objective 1: Implement PLTL-flipped reform model in Organic Chemistry courses at Georgia State University. The implementation of this objective will lead to the development of new curriculum materials and pedagogical model for Organic Chemistry courses at GSU. We will implement the reform model in two sections of Organic Chemistry (~200 students each) beginning Fall 2015. The two sections will either be the same Organic Chemistry course or a combination of the first and second-semester course. In the reform courses, students will meet with their course instructors twice per week for 70 minutes. Students will view the video lecture material before coming to class. After watching the lecture videos, students will be asked to answer quiz questions (online or in-class) relating to the videos. This step is to ensure that students are watching the videos and will be prepared for the face-to-face discussion (Brunsell & Horejsi, 2013). The post-video quizzes will also give instructors feedback on what concepts may need further discussion (Figure 1). Students will then be involved in problem-solving activities that include peer instruction and group discussion. Students will also meet once per week with their Peer Assistants for PLTL to solidify the concepts discuss in class (Figure 1). Peer Assistants will sometimes be utilized in the classroom to assist students with facilitating group learning activities.

Peer Assistant recruitment, training and compensation. We will select undergraduate students who possess leadership potential, have good communication skills, and have received at grade of B or better in Organic Chemistry. Each semester, we will recruit up to 24 Peer Assistants for Organic Chemistry. Dr. Mooring has been responsible for Peer Assistant training at GSU since Spring 2013 and has designed a course for this training. The course covers topics such as group facilitation, problem-solving strategies and questioning techniques. In addition, all Peer Assistants meet once per week with course instructors to review the weekly PLTL workshop problems. Each Peer Assistant will be paid $410 per semester.

Production of Video Lectures. Drs. Comar and Mooring have already had success producing videos for their current Organic Chemistry courses and will be making more videos this summer (2015) to completely implement a flipped classroom model. An iPad app called Explain Everything^TM is used to create the videos. The app records over uploaded PowerPoint presentations or PDF documents. In addition, free-hand drawing and writing could be done using a stylus. The video can then be directly uploaded from the app as a YouTube video. Students have access to the YouTube video as a link embedded into our course management system, BrightSpace. Dr. Mooring received a technology grant from the GSU Center for Instructional Innovation in 2013 to purchase 20 iPad tablets to support chemistry learning. Therefore, the iPads will be readily available to adequately support this work.

Objective 2: Investigate the impact of the reform model on student success in Organic Chemistry courses. This objective will generate data to assess the efficacy of the reform course on student success. To assess the impact of the reform course on students’ performance and persistence in the course, we will use statistical methods to answer the following questions: 1) What is the impact of the reform model on the course pass rate (ABC%)?

2) What is the impact of the PLTL-flipped reform model on students’ American Chemical Society (ACS) Exam Scores? and 3) What is the impact of the reform course on the exam scores, pass rate and ACS Exam scores students based on gender, race/ethnicity and first-generation status? We will compare the results of these variables on the reform course compared to courses before the reform.

Objective 3: Assess the impact of the reform model on student attitude and perceptions towards Organic Chemistry. We will use students’ responses to a pre- and post administration of the 8-item Attitude toward the Subject of Chemistry Inventory version 2 (ASCIv2) (Xu & Lewis, 2011) to gauge their attitude towards Organic Chemistry. An additional survey will be used to evaluate students’ engagement and perception of various aspects of the course components. This survey will include Likert scaled and open-ended questions regarding PLTL, their Peer Assistants, active learning materials, and the video materials they watch before class.

Sustainability. The video lectures, active-learning problems, and PLTL worksheets will already be in place beyond the grant period and can continue to be used by other instructors. Additionally, if the reform effort shows good results, the chemistry department chair has pledged a yearly commitment of $20,000 to continue to support Peer Assistant stipends and sustain the reform effort for two courses per semester.  The cost of implementing PLTL within the reform is $25/student or around $2.50 per PLTL session.

Timeline. Drs. Mooring and Mutanyatta-Comar will be responsible for all project activities in the following timeline, including teaching the organic chemistry courses:

Evaluation Plan

The proposed evaluation plan is designed to provide formative and summative feedback from the logic model (shown below) and from the objectives of this proposal.  The following outlines a system for monitoring the project activities, outputs and outcomes:

Objective 1: Implement PLTL-flipped reform model in Organic Chemistry courses at Georgia State University. The activities related to this objective include: 1) design of active-learning activities, 2) production of video lectures, 3) training of Peer Assistants to implement PLTL sessions, and 4) implementation of flipped classroom-PLTL model. We will use a project checklist and spreadsheet to monitor the outputs related to this objective that will include: 1) the number and nature of active learning curriculum materials developed, 2) the number of video lectures used throughout each semester, 3) the number of Peer Assistants trained, 4) the number of views of the video lectures, and 5) the number of PLTL sessions facilitated each semester.

Objective 2: Assess the impact of the reform model on student success. The outcomes related to objective 2 include: 1) increased percentage of students receiving a passing grade in the course, 2) improved student performance on the ACS National Examination, and 3) improved course performance by women, underrepresented minorities and first-generation students.

We will collect the following data from the course instructor and the university registrar:  basic demographics (gender, race/ethnicity, first-generation status), semester-exam scores, ACS examination scores, final course grade, GPA upon entering the course, and grade in general chemistry courses. We already have IRB authorization to collect this data and will further modify the IRB to collect additional data. We will use IBM SPSS Statistical Software to do our analysis and answer the following questions:

1. Is there a difference in ACS exam scores between reform and non-reform courses? It is well established that factors other than the pedagogical reform will impact student outcomes in the course (Cracolice & Deming, 2005). Therefore, we will use an ANCOVA to answer this research question. Unlike the independent t-test, the ANCOVA can isolate the effect of the pedagogical reform that is independent of the effects that can be attributed to covariates. Since current GPA and grade in general chemistry have been identified as predictors of student success in the Organic Chemistry course, it is reasonable to account for and quantify its effect (if any) on ACS examination scores, the dependent variable.

2. What is the impact of the pedagogical reform on the percentage of students receiving passing grades? We will use a test of proportions to compare the pass rate of students in the reform course and students in the traditional course before the implementation of the reform course.

3. What is the impact of the pedagogical reform on the exam scores, final exam grade and ACS Exam score of women, underrepresented minorities and first-generation students? We will disaggregate our findings by gender, race/ethnicity and first-generation/continuing generation for questions 1 and 2 above.

Objective 3: Assess student attitude and engagement in the reform course. From the logic model, the outcomes related to this objective are: 1) increased student engagement in Organic Chemistry course and with course material and 2) more positive student attitude towards Organic Chemistry course. We will use students’ responses to the pre- and post 8-item ASCIv2 to assess any changes in their attitude towards Organic Chemistry. We will use an additional survey to assess their perception of their Peer Assistants, active learning tasks and the video lectures. The survey will utilize a 5-point Likert-scale design (strongly disagree to strongly agree) and open-ended questions. All data will be analyzed using SPSS. The open-ended questions will also be analyzed for general themes.

Budget

Logic Model

Context:  There is a need to increase student performance and retention in Organic Chemistry courses at Georgia State University.  This is a key course that can affect students’ retention and graduation in STEM

Appendix: Bibliography

Akinyele, A. F. (2010). Peer-Led Team Learning and Improved Performance in an Allied Health Chemistry Course. The Chemical Educator, 15, 353–360.

Barr, D. A., Gonzalez, M. E., & Wanat, S. F. (2008). The leaky pipeline: factors associated with early decline in interest in premedical studies among underrepresented minority undergraduate students. Academic medicine : Journal of the Association of American Medical Colleges, 83(5), 503-511. doi: 10.1097/ACM.0b013e31816bda16

Bergmann, J. a. S. A. (2012). How the flipped classroom is radically transforming learning. The Daily Riff.

Brunsell, E., & Horejsi, M. (2013). Science 2.0. Science Teacher, 80(2), 8.

Cracolice, M. S., & Deming, J. C. (2005). Measuring the effect of Peer-led Team Learning. Progressions: Peer-Led Team Learning. The Workshop Project Newsletter, 6(2), 3-4. https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxxd...

Eddy, S. L., & Hogan, K. A. (2014). Getting under the hood: how and for whom does increasing course structure work? CBE-Life Sciences Education, 13(3), 453-468.

Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410-8415.

Gafney, L., & Varma-Nelson, P. (2007). Evaluating peer-led team learning: A study of long-term effects on former workshop peer leaders. Journal of Chemical Education, 84(3), 535.

Gosser, D. K., & Roth, V. (1998). The Workshop Chemistry Project: Peer-Led Team-Learning. Journal of Chemical Education, 75(2), 185. doi: 10.1021/ed075p185

Gosser, D. K. J. (2012). Promising and Practical Strategy:  Peer-led Team Learning Promising and Practical Strategies to Increase Postsecondary Success Retrieved September 25, 2014

Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66(1), 64-74.

Hockings, S. C., DeAngelis, K. J., & Frey, R. F. (2008). Peer-led team learning in general chemistry: implementation and evaluation. Journal of Chemical Education, 85(7), 990.

Horowitz, G., Rabin, L. A., & Brodale, D. L. (2013). Improving Student Performance in Organic Chemistry: Help Seeking Behaviors and Prior Chemistry Aptitude. Journal of the Scholarship of Teaching and Learning, 13(3), 120-133.

Horwitz, S., Rodger, S. H., Biggers, M., Binkley, D., Frantz, C. K., Gundermann, D., . . . Ryder, B. (2009). Using peer-led team learning to increase participation and success of under-represented groups in introductory computer science. Paper presented at the ACM SIGCSE Bulletin.

Knight, J. K., & Wood, W. B. (2005). Teaching more by lecturing less. Cell biology education, 4(4), 298-310.

Lage, M. J., Platt, G. J., & Treglia, M. (2000). Inverting the Classroom: A Gateway to Creating an Inclusive Learning Environment. The Journal of Economic Education, 31(1), 30-43. doi: 10.2307/1183338

Lewis, S. E. (2011). Retention and reform: an evaluation of peer-led team learning. Journal of Chemical Education, 88(6), 703-707.

Lovecchio, K., & Dundes, L. (2002). Premed survival: understanding the culling process in premedical undergraduate education. Acad Med, 77(7), 719-724.

Mitchell, Y. D., Ippolito, J., & Lewis, S. E. (2012). Evaluating Peer-Led Team Learning across the two semester General Chemistry sequence. Chemistry Education Research and Practice, 13(3), 378-383.

Preszler, R. W. (2009). Replacing lecture with peer-led workshops improves student learning. CBE-Life Sciences Education, 8(3), 182-192.

Varma-Nelson, P., Cracolice, M. S., & Gosser, D. (2004). Peer-Led Team Learning: A Student–Faculty Partnership for Transforming the Learning Environment. Invention and Impact: Building Excellence in Undergraduate Science, Technology, Engineering, and Mathematics (STEM) Education, 16-18.

Wamser, C. C. (2006). Peer-Led Team Learning in Organic Chemistry: Effects on Student Performance, Success, and Persistence in the Course. Journal of Chemical Education, 83(10), 1562. doi: 10.1021/ed083p1562

Xu, X., & Lewis, J. E. (2011). Refinement of a chemistry attitude measure for college students. Journal of Chemical Education, 88(5), 561-568. 

Appendix: Baseline Data

Figure 1.  Average Grade in Organic Chemistry (spring 2012 through Spring 2015)

Table 1. Failure/Withdrawal Rates (below C- and Withdrawal) for Organic Chemistry (Spring 2012 through Spring 2015)

Table 2. ACS Exam National Percentiles Organic Chemistry (Spring 2013 through Spring 2015)

Table 3. Comparison of pass rate (ABC %) and failure/withdrawal rates for PLTL versus non-PLTL sections of CHEM 3410  (Spring 2015 through Spring 2013).  Two instructors are compared.