Submitted by Tim Hickey, Computer Science Department


Over the past two years I have been formally experimenting with the use of technology in my classroom and I’ve come to a number of realizations which I would like to share in this essay.  

The first, perhaps not surprising, realization is that technology itself offers no additional benefit to teaching unless it changes the pedagogy in some meaningful way.  At first I hypothesized that the mere use of technology might serve to enthuse students and keep them more engaged, but an experiment I carried out in Fall 2014 suggested this is not the case. Some of my colleagues had the opposite intuition, they felt that requiring students to use computers in class would not improve their learning. In fact, they felt it would be a disaster with students getting distracted – shopping on Ebay, browsing Facebook, etc. I decided to test these hypotheses about the effects of requiring or banning computers by designing and carrying out an experiment with my own class.  

Since this experiment involved human subjects (my students) and was designed to produce generalizable knowledge about the effects of technology on learning, I was required to get permission from the Institutional Review Board by filling out an IRB form describing the experiment, the risks, the related literature, the protocol, etc.  This was a bit daunting the first time, but wasn’t really that formidable a task.  If anyone else wants to try it, you can get support from the Center for Teaching and Learning which has copies of several successful IRB proposals and I would be happy to guide you through the process.  

I was teaching two large sections of Introduction to Programming in Java (CS11a) in Fall 2014 and I had decided to flip the classroom.  Class time would be involved mostly with activities that required every student to think about the concepts we were covering and to practice the skills they were developing.  There was some lecturing, but this was kept to a minimum.  We made extensive use of two web-based applications created by my graduate students:

  • TeachBack: an Audience Response System created by William Tarimo with several experimental features that we were testing, and
  • Spinoza: an online programming environment created by Fatima Abu Deeb which was designed to support in-class coding exercises.

Both of these supported an active learning paradigm to help students learn to program by solving problems individually or in small groups. There is a commercial product (Learning Catalytics) which is similar to TeachBack, but I was doing research on Educational Technology so we chose to use our home grown system.

I designed the course to have four units, each concluding with a summative exam. Each section had exactly the same readings, assignments, exams, and lecture plans but I decided to ban computers in unit 3 for section 1 to see the effect, and then, for symmetry and fairness, I banned computers in unit 4 for section 2.  Units 1 and 2 both used the technology extensively and students became used to working with these tools.  The IRB approved my experimental design.

Even when computers were banned, I used the same active learning pedagogy. Both sections were given programming problems to work on using a very effective and well-known collaborative learning activity called Think/Pair/Share (TPS). The instructions for this TPS activity were to have students work individually to try to write a program to solve a particular problem.  After a few minutes, I asked them to talk to their neighbors about their solutions and help each other debug. After a few more minutes I would select a few individuals and share their solutions with the class.  This activity is easy to add to any class and has been shown in numerous studies to help students gain a deeper understanding of the material than they would by just listening to a lecture and taking notes.  

In the computer-free section, students had to write their code on paper with pen or pencil, while in the computer-required section they interacted with a code editor and compiler developed by my graduate student and this experience was closer to the skills I was trying to teach as they would be using a real system.  When we shared a particular student’s solution with the class in the computer-free section, they had to tell me what to type on the screen so everyone could see it, rather than just letting me click on their name in the online version which popped up their code. In the computer-free section, I would ask them to think about questions and then would have them raise hands rather than having them all answer the question using the online TeachBack system. I was confident that students would learn more in the computer-required section because it held them more accountable for being engaged and gave them a more genuine experience of problem solving with a computer.

We measured the effect of requiring or banning computers by looking at grades on the unit tests and results of student surveys (all approved by the Institutional Review Board!).  It surprised us that there was no statistical difference between the two classes on any of the measures we examined, except that many students said they preferred using their computers in class.  The good news is that my colleagues fear that students would be distracted was unfounded. The active learning kept them too busy to spend much time on Ebay or Facebook!  We published a paper showing that requiring computers in class “does no harm,” at least for this class.

Interestingly, most students objected vociferously when I banned computers from their section. A  few, though, were relieved because they found they were easily distracted by their laptops.  Some students felt so strongly they surreptitiously switched sections, or they would go home and watch the screen-recorded lectures with their computers open! The lesson here is that in a large class, there is no one-size-fits-all pedagogy that will work well for all students.

More recently I’ve been exploring how I could have used the technology in that class to actually improve student learning by rapidly identifying students at risk of doing poorly.  The idea was to use the information collected from the students online interactions to help identify students that were struggling on the concepts and skills covered that day. Our initial studies suggest that this may be feasible and would help us implement a proactive form of advising where we can give detailed daily feedback to students about what they need to study and how they can get help to master those concepts or skills.

I’ve now adopted the “flipped classroom” model in all of my classes to some extent. The key idea behind flipping a classroom is to introduce students to the content before class, and give them practice working with the concepts and skills in class.  One tool that has revolutionized the way I approach pre-class assignments is the free and open source Nota Bene system from MIT ( This website allows you to upload pdfs for students to read and study.  The students login to the system and jointly annotate the pdf.  They can make observations, ask questions, answer questions, have discussions and all of this happens before class has started.  By scanning their comments before class I have a very good idea of which topics need more coverage and which they seem to have mastered!

I’ve learned that one does need to spend some time in class talking about how to ask a good question. One easy suggestion is that if a student has a question that they could answer using a quick google search, then they should make that search and upload their answer as a comment, rather than just adding a question. Requiring students to make five good comments or questions the night before the next class is enough to stimulate a flurry of intellectual activity while students are reading the text at home.  By perusing the students interactions with the text the next morning, I can decide which skills and concepts the students need to focus on in class that day and which they have mostly mastered. This is an example of a technique called “Just in time Teaching” where the classroom activity is determined at the last minute after looking at the students’ responses to the required reading.  For those who use copyrighted materials for class, you can use the Perusall system which will negotiate the copyrights for your class (for a fee) .

In the flipped classes I teach, I set explicit learning goals for each day and develop activities to engage them with those skills and concepts.  To assess their developing mastery, I use the TeachBack system to assess how well students are learning those concepts and skills.  There are other commercial systems such as LearningCatalytics that offer features similar to TeachBack (again, for a fee!). I also use Computer Science specific tools such as Spinoza and similar interactive tools that aren’t useful in other departments.

A third type of technology that I’ve been using recently in all my large classes is the Echo360 screen recording and live streaming system installed in half a dozen of our large classrooms. LTS will set it up for you at your request and it will automatically screen record your class and simultaneously live stream it to any Brandeis students who want to join remotely. In the Computer Graphics class I taught in Fall 2015, I introduced live streaming half way through the course and told students they could attend class remotely through live streaming as long as they also fully participated in class using the TeachBack system — answering questions I posed, asking questions of the TA using the forum during class, working on Think/Pair/Share problems with other students using TeachBack, etc.  We found 75% of the class tried live streaming at least once, and every day about a fifth of the class was live streaming (though it was a different group of students each day). Most students preferred being physically present in class but most thought livestreaming should be an option for all classes at Brandeis.  It allowed them to attend class if they were sick, or overslept, or were traveling for personal or sports reasons. The newest version of echo360 has some of the audience response system features of TeachBack built in and this should make it even easier for faculty to experiment with allowing students to live-stream their flipped classes.

The overall lesson I’ve learned in the past two years is that technology, if properly used, can support a more active and flexible teaching style that helps me reach more of my students, especially in large classes. I’ve also discovered that I can apply the scientific method to test hypotheses I might have about ways to improve my teaching.  It’s sobering when I discover that a hypothesis I had about my teaching is wrong, but it is also exciting to start using evidence-based methods to guide and improve my teaching.  I would encourage you to try some of these tools and to assess their effect (perhaps using an anonymous survey).  Who knows, you might discover something about the effectiveness of your pedaogical explorations that surprises you!