What Do Students Actually Think About Digital Pedagogy? (Pt. 3)

{This post continues my series of reflections on quantified student feedback regarding the use of digital pedagogy in the classroom. Be sure to check out Part 1 and Part 2, as well.}

My previous posts on digital pedagogy tended to emphasize the helpfulness of resources drawn from the digital humanities in lower-level undergraduate coursework. And while it might seem more obvious to deploy such resources in a large undergraduate survey, I’m confident that digital tools can still help us get the job done in senior-level courses, as well.

Examples of such resources that reward more sustained engagement would include:

(1) Podcasts: While podcasts might strike some as primarily entertainment media, there are a number of series out there that aim to bridge any supposed gap between the ivory tower and the public square. One of the best, in my view, is Peter Adamson’s History of Philosophy Without Any Gaps podcast, which breaks much of ancient and medieval intellectual history down into bite-size, twenty-minute chunks. Adamson manages to capture the best of both worlds, crafting episodes that are as listenable as they are edifying.

(2) Apps: One of the most labour-intensive yet rewarding applications in the field of the Digital Humanities is the creation of apps tailored to certain historical figures or texts. Villanova University has recently developed (and continues to develop) an app devoted to the Confessions of Augustine of Hippo, which happens to be a text near and dear to my own heart. Having used this app in a classroom setting already, I can attest to its efficacy. It really succeeds in its mission of making Augustine’s masterwork more accessible for students living about sixteen-hundred years after the Saint’s death.

In the end, it’s all about calibration. It’s all about fitting the amount and usage of digital resources to the level of the course, as well as to its theme and desired learning outcomes. If those three factors are kept in mind, digital pedagogy can work at almost any stage of undergraduate education.

To back up my claim that digital pedagogy can work at higher levels of undergraduate learning as well, please see the following data drawn from a fourth-year seminar on theories of time in antiquity. This course included 13 students, all of whom decided to participate in the feedback questionnaire. Again for reference purposes, here are the questions to which the charts supply quantified responses:

  1. Overall, I found completing the digital humanities assignments to be:
    1. Educational: 13/13 = 100%
    2. Confusing: 0/13 = 0%
    3. Pointless: 0/13 = 0%
    4. None of the Above: 0/13 = 0%
  2. Before the digital humanities assignments, my sense of the digital humanities was:
    1. Quite Good, Actually: 1/13 = 8%
    2. Limited & Incomplete: 10/13 = 77%
    3. Non-Existent: 2/13 = 15%
  3. These assignments improved my grasp of what “digital humanities” means:
    1. True: 12/13 = 92%
    2. False: 1/13 = 8%
  4. The difficulty-level of the digital humanities assignments was:
    1. Too Hard: 1/13 = 8%
    2. Too Easy: 1/13 = 8%
    3. Just Right: 11/13 = 84%
  5. If these kinds of assignments required deeper engagement with digital resources, that would:
    1. Be More Fun: 4/13 = 31%
    2. Be Too Burdensome: 4/13 = 31%
    3. Not Change Anything: 5/13 = 38%
  6. What is your attitude about the idea of the “digital humanities” generally?
    1. It Was a Refreshing Change: 8/13 = 62%
    2. It Sounded Easier But I Didn’t Really Get Much Out of It: 2/13 = 15%
    3. I Would Prefer to Just Write a One-Page Reading Response: 3/13 = 23%
  7. Which digital humanities assignment did you find most useful?
    1. Podcast Review: 8.5/13 = 65%
    2. App Reflection: 4.5/13 = 35%
    3. Neither Were Useful: 0/13 = 0%
  8. How did listening to a podcast episode compare to an in-class lecture on the same material?
    1. Listening to a Podcast is Preferable: 1/13 = 8%
    2. Listening to an In-Class Lecture is Preferable: 4.5/13 = 35%
    3. Both Are About Equally Effective: 7.5/13 = 57%
  9. What did access to the Confessions app change about your approach to the text itself?
    1. The App’s Resources Helped Me Understand Augustine Better: 10/13 = 77%
    2. App Was Cool But Didn’t Help Me Understand Augustine Better: 3/13 = 23%
    3. The App In No Way Aided My Understanding of Augustine: 0/13 = 0%

And because, once again, numbers never tell the whole story, here’s a selection of student comments regarding the use of digital resources in History 476:

If you could pass any advice on to Dr. Peter Adamson, proprietor of the History of Philosophy podcast, what would it be?

  • “If possible. I would love to get an outline or actual script of his podcast, since I usually have to listen to the podcast multiple times (which is a good thing). But I still hope there is something I can actually read.”
  • “Make his voice less soothing.”
  • “I enjoyed the emphasis on the history aspect, but there wasn’t quite as much info on the actual philosophy I was trying to focus on.”
  • “Make it more user-friendly to find subjects. Have a stronger search engine.”
  • “I found the podcast to be very enlightening. Having some more in-depth discussions and info would be great!”
  • “Keep doing what you’re doing! I loved it.”
  • “Keep doing what you’re doing. I found the podcast to be both interesting and useful.”
  • “Make them longer, since some sections seem too oriented toward surveying topics that merit more depth.”
  • “Subtitles, please! Sometimes I cannot hear the words clearly, especially during the interview episodes.”

If you could pass any advice on to the team at Villanova University that made the Confessions app, what would it be?

  • “It helped my understanding of the text, but the app needs a little work.”
  • “Break up the long book audio clip segments.”
  • “Emphasize external sources less.”
  • “Involve more reflections and commentaries (possibly longer ones).”
  • “The commentaries in the app gave me just a taste, but not enough.”
  • “I found the app to be very useful and didn’t have many problems with it.”
  • “As somebody not entirely familiar with sections of the Bible, abbreviated citations were initially confusing.”
  • “I learned to make a pretty good cake from it!”
  • “The app is well made and helped me quite a bit. I would suggest having discussion boards where app users can have discussions with the app creators.”
  • “Work out the small bugs to make the whole process smoother.”
  • “Great job! Thank you for the hard work. My only advice has to do with the audio format. A different interface might be easier. Other than that, it’s fantastic.”
  • “The app was useful, but also expensive. It needs work.”
  • “The difference between our translation [Hammond] and the app’s translation was too big. I understand why, but it made it hard to use in conjunction with our text.”
  • “There should be a better way of finding quotes.”
  • “Keep doing what you’re doing. I found the app to be very helpful.”
  • “Reformatting the footnotes would be appreciated, as would an index, specifically for the interpretive articles.”
  • “Make it easier to search for keywords while reading through the chapters.”Do you have any ideas about other digital resources that you’d like to use when completing assignments in a Humanities context?

Is there anything else you’d like to recommend to further the work that the digital humanities can accomplish in the classroom?

  • “Visual clips.”
  • “No, but I do think it’s a good idea to do assignments like this. It was very interesting to work with podcasts.”
  • “More podcasts would be helpful, not as an assignment but as an ‘extra’ for topics in class.”
  • “I personally use YouTube when I don’t understand a concept (in all of my courses). This may be useful to other students if credible videos are available.”
  • “I would like the addition of more audio books.”
  • “Creating a podcast or other digital project such as a wiki or website would be fun.”
  • “Video resources would be cool.”
  • “Perhaps some form of video content or supplementary lecture material from other speakers or authorities or YouTube or online lectures.”

Digital Pedagogy Post #1 Featured by Humanities Commons

In a rather pleasant turn of events, the powers that be at Humanities Commons have seen fit to feature my post on quantified student feedback to digital pedagogy!

You can find it on the Humanities Commons homepage, alongside some really excellent exemplars of what this platform is capable of, not least of which is Dr. Meredith Warren’s site (one of the best I’ve seen on here).

Hopefully we can all keep the momentum up as the summer carries on…

What Do Students Actually Think About Digital Pedagogy? (Pt. 2)

Following up on my earlier post on quantifying student feedback in the digital humanities, here are some rough-and-ready charts that aim to visualize some of the feedback data I’ve collected regarding the efficacy of digital pedagogy in the classroom. In general, it tends to demonstrate that the properly calibrated use of digital resources can see real results in a humanities setting.

This dataset communicates feedback findings drawn from a roughly 40-student session of History 101 (“Foundations of the Modern World up to 1500 CE:” a modest topic!).  22 of the 40 students responded to the questionnaire. For reference, here are the seven questions to which the charts supply quantified responses:

  1. Overall, I found completing the digital assignments to be:
    1. Educational: 22/22 = 100%
    2. Confusing: 0/0 = 0%
    3. Pointless: 0/0 = 0%
    4. None of the Above: 0/0 = 0%
  2. Before the digital assignments, my sense of the digital humanities was:
    1. Quite Good, Actually: 5/22 = 23%
    2. Limited & Incomplete: 8/22 = 36%
    3. Non-Existent: 9/22 = 41%
  3. These assignments improved my grasp of what “digital humanities” means:
    1. True: 22/22 = 100%
    2. False: 0/22 = 0%
  4. The difficulty-level of the digital humanities assignments was:
    1. Too Hard: 0/22 = 0%
    2. Too Easy: 8/22 = 36%
    3. Just Right: 14/22 = 64%
  5. If these kinds of assignments required deeper engagement with digital resources, that would:
    1. Be More Fun: 12/22 = 54%
    2. Be Too Burdensome: 5/22 = 23%
    3. Not Change Anything: 5/22 = 23%
  6. What is your attitude about the idea of “digital assignments” generally?
    1. It Was a Refreshing Change: 19.5/22 = 88%
    2. It Sounded Easier But I Didn’t Really Get Much Out of It: 5/22 = 7%
    3. I Would Prefer to Just Write a One-Page Reading Response: 1/22 = 5%
  7. Which digital assignment did you find most useful?
    1. Digital Mapping (DA #1): 4.5/22 = 20%
    2. Timeline Creation (DA #2): 5/22 = 71%
    3. Textual Analysis (DA #3): 2/22= 9%
    4. None Were Useful: 0/0 = 0%

Because numbers never tell the whole story, here’s a selection of student comments regarding the use of digital resources in History 101:

  • “No a million times over to short written assignments!”
  • “Timeline creation was most useful, followed by digital mapping and then textual analysis.”
  • “DH does make history a little more engaging.”
  • “It would be interesting to see other types of digital assignments, as well.”
  • “Some assignments were a bit confusing, but after a bit of time it was fine.”
  • “I think DH helps us to understand history much more easily.”
  • “I think it would be helpful to have a little more direction in where to find historical sources.”
  • “I enjoyed the assignments. However, it would be nice to have one written assignment. It would help with greater understanding.”
  • “I just really enjoyed the textual analysis! It was interesting. J”
  • “It was interesting in the sense that it enlightened me to a subsection of history I have not heard about before. However, the assignments were very easy, specifically the textual analysis. The map and timeline were good.”
  • “The digital assignments were educational to a certain extent.”
  • “I do like reading responses as a way to explore the material. However, the digital assignments were very beneficial and enjoyable. I use mapping and timelines a lot for studying, so this really helps me expand my toolbox.”
  • “The mapping and timeline assignments were helpful for understanding how the things we were studying related to each other. The textual analysis assignment seemed less helpful for that purpose, and it didn’t seem to relate.”
  • “The timeline and mapping assignments helped cement my knowledge of the material. However, the textual analysis really didn’t help with my understanding even though it was interesting.”
  • “I felt that if the assignments were a little more comprehensive I would’ve enjoyed them more. Since I thought they were super-easy, I sort of blew them off and then did the bare minimum at the end. I would’ve liked it to be harder. I loved them though!”
  • “I loved the mixture of assignments in this class. They did seem a bit easy or a lot of it was plug and play. But it was nice not having to write sixteen billion papers for this class.”
  • “Thanks for teaching this class! You made it fun and enjoyable. Keep up the good work.”
  • “I think you should drop Voyant [textual analysis tool], because it is not educational and doesn’t improve understanding of the course material.”
  • “I thought the digital assignments were interesting. I enjoyed them and I did learn some things. They helped me remember things about historical events and places.”
  • “The mapping and timeline assignments were useful. They put things into perspective. I didn’t get much out of the textual analysis assignment.”
  • “I think the mapping and timeline assignments were very helpful for the course. They helped me study and visualize. The textual analysis assignment, meanwhile, didn’t do much to increase knowledge of the course. It was interesting to do, just did not have the educational aspect that the mapping and timeline assignments did.”
  • “I would have liked to do maybe one or two more different ones for a little more engagement.”

For now, this remains mostly a snapshot of raw data, with a few bare-bones visualizations thrown in. But, when these numbers are combined with comparable datasets from other courses, I’m cautiously optimistic that more sophisticated and usable conclusions will result. Stay tuned for Part 3!

Notes on Macroanalysis

Strictly speaking, macroanalysis has to do with the analysis of textual data on the, well, macroscopic level. Turning to large-scale computing can allow us to make sense of datasets that would be beyond the ken of the average researcher making use of traditional methods.

For basic primers on this kind of work and what it can achieve, take a look at Matthew L. Jockers’ MacroanalysisDavid Berry’s Understanding Digital Humanities, or Witten, Frank, & Hall’s Data Mining: Practical Machine Learning Tools and Techniques. If you’re looking for online readings, I’d recommend Ted Underwood’s “Seven Ways Humanists Are Using Computers to Understand Text,” Scott Weingart’s “Topic Modeling for Humanists: a Guided Tour,” and an e-book by David Easley and Jon Kleinberg, Networks, Crowds, and Markets: Reasoning About a Highly Connected World.

The purpose of pursuing this kind of work is to find new ways of recognizing patterns and spotting anomalies across an impressively wide range of sources. Within the context of intellectual history, for example, macroanalysis might allow us to trace the transmission and even the development of key terms and concepts over increasingly long stretches of time.

When it comes to digitizing texts in the first place, we can choose from a number of effective tools. While the best digitization tends to be done by technologically well-endowed research libraries, there are also programs that anyone can use to get engaged in similar kinds of work. Optical Character Recognition (OCR) programs like Abbyy and AcrobatPro can prove to be especially powerful resources, allowing us to swiftly translate bare script into encoded text. Once that’s done, we can turn to tools like OpenRefine in order to clean our data up and make it usable for analysis–although the human touch is usually needed to ensure that the text-based dataset is as clean as it really needs to be.

For a figure like Augustine of Hippo, we could also draw upon existing digitized versions of his corpus in order to get a better picture of how major pieces of the Augustinian vocabulary–from confessio to distentio and beyond–fall into place over a period of decades. Vital resources here would include the digitization efforts undertaken by Belgium’s CETEDOC (Centre de Traitement Electronique des Documents) and the careful, prescient work of James J. O’Donnell (Confessiones online).

Once we have a clean corpus of data to draw from, we can manage our data so that it can easily translate into polished final products. In addition to the obvious (Excel), there are more advanced resources out there that can help streamline our data management, from RStudio (which puts the statistical language R to work for any number of projects) to Stanford’s capacious Palladio, which also includes an NEH-funded visualization program.

Getting even more macroanalytic, scholars can even turn to larger-scale database systems like MySQL, SQLite, MariaDB, and PostgreSQL. A great example of what databases are capable of in the context of the Humanities can be found in OCHRE, hosted by the Oriental Institute at the University of Chicago.

Again, while gathering up all this clean data is already a significant achievement, it is also a prelude to the fun part: actually analyzing the data and seeing what we can discover. There are a number of ways we can approach this stage of macroanalysis. Language processing, as exemplified by the Stanford Natural Language Processing Group and SAMTLA, can harness the power of technology to rapidly decode and parse texts from all over the globe. Markup tools and topic models can then help us give structure to our textual data and collaborate with others who may be analyzing the same datasets. MALLET provides us with a way of modelling word-clusters in order to draw conclusions about terminological trends and semantic trajectories, while (for something completely different) MARKUS stands as an excellent example of how a markup tool can deepen our study of an immense literary corpus (in this case, that of Chinese).

Lessons learned from methods in macroanalysis can also bear fruit in what we might call not-so-macroanalysis. (‘Microanalysis’ would seem to undersell things.)

For example, even taking a relatively constrained dataset–such as the digitized text of Book XI of Augustine’s Confessions–can allow us to track the use of key terms more precisely than any eyeball test. We don’t need language processing tools to tell us that Book XI deals with the themes of time and temporality, but those same tools can indeed help us determine how exactly Augustine chooses to deploy time-related words (tempus, tempora, distentio, etc.) over the course of the entire book.

Translating that processed data into easily interpretable visual media can then offer us a straightforward way to inform others about the terminological breakdown of Augustine’s writing. There are a range of advanced resources aimed at visualizing data in the most analytically responsible way possible: think here of D3, Gephi, and NodeXL. Yet even sites like Wordle or Jason Davies’ generator or (my fav) Voyant can give us the chance to create a visualization as simple as a wordcloud (see below), which can help people see the intensity with which Augustine pursues the topic of time over the course of Book XI, oftentimes better than would a paragraph of explanatory prose.

While constructing a wordcloud like this may seem like a rather straightforward enterprise, it can actually raise a number of thoughtful questions about ‘data-mining’ ancient texts. When it comes to grammar, for example, a language like Latin offers up some challenges that we seldom face when the object of our language processing is English. Trying to account for the cases of various nouns, for example, can add new layers of complexity to the process of turning raw text into usable data.

Digitally analyzing a text like Confessions XI can also raise questions having to do with rhetoric (as many scholars are already demonstrating in the field of stylometry). One of the most obvious steps we must take when cleaning up textual data is to get rid of all of the ‘stop-words:’ terms that recur so frequently in a language that they become almost statistically irrelevant. (In this case, think of non, et, de, and so on.)

But what about a word like te (you; accusative or ablative singular)? It would seem to be a statistically irrelevant term in many settings, and yet in Confessions XI that may not be the case. Throughout this work of his, Augustine frequently refers directly to God in the second person: tu, tibi, te… And so what is the burgeoning young digital humanist to do?

Regardless of what we decide to do with such data-points, the fact remains that the very exercise of trying to translate Augustine into analyzable digital data can give us reason to reflect on the grammatical, rhetorical, and perhaps even conceptual content of the text proper.

macro corpora.png

HathiTrust Digital Library

ECCO: Eighteenth-Century Collections Online

Project Gutenberg

Google Books

Google Ngram Viewer


What Do Students Actually Think About Digital Pedagogy? (Pt. 1)

One of the first questions that comes to mind when one hears the troublesome phrase “Digital Humanities” is an exceedingly basic one: “What does Digital Humanities *mean,* anyway?”

Sometimes, this question is asked in a spirit of honest humility and with a desire to learn more: “What does Digital Humanities mean? What kinds of tools and objects and practices count as DH? What can I learn from DH methods?”

At other times, the question seems to be asked with a bit more of an edge to it: “What even *is* DH, anyway? Does the term mean anything at all? Are we simply dealing with empty buzzwords here? What ever happened to good ol’-fashioned book-learnin’?” (And so on…)

In my experience, most students tend to pose the question in the former sort of way, while some of my fellow instructors prefer to opt for the latter. This is not a hard and fast rule, of course. There are a few students who could be described as hardline anti-DH-ers, while we all have a number of colleagues who either hold a genuine interest in DH or are actively engaged in doing DH work. But, in my admittedly anecdotal experience, it does seem like students–especially younger undergraduates–can be more open to exploring the potential of what DH could mean for them.

If we want to provide straightforward, concrete examples of what great DH work can look like, we could simply point toward huge collaborative efforts like Orbis and Pleiades, or toward specific scholars who have established themselves as preeminent pioneers of DH in their respective fields. In the world of classics and late antiquity, for example, I tend to point students in the direction of scholars like Dr. Sarah Bond or Dr. Jennifer Barry, both of whom run DH workshops for the North American Patristics Society.

When it comes to my meagre corner of the academic world, however, I usually focus in more closely on Digital Pedagogy. Here the operative question becomes: how can we best go about using digital resources to augment students’ own attempts to engage with the historical cultures they’re studying? (In my case, once again, the most relevant historical epoch is antiquity, although I also teach courses in medieval and early modern history.)

If that’s our operative question, the next step should be to ask students themselves what they think about the digital tools we use in class. It’s one thing to experiment with digital pedagogies in a course; it’s another to gather instructor-side data about new assignment-models; but it’s yet another thing to actually go out there and listen to what students have to say about the whole experiment in all of its unwieldy facets.

To that end, I’ve spent the first year of my position here at MacEwan compiling student-side feedback about a wide range of digitally tinged assignments. In a first-year world-history survey, we talked timelines and textual analysis; in a second-year medieval Europe course, we made multiple mock-up maps of the Mediterranean world in the Middle Ages; and in a fourth-year seminar, we worked to incorporate both podcasts and apps into our discussion of some challenging material from antiquity. In every case, students voluntarily submitted feedback info that I was then able to preserve as data and, in many cases, visualize for popular consumption.

Part of my goal with this site is to get some of my data on digital pedagogy out there, so that from its rudimentary roots something more substantive might grow. There are any number of other scholars and instructors out there who have delved deeper into these issues than have I, but I’m hopeful that offering up my feedback as open data will be a not-entirely-pointless exercise.

To that end: in future posts, I plan to share some of the data I’ve collected, as well as the contextual information needed to make sense of the raw numbers. As I continue to experiment with digital pedagogy in the months and years to come, I’ll aim to tweak my approach both to assignment-design and data-collection, so that a fuller picture of the capability of these new resources will begin to come into view.