Tag Archives: presenting

The Cleverness of Intent Over Content

I tweeted the other day about some quizzes I had taken that yielded results that were…unexpected. Resulting conversations ran the gamut from the relative merits of requiring leaders of technical teams to be technical folk to simply commiserating about the “impostor syndrome”-triggering nature of failing even a badly done test—and I fully intend on writing about some of these—but that’s not this post. This post is about a different concept that the assessments made me think about; the incredible difficulty of making good testing materials and one strategy for making better ones.

The fact that teaching people is a tricky and difficult thing should come as no surprise, but what I found only when I had been doing it for a couple of years was that the hard part isn’t the actual teaching itself—that part is actually fairly simple if you know the topic really well and can communicate with any degree of clarity. The hard part is honoring your initial intent with all of the materials, but especially exams and the like.

I started my undergrad—well, the final time I started my undergrad—a working software developer fully a decade into my career. By the time I sat down to take one of my first exams in a class purported to be an introduction to programming logic, I had been writing programs for double that time. That test bothered me to such a degree that it haunted my thoughts for years to come as I continued my career both as an instructor and a developer. Why was it such a terrible exam? How could a fantastic teacher create an such a bad evaluation tool?

More than half of the test was comprised of questions best described by the following template:

What is the definition of {word}?

  1. Obviously wrong answer
  2. Answer that looks right save for one very small error
  3. Answer that could be correct, but is clearly wrong
  4. Correct answer

Of the remaining questions, most only deviated from this formula by not specifically requesting a definition. My favorite example from this particular type of question (presented to you by virtue of the fact that I’m a digital hoarder with decades of bullshit on my hard drive):

An array is:

  1. A collection of values stored in one variable referenced by index 1 to n
  2. A collection of values stored in one variable reference by index 0 to n-1
  3. A single beam of light
  4. A list of similar but unrelated items

There is a host of problems with this question, but for someone who spent some time programming in Pascal and Fortran both in school and professionally in the years prior to this exam, this question was really galling.1

The crux of the problem is, there wasn’t even any point in the latter half of the text of the “correct” answer. It’s clearly very clumsily tacked on as a counter to the “trick” answer. Getting this question “wrong” by answering (1) doesn’t indicate that you don’t understand the material—at best it indicates that you were unclear on a nuance. More importantly, answering this question “correctly” doesn’t even indicate a fundamental understanding of what an array is—as evidenced by the lackluster results of the first practical exercises when we used arrays.

The instructor took their eyes off the prize and forgot what their intent was in giving the exam in the first place. So many tests make this exact mistake. The purpose of this exam was stated in print at the top of the first page:

The purpose of this exam is to demonstrate a basic understanding of how [to] use the foundational components of a computer program…

In most applications, simply knowing the definition of a word—especially to a pedantic degree—does not afford one any more ability to be proficient in a thing that not knowing the definition.2 Wouldn’t the following question have better suited the purpose?

For the following questions, use an array defined in C as follows:

char letters[5] = {“h”, “e”, “l”, “l”, “o”};

What index would you use in C to request the letter ‘e’?

Rather than the definition, if you correctly answer this question I now know if you know how to USE this foundational component of a computer program. It’s still an imperfect question, but already it is more aligned with my exam’s intent. But wait! What if you need to ask a vocabulary question in order to satisfy the intent of the exam? This isn’t uncommon, but the vocabulary question should be phrased in a way as to satisfy understanding over recitation.

In the bad array question above, the language used in the potential answers was directly from the teaching material. This is often done for a very rational, well intended reason: to AVOID tricking students by changes in wording. The problem is that it doesn’t really prove that the student understands the concept which—again—was the stated intent of the exam. It provides evidence that they can recite the verbiage that you provided already, not that they know what it means. Validating understanding of vocabulary in a way that is quickly and easily gradable (read: non-short answer) is tricky, but there are a number of strategies that have been shown to have success.

Most commonly, multiple choice (as above) but with the actual correct answer being a derivation of the textbook answer and the other answers being derivations of other vocabulary items in the material being taught. This can be done in the single format (again, as above), or it can be done in a many-to-many format (as in “draw a line between the word and its definition”). Asking the respondent to select synonyms and/or antonyms can also be valuable in some cases.

Strategies notwithstanding (and if you’d like more in depth information on strategies like these, I highly recommend How Learning Works by Abrose, Bridges, et al), all of this is secondary to resolutely ensuring that you choose mechanisms for evaluation that adhere to the reason that you chose to evaluate the student to begin with.

It is challenging. Even with this knowledge, and even after taking numerous courses on pedagogy, I still struggled with making my testing materials valuable to students. Some time after I had taken this fraught exam, I found myself giving exams that were in no way better than those I am describing here. During one frustrating exam creation session, I got up, walked to the dry erase board in my office, and wrote the following3 in huge letters directly in my line of sight:

I want to know that students that pass this exam will know exactly how to use the things I test them on here in practical ways, and that students that get questions wrong will know exactly what they need to study to be able to use those things.

I want no students that know the answer to a question to get it wrong.

I don’t want my exam to be clever, I want my students to be clever.

Simply, I wrote my objective statement where I could see it. I made my intent…well…intentional, and I did so in a manner that increased the likelihood that it would impact my actual behavior. The positive direction that this pushed my teaching and my students was palpable. The test that I was writing at the time immediately felt more “right” to me than any I had created before. Each subsequent quiz and exam moved closer and closer to the ideal I had in mind because each time I looked at my material I found new ways that it wasn’t honoring my intention. As my skill as an instructor improved, so, too, did my ability to find ways to meet those objectives.

The results weren’t simply gut feel, though. Exam scores improved, but more importantly so did the results of all project work and labs. My sample was small, but my pass rate went up by a small-but-measurable percentage. Better still, the students that came out of my classes started being lauded as particularly “well prepared” for higher-level courses to follow. In short, I hadn’t made the tests easier—I had made them more effective.

Years after this epiphany (and I use that term very loosely, here), I had the pleasure of getting positive feedback from a student at the end of my course. She was switching careers from a decidedly non-technical field to that of a developer, and among the things she said one that stood out to me was the observation that her test anxiety and impostor syndrome did not manifest so intensely on my exams; that, as she opined, the exams “didn’t try to make her feel stupid.” Her software career has surpassed my own at this point and I delight in the idea that this change in course might have played some small part in that.

In my experience, there’s no magic bullet that creates great exams. It is only through conscious, mindful attention to the goals of the exercise that you can hope to end up with the desired result. Conscious, mindful attention…and a ridiculous amount of practice. As an aside to the armchair quarterbacks out there mean-spiritedly snarking about “shitty tests” I extend this invitation: create an exam about something you know very well and see how difficult it is to make something of which you can be proud. I think you’ll be surprised.

1 The “correct” answer was 2, but Pascal, Fortran, and numerous other languages start their indices at 1 rather than 0. Further, most modern languages (even at that time) allow for non-numeric indices, making the question even more grossly inaccurate.

2 There are exceptions, obviously—knowing what “flammable” or “caustic” means could be pretty important in a lab setting, for example.

3 In reality, it was probably something very similar, it subtly mutated over time, but this is pretty close to what I wrote.

Penguicon 2018 Panels

I began planning for this year’s Penguicon with the best of intentions. I put together a handful of panels that I wanted to do and submitted them on time, like a proper planner. Then I agreed to be a panelist on a couple that seemed to be a good fit. Of course, I forgot about the recurring board panel. So now my current load for Penguicon is 6 sessions. I’ve done worse, but, I’ve certainly done better :) I don’t know exactly when they’ll be at the moment, but this is WHAT they’ll be: Continue reading Penguicon 2018 Panels

White, Male Speaker Seeks Microphone

Another white, male conference speaker has sounded off about the “quotas” that are “stealing” “his” speaking gigs and “giving” them to women or people of color despite the fact that they are “inferior.”

In case my liberal use of quotation marks above didn’t sufficiently convey my opinion on the matter, this strikes me as absolute nonsense!

Continue reading White, Male Speaker Seeks Microphone

Manager vs Leader Talk at Penguicon 2016

Dawn Kuczwara (@DigitalDawn) and I talked a bit about the difference between managers and leaders at Penguicon this weekend. Penguicon always pulls a different sort of talk out of us, and this is no exception. The informality of the panel-style discussion lent itself to several things…

Continue reading Manager vs Leader Talk at Penguicon 2016

Moving to Agile: Training

I haven’t really had the mental energy to write much about our transition to agile for the last month or two because I have been spending so much of that time period putting together and executing trainings. Even with as much enthusiasm as I have for this, it has been a draining several weeks.

The human urge to generate complexity when something seems too simple makes teaching simple things a weird chore. When I walk someone through the thought process behind answering a specific Scrum question, it’s often perceived as too simple—I get wary looks from the audience as if I’m trying to trick them. There is no trick, it’s really that simple. Continue reading Moving to Agile: Training

Training Days

This and next week I will be performing a series of trainings for groups within our organization to describe how we’re using Scrum (initially, at least). This is easily my favorite part of my job.

Not the “performing training” part; while I enjoy that considerably, it is also utterly exhausting. No, my favorite part of the job is helping others understand things. Anybody can tell someone the answer. Some of those people can even tell someone the right answer. It is immeasurably more satisfying to walk someone through the though process by which the right answer was derived so that in the future you can watch them solve the next problem correctly. Continue reading Training Days

Strong Teams as Healthy Communities

This is a placeholder that I will almost certainly fail to update later; but I hope that this will one day be a blog post version of the talk I gave at Self.conference 2015.

Notable: I used slides for the first time in a while. It wasn’t the worst. It wasn’t my favorite way, but it wasn’t the worst :)

The slides in PDF format.

Dearly Beloved…

This weekend I served as last-minute officiant for the wedding of a friend. On Wednesday, I was asked if I could fill in for a wedding to take place on Saturday. I had availability, I wanted to help out, I said yes!

In all, it was an awesome experience. I am really glad that I could help out…and Krista and Trevor are clearly fantastic together. I was a bit nervous that the short notice was going to be a problem, but cramming a lecture into my head on short notice has become par for the course for me; by Friday, I was golden.

By Saturday morning, I was less golden. For nearly two weeks, I have been having what felt like a constant, low-grade asthma attack—like someone was applying uneven, firm pressure on my chest. Uneven, firm, completely-unaffected-by-Albuterol pressure. By Friday night and early Saturday morning, my “asthma” had become a phlegmy mess replete with a slightly wet cough, extreme shortness of breath, and a deeper, more rumbly, more difficult to hear speaking voice. Pneumonia was sounding more and more like a possibility.

Because—and this is the important bit—what every public speaker needs immediately before speaking is to lose the ability to talk for any length of time without becoming dizzy from lack of oxygen while at the same time losing any ability to make audible sounds with his or her voice.

I spent the last hours before the ceremony trying to change my natural pattern of speech to allow for breathing breaks, lest I lose consciousness mid-word. Ger even admonished me, at one point, not to pass out during Krista’s wedding. That, as Carl[1] is fond of saying, is my least favorite thing to do.

Ultimately, I made it through without passing out. I was quiet (Ger says that at the back of the room, she could hear that I was speaking, but could not hear the precise words mostly, which is precisely the effect that having a sick-induced low, rumbly voice generates), and I had difficulty remembering what to say while at the same time remembering to change my cadence and take breathing breaks, but it came off well and both the bride and groom seemed happy.

There was an awkward moment, however, early on where things got dicey: I forgot to take a breathing break, so midway into the next clause I suddenly got slightly dizzy. Concerned about losing consciousness, I paused as soon as I could, which was not at a very natural breaking point. It wasn’t at a natural breaking point, but it was just before I was going to have to use the names of the bride and groom.

JUST before. Like, two words before.

I blanked for what felt like a half hour but was instead exactly long enough for all gathered to notice, before I pulled it back together, remembered where I was in my spiel, and soldiered onward.

Not, as it turns out, my favorite way to speak in front of people.

As I said though, I consider it an entirely positive experience and I hope that they were genuinely as happy with their wedding as they appeared to be.

Because I’ll want to find it later, here is the text of the ceremony that I wrote. I’m not a super sappy sort of guy, but I feel like it said the things I wanted it to say and didn’t come off either too cliche or too casual: both concerns I had going into this.

(For the record, I did not, in fact, forget to get the last name!)