Tag Archives: teaching

The Cleverness of Intent Over Content

I tweeted the other day about some quizzes I had taken that yielded results that were…unexpected. Resulting conversations ran the gamut from the relative merits of requiring leaders of technical teams to be technical folk to simply commiserating about the “impostor syndrome”-triggering nature of failing even a badly done test—and I fully intend on writing about some of these—but that’s not this post. This post is about a different concept that the assessments made me think about; the incredible difficulty of making good testing materials and one strategy for making better ones.

The fact that teaching people is a tricky and difficult thing should come as no surprise, but what I found only when I had been doing it for a couple of years was that the hard part isn’t the actual teaching itself—that part is actually fairly simple if you know the topic really well and can communicate with any degree of clarity. The hard part is honoring your initial intent with all of the materials, but especially exams and the like.

I started my undergrad—well, the final time I started my undergrad—a working software developer fully a decade into my career. By the time I sat down to take one of my first exams in a class purported to be an introduction to programming logic, I had been writing programs for double that time. That test bothered me to such a degree that it haunted my thoughts for years to come as I continued my career both as an instructor and a developer. Why was it such a terrible exam? How could a fantastic teacher create an such a bad evaluation tool?

More than half of the test was comprised of questions best described by the following template:

What is the definition of {word}?

  1. Obviously wrong answer
  2. Answer that looks right save for one very small error
  3. Answer that could be correct, but is clearly wrong
  4. Correct answer

Of the remaining questions, most only deviated from this formula by not specifically requesting a definition. My favorite example from this particular type of question (presented to you by virtue of the fact that I’m a digital hoarder with decades of bullshit on my hard drive):

An array is:

  1. A collection of values stored in one variable referenced by index 1 to n
  2. A collection of values stored in one variable reference by index 0 to n-1
  3. A single beam of light
  4. A list of similar but unrelated items

There is a host of problems with this question, but for someone who spent some time programming in Pascal and Fortran both in school and professionally in the years prior to this exam, this question was really galling.1

The crux of the problem is, there wasn’t even any point in the latter half of the text of the “correct” answer. It’s clearly very clumsily tacked on as a counter to the “trick” answer. Getting this question “wrong” by answering (1) doesn’t indicate that you don’t understand the material—at best it indicates that you were unclear on a nuance. More importantly, answering this question “correctly” doesn’t even indicate a fundamental understanding of what an array is—as evidenced by the lackluster results of the first practical exercises when we used arrays.

The instructor took their eyes off the prize and forgot what their intent was in giving the exam in the first place. So many tests make this exact mistake. The purpose of this exam was stated in print at the top of the first page:

The purpose of this exam is to demonstrate a basic understanding of how [to] use the foundational components of a computer program…

In most applications, simply knowing the definition of a word—especially to a pedantic degree—does not afford one any more ability to be proficient in a thing that not knowing the definition.2 Wouldn’t the following question have better suited the purpose?

For the following questions, use an array defined in C as follows:

char letters[5] = {“h”, “e”, “l”, “l”, “o”};

What index would you use in C to request the letter ‘e’?

Rather than the definition, if you correctly answer this question I now know if you know how to USE this foundational component of a computer program. It’s still an imperfect question, but already it is more aligned with my exam’s intent. But wait! What if you need to ask a vocabulary question in order to satisfy the intent of the exam? This isn’t uncommon, but the vocabulary question should be phrased in a way as to satisfy understanding over recitation.

In the bad array question above, the language used in the potential answers was directly from the teaching material. This is often done for a very rational, well intended reason: to AVOID tricking students by changes in wording. The problem is that it doesn’t really prove that the student understands the concept which—again—was the stated intent of the exam. It provides evidence that they can recite the verbiage that you provided already, not that they know what it means. Validating understanding of vocabulary in a way that is quickly and easily gradable (read: non-short answer) is tricky, but there are a number of strategies that have been shown to have success.

Most commonly, multiple choice (as above) but with the actual correct answer being a derivation of the textbook answer and the other answers being derivations of other vocabulary items in the material being taught. This can be done in the single format (again, as above), or it can be done in a many-to-many format (as in “draw a line between the word and its definition”). Asking the respondent to select synonyms and/or antonyms can also be valuable in some cases.

Strategies notwithstanding (and if you’d like more in depth information on strategies like these, I highly recommend How Learning Works by Abrose, Bridges, et al), all of this is secondary to resolutely ensuring that you choose mechanisms for evaluation that adhere to the reason that you chose to evaluate the student to begin with.

It is challenging. Even with this knowledge, and even after taking numerous courses on pedagogy, I still struggled with making my testing materials valuable to students. Some time after I had taken this fraught exam, I found myself giving exams that were in no way better than those I am describing here. During one frustrating exam creation session, I got up, walked to the dry erase board in my office, and wrote the following3 in huge letters directly in my line of sight:

I want to know that students that pass this exam will know exactly how to use the things I test them on here in practical ways, and that students that get questions wrong will know exactly what they need to study to be able to use those things.

I want no students that know the answer to a question to get it wrong.

I don’t want my exam to be clever, I want my students to be clever.

Simply, I wrote my objective statement where I could see it. I made my intent…well…intentional, and I did so in a manner that increased the likelihood that it would impact my actual behavior. The positive direction that this pushed my teaching and my students was palpable. The test that I was writing at the time immediately felt more “right” to me than any I had created before. Each subsequent quiz and exam moved closer and closer to the ideal I had in mind because each time I looked at my material I found new ways that it wasn’t honoring my intention. As my skill as an instructor improved, so, too, did my ability to find ways to meet those objectives.

The results weren’t simply gut feel, though. Exam scores improved, but more importantly so did the results of all project work and labs. My sample was small, but my pass rate went up by a small-but-measurable percentage. Better still, the students that came out of my classes started being lauded as particularly “well prepared” for higher-level courses to follow. In short, I hadn’t made the tests easier—I had made them more effective.

Years after this epiphany (and I use that term very loosely, here), I had the pleasure of getting positive feedback from a student at the end of my course. She was switching careers from a decidedly non-technical field to that of a developer, and among the things she said one that stood out to me was the observation that her test anxiety and impostor syndrome did not manifest so intensely on my exams; that, as she opined, the exams “didn’t try to make her feel stupid.” Her software career has surpassed my own at this point and I delight in the idea that this change in course might have played some small part in that.

In my experience, there’s no magic bullet that creates great exams. It is only through conscious, mindful attention to the goals of the exercise that you can hope to end up with the desired result. Conscious, mindful attention…and a ridiculous amount of practice. As an aside to the armchair quarterbacks out there mean-spiritedly snarking about “shitty tests” I extend this invitation: create an exam about something you know very well and see how difficult it is to make something of which you can be proud. I think you’ll be surprised.


1 The “correct” answer was 2, but Pascal, Fortran, and numerous other languages start their indices at 1 rather than 0. Further, most modern languages (even at that time) allow for non-numeric indices, making the question even more grossly inaccurate.

2 There are exceptions, obviously—knowing what “flammable” or “caustic” means could be pretty important in a lab setting, for example.

3 In reality, it was probably something very similar, it subtly mutated over time, but this is pretty close to what I wrote.

Moving to Agile: Training

I haven’t really had the mental energy to write much about our transition to agile for the last month or two because I have been spending so much of that time period putting together and executing trainings. Even with as much enthusiasm as I have for this, it has been a draining several weeks.

The human urge to generate complexity when something seems too simple makes teaching simple things a weird chore. When I walk someone through the thought process behind answering a specific Scrum question, it’s often perceived as too simple—I get wary looks from the audience as if I’m trying to trick them. There is no trick, it’s really that simple. Continue reading Moving to Agile: Training

Training Days

This and next week I will be performing a series of trainings for groups within our organization to describe how we’re using Scrum (initially, at least). This is easily my favorite part of my job.

Not the “performing training” part; while I enjoy that considerably, it is also utterly exhausting. No, my favorite part of the job is helping others understand things. Anybody can tell someone the answer. Some of those people can even tell someone the right answer. It is immeasurably more satisfying to walk someone through the though process by which the right answer was derived so that in the future you can watch them solve the next problem correctly. Continue reading Training Days

The Whys of Learning

In my experience both inside of and in front of classrooms, students can be broken into three general categories. Those that are simply filling a seat is the smallest group by far, and I’ve never entirely understood why those students are there. This group is not really the focus of my post, though. The last two groups are those that I wish to discuss: the Knowers and the Learners.

Knowers want to know facts…they want to be in possession of knowledge. I envision them as the intellectual version of hoarders. They collect information in a way that is astounding to me. My brain simply doesn’t work in a way that would allow me to amass trivia in that way. This shortcoming has been the bane of my existence throughout most of my education. I struggled my way through history, English, and mathematics courses for a long time because of this. Memorizing streams of date-event combinations, completely inane grammatical dictums, or seemingly arbitrary mathematical rules seemed so boring, pointless, and useless.

It is fairly simple to find the Knowers in a classroom. Knowers are the ones that have their heads down as they frantically scribble every word that the lecturer says onto a piece of paper. They are the students whose hands shoot up into the air to waste the valuable time that follows an instructor’s request for questions with the pointless “Will this be on the exam?” They are the students who will argue with an professor over some obscure piece of minutia or semantic quibble on an evaluation rather than acknowledge that there is a fundamental lack of understanding in play. They revel in multiple choice and in concepts being made black and white.

Sometimes, you will find the Knower in front of a classroom, although they are harder to spot.1 These are characterized by educators issuing a barrage of vocabulary and trivia on students. The Knower-Educator will teach ‘to the test’ oftener than not. Closed-book, closed-note, multiple-choice exams are often their mode of evaluation. It is possible to pass one of their courses—to excel, even—and still not understand any of the material at all. In many cases, it is laughably easy to do so, and I certainly made it a habit through much of my K-12 schooling.

Conversely, Learners want to know why the facts are…well…the facts. They want to be able to figure out the facts on their own. They are rarely satisfied with concepts in the form of “A is true, B is false” when there is a chance to learn in the form of “A is true because of X and Y, B is false because of Y and Z” or better still “We know X, and we know Y, what does this tell us? Yes, A!” This is the group I identify with the most. I learn because I want to know more than mere facts. Facts are largely boring things of no consequence in my life. What good is a pile of facts? In what way is my life going to be bettered by knowing the year that the Magna Carta happened2 or the formula for the quadratic equation? I want to be given the reasons for those facts. Given sufficient reasons, I can almost always backtrack and find the fact itself. Malcolm Gladwell has an example in Outliers: The Story of Success wherein a young lady is re-learning how to obtain the slope of a line and runs into a concept that derails many a math student: the slope of a vertical line. By working through the concept of slope, rather than just memorizing the formula, she was able to grasp and understand why such a line would be considered to have no slope3.

Learners are a little bit harder to distinguish, but there are some indicators. Learners will often sit back and watch a lecturer speak. They take infrequent, short notes. They interrupt the class to ask questions. They veer off topic as they make mental connections. They will argue an infrequently seen boundary case endlessly until they grasp why that case doesn’t jibe with the rest. The fact that many of these traits are often found in the group of students who just don’t care makes it hard to pick Learners out. One point of differentiation, though, is questions that begin with “Why” or “How come.” It is no coincidence that the period of our lives during which we are learning the most (and capable of verbalizing) is characterized by repetitions of these two questions, “Why” is the sound that learning makes.

A Learner-Teacher tends to teach in metaphors and examples. They will sometimes deliver new material as a narrative, as if they were sharing a story rather than a concept. Sometimes they will work from theory to practice, other times from practice to theory, but they always find a way to put the theory in there. They often fail entirely to answer a specific question directly, but answer in a series of leading questions; it is my understanding that this can be maddening to Knowers in exactly the same way that rote memorization is maddening to Learners.

I have never understood collecting things. I have trouble throwing things away that I think there’s even an the remotest possibility that I might use, but collecting for the sake of collecting baffles me. I have an extensive collection of books, but I have read (or intend to read) every single one of them. Among those that I have read, I have read most more than once, and intend to read them again. If I don’t expect to read a book (or to revisit it), then I don’t keep it. If I decide at some future time to read something, I can always obtain it through borrowing, a library, or a purchase. I don’t understand owning something for the sake of owning it.

I feel the same about knowing things. I cannot fathom knowing something for the sake of knowing it. I want to know why and how, because with the why and the how, I can open up a vast, boundless expanses of knowledge. I don’t have to memorize the formula for the slope of a line, I can figure it out because I understand how it was derived. If I don’t need the knowledge, I don’t bother with it; I can always look it up later. Anything that is more trouble than it is worth to look up or calculate I will take the time to memorize, but that is a last resort not a manner of learning.

I write about this today because I am frustrated by what I see as a shift in percentage between Knowers and Learners. It seems as though there is an increased focus on knowing fact in classrooms, and it comes at the expense of learning concepts. Every semester, it seems that I sit alongside more and more memorizers and less and less people inclined to work things out. Each class I stand in front of seems comprised less of people asking “why is that true” and more of people asking “will we be graded on this?” Students seem genuinely angry when they aren’t taught ‘to the test’ or when they aren’t presented a list of bullet-pointed facts to memorize and regurgitate like intellectual bulimia. In a time when acting out in classrooms or demanding better grades has become depressingly commonplace, I have seen the toll being taken on fellow instructors by Knowers wielding a sense of entitlement and misplaced senses of righteous indignation.

I don’t really know how to teach to a Knower…well…technically, I suppose that’s not entirely true. I do know how to teach to a Knower, what I don’t know how to do is to teach to both Knowers and Learners at the same time, and I am entirely unwilling to teach to the Knowers at the Expense of the Learners. This is something new that I am learning about myself. The first practical result of this new knowledge is going to be some changes to my future syllabi; the first of which is an addition to my FAQ:

Is this going to be on the exam? The short answer to this is “Yes, if I took the time to put it on a slide or talk about it, it is potential exam fodder.” The more complete answer is this: I do not require you to know many facts, but I do require you to understand concepts. When I am evaluating your understanding of the material, whether that be through assignments or exams, I will be evaluating your ability to use that conceptual understanding in some practical way. For example, I will not ask a C class to list the data types, I will ask which data type should be used for a given piece of data. As a result, yes, everything we discuss is likely to be on the exam, because when you put all of this stuff together, it dovetails beautifully to form a complete understanding of the thing you are learning about. You should probably just stick to the short answer.

I’m sure as I put together a more comprehensive set of thoughts on the subject, there will be many, many more changes.

1 It can be difficult to spot a Knower as an educator because many of the earmarks of a Knower are also found in bad teachers regardless of the sort of learner they are. Learner-Educators often teach like Knowers because they don’t have a better way at their disposal.

2 1215. I can still recite this off the top of my head. I could not, however, tell you what it is, why it happened, or anything practical about it…but even after 20 years, I can tell you when it happened. That, my friends, is being taught by a Knower.

3 No slope, not zero slope. You could also say the slope is ‘infinite’ or ‘undefined’ with equivalent impenetrability to those who do not understand the underlying concepts.

The Teacher Tax

I am writing in response to Luke Duncan’s October 11th article, “The Teacher Tax.” While I admire Mr. Duncan’s concern for the finances of fellow students and the economics of education, I feel compelled to put to rest some of the fallacious conclusions contained in the article. A “Teacher Tax”, while an entertaining rhetorical device, is not an apt description of the course packets on sale in the bookstore.

Instructors throughout academia publish their work with an eye toward widespread distribution. These publications are the vehicle that drives our educational system, financial compensation for such work is the fuel that powers that vehicle. While it is romantic to hold onto the ideal that teachers teach for love of the educational process, and while that is certainly true in many cases, even teachers have to pay the bills. Published works such as textbooks, study guides, workbooks, and course packets are intended for an audience that far exceeds the author’s classroom inhabitants; they are intended for use on a much broader scale. The amount of time and effort that writing and organizing such a work requires is way above and beyond that which an instructor’s salary is designed to cover. While it would be appreciated if instructors would give of their spare time for free, such charity should not be required.

The article goes on to make the statement that course packets should be sold at cost; and on this point I agree wholeheartedly. The students should be charged only for the expenses that go into the creation of these valuable learning materials (plus the bookstore’s profit margin). Of course, those costs should include the time and effort required to produce such a text in addition to those expenses incurred through publication and distribution. While new ideas are often wrought of self-sacrifice, comparatively few innovations are derived purely from altruism. Even Lawrence Lessig, the creator of the Creative Commons licensing movement, recognizes the importance of revenue-generation to the creative process. One of the components of the “copy-left” system he has created is exclusively used to license for-profit publishing.

Have some teachers found a nice side income? The question is asked with the implied timbre of accusation, as if the desire to acquire money in exchange for performing a task is something of which we should be ashamed. The concept of payment for services rendered is one of the ideas on which our economy is based. An engineer at an automotive company is paid for his work at the plant. When he uses that knowledge to repair a car on his own time, it is perfectly reasonable for that engineer to expect payment for his time. Time is a valuable commodity, and should be valued as greatly as the material cost of text production. It is safe to assume that most of the students here at Henry Ford are spending their tuition money in the hopes that they, too, can find a job where their time will be compensated for.

I, too, feel the strain of my financial situation each semester when I join the long lines in the student bookstore. Make no mistake, it is a problem shared by the bulk of the student body, and there is, perhaps, a need for some type of reform. Rest assured, however, that a supposed “Teacher Tax”, as defined by Mr. Duncan’s article, is not the source of said reform. Without the ability to make money from publishing, innovation in the area of schoolbook development would almost certainly grind to a halt.

A Change of Plans

Returning to school at thirty years old, I found myself somewhat jaded at the prospect of my career after school. I was certainly not excited about the prospect of obtaining a degree so that I could return to a career that I considered to be, at best, a chore. As a result, I initially entered into the co-op program at Henry Ford Community College with the expectation that I would gain very little outside of a grade and another entry on my resume, and the further expectation that it would be more of the same, uninteresting thing that I have been doing for years. Surely, there would be nothing of importance that I would be able to take away from a part-time job that I had not already gained from nearly a decade in corporate Information Technology. I could not have been more wrong. This co-op did not just give me a grade. Instead, I can say with no exaggeration that my time with this work-study program has been a life altering experience.

As a Lab Technician, my official responsibilities could be summarized best as an “equipment babysitter” of sorts; someone to ensure that the lab rules are followed while signing in students and helping students with the equipment. Unofficially, I spent much of my day during the first semester helping the Computer Information Systems (CIS) students understand their coursework. It is in this capacity that I found an exciting, enjoyable niche. I learned that I enjoy teaching college students. Suddenly, a new branch of opportunity sprouted before me, and my future prospects were no longer the bleak, dismal things that I had been dreading.

Now that I have adjusted my collegiate plans to those of an aspiring college instructor, my time as a co-op student has become even more useful. The time I spend at work is now time spent learning how to deal with a variety of different students with varying needs, goals, abilities, learning methods, and skill levels. Just two short semesters has already improved my ability to explain a concept from many different perspectives–trying repeatedly until I can find one that will work. This time spent has also served to teach me new ways to communicate effectively with students. These improvements have been key factors in the considerable expansion of my role as unofficial tutor this semester. I have been invited to sit in on the lab time for two separate CIS125 classes so that I can provide supplemental instruction to the students as they are actively trying to complete their assignments.

The experience, as a whole, has been extremely gratifying. The feeling of working with someone who is having a problem with a concept and being there when they have that “breakthrough moment,” and to know that I was a part of that process, has made the entire process something I strongly wish to remain a part of. I cannot imagine having ever gotten a chance to take part in this type of work without this work-study position. Had this position not been available to me, I would be continuing on my path back into a job field that I have never particularly enjoyed, merely to make money.

A significant percentage of the reason that this program has worked so well for me is that my employers are also educators. As such, they have an understanding of student life, of going back to school, or choosing a career path that suits your strengths, and of finding a path through the educational experience that will end some place meaningful. Certainly, the value of the time I have spent at work would be lessened, had I not been working with a corps of fine educators who have worked hard to help me succeed. The influence that each of these instructors have exerted upon my experience has been a great boon in guiding me down that path.

If I could make any suggestion to potential work-study students, it would be this: Try something new. Do not simply do what you have always done; instead, take this relatively risk-free opportunity to spend a semester doing something different. It might very well open up a completely new avenue of possibility for you that you had no idea existed.

When I was growing up, my father and I used to discuss what I would do when I grew up. I used to say “doctor” or “lawyer” because those were jobs that seemed to make a great deal of money. My father’s response was always the same. He would always reply, “A paycheck is a fringe benefit to doing a job that you enjoy.” To me, that always felt like an unrealistic goal, but today, having the benefit of my co-op position under my belt, I can say with absolute assurance that my paycheck most certainly is a fringe benefit of a job that I enjoy doing. For the first time since I came back to school, I am genuinely excited about finishing school and taking the next step forward, into my new, “grown up” life.