Quality of assessment

Partly by design – but mostly due to the correct alignment of the stars – I’m one of the few lucky people who does kendo as part of their job. Depending on the time of year it can pretty much be non-stop. Believe me, it’s neither as easy or exciting as it sounds and, of course, there are times when all of it gets too much (both physically and mentally/emotionally)… but in general I’d say that because of this strong kendo element within my job I mostly enjoy my working life.

Sometimes, the non-kendo things are a real pain though, and one such thing rolled around last week: an annual “training” seminar. This year a professor was invited from a prestigious private university in Tokyo to lecture on the topic of “classroom assessments” …. brilliant.

Actually, the content of the lecture wasn’t actually that bad, it was just mostly irrelevant to my actual day-to-day work. Cue my brain to – as it generally does in situations like this – switch into kendo mode (I think this is actually the default setting). One topic in particular during the lecture caught my attention: “quality of assessment.”

Gradings (i.e. assessments) are something that we all go through, and I’m betting that all of us have experienced failure as well as success. This seems to be the normal way of the world and it’s probably healthy that we face a mix of each. Anyway, one thing that I’ve noted repeatedly over the past few years is that – despite my increased knowledge about and experience in kendo – I seem to have difficulty predicting if someone will pass or fail with accuracy. Either this is because I simply am not yet experienced enough (or smart enough) to understand the intricacies of the grading procedure, or it’s because of some sort of strong element of subjectiveness (even randomness?) within the procedure itself.

Last week at the seminar a couple of thoughts struck me (all though I am of course considering kendo in Japan here, I’m pretty sure the same questions can be applied to any national organisation):

– The ZNKR is quite consistent in the percentage of people who pass grades, how is this done?

– At gradings emphasis is always on the examinee, not the examiner. Are examiners trained and are their choices judged? Are “bad” examiners removed or re-trained?

Hmmmmm, I see the potential for some worms and a can.

Anyway, here are some points regarding the “quality of assessment” from last weeks lecture (in bold), with a few brainstormed questions from yours truly. Please feel free to consider, argue, or add in your own ideas in the comments.


Points to consider when looking at the quality of an assessment

1. Validity

The degree to which an assessment taps into what one intends to measure.

Do gradings really reflect what kendo practitioners really do during their keiko and in shiai, or do they have to show some something else (an idealised version of what they are supposed to do)?

Does the required content of gradings actually progress through levels, or does it remain somewhat the same between them?

Is there any bias? This could be age or gender bias, or perhaps questions about impartiality (especially pertinent in smaller organisations, or in arts where examinees are not anonymous).

Are participants being judged on what they can do or are they being compared to their opponents? If the latter is true, is it fair to match people who have wildly different ages or to mix genders?

etc.

2. Reliability

The degree to which assessment results are consistent no matter when and where a student takes an assessment or who scores the student’s response.

Is judging consistent across all examiners?

Is judging consistent across grading locations?

Is the content and task difficultly consistent across all parts of the grading process (shiai, kata, written)?

etc.

3. Practicality

The degree to which an assessment can be administered and maintained with available resources.

Does the organisation have enough people with the required experience (and training) to host a grading?

etc.

(True story: I remember being asked to read, then pass or fail the grading questions for 4dan in London years and years ago… I was 3dan at the time)

4. Impact

The degree to which an assessment gives positive and/or negative effects on test takers, teachers, students, and society.

Are participants simply “failed” or are they given useful feedback to promote future improvement?

Do the overall results provide useful information for kendo teachers to aid in the development of kendo for the future?

Are examiners fully aware of the ramifications for the future of kendo should people of sub-par ability be promoted?

etc.


I guess what I am sort of addressing here is the very obvious difficulty in ensuring that the grading process is done accurately/fairly. The current system seems to be highly subjective and seems to have – at least here in Japan (where grading times are extremely short and examinees are somewhat anonymous) – an element of randomness within it. After much thought on the matter, I’ve come to the tentative conclusion that the grading system is probably the weakest area (most open to problems) in modern kendo.

Anyway, these are just some thoughts that I’ve had for a while but which re-surfaced and became re-packaged based on the content of the lecture I listened to last week. If you have any ideas/thoughts/opinions on the matter please feel free to discuss in the comments, either here or on facebook. Cheers.

Published by

George

I'm the founder and chief editor of kenshi247.net. Amongst other things I am a high school kendo club coach, an avid practitioner of classical swordsmanship, a history student, and a vegetarian.

One thought on “Quality of assessment”

  1. I had a discussion with the staff at the Kagawa Federation a few weeks back because I wanted to have a list of criteria for each grade (starting with the kyus since I am working with a few beginners at the moment and I’d like to have them pass their kyus as soon as they can). I even had a similar discussion with an HS teacher here who’s been doing that for about 3 decades… The answer astonished me: neither the people at the federation nor the teacher were able to give me objective criteria for kyus of dans. The agreed answer was: we evaluate the examinees by comparing them to the examined batch… I understand that examiners experience does create a framework in which they can evaluate a majority of people with some accuracy, but it certainly does not look like there is a *system* to avoid obvious inconsistencies. And thank you again for the interesting post !

Leave a Reply