Quality of assessment

Partly by design – but mostly due to the correct alignment of the stars – I’m one of the few lucky people who does kendo as part of their job. Depending on the time of year it can pretty much be non-stop. Believe me, it’s neither as easy or exciting as it sounds and, of course, there are times when all of it gets too much (both physically and mentally/emotionally)… but in general I’d say that because of this strong kendo element within my job I mostly enjoy my working life.

Sometimes, the non-kendo things are a real pain though, and one such thing rolled around last week: an annual “training” seminar. This year a professor was invited from a prestigious private university in Tokyo to lecture on the topic of “classroom assessments” …. brilliant.

Actually, the content of the lecture wasn’t actually that bad, it was just mostly irrelevant to my actual day-to-day work. Cue my brain to – as it generally does in situations like this – switch into kendo mode (I think this is actually the default setting). One topic in particular during the lecture caught my attention: “quality of assessment.”

Gradings (i.e. assessments) are something that we all go through, and I’m betting that all of us have experienced failure as well as success. This seems to be the normal way of the world and it’s probably healthy that we face a mix of each. Anyway, one thing that I’ve noted repeatedly over the past few years is that – despite my increased knowledge about and experience in kendo – I seem to have difficulty predicting if someone will pass or fail with accuracy. Either this is because I simply am not yet experienced enough (or smart enough) to understand the intricacies of the grading procedure, or it’s because of some sort of strong element of subjectiveness (even randomness?) within the procedure itself.

Last week at the seminar a couple of thoughts struck me (all though I am of course considering kendo in Japan here, I’m pretty sure the same questions can be applied to any national organisation):

– The ZNKR is quite consistent in the percentage of people who pass grades, how is this done?

– At gradings emphasis is always on the examinee, not the examiner. Are examiners trained and are their choices judged? Are “bad” examiners removed or re-trained?

Hmmmmm, I see the potential for some worms and a can.

Anyway, here are some points regarding the “quality of assessment” from last weeks lecture (in bold), with a few brainstormed questions from yours truly. Please feel free to consider, argue, or add in your own ideas in the comments.


Points to consider when looking at the quality of an assessment

1. Validity

The degree to which an assessment taps into what one intends to measure.

Do gradings really reflect what kendo practitioners really do during their keiko and in shiai, or do they have to show some something else (an idealised version of what they are supposed to do)?

Does the required content of gradings actually progress through levels, or does it remain somewhat the same between them?

Is there any bias? This could be age or gender bias, or perhaps questions about impartiality (especially pertinent in smaller organisations, or in arts where examinees are not anonymous).

Are participants being judged on what they can do or are they being compared to their opponents? If the latter is true, is it fair to match people who have wildly different ages or to mix genders?

etc.

2. Reliability

The degree to which assessment results are consistent no matter when and where a student takes an assessment or who scores the student’s response.

Is judging consistent across all examiners?

Is judging consistent across grading locations?

Is the content and task difficultly consistent across all parts of the grading process (shiai, kata, written)?

etc.

3. Practicality

The degree to which an assessment can be administered and maintained with available resources.

Does the organisation have enough people with the required experience (and training) to host a grading?

etc.

(True story: I remember being asked to read, then pass or fail the grading questions for 4dan in London years and years ago… I was 3dan at the time)

4. Impact

The degree to which an assessment gives positive and/or negative effects on test takers, teachers, students, and society.

Are participants simply “failed” or are they given useful feedback to promote future improvement?

Do the overall results provide useful information for kendo teachers to aid in the development of kendo for the future?

Are examiners fully aware of the ramifications for the future of kendo should people of sub-par ability be promoted?

etc.


I guess what I am sort of addressing here is the very obvious difficulty in ensuring that the grading process is done accurately/fairly. The current system seems to be highly subjective and seems to have – at least here in Japan (where grading times are extremely short and examinees are somewhat anonymous) – an element of randomness within it. After much thought on the matter, I’ve come to the tentative conclusion that the grading system is probably the weakest area (most open to problems) in modern kendo.

Anyway, these are just some thoughts that I’ve had for a while but which re-surfaced and became re-packaged based on the content of the lecture I listened to last week. If you have any ideas/thoughts/opinions on the matter please feel free to discuss in the comments, either here or on facebook. Cheers.

Shinsa – things to think about

The following is a translation of three very short pieces by Sakuma Saburo hanshi on the subject of gradings. Obviously there is some overlap between the articles. I don’t know about you, but I personally hate gradings and need all the advice I can get !!


Things to be careful about in gradings

  1. Improve your posture
  2. In other-words, ensure that you learn kendo-no-kata thoroughly.

  3. Kiai with a loud voice
  4. This serves not only to rouse your own spirit but intimidate your opponent.

    If I were to give an example from my past, there was a time I went on a 10-day gasshuku. By the 3rd keiko of the day my voice would be hoarse and dry and I couldn’t kiai anymore. Around about the 5th day my voice started to survive even the 3rd keiko session. Going through this pattern over-and-over you will finally develop a loud and sharp kiai that resonates in your opponents stomach.

  5. Attack with abandon (fervour)
  6. “Now!!!” – the very instant you think you see an opening for attack you should attack with full abandon irregardless of what your opponent may attempt to do to you or your shinai.

    If your strike doesn’t land then you should – in the same breath as your first attack – continue striking over-and-over until a valid yuko-datotsu is struck.

    To develop a nimble and flexible kendo style (so you can do the above) you should do intense kakarigeiko with your sensei or sempai (of about 50 seconds to a minute)

  7. Express zanshin
  8. If you think you have struck a valid yuko-datotsu then take an appropriate distance and express your confidence in that strike.

  9. Only do keiko with people better than you
  10. Never do kendo with people of lesser ability than yourself.

    If you are currently practising with the intent of taking a kendo exam then you have to be a little selfish and decline doing keiko with those that aren’t at your level. If you do kendo with these people then your focus will relax and your level will drop. If for whatever reason you can’t refuse, use the keiko as a chance to practise your techniques.

    Against a more senior opponent, first fight for the first strike (shotachi). After that is over continually attack them until the keiko is finished. At that time, be sure and get advice from them.

  11. Acquire various techniques
  12. Do lots of kihon and oji-waza practise. You will face many kenshi who have many different styles of kendo. It’s important that you learn enough techniques so that you can deal with any style of kenshi that you face (i.e. have a large repertoire of techniques which you can select and apply appropriately depending on your opponents style).


Ten points on gradings

  1. Sink slowly and composed into sonkyo. Resolve yourself to feeling “When I stand up I’m going to strike the instant you move (debana).” Stand up deliberately with this in mind.
  2. From the pit of your stomach kiai so as to rouse your spirit and intimidate your opponents.
  3. It doesn’t matter what happens during the shiai, never move back.
  4. Don’t attack recklessly – aim for debana.
  5. If your attack isn’t successful don’t stop and rest – continue striking until you land a valid yuko-datotsu.
  6. If you think your strike is successful take an appropriate distance, ensure that the extension of your kensen is aimed towards your opponents throat, and express zanshin.
  7. Aim for ai-uichi, that is, strike at a hairs-breadth before your opponent.
  8. Get out of tsubazeriai quickly.
  9. Don’t face your back towards the examiners.
  10. After doing the final bow move backwards 3 steps before leaving the area.

About kendo gradings

Some people thing that gradings should be approached in a special manner, but I believe that you should just do your normal kendo, nothing special. Just do what you have been taught by your sensei and sempai.

Here are things that you should be doing as a matter of course:

  1. Wear your uniform correctly.
  2. Act respectfully (i.e. proper emphasis on reigi).
  3. Fight energetically and with a strong spirit.

Here are some extra points worthy of note:

  1. Don’t just attack men.
  2. Some teachers say “Strike men, strike men… who cares about dou etc.” but following this advice can make it difficult for you to pass.

  3. Strike gyaku-do.
  4. There are people that don’t strike gyaku-dou even when it is wide-open. Left and right dou hold the same value in kendo.

    In competitions sometimes shinpan haven’t read the rule book carefully on this point (and thats why they don’t award it and hence why people don’t do it in gradings). Some people, however, end up hitting the floor after striking gyaku-dou, that shouldn’t be considered ippon.

  5. Sometimes I see people strike ai-uchi and then they turn and look at each other as if they are mutually resting… I have no idea why they do this.
  6. (editor: little bit hard to understand this point)

    If your opponent seems to be resting, strike him immediately. If the distance is relatively close people tend just to strike men, but at such a distance it’s simple to defend against. At this time you should tsuki your opponent back, breaking their posture, then strike.

  7. Half-baked strikes are minus points.
  8. Don’t strike randomly.

  9. People often lose (fail) because their movements become ‘stuck’ or their kensen is often off-centre.

grading2


About the author

Sakuma Saburo was born in Fukushima prefecture in 1912 and started kendo in primary school. Before the war he taught kendo in various places. After the war he trained under Mochida Moriji at the Mitsubishi dojo before opening his own. He held a senior position in the Tokyo kendo renmei. He passed away at the age of 84 in 1997.


Source

平成・剣道読本(下)。佐久間三郎。平成9年発行。