True or False

Well I’m a bit late to wish you all a Happy New Year, but I do hope that you’ve all had an excellent start to 2009!

While enjoying an extra week off at the end of the winter break, there were a couple of things that caught my attention. The first was an interview on the radio where Tom Burkard was making a case for replacing Sats essay questions with MCQs (a step too far perhaps?) and the second was watching the lead singer of the Stooges hawking insurance on the TV - it’s a strange world true enough.

Well, now that I’m back at my desk, my friend Bessie here (pictured to the left and standing in for the ox) has told me that it’s going to be a great year for e-Assessment. She’s particularly looking forward to:

• JISC’s ‘Review of Advanced e-assessment techniques‘ due for release sometime in March, which describes state-of-the-art techniques in e-assessment in HE and FE setting
• Games-based assessment, utilising virtual world technology, being developed by the SQA, scheduled to go live in the next academic term
• Due for release in May, 14 subject areas being developed for the Solar e-Assessment platform as part of the TranSETT project (you’ll need to scroll down a bit to see the section on TranSETT for that link)

And frankly a whole lot more, which will be covering in the blog. As always, if there’s anything of interest happening in your part of the world, I’d love to hear from you.

All the best from the Bessie and me!

I came across the Safe Exam Browser (SEB) the other day, an open source project to develop a secure web browser for online examination (essentially a customised version of Firefox). Currently, there is a Windows-only version that works with the Moodle and Ilias VLEs.  However, the roadmap shows that support for MacOS and Linux is a priority for future releases.

Features include:

• Fullscreen mode (without any navigation elements)
• Disabled actions (default):
  • Close or leave the window the test unless submitted
  • Shortcuts/Keys as Win, CTRL+ALT+DEL, ALT+F4, F1, CTRL+P, Printscreen, etc.
  • Right-click
  • Switch to other applications (unless permitted)
  • Surf the internet (unless permitted)

I’ve tried it with Moodle and it’s not that difficult to set up; there are some automated scripts to configure the VLE and database (though you may feel safer making the adjustments manually) and a couple of SEB .ini files that you’ll need to tweak.

It’s certainly an improvement over the ‘Show quiz in a “secure” window’ option within Moodle (though to be fair, it does warn that the security isn’t watertight). One difficulty with this kind of software is the extra barrier that it can introduce to students who are making use of assistive technology during an assessment. You can find useful advice on this area in the e-Assessment section of the JISC TechDis website.

Obviously, even the most secure browser will not ensure that the system won’t be abused. Still, I’d be keen to hear how you get on if you try this out - or indeed of any other methods that you use to make exams more secure.

One of my more ridiculous moments of teenage rebellion occurred while sitting the first part of my O-Grade chemistry prelim (a series of multiple choice questions). Five minutes into the exam, I grabbed my ruler, drew a line down the C column of both sides of the paper and strode out of the hall.

Now if you’re anticipating an epic tale of ‘dropout makes good’, I’m afraid you’re going to be disappointed. After the euphoria of rebellion had worn off, I began to panic, spent the next couple of hours revising and returned to the exam hall to sit the second half of the prelim (a written paper worth about 60% of the overall mark).

Expecting the worse, my teacher handed out the results in class two weeks later. To my amazement, I discovered that I’d passed! There was a terse, ‘See me after class’ message at the bottom of the first paper, but it was the 39% mark at the top that really caught my attention. Long live the revolution!

How to Pass Tests 101

I don’t suggest that you follow my example, but it turns out that what happened to me all those years ago wasn’t that strange after all. If you know what to look for, there’s a good chance that’ll you’ll be able to beat the odds too.

There was a great posting on Donald Clark’s Plan B blog on the subject ‘All of the Above - how to cheat Multiple Choice questions‘ that lists 20 tips on how you can beat any MCQ. Below are the top three:

1. Skip the hard questions, mark them with a cross, and go back to them. This means you’ll not lose marks for unanswered easy questions.
2. If in doubt choose ‘C’, poor questions designers do not truly randomise the right options and have a bias towards ‘C’. Next best is ‘B’.
3. If in doubt choose the ‘longest option’. Question designers often cannot make a right option any shorter, but have complete freedom with wrong options.

The heart of the issue of course is not how to ‘cheat’ on exams, but to write better questions. This doesn’t mean questions that your students won’t be able to guess the answers to, but valid questions that really test their understanding of a subject.

Not the best way to pass an exam

Phil Race came up with an interesting set of questions in his ‘content-free quiz’ that highlighted some of the common errors made when writing multiple choice questions. We’ve produced a variant of his quiz for you to try below. There are 8 questions, don’t panic if the questions don’t seem to make any sense - just select the answer that you think might be right!

How did you do? Better than you expected? Well, you can download the quiz from the link below where you’ll find some feedback to explain the choices that you might have made.

Content Free Quiz (PDF, 12KB)

One of the most important aspects of formative assessment is the feedback given to learners that allows them to advance their knowledge or skills. Dylan Wiliam argues that without this mechanism, assessment can’t be considered to be ‘formative’,

“Assessment is formative only if the information fed back to the learner is used by the learner in making improvements”

The key in that sentence for me is the bit about ‘only if… used by the learner’. I’ve spent a lot of time in the past writing detailed feedback on assignments only to find students scanning the page for the grade and ignoring the additional commentary. I know this to be true, because I used to ask the same students about something I’d written and be faced with blank stares.

Sigh.

That’s why I was quite taken by the JISC- funded project, ‘Sounds Good: Quicker, better assessment using audio feedback‘. Although as a language teacher I’d used audio feedback before (recording comments onto cassette tape in a language lab), I hadn’t come across the practice in other disciplines. The initial findings from the project seem to show that students are genuinely engaged with this style of feedback, considering it to be more personalised and detailed than the traditional written forms.

The more I think about it, with open source applications like Audacity, recording feedback should be a breeze. I’m conscious that not everyone is keen to record themseleves speaking - it’s something that takes a bit of practice to get right (you’ll find some useful tips in the initial findings linked above). If you do decide to give it a go, then please let me know how you get on!

(Oh, I noticed the project has two sites - the older site has a few interesting documents not available on the newer site that are worth checking out)

Wow, has it really been 4 and half months since the last post? Where has all of the time gone? And why is my local supermarket selling Xmas decorations ? Surely, it can’t be that time of year already…

Ah well, I’ll try and make for my absence with some interesting bits and pieces that I’ve across:

Launched at ALT-C in September 2008, Effective Practice with e-Portfolios (PDF, 781kB) draws out key points from established and innovative examples of practice in further and higher education and from a selection of significant JISC-funded projects on e-portfolios.

The publication complements an infoKit on e-portfolios which synthesises the main outcomes from JISC-funded projects in this field and covers the main drivers, purposes, processes, perspectives and issues around e-portfolios. Audiences for both resources will be those whose roles might involve use or implementation of e-portfolios as a tool for learning: learners, practitioners and managers in FE and HE, including IT and network staff; professional bodies; those involved in initial teacher training and in the management and implementation of PDP practice.

Now I have to admit that I don’t actually own a mobile phone, though I do occasionally borrow my wife’s (much to her annoyance). It’s amazing what you can now do on such a device - even those that aren’t ‘smart‘.

Mobile Study off you the ability to create quizzes (with images) that can be downloaded onto most modern mobile phones either directly or via a computer. These quizzes then run as standalone java applications. For the techies out there, if your phone conforms to the MIDP 1.0 specification (most of them), then all should be well.

There’s also a Moodle plugin that allows you to export quizzes created in the VLE to your students’ phones. The scores of these quizzes aren’t recorded in Moodle, this shouldn’t be a problem for most.

Multiple choice questions are supported by feedback, though this is limited to text (without hyperlinks).

It’s been my experience that students respond well to these simple short quizzes, especially if they are delivered in a format that allows you to try them ‘anytime, anywhere’.

We’d love to hear from any of you that have tried to use mobile quizzes with your students. Tell us what you think of the format and how well your students have responded by leaving a comment below!

The JISC-funded WebPA project has won a Bronze at the IMS 2008 Learning Impact Awards in Austin Texas. The open source online peer assessment system was the only UK entry in the field of twenty-three nominees from across the world.

I recently attended a presentation by Nicola Wilkinson from the WebPA development team and was impressed by the simple but effective model that underlies the system.

Students assign themselves and members of their group grades according to a defined set of criteria: e.g. ‘Participation’, ‘Attendance of group meetings’, etc. The lecturer then awards the project an overall grade, which the system adjusts for individual scores according to the input of the students.

It’s a neat uncomplicated solution that allows students to have a say in the assessment process, especially the part that isn’t always visible to the lecturer.

Download the software here.

The NHS e-Portfolio was a finalist in the 2007 e-Government National Awards in the category of ‘Excellence in Learning & Skills’ (though the prize finally went to Gloucestershire College for their StARs & StRIPEs system).

It’s currently being used by over 16,000 doctors in training across the UK as well as 700 dentists and over 100 pharmacists in Scotland. Activity on the system remains high, with 81,000 clinical (skills) assessments, 55,000 multi-source feedback assessments and 130,000 educational log entries in the system to date. Summative assessments submitted by other healthcare professionals to demonstrate competency of individual doctors account for 60% of these entries.

August 2008 will see the launch of the second version of the portfolio. Select this link for a sneak preview of e-Portfolio 2.0 and more information on the project. If you’d like to compare it with the current system, then demo logins for the original portfolio are also available.

I was looking around the Microsoft website for a solution to a problem I was having with Word (a ruler line that refused to be deleted no matter what I tried!). After ten minutes of fruitless searching, my mind (and cursor) started to wander and I came across a section on Office Quizzes.

Obviously, these only cover Microsoft products, but could be useful to students to want to quickly revise their understanding of particular applications, or as part of some additional ECDL/PC Passport classes?

Do you know of any other useful assessment resources on the web? If so, please add a comment and share the knowledge!

As I was trawling through the net, I came across this concept map on the University of Bath site describing some of the key issues surrounding e-Assessment. It’s somewhat dated, and the embedded links don’t really lead anywhere, but it’s still a neat representation. If you happen to know of something better out there, please let me know!