[QTI] manually scored items

[Steve Lay]

Yvonne Winkelmann wrote:
> We have just implemented an imsqti-2.1-engine and java-based player
> in conjunction with the learning management system OLAT.

I know this isn't the point of your message, but is OLAT (still?) an 
open source development?  Are you developing it using an open system 
like Sourceforge?  I think lots of people on this list are interested in 
this.

 >Now we have
 > got problems with combination automatically items and such who
 > assessed by scorer. Unfortunately I am not able to find detailed
 > information about the workflow with manually scored items. Firstly, I
 > long to know the general handling. We use empty outcome variables,
 > who are used for the manually score after the test is finished. Can I
 > enter the score in the report file or is it exclusive for
 > automatically generate outcome variables?

QTI doesn't deal with manually marked questions specifically.  But, 
there is no requirement for an outcome variable declared in an 
assessmentItem to have its value filled by a rule in the corresponding 
response processing.  Indeed, the responseProcessing element is optional 
to allow you to distribute questions with no marking information at all 
if you want.

It is envisaged that the report file generated by a test engine could be 
processed later by another system (either automated scoring or human 
marking workflow) and that the additional outcome information would then 
be added to the report.

Notice that the results reporting specification has a sessionStatus 
attribute on the itemResult which can take the value 
"pendingResponseProcessing" to represent the state in which the 
candidate has submitted responses but these have not yet been processed.

There is an issue here, which *isn't* that clear from reading the 
specification.  How do we tell the difference between an item with 
externally defined response processing (like your human marked items) 
and one that simply has no response processing at all?  For example, an 
item that purely asks for information like "What is your favourite 
colour?" might well have no need of response processing at all.

It is also possible for an item to use a hybrid solution with some 
outcomes being generated by rules defined by QTI in the 
responseProcessing element and others being assigned by humans or other 
external systems.  It is even possible for external systems to update 
the outcome values already set by the responseProcessing rules.  You 
might even have discovered an error in the original responseProcessing 
rules and need to remark an item (of course that never really happens ;-).

In these cases of external processing (or hybrid) it is not clear what 
sessionStatus value should be used and how the delivery engine knows if 
response processing is really complete.  Does it matter?  Sometimes, I 
think it does.....

 >Can I use the manually
> scores in the test outcomeprocessing or for preconditions if the
> learner continue his test?  And how is it intended if the manually
> scoring take place outside the assessmentDeliverySystem?

Outcomes processing is supposed to take place after response processing, 
at each submission point.  It is always safe to re-run the outcomes 
processing rules following an update to the individual item outcomes. 
This is useful if the outcomes have been updated by an external system 
as described above.

You will need to define branches and pre-conditions carefully in your 
assessmentTests to ensure that they only rely on item outcomes that are 
assigned by QTI-defined rules in the items themselves.  The same is true 
for test-level feedback marked with access="during".

Test-level feedback with access="atEnd" is more of a problem.  How does 
the delivery system know if more response processing is required?

To give an example.  If all of your items define SCORE as an integer 
outcome and are marked by humans then at the end of the test all the 
item SCORES will have defaulted to the value 0 while they are waiting 
for marking.  However, if the outcomes processing adds up these values 
the result will be a test score of 0 which could be used to show 
inappropriate feedback to the student!

Yvonne, you are asking some interesting questions which are providing 
really useful input into the specification process.  Hopefully this type 
of issue can be clarified by the project team in the final version of 
the documents.

If others on the list have views about how this slight ambiguity should 
be resolved then please contribute to this thread.

Steve