Chittaranjan.Mandal@iitkgp.edu.eu.org
School of IT, IIT Kharagpur, India
(Contact Author) Chris.Reade@kingston.ac.uk
Tel: 020 8547 7232
Kingston Business School,
Kingston University, Kingston upon Thames, UK
Web.Based.Course.Management@sit.iitkgp.ernet.in
School of IT, IIT Kharagpur, India
Keywords: automatic evaluation, programming, assignment
We describe a scheme for the automatic evaluation of programming-oriented assignments. The scheme is motivated by the need to handle programming assignments with large cohorts of students (approximately eight hundred students) and by the recognition that a large part of common programming assignments can be designed for systematic evaluation.
The work reported here is compared with other work on automatic evaluation of programs reported in the literature such as the GAME system of Blumenstein, Green, Nguyen and Muthukkumarasamy (2004). Although this work is not yet as generic as the GAME system, it is designed for simplicity of use and is implemented as a system that is usable over the web.
This paper focusses on four key areas of concern that were considered and dealt with in the design of the scheme.
Choices for automation of the evaluation process
Security of the process
Marking issues
Overheads for assignment setup
Automation of the evaluation process is the key problem and has varying solutions depending on what is found acceptable. The paper describes what the authors found acceptable in their specific context as well as looking at other possibilities. A particularly important decision was whether submissions should be treated as single 'black boxes' with only overall I/O behaviour to be tested; 'grey boxes' that allow components and functions to be exercised, or 'white boxes' which allow structure as well as behaviour to be evaluated. The first approach is, in general, unworkable in a class where students are learning to program. This is because conformance to overall input/output requirements is particularly hard to achieve and evaluations that relied on this would exclude constructive evaluation for a majority of students. The other approaches involve exercising individual functions within the student's programs and are, unfortunately, language sensitive. This entails per language support for automatic evaluation. At present we have restricted ourselves to programs written in 'C'.
Security is a non-trivial concern because automatic evaluation, almost invariably, involves executing potentially unsafe programs. One cannot rule out the possibility of a malicious program or one that accidentally causes damage to the execution environment. This system is designed under the assumption that programs may be unsafe and executes programs within a 'sand-box' using regular system restrictions.
Marking issues cover what can be assessed, such as (i) correctness of behaviour under tests; (ii) balancing of assessment for partially working programs or parts, and (iii) performance based marking.
The overheads for assignment setup to allow for automatic evaluation include: (i) overall design in terms of functions/parts that can be specified for separate evaluation, and (ii) supplying a minimal body of code to exercise the parts/functions supplied in a submission. The system provides support to reduce the work that an instructor needs to do to set up an assignment.