Select Page

What is it?

Accessed through

https://www.cs.auckland.ac.nz/automated-marker

the automated-marker service compiles and evaluates source code uploaded by users. The source can be coded in a number of programming languages including:

  • C/C++
  • java
  • python
  • sage
  • ruby

The service allows staff (admin role) to define certain parameters and requirements for a single file source code submission. These parameters will include processing and performance requirements for the compiled code, and may include limits to submission count, file sizes, etc. Each defined submission template would normally be associated with a course. 

The service is used with courses in Computer Science and Software Engineering.

Screen shots

The student view
The student view

The system decides what courses a student is in are in and will show the student user any assignments associated with those. Current assignments are listed above past ones.

 

Admin view
The admin view

An admin user can see anything going on in the system. Tools are made available for monitoring system activity, creating assignments, and managing class activity.

The scoreboard
The scoreboard

The scoreboard shows student activity within an assignment. Submissions associated with students are listed with various counters attached. The *s are the number of submissions, the (n) values are the run time associated with the last submission (a performance measure), the larger number is an cumulative time total. 

When assignments are active, this list will include running status information.

 

How does it work?

There are two key functional elements to the service, a web front-end, and a back-end. These communicate through a database, and share access to a file service. The web part handles authorization, information capture, and application logic. The back-end loads and compiles submissions, then tests them against performance requirements, feeding results back to the database.

The automated-marker eco-system

The automated-marker eco-system

Authentication is handled by SSO (shibboleth), and authorization is a mix of local data (for the admin role), and LDAP (for student-course information).

Automarker, the back-end, polls the database for queued submissions every 5 minutes or so. Typical compile and run would be measured in seconds or minutes. Long delays in processing a submission could be indicative of a service failure on the back-end itself, or of the file service.

The agent that does the work on automarker requires manual startup -this is done as the marker user by the authors. This agent is a java application.

Is it working?

While it is not impossible that code submission could cause the service to fail, in recent times the most likely causes are database and file service failures. When testing to see where issues may be in the service check the key components highlighted in the diagram above.

The front end service

Jump to https://www.cs.auckland.ac.nz/automated-marker/ -you should be asked to log in (via shibboleth/SSO), and then (if you are in the right group), you should see something looking like the interfaces shown above -The web side is (probably) fine! The right group is automated-marker-students.cs (in the EC LDAP directory), a combination of Computer Science staff, Science IT staff, Computer Science and Software Engineering  students.

The NetAccount view of the group...

The NetAccount view of the group…

If not,  check https://www.cs.auckland.ac.nz/, which is delivered via webroute+AEM. You should see a University styled departmental web page for Computer Science. If this does not work, check local network connectivity before calling the servicedesk.

The shibboleth interface is specific for automated marker, but the general IAM service can be checked by going to any SSO wrapped site, e.g.. https://www.sit.auckland.ac.nz/sso/protected/.

Automated-marker lives on a distinct web service, try http://130.216.33.163/our_staff/profile.php?id=ramo001, which connects to the webcluster2 hosted part of the CS web site, as does http://[2001:df0:0:2001::163]/our_staff/profile.php?id=ramo001. These links should show the staff profile of the current HoD (@April 2014).

The backend and supporting services

Ping automarker.cs.auckland.ac.nz. An echo indicates that the virtual machine is reachable. If there is no echo, then the virtual host (or hosting service) may be having issues – call the servicedesk

CS admins (aka. IT staff) can log on to the box via ssh. If you are able to, then check for the automarker process:

$ ps aux |grep java
marker    3130  0.1  1.8 663064 39140 ?   Sl   Mar29   8:01 java -Xmx512m -cp ./mysql-connector-java-5.1.15-bin.jar:. Server

This java process is run manually by the marker user.

The system relies on a database hosted on the helix-gate cluster. Loading on this service can cause delays in responses. If the database is unavailable, or slow, then the communication between the front and back end will be disrupted. Issues with the database server would show up in the web interface -submissions or retrieving assignment information would fail or be slow.

The system also shares files through an NFS mount -check that smb://files.fos.auckland.ac.nz/web (\files.fos.auckland.ac.nzweb) is available and has free space.

On automarker (compile and evaluate):

$ mount |grep automated-marker
nfs.fos.auckland.ac.nz:/mnt/webdata/html/www.cs.auckland.ac.nz/automated-marker/ on /mnt/www.cs/automated-marker type nfs (rw,acl,addr=130.216.35.126)

On webcluster2 (submissions):

$ mount |grep webdata
130.216.35.107:/mnt/webdata on /var/www type nfs (rw,nosuid,nodev,addr=130.216.35.107)

A simple check, if you have access to automarker, webcluster2, [and some other mount of the file system]

login@automarker $ ls -1 /mnt/www.cs/automated-marker/ |wc
login@webcluster2 $ ls -1 /var/www/html/www.cs.auckland.ac.nz/automated-marker/ |wc

and optionally

login@mymac $ ls -1 /Volumes/web/www.cs.auckland.ac.nz/automated-marker/ |wc

should return the same results, and

login@automarker $ df -h |grep automated-marker
nfs.fos.auckland.ac.nz:/mnt/webdata/html/www.cs.auckland.ac.nz/automated-marker/
                      1.2T  1.1T   84G  93% /mnt/www.cs/automated-marker

should show some free space (84GB is OK for automated-marker).

The LDAP service determines what assignments are available to a student. Issues relating to seeing incorrect course association may signal an LDAP problem, but group membership issues would normally indicate a system or process problem (ie. enrolment, etc). Service availability and group membership can be tested through the EC tool in the matrix -try checking an affect UPI using https://www.sit.auckland.ac.nz/sso/protected/support_tools/eDir.php -look for group memberships matching specific assignment names.

Contacts

Mark Wilson and Michael Dinneen in Computer Science.

Skip to toolbar