Overview

This documentation is not complete. If you have any additions or questions, please do not hesitate to contact us.

My Courses

The management of the course happens under My Courses from the main menu.

If you have more than one course you can select the course and semester in the dropdown in the top left corner.

Cherry-pick Mechanism

You can choose currently between two different modes, how submissions are automatically selected for the grading.

last: The last submission a student submits before the due date will be automatically selected. This is the default.
best: The best submission a student submits before the due date will be automatically selected.

An TA can always choose another submission of the student to select it for the final grading.

Enroll Url

Provide your student with this url. They then can enroll in your course via this url.

Messages

Here you can answer messages send by students to your course. Or send a course notification to all students currently enrolled in the course.

Exercises

You can create exercises in this panel. See setup of project for more information about how to setup projects.

Code Examples

You can create code examples in this panel, which are projects that will not be reviewed.

Groups

You can organise your course into groups. The students will see when they opened the enroll link a list of groups and space left in the groups. They can then enroll into a single group.

You can define one of the groups as LTI groups. All lti users will be automatically enrolled in this group.

Students

In this panel you see an overview about all the students in a course and their submissions.

Via the actions you can change the group of a student, write an email or impersonate the user. If you click inside the table you can open the submission of the student or remove the submission of the student.

Grading

On this page you can regrade student submissions, that have been graded by an autograder already.

A word of caution: regrading should take place when you don’t expect any new submissions to a task (e.g. after the due date or when an exam has been closed).

At first you need to select a task you wish to regrade and create a new regrade template for it. The IDE with a new project derived from the master solution of the task will open up. Here you can change any file that is not writable for the students. Once you’re done, you regrade all submissions. During this step a copy of the student’s submission project is created and any non-writable file is replaced by the corresponding file of your regrade template. This project is then executed with the modifications. When regrading has completed, the new regrade will show up in the grading table.

The grading table allows you to compare the outcome of multiple gradings in detail. It lists all students and their results of default grading and as well as any regrade template you created. The heading of each regrade template allows you to open up the template again, by clicking on its name, to delete the template (and associated data) or set it as the active grading scheme for this task, which will be reflected throughout the course (e.g. in students table and on students’ exercise page). The triangle with an exclamation mark next to a template allows you to filter the table by differences in additional results compared to the currently active grading. In case submissions occurred after a regrade template was executed an indicator is shown next to the templates’ toolbar. In order to include such submissions you need to re-execute the regrade template (and perhaps delete the old one). If a student has multiple submissions, a + symbol at the left side of the row expands a table that provides details on all submissions.

Below the grading table aggregated statistics are shown that support you in comparing the outcomes of multiple gradings. The table Result Changes between Regrade Templates shows how students change in score when going from a template listed in the rows to one listed in the columns, e.g. when going from template in row i to the one in column j, cell_i,j will show how many students will in increase (green, left), stay the same (grey, middle) and decrease (red, right) in score. The Result Distribution shows boxplots of the score distribution for each regrade template.

LTI Grading

In case your course is provided on an external LTI consumer (e.g. Moodle), your students are not able to submit (for best courses, the last submit is mostly missing). Therefore you need to create the submissions here once the exam / course / exercise is over. Additionally you can also send the results to the LTI consumer if you want. Please read the integrated “How it works” documentation on the LTI Grading page of your course for details.

Live Submissions

In this table you see all submissions from your students in the course.

Analytics

You can run some analytics on your students and groups.

It sometimes might take some time (up to 60 seconds) for large courses until the analytics will be generated. Please be patient.

Backup

You can create a backup for your course.

Write Task Descriptions for Projects

In Code Expert projects you can write a task description which is displayed in the task panel on the right side of the IDE.

Flexible Task Description

To provide multiple task descriptions, which students can choose from, your project needs to include a folder with the exact name cx_description. This folder needs to include a conf.yml file and at least one markdown file.

Configuration

Inside the conf.yml you must specify your tasks descriptions. This is an example of a simple configuration.

descriptions:
 de:
   name: Deutsch
   default: true
 en:
   name: English

Inside the descriptions key you should define your individual task descriptions and how they will be displayed to the students. This example expects two description files with the names de.md and en.md in the cx_description folder. The file name is excepted to start with the key used in the conf.yml. In this example therefore en and de. The name key defines the name of the description displayed to students. Moreover, the file de.md is loaded as default task description until a student switches to the other version. It is good practice to define a default.

For this example a project structure might look like:

/
cx_description/
    conf.yml
    en.md
    de.md
main.cpp

Simple Task Description

Deprecated please do not use Simple Task Description for new projects.

Generate in the root of your project a file with the exact name DESCRIPTION.md. This file should be only writable by course administrators. If you want to hide the file in the project of the students, you can also restrict the read access to either course administrators or TAs. The description is always displayed in the task panel, independently from its permissions.

In this example a project structure might look like:

/
DESCRIPTION.md
main.cpp

Markdown Syntax

In description files you can write using markdown. See markdown Syntax reference for more details on markdown.

Math

You can write math formulas inside your markdown files. To write inline equations enclose you Latex code inside single $ signs. To write block equations use double $$.

Example

this content

Some inline formula $\sum_i i \neq 0$ some more text.

renders to

Some inline formula $\sum_i i \neq 0$ some more text.

and this content

some text
$$
\sum_i i \neq 0
$$
another text

renders to

some text $\sum_i i \neq 0$ another text

Code

Code need to be escaped with backticks (`), inline code with singe backticks and code blocks with triple backticks (```). Additional options can be passed to code blocks after the three backticks of a markdown code block.

We superimpose the following syntax: (<language>)(@(<linenumber>)).

<language>: Force <language>, do not display line numbers (git-style markdown)
@: Auto-guess the language, show line numbers starting with 1
@<linenumber>: Auto-guess the language, show line number starting with <linenumber>
<language>@: Force <language>, show line numbers starting with 1
<language>@<linenumber>: Force <language>, show line numbers starting with <linenumber>

Use ‘plain’, ‘text’, or ‘none’ to switch off syntax highlighting.

Inline code can be optionally prefixed with <language>| to define the language for syntax highlighting, otherwise it is guessed. Again, use ‘plain’, ‘text’, or ‘none’ to switch off highlighting.

Images or Files

You can drag and drop images and files you want to include in your markdown content inside the editor and they are automatically uploaded and stored into the cx_data folder.

You should only add manual links or files to files hosted on external systems. For files stored on code expert, do not manually create the link.

Setup of Projects

To setup a project for your students you should start with creating the solution. The you should switch to the template and remove code and files the students should write.

Possible File Settings

Editable

The files will be editable by the students.

Use this setting for files, the student should add their code. Or for folders, where the students should add some new files.

Editable (default file)

The file will be editable and will open of no file is selected by default. This is only possible for one file per project.

Use this setting for the main file the student should edit.

Hidden

These files will be not displayed to the students.

Use this setting for test files and internal scripts the students should not see or edit.

Read Only

These files will be displayed to the students but cannot be edited by the students.

Use this setting e.g. for header files where the student should provide the implementation. Or files which provide some api the students should use but not be able to edit it.

Custom

This let you specify the settings of the file in greater details.

There should be rarely a use case, where you need this setting.

Runner and cxEnvironments

We first need you to understand how Code Expert executes code. There are 2 applications:

The frontend app, cxweb. That is the web application, where a student or a lecturer writes his code in a project from the Code Expert IDE. A new job is written to the database when on clicks on an action button like ‘run’ or ‘test’ or when a student submits a project.
The runner or backend app, cxrun. It looks for new jobs inserted to the database, deserialize the files from the belonging project. Then it starts a cxEnvironment container (like Java or Python), mounts the projectfiles into it and passes the associated action as environment variable. Moreover cxrun tracks the container during the execution, stops it in case the user want to, parses the output for tags and updates the job state continously. During a job is executed, cxrun redirects the output to cxweb which redirects it to the users browser, all in duplex.

Code Expert is highly scalable, therefore both apps run on multiple hosts in parallel with the Docker Swarm service.

A cxEnvironment (short cxEnv) is a Docker container where the students code is executed in. Code Expert comes with some predefined cxEnvs images ready to use: Java, GCC, Python, Generic and Plain Text. If you need more or just want to have a customized one, you are free to build you own cxEnv.

Execution

A cxEnv container is always started with the ACTION environment variable set to the name of the action chosen by the user (e.g. run). The start script locates the action script and executes it. A lecturer can override the action script or command within the solution or template project configurations conf.yml (in the projects root directory):

# You can either define a cmd or a script file to be executed by uncommenting it.
# By default the action script defined in the cxEnv is executed.
cmd:
#  compile: echo "COMPILE"
#  run: echo "RUN"
#  test: echo "TEST"
#  debug: echo "DEBUG"
#  submit: echo "SUBMIT"

# Write you script somewhere in the project and fill in its path below.
script:
#  compile: /scripts/compile.sh
#  run: /scripts/run.sh
#  test: /scripts/test.sh
#  debug: /scripts/debug.sh
#  submit: /scripts/submit.sh

There is a so-called ‘testrunner’ which have been written for Java and GCC cxEnvs. This is a file in the project which is executed on all actions and runs a different programm depending on the ACTION environment variable. This file was written by lecturers and is not maintained by the Code Expert team. We are happy to help you getting in touch with its creator if you want.

Users and Permissions

cxEnv containers are run as cxrun:cxrun user. The deserialized files of a project are fully accessible to cxrun while the scripts are only readable and executable. To prevent your students to read the hidden files of the project, you can use the cxuser:cxuser (602:602) unpriviledged user. To do so, override the scripts (see Execution above) to e.g. compile as cxrun, grant cxuser access to whatever is needed (default = no access) and sudo to cxuser before executing the code.

Build Your Own

For this, you will need access to our repositories, please drop us an email to expert@inf.ethz.ch.

You best start by copying the Java 11 cxEnv.
Rename it to whatever makes sense. As a naming convention, we use a significant name followed by a dash and its major version (e.g. java-11).
Edit the Dockerfile:
- It needs to be FROM cxhub.ethz.ch/cx/cxenv/base-rhel8:latest.
- Define the name to be displayed in Code Expert (e.g. in the tasks project selector) in LABEL cxrun.name="YOUR CXENV NAME".
- Set the user to USER ${USER} and workdir to WORKDIR ${WORKDIR} (they both are defined in the cxEnv base image).
Define the action scripts:
- Simply add (or do not remove) files named to the actions your image should support (compile, run, test, debug and submit) into the scripts directory. The base image has already defined the copy job for the scripts directory.
- When starting your container with the ACTION variable set to actions it will output a list of the supported actions (docker run --env=ACTION=actions you-image).
- Each action can be overridden from the project configuration, see Execution above.
Update the .gitignore and .dockerignore, usually we ignore everything and explicitly add the files we want to track.
Ignore the deploy-docker.cfg, we will set this up for you later.
You may want to expose some configuration options to the project creators:
- All configuration is defined in the conf.yml in the root directory of the project like the action scripts themselves, see Execution above.
- Please add a short documentation of your config options in your repo at templates/conf.yml.
To read the configuration you can import ${SCRIPTDIR}/.func in your action script and use the following shell functions:
- conf() which takes two argument: the key defined in the conf.yml and a default value, in case the key is not defined or the conf.yml file doesn’t exist.
- confpath() takes the same arguments as conf() but its return value is the absolute path in the container. In conf.yml paths needs to be absolute from the projects root.
Happy scripting! While writing your scripts, you may want to:
- Mount a sample project from your file system with docker run --env=ACTION=run -v $(pwd):/var/lib/cxrun/projectfiles:rw your-image.
- Run the container interactively with docker run -ti ... you-image bash having no ACTION set and executing/debugging the script directly within the ${SCRIPTDIR}.
- Automatically remove exited container with docker run --rm ... avoiding to remove them all manually later on your system.
- Run the container as root with docker run -u root ....

cxEnvs have a wide range of ajustable limitations like CPU and memory consumption, execution timeouts, input/output length and file size and count. Please contact us at expert@inf.ethz.ch, we are happy to find out the right settings according to your needs.

All cxEnv containers are run without privileged access to the kernel. Please consider this while building your own cxEnv.

Return Results

Provide the score of a program

For the score of the submission the last occurrence of the tag <cx:result value="VALUE" /> is used. Where VALUE is expected to be a floating point number between 0 and 1.

To ensure that the student does not provide his own value a lecturer has to ensure that the test runner returns at least on <cx:result> tag at the end of the execution of the tests.

Provide additional result values

To provide additional results for a program one can use the tag <cx:results key="value" /> to return key:value pairs of result data. One can add several of the tags with different keys. If there are several <cx:results > tags with the same key the last occurrence overwrites the previous value. One can provide one or multiple key value pairs per tag. The tags need to be self enclosing and valid xml tags. If the value of the additional result is OK the result will be represented as green tag, otherwise the tag is orange.

Preliminary results

To provide preliminary results the last occurrence of the tag <cx:pre-result value="VALUE" /> is used. Where VALUE is expected to be a floating point number between 0 and 1.

The student will then only see the preliminary result, until his project was reviewed by an TA. This can be used to provide a partial result, (e.g. during exams) and then later provide the student with the correct score.

A course admin can bulk review all submissions under: My Courses > Grading > Fake Review. This can be used to show all students the final result instead of the preliminary result. This should be only used if there is no manual review process.

IDE Views

Code Expert provides several default views

Terminal View

Code Experts provide a terminal like window, which is used to interact with interactive programs. The terminal view can configured via <cx:terminal attributes /> self enclosing tags.

Attributes

mode: “raw”|“canonical”
echo: “on”|“off”|“password”

Mode raw: all keyboard inputs are sent directly to process (backend).

Mode canonical: Line editing mode where backspace removes characters and enter sends the line to backend. Navigation keys are disabled.

Echo on: All keys are echoed in the terminal view.

Echo off: All pressed keys are send to server and not echoed in the terminal view.

If raw mode is enabled echo is turned off.

HTML View

This view renders the output between <cx:html> and </cx:html> as html.

Attributes

You can pass the following attributes to the starting tag:

append If <cx:html append> is used to start an html stream the payload will be appended to the previous html content. If not provided the html view is cleared and rebuild with the new content.

HTML Sanitize

The html is sanitized before it is rendered. Therefore sanitize-html is used with the following options:

allowedTags: [ 'h3', 'h4', 'h5', 'h6', 'blockquote', 'p', 'a', 'ul', 'ol',
  'nl', 'li', 'b', 'i', 'strong', 'em', 'strike', 'code', 'hr', 'br', 'div',
  'table', 'thead', 'caption', 'tbody', 'tr', 'th', 'td', 'pre', 'img'],
allowedAttributes: {
  a: [ 'href', 'name', 'target' ],
  // We don't currently allow img itself by default, but this
  // would make sense if we did
  img: [ 'src' ]
},
// Lots of these won't come up by default because we don't allow them
selfClosing: [ 'img', 'br', 'hr', 'area', 'base', 'basefont', 'input', 'link', 'meta' ],
// URL schemes we permit
allowedSchemes: [ 'http', 'https', 'ftp', 'mailto' ],
allowedSchemesByTag: {},
allowedSchemesAppliedToAttributes: [ 'href', 'src', 'cite' ],
allowProtocolRelative: true,

For more information click here.

File View

The file view provides a list of generated files, that can be downloaded. To generate a file it is required that the file is written by the program into the folder ./cx_out. All files in this folder are uploaded to the database and stored.

Limitations

Destination: Files written to the folder cx_out from the student’s program are automatically uploaded and the <cx:file ... > string added to the end of the output.
Number of files: If more than 20 files are written all files are ignored and an error is displayed.
File size: Files with more than 2MB are ignored and an error is displayed.
Availability: All files are stored for 1h or if it is a submission forever.

Advanced usage

The file view listens to any content between <cx:file fileName=test.png fileId=123 /> and render it as a list.

Code Expert will automatically provide the <cx:file ... > strings for files which are written to the folder ./cx_out.

Attributes

You can pass the following attributes to the starting tag:

fileId The id of the file in the ProjectUploads collection.
fileName The display name for the file.
type The type of the file.
url The url via which the file can be accessed.

Moodle Integration

Code Expert can be integrated into a Moodle exam by using the question type External connection (ETH).

You should never add real exam questions on the code expert instance used for self-assessment during the semester.

Please contact us prior, if you want to generate some content used for exams. We will then explain you the required security rules you need to follow.

Current Project State

The integration is currently in project state and not freely open to all. Please contact LET in case you are interested in using Code Expert in your Moodle exams.

Exam Setup

For Online-Prüfungen-Team

Add a new course (for each exam) in Code Expert
- Go to Admin > Courses > Add Course.
- Set name to the same value as the Moodle course.
- Select Cherry-pick Mechanism, best provides a submit button to students, last does not.
- DON’T set Enrollment Open in Semesters

For Lecturers

Create a new exercise
- Go to MyCourses, select a Semester and <your course>, go to Exercises > Add Exercise.
- Choose a good name, make sure the Handout, Due and Solution Date are set to Manual - Hidden and Save.
- Expand your new exercise.
Create one or multiple tasks you want to use for your exam
- Select Add Task.
- Enter an awesome task name (the students will see the name).
- Make sure Grading is enabled, define the maximum points as needed (to simplify your correction work, the result sent to Moodle is always 0-1, the final grading points are set in Moodle).
- Select a New Project suiting your needs and Save the task.
Set up your project by clicking on the red Project link.
Once ready, Publish Solution & Template.
You’re done so far in Code Expert.
- Go to Moodle, set up your exam and the LTI questions (use this instruction sheet for details).
- Come back to Code Expert to copy the LTI Urls in the task overview by clicking on the small blue button and then paste them into Moodle.

After the Exam

For Lecturers

You need to manually grade the Code Expert results:
- Go to MyCourses, select a Semester and <your course>, go to LTI Grading and select your exercise.
- You should see your tasks and the number of student attempts next to them.
- Create New Submissions for the selected tasks and wait until all submission have been run.
- Then Send Results to LTI.
Go to the Moodle exam and Regrade all on the quiz’s results page.
Don’t forget the backup
- In Moodle (see there).
- In Code Expert by selecting the Backup tab, clicking Create Backup and downloading the archive to a safe place.

For Online-Prüfungen-Team

Do a full backup of the code expert database (sysadmin access required to host /import/cxweb/mongobkp/<current time>).

Documentation for Lecturers

Overview

My Courses

Cherry-pick Mechanism

Enroll Url

Messages

Exercises

Code Examples

Groups

Students

Grading

LTI Grading

Live Submissions

Analytics

Backup

Write Task Descriptions for Projects

Flexible Task Description

Configuration

Simple Task Description

Markdown Syntax

Math

Code

Images or Files

Setup of Projects

Possible File Settings

Editable

Editable (default file)

Hidden

Read Only

Custom

Runner and cxEnvironments

Execution

Users and Permissions

Build Your Own

Return Results

Provide the score of a program

Provide additional result values

Preliminary results

IDE Views

Terminal View

Attributes

HTML View

Attributes

HTML Sanitize

File View

Limitations

Advanced usage

Attributes

Moodle Integration

Current Project State

Exam Setup

For Online-Prüfungen-Team

For Lecturers

After the Exam

For Lecturers

For Online-Prüfungen-Team