Overview

If you want to be informed about Code Expert releases and news please sign up to our mailing list.

If you want to use Code Expert for your courses. Please use the contact form to get in contact, and we will help you set up Code Expert.

This documentation is not complete. If you have any additions or questions, please do not hesitate to contact us.

My Courses

The management of the course happens under My Courses from the main menu.

If you have more than one course you can select the course and semester in the dropdown in the top left corner.

Messages

Here you can answer messages send by students to your course. Or send a course notification to all students currently enrolled in the course.

You can disable messages for your course in the course settings.

Exercises

You can create exercises in this panel. See setup of project for more information about how to setup projects.

Code Examples

You can create code examples in this panel, which are projects that will not be reviewed.

You can disable code examples for your course in the course settings.

Groups

Typically courses are organized into groups of students, each with its own TAs. You need to add at least one group (besides the Automatic Staff Group) to your course. When students open the enroll link, they will see a list of groups and space left in each. They can then enroll into a single group.

You can define one of the groups as LTI groups. All lti users will be automatically enrolled in this group.

In the Automatic Staff Group, all TAs and course admins will be included. This allows them to access the enrollment page, without the need to manually enroll in your course.

Students

In this panel you see an overview about all the students in a course and their submissions.

Via the actions you can change the group of a student, write an email or impersonate the user. If you click inside the table you can open the submission of the student or remove the submission of the student.

Use Impersonate with care. All your actions will be executed in the context of the student until you finish impersonating.

Analytics

Here statistics about your course are displayed. This can be used to gain a deeper understanding of your course.

It sometimes might take some time (up to 60 seconds) for large courses until the analytics will be generated. Please be patient.

Group Overview

Here you can compare some indicators of your exercise groups against each other.

Group Details

Show detailed data about the different groups and tasks.

This can be useful to find out if some TA is not reviewing or the results differs from other groups.

Job Frequency

Here you can see some information about how many attempts and how much time students’ spend to solve a task. This allows to detect unusual behaviour.

Jobs per Task

This displays the distribution of the jobs for each task. This allows one to see if there is a task, where the student e.g. need lot of submission to solve it, which might indicate some problems in the task.

Highscore

This displays the score distribution for a course or a certain exercise.

This can be disabled via the course settings.

Usage

Displays the resources your course is using on our system.

Live Submissions

Here you see all submissions of your students. And the achieved results. This can be used to detect unusual behaviour.

Exam

Used to create standalone code-expert tasks.

Grading

On this page you can regrade student submissions, that have been graded by an autograder already.

A word of caution: regrading should take place when you don’t expect any new submissions to a task (e.g. after the due date or when an exam has been closed).

At first you need to select a task you wish to regrade and create a new regrade template for it. The IDE with a new project derived from the master solution of the task will open up. Here you can change any file that is not writable for the students. Once you’re done, you regrade all submissions. During this step a copy of the student’s submission project is created and any non-writable file is replaced by the corresponding file of your regrade template. This project is then executed with the modifications. When regrading has completed, the new regrade will show up in the grading table.

The grading table allows you to compare the outcome of multiple gradings in detail. It lists all students and their results of default grading and as well as any regrade template you created. The heading of each regrade template allows you to open up the template again, by clicking on its name, to delete the template (and associated data) or set it as the active grading scheme for this task, which will be reflected throughout the course (e.g. in students table and on students’ exercise page). The triangle with an exclamation mark next to a template allows you to filter the table by differences in additional results compared to the currently active grading. In case submissions occurred after a regrade template was executed an indicator is shown next to the templates’ toolbar. In order to include such submissions you need to re-execute the regrade template (and perhaps delete the old one). If a student has multiple submissions, a + symbol at the left side of the row expands a table that provides details on all submissions.

Below the grading table aggregated statistics are shown that support you in comparing the outcomes of multiple gradings. The table Result Changes between Regrade Templates shows how students change in score when going from a template listed in the rows to one listed in the columns, e.g. when going from template in row i to the one in column j, cell_i,j will show how many students will in increase (green, left), stay the same (grey, middle) and decrease (red, right) in score. The Result Distribution shows boxplots of the score distribution for each regrade template.

Moodle Integration

In case your course is provided on an external LTI consumer (e.g. Moodle), it might be that the exam is closed and your student were not able to submit their project. Therefore you need to create the submissions here once the exam / course / exercise is over. Additionally, you can also send the results to the LTI consumer if you want. Please read the integrated “How it works” documentation on the LTI Grading page of your course for details.

Settings

Here the details about a course are displayed and can be edited.

Main Admin

Set a main admin. The main admin can be used, to send messages to the students in his name.

Cherry-pick Mechanism

You can choose currently between two different modes, how submissions are automatically selected for the grading.

• last: The last submission a student submits before the due date will be automatically selected. This is the default.
• best: The best submission a student submits before the due date will be automatically selected.

An TA can always choose another submission of the student to select it for the final grading.

Default Environment

Here you can select a default environment for your course, this will be preselected on task generation.

Semesters and Enroll Url

You can select in which semesters your course is available and if students can enroll in your course. If you want students to enroll in your course share the enroll-url with them. To avoid that students can enroll in your course, you can close the enrollment.

Course Admins

Add here other admins of your course. Here you want to add TAs, which can generate exercises and tasks for your course. Or can help you to answer students messages.

Features

Here you can disable certain features, which reduce the displayed data in your course.

Analytics

If this is enabled a highscore is displayed to the students. Depending on the selected parameters different distributions will be displayed. If you want to display additional results it is required that you only choose numerical values.

Code Examples

If this is enabled the code example section is displayed for course admins and students.

Messages

If this is enabled, students can send you messages via code expert. You can also configure a hint, which is displayed to the student before they send their message.

Backup & Export

You can create a backup for your course.

Use the binary backup to store a complete backup of your course.

The export student projects can be used to export the projects of the students in a human readable way.

Write Task Descriptions for Projects

In Code Expert projects you can write a task description which is displayed in the task panel on the right side of the IDE.

Flexible Task Description

To provide multiple task descriptions, which students can choose from, your project needs to include a folder with the exact name cx_description. This folder needs to include a conf.yml file and at least one markdown file.

Configuration

Inside the conf.yml you must specify your tasks descriptions. This is an example of a simple configuration.

descriptions:
 de:
   name: Deutsch
   default: true
 en:
   name: English

Inside the descriptions key you should define your individual task descriptions and how they will be displayed to the students. This example expects two description files with the names de.md and en.md in the cx_description folder. The file name is excepted to start with the key used in the conf.yml. In this example therefore en and de. The name key defines the name of the description displayed to students. Moreover, the file de.md is loaded as default task description until a student switches to the other version. It is good practice to define a default.

For this example a project structure might look like:

/
cx_description/
    conf.yml
    en.md
    de.md
main.cpp

Simple Task Description

Deprecated please do not use Simple Task Description for new projects.

Generate in the root of your project a file with the exact name DESCRIPTION.md. This file should be only writable by course administrators. If you want to hide the file in the project of the students, you can also restrict the read access to either course administrators or TAs. The description is always displayed in the task panel, independently from its permissions.

In this example a project structure might look like:

/
DESCRIPTION.md
main.cpp

Markdown Syntax

In description files you can write using markdown. See markdown Syntax reference for more details on markdown.

Math

You can write math formulas inside your markdown files. To write inline equations enclose you Latex code inside single $ signs. To write block equations use double $$.

Example

this content

Some inline formula $\sum_i i \neq 0$ some more text.

renders to

Some inline formula $\sum_i i \neq 0$ some more text.

and this content

some text
$$
\sum_i i \neq 0
$$
another text

renders to

some text \(\sum_i i \neq 0\) another text

Code

Code need to be escaped with backticks (`), inline code with singe backticks and code blocks with triple backticks (```). Additional options can be passed to code blocks after the three backticks of a markdown code block.

We superimpose the following syntax: (<language>)(@(<linenumber>)).

• <language>: Force <language>, do not display line numbers (git-style markdown)
• @: Auto-guess the language, show line numbers starting with 1
• @<linenumber>: Auto-guess the language, show line number starting with <linenumber>
• <language>@: Force <language>, show line numbers starting with 1
• <language>@<linenumber>: Force <language>, show line numbers starting with <linenumber>

Use ‘plain’, ‘text’, or ‘none’ to switch off syntax highlighting.

Inline code can be optionally prefixed with <language>| to define the language for syntax highlighting, otherwise it is guessed. Again, use ‘plain’, ‘text’, or ‘none’ to switch off highlighting.

Images or Files

You can drag and drop images and files you want to include in your markdown content inside the editor and they are automatically uploaded and stored into the cx_data folder.

You should only add manual links or files to files hosted on external systems. For files stored on code expert, do not manually create the link.

Setup of Projects

To setup a project for your students you should start with creating the solution. The you should switch to the template and remove code and files the students should write.

Possible File Settings

Editable

The files will be editable by the students.

Use this setting for files, the student should add their code. Or for folders, where the students should add some new files.

Editable (default file)

The file will be editable and will open of no file is selected by default. This is only possible for one file per project.

Use this setting for the main file the student should edit.

Hidden

These files will be not displayed to the students.

Use this setting for test files and internal scripts the students should not see or edit.

Read Only

These files will be displayed to the students but cannot be edited by the students.

Use this setting e.g. for header files where the student should provide the implementation. Or files which provide some api the students should use but not be able to edit it.

Custom

This let you specify the settings of the file in greater details.

There should be rarely a use case, where you need this setting.

Create files on runtime

I you want to generate files on runtime you need to add a conf.yml file to your project root.

UseCases

You can use this feature for the following useCases:

• Your test runner can create some additional files, during submissions which can be viewed by assistants only. Allowing then to gain a better understanding why the program was not successful.
• You can create some additional output in your test runner, which can be shown to students to allow them to better understand why their program crashed.

Settings

Add the following settings for runtimeFileGeneration to your root conf.yml file:

runtimeFileGeneration:
  enabled: true | false
  path: /^(\w|-| |_){1,80}$/
  visible: 'everyone' | 'student' | 'assistant' | 'admin'

The following are the default parameters:

runtimeFileGeneration:
  enabled: false
  path: 'cx_audit'
  visible: 'assistant' 

With the parameter enabled you decide if you want to enable the generation of files during runtime. If enabled you can write your files into the folder defined by the parameter path. With the last parameter visible you can define who has read access to the folder.

Behaviour

There is a limit of 5 files and with each 16kB which can be added to the project.

If you publish a project it checks if it includes one of the runtimeFileGeneration directories and asks you to take some actions (either publish it or delete it before publishing).

Runner and cxEnvironments

We first need you to understand how Code Expert executes code. There are 2 applications:

1. The frontend app, cxweb. That is the web application, where a student or a lecturer writes his code in a project from the Code Expert IDE. A new job is written to the database when on clicks on an action button like ‘run’ or ‘test’ or when a student submits a project.
2. The runner or backend app, cxrun. It looks for new jobs inserted to the database, deserialize the files from the belonging project. Then it starts a cxEnvironment container (like Java or Python), mounts the projectfiles into it and passes the associated action as environment variable. Moreover cxrun tracks the container during the execution, stops it in case the user want to, parses the output for tags and updates the job state continously. During a job is executed, cxrun redirects the output to cxweb which redirects it to the users browser, all in duplex.

Code Expert is highly scalable, therefore both apps run on multiple hosts in parallel with the Docker Swarm service.

A cxEnvironment (short cxEnv) is a Docker container where the students code is executed in. Code Expert comes with some predefined cxEnvs images ready to use: Java, GCC, Python, Generic and Plain Text. If you need more or just want to have a customized one, you are free to build you own cxEnv.

Execution

A cxEnv container is always started with the ACTION environment variable set to the name of the action chosen by the user (e.g. run). The start script locates the action script and executes it. A lecturer can override the action script or command within the solution or template project configurations conf.yml (in the projects root directory):

# You can either define a cmd or a script file to be executed by uncommenting it.
# By default the action script defined in the cxEnv is executed.
cmd:
#  compile: echo "COMPILE"
#  run: echo "RUN"
#  test: echo "TEST"
#  debug: echo "DEBUG"
#  submit: echo "SUBMIT"

# Write you script somewhere in the project and fill in its path below.
script:
#  compile: /scripts/compile.sh
#  run: /scripts/run.sh
#  test: /scripts/test.sh
#  debug: /scripts/debug.sh
#  submit: /scripts/submit.sh

There is a so-called ‘testrunner’ which have been written for Java and GCC cxEnvs. This is a file in the project which is executed on all actions and runs a different programm depending on the ACTION environment variable. This file was written by lecturers and is not maintained by the Code Expert team. We are happy to help you getting in touch with its creator if you want.

Environment Variables

Decide which file are compiled

If one wants to specify for each phase (run/test/submit) which source files are compiled. One needs to overwrite the compile, run, and test script and decide in each script which files should be compiled by the compiler.

Users and Permissions

cxEnv containers are run as cxrun:cxrun user. The deserialized files of a project are fully accessible to cxrun while the scripts are only readable and executable. To prevent your students to read the hidden files of the project, you can use the cxuser:cxuser (602:602) unpriviledged user. To do so, override the scripts (see Execution above) to e.g. compile as cxrun, grant cxuser access to whatever is needed (default = no access) and sudo to cxuser before executing the code.

An example project for gcc can be downloaded here (You can import this direct in the ide). In this project, the run.sh script is changed to allow the student to only access the visible.txt and the compiled application. The secret.txt and secret_code.h is not accessible by the student. However, the method in printSecret in secret_code.h can be accessed by the code of the student. See the main.cpp file for details.

Build Your Own

For this, you will need access to our repositories, please drop us an email to expert@inf.ethz.ch.

1. You best start by copying the Java 11 cxEnv.
2. Rename it to whatever makes sense. As a naming convention, we use a significant name followed by a dash and its major version (e.g. java-11).
3. Edit the Dockerfile:
   • It needs to be FROM cxhub.ethz.ch/cx/cxenv/base-rhel8:latest.
   • Define the name to be displayed in Code Expert (e.g. in the tasks project selector) in LABEL cxrun.name="YOUR CXENV NAME".
   • Set the user to USER ${USER} and workdir to WORKDIR ${WORKDIR} (they both are defined in the cxEnv base image).
4. Define the action scripts:
   • Simply add (or do not remove) files named to the actions your image should support (compile, run, test, debug and submit) into the scripts directory. The base image has already defined the copy job for the scripts directory.
   • When starting your container with the ACTION variable set to actions it will output a list of the supported actions (docker run --env=ACTION=actions you-image).
   • Each action can be overridden from the project configuration, see Execution above.
5. Update the .gitignore and .dockerignore, usually we ignore everything and explicitly add the files we want to track.
6. Ignore the deploy-docker.cfg, we will set this up for you later.
7. You may want to expose some configuration options to the project creators:
   • All configuration is defined in the conf.yml in the root directory of the project like the action scripts themselves, see Execution above.
   • Please add a short documentation of your config options in your repo at templates/conf.yml.
8. To read the configuration you can import ${SCRIPTDIR}/.func in your action script and use the following shell functions:
   • conf() which takes two argument: the key defined in the conf.yml and a default value, in case the key is not defined or the conf.yml file doesn’t exist.
   • confpath() takes the same arguments as conf() but its return value is the absolute path in the container. In conf.yml paths needs to be absolute from the projects root.
9. Happy scripting! While writing your scripts, you may want to:
   • Mount a sample project from your file system with docker run --env=ACTION=run -v $(pwd):/var/lib/cxrun/projectfiles:rw your-image.
   • Run the container interactively with docker run -ti ... you-image bash having no ACTION set and executing/debugging the script directly within the ${SCRIPTDIR}.
   • Automatically remove exited container with docker run --rm ... avoiding to remove them all manually later on your system.
   • Run the container as root with docker run -u root ....

cxEnvs have a wide range of ajustable limitations like CPU and memory consumption, execution timeouts, input/output length and file size and count. Please contact us at expert@inf.ethz.ch, we are happy to find out the right settings according to your needs.

All cxEnv containers are run without privileged access to the kernel. Please consider this while building your own cxEnv.

Return Results

Provide the score of a program

For the score of the submission the last occurrence of the tag <cx:result value="VALUE" /> is used. Where VALUE is expected to be a floating point number between 0 and 1.

To ensure that the student does not provide his own value a lecturer has to ensure that the test runner returns at least on <cx:result> tag at the end of the execution of the tests.

Provide additional result values

To provide additional results for a program one can use the tag <cx:results key="value" /> to return key:value pairs of result data. One can add several of the tags with different keys. If there are several <cx:results > tags with the same key the last occurrence overwrites the previous value. One can provide one or multiple key value pairs per tag. The tags need to be self enclosing and valid xml tags. If the value of the additional result is OK the result will be represented as green tag, otherwise the tag is orange.

Preliminary results

To provide preliminary results the last occurrence of the tag <cx:pre-result value="VALUE" /> is used. Where VALUE is expected to be a floating point number between 0 and 1.

The student will then only see the preliminary result, until his project was reviewed by an TA. This can be used to provide a partial result, (e.g. during exams) and then later provide the student with the correct score.

A course admin can bulk review all submissions under: My Courses > Grading > Fake Review. This can be used to show all students the final result instead of the preliminary result. This should be only used if there is no manual review process.

IDE Views

Code Expert provides several default views.

In exam mode due to security concerns only the terminal view is available.

Terminal View

Code Experts provide a terminal like window, which is used to interact with interactive programs. It will work as a normal linux terminal.

The terminal font only supports some unicode characters (see here). Glyphs not supported by the font might be displayed different on different browsers and systems, or might be even not available.

HTML View

This view renders the output between <cx:html> and </cx:html> as html.

Attributes

You can pass the following attributes to the starting tag:

• append If <cx:html append> is used to start an html stream the payload will be appended to the previous html content. If not provided the html view is cleared and rebuild with the new content.

HTML Security

The html is rendered in a sandboxed iframe.

File View

The file view provides a list of generated files, that can be downloaded. To generate a file it is required that the file is written by the program into the folder ./cx_out. All files in this folder are uploaded to the database and stored.

Limitations

Destination

Files written to the folder cx_out from the student’s program are automatically uploaded and the <cx:file ... > string added to the end of the output.

Number of files

If more than 20 files are written all files are ignored and an error is displayed.

File size

Files with more than 2MB are ignored and an error is displayed.

Availability

All files are stored for 1h or if it is a submission forever.

Advanced usage

The file view listens to any content between <cx:file fileName=test.png fileId=123 /> and render it as a list.

Code Expert will automatically provide the <cx:file ... > strings for files which are written to the folder ./cx_out.

Attributes

You can pass the following attributes to the starting tag:

• fileId The id of the file in the ProjectUploads collection.
• fileName The display name for the file.
• type The type of the file.
• url The url via which the file can be accessed.

Moodle Integration

Code Expert can be integrated into a Moodle exam by using the question type External connection (ETH).

We only support the integration of Code Expert into moodle for exams. For the regular exercises, Code Expert should be used as standalone product, without moodle.

You should never add real exam questions on the code expert instance used for self-assessment during the semester.

Please contact us prior, if you want to generate some content used for exams. We will then explain you the required security rules you need to follow.

Current Project State

The integration is currently in project state and not freely open to all. Please contact LET in case you are interested in using Code Expert in your Moodle exams.

Exam Setup

For Online-Prüfungen-Team

• Add a new course (for each exam) in Code Expert
  • Go to Admin > Courses > Add Course.
  • Set name to the same value as the Moodle course.
  • Select Cherry-pick Mechanism, best provides a submit button to students, last does not.
  • DON’T set Enrollment Open in Semesters

For Lecturers

• Create a new exercise
  • Go to MyCourses, select a Semester and <your course>, go to Exercises > Add Exercise.
  • Choose a good name, make sure the Handout, Due and Solution Date are set to Manual - Hidden and Save. Note: if you want to allow access to the master solution of an exercise during review, the solution status needs to be set to Manual - published or Automatic - Hidden until handout date with an appropriate date configured.
  • Expand your new exercise.
• Create one or multiple tasks you want to use for your exam
  • Select Add Task.
  • Enter an awesome task name (the students will see the name).
  • Make sure Grading is enabled, define the maximum points as needed (to simplify your correction work, the result sent to Moodle is always 0-1, the final grading points are set in Moodle).
  • Select a New Project suiting your needs and Save the task.
• Set up your project by clicking on the red Project link.
• Once ready, Publish Solution & Template.
• You’re done so far in Code Expert.
  • Go to Moodle, set up your exam and the LTI questions (use this instruction sheet for details).
  • Come back to Code Expert to copy the LTI Urls in the task overview. They appear when you hover over the next to Project from where you can copy them by clicking on and then paste them into Moodle.

After the Exam

For Lecturers

If you’re using Code Expert for mock exams running on an instance not in exam mode:

• Student submissions are generated automatically when student review their test
• LTI Grading is not available, Results are not sent to Moodle

If you’re using Code Expert for real exams running on an instance in exam mode:

• You need to manually grade the Code Expert results:
  • Go to MyCourses, select a Semester and <your course>, go to LTI Grading and select your exercise.
  • You should see your tasks and the number of student attempts next to them.
  • Create New Submissions for the selected tasks and wait until all submission have been run.
  • Then Send Results to LTI.
• Go to the Moodle exam and Regrade all on the quiz’s results page.
• Don’t forget the backup
  • In Moodle (see there).
  • In Code Expert by selecting the Backup tab, clicking Create Backup and downloading the archive to a safe place.

For Online-Prüfungen-Team

• Do a full backup of the code expert database (sysadmin access required to host /import/cxweb/mongobkp/<current time>).

FAQ

How to give XP to students that did not submit anything for a task

If you want to give a student XP if the student did not submit anything for a task you need:

1. Impersonate the student from the student table
2. Create a submission for the task.
3. Finish impersonating and cherry pick and mark that task via the review panel.

Then the student should get the XP points.