Overview
If you want to be informed about Code Expert releases and news please sign up to our mailing list.
If you want to use Code Expert for your courses. Please use the contact form to get in contact, and we will help you set up Code Expert.
This documentation is not complete. If you have any additions or questions, please do not hesitate to contact us.
My Courses
The management of the course happens under My Courses from the main menu.
If you have more than one course you can select the course and semester in the dropdown in the top left corner.
Messages
Here you can answer messages send by students to your course. Or send a course notification to all students currently enrolled in the course.
You can disable messages for your course in the course settings.
Exercises
You can create exercises in this panel. See setup of project for more information about how to setup projects.
Code Examples
You can create code examples in this panel, which are projects that will not be reviewed.
You can disable code examples for your course in the course settings.
Groups
Typically courses are organized into groups of students, each with its own TAs. You need to add at least one group (besides the Automatic Staff Group) to your course. When students open the enroll link, they will see a list of groups and space left in each. They can then enroll into a single group.
You can define one of the groups as LTI groups. All lti users will be automatically enrolled in this group.
In the Automatic Staff Group, all TAs and course admins will be included. This allows them to access the enrollment page, without the need to manually enroll in your course.
Students
In this panel you see an overview about all the students in a course and their submissions.
Via the actions you can change the group of a student, write an email or impersonate the user. If you click inside the table you can open the submission of the student or remove the submission of the student.
Use Impersonate with care. All your actions will be executed in the context of the student until you finish impersonating.
Analytics
Here statistics about your course are displayed. This can be used to gain a deeper understanding of your course.
It sometimes might take some time (up to 60 seconds) for large courses until the analytics will be generated. Please be patient.
Group Overview
Here you can compare some indicators of your exercise groups against each other.
Group Details
Show detailed data about the different groups and tasks.
This can be useful to find out if some TA is not reviewing or the results differs from other groups.
Job Frequency
Here you can see some information about how many attempts and how much time students’ spend to solve a task. This allows to detect unusual behaviour.
Jobs per Task
This displays the distribution of the jobs for each task. This allows one to see if there is a task, where the student e.g. need lot of submission to solve it, which might indicate some problems in the task.
Highscore
This displays the score distribution for a course or a certain exercise.
This can be disabled via the course settings.
Usage
Displays the resources your course is using on our system.
Live Submissions
Here you see all submissions of your students. And the achieved results. This can be used to detect unusual behaviour.
Exam
Used to create standalone code-expert tasks.
Grading
On this page you can regrade student submissions, that have been graded by an autograder already.
A word of caution: regrading should take place when you don’t expect any new submissions to a task (e.g. after the due date or when an exam has been closed).
At first you need to select a task you wish to regrade and create a new regrade template for it. The IDE with a new project derived from the master solution of the task will open up. Here you can change any file that is not writable for the students. Once you’re done, you regrade all submissions. During this step a copy of the student’s submission project is created and any non-writable file is replaced by the corresponding file of your regrade template. This project is then executed with the modifications. When regrading has completed, the new regrade will show up in the grading table.
The grading table allows you to compare the outcome of multiple gradings in detail. It lists all students and their results of default grading and as well as any regrade template you created. The heading of each regrade template allows you to open up the template again, by clicking on its name, to delete the template (and associated data) or set it as the active grading scheme for this task, which will be reflected throughout the course (e.g. in students table and on students’ exercise page). The triangle with an exclamation mark next to a template allows you to filter the table by differences in additional results compared to the currently active grading. In case submissions occurred after a regrade template was executed an indicator is shown next to the templates’ toolbar. In order to include such submissions you need to re-execute the regrade template (and perhaps delete the old one). If a student has multiple submissions, a + symbol at the left side of the row expands a table that provides details on all submissions.
Below the grading table aggregated statistics are shown that support you in comparing the outcomes of multiple gradings. The table Result Changes between Regrade Templates shows how students change in score when going from a template listed in the rows to one listed in the columns, e.g. when going from template in row i to the one in column j, cell_i,j will show how many students will in increase (green, left), stay the same (grey, middle) and decrease (red, right) in score. The Result Distribution shows boxplots of the score distribution for each regrade template.
Moodle Integration
In case your course is provided on an external LTI consumer (e.g. Moodle), it might be that the exam is closed and your student were not able to submit their project. Therefore you need to create the submissions here once the exam / course / exercise is over. Additionally, you can also send the results to the LTI consumer if you want. Please read the integrated “How it works” documentation on the LTI Grading page of your course for details.
Settings
Here the details about a course are displayed and can be edited.
Main Admin
Set a main admin. The main admin can be used, to send messages to the students in his name.
Cherry-pick Mechanism
You can choose currently between two different modes, how submissions are automatically selected for the grading.
- last: The last submission a student submits before the due date will be automatically selected. This is the default.
- best: The best submission a student submits before the due date will be automatically selected.
An TA can always choose another submission of the student to select it for the final grading.
Default Environment
Here you can select a default environment for your course, this will be preselected on task generation.
Semesters and Enroll Url
You can select in which semesters your course is available and if students can enroll in your course. If you want students to enroll in your course share the enroll-url with them. To avoid that students can enroll in your course, you can close the enrollment.
Course Admins
Add here other admins of your course. Here you want to add TAs, which can generate exercises and tasks for your course. Or can help you to answer students messages.
Features
Here you can disable certain features, which reduce the displayed data in your course.
Analytics
If this is enabled a highscore is displayed to the students. Depending on the selected parameters different distributions will be displayed. If you want to display additional results it is required that you only choose numerical values.
Code Examples
If this is enabled the code example section is displayed for course admins and students.
Messages
If this is enabled, students can send you messages via code expert. You can also configure a hint, which is displayed to the student before they send their message.
Backup & Export
You can create a backup for your course.
Use the binary backup to store a complete backup of your course.
The export student projects can be used to export the projects of the students in a human readable way.
Write Task Descriptions for Projects
In Code Expert projects you can write a task description which is displayed in the task panel on the right side of the IDE.
Flexible Task Description
To provide multiple task descriptions, which students can choose from, your project needs to include a folder with the exact name cx_description
. This folder needs to include a conf.yml
file and at least one markdown file.
Configuration
Inside the conf.yml
you must specify your tasks descriptions. This is an example of a simple configuration.
descriptions:
de:
name: Deutsch
default: true
en:
name: English
Inside the descriptions
key you should define your individual task descriptions and how they will be displayed to the students. This example expects two description files with the names de.md
and en.md
in the cx_description
folder. The file name is excepted to start with the key used in the conf.yml
. In this example therefore en
and de
. The name
key defines the name of the description displayed to students. Moreover, the file de.md
is loaded as default task description until a student switches to the other version. It is good practice to define a default.
For this example a project structure might look like:
/
cx_description/
conf.yml
en.md
de.md
main.cpp
Simple Task Description
Deprecated please do not use Simple Task Description for new projects.
Generate in the root of your project a file with the exact name DESCRIPTION.md
. This file should be only writable by course administrators. If you want to hide the file in the project of the students, you can also restrict the read access to either course administrators or TAs. The description is always displayed in the task panel, independently from its permissions.
In this example a project structure might look like:
/
DESCRIPTION.md
main.cpp
Markdown Syntax
In description files you can write using markdown. See markdown Syntax reference for more details on markdown.
Math
You can write math formulas inside your markdown files. To write inline equations enclose you Latex code inside single $
signs. To write block equations use double $$
.
Example
this content
Some inline formula $\sum_i i \neq 0$ some more text.
renders to
Some inline formula $\sum_i i \neq 0$ some more text.
and this content
some text
$$
\sum_i i \neq 0
$$
another text
renders to
some text \(\sum_i i \neq 0\) another text
Code
Code need to be escaped with backticks (`), inline code with singe backticks and code blocks with triple backticks (```). Additional options can be passed to code blocks after the three backticks of a markdown code block.
We superimpose the following syntax: (<language>)(@(<linenumber>))
.
<language>
: Force<language>
, do not display line numbers (git-style markdown)@
: Auto-guess the language, show line numbers starting with 1@<linenumber>
: Auto-guess the language, show line number starting with<linenumber>
<language>@
: Force<language>
, show line numbers starting with 1<language>@<linenumber>
: Force<language>
, show line numbers starting with<linenumber>
Use ‘plain’, ‘text’, or ‘none’ to switch off syntax highlighting.
Inline code can be optionally prefixed with <language>|
to define the language for syntax highlighting, otherwise it is guessed. Again, use ‘plain’, ‘text’, or ‘none’ to switch off highlighting.
Images or Files
You can drag and drop images and files you want to include in your markdown content inside the editor and they are automatically uploaded and stored into the cx_data
folder.
You should only add manual links or files to files hosted on external systems. For files stored on code expert, do not manually create the link.
Setup of Projects
To setup a project for your students you should start with creating the solution. The you should switch to the template and remove code and files the students should write.
Possible File Settings
Editable
The files will be editable by the students.
Use this setting for files, the student should add their code. Or for folders, where the students should add some new files.
Editable (default file)
The file will be editable and will open of no file is selected by default. This is only possible for one file per project.
Use this setting for the main file the student should edit.
Hidden
These files will be not displayed to the students.
Use this setting for test files and internal scripts the students should not see or edit.
Read Only
These files will be displayed to the students but cannot be edited by the students.
Use this setting e.g. for header files where the student should provide the implementation. Or files which provide some api the students should use but not be able to edit it.
Custom
This let you specify the settings of the file in greater details.
There should be rarely a use case, where you need this setting.
Create files on runtime
I you want to generate files on runtime you need to add a conf.yml
file to your project root.
UseCases
You can use this feature for the following useCases:
- Your test runner can create some additional files, during submissions which can be viewed by assistants only. Allowing then to gain a better understanding why the program was not successful.
- You can create some additional output in your test runner, which can be shown to students to allow them to better understand why their program crashed.
Settings
Add the following settings for runtimeFileGeneration to your root conf.yml
file:
runtimeFileGeneration:
enabled: true | false
path: /^(\w|-| |_){1,80}$/
visible: 'everyone' | 'student' | 'assistant' | 'admin'
The following are the default parameters:
runtimeFileGeneration:
enabled: false
path: 'cx_audit'
visible: 'assistant'
With the parameter enabled
you decide if you want to enable the generation of files during runtime.
If enabled you can write your files into the folder defined by the parameter path
. With the last
parameter visible
you can define who has read access to the folder.
Behaviour
There is a limit of 5 files and with each 16kB which can be added to the project.
If you publish a project it checks if it includes one of the runtimeFileGeneration directories and asks you to take some actions (either publish it or delete it before publishing).
Runner and cxEnvironments
We first need you to understand how Code Expert executes code. There are 2 applications:
- The frontend app, cxweb. That is the web application, where a student or a lecturer writes his code in a project from the Code Expert IDE. A new job is written to the database when on clicks on an action button like ‘run’ or ‘test’ or when a student submits a project.
- The runner or backend app, cxrun. It looks for new jobs inserted to the database, deserialize the files from the belonging project. Then it starts a cxEnvironment container (like Java or Python), mounts the projectfiles into it and passes the associated action as environment variable. Moreover cxrun tracks the container during the execution, stops it in case the user want to, parses the output for tags and updates the job state continously. During a job is executed, cxrun redirects the output to cxweb which redirects it to the users browser, all in duplex.
Code Expert is highly scalable, therefore both apps run on multiple hosts in parallel with the Docker Swarm service.
A cxEnvironment (short cxEnv) is a Docker container where the students code is executed in. Code Expert comes with some predefined cxEnvs images ready to use: Java, GCC, Python, Generic and Plain Text. If you need more or just want to have a customized one, you are free to build you own cxEnv.
Execution
A cxEnv container is always started with the ACTION
environment variable set to the name of the action chosen by the user (e.g. run
). The start script locates the action script and executes it. A lecturer can override the action script or command within the solution or template project configurations conf.yml
(in the projects root directory):
# You can either define a cmd or a script file to be executed by uncommenting it.
# By default the action script defined in the cxEnv is executed.
cmd:
# compile: echo "COMPILE"
# run: echo "RUN"
# test: echo "TEST"
# debug: echo "DEBUG"
# submit: echo "SUBMIT"
# Write you script somewhere in the project and fill in its path below.
script:
# compile: /scripts/compile.sh
# run: /scripts/run.sh
# test: /scripts/test.sh
# debug: /scripts/debug.sh
# submit: /scripts/submit.sh
There is a so-called ‘testrunner’ which have been written for Java and GCC cxEnvs. This is a file in the project which is executed on all actions and runs a different programm depending on the ACTION
environment variable. This file was written by lecturers and is not maintained by the Code Expert team. We are happy to help you getting in touch with its creator if you want.
Environment Variables
NAME | Description | Values | Additional Information |
ACTION | The current type of jobs, defines which scripts are executed. | test , submit , run , debug , compile , repl |
always set |
USECASE | The useCase of the project. | environmentTemplate , masterSolution , studentTemplate , studentAttempt , submissionFeedback , standalone |
is not set for submit jobs |
USERROLE | The role of the user executing the job. | admin , assistant , student , everyone |
is not set for submit jobs |
CPUTIME | The maximum cpu time the Job can use. | time in sec | always set |
MAXFILESIZE | The maximum file size which can be created in the container. | size in kb | always set |
TIMEOUT | The maximum run time the container can run. | time in sec | always set |
Decide which file are compiled
If one wants to specify for each phase (run/test/submit) which source files are compiled. One needs to overwrite the compile, run, and test script and decide in each script which files should be compiled by the compiler.
Users and Permissions
cxEnv containers are run as cxrun:cxrun
user. The deserialized files of a project are fully accessible to cxrun
while the scripts are only readable and executable. To prevent your students to read the hidden files of the project, you can use the cxuser:cxuser (602:602)
unpriviledged user. To do so, override the scripts (see Execution above) to e.g. compile as cxrun
, grant cxuser
access to whatever is needed (default = no access) and sudo
to cxuser
before executing the code.
An example project for gcc can be downloaded here (You can import this direct in the ide). In this project, the run.sh
script is changed to allow the student to only access the visible.txt
and the compiled application. The secret.txt
and secret_code.h
is not accessible by the student. However, the method in printSecret
in secret_code.h
can be accessed by the code of the student. See the main.cpp
file for details.
Build Your Own
For this, you will need access to our repositories, please drop us an email to expert@inf.ethz.ch.
- You best start by copying the Java 11 cxEnv.
- Rename it to whatever makes sense. As a naming convention, we use a significant name followed by a dash and its major version (e.g.
java-11
). - Edit the Dockerfile:
- It needs to be
FROM cxhub.ethz.ch/cx/cxenv/base-rhel8:latest
. - Define the name to be displayed in Code Expert (e.g. in the tasks project selector) in
LABEL cxrun.name="YOUR CXENV NAME"
. - Set the user to
USER ${USER}
and workdir toWORKDIR ${WORKDIR}
(they both are defined in the cxEnv base image).
- It needs to be
- Define the action scripts:
- Simply add (or do not remove) files named to the actions your image should support (
compile
,run
,test
,debug
andsubmit
) into thescripts
directory. The base image has already defined the copy job for the scripts directory. - When starting your container with the
ACTION
variable set toactions
it will output a list of the supported actions (docker run --env=ACTION=actions you-image
). - Each action can be overridden from the project configuration, see Execution above.
- Simply add (or do not remove) files named to the actions your image should support (
- Update the
.gitignore
and.dockerignore
, usually we ignore everything and explicitly add the files we want to track. - Ignore the
deploy-docker.cfg
, we will set this up for you later. - You may want to expose some configuration options to the project creators:
- All configuration is defined in the
conf.yml
in the root directory of the project like the action scripts themselves, see Execution above. - Please add a short documentation of your config options in your repo at
templates/conf.yml
.
- All configuration is defined in the
- To read the configuration you can import
${SCRIPTDIR}/.func
in your action script and use the following shell functions:conf()
which takes two argument: the key defined in theconf.yml
and a default value, in case the key is not defined or theconf.yml
file doesn’t exist.confpath()
takes the same arguments asconf()
but its return value is the absolute path in the container. Inconf.yml
paths needs to be absolute from the projects root.
- Happy scripting! While writing your scripts, you may want to:
- Mount a sample project from your file system with
docker run --env=ACTION=run -v $(pwd):/var/lib/cxrun/projectfiles:rw your-image
. - Run the container interactively with
docker run -ti ... you-image bash
having noACTION
set and executing/debugging the script directly within the${SCRIPTDIR}
. - Automatically remove exited container with
docker run --rm ...
avoiding to remove them all manually later on your system. - Run the container as root with
docker run -u root ...
.
- Mount a sample project from your file system with
cxEnvs have a wide range of ajustable limitations like CPU and memory consumption, execution timeouts, input/output length and file size and count. Please contact us at expert@inf.ethz.ch, we are happy to find out the right settings according to your needs.
All cxEnv containers are run without privileged access to the kernel. Please consider this while building your own cxEnv.
Return Results
Provide the score of a program
For the score of the submission the last occurrence of the tag <cx:result value="VALUE" />
is used.
Where VALUE is expected to be a floating point number between 0 and 1.
To ensure that the student does not provide his own value a lecturer has to ensure that the test
runner returns at least on <cx:result>
tag at the end of the execution of the tests.
Provide additional result values
To provide additional results for a program one can use the tag <cx:results key="value" />
to
return key:value pairs of result data. One can add several of the tags with different keys.
If there are several <cx:results >
tags with the same key the last occurrence overwrites the previous
value. One can provide one or multiple key value pairs per tag. The tags need to be self enclosing and valid xml tags.
If the value of the additional result is OK the result will be represented as green tag, otherwise the tag is orange.
Preliminary results
To provide preliminary results the last occurrence of the tag <cx:pre-result value="VALUE" />
is used.
Where VALUE is expected to be a floating point number between 0 and 1.
The student will then only see the preliminary result, until his project was reviewed by an TA. This can be used to provide a partial result, (e.g. during exams) and then later provide the student with the correct score.
A course admin can bulk review all submissions under: My Courses > Grading > Fake Review. This can be used to show all students the final result instead of the preliminary result. This should be only used if there is no manual review process.
IDE Views
Code Expert provides several default views.
In exam mode due to security concerns only the terminal view is available.
Terminal View
Code Experts provide a terminal like window, which is used to interact with interactive programs. It will work as a normal linux terminal.
The terminal font only supports some unicode characters (see here). Glyphs not supported by the font might be displayed different on different browsers and systems, or might be even not available.
HTML View
This view renders the output between <cx:html>
and </cx:html>
as html.
Attributes
You can pass the following attributes to the starting tag:
append
If<cx:html append>
is used to start an html stream the payload will be appended to the previous html content. If not provided the html view is cleared and rebuild with the new content.
HTML Security
The html is rendered in a sandboxed iframe.
File View
The file view provides a list of generated files, that can be downloaded.
To generate a file it is required that the file is written by the program into the folder ./cx_out
.
All files in this folder are uploaded to the database and stored.
Limitations
- Destination
- Files written to the folder
cx_out
from the student’s program are automatically uploaded and the<cx:file ... >
string added to the end of the output. - Number of files
- If more than 20 files are written all files are ignored and an error is displayed.
- File size
- Files with more than 2MB are ignored and an error is displayed.
- Availability
- All files are stored for 1h or if it is a submission forever.
Advanced usage
The file view listens to any content between <cx:file fileName=test.png fileId=123 />
and render it as a list.
Code Expert will automatically provide the <cx:file ... >
strings for files which are written to the folder ./cx_out
.
Attributes
You can pass the following attributes to the starting tag:
fileId
The id of the file in the ProjectUploads collection.fileName
The display name for the file.type
The type of the file.url
The url via which the file can be accessed.
Moodle Integration
Code Expert can be integrated into a Moodle exam by using the question type External connection (ETH)
.
We only support the integration of Code Expert into moodle for exams. For the regular exercises, Code Expert should be used as standalone product, without moodle.
You should never add real exam questions on the code expert instance used for self-assessment during the semester.
Please contact us prior, if you want to generate some content used for exams. We will then explain you the required security rules you need to follow.
Current Project State
The integration is currently in project state and not freely open to all. Please contact LET in case you are interested in using Code Expert in your Moodle exams.
Exam Setup
For Online-Prüfungen-Team
- Add a new course (for each exam) in Code Expert
- Go to
Admin > Courses > Add Course
. - Set
name
to the same value as the Moodle course. - Select
Cherry-pick Mechanism
,best
provides a submit button to students,last
does not. - DON’T set
Enrollment Open in Semesters
- Go to
For Lecturers
- Create a new exercise
- Go to
MyCourses
, select aSemester
and<your course>
, go toExercises
>Add Exercise
. - Choose a good
name
, make sure theHandout, Due and Solution Date
are set toManual - Hidden
andSave
. Note: if you want to allow access to the master solution of an exercise during review, the solution status needs to be set toManual - published
orAutomatic - Hidden until handout date
with an appropriate date configured. - Expand your new exercise.
- Go to
- Create one or multiple tasks you want to use for your exam
- Select
Add Task
. - Enter an awesome task name (the students will see the name).
- Make sure
Grading
is enabled, define the maximum points as needed (to simplify your correction work, the result sent to Moodle is always 0-1, the final grading points are set in Moodle). - Select a
New Project
suiting your needs andSave
the task.
- Select
- Set up your project by clicking on the red Project link.
- Once ready,
Publish Solution & Template
. - You’re done so far in Code Expert.
- Go to Moodle, set up your exam and the LTI questions (use this instruction sheet for details).
- Come back to Code Expert to copy the LTI Urls in the task overview. They appear when you hover over the next to Project from where you can copy them by clicking on and then paste them into Moodle.
After the Exam
For Lecturers
If you’re using Code Expert for mock exams running on an instance not in exam mode:
- Student submissions are generated automatically when student review their test
LTI Grading
is not available, Results are not sent to Moodle
If you’re using Code Expert for real exams running on an instance in exam mode:
- You need to manually grade the Code Expert results:
- Go to
MyCourses
, select aSemester
and<your course>
, go toLTI Grading
and select your exercise. - You should see your tasks and the number of student attempts next to them.
Create New Submissions
for the selected tasks and wait until all submission have been run.- Then
Send Results to LTI
.
- Go to
- Go to the Moodle exam and
Regrade all
on the quiz’s results page. - Don’t forget the backup
- In Moodle (see there).
- In Code Expert by selecting the
Backup
tab, clickingCreate Backup
and downloading the archive to a safe place.
For Online-Prüfungen-Team
- Do a full backup of the code expert database (sysadmin access required to host
/import/cxweb/mongobkp/<current time>
).
FAQ
How to give XP to students that did not submit anything for a task
If you want to give a student XP if the student did not submit anything for a task you need:
- Impersonate the student from the student table
- Create a submission for the task.
- Finish impersonating and cherry pick and mark that task via the review panel.
Then the student should get the XP points.