Reference

Aga grades assignments: a library for easily producing autograders for code.

Anything not explicitly documented here should not be used directly by clients and is only exposed for testing, the CLI, and type hinting.

aga.group(weight=1, value=0.0, extra_credit=0.0)

Declare a group of problems.

Parameters:

weight (int) – The group’s relative weight to the problem’s score. See Determining Score for details.
value (float) – The group’s absolute score. See Determining Score for details.
extra_credit (float) – The group’s extra credit. See Determining Score for details.

Returns:

A decorator which adds the group to a problem.

Return type:

Callable[[Problem[T]], Problem[T]]

aga.param: alias of _TestParam

aga.problem(name=None, script=False, check_stdout=None, mock_input=None, ctx=())

Declare a function as the golden solution to a problem.

This method should decorate a function which is known to produce the correct outputs, which we refer to as the “golden solution”. It also provides facilities for testing that solution, via golden test cases, constructed by passing the output argument to the test_case decorator.

Parameters:

name (Optional[str]) – The problem’s name. If None (the default), the wrapped function’s name will be used.
script (bool) – Whether the problem represents a script, as opposed to a function. Implies check_stdout and mock_input unless they are passed explicitly.
check_stdout (Optional[bool]) – Overrides the problem.check_stdout configuration option. If True, check the golden solution’s stdout against the student submission’s for all test cases.
mock_input (Optional[bool]) – Overrides the problem.mock_input configuration option. If True, test cases for this problem will be interpreted as mocked outputs of builtins.input, rather than inputs to the function.
ctx (Iterable[str]) – The context values required in the submission and will be captured

Returns:

A decorator which turns a golden solution into a problem.

Return type:

Callable[[Callable[ProblemInput, T]], Problem[T]]

aga.test_case: alias of _TestParam

aga.test_cases: alias of _TestParams

Core

The core library functionality.

class aga.core.AgaTestCase(test_input, golden, under_test, metadata)

A TestCase which runs a single test of a Problem.

property description: str | None

Get the problem’s description.

Return type:: UnionType[str, None]

property metadata: TestMetadata

Get the test’s metadata.

Return type:: TestMetadata

property name: str

Format the name of the test case.

Return type:: str

runTest()

Run the test case.

Return type:: None

shortDescription()

Dynamically generate the test name.

This method is called by unittest.

Return type:: str

property test_input: _TestInputs[Output]

Get the test input.

Return type:: _TestInputs[~Output]

class aga.core.AgaTestSuite(config, tests): A thin wrapper around TestSuite that store a config.

class aga.core.Problem(golden, name, config, is_script, ctx_targets=())

Stores tests for a single problem.

add_group(grp)

Add a group to the problem.

Return type:: None

add_prize(prize)

Add a prize to the current group.

Return type:: None

add_test_case(param)

Add a test case to the current group.

Student solutions will be checked against the golden solution; i.e., this method does _not_ produce a test of the golden solution.

Return type:: None

check()

Check that the problem is correct.

Currently, this runs all tests of the golden solution.

Return type:: None

config()

Get access to the problem’s config.

Return type:: AgaConfig

expected_symbol()

Get the name of the symbol that should be tested against.

Return type:: str

generate_test_suite(under_test, metadata)

Generate a TestSuite for the student submitted function.

Neither the generated test suite nor the body of this function will run golden tests; instead, golden test cases are treated as equivalent to ordinary ones. To test the golden function, check should be used instead.

Parameters:

under_test (Callable[ProblemParamSpec, ProblemOutputType]) – The student submitted function.
metadata (SubmissionMetadata) – The submission metadata.

Return type:

tuple[AgaTestSuite, list[ScoredPrize]]

Returns:

AgaTestSuite – A unittest test suite containing one test for each TestInput in this problem, checking the result of the problem’s golden solution against under_test.
list[ScorePrize] – The prizes for the problem, with scores assigned.

property golden: Callable[[ProblemParamSpec], ProblemOutputType]

The gold solution property.

Return type:: Callable[[ParamSpec], ~ProblemOutputType]

name()

Get the problem’s name.

Return type:: str

property submission_context: SubmissionContext

The environment values captured from the problem module.

Return type:: SubmissionContext

update_config_weak(config)

Update any non-default items in self.config.

Return type:: None

class aga.core.SubmissionMetadata(total_score, time_since_due, previous_submissions)

Metadata for testing a submission, collected from the frontend.

total_score

The problem’s total score.

Type:: float

time_since-due

The delta _from_ the due date _to_ the submission date, i.e. it’s negative if the problem was submitted before the due date.

Type:: timedelta

previous_submissions

The number of previous submissions.

Type:: int

is_on_time()

Return true of the submission was on time.

Return type:: bool

class aga.core.TestMetadata(max_score, config, check_stdout, mock_input, hidden=False): Stores metadata about a specific test case.

aga.core.group(weight=1, value=0.0, extra_credit=0.0)

Declare a group of problems.

Parameters:

weight (int) – The group’s relative weight to the problem’s score. See Determining Score for details.
value (float) – The group’s absolute score. See Determining Score for details.
extra_credit (float) – The group’s extra credit. See Determining Score for details.

Returns:

A decorator which adds the group to a problem.

Return type:

Callable[[Problem[T]], Problem[T]]

aga.core.param: alias of _TestParam

aga.core.problem(name=None, script=False, check_stdout=None, mock_input=None, ctx=())

Declare a function as the golden solution to a problem.

This method should decorate a function which is known to produce the correct outputs, which we refer to as the “golden solution”. It also provides facilities for testing that solution, via golden test cases, constructed by passing the output argument to the test_case decorator.

Parameters:

name (Optional[str]) – The problem’s name. If None (the default), the wrapped function’s name will be used.
script (bool) – Whether the problem represents a script, as opposed to a function. Implies check_stdout and mock_input unless they are passed explicitly.
check_stdout (Optional[bool]) – Overrides the problem.check_stdout configuration option. If True, check the golden solution’s stdout against the student submission’s for all test cases.
mock_input (Optional[bool]) – Overrides the problem.mock_input configuration option. If True, test cases for this problem will be interpreted as mocked outputs of builtins.input, rather than inputs to the function.
ctx (Iterable[str]) – The context values required in the submission and will be captured

Returns:

A decorator which turns a golden solution into a problem.

Return type:

Callable[[Callable[ProblemInput, T]], Problem[T]]

aga.core.test_case: alias of _TestParam

aga.core.test_cases: alias of _TestParams

Core - Parameters

class aga.core.parameter._TestParam(*args: Any, aga_expect: Any = None, aga_expect_stdout: str | Sequence[str] | None = None, aga_hidden: bool = False, aga_name: str | None = None, aga_description: str | None = None, aga_weight: int = 1, aga_value: float = 0.0, aga_extra_credit: float = 0.0, aga_override_check: Callable[[...], Any] | None = None, aga_override_test: Callable[[...], Any] | None = None, aga_is_pipeline: bool = False, **kwargs: Any)

class aga.core.parameter._TestParam(*args: Any, **kwargs: Any)

property aga_kwargs: AgaKeywordDictType

Return the aga_* keyword arguments of the test.

Return type:: AgaKeywordDictType

aga_kwargs_repr(sep=',')

Return a string representation of the test’s aga_* keyword arguments.

Return type:: str

property args: Tuple[Any, ...]

Return the arguments to be passed to the functions under test.

Return type:: Tuple[Any, …]

args_repr(sep=',')

Return a string representation of the test’s arguments.

Return type:: str

property description: str | None

Get the description of the test case.

Return type:: UnionType[str, None]

ensure_aga_kwargs()

Ensure that the aga_* keywords are handled correct.

Return type:: AgaKeywordContainer

ensure_default_aga_values()

Ensure that the aga_* keywords all have default.

Return type:: AgaKeywordContainer

ensure_valid_kwargs()

Ensure that the aga_* keywords are handled correct.

Return type:: _TestParam

property expect: Any

Get the expected aga_expect of the test case.

Return type:: Any

property expect_stdout: str | None

Get the expected aga_expect_stdout of the test case.

Return type:: UnionType[str, None]

property extra_credit: float

Get the extra credit aga_extra_credit of the test case.

Return type:: float

generate_test_case(prob)

Generate a test case for the given problem.

Return type:: Problem[ProblemParamSpec, ProblemOutputType]

property hidden: bool

Get the hidden aga_hidden of the test case.

Return type:: bool

property is_pipeline: bool

Get the is_pipeline aga_is_pipeline of the test case.

Return type:: bool

property kwargs: Dict[str, Any]

Return the keyword arguments to be passed to the functions under test.

Return type:: Dict[str, Any]

kwargs_repr(sep=',')

Return appropriate string representation of the test’s keyword arguments.

Return type:: str

property name: str | None

Get the name of the test case.

Return type:: UnionType[str, None]

property override_check: Callable[[...], Any] | None

Get the override_check aga_override_check of the test case.

Return type:: Optional[Callable[…, Any], None]

property override_test: Callable[[...], Any] | None

Get the override_test aga_override_test of the test case.

Return type:: Optional[Callable[…, Any], None]

sep_repr(sep=',')

Return sep if both exist, “” otherwise.

Return type:: str

update_aga_kwargs(**kwargs)

Update the keyword arguments to be passed to the functions under test.

Return type:: AgaKeywordContainer

property value: float

Get the value aga_value of the test case.

Return type:: float

property weight: int

Get the weight aga_weight of the test case.

Return type:: int

class aga.core.parameter._TestParams(*args: Any, aga_expect: Any = None, aga_expect_stdout: str | Sequence[str] | None = None, aga_hidden: bool = False, aga_name: str | None = None, aga_description: str | None = None, aga_weight: int = 1, aga_value: float = 0.0, aga_extra_credit: float = 0.0, aga_override_check: Callable[[...], Any] | None = None, aga_override_test: Callable[[...], Any] | None = None, aga_is_pipeline: bool = False, aga_product: bool = False, aga_zip: bool = False, aga_params: bool = False, aga_singular_params: bool = False, **kwargs: Any)

class aga.core.parameter._TestParams(*args: Any, **kwargs: Any)

A class to store the parameters for a test.

static add_aga_kwargs(aga_kwargs, final_params)

Add aga_kwargs to the finalized parameters.

Return type:: None

params: ClassVar[partial[_TestParams]] = functools.partial(<class 'aga.core.parameter._TestParams'>, aga_params=True)

static parse_no_flag(*args, **kwargs)

Parse the parameters for no flag.

Return type:: List[_TestParam]

static parse_params(*args, **kwargs)

Parse the parameters for param sequence.

Return type:: List[_TestParam]

static parse_singular_params(*args, **kwargs)

Parse the parameters for param sequence.

Return type:: List[_TestParam]

static parse_zip_or_product(*args, aga_product=False, aga_zip=False, **kwargs)

Parse parameters for zip or product.

Return type:: List[_TestParam]

product: ClassVar[partial[_TestParams]] = functools.partial(<class 'aga.core.parameter._TestParams'>, aga_product=True)

singular_params: ClassVar[partial[_TestParams]] = functools.partial(<class 'aga.core.parameter._TestParams'>, aga_singular_params=True)

zip: ClassVar[partial[_TestParams]] = functools.partial(<class 'aga.core.parameter._TestParams'>, aga_zip=True)

Prizes

Add points prizes to problems.

This module contains the prize decorator, which lets you define custom post-test-run points hooks for things like correctness and lateness. It also contains several prizes, defined for convenience.

class aga.prize.SubmissionMetadata(total_score, time_since_due, previous_submissions)

Metadata for testing a submission, collected from the frontend.

total_score

The problem’s total score.

Type:: float

time_since-due

The delta _from_ the due date _to_ the submission date, i.e. it’s negative if the problem was submitted before the due date.

Type:: timedelta

previous_submissions

The number of previous submissions.

Type:: int

is_on_time()

Return true of the submission was on time.

Return type:: bool

class aga.prize.TcOutput(score, max_score, name, status=None, hidden=False, description=None, error_description=None)

Stores information about a completed test case.

score

The test’s score.

Type:: float

max_score

The max score for the test.

Type:: float

name

The test’s name.

Type:: str

description

Human-readable text description of the test. Some frontends distinguish between no output and empty output, i.e. in terms of showing UI elements.

Type:: Optional[str]

error_description

Human-readable error description of the test.

Type:: Optional[str]

hidden

The test’s visibility.

Type:: bool

static format_description(desc)

Format a description.

Return type:: str

static format_error_description(error)

Format an error description.

Return type:: str

static format_rich_output(description=None, error_description=None)

Format a rich output.

Return type:: str

is_correct()

Check whether the problem received full credit.

Return type:: bool

property rich_output: str

Output of all the descriptions.

Return type:: str

aga.prize.all_correct(tests, _)

1.0 if all tests passed, 0.0 otherwise.

For use as a prize.

Return type:: tuple[float, str]

aga.prize.correct_and_on_time(tests, metadata)

1.0 if the submission was correct and passed all tests, 0.0 otherwise.

For use as a prize.

Return type:: tuple[float, str]

aga.prize.on_time(_, metadata)

1.0 if the submission was on time, 0.0 otherwise.

For use as a prize.

Return type:: tuple[float, str]

aga.prize.prize(criteria, name='Prize', weight=1, value=0.0, extra_credit=0.0, hidden=False)

Add a points prize to the problem.

Parameters:

criteria (Callable[[list[TcOutput], SubmissionMetadata], tuple[float, str]) – The criteria for awarding the prize’s points. The first returned value should be a float from 0 to 1 which determines the fraction of points to assign. The second should be a string which will be displayed to the student.
name (str) – The name of the prize, to be displayed to the student.
weight (int) – The prize’s weight. See Determining Score for details.
value (int) – The prize’s absolute score. See Determining Score for details.
extra_credit (int) – The prize’s extra credit. See Determining Score for details.
hidden (bool) – Whether the prize should be hidden from the student.

Returns:

A decorator which adds the prize to a problem.

Return type:

Callable[[Problem[T]], Problem[T]]

Checks

Additional checks and filters for problems.

class aga.checks.Disallow(functions=None, binops=None, nodes=None)

A list of items to disallow in code.

functions

The names of functions which the student should not be able to call.

Type:: list[str]

binops

The types of binary operations wihch the student should not be able to use. E.x., to forbid floating-point division, use ast.Div. See here for a list.

Type:: list[type]

nodes

The types of any ast nodes wihch the student should not be able to use. E.x., to forbid for loops, use ast.For. See the docs for a list.

Type:: list[type]

Examples

To disallow the built-in map function: Disallow(functions=["map"]).

To disallow the built-in str.map function: Disallow(functions=["count"]). Note that for class method names, you just use the name of the function.

Note that there is no way to disallow += without also disallowing + with this API.

search_on_object(obj)

Search for disallowed AST objects in a python object.

Return type:: Iterable[tuple[str, int]]

search_on_src(src)

Search for disallowed AST objects in a source string.

Return type:: Iterable[tuple[str, int]]

to_test()

Generate a test method suitable for aga_override_test of test_case.

You can pass the output of this method directly to aga_override_test.

You can also use the lower-level methods search_on_object or search_on_src if you want to generate your own error message.

Return type:: Callable[[TestCase, Callable[…, ~Output], Callable[…, ~Output]], None]

aga.checks.Site: alias of tuple[str, int]