Reference

Aga grades assignments: a library for easily producing autograders for code.

Anything not explicitly documented here should not be used directly by clients and is only exposed for testing, the CLI, and type hinting.

aga.group(weight=1, value=0.0, extra_credit=0.0)

Declare a group of problems.

Parameters
  • weight (int) – The group’s relative weight to the problem’s score. See Determining Score for details.

  • value (float) – The group’s absolute score. See Determining Score for details.

  • extra_credit (float) – The group’s extra credit. See Determining Score for details.

Returns

A decorator which adds the group to a problem.

Return type

Callable[[Problem[T]], Problem[T]]

aga.param

alias of aga.core.parameter._TestParam

aga.problem(name=None, script=False, check_stdout=None, mock_input=None, ctx=())

Declare a function as the golden solution to a problem.

This method should decorate a function which is known to produce the correct outputs, which we refer to as the “golden solution”. It also provides facilities for testing that solution, via golden test cases, constructed by passing the output argument to the test_case decorator.

Parameters
  • name (Optional[str]) – The problem’s name. If None (the default), the wrapped function’s name will be used.

  • script (bool) – Whether the problem represents a script, as opposed to a function. Implies check_stdout and mock_input unless they are passed explicitly.

  • check_stdout (Optional[bool]) – Overrides the problem.check_stdout configuration option. If True, check the golden solution’s stdout against the student submission’s for all test cases.

  • mock_input (Optional[bool]) – Overrides the problem.mock_input configuration option. If True, test cases for this problem will be interpreted as mocked outputs of builtins.input, rather than inputs to the function.

  • ctx (Iterable[str]) – The context values required in the submission and will be captured

Returns

A decorator which turns a golden solution into a problem.

Return type

Callable[[Callable[ProblemInput, T]], Problem[T]]

aga.test_case

alias of aga.core.parameter._TestParam

aga.test_cases

alias of aga.core.parameter._TestParams

Core

The core library functionality.

class aga.core.AgaTestCase(test_input, golden, under_test, metadata)

A TestCase which runs a single test of a Problem.

property description: str | None

Get the problem’s description.

Return type

UnionType[str, None]

property metadata: aga.core.suite.TestMetadata

Get the test’s metadata.

Return type

TestMetadata

property name: str

Format the name of the test case.

Return type

str

runTest()

Run the test case.

Return type

None

shortDescription()

Dynamically generate the test name.

This method is called by unittest.

Return type

str

property test_input: aga.core.suite._TestInputs[aga.core.suite.Output]

Get the test input.

Return type

_TestInputs[~Output]

class aga.core.AgaTestSuite(config, tests)

A thin wrapper around TestSuite that store a config.

class aga.core.Problem(golden, name, config, is_script, ctx_targets=())

Stores tests for a single problem.

add_group(grp)

Add a group to the problem.

Return type

None

add_prize(prize)

Add a prize to the current group.

Return type

None

add_test_case(param)

Add a test case to the current group.

Student solutions will be checked against the golden solution; i.e., this method does _not_ produce a test of the golden solution.

Return type

None

check()

Check that the problem is correct.

Currently, this runs all tests of the golden solution.

Return type

None

config()

Get access to the problem’s config.

Return type

AgaConfig

expected_symbol()

Get the name of the symbol that should be tested against.

Return type

str

generate_test_suite(under_test, metadata)

Generate a TestSuite for the student submitted function.

Neither the generated test suite nor the body of this function will run golden tests; instead, golden test cases are treated as equivalent to ordinary ones. To test the golden function, check should be used instead.

Parameters
  • under_test (Callable[ProblemParamSpec, ProblemOutputType]) – The student submitted function.

  • metadata (SubmissionMetadata) – The submission metadata.

Return type

tuple[AgaTestSuite, list[ScoredPrize]]

Returns

  • AgaTestSuite – A unittest test suite containing one test for each TestInput in this problem, checking the result of the problem’s golden solution against under_test.

  • list[ScorePrize] – The prizes for the problem, with scores assigned.

property golden: Callable[[~ProblemParamSpec], aga.core.problem.ProblemOutputType]

The gold solution property.

Return type

Callable[[ParamSpec], ~ProblemOutputType]

name()

Get the problem’s name.

Return type

str

property submission_context: aga.core.context.SubmissionContext

The environment values captured from the problem module.

Return type

SubmissionContext

update_config_weak(config)

Update any non-default items in self.config.

Return type

None

class aga.core.SubmissionMetadata(total_score, time_since_due, previous_submissions)

Metadata for testing a submission, collected from the frontend.

total_score

The problem’s total score.

Type

float

time_since-due

The delta _from_ the due date _to_ the submission date, i.e. it’s negative if the problem was submitted before the due date.

Type

timedelta

previous_submissions

The number of previous submissions.

Type

int

is_on_time()

Return true of the submission was on time.

Return type

bool

class aga.core.TestMetadata(max_score, config, check_stdout, mock_input, hidden=False)

Stores metadata about a specific test case.

aga.core.group(weight=1, value=0.0, extra_credit=0.0)

Declare a group of problems.

Parameters
  • weight (int) – The group’s relative weight to the problem’s score. See Determining Score for details.

  • value (float) – The group’s absolute score. See Determining Score for details.

  • extra_credit (float) – The group’s extra credit. See Determining Score for details.

Returns

A decorator which adds the group to a problem.

Return type

Callable[[Problem[T]], Problem[T]]

aga.core.param

alias of aga.core.parameter._TestParam

aga.core.problem(name=None, script=False, check_stdout=None, mock_input=None, ctx=())

Declare a function as the golden solution to a problem.

This method should decorate a function which is known to produce the correct outputs, which we refer to as the “golden solution”. It also provides facilities for testing that solution, via golden test cases, constructed by passing the output argument to the test_case decorator.

Parameters
  • name (Optional[str]) – The problem’s name. If None (the default), the wrapped function’s name will be used.

  • script (bool) – Whether the problem represents a script, as opposed to a function. Implies check_stdout and mock_input unless they are passed explicitly.

  • check_stdout (Optional[bool]) – Overrides the problem.check_stdout configuration option. If True, check the golden solution’s stdout against the student submission’s for all test cases.

  • mock_input (Optional[bool]) – Overrides the problem.mock_input configuration option. If True, test cases for this problem will be interpreted as mocked outputs of builtins.input, rather than inputs to the function.

  • ctx (Iterable[str]) – The context values required in the submission and will be captured

Returns

A decorator which turns a golden solution into a problem.

Return type

Callable[[Callable[ProblemInput, T]], Problem[T]]

aga.core.test_case

alias of aga.core.parameter._TestParam

aga.core.test_cases

alias of aga.core.parameter._TestParams

Core - Parameters

class aga.core.parameter._TestParam(*args: Any, aga_expect: Any = 'None', aga_expect_stdout: Optional[Union[str, Sequence[str]]] = 'None', aga_hidden: bool = 'False', aga_name: str | None = 'None', aga_description: str | None = 'None', aga_weight: int = '1', aga_value: float = '0.0', aga_extra_credit: float = '0.0', aga_override_check: Optional[Callable[[...], Any]] = 'None', aga_override_test: Optional[Callable[[...], Any]] = 'None', aga_is_pipeline: bool = 'False', **kwargs: Any)
class aga.core.parameter._TestParam(*args: Any, **kwargs: Any)
property aga_kwargs: aga.core.parameter.AgaKeywordDictType

Return the aga_* keyword arguments of the test.

Return type

AgaKeywordDictType

aga_kwargs_repr(sep=',')

Return a string representation of the test’s aga_* keyword arguments.

Return type

str

property args: Tuple[Any, ...]

Return the arguments to be passed to the functions under test.

Return type

Tuple[Any, …]

args_repr(sep=',')

Return a string representation of the test’s arguments.

Return type

str

property description: str | None

Get the description of the test case.

Return type

UnionType[str, None]

ensure_aga_kwargs()

Ensure that the aga_* keywords are handled correct.

Return type

AgaKeywordContainer

ensure_default_aga_values()

Ensure that the aga_* keywords all have default.

Return type

AgaKeywordContainer

ensure_valid_kwargs()

Ensure that the aga_* keywords are handled correct.

Return type

_TestParam

property expect: Any

Get the expected aga_expect of the test case.

Return type

Any

property expect_stdout: str | None

Get the expected aga_expect_stdout of the test case.

Return type

UnionType[str, None]

property extra_credit: float

Get the extra credit aga_extra_credit of the test case.

Return type

float

generate_test_case(prob)

Generate a test case for the given problem.

Return type

Problem[ProblemParamSpec, ProblemOutputType]

property hidden: bool

Get the hidden aga_hidden of the test case.

Return type

bool

property is_pipeline: bool

Get the is_pipeline aga_is_pipeline of the test case.

Return type

bool

property kwargs: Dict[str, Any]

Return the keyword arguments to be passed to the functions under test.

Return type

Dict[str, Any]

kwargs_repr(sep=',')

Return appropriate string representation of the test’s keyword arguments.

Return type

str

property name: str | None

Get the name of the test case.

Return type

UnionType[str, None]

property override_check: Optional[Callable[[...], Any]]

Get the override_check aga_override_check of the test case.

Return type

Optional[Callable[…, Any], None]

property override_test: Optional[Callable[[...], Any]]

Get the override_test aga_override_test of the test case.

Return type

Optional[Callable[…, Any], None]

sep_repr(sep=',')

Return sep if both exist, “” otherwise.

Return type

str

update_aga_kwargs(**kwargs)

Update the keyword arguments to be passed to the functions under test.

Return type

AgaKeywordContainer

property value: float

Get the value aga_value of the test case.

Return type

float

property weight: int

Get the weight aga_weight of the test case.

Return type

int

class aga.core.parameter._TestParams(*args: Any, aga_expect: Any = 'None', aga_expect_stdout: Optional[Union[str, Sequence[str]]] = 'None', aga_hidden: bool = 'False', aga_name: str | None = 'None', aga_description: str | None = 'None', aga_weight: int = '1', aga_value: float = '0.0', aga_extra_credit: float = '0.0', aga_override_check: Optional[Callable[[...], Any]] = 'None', aga_override_test: Optional[Callable[[...], Any]] = 'None', aga_is_pipeline: bool = 'False', aga_product: bool = 'False', aga_zip: bool = 'False', aga_params: bool = 'False', aga_singular_params: bool = 'False', **kwargs: Any)
class aga.core.parameter._TestParams(*args: Any, **kwargs: Any)

A class to store the parameters for a test.

static add_aga_kwargs(aga_kwargs, final_params)

Add aga_kwargs to the finalized parameters.

Return type

None

params: ClassVar[functools.partial[aga.core.parameter._TestParams]] = functools.partial(<class 'aga.core.parameter._TestParams'>, aga_params=True)
static parse_no_flag(*args, **kwargs)

Parse the parameters for no flag.

Return type

List[_TestParam]

static parse_params(*args, **kwargs)

Parse the parameters for param sequence.

Return type

List[_TestParam]

static parse_singular_params(*args, **kwargs)

Parse the parameters for param sequence.

Return type

List[_TestParam]

static parse_zip_or_product(*args, aga_product=False, aga_zip=False, **kwargs)

Parse parameters for zip or product.

Return type

List[_TestParam]

product: ClassVar[functools.partial[aga.core.parameter._TestParams]] = functools.partial(<class 'aga.core.parameter._TestParams'>, aga_product=True)
singular_params: ClassVar[functools.partial[aga.core.parameter._TestParams]] = functools.partial(<class 'aga.core.parameter._TestParams'>, aga_singular_params=True)
zip: ClassVar[functools.partial[aga.core.parameter._TestParams]] = functools.partial(<class 'aga.core.parameter._TestParams'>, aga_zip=True)

Prizes

Add points prizes to problems.

This module contains the prize decorator, which lets you define custom post-test-run points hooks for things like correctness and lateness. It also contains several prizes, defined for convenience.

class aga.prize.SubmissionMetadata(total_score, time_since_due, previous_submissions)

Metadata for testing a submission, collected from the frontend.

total_score

The problem’s total score.

Type

float

time_since-due

The delta _from_ the due date _to_ the submission date, i.e. it’s negative if the problem was submitted before the due date.

Type

timedelta

previous_submissions

The number of previous submissions.

Type

int

is_on_time()

Return true of the submission was on time.

Return type

bool

class aga.prize.TcOutput(score, max_score, name, status=None, hidden=False, description=None, error_description=None)

Stores information about a completed test case.

score

The test’s score.

Type

float

max_score

The max score for the test.

Type

float

name

The test’s name.

Type

str

description

Human-readable text description of the test. Some frontends distinguish between no output and empty output, i.e. in terms of showing UI elements.

Type

Optional[str]

error_description

Human-readable error description of the test.

Type

Optional[str]

hidden

The test’s visibility.

Type

bool

static format_description(desc)

Format a description.

Return type

str

static format_error_description(error)

Format an error description.

Return type

str

static format_rich_output(description=None, error_description=None)

Format a rich output.

Return type

str

is_correct()

Check whether the problem received full credit.

Return type

bool

property rich_output: str

Output of all the descriptions.

Return type

str

aga.prize.all_correct(tests, _)

1.0 if all tests passed, 0.0 otherwise.

For use as a prize.

Return type

tuple[float, str]

aga.prize.correct_and_on_time(tests, metadata)

1.0 if the submission was correct and passed all tests, 0.0 otherwise.

For use as a prize.

Return type

tuple[float, str]

aga.prize.on_time(_, metadata)

1.0 if the submission was on time, 0.0 otherwise.

For use as a prize.

Return type

tuple[float, str]

aga.prize.prize(criteria, name='Prize', weight=1, value=0.0, extra_credit=0.0, hidden=False)

Add a points prize to the problem.

Parameters
  • criteria (Callable[[list[TcOutput], SubmissionMetadata], tuple[float, str]) – The criteria for awarding the prize’s points. The first returned value should be a float from 0 to 1 which determines the fraction of points to assign. The second should be a string which will be displayed to the student.

  • name (str) – The name of the prize, to be displayed to the student.

  • weight (int) – The prize’s weight. See Determining Score for details.

  • value (int) – The prize’s absolute score. See Determining Score for details.

  • extra_credit (int) – The prize’s extra credit. See Determining Score for details.

  • hidden (bool) – Whether the prize should be hidden from the student.

Returns

A decorator which adds the prize to a problem.

Return type

Callable[[Problem[T]], Problem[T]]

Checks

Additional checks and filters for problems.

class aga.checks.Disallow(functions=None, binops=None, nodes=None)

A list of items to disallow in code.

functions

The names of functions which the student should not be able to call.

Type

list[str]

binops

The types of binary operations wihch the student should not be able to use. E.x., to forbid floating-point division, use ast.Div. See here for a list.

Type

list[type]

nodes

The types of any ast nodes wihch the student should not be able to use. E.x., to forbid for loops, use ast.For. See the docs for a list.

Type

list[type]

Examples

To disallow the built-in map function: Disallow(functions=["map"]).

To disallow the built-in str.map function: Disallow(functions=["count"]). Note that for class method names, you just use the name of the function.

Note that there is no way to disallow += without also disallowing + with this API.

search_on_object(obj)

Search for disallowed AST objects in a python object.

Return type

Iterable[tuple[str, int]]

search_on_src(src)

Search for disallowed AST objects in a source string.

Return type

Iterable[tuple[str, int]]

to_test()

Generate a test method suitable for aga_override_test of test_case.

You can pass the output of this method directly to aga_override_test.

You can also use the lower-level methods search_on_object or search_on_src if you want to generate your own error message.

Return type

Callable[[TestCase, Callable[…, ~Output], Callable[…, ~Output]], None]

aga.checks.Site

alias of tuple[str, int]