Reference
Aga grades assignments: a library for easily producing autograders for code.
Anything not explicitly documented here should not be used directly by clients and is only exposed for testing, the CLI, and type hinting.
- aga.group(weight=1, value=0.0, extra_credit=0.0)
Declare a group of problems.
- Parameters
weight (int) – The group’s relative weight to the problem’s score. See Determining Score for details.
value (float) – The group’s absolute score. See Determining Score for details.
extra_credit (float) – The group’s extra credit. See Determining Score for details.
- Returns
A decorator which adds the group to a problem.
- Return type
- aga.param
alias of
aga.core.parameter._TestParam
- aga.problem(name=None, script=False, check_stdout=None, mock_input=None, ctx=())
Declare a function as the golden solution to a problem.
This method should decorate a function which is known to produce the correct outputs, which we refer to as the “golden solution”. It also provides facilities for testing that solution, via golden test cases, constructed by passing the output argument to the test_case decorator.
- Parameters
name (Optional[str]) – The problem’s name. If None (the default), the wrapped function’s name will be used.
script (bool) – Whether the problem represents a script, as opposed to a function. Implies
check_stdout
andmock_input
unless they are passed explicitly.check_stdout (Optional[bool]) – Overrides the
problem.check_stdout
configuration option. If True, check the golden solution’s stdout against the student submission’s for all test cases.mock_input (Optional[bool]) – Overrides the
problem.mock_input
configuration option. If True, test cases for this problem will be interpreted as mocked outputs ofbuiltins.input
, rather than inputs to the function.ctx (Iterable[str]) – The context values required in the submission and will be captured
- Returns
A decorator which turns a golden solution into a problem.
- Return type
Callable[[Callable[ProblemInput, T]], Problem[T]]
- aga.test_case
alias of
aga.core.parameter._TestParam
- aga.test_cases
alias of
aga.core.parameter._TestParams
Core
The core library functionality.
- class aga.core.AgaTestCase(test_input, golden, under_test, metadata)
A
TestCase
which runs a single test of aProblem
.- property description: str | None
Get the problem’s description.
- Return type
UnionType
[str
,None
]
- property metadata: aga.core.suite.TestMetadata
Get the test’s metadata.
- Return type
- property name: str
Format the name of the test case.
- Return type
str
- runTest()
Run the test case.
- Return type
None
- shortDescription()
Dynamically generate the test name.
This method is called by unittest.
- Return type
str
- property test_input: aga.core.suite._TestInputs[aga.core.suite.Output]
Get the test input.
- Return type
_TestInputs
[~Output]
- class aga.core.AgaTestSuite(config, tests)
A thin wrapper around TestSuite that store a config.
- class aga.core.Problem(golden, name, config, is_script, ctx_targets=())
Stores tests for a single problem.
- add_group(grp)
Add a group to the problem.
- Return type
None
- add_prize(prize)
Add a prize to the current group.
- Return type
None
- add_test_case(param)
Add a test case to the current group.
Student solutions will be checked against the golden solution; i.e., this method does _not_ produce a test of the golden solution.
- Return type
None
- check()
Check that the problem is correct.
Currently, this runs all tests of the golden solution.
- Return type
None
- config()
Get access to the problem’s config.
- Return type
AgaConfig
- expected_symbol()
Get the name of the symbol that should be tested against.
- Return type
str
- generate_test_suite(under_test, metadata)
Generate a
TestSuite
for the student submitted function.Neither the generated test suite nor the body of this function will run golden tests; instead, golden test cases are treated as equivalent to ordinary ones. To test the golden function,
check
should be used instead.- Parameters
under_test (Callable[ProblemParamSpec, ProblemOutputType]) – The student submitted function.
metadata (SubmissionMetadata) – The submission metadata.
- Return type
tuple
[AgaTestSuite
,list
[ScoredPrize
]]- Returns
AgaTestSuite – A unittest test suite containing one test for each TestInput in this problem, checking the result of the problem’s golden solution against
under_test
.list[ScorePrize] – The prizes for the problem, with scores assigned.
- property golden: Callable[[~ProblemParamSpec], aga.core.problem.ProblemOutputType]
The gold solution property.
- Return type
Callable
[[ParamSpec
], ~ProblemOutputType]
- name()
Get the problem’s name.
- Return type
str
- property submission_context: aga.core.context.SubmissionContext
The environment values captured from the problem module.
- Return type
SubmissionContext
- update_config_weak(config)
Update any non-default items in self.config.
- Return type
None
- class aga.core.SubmissionMetadata(total_score, time_since_due, previous_submissions)
Metadata for testing a submission, collected from the frontend.
- total_score
The problem’s total score.
- Type
float
- time_since-due
The delta _from_ the due date _to_ the submission date, i.e. it’s negative if the problem was submitted before the due date.
- Type
timedelta
- previous_submissions
The number of previous submissions.
- Type
int
- is_on_time()
Return true of the submission was on time.
- Return type
bool
- class aga.core.TestMetadata(max_score, config, check_stdout, mock_input, hidden=False)
Stores metadata about a specific test case.
- aga.core.group(weight=1, value=0.0, extra_credit=0.0)
Declare a group of problems.
- Parameters
weight (int) – The group’s relative weight to the problem’s score. See Determining Score for details.
value (float) – The group’s absolute score. See Determining Score for details.
extra_credit (float) – The group’s extra credit. See Determining Score for details.
- Returns
A decorator which adds the group to a problem.
- Return type
- aga.core.param
alias of
aga.core.parameter._TestParam
- aga.core.problem(name=None, script=False, check_stdout=None, mock_input=None, ctx=())
Declare a function as the golden solution to a problem.
This method should decorate a function which is known to produce the correct outputs, which we refer to as the “golden solution”. It also provides facilities for testing that solution, via golden test cases, constructed by passing the output argument to the test_case decorator.
- Parameters
name (Optional[str]) – The problem’s name. If None (the default), the wrapped function’s name will be used.
script (bool) – Whether the problem represents a script, as opposed to a function. Implies
check_stdout
andmock_input
unless they are passed explicitly.check_stdout (Optional[bool]) – Overrides the
problem.check_stdout
configuration option. If True, check the golden solution’s stdout against the student submission’s for all test cases.mock_input (Optional[bool]) – Overrides the
problem.mock_input
configuration option. If True, test cases for this problem will be interpreted as mocked outputs ofbuiltins.input
, rather than inputs to the function.ctx (Iterable[str]) – The context values required in the submission and will be captured
- Returns
A decorator which turns a golden solution into a problem.
- Return type
Callable[[Callable[ProblemInput, T]], Problem[T]]
- aga.core.test_case
alias of
aga.core.parameter._TestParam
- aga.core.test_cases
alias of
aga.core.parameter._TestParams
Core - Parameters
- class aga.core.parameter._TestParam(*args: Any, aga_expect: Any = 'None', aga_expect_stdout: Optional[Union[str, Sequence[str]]] = 'None', aga_hidden: bool = 'False', aga_name: str | None = 'None', aga_description: str | None = 'None', aga_weight: int = '1', aga_value: float = '0.0', aga_extra_credit: float = '0.0', aga_override_check: Optional[Callable[[...], Any]] = 'None', aga_override_test: Optional[Callable[[...], Any]] = 'None', aga_is_pipeline: bool = 'False', **kwargs: Any)
- class aga.core.parameter._TestParam(*args: Any, **kwargs: Any)
- property aga_kwargs: aga.core.parameter.AgaKeywordDictType
Return the aga_* keyword arguments of the test.
- Return type
AgaKeywordDictType
- aga_kwargs_repr(sep=',')
Return a string representation of the test’s aga_* keyword arguments.
- Return type
str
- property args: Tuple[Any, ...]
Return the arguments to be passed to the functions under test.
- Return type
Tuple
[Any
, …]
- args_repr(sep=',')
Return a string representation of the test’s arguments.
- Return type
str
- property description: str | None
Get the description of the test case.
- Return type
UnionType
[str
,None
]
- ensure_aga_kwargs()
Ensure that the aga_* keywords are handled correct.
- Return type
AgaKeywordContainer
- ensure_default_aga_values()
Ensure that the aga_* keywords all have default.
- Return type
AgaKeywordContainer
- ensure_valid_kwargs()
Ensure that the aga_* keywords are handled correct.
- Return type
- property expect: Any
Get the expected aga_expect of the test case.
- Return type
Any
- property expect_stdout: str | None
Get the expected aga_expect_stdout of the test case.
- Return type
UnionType
[str
,None
]
- property extra_credit: float
Get the extra credit aga_extra_credit of the test case.
- Return type
float
- generate_test_case(prob)
Generate a test case for the given problem.
- Return type
Problem[ProblemParamSpec, ProblemOutputType]
Get the hidden aga_hidden of the test case.
- Return type
bool
- property is_pipeline: bool
Get the is_pipeline aga_is_pipeline of the test case.
- Return type
bool
- property kwargs: Dict[str, Any]
Return the keyword arguments to be passed to the functions under test.
- Return type
Dict
[str
,Any
]
- kwargs_repr(sep=',')
Return appropriate string representation of the test’s keyword arguments.
- Return type
str
- property name: str | None
Get the name of the test case.
- Return type
UnionType
[str
,None
]
- property override_check: Optional[Callable[[...], Any]]
Get the override_check aga_override_check of the test case.
- Return type
Optional
[Callable
[…,Any
],None
]
- property override_test: Optional[Callable[[...], Any]]
Get the override_test aga_override_test of the test case.
- Return type
Optional
[Callable
[…,Any
],None
]
- sep_repr(sep=',')
Return sep if both exist, “” otherwise.
- Return type
str
- update_aga_kwargs(**kwargs)
Update the keyword arguments to be passed to the functions under test.
- Return type
AgaKeywordContainer
- property value: float
Get the value aga_value of the test case.
- Return type
float
- property weight: int
Get the weight aga_weight of the test case.
- Return type
int
- class aga.core.parameter._TestParams(*args: Any, aga_expect: Any = 'None', aga_expect_stdout: Optional[Union[str, Sequence[str]]] = 'None', aga_hidden: bool = 'False', aga_name: str | None = 'None', aga_description: str | None = 'None', aga_weight: int = '1', aga_value: float = '0.0', aga_extra_credit: float = '0.0', aga_override_check: Optional[Callable[[...], Any]] = 'None', aga_override_test: Optional[Callable[[...], Any]] = 'None', aga_is_pipeline: bool = 'False', aga_product: bool = 'False', aga_zip: bool = 'False', aga_params: bool = 'False', aga_singular_params: bool = 'False', **kwargs: Any)
- class aga.core.parameter._TestParams(*args: Any, **kwargs: Any)
A class to store the parameters for a test.
- static add_aga_kwargs(aga_kwargs, final_params)
Add aga_kwargs to the finalized parameters.
- Return type
None
- params: ClassVar[functools.partial[aga.core.parameter._TestParams]] = functools.partial(<class 'aga.core.parameter._TestParams'>, aga_params=True)
- static parse_no_flag(*args, **kwargs)
Parse the parameters for no flag.
- Return type
List
[_TestParam
]
- static parse_params(*args, **kwargs)
Parse the parameters for param sequence.
- Return type
List
[_TestParam
]
- static parse_singular_params(*args, **kwargs)
Parse the parameters for param sequence.
- Return type
List
[_TestParam
]
- static parse_zip_or_product(*args, aga_product=False, aga_zip=False, **kwargs)
Parse parameters for zip or product.
- Return type
List
[_TestParam
]
- product: ClassVar[functools.partial[aga.core.parameter._TestParams]] = functools.partial(<class 'aga.core.parameter._TestParams'>, aga_product=True)
- singular_params: ClassVar[functools.partial[aga.core.parameter._TestParams]] = functools.partial(<class 'aga.core.parameter._TestParams'>, aga_singular_params=True)
- zip: ClassVar[functools.partial[aga.core.parameter._TestParams]] = functools.partial(<class 'aga.core.parameter._TestParams'>, aga_zip=True)
Prizes
Add points prizes to problems.
This module contains the prize
decorator, which lets you define custom post-test-run
points hooks for things like correctness and lateness. It also contains several prizes,
defined for convenience.
- class aga.prize.SubmissionMetadata(total_score, time_since_due, previous_submissions)
Metadata for testing a submission, collected from the frontend.
- total_score
The problem’s total score.
- Type
float
- time_since-due
The delta _from_ the due date _to_ the submission date, i.e. it’s negative if the problem was submitted before the due date.
- Type
timedelta
- previous_submissions
The number of previous submissions.
- Type
int
- is_on_time()
Return true of the submission was on time.
- Return type
bool
- class aga.prize.TcOutput(score, max_score, name, status=None, hidden=False, description=None, error_description=None)
Stores information about a completed test case.
- score
The test’s score.
- Type
float
- max_score
The max score for the test.
- Type
float
- name
The test’s name.
- Type
str
- description
Human-readable text description of the test. Some frontends distinguish between no output and empty output, i.e. in terms of showing UI elements.
- Type
Optional[str]
- error_description
Human-readable error description of the test.
- Type
Optional[str]
The test’s visibility.
- Type
bool
- static format_description(desc)
Format a description.
- Return type
str
- static format_error_description(error)
Format an error description.
- Return type
str
- static format_rich_output(description=None, error_description=None)
Format a rich output.
- Return type
str
- is_correct()
Check whether the problem received full credit.
- Return type
bool
- property rich_output: str
Output of all the descriptions.
- Return type
str
- aga.prize.all_correct(tests, _)
1.0 if all tests passed, 0.0 otherwise.
For use as a prize.
- Return type
tuple[float, str]
- aga.prize.correct_and_on_time(tests, metadata)
1.0 if the submission was correct and passed all tests, 0.0 otherwise.
For use as a prize.
- Return type
tuple[float, str]
- aga.prize.on_time(_, metadata)
1.0 if the submission was on time, 0.0 otherwise.
For use as a prize.
- Return type
tuple[float, str]
- aga.prize.prize(criteria, name='Prize', weight=1, value=0.0, extra_credit=0.0, hidden=False)
Add a points prize to the problem.
- Parameters
criteria (Callable[[list[TcOutput], SubmissionMetadata], tuple[float, str]) – The criteria for awarding the prize’s points. The first returned value should be a float from 0 to 1 which determines the fraction of points to assign. The second should be a string which will be displayed to the student.
name (str) – The name of the prize, to be displayed to the student.
weight (int) – The prize’s weight. See Determining Score for details.
value (int) – The prize’s absolute score. See Determining Score for details.
extra_credit (int) – The prize’s extra credit. See Determining Score for details.
hidden (bool) – Whether the prize should be hidden from the student.
- Returns
A decorator which adds the prize to a problem.
- Return type
Checks
Additional checks and filters for problems.
- class aga.checks.Disallow(functions=None, binops=None, nodes=None)
A list of items to disallow in code.
- functions
The names of functions which the student should not be able to call.
- Type
list[str]
- binops
The types of binary operations wihch the student should not be able to use. E.x., to forbid floating-point division, use
ast.Div
. See here for a list.- Type
list[type]
- nodes
The types of any ast nodes wihch the student should not be able to use. E.x., to forbid for loops, use
ast.For
. See the docs for a list.- Type
list[type]
Examples
To disallow the built-in
map
function:Disallow(functions=["map"])
.To disallow the built-in
str.map
function:Disallow(functions=["count"])
. Note that for class method names, you just use the name of the function.Note that there is no way to disallow
+=
without also disallowing+
with this API.- search_on_object(obj)
Search for disallowed AST objects in a python object.
- Return type
Iterable
[tuple
[str
,int
]]
- search_on_src(src)
Search for disallowed AST objects in a source string.
- Return type
Iterable
[tuple
[str
,int
]]
- to_test()
Generate a test method suitable for
aga_override_test
oftest_case
.You can pass the output of this method directly to
aga_override_test
.You can also use the lower-level methods
search_on_object
orsearch_on_src
if you want to generate your own error message.- Return type
Callable
[[TestCase
,Callable
[…, ~Output],Callable
[…, ~Output]],None
]
- aga.checks.Site
alias of
tuple
[str
,int
]