ksuit.providers.metric_property_provider¶
Classes¶
Defines the direction of improvement for a metric. |
|
Provider for properties about metrics, mainly the ordinality of a metric, i.e., is it better if it is higher |
Module Contents¶
- class ksuit.providers.metric_property_provider.Ordinality(*args, **kwds)¶
Bases:
enum.EnumDefines the direction of improvement for a metric.
- HIGHER_IS_BETTER¶
- LOWER_IS_BETTER¶
- NEUTRAL¶
- class ksuit.providers.metric_property_provider.MetricPropertyProvider¶
Provider for properties about metrics, mainly the ordinality of a metric, i.e., is it better if it is higher (e.g, accuracy of a classifier) or better if it is lower (e.g., a mean-squared error or any other loss). Additionally, a concept of “neutral” metrics is introduced, which are things that are logged throughout training but there is no concept of “best” metric (e.g., a learning rate that is scheduled, weight decay, …).
By default, the following patterns are handled: - “loss” (lower is better) - “error” (lower is better) - “accuracy” (higher is better)
Callbacks or trainers can freely add patterns if they introduce new metrics that are not handled.
The MetricPropertyProvider is used by the experiment tracker to automatically create summary values and log them to the online interface/log files and also by early stopping mechanisms to enable flexible early stopping where the early stopping criteria can use loss values, but also, e.g., validation accuracies.
- static register_pattern(pattern, ordinality)¶
Allows users (Trainers/Callbacks) to add new patterns easily.
- Parameters:
pattern (str)
ordinality (Ordinality)
- logger¶
- static get_ordinality(key)¶
Determines the ordinality of a metric key based on registered patterns. The search is case-insensitive.
- Parameters:
key (str)
- Return type:
- is_neutral_key(key)¶
Returns True if the passed key is a neutral metric.