Token Type Measures¶
TokenType(gold, disc, output_folder=None)¶
Token Type measure
The Token measures how many ‘Gold’ token were found. The Type measures how many ‘Gold’ type were found. A ‘Type’ is a word type, and a ‘token’ is a specific occurence of a ‘type’. See https://docs.syntheticlearner.net/tde/measures/index.html for a summary of all measures.
Input :param disc: Discovered Object, contains the discovered boundaries :param gold: Gold object, contains all the gold boundaries :param output_folder: string, path to the output folder
Output :param token_precision: Token Precision :param token_recall: Token Recall :param type_precision: Type Precision :param type_recall: Type Recall
Return Token and Type precision
Return Token and Type recall
Return Token and Type fscore
Loop over all intervals and compute token type measure.
The Token measure is computed by counting all the gold words discovered by the system. The Type measure is computed by counting all the unique type of gold words discovered by the system.
Input: :param gold_phn: the gold phone alignment
stored as an interval tree
gold_wrd (Interval Tree) – the gold word alignment stored as an interval tree
disc (list of tuples) – a list of all the discovered intervals
Output: :return: The Token Type measure