Results from Each Event Individually

The following two links lead to interactive visualizations of the results for each event individually. The suggested use of the table is to drag “event” to the left column and “label” to the upper bar. Then select a viewing option, such as bar chart, and select to view “list unique values”, and then select “fscore”. The image below shows how to configure the options on the table to display the results.

png

Another view that is interesting is the heatmap, that shades cells of the table darker for better fscores, this can be configured as shown in the image below.

png

Finally, there is also the option to filter by any event or label as desired, and this can be done as shown in the following two images.

png

png

To use the interactive visualizations for the events, click the following two links.

Excerpt Based Results

Sentence Based Results

Results from Full Data

The results from all events together are shown in the following table, with both the sentence based results, and the excerpt based results.

fscore_excerpts fscore_sentences precision_excerpts precision_sentences recall_excerpts recall_sentences
label
ACCOUNT 0.646427 0.593052 0.600271 0.520519 0.701298 0.68939
EVENT 0.804812 0.724264 0.760016 0.669544 0.85576 0.788855
GRIEF 0.710274 0.626353 0.730314 0.594215 0.691717 0.66238
HERO 0.623459 0.421671 0.657407 0.458261 0.601905 0.390779
INVESTIGATION 0.646376 0.592606 0.600388 0.549435 0.700388 0.645075
JOURNEY 0.505202 0.538554 0.51264 0.552931 0.499084 0.527772
LEGAL 0.607149 0.562588 0.545679 0.517006 0.684539 0.617024
MEDIA 0.543683 0.488292 0.646586 0.608507 0.469642 0.409108
MISCELLANEOUS 0.1531 0.163613 0.358333 0.316412 0.0979628 0.111955
MOURNING 0.737217 0.65369 0.72905 0.618216 0.745644 0.694081
PERPETRATOR 0.635561 0.568497 0.607743 0.521118 0.666198 0.625522
PHOTO 0.867822 0.810145 0.962235 0.903715 0.791482 0.734191
POLICY 0.76198 0.688665 0.719699 0.624043 0.809883 0.768284
RACECULTURE 0.526538 0.472681 0.485271 0.411426 0.577077 0.555978
RESOURCES 0.714699 0.607568 0.720465 0.664053 0.710595 0.561214
SAFETY 0.699861 0.626388 0.688813 0.598168 0.71184 0.658571
SOCIALSUPPORT 0.534268 0.479476 0.596018 0.498682 0.487058 0.462509
THREAT 0.412299 0.433208 0.4785 0.509968 0.366263 0.378187
TRAUMA 0.60162 0.552743 0.57845 0.494055 0.628757 0.629151

This table identifies which f-scores, precision and recall values were the best, and compares the sentence based and excerpt based results. The colour green indicates good performance and red indicates poor performance. It is nice to see that by far MISCELLANEOUS label had the worst performance. Another thing to note, is that these results show that overall recall does fairly well. This means that the classifiers are performing well at identifying the excerpts as having the correct label. The lower precision indicates that it is returning too many results (i.e. too many false positives). This could be a good thing in this application, since the false positives may actually be true positives due to errors or inconsistencies in the annotations. Also, in this application it may be better to bring to attention more of the results rather than less, since it may be better to identify more cases relevant to the label, as opposed to missing too many of them.