Data Annotation Tutorial

During the UbiComp / ISWC 2023 conference our group presented a tutorial on data annotation and more specific – how to create a data annotation protocol. During the event we explained different techniques for data annotation and different methodologies to collect ground truth data. Additionally we discussed different sources of bias in annotation tasks, demonstrated how to measure agreement between annotators and discussed the best practices for creating annotation guidelines. We also included recent topics such as annotation using Large Language Models (LLMs) by going through several case studies from the literature. The participants also performed annotations using some of the state-of-the-art tools for data annotations.

• All materials from the tutorial could be found on github.