Units (crossing the sentence boundary) reflect the communicative function of the sentence
What is the sentence about?
What information about the topic is asserted?
subordinating, coordinating, adverbial
, and implicit
.Because [the drought reduced U.S. stockpiles], [they have more than enough storage space for their new crop], and that permits them to wait for prices to rise.
[William Gates and Paul Allen in 1975 developed an early language-housekeeper system for PCs], and [Gates became an industry billionaire six years after IBM adapted one of these versions in 1981].
Crucial issue: are the annotations correct?
(Artstein and Poesio, 2008)
validity
of the manual annotation i.e. whether the annotated categories are correct, but there is no "ground truth":
(Artstein and Poesio, 2008)
Instead measure reliability
of annotation
How can reliability be determined?
In all cases, measure of reliability is to calculate the coefficients of agreement
.
In some rare cases, there exists a "correct" annotation (gold standard
).
\[ Recall = \frac{Nb of correct found annotations}{Nb of correct expected annotations} \]
\[ Precision = \frac{Nb of correct found annotations}{Total nb of found annotations} \]
F1-score
: Harmonic mean of precision and recall or balanced \[ F1 = 2 * \frac{P*R}{P+R} \]
\(S\), \(\kappa\), and \(\pi\) measure.