Corpus Linguistics

Today's Topics

1. Linguistic Annotation [1]
2. Seminar presentation: morpho-syntactic tagging
3. [Lab] Word Sketch Engine

Annotation: a major step toward the interoperability of language resources

Annotation invovles a methodology for adding information to a document at some level—a word or phrase, paragraph or section or the entire document..... The ultimate goal is to enable interoperability among annotations for different linguistic phenomena for the same language, together with linguistic annotations applied to different languages and modalities.

Annotation Science

• Levels of corpus annotation
• Framework
• Collaboration Tools Systems
• Quality control

Annotation: what to annotate?

• Syntactic annotation
• Semantic role/Word sense/Ontological annotation
• Upper-level (pragmatic / discourse) annotation
• Emotion and opinion annotation
• Named Entity
• Temporal relation

Annotation: standardized framework?

• Lexical Markup Language (LMF);
• Lemon-models (Lemon)

Annotation: standardized framework?

Annotation: quality control

• Annotation quality testing
  1. Inter-annotator agreement
  2. Intra-annotator agreement
  3. Automatic quality testing
  4. The kappa measure

Annotation: quality control

Inter-annotator agreement

• Two (or more) annotators annotate the same text.
• How often do their analyses agree?
• Time-consuming, since time will be spent re-annotating the same text rather than annotating new text.

Annotation: quality control

Intra-annotator agreement

• How consistent is a single annotator?
• Re-annotate a text you have annotated a few months ago, assess disagreement with yourself.

Annotation: quality control

How to measure agreement between annotators?

• Simplest measure: percentage of agreement
  • But what does it mean? How good is 50% agreement?
  • Just 2 choices, e.g. distinguishing between "celestial body" and "well-known person" sense of "star": 50% is very bad.
  • 40 choices, e.g. word senses of a high frequency verb like "go": 50% not great, but not abysmal either.

Annotation: quality control

• Machine annotation and the problems

How good is a given annotation? Is it correct? Is it consistent? How can you check this for thousands of sentences? The annotation manual may easily be 50 or 100 pages long, and annotation takes a lot of time. E.g. SALSA: 20,000 sentences, about 4 years

Is there any way we can speed this up?

Conducting a corpus annotation project

Things must be taken into consideration: (Palmer and Xue, 2009)

• Target phenomena definition
• Corpus selection
• Annotation efficiency and consistency
• Annotation infrastructure
• Annotation evaluation
• The use of machine learning for preprocessing and sampling.

Annotation schemes: Morpho-Syntactic Tagging

• tagset : design criteria; granularity
• tagger: different algorithms
  • TreeTagger
  • TnT Tagger
  • Stanford Log-linear Part-Of-Speech Tagger

Annotation schemes: Syntactic structure (e.g., treebanking)

parser demo

Exercise [1]

• Online POS tagging service

For advanced use:

• Tree tagger: a language independent part-of-speech tagger.
  • Download the tool, read the document and test it with the texts in your languages.

Exercise [2]

• Sense tagging the word 'go' (BNC sentences on the first page with WSE)

Homework

Preparing for the quiz. (Antconc, BNC-WEB and WSE)

Lab session Word Sketch Engine