SENSEVAL II System Descriptions
Also, here is a link to the new SENSEVAL web site
Page by Michael Oakes, University of Sunderland, July 2nd 2000
Last revision: October 30th 2002 Diana McCarthy
SYSTEMS GROUPED BY TASK
Czech All Words
JHU-Czech: Yarowsky, Cucerzan, Florian, Schafer, & Wicentowski
English All Words
ehu-dlist-all: Agirre & Martinez
Sussex-sel: Carroll & McCarthy
Sussex-sel-ospd: Carroll & McCarthy
Sussex-sel-ospd-ana: Carroll, McCarthy & Preiss
University of California: Chao
LIA-Sinequa_AllWords: Crestan, El-Beze & de Loupy
usm_english_tagger,usm_english_tagger2,usm_english_tagger3: Guo
Estonian All Words
JHU-Estonian: Yarowsky, Cucerzan, Florian, Schafer, & Wicentowski
ehu-dlist-all: Agirre & Martinez
ehu-dlist-best: Agirre & Martinez
UMD-SST: Resnik, Stevens & Cabezas
JHU-Basque: Yarowsky, Cucerzan, Florian, Schafer, & Wicentowski
Chinese Lexical Sample
Danish Lexical Sample
ehu-dlist-all: Agirre & Martinez
ehu-dlist-best: Agirre & Martinez
LIA-Sinequa_Lexsample: Crestan, El-Beze & de Loupy
Univ._Alicante_System: Montoyo
UMD-SST: Resnik, Stevens & Cabezas
JHU-English: Yarowsky, Cucerzan, Florian, Schafer, & Wicentowski
Italian Lexical Sample
JHU-Italian: Yarowsky, Cucerzan, Florian, Schafer, & Wicentowski
Korean Lexical Sample
Univ._Alicante_System: Montoyo
UMD-SST: Resnik, Stevens & Cabezas
JHU-Spanish: Yarowsky, Cucerzan, Florian, Schafer, & Wicentowski
Prolog Word Experts: Lager & Zinovjeva
Linköping University: Ahrenberg, Merkel & Anderson
Språkdata/Machine-Learning: Kokkinakis
Språkdata/Common-Features: Kokkinakis
UMD-SST: Resnik, Stevens & Cabezas
JHU-Swedish: Yarowsky, Cucerzan, Florian, Schafer, & Wicentowski
SYSTEMS DESCRIPTIONS (in alphabetical order of first authors).
2. Your contact details
name: Eneko Agirre & David Martinez
email: {eneko,jibmaird}@si.ehu.es
organisation: University of the Basque Country
3. Task/s: English lexical, English all-words, Basque lexical
4. Did you use any training data provided in an automatic training procedure? Yes
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This supervised system trains on the provided training data. It extracts a basic feature set:
i) local features for english: bigrams and trigrams around the target word, consisting on lemmas or word forms or parts of speech. Also a bag of lemmas constructed using the content words in a +/- 4 word window around the target.
ii) local features for Basque: being Basque an agglutinative language, part of the syntactic information is in the inflectional suffixes. We therefore have used unigrams, bigrams and trigrams of word forms, lemmas, and parts of speech including declension case and number information.
iii) global features: a bag of lemmas with the content words included in the whole context provided for the target word.
The system is based on Yarowsky's decision list. It sorts the features according to the log-likelihood value and chooses the sense of the feature with the highest value. Features occurring only once were pruned.
In the case of the English all-words task, Semcor 1.6 was used for training, via a automatically produced WordNet 1.6-1.7 map. Adjectives and adverbs were not treated in this case.
Tags P and U have not been used. There is no special treatment for multiword detection.
7. keywords: Supervised learning, decision lists, agglutinative languages.
8. URL containing additional information (optional):Click here
1. System name: ehu-dlist-best
2. Your contact details
name: Eneko Agirre & David Martinez
email: {eneko,jibmaird}@si.ehu.es
organisation: University of the Basque Country
3. Task/s: English lexical, Basque lexical
4. Did you use any training data provided in an automatic training procedure? Yes
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This supervised system is a variation of ehu-dlist-all. We used 10-fold cross-validation on the training data to select those features with a precision higher than 0.85. The tagging was only done with those features, getting a precision close to 0.85 at the cost of losing coverage.
7. keywords: Supervised learning, decision lists, agglutinative languages, feature selection.
8. URL containing additional information (optional):Click here
2. Your contact details
name: Lars Ahrenberg, Magnus Merkel, Mikael Andersson
email: {lah,magme,miand}@ida.liu.se
organisation: Dept of Computer and Information Science, Linköping University
3. Task/s: Swedish Lexical Sample Task
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)?
6. Description: Our system is based on supervised learning of contextual features for each given word, where the features include word forms, bigrams, lemmas and parts-of-speech in specific positions, and bigrams and lemmas for a full context. In disambiguation, these features are combined according to a voting scheme.
In the learning phase we consider features that have a total frequency above a set threshold (3), and calculate for each of these:
1) the relative frequency of each sense for the feature,
2) a probability measure, using a Student-t distribution to test the hypothesis, that the observed distribution for a feature is significantly different from the overall distribution of senses in the training data.
To decide the sense of a given instance we use the following algorithm:
1) if no significant feature exists, the most common sense is selected,
2) else, if for all significant features only one sense has relative frequency 1.0, that sense is selected,
3) else consider the t-values for all featuresa) for each feature, keep t-values that are above the 95% signifcance level, and single out the sense, that has the highest t-value, and give the vote to this sense.
b) order the senses according to vote,i) pick the sense with most votes,
ii) if two are tied, pick the sense with most second places,
iii) if there is still a tie,
- merge subsenses into main senses,
- pick the main sense most frequent,
- if tied, choose most frequent original subsense.
7. keywords: supervised learning, voting
8. URL containing additional information (optional):
2. Your contact details
name: Eiji Aramaki
email: aramaki@pine.kuee.kyoto-u.ac.jp
organisation: Kyoto University
3. Task/s: Japanese lexical sample (Translation Memory)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: The system selects the most similar Japanese expression in the TM, by a bottom-up, shared-memory based matching algorithm.
7. keywords: bottom-up matching, shared-memory
8. URL containing additional information (optional):
1. System name: Stanford-Titech1
2. Your contact details
name: Timothy BALDWIN
email: tim@cl.cs.titech.ac.jp
organisation: Stanford and Titech
3. Task/s: Japanese lexical sample (Translation Memory)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: The system selects the appropriate translation record based on the character-bigram-based similarity, as follows.
1. For each input and translation record, calculate the string similarity by way of Dice's coefficient over character bigrams
2. For each pair of inputs, calculate the string similarity by way of Dice's coefficient over character bigrams
3. For each input and translation record combination, calculate the maximum "linked similarity" via each other input, as the product of the input-input and input-translation record similarities
4. For each input, return the translation record for which the sum of the input-translation record string similarity and maximum linked similarity is the greatest.
No language resources were used.
7. keywords: character bigram, Dice's coefficient, input-input similarity
8. URL containing additional information (optional):
1. System name: Stanford-Titech2
2. Your contact details
name: Timothy BALDWIN
email: tim@cl.cs.titech.ac.jp
organisation: Stanford and Titech
3. Task/s: Japanese lexical sample (Translation Memory)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: The system selects the appropriate translation record
based on the
case-frame-based similarity, as follows:
1. Parse each input and translation record, and generate a "case frame" for each
2. For each input, determine that translation record which has the most similar case frame, working outwards from the target word through each case slot/predicate in order of proximity of dependence, in the case of a matching case slot, evaluate the quality of match of the filler using a thesaurus
3. Return the translation record for which the most case slot matches are produced, breaking ties according to the overall quality of match
Goi-Taikei thesaurus, and segmentation/tagging and parsing tools were employed.
7. keywords: case frame, conceptual matching, parsing
8. URL containing additional information (optional):
2. Your contact details
name: Yvonne Canning, Michael Oakes, John Tait
email: Yvonne.Canning@sunderland.ac.uk, Michael.Oakes@sunderland.ac.uk, John.Tait@sunderland.ac.uk
organisation: The University of Sunderland
3. Task/s: English All Words
4. Did you use any training data provided in an automatic training procedure? N
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? N
6. Description: We searched for collocations within a window of one word only. For instance, if the word to be disambiguated was "art" and the sentence was the "the pop art collection is owned by ...", we would have three sequences: 1. pop art, 2. pop art collection, 3. art collection. We would return the sense of the longest of these sequences to be found in the sense index. If none of these sequences were found, we would consider the word "art" in isolation. If there was just one sense in the sense index for "art", that sense would be returned, if there were no sense, then "sense unknown" would be returned. If there were more than one sense in the sense index, then either a) the first sense encountered would be returned, or b) lexical chaining (see Jeremy Ellman, "Using Roget's Thesaurus to Determine the Similarity of Texts", Ph.D. Thesis, University of Sunderland, 2000) would be used for disambiguation of nouns, verbs and adjectives. In this case, the sense tag selected would be the sense tag (of those tags which could apply to "art") most frequently assigned to the other words with the same part of speech in the same section of text.
7. keywords: collocations, lexical chains.
8. URL containing additional information (optional):
2. Your contact details
name: John Carroll, Diana McCarthy
email: {johnca,dianam}@cogs.susx.ac.uk
organisation: The University of Sussex
3. Task/s: English All Words
4. Did you use any training data provided in an automatic training procedure? N
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? Yes
6. Description: Sussex-sel was applied to the all-words task, but processed only the plain text, ignoring the supplied bracketings. The system parses the text to identify subject/verb and verb/direct object relationships, and the nouns and verbs involved are disambiguated using class-based selectional preferences acquired from unsupervised training.
The selectional preferences are acquired for subject and direct object slots. For each slot, the verb and argument head training data is obtained from grammatical relations automatically extracted from parses of the BNC produced by a shallow parser. These are then used to populate the verb and noun WordNet hypernym hierarchy with frequencies. From these frequencies, probability distributions as sets of noun classes which partition the terminal noun senses are obtained using the minimum description length principle. These are conditioned on verb classes, so only the noun data which has occurred in the relationship specified by the slot with a member of the verb class is used. For selectional preference acquisition, the members of a verb class include direct members and hyponyms which have 10 senses or less and have occurred in the BNC with a frequency of 20 or more. Bayes rule is used to obtain probabilities for the verb classes given any noun class.
7. keywords: selectional preferences, grammatical relations
8. URL containing additional information (optional):
1. System name: Sussex-sel-ospd
2. Your contact details name: John Carroll, Diana McCarthy
email: {johnca,dianam}@cogs.susx.ac.uk
organisation: The University of Sussex
3. Task/s English All Words
4. Did you use any training data provided in an automatic training procedure? N
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? Y
6. Description: The sussex system2ospd was applied to the all-words task, but processed only the plain text, ignoring the supplied bracketings. The system parses the text to identify subject/verb and verb/direct object relationships, and the nouns and verbs involved are disambiguated using class-based selectional preferences acquired from unsupervised training. This system is the same as our Sussex-sel, but we also employ Yarowsky's one sense per discourse (OSPD) heuristic provided that there is no conficting evidence for a target instance within the discourse.
The selectional preferences are acquired for subject and direct object slots. For each slot, the verb and argument head training data is obtained from grammatical relations automatically extracted from parses of the BNC produced by a shallow parser. These are then used to populate the verb and noun WordNet hypernym hierarchy with frequencies. From these frequencies, probability distributions as sets of noun classes which partition the terminal noun senses are obtained using the minimum description length principle. These are conditioned on verb classes, so only the noun data which has occurred in the relationship specified by the slot with a member of the verb class is used. For selectional preference acquisition, the members of a verb class include direct members and hyponyms which have 10 senses or less and have occurred in the BNC with a frequency of 20 or more. Bayes rule is used to obtain probabilities for the verb classes given any noun class.
7. keywords: selectional preferences, grammatical relations, one sense per discourse
8. URL containing additional information (optional):
1. System name: Sussex-sel-ospd-ana
2. Your contact details
name: John Carroll, Diana McCarthy, Judita Preiss
email: {johnca,dianam}@cogs.susx.ac.uk jp233@hermes.cam.ac.uk
organisation: The University of Sussex and the University of Sheffield
3. Task/s: English All Words
4. Did you use any training data provided in an automatic training procedure? N
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? Yes
6. Description: Sussex-sel-ospd-ana parses the plain text to identify subject/verb and verb/direct object relationships, and the nouns and verbs involved are disambiguated using class-based selectional preferences acquired from unsupervised training. We employ Yarowsky's one sense per discourse (OSPD) heuristic provided that there is no conficting evidence for a target instance within the discourse. This system is the same as our Sussex-sel-ospd, but additionally, anaphor resolution (ANA) is used during sense tagging so that evidence is taken across anaphoric links, for example disambiguating a verb which occurs with a pronoun as subject with reference to the antecedent of the pronoun.
The selectional preferences are acquired for subject and direct object slots. For each slot, the verb and argument head training data is obtained from grammatical relations automatically extracted from parses of the BNC. These are then used to populate the verb and noun WordNet hypernym hierarchy with frequencies. From these frequencies, probability distributions as sets of noun classes which partition the terminal noun senses are obtained using the minimum description length principle. These are conditioned on verb classes, so only the noun data which has occurred in the relationship specified by the slot with a member of the verb class is used. For selectional preference acquisition, the members of a verb class include direct members and hyponyms which have 10 senses or less and have occurred in the BNC with a frequency of 20 or more. Bayes rule is used to obtain probabilities for the verb classes given any noun class.
7. keywords: selectional preferences, grammatical relations, one sense per discourse, anaphor resolution
8. URL containing additional information (optional):
1. System name: none at the moment
2. Your contact details
name: Gerald Chao
email: gerald@cs.ucla.edu
organisation: Dept. of Computer Science, University of California Los Angeles
3. Task/s: English all words
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)?
6. Description: This probabilistic WSD system uses synonym permutations to form n-grams, then queries AltaVista for word counts as the basis for establishing the probabilities. These parameters are then smoothed by training data from SemCor, weighted based on both semantic distance and density. The parameter smoothing is performed by recasting the WordNet hierarchy as Bayesian networks, in a processed called Semantic Backoff. The sentential structure is then used to construct another Bayesian network, quantitated via the parameters established earlier. WSD is performed by the Maximum A Posteriori estimation, using the Join Tree inferencing algorithm. Lastly, the one-sense-per-discourse is applied to test its effectiveness.
7. keywords: Bayesian networks, semantic distance and density, parameter smoothing, MAP
8. URL containing additional information (optional):
2. Your contact details
name:Hoa Trang Dang
email: htd@linc.cis.upenn.edu
organisation: University of Pennsylvania
3. Task/s: English lexical sample (verbs only, unofficial)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: VB is a supervised word sense disambiguation system that uses a maximum entropy framework to combine linguistic contextual features from corpus instances of each verb to be tagged. The features for a verb w are:
(1) The word w, the part of speech of w, and whether or not the sentence containing w is passiveThe system computes the probablity of each sense for a test instance based on the maximum entropy model, filters out senses using the satellites, and outputs the senses that have probability within a factor of .80 of the highest probablity sense.
(2) Whether there is a sentential complement, subject, direct object, or indirect object
(3) The words (if any) in the positions of subject, direct object, indirect object, particle, prepositional complement (and its object)
(4) WordNet synsets and hypernyms for the nouns appearing in the positions in (3)
(5) A Named Entity tag (PERSON, ORGANIZATION, LOCATION) for proper nouns appearing in (3)
(6) Words at positions -2, -1, +1, +2, relative to w
(7) All keywords that occur anywhere in the context supplied for w. Keywords were chosen to minimize the entropy of the probability of a sense given the keyword, estimated as the ML probability in the training data.
7. keywords: maximum entropy model, verb arguments, selectional restrictions, collocations
8. URL containing additional information (optional):
Thanks to previous experiments, we noticed that for some words or even some synsets,
the information useful for disambiguation purpose are present in variable window
sizes around the term to be disambiguated. In order to select automatically the
appropriate window size, we have designed a mixed approach combining the SCT and
a long-range similarity measure (like in document retrieval), in some particular
cases. The similarities computed with the Cosine measure between the test sentence
and the union of examples for each sense are used to decision which synset is
the best one among those produced using SCT with window size k=3, 5 or 7. Applying
this process on SENSEVAL-1, we achieved 85.7% average precision on nouns and 72.8%
average precision on verbs.
7. keywords: Semantic Classification Tree, Semantic Classes, Cosine Similarity
Measure
8. URL containing additional information:
Click here
name: Eric Crestan
email: eric.crestan@lia.univ-avignon.fr;crestan@sinequa.com
organisation: Laboratoire Informatique d'Avignon; Sinequa
name: Marc El-Beze
email: marc.elbeze@lia.univ-avignon.fr
organisation: Laboratoire Informatique d'Avignon
name: Claude de Loupy
email: loupy@sinequa.com
organisation: Sinequa
3. Task/s (e.g. English all words): English all words
4. Did you use any training data provided in an automatic training procedure?
5. (if the answer to (4) is no) did you use any training data provided in
any way (eg as a test set for debugging)?
6. Description:
Our approach for the all-words disambiguation task is based on statistical models.
Since no training corpus has been provided, we had to revise our approach, considering
a method that needs less training data. It can be broken up into two distinctive
phases. The first phase consists in identifying the coarse Semantic Classes
(SC) related to each word in the test. For this purpose, we have applied the
Viterbi algorithm, using a trisem model, in order to select the most likely
path in the SC graph built for each sentence. The second phase consists in assigning
for each word, its most likely sense given the SC determined in the first step.
In order to train the models, we took advantage of the Semcor (release 1.6).
Moreover, we chose to apply a special treatment on words fulfilling two constraints:
To be
i) one of the most words to be disambiguated in the all-words task,
ii)
one of the words to be disambiguated in the lexical sample task.
For 2 nouns
and 4 verbs, we used the same approach as defined for the lexical sample task
(i.e. Semantic Classification Trees). Previous experiments [Loupy, 2000] using
the Semcor have shown some good
results for word SC affectation (about 90%). For this reason, we could expect
to achieve results at least as good as those obtained with an approach based
on unisem model. However, the absence of mapping between Wordnet 1.6 and 1.7
senses may have a negative effect on the final results.
7. keywords: Trisem Model, Semantic Classes, Semantic Classification Tree
8. URL containing additional
information: Click here
1. System name: UMD-UST (several versions)
2. Your contact details
name: Mona Diab, Philip Resnik
email: {mdiab,resnik}@umiacs.umd.edu
organisation: University of Maryland, College Park, Linguistics Department & UMIACS, USA
3. Task/s: English all words
4. Did you use any training data provided in an automatic training procedure? N
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? Y
6. Description: UST is an unsupervised system for word sense tagging. It exploits the translator's choice of words in a foreign language to disambiguate the senses of an ambiguous word in the target language. It requires a parallel corpus that is sentence aligned and a sense inventory for the language that needs to be sense tagged. It produces both corpora tagged with sense IDs from the sense inventory. Basically, the system assumes that the parallel corpus is token aligned. It then clusters all the words that mapped to the same orthographic form on the foreign side and uses a similarity measure on the clustered words and their corresponding senses. The assumption is that if words clustered together based on their translations then the relevant senses will get higher weights given the appropriate similarity measure.
7. keywords: parallel corpora, token alignments, WordNet, information-theoretic similarity measure
8. URL containing additional information (optional):
1. System name: TALP
2. Your contact details
name: Gerard Escudero
email: escudero@lsi.upc.es
organisation: TALP Research Center - Technical University of Catalonia
3. Task/s: English Lexical Sample Task
4. Did you use any training data provided in an automatic training procedure?
Y
6. Description:
The TALP system can be defined as Hierarchical LazyBoosting. It works as Yarowsky's
hierarchical decision lists, but using LazyBoosting instead of Decision Lists.
LazyBoosting belongs to the boosting-algorithm family set. The main idea of
boosting algorithms is to combine many simple and moderately accurate hypotheses
(weak classifiers) into a single, highly accurate classifier. More specifically,
LazyBoosting is a simple modification of AdaBoost.MH algorithm, wich consists
in reducing the feature space that is explored whenever a weak classifier is
learnt. That is, a small amount of attributes is randomly selected and the best
weak rule among then is selected.
The information used to represent examples is:
- Local information: using part-of-speech and lemmas of words placed in a 7-word window around the target word.
- Topic information: treating open-class words as a bag of words. These open-class words are placed in a 21-word window around the target word.
- Semantic domain information: using a semantic hierarchy linked to WordNet 1.6. The domain weigths depends on the words of the context, the number of senses of these words and their distribution on the context.
Multiwords have been preprocessed separately in a previous task.
Examples labeled with "U" or "P" have not been considered
in the training process. So, no test example is tagged with one of these labels.
7. keywords: hierarchical LazyBoosting, semantic domain attributes,
multiword preprocessing, AdaBoost.MH
8. URL containing additional information:
Click here
2. Your contact details
name: David Fernandez-Amoros
email: david@lsi.uned.es
organisation: UNED University (Spain)
3. Task/s: English all words
4. Did you use any training data provided in an automatic training procedure? No
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? : No
6. Description: We have used tokenized, lemmatized, stripped the stop words out of the contexts and detected person names and numbers. We have detected multiword terms as present in WordNet.
According to the file cntlist we have established a filter, discarding in the first heuristic the senses that have not appeared more than 10% in the wordnet files.
We have made a relevance matrix between words with the data from 3200
books in English from the Gutenberg Project ( Click here)
. This matrix is sensitive to the distances between words in the corpus.
With the aid of the matrix we have enriched sense descriptions and also we have used it to filter the context of the words to be disambiguated taking into account the part of speech involved.
As back-off strategies we have used the same one discarding the frequency filter and for the few words left the first sense.
7. keywords:dictionary_definitions relevance matrix
8. URL containing additional information (optional):
2. Your contact details
name: David Fernandez-Amoros
email: david@lsi.uned.es
organisation: UNED University (Spain)
3. Task/s: English all words
4. Did you use any training data provided in an automatic training procedure? No
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? : No
6. Description: We have used tokenized, lemmatized, stripped the stop words out of the contexts and detected person names and numbers. We have detected multiword terms as present in WordNet.
According to the file cntlist we have established a filter, discarding in the first heuristic the senses that have not appeared more than 10% in the wordnet files.
We have made a relevance matrix between words with the data from 3200
books in English from the Gutenberg Project (Click here).
This matrix is sensitive to the distances between words in the corpus.
With the aid of the matrix we have enriched sense descriptions and also we have used it to filter the context of the words to be disambiguated taking into account the part of speech involved.
As back-off strategies we have used the same one discarding the frequency filter and for the few words left the first sense.
7. keywords:dictionary_definitions relevance matrix
8. URL containing additional information (optional):
2. Your contact details
name: David Fernandez-Amoros
email: david@lsi.uned.es
organisation: UNED University (Spain)
3. Task/s: English lexical-sample
4. Did you use any training data provided in an automatic training procedure? No
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? : Yes
6. Description: We have used tokenized, lemmatized, stripped the stop words out of the contexts and detected person names and numbers. We have detected multiword terms as present in WordNet.
According to the file cntlist we have established a filter, discarding in the first heuristic the senses that have not appeared more than 10% in the wordnet files.
We have made a relevance matrix between words with the data from 3200
books in English from the Gutenberg Project ( Click here).
This matrix is sensitive to the distances between words in the corpus.
With the aid of the matrix we have enriched sense descriptions. We have added the information of the five first hyponyms where possible and also we have used the matrix to filter the context of the words to be disambiguated.
As back-off strategies we have used the same one discarding the frequency filter and for the few words left the first sense.
7. keywords:dictionary_definitions relevance matrix
8. URL containing additional information (optional):
2. Your contact details
name: David Fernandez-Amoros
email: david@lsi.uned.es
organisation: UNED University (Spain)
3. Task/s: English lexical-sample
4. Did you use any training data provided in an automatic training procedure? Yes
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: We have used tokenized, lemmatized, stripped the stop words out of the contexts and detected person names and numbers. We have detected multiword terms as present in WordNet.
According to the file cntlist we have established a filter, discarding in the first heuristic the senses that have not appeared more than 10% in the wordnet files.
We have made a relevance matrix between words with the data from 3200
books in English from the Gutenberg Project (Click here).
This matrix is sensitive to the distances between words in the corpus.
With the aid of the matrix we have enriched sense descriptions. We have added the information of the five first hyponyms where possible. We have added the training information to the definitions. Also we have used the matrix to filter the context of the words to be disambiguated.
As back-off strategies we have used the same one discarding the frequency filter and for the few words left the first sense.
7. keywords:dictionary_definitions relevance matrix
8. URL containing additional information (optional):
1. System name: usm_english_tagger, usm_english_tagger2, usm_english_tagger3
2. Your contact details
name: Chengming Guo
email: cmguo@cs.usm.my
organisation: Universiti Sains Malaysia
3. Task/s: English all words
4. Did you use any training data provided in an automatic training procedure? No
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? : N
6. Description: Our systems are descriptive-semantic-primitive-based, general-domain systems that do not require training or supervision.
There are three components to the systems: a machine tool level machine-tractable dictionary (MTD), a semantic distance matrix of the primitives, and a semantic tagger that uses a simple summation algorithm.
In tackling the SENSEVAL2 English all-word task, the first 7 words of the definition text of the Wordnet dictionary were first disambiguated using information from the MTD. The disambiguated version of the Wordnet dictionary was then used in handling the test data.
In the sense-tagging of the test sentences, all nonheads (words that ddi not require tagging) were removed from each sentence, leaving only the heads (words to be sense-tagged) behind. The lists of heads were then cut into chunks of three for the actural tagging process, with each chunk of three successive heads to be tagged in seconds.
The three systems are different in the number of primitives used in the MTD as well as in the semantic distance matrix. They could either be a little over 4,000 or less than 500.
7. keywords: descriptive-semantic-primitives; machine-tractable-dictionary; sense-tagging; semantic-disambiguation.
8. URL containing additional information (optional):
1. System name: IIT1, IIT2, IIT3
2. Your contact details
name: Woody Haynes
email:skhii@mindspring.com
organisation: Illinois Institute of Technology3. Task/s: English Lexical Sample (IIT1, IIT2), English All Word (IIT1, IIT2, IIT3)
4. Did you use any training data provided in an automatic training procedure? N
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? N
6. Description: For a target word, accumulate all WordNet 1.7 examples (stuff in quotes) that are related to the word's plausible synsets and any of their related synsets. (Related synsets include all ancestors for parent relations, immediate children for child relations, and transitive closure within a relation for all other relations.)
Since each example should have one of its synset words or collocations in it, it can be viewed as a mini-corpus of tagged lexical-sample instances. So, consider how well each example matches the target context. Align the context target word to the synset word in the example.
For each example word (working out from the synset word) find the closet word in the target context that is related under WordNet relations. (Two words are considered related if they have a common ancestor under a parent/child relation or have common elements in the transitive closure under another relation.)
Score the match quality of each example. The primary components of the score are (1) lexical proximity of matches of open class words, (2) exact match or POS match for closed class words (3) position shifts of matched words, (4) ordering of matched words, and (5) lexical proximity of the example synset to the candidate target word sense.
IIT2 reduces the effect of mismatches distant from the target word over IIT1. IIT3 (All Word only) restricts senses of context words to the "best" sense for words to the left of the target word before beginning the example match. It uses the IIT1 scoring methodology.
7. keywords: WordNet Examples, Untrained. Pattern Matching
8. URL containing additional information: Click here
2. Your contact details
name: Veronique Hoste
email: hoste@uia.ua.ac.be
organisation: CNTS Language Technology Group, University of Antwerp
3. Task/s: English all words
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)?
6. Description: We apply a machine learning approach for the automatic desambiguation of word senses in the English all words task. Based on the SEMCOR corpus, semantic word experts are trained for the multi-sense words. The word experts combine different types of learning algorithms, viz. memory-based learning (TiMBL) and rule induction (Ripper), which take as input different knowledge sources:
- The input for the memory-based learner is a feature vector consisting of the target word and lemma and three words to its right and left, along with a more fine-grained part-of-speech.
- A second memory-based learner is fed with a co-occurence vector consisting of possible desambiguating keywords above a predefined threshold of one sentence to the right and one to the left of the focus. This vector further contains the sense words available in the WordNet definitions.
- The rule induction method takes as input both context information and all possible keywords within the context of three sentences. Both memory-based learners are cross-validated to determine the optimal parameter settings for each word expert. On these combined classifier outputs and the WordNet most frequent sense, majority voting and weighted voting are performed. The architecture of the fairly large (ca. 2000) amount of word experts makes it also easy to parallelise the training process. For the classification of a given test item, it is first checked whether a word expert is available. If so, the best performing algorithm on the train set is applied with its optimal parameter settings to classify the item. If not, the most frequent WordNet sense is returned.
7. keywords: memory-based learning, rule induction, classifier combination, word experts
8. URL containing additional information (optional):
1. System name: Språkdata/Common-Features
2. Your contact details
name: Dimitrios Kokkinakis
email: Dimitrios.Kokkinakis@svenska.gu.se
organisation:Språkdata, Göteborg University
3. Task/s: Swedish Lexical Sample
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: The method is basically an overlap "LESK"-variant using infrormation both from the corpus and the provided dictionary.
7. keywords:
8. URL containing additional information (optional):
1. System name: Språkdata/Machine-Learning
2. Your contact details
name: Dimitrios Kokkinakis
email: Dimitrios.Kokkinakis@svenska.gu.se
organisation:Språkdata, Göteborg University
3. Task/s: Swedish Lexical Sample
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: achine learning approach for the automatic disambiguation of word senses in the Swedish lexical sample task. The memory-based learning (TiMBL) was used. The input for the learner was a feature vector consisting of 100 features. The training data was taken i)from the syntactic examples in the dictionary and ii)the training corpus.
The lemma of the head word, three tokens to its right and left, their semantic tags taken from the Swedish SIMPLE, or Named-Entity Recognition, along with a a number of content words to the left and right of the head-word. Also, the most frequent semantic tags of the whole available context. We chose the ib1 algorithm with weigheted overlap metric + gain ration weighting, parameters that gave best results in a similar exercise for Swedish conducted in the past.
7. keywords:memory-based learning, feature vector
8. URL containing additional information (optional):
2. Your contact details
name: Tadashi Kumano
email: tkumano@slt.atr.co.jp
organisation: ATR
3. Task/s: Japanese lexical sample (Translation Memory)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: The system selects the most similar TM entry based on the cosine similarity between context vectors, which were constructed from semantic features and syntactic relations of neighboring words of the target word.
A Japanese morphological analyzer (JUMAN) and parser (KNP) were employed.
7. keywords: context vector, cosine similarity, semantic feature, syntactic relation
8. URL containing additional information (optional):
1. System name: Prolog Word Experts (PWEs -- or "Peewees")
2. Your contact details
name: Torbjörn Lager, Natalia Zinovjeva
email: {torbjorn,natalia}@stp.ling.uu.se
organisation: Dept of Linguistics, Uppsala University
3. Task/s: Swedish lexical sample
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: The 'knowledge' available to a Prolog Word Expert is a sequence of
transformation rules. Such sequences are learned from sense-tagged data
using transformation-based learning. The rules are just syntactic sugar
for formulas of first order predicate logic, and the assignment of a tag
to a particular word W follows deductively from the set of formulas
corresponding to a rule sequence plus a description (also a set of
formulas) of the local context of W. The system as such consists of a
PWE compiler which translates PWE specifications into Horn clause
formulas (similar to a DCG compiler). The rest is just a matter of
performing Prolog-style constructive proofs. An online demo of a word
expert for the word "interest" is available at
Click here
7. keywords: word experts, word sense disambiguation as deduction, supervised learning, transformation-based learning
8. URL containing additional information (optional):Click here
1. System name: DIMAP Disambiguation System
2. Your contact details
name: Ken Litkowski
email: ken@clres.com
organisation: CL Research
3. Task/s English all words and English lexical sample
4. Did you use any training data provided in an automatic training procedure? N
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? N
6. Description: The CL Research disambiguation system is part of the DIMAP dictionary software, which has been designed to use any full dictionary as the basis for disambiguation. Senseval-2 results were generated using WordNet, but also using the New Oxford Dictionary of English (NODE). The disambiguation functionality exploits whatever information is made available by the lexical database. The basic design is similar to CL Research's system for Senseval-1; however, many of the disambiguation routines were not able to be reimplemented by the submission date. For Senseval-2, the implemented routines included special routines for examining multiword units and examining contextual clues (both specific word, Lesk-style use of definition content words, and subject matter analyses); syntactic constraints have not yet been employed. The official submission used only information available from WordNet. Subsequently, NODE was used on the Senseval-2 training data (which had not otherwise been used). NODE definitions were then automatically mapped into WordNet, so that results could be compared with the use of WordNet on the training data. Despite using two entirely different sense inventories, with one going through a further stage of imperfect mapping, results were quite comparable (at about 0.30 precision). With system design facilitating analysis of the contribution of different types of information, further implementation (using Senseval-1 data) will allow some useful assessments of the importance of various lexical information.
7. keywords: dictionary definitions, sense inventory mapping, context assessment, multiword units
8. URL containing additional information: Click here
2. Your contact details
name: Bernardo Magnini
email: magnini@irst.itc.it
organisation: ITC-irst
3. Task/s: English all words
4. Did you use any training data provided in an automatic training procedure? No
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? : Yes
6. Description: The system uses "semantic domains" (e.g. Medicine, Sport, Architecture) associated to wordnet synsets. The algorithm follows two steps: first a domain is chosen for a word (among those allowed by the word senses in wordnet); then a sense, among those belonging to the preferred domain, is selected. The first step (i.e. word domain disambiguation) considers, for each word, a text window of about 100 words. A score is computed which takes into account the domains of the words within the window as well as their distance from the target word. The second step (sense selection) implements a most frequent algorithm.
7. keywords: semantic domains, domain driven disambiguation.
8. URL containing additional information (optional):
1. System name: irst-eng-sample
2. Your contact details
name: Bernardo Magnini
email: magnini@irst.itc.it
organisation: ITC-irst
3. Task/s: English Lexical Sample
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: The system uses "semantic domains" (e.g. Medicine, Sport, Architecture) associated to wordnet synsets. For each word in the test data, a "domain vector" is built considering a text window around the target word. Then, the resulting domain vector is compared with the domain vectors previously acquired for each word sense from the training data, and the most similar one is selected.
7. keywords: semantic domains, domain driven disambiguation.
8. URL containing additional information (optional):
1. System name: irst-ita-sample
2. Your contact details
name: Bernardo Magnini
email: magnini@irst.itc.it
organisation: ITC-irst
3. Task/s: Italian Lexical Sample
4. Did you use any training data provided in an automatic training procedure? No
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? : Yes
6. Description: The system tries to exploit the idea of "Domain Driven Disambiguation". It is based on the annotation of the wordnet synsets with semantic domains (i.e. Medicine, Sport, Architecture). The algorithm follows two steps: first a domain is chosen for a word (among those allowed by the word senses in wordnet); then a sense, among those belonging to the preferred domain, is selected. The first step (i.e. word domain disambiguation) is based on a similarity function among semantic domains, which was trained on an English corpus. The second step (sense selection) implements a most frequent algorithm.
7. keywords: semantic domains, domain driven disambiguation, similarity.
8. URL containing additional information (optional):
1. System name: CS224N (Stanford)
2. Your contact details
name: Christopher Manning
email: manning@cs.stanford.edu
organisation: Stanford University
3. Task/s: English lexical sample, Spanish lexical sample
4. Did you use any training data provided in an automatic training procedure? Yes
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: The system used entirely supervised WSD methods, based solely on the provided training data. There were a collection of first level word sense classifiers, mainly using Naive Bayes methods, but also including vector space, n-gram, and KNN classifiers, and implementing a range of windowing, distance weighting, and smoothing techniques. These were combined by a second level classifier, which could variously use simple voting, weighted voting, or a loglinear model. The choice of combination method and the parameters of weighted voting and the features and weights of the loglinear model were chosen by crossvalidation on the training data. The first level and combined classifiers simply reported a single most likely sense choice for each test word.
7. keywords: supervised WSD, naive Bayes, classifier combination
8. URL containing additional information (optional):
2. Your contact details
name: Rada Mihalcea
email: rada@seas.smu.edu
organisation: SMU, Dallas, TX
3. Task/s: English, all words.
4. Did you use any training data provided in an automatic training procedure? Y (SemCor)
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: The semantic disambiguation of a word is performed based
on its relations with the preceding and succeeding words. A very large corpus
of word-word relations, and the corresponding senses, has been created off-line,
using:
1) The examples from WordNet 1.7, in which the words from the synsets they correspond to are disambiguated.
2) SemCor - which was translated such that it points to senses from WordNet 1.7.
3) A large additional set of sense tagged word-word pairs, generated from the pairs created at step 1 and 2, based on a set of heuristics.
The recall of this algorithm is not 100%, and therefore we have applied a cache-like methodology to propagate senses of disambiguated words to the still ambiguous words found in the immediate context.
For the few words which are still ambiguous at this point, we assign the most frequent sense from WordNet.
(An initial,
simpler version of this algorithm is described in "An Iterative Approach
to Word Sense Disambiguation", Mihalcea & Moldovan, in Proceedings
of Flairs-2000)
7. keywords: sense tagged word-word pairs, dictionary examples, generated
corpus
8. URL containing additional information (optional):
2. Your contact details
name: Rada Mihalcea
email: rada@seas.smu.edu
organisation: SMU, Dallas, TX
3. Task/s: English lexical sample
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: There are four main steps in the algorithm we have used
during the lexical sample task:
1) The data is preprocessed: SGML tags are eliminated, the text is tokenized, part of speech tagged and Named Entities are identified.
2) Compound concepts are identified: we determine the maximum sequence of words which form a compound word defined in WordNet. The training and
test data is split based on the word to be tagged. For example, the context examples containing the verb "dress down" are separated from the examples
containing only "dress". Words which are monosemous are eliminated at this step, as well as words which can be tagged as proper nouns (if they are tagged as such by the part of speech tagger and if they have a role identified by the Named Entity recognizer).3) We automatically extract "patterns" (using a set of heuristics) for each ambiguous word, based on WordNet examples (with the synset words disambiguated), SemCor examples and the training examples provided. The patterns are validated on the training data, and we keep only those which are 100% accurate. The patterns are then applied on the test data. Only a few instances can be disambiguated this way, but with high confidence: previous experiments have shown that high accuracy is obtained with this procedure.
4) This is the main step of the algorithm and it disambiguates all ambiguous instances which have not been previously disambiguated. We use an instance based learning algorithm and a large pool of features that are actively selected. The learner is trained on the training data provided and then applied on the test instances.
7. keywords: instance based learning, dictionary examples, sense tagged
corpus.
8. URL containing additional information (optional):
1. System name: Univ._Alicante_System
2. Your contact details
name: Andrés Montoyo and Armando Suarez
email: montoyo@dlsi.ua.es
organisation: Universidad de Alicante
3. Task/s: English Lexical Sample Task, Spanish Lexical Sample Task
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: The Univ._Alicante_System has used 2 different methods to the lexical sample task:
1) Knowledge-driven Method. Firts method resolves the lexical ambiguity of nouns in the test data, and although it relies on the semantic relations (Hypernymy and Hyponymy) and the hierarchic organization of WordNet, it does not, however, require any sort of training process, no hand-coding of lexical entries, nor the hand-tagging of texts.
2) Corpus-driven Method. Second method disambiguates all ambiguous verbs and adjectives instances in the test data. This algorithm implements a supervised learning method (Maximum Entropy Probability Models) consisting of the estimation of functions for classifying word senses by learning on a training data provided and then applied on the test instances.
7. keywords:Taxonomy_WordNet, Untrained_nouns, hybrid model
8. URL containing additional information (optional):
2. Your contact details
name: Masaki Murata
email: murata@crl.go.jp
organisation: Communications Research Laboratory
3. Task/s: Japanese lexical sample (Dictionary)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description (250 words max): We used a machine learning technique for constructing the WSD system. The features used in the model is outputs of morphological and syntactic analysis. The learning algorithm is support vector machine.
7. keywords: support vector machine, morphological analysis, syntactic analysis
8. URL containing additional information (optional):
2. Your contact details
name: Masaki Murata
email: murata@crl.go.jp
organisation: Communications Research Laboratory
3. Task/s: Japanese lexical sample (Dictionary)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: We used a machine learning technique for constructing
the WSD system. The features used in the model is outputs of morphological and
syntactic analysis. We used simple Bayes for learning.
7. keywords: simple Bayes, morphological analysis, syntactic analysis
8. URL containing additional information (optional):
2. Your contact details
name: Masaki Murata
email: murata@crl.go.jp
organisation: Communications Research Laboratory
3. Task/s: Japanese lexical sample (Dictionary)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: We used a machine learning technique for constructing the WSD system. The features used in the model is outputs of morphological and syntactic analysis. We used a hybrid model of support vector machine and simple Bayes for learning.
7. keywords: support vector machine, simple Bayes, hybrid model
8. URL containing additional information (optional):
2. Your contact details
name: Masaki Murata
email: murata@crl.go.jp
organisation: Communications Research Laboratory
3. Task/s: Japanese lexical sample (Dictionary)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: We used a machine learning technique for constructing
the WSD system. The features used in the model are the outputs of morphological
and
syntactic analysis. We used a hybrid model of two kinds of support vector machines
and two kinds of simple Bayes for learning.
7. keywords: support vector machine, simple Bayes, hybrid of 4 models
8. URL containing additional information (optional):
2. Your contact details
name: Ted Pedersen
email: tpederse@d.umn.edu
organisation: University of Minnesota Duluth
3. Task: English Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system takes a supervised learning approach to word sense disambiguation, where three Naive Bayesian classifiers are induced from sense-tagged training examples. A weighted vote is taken among these to assign senses to test examples. No information from WordNet is utilized by this system.
Each Naive Bayesian classifier is based on a different set of features that are identified in a filtering step prior to learning.
The first feature set is based on bigrams (two word sequences) that meet the following criteria:
1) occur 2 or more times and
2) have a log-likelihood ratio >= to 6.635 (i.e.,
p=.01) and
3) are not made up of stop-listed words.
The second feature set is based on unigrams (one word sequences) that meet the following criteria:
1)
occur 5 or more times and
2) are not found on the stop-list.
The third feature set is based on bigrams that may include one intervening word that is ignored and that meet the following criteria:
1) occur 2 or more times and
2) have a
log-likelihood ratio >= to 2.706 (i.e., p=.1) and
3) are not made up of stop-listed
words and
4) include the word to be disambiguated
A Naive Bayesian classifier is learned based on each feature set. When presented with a test example, each classifier assigns a probability to each possible sense. These probabilities are summed and sense with the largest value is assigned.
This is loosely based on the NAACL-00 paper "A Simple Approach to Building Ensembles of Naive Bayesian Classifiers for Word Sense Disambiguation" by Ted Pedersen.
7. keywords: supervised learning, Naive Bayesian classifier, ensemble
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details
name: Ted Pedersen
email: tpederse@d.umn.edu
organisation: University of Minnesota Duluth
3. Task: English Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system takes a supervised learning approach to word sense disambiguation, where a decision tree is induced from sense-tagged training examples and then used to assign senses to the test examples. No information from WordNet is utilized by this system.
This system uses a filter to perform feature identification prior to learning. All bigrams (two word sequences) that meet the following criteria form a set of candidate features:
1) occur more
than 2 times and
2) have a log-likelihood ratio >= to 6.635 (i.e., p=.01) and
3) are not made up of stop-listed words.
The training examples are converted into feature vectors, where each feature represents whether a candidate feature occurs in the context of a specific training example.
The feature vectors are the input to the J48 learning algorithm, the Weka implementation of the C4.5 decision tree learner. The parameter settings for pruning are C=0.25 (a confidence threshold) and M=2 (the number of training examples that must be covered by each leaf in the tree).
The decision tree learner is "bagged". The training examples are sampled ten times (with replacement) and a decision tree is learned for each sample. Each test example is assigned a sense based on a vote taken from among the learned trees.
This is based on the NAACL-01 paper "A Decision Tree of Bigrams is an Accurate Predictor of Word Sense" by Ted Pedersen. The use of bagging and a stop list is new for Senseval.
7. keywords: supervised learning, decision tree of bigrams, bagging
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details
name: Ted Pedersen
email: tpederse@d.umn.edu
organisation: University of Minnesota Duluth
3. Task: English Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system is identical to duluth1, except that rather than learning three Naive Bayesian classifiers from three different feature sets, it learns three bagged decision trees from the three feature sets. When presented with a test example, each decision tree outputs probabilities for each possible sense. These probabilities are summed and the sense with the maximum value is assigned to the test example. No information from WordNet is utilized by this system.
Note that a Naive Bayesian classifier has no "internal" feature selection mechanism, and accepts all features provided by the filtering step. The decision tree learner performs its own feature selection based on the gain ratio, which measures how well a feature partitions the training examples into senses.
The bagging process for each decision tree is as described in duluth2, and the features used as the basis for each decision tree are the same as in duluth1.
7. keywords: supervised learning, bagged decision trees, ensemble
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details
name: Ted Pedersen
email: tpederse@d.umn.edu organisation: University of Minnesota Duluth
3. Task: English Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system takes a supervised learning approach to word sense disambiguation, where a Naive Bayesian classifier is learned from sense-tagged training examples. No information from WordNet is utilized by this system.
The Naive Bayesian classifier is based on a set of features that consists of unigrams (one word sequences) that are identified in a filtering set prior to learning. These features must meet the following criteria:
1) occur 5 or
more times and
2) are not found on the stop-list.
Such unigrams form a set of features. The training examples are converted into feature vectors, where each feature represents whether or not a unigram occurs in the context of a specific training example. These features vectors are used to make the estimates of the parameters of the Naive Bayesian classifier.
When presented with a test example, the Naive Bayesian classifier will output the probability associated with each sense. The sense with the highest probability is assigned to the test example.
This system implements a standard benchmark, the Naive Bayesian classifier based on a bag of words feature set.
7. keywords: supervised learning, Naive Bayesian classifier, bag of words
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details name: Ted Pedersen
email: tpederse@d.umn.edu
organisation: University of Minnesota Duluth
3. Task: English Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system takes a supervised learning approach to word sense disambiguation, where a decision tree is induced from sense-tagged training examples. This system is identical to duluth2, except that it relies on a different feature set. No information from WordNet is utilized by this system.
This system uses a filter to perform feature identification prior to learning. Two different kinds of bigrams are identified as candidate features. The first is a consecutive two word sequence that meets the following criteria:
1) occur 2 or more times and
2) have a log-likelihood ratio >= to 6.635 (i.e.,
p=.01) and
3) are not made up of stop-listed words.
The second is a non-consecutive two word sequence, where there may be zero or one intervening word that is ignored. Such bigrams must meet the following criteria:
1) occur 2 or more times and
2) have a log-likelihood ratio >= to 2.706 (i.e., p=.1) and
3) are not made
up of stop-listed words and
4) include the word to be disambiguated
The process of converting the training examples into feature vectors, bagging the decision tree, and making sense assignments is identical to duluth2.
This is loosely based on the NAACL-01 paper "A Decision Tree of Bigrams is an Accurate Predictor of Word Sense" by Ted Pedersen.
7. keywords: supervised learning, decision tree of bigrams, bagging
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details
name: Ted Pedersen
email: tpederse@d.umn.edu
organisation: University of Minnesota Duluth
3. Task: Spanish Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system takes a supervised learning approach to word sense disambiguation, where three Naive Bayesian classifiers are induced from sense-tagged training examples. A weighted vote is taken among these to assign senses to test examples. No information from WordNet is utilized by this system.
Each Naive Bayesian classifier is based on a different set of features that are identified in a filtering step prior to learning.
The first feature set is based on bigrams (two word sequences) that meet the following criteria:
1) occur 2 or more times and
2) have a log-likelihood ratio >= to 6.635 (i.e.,
p=.01) and
3) are not made up of stop-listed words.
The second feature set is based on unigrams (one word sequences) that meet the following criteria:
1)
occur 5 or more times and
2) are not found on the stop-list.
The third feature set is based on bigrams that may include one intervening word that is ignored and that meet the following criteria:
1) occur 2 or more times and
2) have a
log-likelihood ratio >= to 2.706 (i.e., p=.1) and
3) are not made up of stop-listed
words and
4) include the word to be disambiguated
A Naive Bayesian classifier is learned based on each feature set. When presented with a test example, each classifier assigns a probability to each possible sense. These probabilities are summed and sense with the largest value is assigned.
This is loosely based on the NAACL-00 paper "A Simple Approach to Building Ensembles of Naive Bayesian Classifiers for Word Sense Disambiguation" by Ted Pedersen.
This is the same approach as taken in duluth1 for English. The only difference is in the stop list.
7. keywords: supervised learning, Naive Bayesian classifier, ensemble
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details name: Ted Pedersen
email: tpederse@d.umn.edu organisation: University of Minnesota Duluth
3. Task: Spanish Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system takes a supervised learning approach to word sense disambiguation, where a decision tree is induced from sense-tagged training examples and then used to assign senses to the test examples. No information from WordNet is utilized by this system.
This system uses a filter to perform feature identification prior to learning. All bigrams (two word sequences) that meet the following criteria form a set of candidate features:
1) occur more
than 2 times and
2) have a log-likelihood ratio >= to 6.635 (i.e., p=.01) and
3) are not made up of stop-listed words.
The training examples are converted into feature vectors, where each feature represents whether a candidate feature occurs in the context of a specific training example.
The feature vectors are the input to the J48 learning algorithm, the Weka implementation of the C4.5 decision tree learner. The parameter settings for pruning are C=0.25 (a confidence threshold) and M=2 (the number of training examples that must be covered by each leaf in the tree).
The decision tree learner is "bagged". The training examples are sampled ten times (with replacement) and a decision tree is learned for each sample. Each test example is assigned a sense based on a vote taken from among the learned trees.
This is based on the NAACL-01 paper "A Decision Tree of Bigrams is an Accurate Predictor of Word Sense" by Ted Pedersen. The use of bagging and a stop list is new for Senseval.
This is the same approach as taken in duluth2 for English. The only difference is in the stop list.
7. keywords: supervised learning, decision tree of bigrams, bagging
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details
name: Ted Pedersen
email: tpederse@d.umn.edu
organisation: University of Minnesota Duluth
3. Task: Spanish Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system is identical to duluth6, except that rather than learning three Naive Bayesian classifiers from three different feature sets, it learns three bagged decision trees from the three feature sets. When presented with a test example, each decision tree outputs probabilities for each possible sense. These probabilities are summed and the sense with the maximum value is assigned to the test example. No information from WordNet is utilized by this system.
Note that a Naive Bayesian classifier has no "internal" feature selection mechanism, and accepts all features provided by the filtering step. The decision tree learner performs its own feature selection based on the gain ratio, which measures how well a feature partitions the training examples into senses.
This is the same approach as taken in duluth3 for English. The only difference is in the stop list.
7. keywords: supervised learning, bagged decision trees, ensemble
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details
name: Ted Pedersen
email: tpederse@d.umn.edu
organisation: University of Minnesota Duluth
3. Task: Spanish Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system takes a supervised learning approach to word sense disambiguation, where a Naive Bayesian classifier is learned from sense-tagged training examples. No information from WordNet is utilized by this system.
The Naive Bayesian classifier is based on a set of features that consists of unigrams (one word sequences) that are identified in a filtering set prior to learning. These features must meet the following criteria:
1) occur 2 or
more times and
2) are not found on the stop-list.
Such unigrams form a set of features. The training examples are converted into feature vectors, where each feature represents whether or not a unigram occurs in the context of a specific training example. These features vectors are used to make the estimates of the parameters of the Naive Bayesian classifier.
When presented with a test example, the Naive Bayesian classifier will output the probability associated with each sense. The sense with the highest probability is assigned to the test example.
This system implements a standard benchmark, the Naive Bayesian classifier based on a bag of words feature set.
This is the same approach as taken in duluth4 for English. The only differences are in the stop list and the value of the frequency cut off.
7. keywords: supervised learning, Naive Bayesian classifier, bag of words
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details
name: Ted Pedersen
email: tpederse@d.umn.edu
organisation: University of Minnesota Duluth
3. Task: Spanish Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system takes a supervised learning approach to word sense disambiguation, where a decision tree is induced from sense-tagged training examples. This system is identical to duluth7, except that it relies on a different feature set. No information from WordNet is utilized by this system.
This system uses a filter to perform feature identification prior to learning. Two different kinds of bigrams are identified as candidate features. The first is a consecutive two word sequence that meets the following criteria:
1) occur 2 or more times and
2) have a log-likelihood ratio >= to 6.635 (i.e.,
p=.01) and
3) are not made up of stop-listed words.
The second is a non-consecutive two word sequence, where there may be zero or one intervening word that is ignored. Such bigrams must meet the following criteria:
1) occur 2 or more times and
2) have a log-likelihood ratio >= to 2.706 (i.e., p=.1) and
3) are not made
up of stop-listed words and
4) include the word to be disambiguated
This is the same approach as taken in duluth5 for English. The only difference is in the stop list.
This is loosely based on the NAACL-01 paper "A Decision Tree of Bigrams is an Accurate Predictor of Word Sense" by Ted Pedersen.
7. keywords: supervised learning, decision tree of bigrams, bagging
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details
name: Ted Pedersen
email: tpederse@d.umn.edu
organisation: University of Minnesota Duluth
3. Task: English Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system takes a supervised learning approach to word sense disambiguation, where three different classifiers are induced from sense-tagged training examples. Each classifier is based on the same feature set. A weighted vote is taken among these classifiers to assign senses to test examples. No information from WordNet is utilized by this system.
The three classifiers are a bagged J48 decision tree, a Naive Bayesian classifier, and a nearest neighbor classifier (IBk).
This system uses a filter to perform feature identification prior to learning. All non-consecutive bigrams (that may include zero, one, or two intervening words that are ingored) and that meet the following criteria form a set of candidate features:
1) occur more than 2 times and
2)
have a log-likelihood ratio >= to 10.827 (i.e., p=.001) and
3) are not made
up of stop-listed words.
The training examples are converted into feature vectors, where each feature represents whether a candidate feature occurs in the context of a specific training example.
The feature vectors are the input to the J48 learning algorithm, the IBk nearest neighbor learner (where the number of neighbors k=1), and a Naive Bayesian classifier. When presented with a test example, each classifier outputs a probability for each possible sense. These are summed and the sense with the maximum probability is assigned to a test example.
7. keywords: supervised learning, ensemble
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details
name: Ted Pedersen
email: tpederse@d.umn.edu
organisation: University of Minnesota Duluth
3. Task: English Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system takes a supervised learning approach to word sense disambiguation, where a decision stump is induced from sense-tagged training examples. This system is identical to duluth5, except that it relies on a different learning algorithm. Rather than learned a bagged decision tree (as duluth5 does) this system simply learns a decision stump, a one node decision tree.
The features used are the same as duluth5. This system provides a baseline that can be used to compare the benefits of learning an entire decision tree (duluth5) versus identifying a single node tree (duluthB).
This system is motivated by the relative success of decision stumps as reported in the NAACL-01 paper "A Decision Tree of Bigrams is an Accurate Predictor of Word Sense" by Ted Pedersen.
7. keywords: supervised learning, decision stumps
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details name: Ted Pedersen
email: tpederse@d.umn.edu
organisation: University of Minnesota Duluth
3. Task: English Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system takes a supervised learning approach to word sense disambiguation, where the results of the systems duluth1, duluth2, duluth3, duluth4, duluth5, duluthA, and duluthB are combined into an ensemble. Each of those systems outputs probabilities for each sense when presented with a test example, so all of these are summed together and the sense with the maximum probability is assigned to a test example.
7. keywords: supervised learning, ensemble
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details
name: Ted Pedersen
email: tpederse@d.umn.edu
organisation: University of Minnesota Duluth
3. Task: Spanish Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system takes a supervised learning approach to word sense disambiguation, where three different classifiers are induced from sense-tagged training examples. Each classifier is based on the same feature set. A weighted vote is taken among these classifiers to assign senses to test examples. No information from WordNet is utilized by this system.
The three classifiers are a bagged J48 decision tree, a Naive Bayesian classifier, and a nearest neighbor classifier (IBk).
This system uses a filter to perform feature identification prior to learning. All non-consecutive bigrams (that may include zero, one, or two intervening words that are ingored) and that meet the following criteria form a set of candidate features:
1) occur more than 2 times and
2)
have a log-likelihood ratio >= to 0.00 and
3) are not made up of stop-listed
words.
The training examples are converted into feature vectors, where each feature represents whether a candidate feature occurs in the context of a specific training example.
The feature vectors are the input to the J48 learning algorithm, the IBk nearest neighbor learner (where the number of neighbors k=1), and a Naive Bayesian classifier. When presented with a test example, each classifier outputs a probability for each possible sense. These are summed and the sense with the maximum probability is assigned to a test example.
This is the same approach as taken in duluthA for English. The only differences are in the stop list and in the setting of the significance value for the log-likelihood ratio.
7. keywords: supervised learning, ensemble
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details name: Ted Pedersen
email: tpederse@d.umn.edu
organisation: University of Minnesota Duluth
3. Task: Spanish Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system takes a supervised learning approach to word sense disambiguation, where a decision stump is induced from sense-tagged training examples. This system is identical to duluth10, except that it relies on a different learning algorithm. Rather than learned a bagged decision tree (as duluth10 does) this system simply learns a decision stump, a one node decision tree.
The features used are the same as duluth10. This system provides a baseline that can be used to compare the benefits of learning an entire decision tree (duluth10) versus identifying a single node tree (duluthB).
This is the same approach as taken in duluthB for English. The only difference is in the stop list.
This system is motivated by the relative success of decision stumps as reported in the NAACL-01 paper "A Decision Tree of Bigrams is an Accurate Predictor of Word Sense" by Ted Pedersen.
7. keywords: supervised learning, decision stumps
8. URL containing additional information: Complete source code and documentation
for this system will be available by the end of August 2001 at : Click here
2. Your contact details
name: Ted Pedersen
email: tpederse@d.umn.edu
organisation: University of Minnesota Duluth
3. Task: Spanish Lexical Sample
4. Did you use any training data provided in an automatic training procedure? YES
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system takes a supervised learning approach to word sense disambiguation, where the results of the systems duluth6, duluth7, duluth8, duluth9, duluth10, duluthX, and duluthY are combined into an ensemble. Each of those systems outputs probabilities for each sense when presented with a test example, so all of these are summed together and the sense with the maximum probability is assigned to a test example.
This is the same approach as taken in duluthC for English.
7. keywords: supervised learning, ensemble
8. URL containing additional information: Complete source code and documentation for this system will be available by the end of August 2001 at : this site
2. Your contact details
name: Judita Preiss
email: Judita.Preiss@cl.cam.ac.uk
organisation: University of Sheffield
3. Task/s: English all words
4. Did you use any training data provided in an automatic training procedure? N
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? No training data was provided.
6. Description (250 words max): Our Word Sense Disambiguation (WSD) System was entered for the English all words task in Senseval 2001. The system makes use of the WordNet 1.7 hierarchy and an anaphora resolution algorithm to assign precisely one sense to each noun from the text. (The algorithm is only capable of assigning a sense to a noun which occurred in WordNet.)
A distance function based on the WordNet noun hierarchy forms the core of the WSD algorithm. We assign a weight to every node in WordNet; this is proportional to the number of descendants of the node. By combining appropriate weights, we can derive the `distance' between any two noun senses. For a chosen path of senses (this consists of one sense for each noun in the text) we sum all the pairwise distances, giving us an `energy' value for the path. This energy is minimized using simulated annealing. (See Preiss 2001 for a more detailed explanation.)
This method is combined with an anaphora resolution algorithm (Kennedy and Boguraev 1996). Resolving pronouns and replacing them in the text with their antecedents leads to the repetition of some words that are likely to be a ``topic stamp'' (Boguraev et al. 1998). The senses of the repeated words are tied together, thus making any change of sense more noticeable (in the resulting energy value). This leads to a more accurate disambiguation of topic words which in turn lead to an increased disambiguation accuracy of other nouns.
7. keywords: WSD, conceptual distance, anaphora resolution
8. URL containing additional information:
Click here
2. Your contact details
name: Philip Resnik, Jessica Stevens, Clara I. Cabezas
email: {resnik,clarac,stevenjc}@umiacs.umd.edu
organization: University of Maryland, College Park, Linguistics Department & UMIACS, USA
Tasks: English lexical sample (official), Spanish lexical sample (official), Swedish lexical sample (official), Basque lexical sample (unofficial)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: SST is a supervised word sense tagger. This tagger uses support vector machine (SVM) learning to build classifiers from the training data. The IDF-weighted feature vector contains both wide co-occurrence and local collocation features. Wide co-occurrences include all of the words in each instance whereas local collocation uses a window consisting of the 3 tokens immediately before and after the word to be tagged (i.e. features include left_wd3, left_wd2, left_wd1, right_wd1, right_wd2, right_wd3). Training and test data were automatically cleaned and tokenized before being submitted to training and classification. Classification produces a confidence score for every sense; our answers include just the highest scoring sense for each instance.
7. keywords: Support vector machine, SVM, collocations.
8. URL containing additional information (optional):
2. Your contact details
name: Hee-Cheol Seo, Sang-Zoo Lee, Hae-Chang Rim.
email: {hcseo, zoo, rim}@nlp.korea.ac.kr
organisation: Korea University
3. Task/s: English lexical
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)?
6. Description: We used the classification information model where local contexts, topical contexts and bigram contexts are used as features in order to decide the sense of a polysemous word in a context.
*. Feature
A local context
consists of the following features for all words within a window.
Word_Position
: a word and its position
Word_POS : a word and its part-of-speech.
POS_position
: the part-of-speech and position of a word.
The window size of +-3 words was
empirically chosen.
A topical context consists of the following features for
all open-class words within a window.
Word : an open-class word
The window size
of +-1 sentences was empirically chosen.
A bigram context consists of the following
features for all word pairs within a window.
Word+Word : i-th word and j-th
word (i!=j)
Word+POS : i-th word and j-th part-of-speech (i!=j)
POS+Word : i-th
part-of-speech and j-th word (i!=j)
The window size of +-2 words was empirically
chosen.
*. Procedure of sense disambiguation
(1) Filter out senses using the
satellite features.
(2) Disambiguate word sense using the classification information
model.
*. Classification information model(CIM)
CIM disambiguates word sense
considering the discrimination score(DS) of features.
The DS of a feature is
the sum of relevance scores between the feature and each sense.
The relevance
score between a feature and a sense is proportion to the conditional probability
of the feature given the sense.
The equations that we used are in the following web site.
7. keywords: Classification Information model, local context, topical context, bigram context
8. URL containing additional information (optional): Click here
2. Your contact details name: Hee-Cheol Seo, Sang-Zoo Lee, Hae-Chang Rim.
email: {hcseo, zoo, holee, rim}@nlp.korea.ac.kr
organisation: Korea University
3. Task/s: Korean lexical
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)?
6. Description: We used the classification information model where local contexts, topical contexts and bigram contexts are used as features in order to decide the sense of a polysemous word in a context.
*. Feature
-> An "eojeol"
in Korean is a spacing unit and consists of one or more morphemes.
A local context
consists of the following features for all words within a window.
Morpheme_Position
: a morpheme and its position.
Morpheme_POS : a morpheme and its part-of-speech.
POS_Position : the part-of-speech and position of a morpheme
Eojeol_Position
: an "eojeol" and its position.
The window size of +3/-2 words was empirically
chosen.
A topical context consists of the following features for all open-class
morphemes within a window.
Morpheme : an open-class morpheme
The window size
of all sentences was empirically chosen.
A bigram context consists of the following
features for all word pairs within a window.
Morpheme+Morpheme : i-th morpheme
and j-th morpheme (i!=j)
Morpheme+POS : i-th morpheme and j-th part-of-speech
(i!=j)
POS+POS : i-th part-of-speech and j-th part-of-speech (i!=j)
Eojeol+Eojeol
: i-th "eojeol" and j-th "eojeol" (i!=j)
The window size of +3/-2 words was
empirically chosen.
*. Classification information model(CIM)
CIM disambiguates
word sense considering the discrimination score(DS) of features.
The DS of a
feature is the sum of relevance scores between the feature and each sense.
The
relevance score between a feature and a sense is proportion to the conditional
probability of the feature given the sense.
The equations that we used are in the following web site.
7. keywords: Classification Information model, local context, topical context, bigram context
8. URL containing additional information: Click here
2. Your contact details
name: Hiroyuki Shinnou
email: shinnou@dse.ibaraki.ac.jp
organisation: Ibaraki University
3. Task/s: Japanese lexical sample (Translation Memory)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: We manually constructed a training data from newspaper
articles, 170 instances for each entry word. Features were collected in 7-word
window around the target word, and decision list method was used for learning.
7. keywords: training data, 7-word window, decision list
8. URL containing additional information (optional):
1. System name: WASPS-Workbench
2. Your contact details
name: David Tugwell
email: David.Tugwell@itri.brighton.ac.uk
organisation: ITRI, University of Brighton
3. Task/s: English lexical sample
4. Did you use any training data provided in an automatic training procedure? N
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? N (though we downloaded the training data to find out which words were in the evaluation set).
6. Description: The WASPS-Workbench is a browser-based tool which integrates lexicography and automatic WSD for the benefit of both parties. The user enters a word to be analyzed and the Workbench calculates a "Word-Sketch": a page of statistically significant collocation patterns for that word (currently based on the BNC). On the basis of these patterns, the user then draws up a sense inventory and assigns senses to particular patterns. These assignments are then used as seeds for a bootstrapping algorithm (a la Yarowsky, 1995) which disambiguates the whole corpus. The result for the lexicographer is a number of "Sense-Sketches", showing significant patterns for the individual senses of the word, while for automatic WSD we have a decision list of clues for sense disambiguation, consisting of grammatical relation patterns, words-in-context, and n-grams.
For the SENSEVAL task, we had to assign the senses from a fixed inventory (here WordNet). The disadvantage of this is that we are often forced to make difficult, if not impossible, decisions in distingishing between senses. Also as we do not use the training data, we have no knowledge of the relative frequencies of the different senses.
The task of assigning senses to patterns was shared between the entrant and one paid assistant who had never used the system before. For the assistant, the length of the interaction with the system ranged from 3 to 25 mins, with an average under 15 mins.
Due to time constraints we only managed to submit results for nouns within the deadline.
7. keywords: lexicographic input, decision list
8. URL containing additional information (optional):
2. Your contact details
name: Kiyotaka Uchimoto
email: uchimoto@crl.go.jp
organisation: CRL and NYU
3. Task/s: Japanese lexical sample (Translation Memory)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: Given an input sentence, our system returns an entry
number in TM or a translation of head word (or phrase). Our system returns a
TM entry
when the similarity between Japanese example of a TM entry and an input sentence
is over a threshold. The similarity is calculated by considering an agreement
between Japanese examples in TM and an input sentence.
Otherwise, our system returns a translation as follows:
1. TM entries are classified according to the English head word after reinforced by a bilingual corpus.
2. For each cluster, similar examples to Japanese and English examples of TM in the cluster are collected from monolingual corpora (newspaper articles). Similar examples are defined as sentences which share the head word and several words around it.
3. Given a sentence, our system outputs the head word in the closest cluster to the given sentence. The closest cluster is selected based on machine learning systems such as SVM. The features used in the SENSEVAL2 formal run were mainly as follows:1. Several words on the left and on the right of the head word in a given sentence, and their POS assigned by the morphological analyzer JUMAN.
2. N-grams including the head word in a given sentence.
3. English head word in the cluster where the longest string in a given sentence is found, and the length.
4. Frequencies and existence of Japanese content words in a given sentence and those of translations of the content words in each cluster.
7. keywords: SVM, ME, decision list, simple bayes
8. URL containing additional information (optional):
2. Your contact details
name: Kadri Vider
email: kadriv@ut.ee
organisation: University of Tartu
3. Task/s Estonian all words
4. Did you use any training data provided in an automatic training procedure? Yes
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: Our system bases on Eneko Agirre and German Rigau paper "Word Sense Disambiguation Using Conceptual Density": "The method relies on the use of the wide-coverage noun-taxonomy of WordNet and the notion of conceptual distance among concepts, captured by a Conceptual Density formula developed for this purpose." Our idea was to test this method not only for Estonian nouns, but for verbs and phrases as well. System is created and developed by Kaarel Kaljurand.
7. keywords: wordnet-based, conceptual density
8. URL containing additional information: Click here - in Estonian, sorry!
2. Your contact details
name: XXX
email: XXX
organisation: XXX
3. Task/s: Japanese lexical sample
4. Did you use any training data provided in an automatic training procedure? N
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? N
6. Description: A commercial MT system was utilized as it was. The MT
system employs a translation dictionary and standard Japanese and English grammars,
and performs morpho-syntactic and semantic analyses of source (Japanese) sentences.
7. keywords: machine translation system, translation dictionary, semantic analysis
8. URL containing additional information (optional):
2. Your contact details
name: XXX
email: XXX
organisation: XXX
3. Task/s: Japanese lexical sample (Translation Memory)
4. Did you use any training data provided in an automatic training procedure? N
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? N
6. Description (250 words max): A commercial MT system was utilized as it was. The MT system is based on an interlingua-approach.
7. keywords: machine translation system, interlingua
8. URL containing additional information (optional):
2. Your contact details
name: XXX
email: XXX
organisation: XXX
3. Task/s: Japanese lexical sample (Translation Memory)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: A sentence (TM entries for learning, and an input for testing) is morphologically analyzed and converted into a semantic tag sequence, and maximum entropy method was used for learning.
7. keywords: morphological analysis, semantic tag, maximum entropy
8. URL containing additional information (optional):
2. Your contact details
name: Yutaka Yagi
email: yutaka@cl.cs.titech.ac.jp
organisation: Tokyo Institute of Technology
3. Task/s: Japanese dictionary-based task (lexical sample)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: We learned decision lists from the training data. The features used in decision lists are content words and part-of-speech tags in a window.
In Iwanami Kokugo Jiten (a Japanese dictionary), sense inventory of this task, word sense descriptions contain some example sentences. We also used content words in these example sentences as features of decision lists.
7. keywords: decision list, example sentences in sense description
8. URL containing additional information (optional):
2. Your contact details
name: Yutaka Yagi
email: yutaka@cl.cs.titech.ac.jp
organisation: Tokyo Institute of Technology
3. Task/s: Japanese dictionary-based task (lexical sample)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: This system is almost the same as `Titech1'. The only difference is that morphological information in the evaluation data was corrected automatically. We learned the transformation rules like Brill's POS tagger to correct morphological information.
7. keywords: decision list, correcting morphological information, transformation rules
8. URL containing additional information (optional):
1. System name: JHU-Basque, JHU-Spanish and JHU-Swedish
2. Your contact details
name:David Yarowsky, Silviu Cucerzan, Radu Florian, Charles Schafer
and Richard Wicentowski
email {yarowsky,silviu,rflorian,cschafer,richardw}@cs.jhu.edu
organisation: Computer Science Department and Center for Language and Speech Processing, Johns Hopkins University
3. Task/s: Spanish lexical choice, Swedish lexical choice, Basque lexical choice
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: The JHU SENSEVAL-2 system for the lexical sample tasks consists of 6 diverse supervised learning subsystems integrated via classifier combination. The subsystems included decision lists (Yarowsky, 2000), transformation-based error-driven learning (Brill, 1995; Florian and Ngai, 2001), cosine-based vector models, decision stumps and two feature-enhanced naive Bayes systems (one trained on words, one trained on lemmas). For every subsystem the features included not only bag-of-words in a fixed context window and contextual n-grams, but also a rich variety of syntactic features including subjects, objects and objects of prepositions of verbs and several modification relationships for nouns and adjectives. For Spanish, Swedish and Basque these relationships were approximated using heuristic patterns. Additional features included parts-of-speech and lemmas in all syntactic positions, extracted using JHU-developed algorithms based on minimally supervised learning (including Yarowsky and Wicentowski, 2000; Cucerzan and Yarowsky, 2000). The output of each subsystem was merged by a classifier combination algorithm using weighted and thresholded voting and score combination.
7. keywords:classifier combination, word sense disambiguation, bayes similarity, cosine similarity, decision lists, transformation based learning
8. URL containing additional information (optional):
2. Your contact details
name: David Yarowsky, Silviu Cucerzan, Radu Florian, Charles Schafer
and Richard Wicentowski
email {yarowsky,silviu,rflorian,cschafer,richardw}@cs.jhu.edu
organisation: Computer Science Department and Center for Language and Speech Processing, Johns Hopkins University
3. Task/s: Czech all words
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: Because of the importance of morphological analysis in a highly inflected language such as Czech, a part-of-speech tagger and lemmatizer kindly provided by Jan Hajic of Charles University was first applied to the data. Consistent with the spirit of evaluating sense disambiguation rather than morphology, the JHU system focused on those words where more than one sense was possible for a root word (e.g. the -1 and -2 suffixes in the Czech inventory). In these cases, the fine-grained output of the Czech lemmatizer was ignored (in both training and test) and a generic lexical sample sense classifier was trained on the training data (see JHU_Swedish for further details of this). Whenever insufficient numbers of minority tagged examples were available for training a word-specific classifier, the majority sense for the POS-level lemma was returned. Likewise, if only one possible sense tag was observed for any POS-level lemma analysis, then this unambiguous sense tag was also returned.
7. keywords:word sense disambiguation, morphological analysis, part-of-speech tagging, highly inflected languages
8. URL containing additional information (optional):
2. Your contact details
name: David Yarowsky, Silviu Cucerzan, Radu Florian, Charles Schafer
and Richard Wicentowski
email {yarowsky,silviu,rflorian,cschafer,richardw}@cs.jhu.edu
organisation: Computer Science Department and Center for Language and Speech Processing, Johns Hopkins University
3. Task/s: English Lexical choice
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: The JHU SENSEVAL-2 system for the lexical sample tasks consists of 6 diverse supervised learning subsystems integrated via classifier combination. The subsystems included decision lists (Yarowsky, 2000), transformation-based error-driven learning (Brill, 1995; Florian and Ngai, 2001), cosine-based vector models, decision stumps and two feature-enhanced naive Bayes systems (one trained on words, one trained on lemmas). For every subsystem the features included not only bag-of-words in a fixed context window and contextual n-grams, but also a rich variety of syntactic features including subjects, objects and objects of prepositions of verbs and several modification relationships for nouns and adjectives. These relationships were approximated using heuristic patterns over base noun phrase bracketed sentences (Florian and Ngai, 2001). Additional features included parts-of-speech and lemmas in all syntactic positions, extracted using a Brill-style POS tagger and morphological analysis based on Yarowsky and Wicentowski, 2000). The output of each subsystem was merged by a classifier combination algorithm using weighted and thresholded voting and score combination.
The JHU-English system differed slightly from our other lexical sample systems in its treatment of phrasal senses. Because phrasal compounds such as verb-particle pairs were explicitly marked, both in training and test data, this additional provided information was used as follows: If a phrasal compound was marked in the data, then only compound senses (e.g. verb-particle) were considered. Likewise, if phrasal compounds were not marked in the data, the compound senses were excluded from consideration.
7. keywords: classifier combination, word sense disambiguation, bayes classifiers, cosine similarity, decision lists, transformation based learning
8. URL containing additional information (optional):
2. Your contact details
name:David Yarowsky, Silviu Cucerzan, Radu Florian, Charles Schafer
and Richard Wicentowski
email{yarowsky,silviu,rflorian,cschafer,richardw}@cs.jhu.edu
organisation: Computer Science Department and Center for Language and Speech Processing, Johns Hopkins University
3. Task/s: Estonian all words
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: Because of the importance of morphological analysis in a highly inflected language such as Estonian, a lemmatizer based on Yarowsky and Wicentowski (2000) was first applied to all words in the training data (and, at evaluation time, the test data). For each lemma, the P(sense|lemma) distribution was measured on the training data. For all lemmas exhibiting only one sense in the training data, this sense was returned. Likewise, if there was insufficient data for word-specific training (the sum of the minority sense examples for the word in training data was below a threshold) the majority sense in training was returned for all instances of that lemma. In the remaining cases where a lemma had more than one sense in training, with sufficient minority examples to adequately modeled, the generic JHU lexical sample sense classifier was trained and applied (see JHU_Swedish for further details).
7. keywords:word sense disambiguation, morphological analysis, classifier combination
8. URL containing additional information (optional):
2. Your contact details
name:David Yarowsky, Silviu Cucerzan, Radu Florian, Charles Schafer
and Richard Wicentowski
email{yarowsky,silviu,rflorian,cschafer,richardw}@cs.jhu.edu
organisation: Computer Science Department and Center for Language and Speech Processing, Johns Hopkins University
3. Task/s: Italian lexical choice
4. Did you use any training data provided in an automatic training procedure? N
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? N
6. Description: Because no training data was provided for Italian, the JHU_Italian system was unsupervised, exploiting hierarchical cluster models induced from the Italian WordNet. Because several relationship types (e.g. hypernymy) are represented in the Italian WordNet, each relationship type was arbitrarily assigned a generic weight indicating a rough semantic similarity implied by that relationship (e.g. synonymy received a 0.5, while meronomy received a 10). Then a weighted graph with all the relationships was created, and for each word w to be disambiguated, the graph was examined to determine the words v with a direct relationship to word w (one link away in the graph).
From an unannotated Italian web-mined corpus (approx 6M words), we selected all the sentences containing these words (the words v). We considered the contexts containing the word as being representative for the corresponding sense with the appropriate weight, and we used a Bayes similarity-based model to run an adaptive clustering model, in a k-means fashion. The classification assigned to each test sample after the convergence of the clustering algorithm was output as its classification.
Because no sense-tagged training data was used, it was not possible to know the lexical priors or even majority sense for each word. Thus baseline performance is significantly less than for other lexical-sample tasks.
7. keywords:unsupervised word sense disambiguation, bayes similarity, adaptive clustering, unsupervised learning
8. URL containing additional information (optional):
2. Your contact details
name: Kaoru Yamamoto
email: kaoru-ya@is.aist-nara.ac.jp
organisation: Nara Institute of Science and Technology
3. Task/s: Japanese lexical sample (Dictionary)
4. Did you use any training data provided in an automatic training procedure? Y
5. (if the answer to (4) is no) did you use any training data provided in any way (eg as a test set for debugging)? :
6. Description: We used SVM, PCA and ICA for learning. The features
used in the model are outputs of morphological and syntactic analysis. We also
use
Mainichi Simbun newspaper articles for 1994 as additional training data.
7. keywords: SVM, PCA, ICA
8. URL containing additional information (optional):
TASK DESCRIPTIONS (in alphabetical order of Tasks).
1. Task name: Basque Lexical Sample
2. Your contact details
name: Eneko Agirre
email: eneko@si.ehu.es
organisation: University of the Basque Country
3. Description:
The Basque task consists of lexical samples for 40 words. There will be approx. 75+15*senses+6*multiword_terms samples per word. The samples would comprise 5 sentences centered in the target word, taken from a newspaper corpus, and, if there is interest, the whole documents will be available via internet. The lexicon used is the reference dictionary Euskal Hiztegia, in TEI-sgml format. The senses are hierarchically organized, and the definition, synonyms and examples are provided for each sense, among other lexicographical data. Due to the complex structure of the dictionary, a flat list of word senses and multiword terms is also provided.
8. URL containing additional information (optional):
2. Your contact details
name: Antal van den Bosch
email:antalb@kub.nl
organisation: Tilburg University
3. Description:
The corpus is a concatenation of the texts of 102 illustrated children books for the age range of 4 to 12. Each word in these texts is manually annotated with its appropriate sense, by six persons who all processed a different part of the data. The sense inventory is roughly based on a Dutch children's dictionary.
Sense tags are non-hierarchical. Each tag is realised as a mnemonic description of the specific meaning the word has in the sentence. It is usually composed of the word's lemma and a sense circumscription of one or two words, often using a related term (drogen_nat, "dry_wet") or a reference of the grammatical category (fiets_N, fietsen_V). When a word is not ambiguous, its sense is tagged with "=".
The dataset also contains senses that span over multiple words. The meanings of these multi-word expressions cannot be broken down into the set of meanings of the individual words in the expressions. Multi-word expressions cover idiomatic expressions (in de steek laten, aan de hand zijn), sayings and proverbs (Boontje komt om zijn loontje) and strong collocations (derde wereld, klavertje vier).
Some basic statistics for the complete corpus:
| # tokens | 152.758 |
| # types | 10.263 |
| # sentences | 12.287 |
| # words per sentence | 12.4 |
| # unambiguous words | 9.095 |
| # words that occur once | 4.949 |
| # sense tags | 9319 |
| # word/sense combinations occuring once | 6.702 |
| % of ambiguous tokens in corpus | 54 |
For SENSEVAL-2, the dataset was divided in two parts. The training set consisted of 76 books (about 115.000 words). The test set consisted of the remaining 26 books (about 38.000 words).
8. URL containing additional information (optional):
1. Task name: English Lexical Sample
2. Your contact details
name: Adam Kilgharriff
email:adam@itri.bton.ac.uk
organisation:ITRI University of Brighton
3. Description:
Lexicon used was WordNet 1.7; instances were mostly from the British Natinal Corpus with some from the Wall Street Journal. Approach was very similar to the SENSEVAL-1 task (see Kilgarriff and Rosenzweig paper in Computers and the Humanities 34 (1--2), SENSEVAL Special Issue, 2000.) There were 29 nouns and 15 adjectives, with between 70 and 455 instances per word (total instances: 7567, divided 2:1 between training data and test data). Inter-tagger-agreement was 85.5%.
(verb data is not covered here as that was prepared by Martha Palmer and colleagues at UPenn)
8. URL containing additional information (optional):Click here
(or main SENSEVAL page for results)
1. Task name: Japanese Lexical Sample (translation task)
2. Your contact details
name: Sadao Kurohashi
email: kuro@kc.t.u-tokyo.ac.jp
organisation: University of Tokyo
3. Description: In this task, word sense is defined according to translation distinction (i.e. if the head word is translated differently in the given expressional context, then it is treated as constituting a different sense), and word sense disambiguation involves selecting the appropriate English word/phrase/sentence equivalent for a Japanese word.
All participants is supplied with a translation memory (TM). The TM contains, for each Japanese head word, a list of typical Japanese expressions (phrases/sentences) involving the head word and an English translation for each. Each pair is treated as a distinct sense and has a unique "sense ID".
In evaluation, 40 words (20 nouns and 20 verbs) are selected from the TM as target words, and 30 instances of each target word provided, making for a total of 1,200 instances. The test documents are annotated with morphological information (word segmentation, POS tag, reading and base form, all automatically annotated) for all words. For each target word, the ID of the sense in the TM best approximating that usage must be submitted. Submitted senses are judged to be correct if they are contained in the gold standard sense ID set. Alternatively, submissions can take the form of actual target word translations, or translations of phrases or sentences including each target word. In this case, translation experts are asked to judge whether the supplied translation is appropriate or not. Since participants can return translations of test sentences as answers, several existing MT systems entered the contest.
8. URL containing additional information (optional):
1. Task name: Japanese Lexical Sample (dictionary-based task)
2. Your contact details
name: Kiyoaki Shirai
email:kshirai@jaist.ac.jp
organisation: Japan Advanced Institute of Science and Technology
3. Description: Word senses are defined according to the Iwanami Kokugo Jiten, a Japanese dictionary published by Iwanami Shoten.
The Iwanami Kokugo Jiten is distributed to all participants. For each sense in the dictionary, a corresponding sense ID and morphological information (word segmentation, POS tag, base form and reading, all manually post-edited) will be supplied. A corpus annotated with sense IDs is also distributed as training data, made up of 3,000 newspaper articles extracted from the 1994 Mainiti Shimbun, consisting of 888,000 words. Only 149,556 words in this text is manually annotated for sense. The text is also annotated with morphological information (word segmentation, POS tag and base form and reading, all manually post-edited) for all words. Furthermore, each article is assigned a UDC (Universal Decimal Classification) code representing the text class.
For evaluation, we distribute test documents with marked target words. Participants is required to assign one or more sense IDs to each target word, optionally with associated probabilities. Test documents will take the form of newspaper articles annotated with UDC codes. The text will also be annotated with morphological information (word segmentation, POS tag, reading and base form, all automatically annotated) for all words. Notice that morphological information in the training data is post-edited, but not in the evaluation data, so participants may ignore morphological information in the evaluation data.
The number of target words is 100, 50 nouns and 50 verbs. 100 instances of each target word will be provided, making for a total of 10,000 instances for evaluation.
8. URL containing additional information (optional):
1. Task name: Spanish Lexical Sample
2. Your contact details
name: German Rigau Claramunt
email: g.rigau@lsi.upc.es
organisation: TALP Research Center (UPC), Barcelona.
3. Description: The task for Spanish is a 'lexical sample' for 40 words (18 nouns, 13 verbs and 9 adjectives). The items chosen can only belong to one of the syntactic categories and the sentences have been chosen to illustrated it. The length of corpus samples is the sentence and have been taken from two corpora: 'El Peri&ocute;dico' (a Spanish newspaper) and Lexesp-III (DGICYT APC 99-0105; a collection of texts from different thematic areas).
The lexicon provided has been created specifically for the task and it
consists of a definition for each sense linked to the Spanish version
of EuroWordNet and, thus, to the English WordNet 1.5, the syntactic
category and, sometimes, examples and synonyms are also
provided. Neither proper nouns nor multiwords have been considered. We
can also provide the complete mapping between WordNet 1.5 and 1.6
versions (see this site
).
7. keywords: semantic domains, domain driven disambiguation.
8. URL containing additional information (optional):click here
1. Task name: Swedish Lexical Sample
2. Your contact details
name: Dimitrios Kokkinakis
email: Dimitrios.Kokkinakis@svenska.gu.se
organisation: Språkdata, Göteborg University
3. Description: The lexical sample task in SENSEVAL-2 for Swedish consisted of 40 lemmas; (145 senses, 304 sub-senses)
The lexicon was generated the 2001-04-24 from GLDB/SDB (Click here)20 NOUNS
15 VERBS
5 ADJECTIVES
8. URL containing additional information (optional):Kokkinakis D., Jarborg J. and Cederholm Y. (May, 2001), Swedish
SENSEVAL; a Developers' Perspective. Proceedings of the
NODALIDA
(Nordiska Datalingvistikdagarna) Conference. Uppsala, Sweden;
Click here