Printer Friendly
Dictionary, Encyclopedia and Thesaurus - The Free Dictionary
3,911,093,858 visitors served.
forum Join the Word of the Day Mailing List For webmasters
?
Dictionary/
thesaurus
Medical
dictionary
Legal
dictionary
Financial
dictionary
Acronyms
 
Idioms
Encyclopedia
Wikipedia
encyclopedia
?

lexical analysis
(redirected from Lexical token)

    Also found in: Wikipedia 0.01 sec.
(programming)lexical analysis - (Or "linear analysis", "scanning") The first stage of processing a language. The stream of characters making up the source program or other input is read one at a time and grouped into lexemes (or "tokens") - word-like pieces such as keywords, identifiers, literals and punctutation. The lexemes are then passed to the parser.

["Compilers - Principles, Techniques and Tools", by Alfred V. Aho, Ravi Sethi and Jeffrey D. Ullman, pp. 4-5]
Want to thank TFD for its existence? Tell a friend about us, add a link to this page, add the site to iGoogle, or visit the webmaster's page for free fun content.
?Page tools
Printer friendly
Cite / link 		Feedback
Mentioned in?  References in periodicals archive?   Encyclopedia browser?   Full browser?
No references found
 
Specific research questions, on the other hand, require detailed morphological analyses of lexical tokens in a corpus.
Efficient morphological parsing with a weighted finite state ... by Cavar, Damir; Jazbec, Ivo-Pavao; Runjaic, Sinisa / Informatica
Together the occurrences of these 13 shared lexical words constituted some 19% of the total lexical tokens in the three unrelated essays; by comparison, the suspect group sha red 74 lexical words whose occurrences accounted for almost half (49,3%) of all the lexical tokens.
Making texts speak: the work of the forensic linguist by Coulthard, Malcolm / Studia Anglica Posnaniensia: international review of English Studies
 
 
 
Encyclopedia
?

Terms of Use | Privacy policy | Feedback | Advertise with Us | Copyright © 2012 Farlex, Inc.
Disclaimer
All content on this website, including dictionary, thesaurus, literature, geography, and other reference data is for informational purposes only. This information should not be considered complete, up to date, and is not intended to be used in place of a visit, consultation, or advice of a legal, medical, or any other professional.