Printer Friendly
Dictionary, Encyclopedia and Thesaurus - The Free Dictionary
1,771,255,734 visitors served.
forum mailing list For webmasters
?
New: Language forums
Dictionary/
thesaurus
Medical
dictionary
Legal
dictionary
Financial
dictionary
Acronyms
 
Idioms
Encyclopedia
Wikipedia
encyclopedia
?

lexical analysis
(redirected from Tokenize)

    Also found in: Dictionary/thesaurus, Legal, Idioms, Wikipedia 0.06 sec.
(programming)lexical analysis - (Or "linear analysis", "scanning") The first stage of processing a language. The stream of characters making up the source program or other input is read one at a time and grouped into lexemes (or "tokens") - word-like pieces such as keywords, identifiers, literals and punctutation. The lexemes are then passed to the parser.

["Compilers - Principles, Techniques and Tools", by Alfred V. Aho, Ravi Sethi and Jeffrey D. Ullman, pp. 4-5]


How to thank TFD for its existence? Tell a friend about us, add a link to this page, add the site to iGoogle, or visit webmaster's page for free fun content.
?Page tools
Printer friendly
Cite / link
 Email
Feedback
? Mentioned in ? References in periodicals archive
 
Experimental multiethnic texts such as Borderlands/La Frontera help advance the field of multiethnic literary studies by challenging categories in the larger field of literary studies that too often tokenize multiethnic literatures, co-opting them and making them the current "trend," or relegating them to the margins.
Unsettling calls for national unity: the pedagogy of experimental ... by Berila, Beth / MELUS
The anthologies, and the poems they anthologize, oppose mainstream anthologies that continue to marginalize and tokenize poets of color.
Theorizing difference in Asian American poetry anthologies by McCormick, Adrienne / MELUS
But Santos's stories are not so simple to tokenize.
Up from benevolent assimilation: at home with the Manongs of ... by Bascara, Victor / MELUS
More results
 
Encyclopedia browser? ? Full browser
 
 
Encyclopedia
?

Disclaimer | Privacy policy | Feedback | Copyright © 2009 Farlex, Inc.
All content on this website, including dictionary, thesaurus, literature, geography, and other reference data is for informational purposes only. This information should not be considered complete, up to date, and is not intended to be used in place of a visit, consultation, or advice of a legal, medical, or any other professional. Terms of Use.