token (redirected from Tokenizing)

token

In programming, a string of characters. For example, in the C expression #define MAXAMOUNT 50000, MAXAMOUNT is the token. See also token passing and authentication token.

---

token

1. a gift voucher that can be used as payment for goods of a specified value

2. Philosophy an individual instance: if the same sentence has different truth-values on different occasions of utterance the truth-value may be said to attach to the sentence-token

---

token [′tō·kən]

(communications)

A unique grouping of bits that is transmitted as a unit in a communications network and used as a signal to notify stations in the network when they have control and are free to send information or take other specified actions.

(computer science)

A distinguishable unit in a sequence of characters.

A single byte that is used to represent a keyword in a programming language in order to conserve storage space.

A physical object, such as a badge or identity card, issued to authorized users of a computing system, building, or area.

---

1.	(grammar)	token - A basic, grammatically indivisible unit of a language such as a keyword, operator or identifier. Compare: lexeme.
2.	(convention)	token - (Or "pumpkin") An abstact concept passed between cooperating agents to ensure synchronised access to a shared resource. Such a token is never duplicated or destroyed (unless the resource is) and whoever has the token has exclusive access to the resource it controls. See for example token ring. If several programmers are working on a program, one programmer will "have the token" at any time, meaning that only he can change the program whereas others can only read it. If someone else wants to modify it he must first obtain the token.