The CTokenType class holds a single custom token definition as loaded from a text file. These definitions will be used to create tokens by the tokenizer for use in the parser.
CTokenTypeEdit
This class is very simple and is mainly composed of three members:
class CTokenType {
dword m_ID; /* Unique ID for this token */
string m_Name; /* Token name */
dword m_Flags; /* Token flags */
};
The flags member is a bit field with the possible values:
-
- Keep: Store token value when tokenizing (used for strings, identifiers, etc...).
- Simple: Do not store the token value (most tokens).
- Ignore: Ignore token (not saved to the token stream, like comments).
- TrimFirst: Remove the first character in the token value (for quoted strings).
- TrimLast: Remove the last character in the token value (for quoted strings).
- EndToken: Identifies the token that ends a token sequence (stops parsing).
Data File FormatEdit
Custom tokens are defined from a text file with the following format:
Tokens
[TokenName] = [ID], [Option1], ... [OptionN]
...
End
The [TokenName] is a unique string identifying the token. [ID] is a unique integer value which can also be used to identify the token. Multiple options can be specified after the ID and can be any of:
-
- keep
- simple
- ignore
- trimlast
- trimfirst
- endtoken
Oblivion Token FileEdit
The following token definitions can be used for Oblivion scripts:
Tokens
Unknown = 0, keep
LBracket = 1, simple
RBracket = 2, simple
AddOp = 3, keep
MultOp = 4, keep
RelOp = 5, keep
BoolOp = 6, keep
Comma = 7, ignore
String = 16, keep, trimfirst, trimlast
Identifier = 17, keep
If = 18, simple
Elseif = 19, simple
Else = 20, simple
Endif = 21, simple
Set = 22, simple
Scriptname = 23, simple
Begin = 24, simple
End = 25, simple
Comment = 26, ignore
EndLine = 27, simple
Integer = 28, keep
Float = 29, keep
EndofProgram = 30, simple, endtoken
End