Puma Reference Manual Puma::Token Class Reference



Puma::Token Class Reference

Abstraction of a language token. More...

#include <Puma/Token.h>

Inheritance diagram for Puma::Token:
Inheritance graph

Public Types

enum  {
  ID_END_OF_FILE = -1,
  ID_UNKNOWN = -2,
  ID_ERROR = -3,
  ID_WARNING = -4
}
 Special token types. More...
 

Public Member Functions

 Token (int type=0, LanguageID lang=LanguageID(0), const char *text="")
 Constructor. More...
 
 Token (const Token &copy)
 Copy-constructor. More...
 
virtual ~Token ()
 Destructor. More...
 
void print (ostream &os) const
 Print the token text on the given stream. More...
 
virtual ListElementduplicate ()
 Duplicate this token. More...
 
Unitunit () const
 Get the unit this token belongs to. More...
 
void reset (int type=0, const char *text=0, LanguageID lang=LanguageID(0))
 Reset the token object. More...
 
void location (const Location &loc)
 Set the location of the token (usually the line and column in a file). More...
 
void cont_lines (Array< int > *cl)
 Set the continues lines appeared in this token. More...
 
int line_breaks () const
 Count the number of line breaks in the text of this token. More...
 
void macro_call (bool is_call=true)
 Set the token as being a macro call. More...
 
const char * text () const
 Get the token text. More...
 
DStringdtext () const
 Get the token text. More...
 
char * get_static_text () const
 Get the static token text. More...
 
int type () const
 Get the type of the token (see Puma::CTokens). More...
 
const Locationlocation () const
 Get the location of the token (usually the line and column in a file). More...
 
bool is_macro_generated () const
 Check if the token is macro generated. More...
 
bool is_macro_call () const
 Check if the token is a macro call. More...
 
bool is_core () const
 Check if this is a core language token. More...
 
bool is_identifier () const
 Check if the token is an identifier. More...
 
bool is_keyword () const
 Check if the token is a keyword. More...
 
bool is_wildcard () const
 Check if this is a wildcard token. More...
 
bool is_preprocessor () const
 Check if this is a preprocessor token. More...
 
bool is_directive () const
 Check if this is a compiler directive. More...
 
bool is_whitespace () const
 Check if this is a white-space token. More...
 
bool is_comment () const
 Check if this is a comment. More...
 
bool is_macro_op () const
 Check if this is function-like macro operand. More...
 
bool is_open () const
 Check if this is a left parenthesis. More...
 
bool is_comma () const
 Check if this is a comma. More...
 
bool is_close () const
 Check if this is a right parenthesis. More...
 
void * operator new (size_t)
 Own operator new reusing memory. More...
 
void operator delete (void *)
 Own delete operator. More...
 
- Public Member Functions inherited from Puma::ListElement
 ListElement ()
 Constructor. More...
 
virtual ~ListElement ()
 Destructor. More...
 
Listbelonging_to () const
 Get the list this element belongs to. More...
 

Static Public Member Functions

static char * get_static_text (int token_type)
 Get the static token text for the given token type. More...
 

Static Public Attributes

static LanguageID comment_id
 Comment tokens. More...
 
static LanguageID pre_id
 Preprocessor tokens. More...
 
static LanguageID dir_id
 Compiler directives. More...
 
static LanguageID cpp_id
 C/C++ core language tokens. More...
 
static LanguageID white_id
 White-spaces. More...
 
static LanguageID wildcard_id
 Wildcards. More...
 
static LanguageID keyword_id
 Keywords. More...
 
static LanguageID open_id
 Left parenthesis. More...
 
static LanguageID macro_op_id
 Function-like macro operands. More...
 
static LanguageID identifier_id
 Identifiers. More...
 
static LanguageID comma_id
 Comma. More...
 
static LanguageID close_id
 Right parenthesis. More...
 

Detailed Description

Abstraction of a language token.

A token is created by a scanner (see Puma::CScanner) as part a token chain (Puma::Unit). It encapsulates a lexical unit of the scanned text.

The information provided about a lexical unit is

  • the token type
  • the location in the scanned text
  • the portion of the scanned text represented by this token
  • the language of the token

Member Enumeration Documentation

anonymous enum

Special token types.

Enumerator
ID_END_OF_FILE 

End-of-file token.

ID_UNKNOWN 

Unknown token type.

ID_ERROR 

Error token type.

ID_WARNING 

Warning token type.

Constructor & Destructor Documentation

Puma::Token::Token ( int  type = 0,
LanguageID  lang = LanguageID(0),
const char *  text = "" 
)

Constructor.

Parameters
typeThe token type.
langThe language of the token.
textThe optional token text (defaults to the empty string).
Puma::Token::Token ( const Token copy)

Copy-constructor.

Parameters
copyThe token to copy.
virtual Puma::Token::~Token ( )
virtual

Destructor.

Destroys the token text.

Member Function Documentation

void Puma::Token::cont_lines ( Array< int > *  cl)
inline

Set the continues lines appeared in this token.

Parameters
clThe continues line array.
DString& Puma::Token::dtext ( ) const

Get the token text.

virtual ListElement* Puma::Token::duplicate ( )
inlinevirtual

Duplicate this token.

Returns
The copy of this token. Has to be destroyed by the caller.

Implements Puma::ListElement.

char* Puma::Token::get_static_text ( ) const

Get the static token text.

Only for keywords, operators, and so on with a constant token text.

Returns
The static text or NULL for tokens with dynamic text (like identifiers).
static char* Puma::Token::get_static_text ( int  token_type)
static

Get the static token text for the given token type.

Only for keywords, operators, and so on with a constant token text.

Returns
The static text or NULL for tokens with dynamic text (like identifiers).
bool Puma::Token::is_close ( ) const
inline

Check if this is a right parenthesis.

bool Puma::Token::is_comma ( ) const
inline

Check if this is a comma.

bool Puma::Token::is_comment ( ) const
inline

Check if this is a comment.

bool Puma::Token::is_core ( ) const
inline

Check if this is a core language token.

bool Puma::Token::is_directive ( ) const
inline

Check if this is a compiler directive.

bool Puma::Token::is_identifier ( ) const
inline

Check if the token is an identifier.

bool Puma::Token::is_keyword ( ) const
inline

Check if the token is a keyword.

bool Puma::Token::is_macro_call ( ) const
inline

Check if the token is a macro call.

bool Puma::Token::is_macro_generated ( ) const

Check if the token is macro generated.

bool Puma::Token::is_macro_op ( ) const
inline

Check if this is function-like macro operand.

bool Puma::Token::is_open ( ) const
inline

Check if this is a left parenthesis.

bool Puma::Token::is_preprocessor ( ) const
inline

Check if this is a preprocessor token.

bool Puma::Token::is_whitespace ( ) const
inline

Check if this is a white-space token.

bool Puma::Token::is_wildcard ( ) const
inline

Check if this is a wildcard token.

int Puma::Token::line_breaks ( ) const

Count the number of line breaks in the text of this token.

void Puma::Token::location ( const Location loc)
inline

Set the location of the token (usually the line and column in a file).

Parameters
locThe location.
const Location& Puma::Token::location ( ) const
inline

Get the location of the token (usually the line and column in a file).

void Puma::Token::macro_call ( bool  is_call = true)
inline

Set the token as being a macro call.

void Puma::Token::operator delete ( void *  )

Own delete operator.

void* Puma::Token::operator new ( size_t  )

Own operator new reusing memory.

void Puma::Token::print ( ostream &  os) const

Print the token text on the given stream.

Parameters
osThe output stream.
void Puma::Token::reset ( int  type = 0,
const char *  text = 0,
LanguageID  lang = LanguageID(0) 
)

Reset the token object.

Parameters
typeThe new token type.
textThe new token text.
langThe new token language.
const char* Puma::Token::text ( ) const

Get the token text.

int Puma::Token::type ( ) const
inline

Get the type of the token (see Puma::CTokens).

Unit* Puma::Token::unit ( ) const

Get the unit this token belongs to.

Member Data Documentation

LanguageID Puma::Token::close_id
static

Right parenthesis.

LanguageID Puma::Token::comma_id
static

Comma.

LanguageID Puma::Token::comment_id
static

Comment tokens.

LanguageID Puma::Token::cpp_id
static

C/C++ core language tokens.

LanguageID Puma::Token::dir_id
static

Compiler directives.

LanguageID Puma::Token::identifier_id
static

Identifiers.

LanguageID Puma::Token::keyword_id
static

Keywords.

LanguageID Puma::Token::macro_op_id
static

Function-like macro operands.

LanguageID Puma::Token::open_id
static

Left parenthesis.

LanguageID Puma::Token::pre_id
static

Preprocessor tokens.

LanguageID Puma::Token::white_id
static

White-spaces.

LanguageID Puma::Token::wildcard_id
static

Wildcards.




Puma Reference Manual. Created on Sat Dec 21 2019.