Syntax::Tokenizer

The base class of all tokenizers. It sets up the scanner and manages the looping until all tokens have been extracted. It also provides convenience methods to make sure adjacent tokens of identical groups are returned as a single token.

Constants

EOL

Attributes

group[R]

The current group being processed by the tokenizer

chunk[R]

The current chunk of text being accumulated

Private Class Methods

delegate( sym ) click to toggle source

A convenience for delegating method calls to the scanner.

     # File lib/syntax/common.rb, line 98
 98:       def self.delegate( sym )
 99:         define_method( sym ) { |*a| @text.__send__( sym, *a ) }
100:       end

Public Instance Methods

finish() click to toggle source

Finish tokenizing. This flushes the buffer, yielding any remaining text to the client.

    # File lib/syntax/common.rb, line 57
57:     def finish
58:       start_group nil
59:       teardown
60:     end

option(opt) click to toggle source

Get the value of the specified option.

    # File lib/syntax/common.rb, line 89
89:     def option(opt)
90:       @options ? @options[opt] : nil
91:     end

set( opts={} ) click to toggle source

Specify a set of tokenizer-specific options. Each tokenizer may (or may not) publish any options, but if a tokenizer does those options may be used to specify optional behavior.

    # File lib/syntax/common.rb, line 84
84:     def set( opts={} )
85:       ( @options ||= Hash.new ).update opts
86:     end

setup() click to toggle source

Subclasses may override this method to provide implementation-specific setup logic.

    # File lib/syntax/common.rb, line 52
52:     def setup
53:     end

start( text, &block ) click to toggle source

Start tokenizing. This sets up the state in preparation for tokenization, such as creating a new scanner for the text and saving the callback block. The block will be invoked for each token extracted.

    # File lib/syntax/common.rb, line 42
42:     def start( text, &block )
43:       @chunk = ""
44:       @group = :normal
45:       @callback = block
46:       @text = StringScanner.new( text )
47:       setup
48:     end

step() click to toggle source

Subclasses must implement this method, which is called for each iteration of the tokenization process. This method may extract multiple tokens.

    # File lib/syntax/common.rb, line 69
69:     def step
70:       raise NotImplementedError, "subclasses must implement #step"
71:     end

teardown() click to toggle source

Subclasses may override this method to provide implementation-specific teardown logic.

    # File lib/syntax/common.rb, line 64
64:     def teardown
65:     end

tokenize( text, &block ) click to toggle source

Begins tokenizing the given text, calling # until the text has been exhausted.

    # File lib/syntax/common.rb, line 75
75:     def tokenize( text, &block )
76:       start text, &block
77:       step until @text.eos?
78:       finish
79:     end

Private Instance Methods

append( data ) click to toggle source

Append the given data to the currently active chunk.

     # File lib/syntax/common.rb, line 120
120:       def append( data )
121:         @chunk << data
122:       end

end_region( gr, data=nil ) click to toggle source

     # File lib/syntax/common.rb, line 143
143:       def end_region( gr, data=nil )
144:         flush_chunk
145:         @group = gr
146:         @callback.call( Token.new( data||"", @group, :region_close ) )
147:       end

flush_chunk() click to toggle source

     # File lib/syntax/common.rb, line 149
149:       def flush_chunk
150:         @callback.call( Token.new( @chunk, @group ) ) unless @chunk.empty?
151:         @chunk = ""
152:       end

start_group( gr, data=nil ) click to toggle source

Request that a new group be started. If the current group is the same as the group being requested, a new group will not be created. If a new group is created and the current chunk is not empty, the chunk’s contents will be yielded to the client as a token, and then cleared.

After the new group is started, if data is non-nil it will be appended to the chunk.

     # File lib/syntax/common.rb, line 131
131:       def start_group( gr, data=nil )
132:         flush_chunk if gr != @group
133:         @group = gr
134:         @chunk << data if data
135:       end

start_region( gr, data=nil ) click to toggle source

     # File lib/syntax/common.rb, line 137
137:       def start_region( gr, data=nil )
138:         flush_chunk
139:         @group = gr
140:         @callback.call( Token.new( data||"", @group, :region_open ) )
141:       end

subgroup(n) click to toggle source

Access the n-th subgroup from the most recent match.

     # File lib/syntax/common.rb, line 115
115:       def subgroup(n)
116:         @text[n]
117:       end

subtokenize( syntax, text ) click to toggle source

     # File lib/syntax/common.rb, line 154
154:       def subtokenize( syntax, text )
155:         tokenizer = Syntax.load( syntax )
156:         tokenizer.set @options if @options
157:         flush_chunk
158:         tokenizer.tokenize( text, &@callback )
159:       end

Home Classes Methods

In Files

Parent