CodeRay::Scanners::Scanner

Scanner

The base class for all Scanners.

It is a subclass of Ruby’s great StringScanner, which makes it easy to access the scanning methods inside.

It is also Enumerable, so you can use it like an Array of Tokens:

  require 'coderay'
  
  c_scanner = CodeRay::Scanners[:c].new "if (*p == '{') nest++;"
  
  for text, kind in c_scanner
    puts text if kind == :operator
  end
  
  # prints: (*==)++;

OK, this is a very simple example :) You can also use map, any?, find and even sort_by, if you want.

Constants

ScanError

Raised if a Scanner fails while scanning

DEFAULT_OPTIONS

The default options for all scanner classes.

Define @default_options for subclasses.

KINDS_NOT_LOC

Public Class Methods

file_extension(extension = nil) click to toggle source

    # File lib/coderay/scanner.rb, line 85
85:         def file_extension extension = nil
86:           if extension
87:             @file_extension = extension.to_s
88:           else
89:             @file_extension ||= plugin_id.to_s
90:           end
91:         end

new(code='', options = {}) click to toggle source

Create a new Scanner.

code is the input String and is handled by the superclass StringScanner.
options is a Hash with Symbols as keys. It is merged with the default options of the class (you can overwrite default options here.)
block is the callback for streamed highlighting.

If you set :stream to true in the options, the Scanner uses a TokenStream with the block as callback to handle the tokens.

Else, a Tokens object is used.

     # File lib/coderay/scanner.rb, line 119
119:       def initialize code='', options = {}, &block
120:         raise "I am only the basic Scanner class. I can't scan "           "anything. :( Use my subclasses." if self.class == Scanner
121:         
122:         @options = self.class::DEFAULT_OPTIONS.merge options
123: 
124:         super Scanner.normify(code)
125: 
126:         @tokens = options[:tokens]
127:         if @options[:stream]
128:           warn "warning in CodeRay::Scanner.new: :stream is set, "             "but no block was given" unless block_given?
129:           raise NotStreamableError, self unless kind_of? Streamable
130:           @tokens ||= TokenStream.new(&block)
131:         else
132:           warn "warning in CodeRay::Scanner.new: Block given, "             "but :stream is #{@options[:stream]}" if block_given?
133:           @tokens ||= Tokens.new
134:         end
135:         @tokens.scanner = self
136: 
137:         setup
138:       end

normify(code) click to toggle source

    # File lib/coderay/scanner.rb, line 69
69:         def normify code
70:           code = code.to_s
71:           if code.respond_to? :force_encoding
72:             debug, $DEBUG = $DEBUG, false
73:             begin
74:               code.force_encoding 'utf-8'
75:               code[/\z/]  # raises an ArgumentError when code contains a non-UTF-8 char
76:             rescue ArgumentError
77:               code.force_encoding 'binary'
78:             ensure
79:               $DEBUG = debug
80:             end
81:           end
82:           code.to_unix
83:         end

streamable?() click to toggle source

Returns if the Scanner can be used in streaming mode.

    # File lib/coderay/scanner.rb, line 65
65:         def streamable?
66:           is_a? Streamable
67:         end

Public Instance Methods

code=(code) click to toggle source

Alias for: string=

column(pos = self.pos) click to toggle source

     # File lib/coderay/scanner.rb, line 202
202:       def column pos = self.pos
203:         return 0 if pos <= 0
204:         string = string()
205:         if string.respond_to?(:bytesize) && (defined?(@bin_string) || string.bytesize != string.size)
206:           @bin_string ||= string.dup.force_encoding('binary')
207:           string = @bin_string
208:         end
209:         pos - (string.rindex(\n\, pos) || 0)
210:       end

each(&block) click to toggle source

Traverses the tokens.

     # File lib/coderay/scanner.rb, line 187
187:       def each &block
188:         raise ArgumentError,
189:           'Cannot traverse TokenStream.' if @options[:stream]
190:         tokens.each(&block)
191:       end

lang() click to toggle source

Returns the Plugin ID for this scanner.

     # File lib/coderay/scanner.rb, line 159
159:       def lang
160:         self.class.plugin_id
161:       end

line() click to toggle source

The current line position of the scanner.

Beware, this is implemented inefficiently. It should be used for debugging only.

     # File lib/coderay/scanner.rb, line 198
198:       def line
199:         string[0..pos].count("\n") + 1
200:       end

marshal_dump() click to toggle source

     # File lib/coderay/scanner.rb, line 212
212:       def marshal_dump
213:         @options
214:       end

marshal_load(options) click to toggle source

     # File lib/coderay/scanner.rb, line 216
216:       def marshal_load options
217:         @options = options
218:       end

reset() click to toggle source

     # File lib/coderay/scanner.rb, line 143
143:       def reset
144:         super
145:         reset_instance
146:       end

streaming?() click to toggle source

Whether the scanner is in streaming mode.

     # File lib/coderay/scanner.rb, line 182
182:       def streaming?
183:         !!@options[:stream]
184:       end

string=(code) click to toggle source

     # File lib/coderay/scanner.rb, line 148
148:       def string= code
149:         code = Scanner.normify(code)
150:         super code
151:         reset_instance
152:       end

Also aliased as: code=

tokenize(new_string=nil, options = {}) click to toggle source

Scans the code and returns all tokens in a Tokens object.

     # File lib/coderay/scanner.rb, line 164
164:       def tokenize new_string=nil, options = {}
165:         options = @options.merge(options)
166:         self.string = new_string if new_string
167:         @cached_tokens =
168:           if @options[:stream]  # :stream must have been set already
169:             reset unless new_string
170:             scan_tokens @tokens, options
171:             @tokens
172:           else
173:             scan_tokens @tokens, options
174:           end
175:       end

tokens() click to toggle source

     # File lib/coderay/scanner.rb, line 177
177:       def tokens
178:         @cached_tokens ||= tokenize
179:       end

Protected Instance Methods

raise_inspect(msg, tokens, state = 'No state given!', ambit = 30) click to toggle source

Scanner error with additional status information

     # File lib/coderay/scanner.rb, line 247
247:       def raise_inspect msg, tokens, state = 'No state given!', ambit = 30
248:         raise ScanError, ***ERROR in %s: %s (after %d tokens)tokens:%scurrent line: %d  column: %d  pos: %dmatched: %p  state: %pbol? = %p,  eos? = %psurrounding code:%p  ~~  %p***ERROR*** % [
249:           File.basename(caller[0]),
250:           msg,
251:           tokens.size,
252:           tokens.last(10).map { |t| t.inspect }.join("\n"),
253:           line, column, pos,
254:           matched, state, bol?, eos?,
255:           string[pos - ambit, ambit],
256:           string[pos, ambit],
257:         ]
258:       end

reset_instance() click to toggle source

     # File lib/coderay/scanner.rb, line 240
240:       def reset_instance
241:         @tokens.clear unless @options[:keep_tokens]
242:         @cached_tokens = nil
243:         @bin_string = nil if defined? @bin_string
244:       end

scan_tokens(tokens, options) click to toggle source

This is the central method, and commonly the only one a subclass implements.

Subclasses must implement this method; it must return tokens and must only use Tokens#<< for storing scanned tokens!

     # File lib/coderay/scanner.rb, line 235
235:       def scan_tokens tokens, options
236:         raise NotImplementedError,
237:           "#{self.class}#scan_tokens not implemented."
238:       end

setup() click to toggle source

Can be implemented by subclasses to do some initialization that has to be done once per instance.

Use reset for initialization that has to be done once per scan.

     # File lib/coderay/scanner.rb, line 227
227:       def setup
228:       end

Home Classes Methods

In Files

Parent

Methods

Included Modules