ViewVC logotype

Contents of /code/trunk/HACKING

Parent Directory Parent Directory | Revision Log Revision Log

Revision 602 - (hide annotations) (download)
Wed May 25 08:29:03 2011 UTC (3 years, 11 months ago) by ph10
File size: 19106 byte(s)
Remove OP_OPT by handling /i and /m entirely at compile time. Fixes bug with 
patterns like /(?i:([^b]))(?1)/, where the /i option was mishandled.

1 nigel 41 Technical Notes about PCRE
2     --------------------------
4 nigel 91 These are very rough technical notes that record potentially useful information
5     about PCRE internals.
7 ph10 550
8 nigel 75 Historical note 1
9     -----------------
11 nigel 41 Many years ago I implemented some regular expression functions to an algorithm
12     suggested by Martin Richards. These were not Unix-like in form, and were quite
13     restricted in what they could do by comparison with Perl. The interesting part
14     about the algorithm was that the amount of space required to hold the compiled
15     form of an expression was known in advance. The code to apply an expression did
16 nigel 63 not operate by backtracking, as the original Henry Spencer code and current
17     Perl code does, but instead checked all possibilities simultaneously by keeping
18     a list of current states and checking all of them as it advanced through the
19 nigel 75 subject string. In the terminology of Jeffrey Friedl's book, it was a "DFA
20 nigel 93 algorithm", though it was not a traditional Finite State Machine (FSM). When
21     the pattern was all used up, all remaining states were possible matches, and
22     the one matching the longest subset of the subject string was chosen. This did
23     not necessarily maximize the individual wild portions of the pattern, as is
24     expected in Unix and Perl-style regular expressions.
25 nigel 41
26 ph10 550
27 nigel 75 Historical note 2
28     -----------------
30 nigel 91 By contrast, the code originally written by Henry Spencer (which was
31     subsequently heavily modified for Perl) compiles the expression twice: once in
32     a dummy mode in order to find out how much store will be needed, and then for
33     real. (The Perl version probably doesn't do this any more; I'm talking about
34     the original library.) The execution function operates by backtracking and
35     maximizing (or, optionally, minimizing in Perl) the amount of the subject that
36     matches individual wild portions of the pattern. This is an "NFA algorithm" in
37     Friedl's terminology.
38 nigel 41
39 ph10 550
40 nigel 75 OK, here's the real stuff
41     -------------------------
43 nigel 77 For the set of functions that form the "basic" PCRE library (which are
44     unrelated to those mentioned above), I tried at first to invent an algorithm
45     that used an amount of store bounded by a multiple of the number of characters
46     in the pattern, to save on compiling time. However, because of the greater
47     complexity in Perl regular expressions, I couldn't do this. In any case, a
48 nigel 93 first pass through the pattern is helpful for other reasons.
49 nigel 41
50 ph10 550
51 nigel 93 Computing the memory requirement: how it was
52     --------------------------------------------
54     Up to and including release 6.7, PCRE worked by running a very degenerate first
55     pass to calculate a maximum store size, and then a second pass to do the real
56     compile - which might use a bit less than the predicted amount of memory. The
57     idea was that this would turn out faster than the Henry Spencer code because
58     the first pass is degenerate and the second pass can just store stuff straight
59     into the vector, which it knows is big enough.
61 ph10 550
62 nigel 93 Computing the memory requirement: how it is
63     -------------------------------------------
65     By the time I was working on a potential 6.8 release, the degenerate first pass
66     had become very complicated and hard to maintain. Indeed one of the early
67     things I did for 6.8 was to fix Yet Another Bug in the memory computation. Then
68     I had a flash of inspiration as to how I could run the real compile function in
69     a "fake" mode that enables it to compute how much memory it would need, while
70     actually only ever using a few hundred bytes of working memory, and without too
71 ph10 602 many tests of the mode that might slow it down. So I refactored the compiling
72 nigel 93 functions to work this way. This got rid of about 600 lines of source. It
73     should make future maintenance and development easier. As this was such a major
74     change, I never released 6.8, instead upping the number to 7.0 (other quite
75 ph10 456 major changes were also present in the 7.0 release).
76 nigel 93
77 ph10 456 A side effect of this work was that the previous limit of 200 on the nesting
78 nigel 93 depth of parentheses was removed. However, there is a downside: pcre_compile()
79     runs more slowly than before (30% or more, depending on the pattern) because it
80 ph10 456 is doing a full analysis of the pattern. My hope was that this would not be a
81     big issue, and in the event, nobody has commented on it.
82 nigel 93
83 ph10 550
84 nigel 77 Traditional matching function
85     -----------------------------
87     The "traditional", and original, matching function is called pcre_exec(), and
88     it implements an NFA algorithm, similar to the original Henry Spencer algorithm
89 ph10 456 and the way that Perl works. This is not surprising, since it is intended to be
90     as compatible with Perl as possible. This is the function most users of PCRE
91     will use most of the time.
92 nigel 77
93 ph10 550
94 nigel 77 Supplementary matching function
95     -------------------------------
97     From PCRE 6.0, there is also a supplementary matching function called
98     pcre_dfa_exec(). This implements a DFA matching algorithm that searches
99     simultaneously for all possible matches that start at one point in the subject
100     string. (Going back to my roots: see Historical Note 1 above.) This function
101     intreprets the same compiled pattern data as pcre_exec(); however, not all the
102 nigel 91 facilities are available, and those that are do not always work in quite the
103 nigel 77 same way. See the user documentation for details.
105 nigel 93 The algorithm that is used for pcre_dfa_exec() is not a traditional FSM,
106     because it may have a number of states active at one time. More work would be
107     needed at compile time to produce a traditional FSM where only one state is
108     ever active at once. I believe some other regex matchers work this way.
111 ph10 602 Changeable options
112     ------------------
114     The /i, /m, or /s options (PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL) may
115     change in the middle of patterns. From PCRE 8.13, their processing is handled
116     entirely at compile time by generating different opcodes for the different
117     settings. The runtime functions do not need to keep track of an options state
118     any more.
121 nigel 77 Format of compiled patterns
122     ---------------------------
124 nigel 41 The compiled form of a pattern is a vector of bytes, containing items of
125     variable length. The first byte in an item is an opcode, and the length of the
126 nigel 75 item is either implicit in the opcode or contained in the data bytes that
127     follow it.
128 nigel 41
129 ph10 212 In many cases below LINK_SIZE data values are specified for offsets within the
130     compiled pattern. The default value for LINK_SIZE is 2, but PCRE can be
131 nigel 93 compiled to use 3-byte or 4-byte values for these offsets (impairing the
132 nigel 75 performance). This is necessary only when patterns whose compiled length is
133 nigel 93 greater than 64K are going to be processed. In this description, we assume the
134 ph10 212 "normal" compilation options. Data values that are counts (e.g. for
135     quantifiers) are always just two bytes long.
136 nigel 75
137 nigel 41 Opcodes with no following data
138     ------------------------------
140     These items are all just one byte long
142     OP_END end of pattern
143 ph10 342 OP_ANY match any one character other than newline
144     OP_ALLANY match any one character, including newline
145 nigel 75 OP_ANYBYTE match any single byte, even in UTF-8 mode
146 nigel 41 OP_SOD match start of data: \A
147 nigel 71 OP_SOM, start of match (subject + offset): \G
148 ph10 181 OP_SET_SOM, set start of match (\K)
149 ph10 602 OP_CIRC ^ (start of data)
150     OP_CIRCM ^ multiline mode (start of data or after newline)
151 nigel 41 OP_NOT_WORD_BOUNDARY \W
153     OP_NOT_DIGIT \D
154     OP_DIGIT \d
155 ph10 181 OP_NOT_HSPACE \H
156     OP_HSPACE \h
157 nigel 41 OP_NOT_WHITESPACE \S
158     OP_WHITESPACE \s
159 ph10 181 OP_NOT_VSPACE \V
160     OP_VSPACE \v
161 nigel 41 OP_NOT_WORDCHAR \W
162     OP_WORDCHAR \w
163     OP_EODN match end of data or \n at end: \Z
164     OP_EOD match end of data: \z
165 ph10 602 OP_DOLL $ (end of data, or before final newline)
166     OP_DOLLM $ multiline mode (end of data or before newline)
167 nigel 75 OP_EXTUNI match an extended Unicode character
168 nigel 93 OP_ANYNL match any Unicode newline sequence
169 nigel 75
170 ph10 550 OP_ACCEPT ) These are Perl 5.10's "backtracking control
171     OP_COMMIT ) verbs". If OP_ACCEPT is inside capturing
172     OP_FAIL ) parentheses, it may be preceded by one or more
173     OP_PRUNE ) OP_CLOSE, followed by a 2-byte number,
174     OP_SKIP ) indicating which parentheses must be closed.
175 ph10 212
176 nigel 41
177 ph10 550 Backtracking control verbs with data
178     ------------------------------------
180     OP_THEN is followed by a LINK_SIZE offset, which is the distance back to the
181     start of the current branch.
183     OP_MARK is followed by the mark name, preceded by a one-byte length, and
184     followed by a binary zero. For (*PRUNE), (*SKIP), and (*THEN) with arguments,
185     the opcodes OP_PRUNE_ARG, OP_SKIP_ARG, and OP_THEN_ARG are used. For the first
186     two, the name follows immediately; for OP_THEN_ARG, it follows the LINK_SIZE
187     offset value.
190 ph10 602 Matching literal characters
191     ---------------------------
193     The OP_CHAR opcode is followed by a single character that is to be matched
194     casefully. For caseless matching, OP_CHARI is used. In UTF-8 mode, the
195     character may be more than one byte long. (Earlier versions of PCRE used
196     multi-character strings, but this was changed to allow some new features to be
197     added.)
200 nigel 41 Repeating single characters
201     ---------------------------
203 nigel 75 The common repeats (*, +, ?) when applied to a single character use the
204 ph10 602 following opcodes, which come in caseful and caseless versions:
205 nigel 41
206 ph10 602 Caseful Caseless
216 nigel 41
217 nigel 75 In ASCII mode, these are two-byte items; in UTF-8 mode, the length is variable.
218 nigel 93 Those with "MIN" in their name are the minimizing versions. Those with "POS" in
219     their names are possessive versions. Each is followed by the character that is
220 ph10 602 to be repeated. Other repeats make use of these opcodes:
221 nigel 41
222 ph10 602 Caseful Caseless
227 nigel 41
228 ph10 602 Each of these is followed by a two-byte count (most significant first) and the
229 nigel 41 repeated character. OP_UPTO matches from 0 to the given number. A repeat with a
230     non-zero minimum and a fixed maximum is coded as an OP_EXACT followed by an
231 nigel 93 OP_UPTO (or OP_MINUPTO or OPT_POSUPTO).
232 nigel 41
234     Repeating character types
235     -------------------------
237     Repeats of things like \d are done exactly as for single characters, except
238     that instead of a character, the opcode for the type is stored in the data
239     byte. The opcodes are:
243 nigel 93 OP_TYPEPOSSTAR
244 nigel 41 OP_TYPEPLUS
246 nigel 93 OP_TYPEPOSPLUS
247 nigel 41 OP_TYPEQUERY
249 nigel 93 OP_TYPEPOSQUERY
250 nigel 41 OP_TYPEUPTO
252 nigel 93 OP_TYPEPOSUPTO
253 nigel 41 OP_TYPEEXACT
256 nigel 75 Match by Unicode property
257     -------------------------
259     OP_PROP and OP_NOTPROP are used for positive and negative matches of a
260     character by testing its Unicode property (the \p and \P escape sequences).
261 nigel 91 Each is followed by two bytes that encode the desired property as a type and a
262     value.
263 nigel 75
264 nigel 91 Repeats of these items use the OP_TYPESTAR etc. set of opcodes, followed by
265     three bytes: OP_PROP or OP_NOTPROP and then the desired property type and
266     value.
267 nigel 75
269 nigel 41 Character classes
270     -----------------
272 ph10 602 If there is only one character, OP_CHAR or OP_CHARI is used for a positive
273     class, and OP_NOT or OP_NOTI for a negative one (that is, for something like
274     [^a]). However, in UTF-8 mode, the use of OP_NOT[I] applies only to characters
275     with values < 128, because OP_NOT[I] is confined to single bytes.
276 nigel 41
277 ph10 602 Another set of 13 repeating opcodes (called OP_NOTSTAR etc.) are used for a
278     repeated, negated, single-character class. The normal single-character opcodes
279     (OP_STAR, etc.) are used for a repeated positive single-character class.
280 nigel 63
281 ph10 602 When there is more than one character in a class and all the characters are
282     less than 256, OP_CLASS is used for a positive class, and OP_NCLASS for a
283     negative one. In either case, the opcode is followed by a 32-byte bit map
284     containing a 1 bit for every character that is acceptable. The bits are counted
285     from the least significant end of each byte. In caseless mode, bits for both
286     cases are set.
287 nigel 41
288 nigel 75 The reason for having both OP_CLASS and OP_NCLASS is so that, in UTF-8 mode,
289     subject characters with values greater than 256 can be handled correctly. For
290 ph10 602 OP_CLASS they do not match, whereas for OP_NCLASS they do.
291 nigel 71
292 nigel 63 For classes containing characters with values > 255, OP_XCLASS is used. It
293     optionally uses a bit map (if any characters lie within it), followed by a list
294 ph10 602 of pairs (for a range) and single characters. In caseless mode, both cases are
295     explicitly listed. There is a flag character than indicates whether it is a
296     positive or a negative class.
297 nigel 41
298 nigel 63
299 nigel 41 Back references
300     ---------------
302 ph10 602 OP_REF (caseful) or OP_REFI (caseless) is followed by two bytes containing the
303     reference number.
304 nigel 41
306     Repeating character classes and back references
307     -----------------------------------------------
309 nigel 93 Single-character classes are handled specially (see above). This section
310 ph10 602 applies to OP_CLASS and OP_REF[I]. In both cases, the repeat information
311     follows the base item. The matching code looks at the following opcode to see
312     if it is one of
313 nigel 41
314     OP_CRSTAR
316     OP_CRPLUS
318     OP_CRQUERY
320     OP_CRRANGE
323     All but the last two are just single-byte items. The others are followed by
324 nigel 93 four bytes of data, comprising the minimum and maximum repeat counts. There are
325     no special possessive opcodes for these repeats; a possessive repeat is
326     compiled into an atomic group.
327 nigel 41
329     Brackets and alternation
330     ------------------------
332 nigel 43 A pair of non-capturing (round) brackets is wrapped round each expression at
333 nigel 41 compile time, so alternation always happens in the context of brackets.
334 nigel 53
335 nigel 93 [Note for North Americans: "bracket" to some English speakers, including
336     myself, can be round, square, curly, or pointy. Hence this usage.]
337 nigel 41
338 nigel 93 Non-capturing brackets use the opcode OP_BRA. Originally PCRE was limited to 99
339     capturing brackets and it used a different opcode for each one. From release
340     3.5, the limit was removed by putting the bracket number into the data for
341     higher-numbered brackets. From release 7.0 all capturing brackets are handled
342     this way, using the single opcode OP_CBRA.
343 nigel 53
344 nigel 77 A bracket opcode is followed by LINK_SIZE bytes which give the offset to the
345     next alternative OP_ALT or, if there aren't any branches, to the matching
346     OP_KET opcode. Each OP_ALT is followed by LINK_SIZE bytes giving the offset to
347 nigel 93 the next one, or to the OP_KET opcode. For capturing brackets, the bracket
348     number immediately follows the offset, always as a 2-byte item.
349 nigel 41
350     OP_KET is used for subpatterns that do not repeat indefinitely, while
351     OP_KETRMIN and OP_KETRMAX are used for indefinite repetitions, minimally or
352 nigel 77 maximally respectively. All three are followed by LINK_SIZE bytes giving (as a
353 nigel 93 positive number) the offset back to the matching bracket opcode.
354 nigel 41
355     If a subpattern is quantified such that it is permitted to match zero times, it
356 ph10 335 is preceded by one of OP_BRAZERO, OP_BRAMINZERO, or OP_SKIPZERO. These are
357     single-byte opcodes that tell the matcher that skipping the following
358     subpattern entirely is a valid branch. In the case of the first two, not
359     skipping the pattern is also valid (greedy and non-greedy). The third is used
360     when a pattern has the quantifier {0,0}. It cannot be entirely discarded,
361     because it may be called as a subroutine from elsewhere in the regex.
362 nigel 41
363     A subpattern with an indefinite maximum repetition is replicated in the
364 nigel 75 compiled data its minimum number of times (or once with OP_BRAZERO if the
365     minimum is zero), with the final copy terminating with OP_KETRMIN or OP_KETRMAX
366     as appropriate.
367 nigel 41
368     A subpattern with a bounded maximum repetition is replicated in a nested
369 nigel 75 fashion up to the maximum number of times, with OP_BRAZERO or OP_BRAMINZERO
370     before each replication after the minimum, so that, for example, (abc){2,5} is
371 nigel 93 compiled as (abc)(abc)((abc)((abc)(abc)?)?)?, except that each bracketed group
372     has the same number.
373 nigel 41
374 nigel 93 When a repeated subpattern has an unbounded upper limit, it is checked to see
375     whether it could match an empty string. If this is the case, the opcode in the
376     final replication is changed to OP_SBRA or OP_SCBRA. This tells the matcher
377     that it needs to check for matching an empty string when it hits OP_KETRMIN or
378     OP_KETRMAX, and if so, to break the loop.
379 nigel 41
380 nigel 93
381 nigel 41 Assertions
382     ----------
384     Forward assertions are just like other subpatterns, but starting with one of
385     the opcodes OP_ASSERT or OP_ASSERT_NOT. Backward assertions use the opcodes
386     OP_ASSERTBACK and OP_ASSERTBACK_NOT, and the first opcode inside the assertion
387     is OP_REVERSE, followed by a two byte count of the number of characters to move
388 nigel 49 back the pointer in the subject string. When operating in UTF-8 mode, the count
389     is a character count rather than a byte count. A separate count is present in
390     each alternative of a lookbehind assertion, allowing them to have different
391     fixed lengths.
392 nigel 41
394 nigel 93 Once-only (atomic) subpatterns
395     ------------------------------
396 nigel 41
397     These are also just like other subpatterns, but they start with the opcode
398 nigel 93 OP_ONCE. The check for matching an empty string in an unbounded repeat is
399     handled entirely at runtime, so there is just this one opcode.
400 nigel 41
402     Conditional subpatterns
403     -----------------------
405 nigel 93 These are like other subpatterns, but they start with the opcode OP_COND, or
406     OP_SCOND for one that might match an empty string in an unbounded repeat. If
407 nigel 41 the condition is a back reference, this is stored at the start of the
408 nigel 53 subpattern using the opcode OP_CREF followed by two bytes containing the
409 ph10 460 reference number. OP_NCREF is used instead if the reference was generated by
410     name (so that the runtime code knows to check for duplicate names).
411 nigel 41
412 ph10 460 If the condition is "in recursion" (coded as "(?(R)"), or "in recursion of
413     group x" (coded as "(?(Rx)"), the group number is stored at the start of the
414     subpattern using the opcode OP_RREF or OP_NRREF (cf OP_NCREF), and a value of
415     zero for "the whole pattern". For a DEFINE condition, just the single byte
416     OP_DEF is used (it has no associated data). Otherwise, a conditional subpattern
417     always starts with one of the assertions.
418 nigel 41
419 ph10 460
420 nigel 71 Recursion
421     ---------
423     Recursion either matches the current regex, or some subexpression. The opcode
424     OP_RECURSE is followed by an value which is the offset to the starting bracket
425 nigel 87 from the start of the whole pattern. From release 6.5, OP_RECURSE is
426     automatically wrapped inside OP_ONCE brackets (because otherwise some patterns
427     broke it). OP_RECURSE is also used for "subroutine" calls, even though they
428     are not strictly a recursion.
429 nigel 71
431     Callout
432     -------
434 nigel 75 OP_CALLOUT is followed by one byte of data that holds a callout number in the
435     range 0 to 254 for manual callouts, or 255 for an automatic callout. In both
436     cases there follows a two-byte value giving the offset in the pattern to the
437     start of the following item, and another two-byte item giving the length of the
438     next item.
439 nigel 71
441 nigel 41 Philip Hazel
442 ph10 602 May 2011

ViewVC Help
Powered by ViewVC 1.1.12