/[pcre]/code/trunk/HACKING
ViewVC logotype

Contents of /code/trunk/HACKING

Parent Directory Parent Directory | Revision Log Revision Log


Revision 604 - (hide annotations) (download)
Thu Jun 2 19:04:54 2011 UTC (3 years, 3 months ago) by ph10
File size: 19558 byte(s)
Refactoring to reduce stack usage for possessively quantified subpatterns. Also 
fixed a number of bugs related to repeated subpatterns. Some further tidies 
consequent on the removal of OP_OPT are also in this patch.

1 nigel 41 Technical Notes about PCRE
2     --------------------------
3    
4 nigel 91 These are very rough technical notes that record potentially useful information
5     about PCRE internals.
6    
7 ph10 550
8 nigel 75 Historical note 1
9     -----------------
10    
11 nigel 41 Many years ago I implemented some regular expression functions to an algorithm
12     suggested by Martin Richards. These were not Unix-like in form, and were quite
13     restricted in what they could do by comparison with Perl. The interesting part
14     about the algorithm was that the amount of space required to hold the compiled
15     form of an expression was known in advance. The code to apply an expression did
16 nigel 63 not operate by backtracking, as the original Henry Spencer code and current
17     Perl code does, but instead checked all possibilities simultaneously by keeping
18     a list of current states and checking all of them as it advanced through the
19 nigel 75 subject string. In the terminology of Jeffrey Friedl's book, it was a "DFA
20 nigel 93 algorithm", though it was not a traditional Finite State Machine (FSM). When
21     the pattern was all used up, all remaining states were possible matches, and
22     the one matching the longest subset of the subject string was chosen. This did
23     not necessarily maximize the individual wild portions of the pattern, as is
24     expected in Unix and Perl-style regular expressions.
25 nigel 41
26 ph10 550
27 nigel 75 Historical note 2
28     -----------------
29    
30 nigel 91 By contrast, the code originally written by Henry Spencer (which was
31     subsequently heavily modified for Perl) compiles the expression twice: once in
32     a dummy mode in order to find out how much store will be needed, and then for
33     real. (The Perl version probably doesn't do this any more; I'm talking about
34     the original library.) The execution function operates by backtracking and
35     maximizing (or, optionally, minimizing in Perl) the amount of the subject that
36     matches individual wild portions of the pattern. This is an "NFA algorithm" in
37     Friedl's terminology.
38 nigel 41
39 ph10 550
40 nigel 75 OK, here's the real stuff
41     -------------------------
42    
43 nigel 77 For the set of functions that form the "basic" PCRE library (which are
44     unrelated to those mentioned above), I tried at first to invent an algorithm
45     that used an amount of store bounded by a multiple of the number of characters
46     in the pattern, to save on compiling time. However, because of the greater
47     complexity in Perl regular expressions, I couldn't do this. In any case, a
48 nigel 93 first pass through the pattern is helpful for other reasons.
49 nigel 41
50 ph10 550
51 nigel 93 Computing the memory requirement: how it was
52     --------------------------------------------
53    
54     Up to and including release 6.7, PCRE worked by running a very degenerate first
55     pass to calculate a maximum store size, and then a second pass to do the real
56     compile - which might use a bit less than the predicted amount of memory. The
57     idea was that this would turn out faster than the Henry Spencer code because
58     the first pass is degenerate and the second pass can just store stuff straight
59     into the vector, which it knows is big enough.
60    
61 ph10 550
62 nigel 93 Computing the memory requirement: how it is
63     -------------------------------------------
64    
65     By the time I was working on a potential 6.8 release, the degenerate first pass
66     had become very complicated and hard to maintain. Indeed one of the early
67     things I did for 6.8 was to fix Yet Another Bug in the memory computation. Then
68     I had a flash of inspiration as to how I could run the real compile function in
69     a "fake" mode that enables it to compute how much memory it would need, while
70     actually only ever using a few hundred bytes of working memory, and without too
71 ph10 602 many tests of the mode that might slow it down. So I refactored the compiling
72 nigel 93 functions to work this way. This got rid of about 600 lines of source. It
73     should make future maintenance and development easier. As this was such a major
74     change, I never released 6.8, instead upping the number to 7.0 (other quite
75 ph10 456 major changes were also present in the 7.0 release).
76 nigel 93
77 ph10 456 A side effect of this work was that the previous limit of 200 on the nesting
78 nigel 93 depth of parentheses was removed. However, there is a downside: pcre_compile()
79     runs more slowly than before (30% or more, depending on the pattern) because it
80 ph10 456 is doing a full analysis of the pattern. My hope was that this would not be a
81     big issue, and in the event, nobody has commented on it.
82 nigel 93
83 ph10 550
84 nigel 77 Traditional matching function
85     -----------------------------
86    
87     The "traditional", and original, matching function is called pcre_exec(), and
88     it implements an NFA algorithm, similar to the original Henry Spencer algorithm
89 ph10 456 and the way that Perl works. This is not surprising, since it is intended to be
90     as compatible with Perl as possible. This is the function most users of PCRE
91     will use most of the time.
92 nigel 77
93 ph10 550
94 nigel 77 Supplementary matching function
95     -------------------------------
96    
97     From PCRE 6.0, there is also a supplementary matching function called
98     pcre_dfa_exec(). This implements a DFA matching algorithm that searches
99     simultaneously for all possible matches that start at one point in the subject
100     string. (Going back to my roots: see Historical Note 1 above.) This function
101     intreprets the same compiled pattern data as pcre_exec(); however, not all the
102 nigel 91 facilities are available, and those that are do not always work in quite the
103 nigel 77 same way. See the user documentation for details.
104    
105 nigel 93 The algorithm that is used for pcre_dfa_exec() is not a traditional FSM,
106     because it may have a number of states active at one time. More work would be
107     needed at compile time to produce a traditional FSM where only one state is
108     ever active at once. I believe some other regex matchers work this way.
109    
110    
111 ph10 602 Changeable options
112     ------------------
113    
114     The /i, /m, or /s options (PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL) may
115     change in the middle of patterns. From PCRE 8.13, their processing is handled
116     entirely at compile time by generating different opcodes for the different
117     settings. The runtime functions do not need to keep track of an options state
118     any more.
119    
120    
121 nigel 77 Format of compiled patterns
122     ---------------------------
123    
124 nigel 41 The compiled form of a pattern is a vector of bytes, containing items of
125     variable length. The first byte in an item is an opcode, and the length of the
126 nigel 75 item is either implicit in the opcode or contained in the data bytes that
127     follow it.
128 nigel 41
129 ph10 212 In many cases below LINK_SIZE data values are specified for offsets within the
130     compiled pattern. The default value for LINK_SIZE is 2, but PCRE can be
131 nigel 93 compiled to use 3-byte or 4-byte values for these offsets (impairing the
132 nigel 75 performance). This is necessary only when patterns whose compiled length is
133 nigel 93 greater than 64K are going to be processed. In this description, we assume the
134 ph10 212 "normal" compilation options. Data values that are counts (e.g. for
135     quantifiers) are always just two bytes long.
136 nigel 75
137 nigel 41 Opcodes with no following data
138     ------------------------------
139    
140     These items are all just one byte long
141    
142     OP_END end of pattern
143 ph10 342 OP_ANY match any one character other than newline
144     OP_ALLANY match any one character, including newline
145 nigel 75 OP_ANYBYTE match any single byte, even in UTF-8 mode
146 nigel 41 OP_SOD match start of data: \A
147 nigel 71 OP_SOM, start of match (subject + offset): \G
148 ph10 181 OP_SET_SOM, set start of match (\K)
149 ph10 602 OP_CIRC ^ (start of data)
150     OP_CIRCM ^ multiline mode (start of data or after newline)
151 nigel 41 OP_NOT_WORD_BOUNDARY \W
152     OP_WORD_BOUNDARY \w
153     OP_NOT_DIGIT \D
154     OP_DIGIT \d
155 ph10 181 OP_NOT_HSPACE \H
156     OP_HSPACE \h
157 nigel 41 OP_NOT_WHITESPACE \S
158     OP_WHITESPACE \s
159 ph10 181 OP_NOT_VSPACE \V
160     OP_VSPACE \v
161 nigel 41 OP_NOT_WORDCHAR \W
162     OP_WORDCHAR \w
163     OP_EODN match end of data or \n at end: \Z
164     OP_EOD match end of data: \z
165 ph10 602 OP_DOLL $ (end of data, or before final newline)
166     OP_DOLLM $ multiline mode (end of data or before newline)
167 nigel 75 OP_EXTUNI match an extended Unicode character
168 nigel 93 OP_ANYNL match any Unicode newline sequence
169 nigel 75
170 ph10 550 OP_ACCEPT ) These are Perl 5.10's "backtracking control
171     OP_COMMIT ) verbs". If OP_ACCEPT is inside capturing
172     OP_FAIL ) parentheses, it may be preceded by one or more
173     OP_PRUNE ) OP_CLOSE, followed by a 2-byte number,
174     OP_SKIP ) indicating which parentheses must be closed.
175 ph10 212
176 nigel 41
177 ph10 550 Backtracking control verbs with data
178     ------------------------------------
179    
180     OP_THEN is followed by a LINK_SIZE offset, which is the distance back to the
181     start of the current branch.
182    
183     OP_MARK is followed by the mark name, preceded by a one-byte length, and
184     followed by a binary zero. For (*PRUNE), (*SKIP), and (*THEN) with arguments,
185     the opcodes OP_PRUNE_ARG, OP_SKIP_ARG, and OP_THEN_ARG are used. For the first
186     two, the name follows immediately; for OP_THEN_ARG, it follows the LINK_SIZE
187     offset value.
188    
189    
190 ph10 602 Matching literal characters
191     ---------------------------
192    
193     The OP_CHAR opcode is followed by a single character that is to be matched
194     casefully. For caseless matching, OP_CHARI is used. In UTF-8 mode, the
195     character may be more than one byte long. (Earlier versions of PCRE used
196     multi-character strings, but this was changed to allow some new features to be
197     added.)
198    
199    
200 nigel 41 Repeating single characters
201     ---------------------------
202    
203 nigel 75 The common repeats (*, +, ?) when applied to a single character use the
204 ph10 602 following opcodes, which come in caseful and caseless versions:
205 nigel 41
206 ph10 602 Caseful Caseless
207     OP_STAR OP_STARI
208     OP_MINSTAR OP_MINSTARI
209     OP_POSSTAR OP_POSSTARI
210     OP_PLUS OP_PLUSI
211     OP_MINPLUS OP_MINPLUSI
212     OP_POSPLUS OP_POSPLUSI
213     OP_QUERY OP_QUERYI
214     OP_MINQUERY OP_MINQUERYI
215     OP_POSQUERY OP_POSQUERYI
216 nigel 41
217 nigel 75 In ASCII mode, these are two-byte items; in UTF-8 mode, the length is variable.
218 nigel 93 Those with "MIN" in their name are the minimizing versions. Those with "POS" in
219     their names are possessive versions. Each is followed by the character that is
220 ph10 602 to be repeated. Other repeats make use of these opcodes:
221 nigel 41
222 ph10 602 Caseful Caseless
223     OP_UPTO OP_UPTOI
224     OP_MINUPTO OP_MINUPTOI
225     OP_POSUPTO OP_POSUPTOI
226     OP_EXACT OP_EXACTI
227 nigel 41
228 ph10 602 Each of these is followed by a two-byte count (most significant first) and the
229 nigel 41 repeated character. OP_UPTO matches from 0 to the given number. A repeat with a
230     non-zero minimum and a fixed maximum is coded as an OP_EXACT followed by an
231 nigel 93 OP_UPTO (or OP_MINUPTO or OPT_POSUPTO).
232 nigel 41
233    
234     Repeating character types
235     -------------------------
236    
237     Repeats of things like \d are done exactly as for single characters, except
238     that instead of a character, the opcode for the type is stored in the data
239     byte. The opcodes are:
240    
241     OP_TYPESTAR
242     OP_TYPEMINSTAR
243 nigel 93 OP_TYPEPOSSTAR
244 nigel 41 OP_TYPEPLUS
245     OP_TYPEMINPLUS
246 nigel 93 OP_TYPEPOSPLUS
247 nigel 41 OP_TYPEQUERY
248     OP_TYPEMINQUERY
249 nigel 93 OP_TYPEPOSQUERY
250 nigel 41 OP_TYPEUPTO
251     OP_TYPEMINUPTO
252 nigel 93 OP_TYPEPOSUPTO
253 nigel 41 OP_TYPEEXACT
254    
255    
256 nigel 75 Match by Unicode property
257     -------------------------
258    
259     OP_PROP and OP_NOTPROP are used for positive and negative matches of a
260     character by testing its Unicode property (the \p and \P escape sequences).
261 nigel 91 Each is followed by two bytes that encode the desired property as a type and a
262     value.
263 nigel 75
264 nigel 91 Repeats of these items use the OP_TYPESTAR etc. set of opcodes, followed by
265     three bytes: OP_PROP or OP_NOTPROP and then the desired property type and
266     value.
267 nigel 75
268    
269 nigel 41 Character classes
270     -----------------
271    
272 ph10 602 If there is only one character, OP_CHAR or OP_CHARI is used for a positive
273     class, and OP_NOT or OP_NOTI for a negative one (that is, for something like
274     [^a]). However, in UTF-8 mode, the use of OP_NOT[I] applies only to characters
275     with values < 128, because OP_NOT[I] is confined to single bytes.
276 nigel 41
277 ph10 602 Another set of 13 repeating opcodes (called OP_NOTSTAR etc.) are used for a
278     repeated, negated, single-character class. The normal single-character opcodes
279     (OP_STAR, etc.) are used for a repeated positive single-character class.
280 nigel 63
281 ph10 602 When there is more than one character in a class and all the characters are
282     less than 256, OP_CLASS is used for a positive class, and OP_NCLASS for a
283     negative one. In either case, the opcode is followed by a 32-byte bit map
284     containing a 1 bit for every character that is acceptable. The bits are counted
285     from the least significant end of each byte. In caseless mode, bits for both
286     cases are set.
287 nigel 41
288 nigel 75 The reason for having both OP_CLASS and OP_NCLASS is so that, in UTF-8 mode,
289     subject characters with values greater than 256 can be handled correctly. For
290 ph10 602 OP_CLASS they do not match, whereas for OP_NCLASS they do.
291 nigel 71
292 nigel 63 For classes containing characters with values > 255, OP_XCLASS is used. It
293     optionally uses a bit map (if any characters lie within it), followed by a list
294 ph10 602 of pairs (for a range) and single characters. In caseless mode, both cases are
295     explicitly listed. There is a flag character than indicates whether it is a
296     positive or a negative class.
297 nigel 41
298 nigel 63
299 nigel 41 Back references
300     ---------------
301    
302 ph10 602 OP_REF (caseful) or OP_REFI (caseless) is followed by two bytes containing the
303     reference number.
304 nigel 41
305    
306     Repeating character classes and back references
307     -----------------------------------------------
308    
309 nigel 93 Single-character classes are handled specially (see above). This section
310 ph10 602 applies to OP_CLASS and OP_REF[I]. In both cases, the repeat information
311     follows the base item. The matching code looks at the following opcode to see
312     if it is one of
313 nigel 41
314     OP_CRSTAR
315     OP_CRMINSTAR
316     OP_CRPLUS
317     OP_CRMINPLUS
318     OP_CRQUERY
319     OP_CRMINQUERY
320     OP_CRRANGE
321     OP_CRMINRANGE
322    
323     All but the last two are just single-byte items. The others are followed by
324 nigel 93 four bytes of data, comprising the minimum and maximum repeat counts. There are
325     no special possessive opcodes for these repeats; a possessive repeat is
326     compiled into an atomic group.
327 nigel 41
328    
329     Brackets and alternation
330     ------------------------
331    
332 nigel 43 A pair of non-capturing (round) brackets is wrapped round each expression at
333 nigel 41 compile time, so alternation always happens in the context of brackets.
334 nigel 53
335 nigel 93 [Note for North Americans: "bracket" to some English speakers, including
336     myself, can be round, square, curly, or pointy. Hence this usage.]
337 nigel 41
338 nigel 93 Non-capturing brackets use the opcode OP_BRA. Originally PCRE was limited to 99
339     capturing brackets and it used a different opcode for each one. From release
340     3.5, the limit was removed by putting the bracket number into the data for
341     higher-numbered brackets. From release 7.0 all capturing brackets are handled
342     this way, using the single opcode OP_CBRA.
343 nigel 53
344 nigel 77 A bracket opcode is followed by LINK_SIZE bytes which give the offset to the
345     next alternative OP_ALT or, if there aren't any branches, to the matching
346     OP_KET opcode. Each OP_ALT is followed by LINK_SIZE bytes giving the offset to
347 nigel 93 the next one, or to the OP_KET opcode. For capturing brackets, the bracket
348     number immediately follows the offset, always as a 2-byte item.
349 nigel 41
350     OP_KET is used for subpatterns that do not repeat indefinitely, while
351     OP_KETRMIN and OP_KETRMAX are used for indefinite repetitions, minimally or
352 ph10 604 maximally respectively (see below for possessive repetitions). All three are
353     followed by LINK_SIZE bytes giving (as a positive number) the offset back to
354     the matching bracket opcode.
355 nigel 41
356     If a subpattern is quantified such that it is permitted to match zero times, it
357 ph10 335 is preceded by one of OP_BRAZERO, OP_BRAMINZERO, or OP_SKIPZERO. These are
358     single-byte opcodes that tell the matcher that skipping the following
359     subpattern entirely is a valid branch. In the case of the first two, not
360     skipping the pattern is also valid (greedy and non-greedy). The third is used
361     when a pattern has the quantifier {0,0}. It cannot be entirely discarded,
362     because it may be called as a subroutine from elsewhere in the regex.
363 nigel 41
364     A subpattern with an indefinite maximum repetition is replicated in the
365 nigel 75 compiled data its minimum number of times (or once with OP_BRAZERO if the
366     minimum is zero), with the final copy terminating with OP_KETRMIN or OP_KETRMAX
367     as appropriate.
368 nigel 41
369     A subpattern with a bounded maximum repetition is replicated in a nested
370 nigel 75 fashion up to the maximum number of times, with OP_BRAZERO or OP_BRAMINZERO
371     before each replication after the minimum, so that, for example, (abc){2,5} is
372 nigel 93 compiled as (abc)(abc)((abc)((abc)(abc)?)?)?, except that each bracketed group
373     has the same number.
374 nigel 41
375 nigel 93 When a repeated subpattern has an unbounded upper limit, it is checked to see
376     whether it could match an empty string. If this is the case, the opcode in the
377     final replication is changed to OP_SBRA or OP_SCBRA. This tells the matcher
378     that it needs to check for matching an empty string when it hits OP_KETRMIN or
379     OP_KETRMAX, and if so, to break the loop.
380 nigel 41
381 ph10 604 Possessive brackets
382     -------------------
383 nigel 93
384 ph10 604 When a repeated group (capturing or non-capturing) is marked as possessive by
385     the "+" notation, e.g. (abc)++, different opcodes are used. Their names all
386     have POS on the end, e.g. OP_BRAPOS instead of OP_BRA and OP_SCPBRPOS instead
387     of OP_SCBRA. The end of such a group is marked by OP_KETRPOS. If the minimum
388     repetition is zero, the group is preceded by OP_BRAPOSZERO.
389    
390    
391 nigel 41 Assertions
392     ----------
393    
394     Forward assertions are just like other subpatterns, but starting with one of
395     the opcodes OP_ASSERT or OP_ASSERT_NOT. Backward assertions use the opcodes
396     OP_ASSERTBACK and OP_ASSERTBACK_NOT, and the first opcode inside the assertion
397     is OP_REVERSE, followed by a two byte count of the number of characters to move
398 nigel 49 back the pointer in the subject string. When operating in UTF-8 mode, the count
399     is a character count rather than a byte count. A separate count is present in
400     each alternative of a lookbehind assertion, allowing them to have different
401     fixed lengths.
402 nigel 41
403    
404 nigel 93 Once-only (atomic) subpatterns
405     ------------------------------
406 nigel 41
407     These are also just like other subpatterns, but they start with the opcode
408 nigel 93 OP_ONCE. The check for matching an empty string in an unbounded repeat is
409     handled entirely at runtime, so there is just this one opcode.
410 nigel 41
411    
412     Conditional subpatterns
413     -----------------------
414    
415 nigel 93 These are like other subpatterns, but they start with the opcode OP_COND, or
416     OP_SCOND for one that might match an empty string in an unbounded repeat. If
417 nigel 41 the condition is a back reference, this is stored at the start of the
418 nigel 53 subpattern using the opcode OP_CREF followed by two bytes containing the
419 ph10 460 reference number. OP_NCREF is used instead if the reference was generated by
420     name (so that the runtime code knows to check for duplicate names).
421 nigel 41
422 ph10 460 If the condition is "in recursion" (coded as "(?(R)"), or "in recursion of
423     group x" (coded as "(?(Rx)"), the group number is stored at the start of the
424     subpattern using the opcode OP_RREF or OP_NRREF (cf OP_NCREF), and a value of
425     zero for "the whole pattern". For a DEFINE condition, just the single byte
426     OP_DEF is used (it has no associated data). Otherwise, a conditional subpattern
427     always starts with one of the assertions.
428 nigel 41
429 ph10 460
430 nigel 71 Recursion
431     ---------
432    
433     Recursion either matches the current regex, or some subexpression. The opcode
434     OP_RECURSE is followed by an value which is the offset to the starting bracket
435 nigel 87 from the start of the whole pattern. From release 6.5, OP_RECURSE is
436     automatically wrapped inside OP_ONCE brackets (because otherwise some patterns
437     broke it). OP_RECURSE is also used for "subroutine" calls, even though they
438     are not strictly a recursion.
439 nigel 71
440    
441     Callout
442     -------
443    
444 nigel 75 OP_CALLOUT is followed by one byte of data that holds a callout number in the
445     range 0 to 254 for manual callouts, or 255 for an automatic callout. In both
446     cases there follows a two-byte value giving the offset in the pattern to the
447     start of the following item, and another two-byte item giving the length of the
448     next item.
449 nigel 71
450    
451 nigel 41 Philip Hazel
452 ph10 602 May 2011

webmaster@exim.org
ViewVC Help
Powered by ViewVC 1.1.12