/[pcre]/code/trunk/HACKING
ViewVC logotype

Contents of /code/trunk/HACKING

Parent Directory Parent Directory | Revision Log Revision Log


Revision 612 - (hide annotations) (download)
Sat Jul 2 15:20:59 2011 UTC (2 years, 9 months ago) by ph10
File size: 19676 byte(s)
Fix two study bugs concerned with minimum subject lengths; add features to 
pcretest so that all tests can be run with or without study; adjust tests so 
that this happens.

1 nigel 41 Technical Notes about PCRE
2     --------------------------
3    
4 nigel 91 These are very rough technical notes that record potentially useful information
5 ph10 612 about PCRE internals. For information about testing PCRE, see the pcretest
6     documentation and the comment at the head of the RunTest file.
7 nigel 91
8 ph10 550
9 nigel 75 Historical note 1
10     -----------------
11    
12 nigel 41 Many years ago I implemented some regular expression functions to an algorithm
13     suggested by Martin Richards. These were not Unix-like in form, and were quite
14     restricted in what they could do by comparison with Perl. The interesting part
15     about the algorithm was that the amount of space required to hold the compiled
16     form of an expression was known in advance. The code to apply an expression did
17 nigel 63 not operate by backtracking, as the original Henry Spencer code and current
18     Perl code does, but instead checked all possibilities simultaneously by keeping
19     a list of current states and checking all of them as it advanced through the
20 nigel 75 subject string. In the terminology of Jeffrey Friedl's book, it was a "DFA
21 nigel 93 algorithm", though it was not a traditional Finite State Machine (FSM). When
22     the pattern was all used up, all remaining states were possible matches, and
23     the one matching the longest subset of the subject string was chosen. This did
24     not necessarily maximize the individual wild portions of the pattern, as is
25     expected in Unix and Perl-style regular expressions.
26 nigel 41
27 ph10 550
28 nigel 75 Historical note 2
29     -----------------
30    
31 nigel 91 By contrast, the code originally written by Henry Spencer (which was
32     subsequently heavily modified for Perl) compiles the expression twice: once in
33     a dummy mode in order to find out how much store will be needed, and then for
34     real. (The Perl version probably doesn't do this any more; I'm talking about
35     the original library.) The execution function operates by backtracking and
36     maximizing (or, optionally, minimizing in Perl) the amount of the subject that
37     matches individual wild portions of the pattern. This is an "NFA algorithm" in
38     Friedl's terminology.
39 nigel 41
40 ph10 550
41 nigel 75 OK, here's the real stuff
42     -------------------------
43    
44 nigel 77 For the set of functions that form the "basic" PCRE library (which are
45     unrelated to those mentioned above), I tried at first to invent an algorithm
46     that used an amount of store bounded by a multiple of the number of characters
47     in the pattern, to save on compiling time. However, because of the greater
48     complexity in Perl regular expressions, I couldn't do this. In any case, a
49 nigel 93 first pass through the pattern is helpful for other reasons.
50 nigel 41
51 ph10 550
52 nigel 93 Computing the memory requirement: how it was
53     --------------------------------------------
54    
55     Up to and including release 6.7, PCRE worked by running a very degenerate first
56     pass to calculate a maximum store size, and then a second pass to do the real
57     compile - which might use a bit less than the predicted amount of memory. The
58     idea was that this would turn out faster than the Henry Spencer code because
59     the first pass is degenerate and the second pass can just store stuff straight
60     into the vector, which it knows is big enough.
61    
62 ph10 550
63 nigel 93 Computing the memory requirement: how it is
64     -------------------------------------------
65    
66     By the time I was working on a potential 6.8 release, the degenerate first pass
67     had become very complicated and hard to maintain. Indeed one of the early
68     things I did for 6.8 was to fix Yet Another Bug in the memory computation. Then
69     I had a flash of inspiration as to how I could run the real compile function in
70     a "fake" mode that enables it to compute how much memory it would need, while
71     actually only ever using a few hundred bytes of working memory, and without too
72 ph10 602 many tests of the mode that might slow it down. So I refactored the compiling
73 nigel 93 functions to work this way. This got rid of about 600 lines of source. It
74     should make future maintenance and development easier. As this was such a major
75     change, I never released 6.8, instead upping the number to 7.0 (other quite
76 ph10 456 major changes were also present in the 7.0 release).
77 nigel 93
78 ph10 456 A side effect of this work was that the previous limit of 200 on the nesting
79 nigel 93 depth of parentheses was removed. However, there is a downside: pcre_compile()
80     runs more slowly than before (30% or more, depending on the pattern) because it
81 ph10 456 is doing a full analysis of the pattern. My hope was that this would not be a
82     big issue, and in the event, nobody has commented on it.
83 nigel 93
84 ph10 550
85 nigel 77 Traditional matching function
86     -----------------------------
87    
88     The "traditional", and original, matching function is called pcre_exec(), and
89     it implements an NFA algorithm, similar to the original Henry Spencer algorithm
90 ph10 456 and the way that Perl works. This is not surprising, since it is intended to be
91     as compatible with Perl as possible. This is the function most users of PCRE
92     will use most of the time.
93 nigel 77
94 ph10 550
95 nigel 77 Supplementary matching function
96     -------------------------------
97    
98     From PCRE 6.0, there is also a supplementary matching function called
99     pcre_dfa_exec(). This implements a DFA matching algorithm that searches
100     simultaneously for all possible matches that start at one point in the subject
101     string. (Going back to my roots: see Historical Note 1 above.) This function
102     intreprets the same compiled pattern data as pcre_exec(); however, not all the
103 nigel 91 facilities are available, and those that are do not always work in quite the
104 nigel 77 same way. See the user documentation for details.
105    
106 nigel 93 The algorithm that is used for pcre_dfa_exec() is not a traditional FSM,
107     because it may have a number of states active at one time. More work would be
108     needed at compile time to produce a traditional FSM where only one state is
109     ever active at once. I believe some other regex matchers work this way.
110    
111    
112 ph10 602 Changeable options
113     ------------------
114    
115     The /i, /m, or /s options (PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL) may
116     change in the middle of patterns. From PCRE 8.13, their processing is handled
117     entirely at compile time by generating different opcodes for the different
118     settings. The runtime functions do not need to keep track of an options state
119     any more.
120    
121    
122 nigel 77 Format of compiled patterns
123     ---------------------------
124    
125 nigel 41 The compiled form of a pattern is a vector of bytes, containing items of
126     variable length. The first byte in an item is an opcode, and the length of the
127 nigel 75 item is either implicit in the opcode or contained in the data bytes that
128     follow it.
129 nigel 41
130 ph10 212 In many cases below LINK_SIZE data values are specified for offsets within the
131     compiled pattern. The default value for LINK_SIZE is 2, but PCRE can be
132 nigel 93 compiled to use 3-byte or 4-byte values for these offsets (impairing the
133 nigel 75 performance). This is necessary only when patterns whose compiled length is
134 nigel 93 greater than 64K are going to be processed. In this description, we assume the
135 ph10 212 "normal" compilation options. Data values that are counts (e.g. for
136     quantifiers) are always just two bytes long.
137 nigel 75
138 nigel 41 Opcodes with no following data
139     ------------------------------
140    
141     These items are all just one byte long
142    
143     OP_END end of pattern
144 ph10 342 OP_ANY match any one character other than newline
145     OP_ALLANY match any one character, including newline
146 nigel 75 OP_ANYBYTE match any single byte, even in UTF-8 mode
147 nigel 41 OP_SOD match start of data: \A
148 nigel 71 OP_SOM, start of match (subject + offset): \G
149 ph10 181 OP_SET_SOM, set start of match (\K)
150 ph10 602 OP_CIRC ^ (start of data)
151     OP_CIRCM ^ multiline mode (start of data or after newline)
152 nigel 41 OP_NOT_WORD_BOUNDARY \W
153     OP_WORD_BOUNDARY \w
154     OP_NOT_DIGIT \D
155     OP_DIGIT \d
156 ph10 181 OP_NOT_HSPACE \H
157     OP_HSPACE \h
158 nigel 41 OP_NOT_WHITESPACE \S
159     OP_WHITESPACE \s
160 ph10 181 OP_NOT_VSPACE \V
161     OP_VSPACE \v
162 nigel 41 OP_NOT_WORDCHAR \W
163     OP_WORDCHAR \w
164     OP_EODN match end of data or \n at end: \Z
165     OP_EOD match end of data: \z
166 ph10 602 OP_DOLL $ (end of data, or before final newline)
167     OP_DOLLM $ multiline mode (end of data or before newline)
168 nigel 75 OP_EXTUNI match an extended Unicode character
169 nigel 93 OP_ANYNL match any Unicode newline sequence
170 nigel 75
171 ph10 550 OP_ACCEPT ) These are Perl 5.10's "backtracking control
172     OP_COMMIT ) verbs". If OP_ACCEPT is inside capturing
173     OP_FAIL ) parentheses, it may be preceded by one or more
174     OP_PRUNE ) OP_CLOSE, followed by a 2-byte number,
175     OP_SKIP ) indicating which parentheses must be closed.
176 ph10 212
177 nigel 41
178 ph10 550 Backtracking control verbs with data
179     ------------------------------------
180    
181     OP_THEN is followed by a LINK_SIZE offset, which is the distance back to the
182     start of the current branch.
183    
184     OP_MARK is followed by the mark name, preceded by a one-byte length, and
185     followed by a binary zero. For (*PRUNE), (*SKIP), and (*THEN) with arguments,
186     the opcodes OP_PRUNE_ARG, OP_SKIP_ARG, and OP_THEN_ARG are used. For the first
187     two, the name follows immediately; for OP_THEN_ARG, it follows the LINK_SIZE
188     offset value.
189    
190    
191 ph10 602 Matching literal characters
192     ---------------------------
193    
194     The OP_CHAR opcode is followed by a single character that is to be matched
195     casefully. For caseless matching, OP_CHARI is used. In UTF-8 mode, the
196     character may be more than one byte long. (Earlier versions of PCRE used
197     multi-character strings, but this was changed to allow some new features to be
198     added.)
199    
200    
201 nigel 41 Repeating single characters
202     ---------------------------
203    
204 nigel 75 The common repeats (*, +, ?) when applied to a single character use the
205 ph10 602 following opcodes, which come in caseful and caseless versions:
206 nigel 41
207 ph10 602 Caseful Caseless
208     OP_STAR OP_STARI
209     OP_MINSTAR OP_MINSTARI
210     OP_POSSTAR OP_POSSTARI
211     OP_PLUS OP_PLUSI
212     OP_MINPLUS OP_MINPLUSI
213     OP_POSPLUS OP_POSPLUSI
214     OP_QUERY OP_QUERYI
215     OP_MINQUERY OP_MINQUERYI
216     OP_POSQUERY OP_POSQUERYI
217 nigel 41
218 nigel 75 In ASCII mode, these are two-byte items; in UTF-8 mode, the length is variable.
219 nigel 93 Those with "MIN" in their name are the minimizing versions. Those with "POS" in
220     their names are possessive versions. Each is followed by the character that is
221 ph10 602 to be repeated. Other repeats make use of these opcodes:
222 nigel 41
223 ph10 602 Caseful Caseless
224     OP_UPTO OP_UPTOI
225     OP_MINUPTO OP_MINUPTOI
226     OP_POSUPTO OP_POSUPTOI
227     OP_EXACT OP_EXACTI
228 nigel 41
229 ph10 602 Each of these is followed by a two-byte count (most significant first) and the
230 nigel 41 repeated character. OP_UPTO matches from 0 to the given number. A repeat with a
231     non-zero minimum and a fixed maximum is coded as an OP_EXACT followed by an
232 nigel 93 OP_UPTO (or OP_MINUPTO or OPT_POSUPTO).
233 nigel 41
234    
235     Repeating character types
236     -------------------------
237    
238     Repeats of things like \d are done exactly as for single characters, except
239     that instead of a character, the opcode for the type is stored in the data
240     byte. The opcodes are:
241    
242     OP_TYPESTAR
243     OP_TYPEMINSTAR
244 nigel 93 OP_TYPEPOSSTAR
245 nigel 41 OP_TYPEPLUS
246     OP_TYPEMINPLUS
247 nigel 93 OP_TYPEPOSPLUS
248 nigel 41 OP_TYPEQUERY
249     OP_TYPEMINQUERY
250 nigel 93 OP_TYPEPOSQUERY
251 nigel 41 OP_TYPEUPTO
252     OP_TYPEMINUPTO
253 nigel 93 OP_TYPEPOSUPTO
254 nigel 41 OP_TYPEEXACT
255    
256    
257 nigel 75 Match by Unicode property
258     -------------------------
259    
260     OP_PROP and OP_NOTPROP are used for positive and negative matches of a
261     character by testing its Unicode property (the \p and \P escape sequences).
262 nigel 91 Each is followed by two bytes that encode the desired property as a type and a
263     value.
264 nigel 75
265 nigel 91 Repeats of these items use the OP_TYPESTAR etc. set of opcodes, followed by
266     three bytes: OP_PROP or OP_NOTPROP and then the desired property type and
267     value.
268 nigel 75
269    
270 nigel 41 Character classes
271     -----------------
272    
273 ph10 602 If there is only one character, OP_CHAR or OP_CHARI is used for a positive
274     class, and OP_NOT or OP_NOTI for a negative one (that is, for something like
275     [^a]). However, in UTF-8 mode, the use of OP_NOT[I] applies only to characters
276     with values < 128, because OP_NOT[I] is confined to single bytes.
277 nigel 41
278 ph10 602 Another set of 13 repeating opcodes (called OP_NOTSTAR etc.) are used for a
279     repeated, negated, single-character class. The normal single-character opcodes
280     (OP_STAR, etc.) are used for a repeated positive single-character class.
281 nigel 63
282 ph10 602 When there is more than one character in a class and all the characters are
283     less than 256, OP_CLASS is used for a positive class, and OP_NCLASS for a
284     negative one. In either case, the opcode is followed by a 32-byte bit map
285     containing a 1 bit for every character that is acceptable. The bits are counted
286     from the least significant end of each byte. In caseless mode, bits for both
287     cases are set.
288 nigel 41
289 nigel 75 The reason for having both OP_CLASS and OP_NCLASS is so that, in UTF-8 mode,
290     subject characters with values greater than 256 can be handled correctly. For
291 ph10 602 OP_CLASS they do not match, whereas for OP_NCLASS they do.
292 nigel 71
293 nigel 63 For classes containing characters with values > 255, OP_XCLASS is used. It
294     optionally uses a bit map (if any characters lie within it), followed by a list
295 ph10 602 of pairs (for a range) and single characters. In caseless mode, both cases are
296     explicitly listed. There is a flag character than indicates whether it is a
297     positive or a negative class.
298 nigel 41
299 nigel 63
300 nigel 41 Back references
301     ---------------
302    
303 ph10 602 OP_REF (caseful) or OP_REFI (caseless) is followed by two bytes containing the
304     reference number.
305 nigel 41
306    
307     Repeating character classes and back references
308     -----------------------------------------------
309    
310 nigel 93 Single-character classes are handled specially (see above). This section
311 ph10 602 applies to OP_CLASS and OP_REF[I]. In both cases, the repeat information
312     follows the base item. The matching code looks at the following opcode to see
313     if it is one of
314 nigel 41
315     OP_CRSTAR
316     OP_CRMINSTAR
317     OP_CRPLUS
318     OP_CRMINPLUS
319     OP_CRQUERY
320     OP_CRMINQUERY
321     OP_CRRANGE
322     OP_CRMINRANGE
323    
324     All but the last two are just single-byte items. The others are followed by
325 nigel 93 four bytes of data, comprising the minimum and maximum repeat counts. There are
326     no special possessive opcodes for these repeats; a possessive repeat is
327     compiled into an atomic group.
328 nigel 41
329    
330     Brackets and alternation
331     ------------------------
332    
333 nigel 43 A pair of non-capturing (round) brackets is wrapped round each expression at
334 nigel 41 compile time, so alternation always happens in the context of brackets.
335 nigel 53
336 nigel 93 [Note for North Americans: "bracket" to some English speakers, including
337     myself, can be round, square, curly, or pointy. Hence this usage.]
338 nigel 41
339 nigel 93 Non-capturing brackets use the opcode OP_BRA. Originally PCRE was limited to 99
340     capturing brackets and it used a different opcode for each one. From release
341     3.5, the limit was removed by putting the bracket number into the data for
342     higher-numbered brackets. From release 7.0 all capturing brackets are handled
343     this way, using the single opcode OP_CBRA.
344 nigel 53
345 nigel 77 A bracket opcode is followed by LINK_SIZE bytes which give the offset to the
346     next alternative OP_ALT or, if there aren't any branches, to the matching
347     OP_KET opcode. Each OP_ALT is followed by LINK_SIZE bytes giving the offset to
348 nigel 93 the next one, or to the OP_KET opcode. For capturing brackets, the bracket
349     number immediately follows the offset, always as a 2-byte item.
350 nigel 41
351     OP_KET is used for subpatterns that do not repeat indefinitely, while
352     OP_KETRMIN and OP_KETRMAX are used for indefinite repetitions, minimally or
353 ph10 604 maximally respectively (see below for possessive repetitions). All three are
354     followed by LINK_SIZE bytes giving (as a positive number) the offset back to
355     the matching bracket opcode.
356 nigel 41
357     If a subpattern is quantified such that it is permitted to match zero times, it
358 ph10 335 is preceded by one of OP_BRAZERO, OP_BRAMINZERO, or OP_SKIPZERO. These are
359     single-byte opcodes that tell the matcher that skipping the following
360     subpattern entirely is a valid branch. In the case of the first two, not
361     skipping the pattern is also valid (greedy and non-greedy). The third is used
362     when a pattern has the quantifier {0,0}. It cannot be entirely discarded,
363     because it may be called as a subroutine from elsewhere in the regex.
364 nigel 41
365     A subpattern with an indefinite maximum repetition is replicated in the
366 nigel 75 compiled data its minimum number of times (or once with OP_BRAZERO if the
367     minimum is zero), with the final copy terminating with OP_KETRMIN or OP_KETRMAX
368     as appropriate.
369 nigel 41
370     A subpattern with a bounded maximum repetition is replicated in a nested
371 nigel 75 fashion up to the maximum number of times, with OP_BRAZERO or OP_BRAMINZERO
372     before each replication after the minimum, so that, for example, (abc){2,5} is
373 nigel 93 compiled as (abc)(abc)((abc)((abc)(abc)?)?)?, except that each bracketed group
374     has the same number.
375 nigel 41
376 nigel 93 When a repeated subpattern has an unbounded upper limit, it is checked to see
377     whether it could match an empty string. If this is the case, the opcode in the
378     final replication is changed to OP_SBRA or OP_SCBRA. This tells the matcher
379     that it needs to check for matching an empty string when it hits OP_KETRMIN or
380     OP_KETRMAX, and if so, to break the loop.
381 nigel 41
382 ph10 604 Possessive brackets
383     -------------------
384 nigel 93
385 ph10 604 When a repeated group (capturing or non-capturing) is marked as possessive by
386     the "+" notation, e.g. (abc)++, different opcodes are used. Their names all
387     have POS on the end, e.g. OP_BRAPOS instead of OP_BRA and OP_SCPBRPOS instead
388     of OP_SCBRA. The end of such a group is marked by OP_KETRPOS. If the minimum
389     repetition is zero, the group is preceded by OP_BRAPOSZERO.
390    
391    
392 nigel 41 Assertions
393     ----------
394    
395     Forward assertions are just like other subpatterns, but starting with one of
396     the opcodes OP_ASSERT or OP_ASSERT_NOT. Backward assertions use the opcodes
397     OP_ASSERTBACK and OP_ASSERTBACK_NOT, and the first opcode inside the assertion
398     is OP_REVERSE, followed by a two byte count of the number of characters to move
399 nigel 49 back the pointer in the subject string. When operating in UTF-8 mode, the count
400     is a character count rather than a byte count. A separate count is present in
401     each alternative of a lookbehind assertion, allowing them to have different
402     fixed lengths.
403 nigel 41
404    
405 nigel 93 Once-only (atomic) subpatterns
406     ------------------------------
407 nigel 41
408     These are also just like other subpatterns, but they start with the opcode
409 nigel 93 OP_ONCE. The check for matching an empty string in an unbounded repeat is
410     handled entirely at runtime, so there is just this one opcode.
411 nigel 41
412    
413     Conditional subpatterns
414     -----------------------
415    
416 nigel 93 These are like other subpatterns, but they start with the opcode OP_COND, or
417     OP_SCOND for one that might match an empty string in an unbounded repeat. If
418 nigel 41 the condition is a back reference, this is stored at the start of the
419 nigel 53 subpattern using the opcode OP_CREF followed by two bytes containing the
420 ph10 460 reference number. OP_NCREF is used instead if the reference was generated by
421     name (so that the runtime code knows to check for duplicate names).
422 nigel 41
423 ph10 460 If the condition is "in recursion" (coded as "(?(R)"), or "in recursion of
424     group x" (coded as "(?(Rx)"), the group number is stored at the start of the
425     subpattern using the opcode OP_RREF or OP_NRREF (cf OP_NCREF), and a value of
426     zero for "the whole pattern". For a DEFINE condition, just the single byte
427     OP_DEF is used (it has no associated data). Otherwise, a conditional subpattern
428     always starts with one of the assertions.
429 nigel 41
430 ph10 460
431 nigel 71 Recursion
432     ---------
433    
434     Recursion either matches the current regex, or some subexpression. The opcode
435     OP_RECURSE is followed by an value which is the offset to the starting bracket
436 nigel 87 from the start of the whole pattern. From release 6.5, OP_RECURSE is
437     automatically wrapped inside OP_ONCE brackets (because otherwise some patterns
438     broke it). OP_RECURSE is also used for "subroutine" calls, even though they
439     are not strictly a recursion.
440 nigel 71
441    
442     Callout
443     -------
444    
445 nigel 75 OP_CALLOUT is followed by one byte of data that holds a callout number in the
446     range 0 to 254 for manual callouts, or 255 for an automatic callout. In both
447     cases there follows a two-byte value giving the offset in the pattern to the
448     start of the following item, and another two-byte item giving the length of the
449     next item.
450 nigel 71
451    
452 nigel 41 Philip Hazel
453 ph10 612 July 2011

webmaster@exim.org
ViewVC Help
Powered by ViewVC 1.1.12