Parent Directory | Revision Log

Revision **148** -
(**show annotations**)
(**download**)

*Mon Apr 16 13:25:10 2007 UTC*
(7 years ago)
by *ph10*

File size: 7868 byte(s)

File size: 7868 byte(s)

Fix bugs in the documentation.

1 | .TH PCREMATCHING 3 |

2 | .SH NAME |

3 | PCRE - Perl-compatible regular expressions |

4 | .SH "PCRE MATCHING ALGORITHMS" |

5 | .rs |

6 | .sp |

7 | This document describes the two different algorithms that are available in PCRE |

8 | for matching a compiled regular expression against a given subject string. The |

9 | "standard" algorithm is the one provided by the \fBpcre_exec()\fP function. |

10 | This works in the same was as Perl's matching function, and provides a |

11 | Perl-compatible matching operation. |

12 | .P |

13 | An alternative algorithm is provided by the \fBpcre_dfa_exec()\fP function; |

14 | this operates in a different way, and is not Perl-compatible. It has advantages |

15 | and disadvantages compared with the standard algorithm, and these are described |

16 | below. |

17 | .P |

18 | When there is only one possible way in which a given subject string can match a |

19 | pattern, the two algorithms give the same answer. A difference arises, however, |

20 | when there are multiple possibilities. For example, if the pattern |

21 | .sp |

22 | ^<.*> |

23 | .sp |

24 | is matched against the string |

25 | .sp |

26 | <something> <something else> <something further> |

27 | .sp |

28 | there are three possible answers. The standard algorithm finds only one of |

29 | them, whereas the alternative algorithm finds all three. |

30 | . |

31 | .SH "REGULAR EXPRESSIONS AS TREES" |

32 | .rs |

33 | .sp |

34 | The set of strings that are matched by a regular expression can be represented |

35 | as a tree structure. An unlimited repetition in the pattern makes the tree of |

36 | infinite size, but it is still a tree. Matching the pattern to a given subject |

37 | string (from a given starting point) can be thought of as a search of the tree. |

38 | There are two ways to search a tree: depth-first and breadth-first, and these |

39 | correspond to the two matching algorithms provided by PCRE. |

40 | . |

41 | .SH "THE STANDARD MATCHING ALGORITHM" |

42 | .rs |

43 | .sp |

44 | In the terminology of Jeffrey Friedl's book "Mastering Regular |

45 | Expressions", the standard algorithm is an "NFA algorithm". It conducts a |

46 | depth-first search of the pattern tree. That is, it proceeds along a single |

47 | path through the tree, checking that the subject matches what is required. When |

48 | there is a mismatch, the algorithm tries any alternatives at the current point, |

49 | and if they all fail, it backs up to the previous branch point in the tree, and |

50 | tries the next alternative branch at that level. This often involves backing up |

51 | (moving to the left) in the subject string as well. The order in which |

52 | repetition branches are tried is controlled by the greedy or ungreedy nature of |

53 | the quantifier. |

54 | .P |

55 | If a leaf node is reached, a matching string has been found, and at that point |

56 | the algorithm stops. Thus, if there is more than one possible match, this |

57 | algorithm returns the first one that it finds. Whether this is the shortest, |

58 | the longest, or some intermediate length depends on the way the greedy and |

59 | ungreedy repetition quantifiers are specified in the pattern. |

60 | .P |

61 | Because it ends up with a single path through the tree, it is relatively |

62 | straightforward for this algorithm to keep track of the substrings that are |

63 | matched by portions of the pattern in parentheses. This provides support for |

64 | capturing parentheses and back references. |

65 | . |

66 | .SH "THE ALTERNATIVE MATCHING ALGORITHM" |

67 | .rs |

68 | .sp |

69 | This algorithm conducts a breadth-first search of the tree. Starting from the |

70 | first matching point in the subject, it scans the subject string from left to |

71 | right, once, character by character, and as it does this, it remembers all the |

72 | paths through the tree that represent valid matches. In Friedl's terminology, |

73 | this is a kind of "DFA algorithm", though it is not implemented as a |

74 | traditional finite state machine (it keeps multiple states active |

75 | simultaneously). |

76 | .P |

77 | The scan continues until either the end of the subject is reached, or there are |

78 | no more unterminated paths. At this point, terminated paths represent the |

79 | different matching possibilities (if there are none, the match has failed). |

80 | Thus, if there is more than one possible match, this algorithm finds all of |

81 | them, and in particular, it finds the longest. In PCRE, there is an option to |

82 | stop the algorithm after the first match (which is necessarily the shortest) |

83 | has been found. |

84 | .P |

85 | Note that all the matches that are found start at the same point in the |

86 | subject. If the pattern |

87 | .sp |

88 | cat(er(pillar)?) |

89 | .sp |

90 | is matched against the string "the caterpillar catchment", the result will be |

91 | the three strings "cat", "cater", and "caterpillar" that start at the fourth |

92 | character of the subject. The algorithm does not automatically move on to find |

93 | matches that start at later positions. |

94 | .P |

95 | There are a number of features of PCRE regular expressions that are not |

96 | supported by the alternative matching algorithm. They are as follows: |

97 | .P |

98 | 1. Because the algorithm finds all possible matches, the greedy or ungreedy |

99 | nature of repetition quantifiers is not relevant. Greedy and ungreedy |

100 | quantifiers are treated in exactly the same way. However, possessive |

101 | quantifiers can make a difference when what follows could also match what is |

102 | quantified, for example in a pattern like this: |

103 | .sp |

104 | ^a++\ew! |

105 | .sp |

106 | This pattern matches "aaab!" but not "aaa!", which would be matched by a |

107 | non-possessive quantifier. Similarly, if an atomic group is present, it is |

108 | matched as if it were a standalone pattern at the current point, and the |

109 | longest match is then "locked in" for the rest of the overall pattern. |

110 | .P |

111 | 2. When dealing with multiple paths through the tree simultaneously, it is not |

112 | straightforward to keep track of captured substrings for the different matching |

113 | possibilities, and PCRE's implementation of this algorithm does not attempt to |

114 | do this. This means that no captured substrings are available. |

115 | .P |

116 | 3. Because no substrings are captured, back references within the pattern are |

117 | not supported, and cause errors if encountered. |

118 | .P |

119 | 4. For the same reason, conditional expressions that use a backreference as the |

120 | condition or test for a specific group recursion are not supported. |

121 | .P |

122 | 5. Callouts are supported, but the value of the \fIcapture_top\fP field is |

123 | always 1, and the value of the \fIcapture_last\fP field is always -1. |

124 | .P |

125 | 6. |

126 | The \eC escape sequence, which (in the standard algorithm) matches a single |

127 | byte, even in UTF-8 mode, is not supported because the alternative algorithm |

128 | moves through the subject string one character at a time, for all active paths |

129 | through the tree. |

130 | . |

131 | .SH "ADVANTAGES OF THE ALTERNATIVE ALGORITHM" |

132 | .rs |

133 | .sp |

134 | Using the alternative matching algorithm provides the following advantages: |

135 | .P |

136 | 1. All possible matches (at a single point in the subject) are automatically |

137 | found, and in particular, the longest match is found. To find more than one |

138 | match using the standard algorithm, you have to do kludgy things with |

139 | callouts. |

140 | .P |

141 | 2. There is much better support for partial matching. The restrictions on the |

142 | content of the pattern that apply when using the standard algorithm for partial |

143 | matching do not apply to the alternative algorithm. For non-anchored patterns, |

144 | the starting position of a partial match is available. |

145 | .P |

146 | 3. Because the alternative algorithm scans the subject string just once, and |

147 | never needs to backtrack, it is possible to pass very long subject strings to |

148 | the matching function in several pieces, checking for partial matching each |

149 | time. |

150 | . |

151 | .SH "DISADVANTAGES OF THE ALTERNATIVE ALGORITHM" |

152 | .rs |

153 | .sp |

154 | The alternative algorithm suffers from a number of disadvantages: |

155 | .P |

156 | 1. It is substantially slower than the standard algorithm. This is partly |

157 | because it has to search for all possible matches, but is also because it is |

158 | less susceptible to optimization. |

159 | .P |

160 | 2. Capturing parentheses and back references are not supported. |

161 | .P |

162 | 3. Although atomic groups are supported, their use does not provide the |

163 | performance advantage that it does for the standard algorithm. |

164 | . |

165 | . |

166 | .SH AUTHOR |

167 | .rs |

168 | .sp |

169 | .nf |

170 | Philip Hazel |

171 | University Computing Service |

172 | Cambridge CB2 3QH, England. |

173 | .fi |

174 | . |

175 | . |

176 | .SH REVISION |

177 | .rs |

178 | .sp |

179 | .nf |

180 | Last updated: 06 March 2007 |

181 | Copyright (c) 1997-2007 University of Cambridge. |

182 | .fi |

Name | Value |
---|---|

svn:eol-style |
native |

svn:keywords |
"Author Date Id Revision Url" |

webmaster@exim.org | ViewVC Help |

Powered by ViewVC 1.1.12 |