| 1 |
.TH PCREPARTIAL 3
|
| 2 |
.SH NAME
|
| 3 |
PCRE - Perl-compatible regular expressions
|
| 4 |
.SH "PARTIAL MATCHING IN PCRE"
|
| 5 |
.rs
|
| 6 |
.sp
|
| 7 |
In normal use of PCRE, if the subject string that is passed to
|
| 8 |
\fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP matches as far as it goes, but is
|
| 9 |
too short to match the entire pattern, PCRE_ERROR_NOMATCH is returned. There
|
| 10 |
are circumstances where it might be helpful to distinguish this case from other
|
| 11 |
cases in which there is no match.
|
| 12 |
.P
|
| 13 |
Consider, for example, an application where a human is required to type in data
|
| 14 |
for a field with specific formatting requirements. An example might be a date
|
| 15 |
in the form \fIddmmmyy\fP, defined by this pattern:
|
| 16 |
.sp
|
| 17 |
^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$
|
| 18 |
.sp
|
| 19 |
If the application sees the user's keystrokes one by one, and can check that
|
| 20 |
what has been typed so far is potentially valid, it is able to raise an error
|
| 21 |
as soon as a mistake is made, by beeping and not reflecting the character that
|
| 22 |
has been typed, for example. This immediate feedback is likely to be a better
|
| 23 |
user interface than a check that is delayed until the entire string has been
|
| 24 |
entered. Partial matching can also sometimes be useful when the subject string
|
| 25 |
is very long and is not all available at once.
|
| 26 |
.P
|
| 27 |
PCRE supports partial matching by means of the PCRE_PARTIAL_SOFT and
|
| 28 |
PCRE_PARTIAL_HARD options, which can be set when calling \fBpcre_exec()\fP or
|
| 29 |
\fBpcre_dfa_exec()\fP. For backwards compatibility, PCRE_PARTIAL is a synonym
|
| 30 |
for PCRE_PARTIAL_SOFT. The essential difference between the two options is
|
| 31 |
whether or not a partial match is preferred to an alternative complete match,
|
| 32 |
though the details differ between the two matching functions. If both options
|
| 33 |
are set, PCRE_PARTIAL_HARD takes precedence.
|
| 34 |
.P
|
| 35 |
Setting a partial matching option disables two of PCRE's optimizations. PCRE
|
| 36 |
remembers the last literal byte in a pattern, and abandons matching immediately
|
| 37 |
if such a byte is not present in the subject string. This optimization cannot
|
| 38 |
be used for a subject string that might match only partially. If the pattern
|
| 39 |
was studied, PCRE knows the minimum length of a matching string, and does not
|
| 40 |
bother to run the matching function on shorter strings. This optimization is
|
| 41 |
also disabled for partial matching.
|
| 42 |
.
|
| 43 |
.
|
| 44 |
.SH "PARTIAL MATCHING USING pcre_exec()"
|
| 45 |
.rs
|
| 46 |
.sp
|
| 47 |
A partial match occurs during a call to \fBpcre_exec()\fP whenever the end of
|
| 48 |
the subject string is reached successfully, but matching cannot continue
|
| 49 |
because more characters are needed. However, at least one character must have
|
| 50 |
been matched. (In other words, a partial match can never be an empty string.)
|
| 51 |
.P
|
| 52 |
If PCRE_PARTIAL_SOFT is set, the partial match is remembered, but matching
|
| 53 |
continues as normal, and other alternatives in the pattern are tried. If no
|
| 54 |
complete match can be found, \fBpcre_exec()\fP returns PCRE_ERROR_PARTIAL
|
| 55 |
instead of PCRE_ERROR_NOMATCH. If there are at least two slots in the offsets
|
| 56 |
vector, the first of them is set to the offset of the earliest character that
|
| 57 |
was inspected when the partial match was found. For convenience, the second
|
| 58 |
offset points to the end of the string so that a substring can easily be
|
| 59 |
identified.
|
| 60 |
.P
|
| 61 |
For the majority of patterns, the first offset identifies the start of the
|
| 62 |
partially matched string. However, for patterns that contain lookbehind
|
| 63 |
assertions, or \eK, or begin with \eb or \eB, earlier characters have been
|
| 64 |
inspected while carrying out the match. For example:
|
| 65 |
.sp
|
| 66 |
/(?<=abc)123/
|
| 67 |
.sp
|
| 68 |
This pattern matches "123", but only if it is preceded by "abc". If the subject
|
| 69 |
string is "xyzabc12", the offsets after a partial match are for the substring
|
| 70 |
"abc12", because all these characters are needed if another match is tried
|
| 71 |
with extra characters added.
|
| 72 |
.P
|
| 73 |
If there is more than one partial match, the first one that was found provides
|
| 74 |
the data that is returned. Consider this pattern:
|
| 75 |
.sp
|
| 76 |
/123\ew+X|dogY/
|
| 77 |
.sp
|
| 78 |
If this is matched against the subject string "abc123dog", both
|
| 79 |
alternatives fail to match, but the end of the subject is reached during
|
| 80 |
matching, so PCRE_ERROR_PARTIAL is returned instead of PCRE_ERROR_NOMATCH. The
|
| 81 |
offsets are set to 3 and 9, identifying "123dog" as the first partial match
|
| 82 |
that was found. (In this example, there are two partial matches, because "dog"
|
| 83 |
on its own partially matches the second alternative.)
|
| 84 |
.P
|
| 85 |
If PCRE_PARTIAL_HARD is set for \fBpcre_exec()\fP, it returns
|
| 86 |
PCRE_ERROR_PARTIAL as soon as a partial match is found, without continuing to
|
| 87 |
search for possible complete matches. The difference between the two options
|
| 88 |
can be illustrated by a pattern such as:
|
| 89 |
.sp
|
| 90 |
/dog(sbody)?/
|
| 91 |
.sp
|
| 92 |
This matches either "dog" or "dogsbody", greedily (that is, it prefers the
|
| 93 |
longer string if possible). If it is matched against the string "dog" with
|
| 94 |
PCRE_PARTIAL_SOFT, it yields a complete match for "dog". However, if
|
| 95 |
PCRE_PARTIAL_HARD is set, the result is PCRE_ERROR_PARTIAL. On the other hand,
|
| 96 |
if the pattern is made ungreedy the result is different:
|
| 97 |
.sp
|
| 98 |
/dog(sbody)??/
|
| 99 |
.sp
|
| 100 |
In this case the result is always a complete match because \fBpcre_exec()\fP
|
| 101 |
finds that first, and it never continues after finding a match. It might be
|
| 102 |
easier to follow this explanation by thinking of the two patterns like this:
|
| 103 |
.sp
|
| 104 |
/dog(sbody)?/ is the same as /dogsbody|dog/
|
| 105 |
/dog(sbody)??/ is the same as /dog|dogsbody/
|
| 106 |
.sp
|
| 107 |
The second pattern will never match "dogsbody" when \fBpcre_exec()\fP is
|
| 108 |
used, because it will always find the shorter match first.
|
| 109 |
.
|
| 110 |
.
|
| 111 |
.SH "PARTIAL MATCHING USING pcre_dfa_exec()"
|
| 112 |
.rs
|
| 113 |
.sp
|
| 114 |
The \fBpcre_dfa_exec()\fP function moves along the subject string character by
|
| 115 |
character, without backtracking, searching for all possible matches
|
| 116 |
simultaneously. If the end of the subject is reached before the end of the
|
| 117 |
pattern, there is the possibility of a partial match, again provided that at
|
| 118 |
least one character has matched.
|
| 119 |
.P
|
| 120 |
When PCRE_PARTIAL_SOFT is set, PCRE_ERROR_PARTIAL is returned only if there
|
| 121 |
have been no complete matches. Otherwise, the complete matches are returned.
|
| 122 |
However, if PCRE_PARTIAL_HARD is set, a partial match takes precedence over any
|
| 123 |
complete matches. The portion of the string that was inspected when the longest
|
| 124 |
partial match was found is set as the first matching string, provided there are
|
| 125 |
at least two slots in the offsets vector.
|
| 126 |
.P
|
| 127 |
Because \fBpcre_dfa_exec()\fP always searches for all possible matches, and
|
| 128 |
there is no difference between greedy and ungreedy repetition, its behaviour is
|
| 129 |
different from \fBpcre_exec\fP when PCRE_PARTIAL_HARD is set. Consider the
|
| 130 |
string "dog" matched against the ungreedy pattern shown above:
|
| 131 |
.sp
|
| 132 |
/dog(sbody)??/
|
| 133 |
.sp
|
| 134 |
Whereas \fBpcre_exec()\fP stops as soon as it finds the complete match for
|
| 135 |
"dog", \fBpcre_dfa_exec()\fP also finds the partial match for "dogsbody", and
|
| 136 |
so returns that when PCRE_PARTIAL_HARD is set.
|
| 137 |
.
|
| 138 |
.
|
| 139 |
.SH "PARTIAL MATCHING AND WORD BOUNDARIES"
|
| 140 |
.rs
|
| 141 |
.sp
|
| 142 |
If a pattern ends with one of sequences \ew or \eW, which test for word
|
| 143 |
boundaries, partial matching with PCRE_PARTIAL_SOFT can give counter-intuitive
|
| 144 |
results. Consider this pattern:
|
| 145 |
.sp
|
| 146 |
/\ebcat\eb/
|
| 147 |
.sp
|
| 148 |
This matches "cat", provided there is a word boundary at either end. If the
|
| 149 |
subject string is "the cat", the comparison of the final "t" with a following
|
| 150 |
character cannot take place, so a partial match is found. However,
|
| 151 |
\fBpcre_exec()\fP carries on with normal matching, which matches \eb at the end
|
| 152 |
of the subject when the last character is a letter, thus finding a complete
|
| 153 |
match. The result, therefore, is \fInot\fP PCRE_ERROR_PARTIAL. The same thing
|
| 154 |
happens with \fBpcre_dfa_exec()\fP, because it also finds the complete match.
|
| 155 |
.P
|
| 156 |
Using PCRE_PARTIAL_HARD in this case does yield PCRE_ERROR_PARTIAL, because
|
| 157 |
then the partial match takes precedence.
|
| 158 |
.
|
| 159 |
.
|
| 160 |
.SH "FORMERLY RESTRICTED PATTERNS"
|
| 161 |
.rs
|
| 162 |
.sp
|
| 163 |
For releases of PCRE prior to 8.00, because of the way certain internal
|
| 164 |
optimizations were implemented in the \fBpcre_exec()\fP function, the
|
| 165 |
PCRE_PARTIAL option (predecessor of PCRE_PARTIAL_SOFT) could not be used with
|
| 166 |
all patterns. From release 8.00 onwards, the restrictions no longer apply, and
|
| 167 |
partial matching with \fBpcre_exec()\fP can be requested for any pattern.
|
| 168 |
.P
|
| 169 |
Items that were formerly restricted were repeated single characters and
|
| 170 |
repeated metasequences. If PCRE_PARTIAL was set for a pattern that did not
|
| 171 |
conform to the restrictions, \fBpcre_exec()\fP returned the error code
|
| 172 |
PCRE_ERROR_BADPARTIAL (-13). This error code is no longer in use. The
|
| 173 |
PCRE_INFO_OKPARTIAL call to \fBpcre_fullinfo()\fP to find out if a compiled
|
| 174 |
pattern can be used for partial matching now always returns 1.
|
| 175 |
.
|
| 176 |
.
|
| 177 |
.SH "EXAMPLE OF PARTIAL MATCHING USING PCRETEST"
|
| 178 |
.rs
|
| 179 |
.sp
|
| 180 |
If the escape sequence \eP is present in a \fBpcretest\fP data line, the
|
| 181 |
PCRE_PARTIAL_SOFT option is used for the match. Here is a run of \fBpcretest\fP
|
| 182 |
that uses the date example quoted above:
|
| 183 |
.sp
|
| 184 |
re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/
|
| 185 |
data> 25jun04\eP
|
| 186 |
0: 25jun04
|
| 187 |
1: jun
|
| 188 |
data> 25dec3\eP
|
| 189 |
Partial match: 23dec3
|
| 190 |
data> 3ju\eP
|
| 191 |
Partial match: 3ju
|
| 192 |
data> 3juj\eP
|
| 193 |
No match
|
| 194 |
data> j\eP
|
| 195 |
No match
|
| 196 |
.sp
|
| 197 |
The first data string is matched completely, so \fBpcretest\fP shows the
|
| 198 |
matched substrings. The remaining four strings do not match the complete
|
| 199 |
pattern, but the first two are partial matches. Similar output is obtained
|
| 200 |
when \fBpcre_dfa_exec()\fP is used.
|
| 201 |
.P
|
| 202 |
If the escape sequence \eP is present more than once in a \fBpcretest\fP data
|
| 203 |
line, the PCRE_PARTIAL_HARD option is set for the match.
|
| 204 |
.
|
| 205 |
.
|
| 206 |
.SH "MULTI-SEGMENT MATCHING WITH pcre_dfa_exec()"
|
| 207 |
.rs
|
| 208 |
.sp
|
| 209 |
When a partial match has been found using \fBpcre_dfa_exec()\fP, it is possible
|
| 210 |
to continue the match by providing additional subject data and calling
|
| 211 |
\fBpcre_dfa_exec()\fP again with the same compiled regular expression, this
|
| 212 |
time setting the PCRE_DFA_RESTART option. You must pass the same working
|
| 213 |
space as before, because this is where details of the previous partial match
|
| 214 |
are stored. Here is an example using \fBpcretest\fP, using the \eR escape
|
| 215 |
sequence to set the PCRE_DFA_RESTART option (\eD specifies the use of
|
| 216 |
\fBpcre_dfa_exec()\fP):
|
| 217 |
.sp
|
| 218 |
re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/
|
| 219 |
data> 23ja\eP\eD
|
| 220 |
Partial match: 23ja
|
| 221 |
data> n05\eR\eD
|
| 222 |
0: n05
|
| 223 |
.sp
|
| 224 |
The first call has "23ja" as the subject, and requests partial matching; the
|
| 225 |
second call has "n05" as the subject for the continued (restarted) match.
|
| 226 |
Notice that when the match is complete, only the last part is shown; PCRE does
|
| 227 |
not retain the previously partially-matched string. It is up to the calling
|
| 228 |
program to do that if it needs to.
|
| 229 |
.P
|
| 230 |
You can set the PCRE_PARTIAL_SOFT or PCRE_PARTIAL_HARD options with
|
| 231 |
PCRE_DFA_RESTART to continue partial matching over multiple segments. This
|
| 232 |
facility can be used to pass very long subject strings to
|
| 233 |
\fBpcre_dfa_exec()\fP.
|
| 234 |
.
|
| 235 |
.
|
| 236 |
.SH "MULTI-SEGMENT MATCHING WITH pcre_exec()"
|
| 237 |
.rs
|
| 238 |
.sp
|
| 239 |
From release 8.00, \fBpcre_exec()\fP can also be used to do multi-segment
|
| 240 |
matching. Unlike \fBpcre_dfa_exec()\fP, it is not possible to restart the
|
| 241 |
previous match with a new segment of data. Instead, new data must be added to
|
| 242 |
the previous subject string, and the entire match re-run, starting from the
|
| 243 |
point where the partial match occurred. Earlier data can be discarded.
|
| 244 |
Consider an unanchored pattern that matches dates:
|
| 245 |
.sp
|
| 246 |
re> /\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed/
|
| 247 |
data> The date is 23ja\eP
|
| 248 |
Partial match: 23ja
|
| 249 |
.sp
|
| 250 |
The this stage, an application could discard the text preceding "23ja", add on
|
| 251 |
text from the next segment, and call \fBpcre_exec()\fP again. Unlike
|
| 252 |
\fBpcre_dfa_exec()\fP, the entire matching string must always be available, and
|
| 253 |
the complete matching process occurs for each call, so more memory and more
|
| 254 |
processing time is needed.
|
| 255 |
.P
|
| 256 |
\fBNote:\fP If the pattern contains lookbehind assertions, or \eK, or starts
|
| 257 |
with \eb or \eB, the string that is returned for a partial match will include
|
| 258 |
characters that precede the partially matched string itself, because these must
|
| 259 |
be retained when adding on more characters for a subsequent matching attempt.
|
| 260 |
.
|
| 261 |
.
|
| 262 |
.SH "ISSUES WITH MULTI-SEGMENT MATCHING"
|
| 263 |
.rs
|
| 264 |
.sp
|
| 265 |
Certain types of pattern may give problems with multi-segment matching,
|
| 266 |
whichever matching function is used.
|
| 267 |
.P
|
| 268 |
1. If the pattern contains tests for the beginning or end of a line, you need
|
| 269 |
to pass the PCRE_NOTBOL or PCRE_NOTEOL options, as appropriate, when the
|
| 270 |
subject string for any call does not contain the beginning or end of a line.
|
| 271 |
.P
|
| 272 |
2. Lookbehind assertions at the start of a pattern are catered for in the
|
| 273 |
offsets that are returned for a partial match. However, in theory, a lookbehind
|
| 274 |
assertion later in the pattern could require even earlier characters to be
|
| 275 |
inspected, and it might not have been reached when a partial match occurs. This
|
| 276 |
is probably an extremely unlikely case; you could guard against it to a certain
|
| 277 |
extent by always including extra characters at the start.
|
| 278 |
.P
|
| 279 |
3. Matching a subject string that is split into multiple segments may not
|
| 280 |
always produce exactly the same result as matching over one single long string,
|
| 281 |
especially when PCRE_PARTIAL_SOFT is used. The section "Partial Matching and
|
| 282 |
Word Boundaries" above describes an issue that arises if the pattern ends with
|
| 283 |
\eb or \eB. Another kind of difference may occur when there are multiple
|
| 284 |
matching possibilities, because a partial match result is given only when there
|
| 285 |
are no completed matches. This means that as soon as the shortest match has
|
| 286 |
been found, continuation to a new subject segment is no longer possible.
|
| 287 |
Consider again this \fBpcretest\fP example:
|
| 288 |
.sp
|
| 289 |
re> /dog(sbody)?/
|
| 290 |
data> dogsb\eP
|
| 291 |
0: dog
|
| 292 |
data> do\eP\eD
|
| 293 |
Partial match: do
|
| 294 |
data> gsb\eR\eP\eD
|
| 295 |
0: g
|
| 296 |
data> dogsbody\eD
|
| 297 |
0: dogsbody
|
| 298 |
1: dog
|
| 299 |
.sp
|
| 300 |
The first data line passes the string "dogsb" to \fBpcre_exec()\fP, setting the
|
| 301 |
PCRE_PARTIAL_SOFT option. Although the string is a partial match for
|
| 302 |
"dogsbody", the result is not PCRE_ERROR_PARTIAL, because the shorter string
|
| 303 |
"dog" is a complete match. Similarly, when the subject is presented to
|
| 304 |
\fBpcre_dfa_exec()\fP in several parts ("do" and "gsb" being the first two) the
|
| 305 |
match stops when "dog" has been found, and it is not possible to continue. On
|
| 306 |
the other hand, if "dogsbody" is presented as a single string,
|
| 307 |
\fBpcre_dfa_exec()\fP finds both matches.
|
| 308 |
.P
|
| 309 |
Because of these problems, it is probably best to use PCRE_PARTIAL_HARD when
|
| 310 |
matching multi-segment data. The example above then behaves differently:
|
| 311 |
.sp
|
| 312 |
re> /dog(sbody)?/
|
| 313 |
data> dogsb\eP\eP
|
| 314 |
Partial match: dogsb
|
| 315 |
data> do\eP\eD
|
| 316 |
Partial match: do
|
| 317 |
data> gsb\eR\eP\eP\eD
|
| 318 |
Partial match: gsb
|
| 319 |
.sp
|
| 320 |
.P
|
| 321 |
4. Patterns that contain alternatives at the top level which do not all
|
| 322 |
start with the same pattern item may not work as expected when
|
| 323 |
\fBpcre_dfa_exec()\fP is used. For example, consider this pattern:
|
| 324 |
.sp
|
| 325 |
1234|3789
|
| 326 |
.sp
|
| 327 |
If the first part of the subject is "ABC123", a partial match of the first
|
| 328 |
alternative is found at offset 3. There is no partial match for the second
|
| 329 |
alternative, because such a match does not start at the same point in the
|
| 330 |
subject string. Attempting to continue with the string "7890" does not yield a
|
| 331 |
match because only those alternatives that match at one point in the subject
|
| 332 |
are remembered. The problem arises because the start of the second alternative
|
| 333 |
matches within the first alternative. There is no problem with anchored
|
| 334 |
patterns or patterns such as:
|
| 335 |
.sp
|
| 336 |
1234|ABCD
|
| 337 |
.sp
|
| 338 |
where no string can be a partial match for both alternatives. This is not a
|
| 339 |
problem if \fPpcre_exec()\fP is used, because the entire match has to be rerun
|
| 340 |
each time:
|
| 341 |
.sp
|
| 342 |
re> /1234|3789/
|
| 343 |
data> ABC123\eP
|
| 344 |
Partial match: 123
|
| 345 |
data> 1237890
|
| 346 |
0: 3789
|
| 347 |
.sp
|
| 348 |
.
|
| 349 |
.
|
| 350 |
.SH AUTHOR
|
| 351 |
.rs
|
| 352 |
.sp
|
| 353 |
.nf
|
| 354 |
Philip Hazel
|
| 355 |
University Computing Service
|
| 356 |
Cambridge CB2 3QH, England.
|
| 357 |
.fi
|
| 358 |
.
|
| 359 |
.
|
| 360 |
.SH REVISION
|
| 361 |
.rs
|
| 362 |
.sp
|
| 363 |
.nf
|
| 364 |
Last updated: 29 September 2009
|
| 365 |
Copyright (c) 1997-2009 University of Cambridge.
|
| 366 |
.fi
|
Parent Directory
| 
Revision Log