/[pcre]/code/trunk/doc/pcrepattern.3

Diff of /code/trunk/doc/pcrepattern.3

Parent Directory | Revision Log | View Patch Patch

-revision 535 by ph10,
Thu Jun  3 19:18:24 2010 UTC
+revision 745 by ph10,
Mon Nov 14 11:41:03 2011 UTC
 Line 32 
 Starting a pattern with this sequence is
  option. This feature is not Perl-compatible. How setting UTF-8 mode affects
  pattern matching is mentioned in several places below. There is also a summary
  of UTF-8 features in the
- .\" HTML <a href="pcre.html#utf8support">
- .\" </a>
- section on UTF-8 support
- .\"
- in the main
  .\" HREF
- \fBpcre\fP
+ \fBpcreunicode\fP
  .\"
  page.
  .P
-Line 52 
 such as \ed and \ew to use Unicode prope
+Line 47 
 such as \ed and \ew to use Unicode prope
  instead of recognizing only characters with codes less than 128 via a lookup
  table.
  .P
+ If a pattern starts with (*NO_START_OPT), it has the same effect as setting the
+ PCRE_NO_START_OPTIMIZE option either at compile or matching time. There are
+ also some more of these special sequences that are concerned with the handling
+ of newlines; they are described below.
+ .P
  The remainder of this document discusses the patterns that are supported by
  PCRE when its main matching function, \fBpcre_exec()\fP, is used.
  From release 6.0, PCRE offers a second matching function,
 Line 66 
 discussed in the
  page.
  .
  .
+ .\" HTML <a name="newlines"></a>
  .SH "NEWLINE CONVENTIONS"
  .rs
  .sp
-Line 181 
 The following sections describe the use
+Line 182 
 The following sections describe the use
  .rs
  .sp
  The backslash character has several uses. Firstly, if it is followed by a
- non-alphanumeric character, it takes away any special meaning that character
+ character that is not a number or a letter, it takes away any special meaning
- may have. This use of backslash as an escape character applies both inside and
+ that character may have. This use of backslash as an escape character applies
- outside character classes.
+ both inside and outside character classes.
  .P
  For example, if you want to match a * character, you write \e* in the pattern.
  This escaping action applies whether or not the following character would
-Line 191 
 otherwise be interpreted as a metacharac
+Line 192 
 otherwise be interpreted as a metacharac
  non-alphanumeric with backslash to specify that it stands for itself. In
  particular, if you want to match a backslash, you write \e\e.
  .P
+ In UTF-8 mode, only ASCII numbers and letters have any special meaning after a
+ backslash. All other characters (in particular, those whose codepoints are
+ greater than 127) are treated as literals.
+ .P
  If a pattern is compiled with the PCRE_EXTENDED option, whitespace in the
  pattern (other than in a character class) and characters between a # outside
  a character class and the next newline are ignored. An escaping backslash can
-Line 210 
 Perl, $ and @ cause variable interpolati
+Line 215 
 Perl, $ and @ cause variable interpolati
    \eQabc\eE\e$\eQxyz\eE   abc$xyz        abc$xyz
  .sp
  The \eQ...\eE sequence is recognized both inside and outside character classes.
+ An isolated \eE that is not preceded by \eQ is ignored. If \eQ is not followed
+ by \eE later in the pattern, the literal interpretation continues to the end of
+ the pattern (that is, \eE is assumed at the end). If the isolated \eQ is inside
+ a character class, this causes an error, because the character class is not
+ terminated.
  .
  .
  .\" HTML <a name="digitsafterbackslash"></a>
-Line 223 
 but when a pattern is being prepared by
+Line 233 
 but when a pattern is being prepared by
  one of the following escape sequences than the binary character it represents:
  .sp
    \ea        alarm, that is, the BEL character (hex 07)
-   \ecx       "control-x", where x is any character
+   \ecx       "control-x", where x is any ASCII character
    \ee        escape (hex 1B)
    \ef        formfeed (hex 0C)
    \en        linefeed (hex 0A)
-Line 231 
 one of the following escape sequences th
+Line 241 
 one of the following escape sequences th
    \et        tab (hex 09)
    \eddd      character with octal code ddd, or back reference
    \exhh      character with hex code hh
-   \ex{hhh..} character with hex code hhh..
+   \ex{hhh..} character with hex code hhh.. (non-JavaScript mode)
+   \euhhhh    character with hex code hhhh (JavaScript mode only)
  .sp
  The precise effect of \ecx is as follows: if x is a lower case letter, it
  is converted to upper case. Then bit 6 of the character (hex 40) is inverted.
- Thus \ecz becomes hex 1A, but \ec{ becomes hex 3B, while \ec; becomes hex
+ Thus \ecz becomes hex 1A (z is 7A), but \ec{ becomes hex 3B ({ is 7B), while
-B.
+ \ec; becomes hex 7B (; is 3B). If the byte following \ec has a value greater
- .P
+ than 127, a compile-time error occurs. This locks out non-ASCII characters in
- After \ex, from zero to two hexadecimal digits are read (letters can be in
+ both byte mode and UTF-8 mode. (When PCRE is compiled in EBCDIC mode, all byte
- upper or lower case). Any number of hexadecimal digits may appear between \ex{
+ values are valid. A lower case letter is converted to upper case, and then the
- and }, but the value of the character code must be less than 256 in non-UTF-8
+xc0 bits are flipped.)
- mode, and less than 2**31 in UTF-8 mode. That is, the maximum value in
+ .P
- hexadecimal is 7FFFFFFF. Note that this is bigger than the largest Unicode code
+ By default, after \ex, from zero to two hexadecimal digits are read (letters
- point, which is 10FFFF.
+ can be in upper or lower case). Any number of hexadecimal digits may appear
+ between \ex{ and }, but the value of the character code must be less than 256
+ in non-UTF-8 mode, and less than 2**31 in UTF-8 mode. That is, the maximum
+ value in hexadecimal is 7FFFFFFF. Note that this is bigger than the largest
+ Unicode code point, which is 10FFFF.
  .P
  If characters other than hexadecimal digits appear between \ex{ and }, or if
  there is no terminating }, this form of escape is not recognized. Instead, the
  initial \ex will be interpreted as a basic hexadecimal escape, with no
  following digits, giving a character whose value is zero.
  .P
+ If the PCRE_JAVASCRIPT_COMPAT option is set, the interpretation of \ex is
+ as just described only when it is followed by two hexadecimal digits.
+ Otherwise, it matches a literal "x" character. In JavaScript mode, support for
+ code points greater than 256 is provided by \eu, which must be followed by
+ four hexadecimal digits; otherwise it matches a literal "u" character.
+ .P
  Characters whose value is less than 256 can be defined by either of the two
- syntaxes for \ex. There is no difference in the way they are handled. For
+ syntaxes for \ex (or by \eu in JavaScript mode). There is no difference in the
- example, \exdc is exactly the same as \ex{dc}.
+ way they are handled. For example, \exdc is exactly the same as \ex{dc} (or
+ \eu00dc in JavaScript mode).
  .P
  After \e0 up to two further octal digits are read. If there are fewer than two
  digits, just those that are present are used. Thus the sequence \e0\ex\e07
-Line 314 
 unrecognized escape sequences, they are
+Line 336 
 unrecognized escape sequences, they are
  set. Outside a character class, these sequences have different meanings.
  .
  .
+ .SS "Unsupported escape sequences"
+ .rs
+ .sp
+ In Perl, the sequences \el, \eL, \eu, and \eU are recognized by its string
+ handler and used to modify the case of following characters. By default, PCRE
+ does not support these escape sequences. However, if the PCRE_JAVASCRIPT_COMPAT
+ option is set, \eU matches a "U" character, and \eu can be used to define a
+ character by code point, as described in the previous section.
+ .
+ .
  .SS "Absolute and relative back references"
  .rs
  .sp
-Line 373 
 This is the same as
+Line 405 
 This is the same as
  .\" </a>
  the "." metacharacter
  .\"
- when PCRE_DOTALL is not set.
+ when PCRE_DOTALL is not set. Perl also uses \eN to match characters by name;
+ PCRE does not support this.
  .P
  Each pair of lower and upper case escape sequences partitions the complete set
  of characters into two disjoint sets. Any given character matches one, and only
-Line 422 
 any Unicode letter, and underscore. Note
+Line 455 
 any Unicode letter, and underscore. Note
  \eB because they are defined in terms of \ew and \eW. Matching these sequences
  is noticeably slower when PCRE_UCP is set.
  .P
- The sequences \eh, \eH, \ev, and \eV are Perl 5.10 features. In contrast to the
+ The sequences \eh, \eH, \ev, and \eV are features that were added to Perl at
- other sequences, which match only ASCII characters by default, these always
+ release 5.10. In contrast to the other sequences, which match only ASCII
- match certain high-valued codepoints in UTF-8 mode, whether or not PCRE_UCP is
+ characters by default, these always match certain high-valued codepoints in
- set. The horizontal space characters are:
+ UTF-8 mode, whether or not PCRE_UCP is set. The horizontal space characters
+ are:
  .sp
    U+0009     Horizontal tab
    U+0020     Space
-Line 463 
 The vertical space characters are:
+Line 497 
 The vertical space characters are:
  .rs
  .sp
  Outside a character class, by default, the escape sequence \eR matches any
- Unicode newline sequence. This is a Perl 5.10 feature. In non-UTF-8 mode \eR is
+ Unicode newline sequence. In non-UTF-8 mode \eR is equivalent to the following:
- equivalent to the following:
  .sp
    (?>\er\en|\en|\ex0b|\ef|\er|\ex85)
  .sp
-Line 738 
 Characters with the "mark" property are
+Line 771 
 Characters with the "mark" property are
  preceding character. None of them have codepoints less than 256, so in
  non-UTF-8 mode \eX matches any one character.
  .P
+ Note that recent versions of Perl have changed \eX to match what Unicode calls
+ an "extended grapheme cluster", which has a more complicated definition.
+ .P
  Matching characters by Unicode property is not fast, because PCRE has to search
  a structure that contains data for over fifteen thousand characters. That is
  why the traditional escape sequences such as \ed and \ew do not use Unicode
-Line 772 
 same characters as Xan, plus underscore.
+Line 808 
 same characters as Xan, plus underscore.
  .SS "Resetting the match start"
  .rs
  .sp
- The escape sequence \eK, which is a Perl 5.10 feature, causes any previously
+ The escape sequence \eK causes any previously matched characters not to be
- matched characters not to be included in the final matched sequence. For
+ included in the final matched sequence. For example, the pattern:
- example, the pattern:
  .sp
    foo\eKbar
  .sp
-Line 946 
 The handling of dot is entirely independ
+Line 981 
 The handling of dot is entirely independ
  dollar, the only relationship being that they both involve newlines. Dot has no
  special meaning in a character class.
  .P
- The escape sequence \eN always behaves as a dot does when PCRE_DOTALL is not
+ The escape sequence \eN behaves like a dot, except that it is not affected by
- set. In other words, it matches any one character except one that signifies the
+ the PCRE_DOTALL option. In other words, it matches any character except one
- end of a line.
+ that signifies the end of a line. Perl also uses \eN to match characters by
+ name; PCRE does not support this.
  .
  .
  .SH "MATCHING A SINGLE BYTE"
  .rs
  .sp
  Outside a character class, the escape sequence \eC matches any one byte, both
- in and out of UTF-8 mode. Unlike a dot, it always matches any line-ending
+ in and out of UTF-8 mode. Unlike a dot, it always matches line-ending
  characters. The feature is provided in Perl in order to match individual bytes
- in UTF-8 mode. Because it breaks up UTF-8 characters into individual bytes,
+ in UTF-8 mode, but it is unclear how it can usefully be used. Because \eC
- what remains in the string may be a malformed UTF-8 string. For this reason,
+ breaks up characters into individual bytes, matching one byte with \eC in UTF-8
- the \eC escape sequence is best avoided.
+ mode means that the rest of the string may start with a malformed UTF-8
+ character. This has undefined results, because PCRE assumes that it is dealing
+ with valid UTF-8 strings (and by default it checks this at the start of
+ processing unless the PCRE_NO_UTF8_CHECK option is used).
  .P
  PCRE does not allow \eC to appear in lookbehind assertions
  .\" HTML <a href="#lookbehind">
-Line 968 
 PCRE does not allow \eC to appear in loo
+Line 1007 
 PCRE does not allow \eC to appear in loo
  .\"
  because in UTF-8 mode this would make it impossible to calculate the length of
  the lookbehind.
+ .P
+ In general, the \eC escape sequence is best avoided in UTF-8 mode. However, one
+ way of using it that avoids the problem of malformed UTF-8 characters is to
+ use a lookahead to check the length of the next character, as in this pattern
+ (ignore white space and line breaks):
+ .sp
+   (?| (?=[\ex00-\ex7f])(\eC) |
+       (?=[\ex80-\ex{7ff}])(\eC)(\eC) |
+       (?=[\ex{800}-\ex{ffff}])(\eC)(\eC)(\eC) |
+       (?=[\ex{10000}-\ex{1fffff}])(\eC)(\eC)(\eC)(\eC))
+ .sp
+ A group that starts with (?| resets the capturing parentheses numbers in each
+ alternative (see
+ .\" HTML <a href="#dupsubpatternnumber">
+ .\" </a>
+ "Duplicate Subpattern Numbers"
+ .\"
+ below). The assertions at the start of each branch check the next UTF-8
+ character for values whose encoding uses 1, 2, 3, or 4 bytes, respectively. The
+ character's individual bytes are then captured by the appropriate number of
+ groups.
  .
  .
  .\" HTML <a name="characterclass"></a>
-Line 1043 
 characters in both cases. In UTF-8 mode,
+Line 1103 
 characters in both cases. In UTF-8 mode,
  characters with values greater than 128 only when it is compiled with Unicode
  property support.
  .P
- The character types \ed, \eD, \eh, \eH, \ep, \eP, \es, \eS, \ev, \eV, \ew, and
+ The character escape sequences \ed, \eD, \eh, \eH, \ep, \eP, \es, \eS, \ev,
- \eW may also appear in a character class, and add the characters that they
+ \eV, \ew, and \eW may appear in a character class, and add the characters that
- match to the class. For example, [\edABCDEF] matches any hexadecimal digit. A
+ they match to the class. For example, [\edABCDEF] matches any hexadecimal
- circumflex can conveniently be used with the upper case character types to
+ digit. In UTF-8 mode, the PCRE_UCP option affects the meanings of \ed, \es, \ew
+ and their upper case partners, just as it does when they appear outside a
+ character class, as described in the section entitled
+ .\" HTML <a href="#genericchartypes">
+ .\" </a>
+ "Generic character types"
+ .\"
+ above. The escape sequence \eb has a different meaning inside a character
+ class; it matches the backspace character. The sequences \eB, \eN, \eR, and \eX
+ are not special inside a character class. Like any other unrecognized escape
+ sequences, they are treated as the literal characters "B", "N", "R", and "X" by
+ default, but cause an error if the PCRE_EXTRA option is set.
+ .P
+ A circumflex can conveniently be used with the upper case character types to
  specify a more restricted set of characters than the matching lower case type.
- For example, the class [^\eW_] matches any letter or digit, but not underscore.
+ For example, the class [^\eW_] matches any letter or digit, but not underscore,
+ whereas [\ew] includes underscore. A positive character class should be read as
+ "something OR something OR ..." and a negative class as "NOT something AND NOT
+ something AND NOT ...".
  .P
  The only metacharacters that are recognized in character classes are backslash,
  hyphen (only where it can be interpreted as specifying a range), circumflex
-Line 1171 
 extracts it into the global options (and
+Line 1247 
 extracts it into the global options (and
  extracted by the \fBpcre_fullinfo()\fP function).
  .P
  An option change within a subpattern (see below for a description of
- subpatterns) affects only that part of the current pattern that follows it, so
+ subpatterns) affects only that part of the subpattern that follows it, so
  .sp
    (a(?i)b)c
  .sp
-Line 1212 
 Turning part of a pattern into a subpatt
+Line 1288 
 Turning part of a pattern into a subpatt
  .sp
    cat(aract|erpillar|)
  .sp
- matches one of the words "cat", "cataract", or "caterpillar". Without the
+ matches "cataract", "caterpillar", or "cat". Without the parentheses, it would
- parentheses, it would match "cataract", "erpillar" or an empty string.
+ match "cataract", "erpillar" or an empty string.
  .sp
 . It sets up the subpattern as a capturing subpattern. This means that, when
  the whole pattern matches, that portion of the subject string that matched the
  subpattern is passed back to the caller via the \fIovector\fP argument of
  \fBpcre_exec()\fP. Opening parentheses are counted from left to right (starting
- from 1) to obtain numbers for the capturing subpatterns.
+ from 1) to obtain numbers for the capturing subpatterns. For example, if the
- .P
+ string "the red king" is matched against the pattern
- For example, if the string "the red king" is matched against the pattern
  .sp
    the ((red|white) (king|queen))
  .sp
-Line 1270 
 at captured substring number one, whiche
+Line 1345 
 at captured substring number one, whiche
  is useful when you want to capture part, but not all, of one of a number of
  alternatives. Inside a (?| group, parentheses are numbered as usual, but the
  number is reset at the start of each branch. The numbers of any capturing
- buffers that follow the subpattern start after the highest number used in any
+ parentheses that follow the subpattern start after the highest number used in
- branch. The following example is taken from the Perl documentation.
+ any branch. The following example is taken from the Perl documentation. The
- The numbers underneath show in which buffer the captured content will be
+ numbers underneath show in which buffer the captured content will be stored.
- stored.
  .sp
    # before  ---------------branch-reset----------- after
    / ( a )  (?| x ( y ) z | (p (q) r) | (t) u (v) ) ( z ) /x
-Line 1285 
 or "defdef":
+Line 1359 
 or "defdef":
  .sp
    /(?|(abc)|(def))\e1/
  .sp
- In contrast, a recursive or "subroutine" call to a numbered subpattern always
+ In contrast, a subroutine call to a numbered subpattern always refers to the
- refers to the first one in the pattern with the given number. The following
+ first one in the pattern with the given number. The following pattern matches
- pattern matches "abcabc" or "defabc":
+ "abcabc" or "defabc":
  .sp
    /(?|(abc)|(def))(?1)/
  .sp
-Line 1400 
 items:
+Line 1474 
 items:
    the \eC escape sequence
    the \eX escape sequence (in UTF-8 mode with Unicode properties)
    the \eR escape sequence
-   an escape such as \ed that matches a single character
+   an escape such as \ed or \epL that matches a single character
    a character class
    a back reference (see next section)
-   a parenthesized subpattern (unless it is an assertion)
+   a parenthesized subpattern (including assertions)
-   a recursive or "subroutine" call to a subpattern
+   a subroutine call to a subpattern (recursive or otherwise)
  .sp
  The general repetition quantifier specifies a minimum and maximum number of
  permitted matches, by giving the two numbers in curly brackets (braces),
-Line 1442 
 subpatterns that are referenced as
+Line 1516 
 subpatterns that are referenced as
  .\" </a>
  subroutines
  .\"
- from elsewhere in the pattern. Items other than subpatterns that have a {0}
+ from elsewhere in the pattern (but see also the section entitled
- quantifier are omitted from the compiled pattern.
+ .\" HTML <a href="#subdefine">
+ .\" </a>
+ "Defining subpatterns for use by reference only"
+ .\"
+ below). Items other than subpatterns that have a {0} quantifier are omitted
+ from the compiled pattern.
  .P
  For convenience, the three most common quantifiers have single-character
  abbreviations:
-Line 1668 
 no such problem when named parentheses a
+Line 1747 
 no such problem when named parentheses a
  subpattern is possible using named parentheses (see below).
  .P
  Another way of avoiding the ambiguity inherent in the use of digits following a
- backslash is to use the \eg escape sequence, which is a feature introduced in
+ backslash is to use the \eg escape sequence. This escape must be followed by an
- Perl 5.10. This escape must be followed by an unsigned number or a negative
+ unsigned number or a negative number, optionally enclosed in braces. These
- number, optionally enclosed in braces. These examples are all identical:
+ examples are all identical:
  .sp
    (ring), \e1
    (ring), \eg1
-Line 1684 
 example:
+Line 1763 
 example:
    (abc(def)ghi)\eg{-1}
  .sp
  The sequence \eg{-1} is a reference to the most recently started capturing
- subpattern before \eg, that is, is it equivalent to \e2. Similarly, \eg{-2}
+ subpattern before \eg, that is, is it equivalent to \e2 in this example.
- would be equivalent to \e1. The use of relative references can be helpful in
+ Similarly, \eg{-2} would be equivalent to \e1. The use of relative references
- long patterns, and also in patterns that are created by joining together
+ can be helpful in long patterns, and also in patterns that are created by
- fragments that contain references within themselves.
+ joining together fragments that contain references within themselves.
  .P
  A back reference matches whatever actually matched the capturing subpattern in
  the current subject string, rather than anything matching the subpattern
-Line 1789 
 those that look ahead of the current pos
+Line 1868 
 those that look ahead of the current pos
  that look behind it. An assertion subpattern is matched in the normal way,
  except that it does not cause the current matching position to be changed.
  .P
- Assertion subpatterns are not capturing subpatterns, and may not be repeated,
+ Assertion subpatterns are not capturing subpatterns. If such an assertion
- because it makes no sense to assert the same thing several times. If any kind
+ contains capturing subpatterns within it, these are counted for the purposes of
- of assertion contains capturing subpatterns within it, these are counted for
+ numbering the capturing subpatterns in the whole pattern. However, substring
- the purposes of numbering the capturing subpatterns in the whole pattern.
+ capturing is carried out only for positive assertions, because it does not make
- However, substring capturing is carried out only for positive assertions,
+ sense for negative assertions.
- because it does not make sense for negative assertions.
+ .P
+ For compatibility with Perl, assertion subpatterns may be repeated; though
+ it makes no sense to assert the same thing several times, the side effect of
+ capturing parentheses may occasionally be useful. In practice, there only three
+ cases:
+ .sp
+ (1) If the quantifier is {0}, the assertion is never obeyed during matching.
+ However, it may contain internal capturing parenthesized groups that are called
+ from elsewhere via the
+ .\" HTML <a href="#subpatternsassubroutines">
+ .\" </a>
+ subroutine mechanism.
+ .\"
+ .sp
+ (2) If quantifier is {0,n} where n is greater than zero, it is treated as if it
+ were {0,1}. At run time, the rest of the pattern match is tried with and
+ without the assertion, the order depending on the greediness of the quantifier.
+ .sp
+ (3) If the minimum repetition is greater than zero, the quantifier is ignored.
+ The assertion is obeyed just once when encountered during matching.
  .
  .
  .SS "Lookahead assertions"
-Line 1823 
 lookbehind assertion is needed to achiev
+Line 1921 
 lookbehind assertion is needed to achiev
  If you want to force a matching failure at some point in a pattern, the most
  convenient way to do it is with (?!) because an empty string always matches, so
  an assertion that requires there not to be an empty string must always fail.
- The Perl 5.10 backtracking control verb (*FAIL) or (*F) is essentially a
+ The backtracking control verb (*FAIL) or (*F) is a synonym for (?!).
- synonym for (?!).
  .
  .
  .\" HTML <a name="lookbehind"></a>
-Line 1849 
 is permitted, but
+Line 1946 
 is permitted, but
  .sp
  causes an error at compile time. Branches that match different length strings
  are permitted only at the top level of a lookbehind assertion. This is an
- extension compared with Perl (5.8 and 5.10), which requires all branches to
+ extension compared with Perl, which requires all branches to match the same
- match the same length of string. An assertion such as
+ length of string. An assertion such as
  .sp
    (?<=ab(c|de))
  .sp
-Line 1860 
 branches:
+Line 1957 
 branches:
  .sp
    (?<=abc|abde)
  .sp
- In some cases, the Perl 5.10 escape sequence \eK
+ In some cases, the escape sequence \eK
  .\" HTML <a href="#resetmatchstart">
  .\" </a>
  (see above)
-Line 1964 
 already been matched. The two possible f
+Line 2061 
 already been matched. The two possible f
  .sp
  If the condition is satisfied, the yes-pattern is used; otherwise the
  no-pattern (if present) is used. If there are more than two alternatives in the
- subpattern, a compile-time error occurs.
+ subpattern, a compile-time error occurs. Each of the two alternatives may
+ itself contain nested subpatterns of any form, including conditional
+ subpatterns; the restriction to two alternatives applies only at the level of
+ the condition. This pattern fragment is an example where the alternatives are
+ complex:
+ .sp
+   (?(1) (A|B|C) | (D | (?(2)E|F) | E) )
+ .sp
  .P
  There are four kinds of condition: references to subpatterns, references to
  recursion, a pseudo-condition called DEFINE, and assertions.
-Line 1981 
 matched. If there is more than one captu
+Line 2085 
 matched. If there is more than one captu
  .\" </a>
  section about duplicate subpattern numbers),
  .\"
- the condition is true if any of them have been set. An alternative notation is
+ the condition is true if any of them have matched. An alternative notation is
  to precede the digits with a plus or minus sign. In this case, the subpattern
  number is relative rather than absolute. The most recently opened parentheses
- can be referenced by (?(-1), the next most recent by (?(-2), and so on. In
+ can be referenced by (?(-1), the next most recent by (?(-2), and so on. Inside
- looping constructs it can also make sense to refer to subsequent groups with
+ loops it can also make sense to refer to subsequent groups. The next
- constructs such as (?(+2).
+ parentheses to be opened can be referenced as (?(+1), and so on. (The value
+ zero in any of these forms is not used; it provokes a compile-time error.)
  .P
  Consider the following pattern, which contains non-significant white space to
  make it more readable (assume the PCRE_EXTENDED option) and to divide it into
-Line 1997 
 three parts for ease of discussion:
+Line 2102 
 three parts for ease of discussion:
  The first part matches an optional opening parenthesis, and if that
  character is present, sets it as the first captured substring. The second part
  matches one or more characters that are not parentheses. The third part is a
- conditional subpattern that tests whether the first set of parentheses matched
+ conditional subpattern that tests whether or not the first set of parentheses
- or not. If they did, that is, if subject started with an opening parenthesis,
+ matched. If they did, that is, if subject started with an opening parenthesis,
  the condition is true, and so the yes-pattern is executed and a closing
  parenthesis is required. Otherwise, since no-pattern is not present, the
  subpattern matches nothing. In other words, this pattern matches a sequence of
-Line 2054 
 The syntax for recursive patterns
+Line 2159 
 The syntax for recursive patterns
  .\"
  is described below.
  .
+ .\" HTML <a name="subdefine"></a>
  .SS "Defining subpatterns for use by reference only"
  .rs
  .sp
-Line 2061 
 If the condition is the string (DEFINE),
+Line 2167 
 If the condition is the string (DEFINE),
  name DEFINE, the condition is always false. In this case, there may be only one
  alternative in the subpattern. It is always skipped if control reaches this
  point in the pattern; the idea of DEFINE is that it can be used to define
- "subroutines" that can be referenced from elsewhere. (The use of
+ subroutines that can be referenced from elsewhere. (The use of
  .\" HTML <a href="#subpatternsassubroutines">
  .\" </a>
- "subroutines"
+ subroutines
  .\"
- is described below.) For example, a pattern to match an IPv4 address could be
+ is described below.) For example, a pattern to match an IPv4 address such as
- written like this (ignore whitespace and line breaks):
+ "192.168.23.245" could be written like this (ignore whitespace and line
+ breaks):
  .sp
    (?(DEFINE) (?<byte> 2[0-4]\ed | 25[0-5] | 1\ed\ed | [1-9]?\ed) )
    \eb (?&byte) (\e.(?&byte)){3} \eb
-Line 2102 
 dd-aaa-dd or dd-dd-dd, where aaa are let
+Line 2209 
 dd-aaa-dd or dd-dd-dd, where aaa are let
  .SH COMMENTS
  .rs
  .sp
- The sequence (?# marks the start of a comment that continues up to the next
+ There are two ways of including comments in patterns that are processed by
- closing parenthesis. Nested parentheses are not permitted. The characters
+ PCRE. In both cases, the start of the comment must not be in a character class,
- that make up a comment play no part in the pattern matching at all.
+ nor in the middle of any other sequence of related characters such as (?: or a
+ subpattern name or number. The characters that make up a comment play no part
+ in the pattern matching.
  .P
- If the PCRE_EXTENDED option is set, an unescaped # character outside a
+ The sequence (?# marks the start of a comment that continues up to the next
- character class introduces a comment that continues to immediately after the
+ closing parenthesis. Nested parentheses are not permitted. If the PCRE_EXTENDED
- next newline in the pattern.
+ option is set, an unescaped # character also introduces a comment, which in
+ this case continues to immediately after the next newline character or
+ character sequence in the pattern. Which characters are interpreted as newlines
+ is controlled by the options passed to \fBpcre_compile()\fP or by a special
+ sequence at the start of the pattern, as described in the section entitled
+ .\" HTML <a href="#newlines">
+ .\" </a>
+ "Newline conventions"
+ .\"
+ above. Note that the end of this type of comment is a literal newline sequence
+ in the pattern; escape sequences that happen to represent a newline do not
+ count. For example, consider this pattern when PCRE_EXTENDED is set, and the
+ default newline convention is in force:
+ .sp
+   abc #comment \en still comment
+ .sp
+ On encountering the # character, \fBpcre_compile()\fP skips along, looking for
+ a newline in the pattern. The sequence \en is still literal at this stage, so
+ it does not terminate the comment. Only an actual character with the code value
+x0a (the default newline) does so.
  .
  .
  .\" HTML <a name="recursion"></a>
-Line 2137 
 individual subpattern recursion. After i
+Line 2265 
 individual subpattern recursion. After i
  this kind of recursion was subsequently introduced into Perl at release 5.10.
  .P
  A special item that consists of (? followed by a number greater than zero and a
- closing parenthesis is a recursive call of the subpattern of the given number,
+ closing parenthesis is a recursive subroutine call of the subpattern of the
- provided that it occurs inside that subpattern. (If not, it is a
+ given number, provided that it occurs inside that subpattern. (If not, it is a
  .\" HTML <a href="#subpatternsassubroutines">
  .\" </a>
- "subroutine"
+ non-recursive subroutine
  .\"
  call, which is described in the next section.) The special item (?R) or (?0) is
  a recursive call of the entire regular expression.
-Line 2166 
 We have put the pattern into parentheses
+Line 2294 
 We have put the pattern into parentheses
  them instead of the whole pattern.
  .P
  In a larger pattern, keeping track of parenthesis numbers can be tricky. This
- is made easier by the use of relative references (a Perl 5.10 feature).
+ is made easier by the use of relative references. Instead of (?1) in the
- Instead of (?1) in the pattern above you can write (?-2) to refer to the second
+ pattern above you can write (?-2) to refer to the second most recently opened
- most recently opened parentheses preceding the recursion. In other words, a
+ parentheses preceding the recursion. In other words, a negative number counts
- negative number counts capturing parentheses leftwards from the point at which
+ capturing parentheses leftwards from the point at which it is encountered.
- it is encountered.
  .P
  It is also possible to refer to subsequently opened parentheses, by writing
  references such as (?+2). However, these cannot be recursive because the
  reference is not inside the parentheses that are referenced. They are always
  .\" HTML <a href="#subpatternsassubroutines">
  .\" </a>
- "subroutine"
+ non-recursive subroutine
  .\"
  calls, as described in the next section.
  .P
-Line 2214 
 documentation). If the pattern above is
+Line 2341 
 documentation). If the pattern above is
  .sp
  the value for the inner capturing parentheses (numbered 2) is "ef", which is
  the last value taken on at the top level. If a capturing subpattern is not
- matched at the top level, its final value is unset, even if it is (temporarily)
+ matched at the top level, its final captured value is unset, even if it was
- set at a deeper level.
+ (temporarily) set at a deeper level during the matching process.
  .P
  If there are more than 15 capturing parentheses in a pattern, PCRE has to
  obtain extra memory to store data during a recursion, which it does by using
-Line 2235 
 is the actual recursive call.
+Line 2362 
 is the actual recursive call.
  .
  .
  .\" HTML <a name="recursiondifference"></a>
- .SS "Recursion difference from Perl"
+ .SS "Differences in recursion processing between PCRE and Perl"
  .rs
  .sp
- In PCRE (like Python, but unlike Perl), a recursive subpattern call is always
+ Recursion processing in PCRE differs from Perl in two important ways. In PCRE
- treated as an atomic group. That is, once it has matched some of the subject
+ (like Python, but unlike Perl), a recursive subpattern call is always treated
- string, it is never re-entered, even if it contains untried alternatives and
+ as an atomic group. That is, once it has matched some of the subject string, it
- there is a subsequent matching failure. This can be illustrated by the
+ is never re-entered, even if it contains untried alternatives and there is a
- following pattern, which purports to match a palindromic string that contains
+ subsequent matching failure. This can be illustrated by the following pattern,
- an odd number of characters (for example, "a", "aba", "abcba", "abcdcba"):
+ which purports to match a palindromic string that contains an odd number of
+ characters (for example, "a", "aba", "abcba", "abcdcba"):
  .sp
    ^(.|(.)(?1)\e2)$
  .sp
-Line 2273 
 time we do have another alternative to t
+Line 2401 
 time we do have another alternative to t
  difference: in the previous case the remaining alternative is at a deeper
  recursion level, which PCRE cannot use.
  .P
- To change the pattern so that matches all palindromic strings, not just those
+ To change the pattern so that it matches all palindromic strings, not just
- with an odd number of characters, it is tempting to change the pattern to this:
+ those with an odd number of characters, it is tempting to change the pattern to
+ this:
  .sp
    ^((.)(?1)\e2|.?)$
  .sp
-Line 2303 
 For example, although "abcba" is correct
+Line 2432 
 For example, although "abcba" is correct
  PCRE finds the palindrome "aba" at the start, then fails at top level because
  the end of the string does not follow. Once again, it cannot jump back into the
  recursion to try other alternatives, so the entire match fails.
+ .P
+ The second way in which PCRE and Perl differ in their recursion processing is
+ in the handling of captured values. In Perl, when a subpattern is called
+ recursively or as a subpattern (see the next section), it has no access to any
+ values that were captured outside the recursion, whereas in PCRE these values
+ can be referenced. Consider this pattern:
+ .sp
+   ^(.)(\e1|a(?2))
+ .sp
+ In PCRE, this pattern matches "bab". The first capturing parentheses match "b",
+ then in the second group, when the back reference \e1 fails to match "b", the
+ second alternative matches "a" and then recurses. In the recursion, \e1 does
+ now match "b" and so the whole match succeeds. In Perl, the pattern fails to
+ match because inside the recursive call \e1 cannot access the externally set
+ value.
  .
  .
  .\" HTML <a name="subpatternsassubroutines"></a>
  .SH "SUBPATTERNS AS SUBROUTINES"
  .rs
  .sp
- If the syntax for a recursive subpattern reference (either by number or by
+ If the syntax for a recursive subpattern call (either by number or by
  name) is used outside the parentheses to which it refers, it operates like a
- subroutine in a programming language. The "called" subpattern may be defined
+ subroutine in a programming language. The called subpattern may be defined
  before or after the reference. A numbered reference can be absolute or
  relative, as in these examples:
  .sp
-Line 2331 
 matches "sense and sensibility" and "res
+Line 2475 
 matches "sense and sensibility" and "res
  is used, it does match "sense and responsibility" as well as the other two
  strings. Another example is given in the discussion of DEFINE above.
  .P
- Like recursive subpatterns, a subroutine call is always treated as an atomic
+ All subroutine calls, whether recursive or not, are always treated as atomic
- group. That is, once it has matched some of the subject string, it is never
+ groups. That is, once a subroutine has matched some of the subject string, it
- re-entered, even if it contains untried alternatives and there is a subsequent
+ is never re-entered, even if it contains untried alternatives and there is a
- matching failure. Any capturing parentheses that are set during the subroutine
+ subsequent matching failure. Any capturing parentheses that are set during the
- call revert to their previous values afterwards.
+ subroutine call revert to their previous values afterwards.
  .P
- When a subpattern is used as a subroutine, processing options such as
+ Processing options such as case-independence are fixed when a subpattern is
- case-independence are fixed when the subpattern is defined. They cannot be
+ defined, so if it is used as a subroutine, such options cannot be changed for
- changed for different calls. For example, consider this pattern:
+ different calls. For example, consider this pattern:
  .sp
    (abc)(?i:(?-1))
  .sp
-Line 2420 
 a backtracking algorithm. With the excep
+Line 2564 
 a backtracking algorithm. With the excep
  failing negative assertion, they cause an error if encountered by
  \fBpcre_dfa_exec()\fP.
  .P
- If any of these verbs are used in an assertion or subroutine subpattern
+ If any of these verbs are used in an assertion or in a subpattern that is
- (including recursive subpatterns), their effect is confined to that subpattern;
+ called as a subroutine (whether or not recursively), their effect is confined
- it does not extend to the surrounding pattern. Note that such subpatterns are
+ to that subpattern; it does not extend to the surrounding pattern, with one
- processed as anchored at the point where they are tested.
+ exception: a *MARK that is encountered in a positive assertion \fIis\fP passed
+ back (compare capturing parentheses in assertions). Note that such subpatterns
+ are processed as anchored at the point where they are tested. Note also that
+ Perl's treatment of subroutines is different in some cases.
  .P
  The new verbs make use of what was previously invalid syntax: an opening
  parenthesis followed by an asterisk. They are generally of the form
  (*VERB) or (*VERB:NAME). Some may take either form, with differing behaviour,
- depending on whether or not an argument is present. An name is a sequence of
+ depending on whether or not an argument is present. A name is any sequence of
- letters, digits, and underscores. If the name is empty, that is, if the closing
+ characters that does not include a closing parenthesis. If the name is empty,
- parenthesis immediately follows the colon, the effect is as if the colon were
+ that is, if the closing parenthesis immediately follows the colon, the effect
- not there. Any number of these verbs may occur in a pattern.
+ is as if the colon were not there. Any number of these verbs may occur in a
+ pattern.
  .P
  PCRE contains some optimizations that are used to speed up matching by running
  some checks at the start of each match attempt. For example, it may know the
-Line 2439 
 minimum length of matching subject, or t
+Line 2587 
 minimum length of matching subject, or t
  present. When one of these optimizations suppresses the running of a match, any
  included backtracking verbs will not, of course, be processed. You can suppress
  the start-of-match optimizations by setting the PCRE_NO_START_OPTIMIZE option
- when calling \fBpcre_exec()\fP.
+ when calling \fBpcre_compile()\fP or \fBpcre_exec()\fP, or by starting the
+ pattern with (*NO_START_OPT).
  .
  .
  .SS "Verbs that act immediately"
-Line 2451 
 followed by a name.
+Line 2600 
 followed by a name.
     (*ACCEPT)
  .sp
  This verb causes the match to end successfully, skipping the remainder of the
- pattern. When inside a recursion, only the innermost pattern is ended
+ pattern. However, when it is inside a subpattern that is called as a
- immediately. If (*ACCEPT) is inside capturing parentheses, the data so far is
+ subroutine, only that subpattern is ended successfully. Matching then continues
- captured. (This feature was added to PCRE at release 8.00.) For example:
+ at the outer level. If (*ACCEPT) is inside capturing parentheses, the data so
+ far is captured. For example:
  .sp
    A((?:A|B(*ACCEPT)|C)D)
  .sp
-Line 2462 
 the outer parentheses.
+Line 2612 
 the outer parentheses.
  .sp
    (*FAIL) or (*F)
  .sp
- This verb causes the match to fail, forcing backtracking to occur. It is
+ This verb causes a matching failure, forcing backtracking to occur. It is
  equivalent to (?!) but easier to read. The Perl documentation notes that it is
  probably useful only when combined with (?{}) or (??{}). Those are, of course,
  Perl features that are not present in PCRE. The nearest equivalent is the
-Line 2513 
 indicates which of the two alternatives
+Line 2663 
 indicates which of the two alternatives
  of obtaining this information than putting each alternative in its own
  capturing parentheses.
  .P
+ If (*MARK) is encountered in a positive assertion, its name is recorded and
+ passed back if it is the last-encountered. This does not happen for negative
+ assertions.
+ .P
  A name may also be returned after a failed match if the final path through the
  pattern involves (*MARK). However, unless (*MARK) used in conjunction with
  (*COMMIT), this is unlikely to happen for an unanchored pattern because, as the
-Line 2625 
 following pattern fails to match, the pr
+Line 2779 
 following pattern fails to match, the pr
  searched for the most recent (*MARK) that has the same name. If one is found,
  the "bumpalong" advance is to the subject position that corresponds to that
  (*MARK) instead of to where (*SKIP) was encountered. If no (*MARK) with a
- matching name is found, normal "bumpalong" of one character happens (the
+ matching name is found, normal "bumpalong" of one character happens (that is,
- (*SKIP) is ignored).
+ the (*SKIP) is ignored).
  .sp
    (*THEN) or (*THEN:NAME)
  .sp
- This verb causes a skip to the next alternation if the rest of the pattern does
+ This verb causes a skip to the next innermost alternative if the rest of the
- not match. That is, it cancels pending backtracking, but only within the
+ pattern does not match. That is, it cancels pending backtracking, but only
- current alternation. Its name comes from the observation that it can be used
+ within the current alternative. Its name comes from the observation that it can
- for a pattern-based if-then-else block:
+ be used for a pattern-based if-then-else block:
  .sp
    ( COND1 (*THEN) FOO | COND2 (*THEN) BAR | COND3 (*THEN) BAZ ) ...
  .sp
  If the COND1 pattern matches, FOO is tried (and possibly further items after
- the end of the group if FOO succeeds); on failure the matcher skips to the
+ the end of the group if FOO succeeds); on failure, the matcher skips to the
  second alternative and tries COND2, without backtracking into COND1. The
  behaviour of (*THEN:NAME) is exactly the same as (*MARK:NAME)(*THEN) if the
- overall match fails. If (*THEN) is not directly inside an alternation, it acts
+ overall match fails. If (*THEN) is not inside an alternation, it acts like
- like (*PRUNE).
+ (*PRUNE).
+ .P
+ Note that a subpattern that does not contain a | character is just a part of
+ the enclosing alternative; it is not a nested alternation with only one
+ alternative. The effect of (*THEN) extends beyond such a subpattern to the
+ enclosing alternative. Consider this pattern, where A, B, etc. are complex
+ pattern fragments that do not contain any | characters at this level:
+ .sp
+   A (B(*THEN)C) | D
+ .sp
+ If A and B are matched, but there is a failure in C, matching does not
+ backtrack into A; instead it moves to the next alternative, that is, D.
+ However, if the subpattern containing (*THEN) is given an alternative, it
+ behaves differently:
+ .sp
+   A (B(*THEN)C | (*FAIL)) | D
+ .sp
+ The effect of (*THEN) is now confined to the inner subpattern. After a failure
+ in C, matching moves to (*FAIL), which causes the whole subpattern to fail
+ because there are no more alternatives to try. In this case, matching does now
+ backtrack into A.
+ .P
+ Note also that a conditional subpattern is not considered as having two
+ alternatives, because only one is ever used. In other words, the | character in
+ a conditional subpattern has a different meaning. Ignoring white space,
+ consider:
+ .sp
+   ^.*? (?(?=a) a | b(*THEN)c )
+ .sp
+ If the subject is "ba", this pattern does not match. Because .*? is ungreedy,
+ it initially matches zero characters. The condition (?=a) then fails, the
+ character "b" is matched, but "c" is not. At this point, matching does not
+ backtrack to .*? as might perhaps be expected from the presence of the |
+ character. The conditional subpattern is part of the single alternative that
+ comprises the whole pattern, and so the match fails. (If there was a backtrack
+ into .*?, allowing it to match "b", the match would succeed.)
+ .P
+ The verbs just described provide four different "strengths" of control when
+ subsequent matching fails. (*THEN) is the weakest, carrying on the match at the
+ next alternative. (*PRUNE) comes next, failing the match at the current
+ starting position, but allowing an advance to the next character (for an
+ unanchored pattern). (*SKIP) is similar, except that the advance may be more
+ than one character. (*COMMIT) is the strongest, causing the entire match to
+ fail.
+ .P
+ If more than one such verb is present in a pattern, the "strongest" one wins.
+ For example, consider this pattern, where A, B, etc. are complex pattern
+ fragments:
+ .sp
+   (A(*COMMIT)B(*THEN)C|D)
+ .sp
+ Once A has matched, PCRE is committed to this match, at the current starting
+ position. If subsequently B matches, but C does not, the normal (*THEN) action
+ of trying the next alternative (that is, D) does not happen because (*COMMIT)
+ overrides.
  .
  .
  .SH "SEE ALSO"
-Line 2666 
 Cambridge CB2 3QH, England.
+Line 2874 
 Cambridge CB2 3QH, England.
  .rs
  .sp
  .nf
- Last updated: 18 May 2010
+ Last updated: 14 November 2011
- Copyright (c) 1997-2010 University of Cambridge.
+ Copyright (c) 1997-2011 University of Cambridge.
  .fi

 Legend:



Removed from v.535
 


changed lines


 
Added in v.745
 Legend:



Removed from v.535
 


changed lines


 
Added in v.745
-Removed from v.535
+Added in v.745

webmaster@exim.org	ViewVC Help
Powered by ViewVC 1.1.12