/[pcre]/code/trunk/doc/pcre.txt
ViewVC logotype

Contents of /code/trunk/doc/pcre.txt

Parent Directory Parent Directory | Revision Log Revision Log


Revision 83 - (hide annotations) (download)
Sat Feb 24 21:41:06 2007 UTC (6 years, 2 months ago) by nigel
File MIME type: text/plain
File size: 224275 byte(s)
Load pcre-6.3 into code/trunk.

1 nigel 75 -----------------------------------------------------------------------------
2 nigel 63 This file contains a concatenation of the PCRE man pages, converted to plain
3     text format for ease of searching with a text editor, or for use on systems
4     that do not have a man page processor. The small individual files that give
5     synopses of each function in the library have not been included. There are
6     separate text files for the pcregrep and pcretest commands.
7     -----------------------------------------------------------------------------
8    
9 nigel 41
10 nigel 79 PCRE(3) PCRE(3)
11 nigel 41
12 nigel 79
13 nigel 73 NAME
14     PCRE - Perl-compatible regular expressions
15    
16 nigel 77
17 nigel 75 INTRODUCTION
18 nigel 41
19 nigel 73 The PCRE library is a set of functions that implement regular expres-
20     sion pattern matching using the same syntax and semantics as Perl, with
21     just a few differences. The current implementation of PCRE (release
22 nigel 77 6.x) corresponds approximately with Perl 5.8, including support for
23 nigel 75 UTF-8 encoded strings and Unicode general category properties. However,
24     this support has to be explicitly enabled; it is not the default.
25 nigel 63
26 nigel 77 In addition to the Perl-compatible matching function, PCRE also con-
27     tains an alternative matching function that matches the same compiled
28     patterns in a different way. In certain circumstances, the alternative
29     function has some advantages. For a discussion of the two matching
30     algorithms, see the pcrematching page.
31    
32 nigel 75 PCRE is written in C and released as a C library. A number of people
33 nigel 77 have written wrappers and interfaces of various kinds. In particular,
34     Google Inc. have provided a comprehensive C++ wrapper. This is now
35     included as part of the PCRE distribution. The pcrecpp page has details
36     of this interface. Other people's contributions can be found in the
37     Contrib directory at the primary FTP site, which is:
38 nigel 63
39 nigel 73 ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre
40 nigel 63
41 nigel 73 Details of exactly which Perl regular expression features are and are
42     not supported by PCRE are given in separate documents. See the pcrepat-
43     tern and pcrecompat pages.
44 nigel 63
45 nigel 73 Some features of PCRE can be included, excluded, or changed when the
46     library is built. The pcre_config() function makes it possible for a
47 nigel 75 client to discover which features are available. The features them-
48     selves are described in the pcrebuild page. Documentation about build-
49     ing PCRE for various operating systems can be found in the README file
50     in the source distribution.
51 nigel 63
52 nigel 77 The library contains a number of undocumented internal functions and
53     data tables that are used by more than one of the exported external
54     functions, but which are not intended for use by external callers.
55     Their names all begin with "_pcre_", which hopefully will not provoke
56 nigel 83 any name clashes. In some environments, it is possible to control which
57     external symbols are exported when a shared library is built, and in
58     these cases the undocumented symbols are not exported.
59 nigel 63
60 nigel 77
61 nigel 63 USER DOCUMENTATION
62    
63 nigel 75 The user documentation for PCRE comprises a number of different sec-
64     tions. In the "man" format, each of these is a separate "man page". In
65     the HTML format, each is a separate page, linked from the index page.
66     In the plain text format, all the sections are concatenated, for ease
67     of searching. The sections are as follows:
68 nigel 63
69 nigel 73 pcre this document
70 nigel 77 pcreapi details of PCRE's native C API
71 nigel 73 pcrebuild options for building PCRE
72     pcrecallout details of the callout feature
73     pcrecompat discussion of Perl compatibility
74 nigel 77 pcrecpp details of the C++ wrapper
75 nigel 73 pcregrep description of the pcregrep command
76 nigel 77 pcrematching discussion of the two matching algorithms
77 nigel 75 pcrepartial details of the partial matching facility
78 nigel 73 pcrepattern syntax and semantics of supported
79     regular expressions
80     pcreperform discussion of performance issues
81 nigel 77 pcreposix the POSIX-compatible C API
82 nigel 75 pcreprecompile details of saving and re-using precompiled patterns
83 nigel 73 pcresample discussion of the sample program
84 nigel 75 pcretest description of the pcretest testing command
85 nigel 63
86 nigel 73 In addition, in the "man" and HTML formats, there is a short page for
87 nigel 77 each C library function, listing its arguments and results.
88 nigel 63
89    
90     LIMITATIONS
91    
92 nigel 73 There are some size limitations in PCRE but it is hoped that they will
93     never in practice be relevant.
94 nigel 63
95 nigel 73 The maximum length of a compiled pattern is 65539 (sic) bytes if PCRE
96     is compiled with the default internal linkage size of 2. If you want to
97     process regular expressions that are truly enormous, you can compile
98     PCRE with an internal linkage size of 3 or 4 (see the README file in
99     the source distribution and the pcrebuild documentation for details).
100 nigel 75 In these cases the limit is substantially larger. However, the speed
101 nigel 73 of execution will be slower.
102 nigel 63
103 nigel 73 All values in repeating quantifiers must be less than 65536. The maxi-
104     mum number of capturing subpatterns is 65535.
105 nigel 63
106 nigel 73 There is no limit to the number of non-capturing subpatterns, but the
107     maximum depth of nesting of all kinds of parenthesized subpattern,
108     including capturing subpatterns, assertions, and other types of subpat-
109     tern, is 200.
110 nigel 63
111 nigel 73 The maximum length of a subject string is the largest positive number
112 nigel 77 that an integer variable can hold. However, when using the traditional
113     matching function, PCRE uses recursion to handle subpatterns and indef-
114     inite repetition. This means that the available stack space may limit
115     the size of a subject string that can be processed by certain patterns.
116 nigel 63
117    
118 nigel 75 UTF-8 AND UNICODE PROPERTY SUPPORT
119 nigel 63
120 nigel 77 From release 3.3, PCRE has had some support for character strings
121     encoded in the UTF-8 format. For release 4.0 this was greatly extended
122     to cover most common requirements, and in release 5.0 additional sup-
123 nigel 75 port for Unicode general category properties was added.
124 nigel 63
125 nigel 77 In order process UTF-8 strings, you must build PCRE to include UTF-8
126     support in the code, and, in addition, you must call pcre_compile()
127     with the PCRE_UTF8 option flag. When you do this, both the pattern and
128     any subject strings that are matched against it are treated as UTF-8
129 nigel 73 strings instead of just strings of bytes.
130 nigel 63
131 nigel 77 If you compile PCRE with UTF-8 support, but do not use it at run time,
132     the library will be a bit bigger, but the additional run time overhead
133     is limited to testing the PCRE_UTF8 flag in several places, so should
134 nigel 73 not be very large.
135 nigel 63
136 nigel 75 If PCRE is built with Unicode character property support (which implies
137 nigel 77 UTF-8 support), the escape sequences \p{..}, \P{..}, and \X are sup-
138 nigel 75 ported. The available properties that can be tested are limited to the
139 nigel 77 general category properties such as Lu for an upper case letter or Nd
140     for a decimal number. A full list is given in the pcrepattern documen-
141 nigel 75 tation. The PCRE library is increased in size by about 90K when Unicode
142     property support is included.
143    
144 nigel 73 The following comments apply when PCRE is running in UTF-8 mode:
145 nigel 63
146 nigel 77 1. When you set the PCRE_UTF8 flag, the strings passed as patterns and
147     subjects are checked for validity on entry to the relevant functions.
148 nigel 73 If an invalid UTF-8 string is passed, an error return is given. In some
149 nigel 77 situations, you may already know that your strings are valid, and
150 nigel 73 therefore want to skip these checks in order to improve performance. If
151 nigel 77 you set the PCRE_NO_UTF8_CHECK flag at compile time or at run time,
152     PCRE assumes that the pattern or subject it is given (respectively)
153     contains only valid UTF-8 codes. In this case, it does not diagnose an
154     invalid UTF-8 string. If you pass an invalid UTF-8 string to PCRE when
155     PCRE_NO_UTF8_CHECK is set, the results are undefined. Your program may
156 nigel 73 crash.
157 nigel 63
158 nigel 73 2. In a pattern, the escape sequence \x{...}, where the contents of the
159 nigel 77 braces is a string of hexadecimal digits, is interpreted as a UTF-8
160     character whose code number is the given hexadecimal number, for exam-
161     ple: \x{1234}. If a non-hexadecimal digit appears between the braces,
162 nigel 73 the item is not recognized. This escape sequence can be used either as
163     a literal, or within a character class.
164 nigel 63
165 nigel 77 3. The original hexadecimal escape sequence, \xhh, matches a two-byte
166 nigel 73 UTF-8 character if the value is greater than 127.
167 nigel 63
168 nigel 77 4. Repeat quantifiers apply to complete UTF-8 characters, not to indi-
169 nigel 73 vidual bytes, for example: \x{100}{3}.
170 nigel 63
171 nigel 77 5. The dot metacharacter matches one UTF-8 character instead of a sin-
172 nigel 75 gle byte.
173 nigel 63
174 nigel 77 6. The escape sequence \C can be used to match a single byte in UTF-8
175     mode, but its use can lead to some strange effects. This facility is
176     not available in the alternative matching function, pcre_dfa_exec().
177 nigel 63
178 nigel 75 7. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly
179     test characters of any code value, but the characters that PCRE recog-
180     nizes as digits, spaces, or word characters remain the same set as
181     before, all with values less than 256. This remains true even when PCRE
182     includes Unicode property support, because to do otherwise would slow
183     down PCRE in many common cases. If you really want to test for a wider
184     sense of, say, "digit", you must use Unicode property tests such as
185     \p{Nd}.
186 nigel 63
187 nigel 75 8. Similarly, characters that match the POSIX named character classes
188     are all low-valued characters.
189 nigel 63
190 nigel 75 9. Case-insensitive matching applies only to characters whose values
191     are less than 128, unless PCRE is built with Unicode property support.
192     Even when Unicode property support is available, PCRE still uses its
193     own character tables when checking the case of low-valued characters,
194     so as not to degrade performance. The Unicode property information is
195     used only for characters with higher values.
196 nigel 63
197    
198     AUTHOR
199    
200 nigel 77 Philip Hazel
201 nigel 73 University Computing Service,
202     Cambridge CB2 3QG, England.
203 nigel 63
204 nigel 77 Putting an actual email address here seems to have been a spam magnet,
205     so I've taken it away. If you want to email me, use my initial and sur-
206     name, separated by a dot, at the domain ucs.cam.ac.uk.
207    
208     Last updated: 07 March 2005
209     Copyright (c) 1997-2005 University of Cambridge.
210 nigel 79 ------------------------------------------------------------------------------
211 nigel 63
212    
213 nigel 79 PCREBUILD(3) PCREBUILD(3)
214 nigel 63
215 nigel 79
216 nigel 73 NAME
217     PCRE - Perl-compatible regular expressions
218    
219 nigel 77
220 nigel 63 PCRE BUILD-TIME OPTIONS
221    
222 nigel 73 This document describes the optional features of PCRE that can be
223     selected when the library is compiled. They are all selected, or dese-
224 nigel 75 lected, by providing options to the configure script that is run before
225     the make command. The complete list of options for configure (which
226     includes the standard ones such as the selection of the installation
227     directory) can be obtained by running
228 nigel 63
229 nigel 73 ./configure --help
230 nigel 63
231 nigel 73 The following sections describe certain options whose names begin with
232     --enable or --disable. These settings specify changes to the defaults
233     for the configure command. Because of the way that configure works,
234     --enable and --disable always come in pairs, so the complementary
235     option always exists as well, but as it specifies the default, it is
236     not described.
237 nigel 63
238    
239 nigel 83 C++ SUPPORT
240    
241     By default, the configure script will search for a C++ compiler and C++
242     header files. If it finds them, it automatically builds the C++ wrapper
243     library for PCRE. You can disable this by adding
244    
245     --disable-cpp
246    
247     to the configure command.
248    
249    
250 nigel 63 UTF-8 SUPPORT
251    
252 nigel 73 To build PCRE with support for UTF-8 character strings, add
253 nigel 63
254 nigel 73 --enable-utf8
255 nigel 63
256 nigel 73 to the configure command. Of itself, this does not make PCRE treat
257     strings as UTF-8. As well as compiling PCRE with this option, you also
258     have have to set the PCRE_UTF8 option when you call the pcre_compile()
259     function.
260 nigel 63
261    
262 nigel 75 UNICODE CHARACTER PROPERTY SUPPORT
263    
264     UTF-8 support allows PCRE to process character values greater than 255
265     in the strings that it handles. On its own, however, it does not pro-
266     vide any facilities for accessing the properties of such characters. If
267     you want to be able to use the pattern escapes \P, \p, and \X, which
268     refer to Unicode character properties, you must add
269    
270     --enable-unicode-properties
271    
272     to the configure command. This implies UTF-8 support, even if you have
273     not explicitly requested it.
274    
275     Including Unicode property support adds around 90K of tables to the
276     PCRE library, approximately doubling its size. Only the general cate-
277     gory properties such as Lu and Nd are supported. Details are given in
278     the pcrepattern documentation.
279    
280    
281 nigel 63 CODE VALUE OF NEWLINE
282    
283 nigel 73 By default, PCRE treats character 10 (linefeed) as the newline charac-
284     ter. This is the normal newline character on Unix-like systems. You can
285     compile PCRE to use character 13 (carriage return) instead by adding
286 nigel 63
287 nigel 73 --enable-newline-is-cr
288 nigel 63
289 nigel 73 to the configure command. For completeness there is also a --enable-
290     newline-is-lf option, which explicitly specifies linefeed as the new-
291     line character.
292 nigel 63
293    
294     BUILDING SHARED AND STATIC LIBRARIES
295    
296 nigel 73 The PCRE building process uses libtool to build both shared and static
297     Unix libraries by default. You can suppress one of these by adding one
298     of
299 nigel 63
300 nigel 73 --disable-shared
301     --disable-static
302 nigel 63
303 nigel 73 to the configure command, as required.
304 nigel 63
305    
306     POSIX MALLOC USAGE
307    
308 nigel 75 When PCRE is called through the POSIX interface (see the pcreposix doc-
309     umentation), additional working storage is required for holding the
310     pointers to capturing substrings, because PCRE requires three integers
311 nigel 73 per substring, whereas the POSIX interface provides only two. If the
312     number of expected substrings is small, the wrapper function uses space
313     on the stack, because this is faster than using malloc() for each call.
314     The default threshold above which the stack is no longer used is 10; it
315     can be changed by adding a setting such as
316 nigel 63
317 nigel 73 --with-posix-malloc-threshold=20
318 nigel 63
319 nigel 73 to the configure command.
320 nigel 63
321    
322     LIMITING PCRE RESOURCE USAGE
323    
324 nigel 75 Internally, PCRE has a function called match(), which it calls repeat-
325 nigel 77 edly (possibly recursively) when matching a pattern with the
326     pcre_exec() function. By controlling the maximum number of times this
327     function may be called during a single matching operation, a limit can
328     be placed on the resources used by a single call to pcre_exec(). The
329     limit can be changed at run time, as described in the pcreapi documen-
330     tation. The default is 10 million, but this can be changed by adding a
331     setting such as
332 nigel 63
333 nigel 73 --with-match-limit=500000
334 nigel 63
335 nigel 77 to the configure command. This setting has no effect on the
336     pcre_dfa_exec() matching function.
337 nigel 63
338    
339     HANDLING VERY LARGE PATTERNS
340    
341 nigel 73 Within a compiled pattern, offset values are used to point from one
342     part to another (for example, from an opening parenthesis to an alter-
343 nigel 75 nation metacharacter). By default, two-byte values are used for these
344 nigel 73 offsets, leading to a maximum size for a compiled pattern of around
345     64K. This is sufficient to handle all but the most gigantic patterns.
346     Nevertheless, some people do want to process enormous patterns, so it
347     is possible to compile PCRE to use three-byte or four-byte offsets by
348     adding a setting such as
349 nigel 63
350 nigel 73 --with-link-size=3
351 nigel 63
352 nigel 73 to the configure command. The value given must be 2, 3, or 4. Using
353     longer offsets slows down the operation of PCRE because it has to load
354     additional bytes when handling them.
355 nigel 63
356 nigel 73 If you build PCRE with an increased link size, test 2 (and test 5 if
357     you are using UTF-8) will fail. Part of the output of these tests is a
358     representation of the compiled pattern, and this changes with the link
359     size.
360 nigel 63
361 nigel 73
362     AVOIDING EXCESSIVE STACK USAGE
363    
364 nigel 77 When matching with the pcre_exec() function, PCRE implements backtrack-
365     ing by making recursive calls to an internal function called match().
366     In environments where the size of the stack is limited, this can se-
367     verely limit PCRE's operation. (The Unix environment does not usually
368     suffer from this problem.) An alternative approach that uses memory
369     from the heap to remember data, instead of using recursive function
370     calls, has been implemented to work round this problem. If you want to
371     build a version of PCRE that works this way, add
372 nigel 73
373     --disable-stack-for-recursion
374    
375     to the configure command. With this configuration, PCRE will use the
376 nigel 75 pcre_stack_malloc and pcre_stack_free variables to call memory manage-
377     ment functions. Separate functions are provided because the usage is
378     very predictable: the block sizes requested are always the same, and
379 nigel 73 the blocks are always freed in reverse order. A calling program might
380     be able to implement optimized functions that perform better than the
381     standard malloc() and free() functions. PCRE runs noticeably more
382 nigel 77 slowly when built in this way. This option affects only the pcre_exec()
383     function; it is not relevant for the the pcre_dfa_exec() function.
384 nigel 73
385    
386     USING EBCDIC CODE
387    
388 nigel 77 PCRE assumes by default that it will run in an environment where the
389     character code is ASCII (or Unicode, which is a superset of ASCII).
390     PCRE can, however, be compiled to run in an EBCDIC environment by
391 nigel 75 adding
392 nigel 73
393     --enable-ebcdic
394    
395     to the configure command.
396    
397 nigel 83 Last updated: 15 August 2005
398 nigel 77 Copyright (c) 1997-2005 University of Cambridge.
399 nigel 79 ------------------------------------------------------------------------------
400 nigel 63
401    
402 nigel 79 PCREMATCHING(3) PCREMATCHING(3)
403 nigel 63
404 nigel 79
405 nigel 77 NAME
406     PCRE - Perl-compatible regular expressions
407 nigel 73
408 nigel 77
409     PCRE MATCHING ALGORITHMS
410    
411     This document describes the two different algorithms that are available
412     in PCRE for matching a compiled regular expression against a given sub-
413     ject string. The "standard" algorithm is the one provided by the
414     pcre_exec() function. This works in the same was as Perl's matching
415     function, and provides a Perl-compatible matching operation.
416    
417     An alternative algorithm is provided by the pcre_dfa_exec() function;
418     this operates in a different way, and is not Perl-compatible. It has
419     advantages and disadvantages compared with the standard algorithm, and
420     these are described below.
421    
422     When there is only one possible way in which a given subject string can
423     match a pattern, the two algorithms give the same answer. A difference
424     arises, however, when there are multiple possibilities. For example, if
425     the pattern
426    
427     ^<.*>
428    
429     is matched against the string
430    
431     <something> <something else> <something further>
432    
433     there are three possible answers. The standard algorithm finds only one
434     of them, whereas the DFA algorithm finds all three.
435    
436    
437     REGULAR EXPRESSIONS AS TREES
438    
439     The set of strings that are matched by a regular expression can be rep-
440     resented as a tree structure. An unlimited repetition in the pattern
441     makes the tree of infinite size, but it is still a tree. Matching the
442     pattern to a given subject string (from a given starting point) can be
443     thought of as a search of the tree. There are two standard ways to
444     search a tree: depth-first and breadth-first, and these correspond to
445     the two matching algorithms provided by PCRE.
446    
447    
448     THE STANDARD MATCHING ALGORITHM
449    
450     In the terminology of Jeffrey Friedl's book Mastering Regular Expres-
451     sions, the standard algorithm is an "NFA algorithm". It conducts a
452     depth-first search of the pattern tree. That is, it proceeds along a
453     single path through the tree, checking that the subject matches what is
454     required. When there is a mismatch, the algorithm tries any alterna-
455     tives at the current point, and if they all fail, it backs up to the
456     previous branch point in the tree, and tries the next alternative
457     branch at that level. This often involves backing up (moving to the
458     left) in the subject string as well. The order in which repetition
459     branches are tried is controlled by the greedy or ungreedy nature of
460     the quantifier.
461    
462     If a leaf node is reached, a matching string has been found, and at
463     that point the algorithm stops. Thus, if there is more than one possi-
464     ble match, this algorithm returns the first one that it finds. Whether
465     this is the shortest, the longest, or some intermediate length depends
466     on the way the greedy and ungreedy repetition quantifiers are specified
467     in the pattern.
468    
469     Because it ends up with a single path through the tree, it is rela-
470     tively straightforward for this algorithm to keep track of the sub-
471     strings that are matched by portions of the pattern in parentheses.
472     This provides support for capturing parentheses and back references.
473    
474    
475     THE DFA MATCHING ALGORITHM
476    
477     DFA stands for "deterministic finite automaton", but you do not need to
478     understand the origins of that name. This algorithm conducts a breadth-
479     first search of the tree. Starting from the first matching point in the
480     subject, it scans the subject string from left to right, once, charac-
481     ter by character, and as it does this, it remembers all the paths
482     through the tree that represent valid matches.
483    
484     The scan continues until either the end of the subject is reached, or
485     there are no more unterminated paths. At this point, terminated paths
486     represent the different matching possibilities (if there are none, the
487     match has failed). Thus, if there is more than one possible match,
488     this algorithm finds all of them, and in particular, it finds the long-
489     est. In PCRE, there is an option to stop the algorithm after the first
490     match (which is necessarily the shortest) has been found.
491    
492     Note that all the matches that are found start at the same point in the
493     subject. If the pattern
494    
495     cat(er(pillar)?)
496    
497     is matched against the string "the caterpillar catchment", the result
498     will be the three strings "cat", "cater", and "caterpillar" that start
499     at the fourth character of the subject. The algorithm does not automat-
500     ically move on to find matches that start at later positions.
501    
502     There are a number of features of PCRE regular expressions that are not
503     supported by the DFA matching algorithm. They are as follows:
504    
505     1. Because the algorithm finds all possible matches, the greedy or
506     ungreedy nature of repetition quantifiers is not relevant. Greedy and
507     ungreedy quantifiers are treated in exactly the same way.
508    
509     2. When dealing with multiple paths through the tree simultaneously, it
510     is not straightforward to keep track of captured substrings for the
511     different matching possibilities, and PCRE's implementation of this
512     algorithm does not attempt to do this. This means that no captured sub-
513     strings are available.
514    
515     3. Because no substrings are captured, back references within the pat-
516     tern are not supported, and cause errors if encountered.
517    
518     4. For the same reason, conditional expressions that use a backrefer-
519     ence as the condition are not supported.
520    
521     5. Callouts are supported, but the value of the capture_top field is
522     always 1, and the value of the capture_last field is always -1.
523    
524     6. The \C escape sequence, which (in the standard algorithm) matches a
525     single byte, even in UTF-8 mode, is not supported because the DFA algo-
526     rithm moves through the subject string one character at a time, for all
527     active paths through the tree.
528    
529    
530     ADVANTAGES OF THE DFA ALGORITHM
531    
532     Using the DFA matching algorithm provides the following advantages:
533    
534     1. All possible matches (at a single point in the subject) are automat-
535     ically found, and in particular, the longest match is found. To find
536     more than one match using the standard algorithm, you have to do kludgy
537     things with callouts.
538    
539     2. There is much better support for partial matching. The restrictions
540     on the content of the pattern that apply when using the standard algo-
541     rithm for partial matching do not apply to the DFA algorithm. For non-
542     anchored patterns, the starting position of a partial match is avail-
543     able.
544    
545     3. Because the DFA algorithm scans the subject string just once, and
546     never needs to backtrack, it is possible to pass very long subject
547     strings to the matching function in several pieces, checking for par-
548     tial matching each time.
549    
550    
551     DISADVANTAGES OF THE DFA ALGORITHM
552    
553     The DFA algorithm suffers from a number of disadvantages:
554    
555     1. It is substantially slower than the standard algorithm. This is
556     partly because it has to search for all possible matches, but is also
557     because it is less susceptible to optimization.
558    
559     2. Capturing parentheses and back references are not supported.
560    
561     3. The "atomic group" feature of PCRE regular expressions is supported,
562     but does not provide the advantage that it does for the standard algo-
563     rithm.
564    
565     Last updated: 28 February 2005
566     Copyright (c) 1997-2005 University of Cambridge.
567 nigel 79 ------------------------------------------------------------------------------
568 nigel 77
569    
570 nigel 79 PCREAPI(3) PCREAPI(3)
571 nigel 77
572 nigel 79
573 nigel 73 NAME
574     PCRE - Perl-compatible regular expressions
575    
576 nigel 77
577 nigel 75 PCRE NATIVE API
578 nigel 63
579 nigel 73 #include <pcre.h>
580 nigel 41
581 nigel 73 pcre *pcre_compile(const char *pattern, int options,
582     const char **errptr, int *erroffset,
583     const unsigned char *tableptr);
584 nigel 41
585 nigel 77 pcre *pcre_compile2(const char *pattern, int options,
586     int *errorcodeptr,
587     const char **errptr, int *erroffset,
588     const unsigned char *tableptr);
589    
590 nigel 73 pcre_extra *pcre_study(const pcre *code, int options,
591     const char **errptr);
592 nigel 41
593 nigel 73 int pcre_exec(const pcre *code, const pcre_extra *extra,
594     const char *subject, int length, int startoffset,
595     int options, int *ovector, int ovecsize);
596 nigel 41
597 nigel 77 int pcre_dfa_exec(const pcre *code, const pcre_extra *extra,
598     const char *subject, int length, int startoffset,
599     int options, int *ovector, int ovecsize,
600     int *workspace, int wscount);
601    
602 nigel 73 int pcre_copy_named_substring(const pcre *code,
603     const char *subject, int *ovector,
604     int stringcount, const char *stringname,
605     char *buffer, int buffersize);
606 nigel 63
607 nigel 73 int pcre_copy_substring(const char *subject, int *ovector,
608     int stringcount, int stringnumber, char *buffer,
609     int buffersize);
610 nigel 41
611 nigel 73 int pcre_get_named_substring(const pcre *code,
612     const char *subject, int *ovector,
613     int stringcount, const char *stringname,
614     const char **stringptr);
615 nigel 63
616 nigel 73 int pcre_get_stringnumber(const pcre *code,
617     const char *name);
618 nigel 63
619 nigel 73 int pcre_get_substring(const char *subject, int *ovector,
620     int stringcount, int stringnumber,
621     const char **stringptr);
622 nigel 41
623 nigel 73 int pcre_get_substring_list(const char *subject,
624     int *ovector, int stringcount, const char ***listptr);
625 nigel 41
626 nigel 73 void pcre_free_substring(const char *stringptr);
627 nigel 49
628 nigel 73 void pcre_free_substring_list(const char **stringptr);
629 nigel 49
630 nigel 73 const unsigned char *pcre_maketables(void);
631 nigel 41
632 nigel 73 int pcre_fullinfo(const pcre *code, const pcre_extra *extra,
633     int what, void *where);
634 nigel 43
635 nigel 73 int pcre_info(const pcre *code, int *optptr, int *firstcharptr);
636 nigel 63
637 nigel 77 int pcre_refcount(pcre *code, int adjust);
638    
639 nigel 73 int pcre_config(int what, void *where);
640 nigel 41
641 nigel 73 char *pcre_version(void);
642 nigel 63
643 nigel 73 void *(*pcre_malloc)(size_t);
644 nigel 41
645 nigel 73 void (*pcre_free)(void *);
646 nigel 41
647 nigel 73 void *(*pcre_stack_malloc)(size_t);
648 nigel 41
649 nigel 73 void (*pcre_stack_free)(void *);
650 nigel 41
651 nigel 73 int (*pcre_callout)(pcre_callout_block *);
652 nigel 41
653 nigel 73
654 nigel 75 PCRE API OVERVIEW
655 nigel 41
656 nigel 73 PCRE has its own native API, which is described in this document. There
657     is also a set of wrapper functions that correspond to the POSIX regular
658 nigel 77 expression API. These are described in the pcreposix documentation.
659     Both of these APIs define a set of C function calls. A C++ wrapper is
660     distributed with PCRE. It is documented in the pcrecpp page.
661 nigel 43
662 nigel 77 The native API C function prototypes are defined in the header file
663     pcre.h, and on Unix systems the library itself is called libpcre. It
664 nigel 75 can normally be accessed by adding -lpcre to the command for linking an
665     application that uses PCRE. The header file defines the macros
666     PCRE_MAJOR and PCRE_MINOR to contain the major and minor release num-
667     bers for the library. Applications can use these to include support
668     for different releases of PCRE.
669 nigel 41
670 nigel 77 The functions pcre_compile(), pcre_compile2(), pcre_study(), and
671     pcre_exec() are used for compiling and matching regular expressions in
672     a Perl-compatible manner. A sample program that demonstrates the sim-
673     plest way of using them is provided in the file called pcredemo.c in
674     the source distribution. The pcresample documentation describes how to
675     run it.
676 nigel 49
677 nigel 77 A second matching function, pcre_dfa_exec(), which is not Perl-compati-
678     ble, is also provided. This uses a different algorithm for the match-
679     ing. This allows it to find all possible matches (at a given point in
680     the subject), not just one. However, this algorithm does not return
681     captured substrings. A description of the two matching algorithms and
682     their advantages and disadvantages is given in the pcrematching docu-
683     mentation.
684 nigel 63
685 nigel 77 In addition to the main compiling and matching functions, there are
686     convenience functions for extracting captured substrings from a subject
687     string that is matched by pcre_exec(). They are:
688    
689 nigel 73 pcre_copy_substring()
690     pcre_copy_named_substring()
691     pcre_get_substring()
692     pcre_get_named_substring()
693     pcre_get_substring_list()
694 nigel 75 pcre_get_stringnumber()
695 nigel 63
696 nigel 73 pcre_free_substring() and pcre_free_substring_list() are also provided,
697     to free the memory used for extracted strings.
698 nigel 41
699 nigel 77 The function pcre_maketables() is used to build a set of character
700     tables in the current locale for passing to pcre_compile(),
701     pcre_exec(), or pcre_dfa_exec(). This is an optional facility that is
702     provided for specialist use. Most commonly, no special tables are
703     passed, in which case internal tables that are generated when PCRE is
704     built are used.
705 nigel 49
706 nigel 75 The function pcre_fullinfo() is used to find out information about a
707     compiled pattern; pcre_info() is an obsolete version that returns only
708     some of the available information, but is retained for backwards com-
709     patibility. The function pcre_version() returns a pointer to a string
710 nigel 73 containing the version of PCRE and its date of release.
711 nigel 41
712 nigel 77 The function pcre_refcount() maintains a reference count in a data
713     block containing a compiled pattern. This is provided for the benefit
714     of object-oriented applications.
715    
716 nigel 75 The global variables pcre_malloc and pcre_free initially contain the
717     entry points of the standard malloc() and free() functions, respec-
718 nigel 73 tively. PCRE calls the memory management functions via these variables,
719 nigel 75 so a calling program can replace them if it wishes to intercept the
720 nigel 73 calls. This should be done before calling any PCRE functions.
721 nigel 41
722 nigel 75 The global variables pcre_stack_malloc and pcre_stack_free are also
723     indirections to memory management functions. These special functions
724     are used only when PCRE is compiled to use the heap for remembering
725 nigel 77 data, instead of recursive function calls, when running the pcre_exec()
726     function. This is a non-standard way of building PCRE, for use in envi-
727     ronments that have limited stacks. Because of the greater use of memory
728     management, it runs more slowly. Separate functions are provided so
729     that special-purpose external code can be used for this case. When
730     used, these functions are always called in a stack-like manner (last
731     obtained, first freed), and always for memory blocks of the same size.
732 nigel 41
733 nigel 73 The global variable pcre_callout initially contains NULL. It can be set
734 nigel 77 by the caller to a "callout" function, which PCRE will then call at
735     specified points during a matching operation. Details are given in the
736 nigel 73 pcrecallout documentation.
737 nigel 41
738 nigel 73
739 nigel 63 MULTITHREADING
740    
741 nigel 77 The PCRE functions can be used in multi-threading applications, with
742 nigel 73 the proviso that the memory management functions pointed to by
743     pcre_malloc, pcre_free, pcre_stack_malloc, and pcre_stack_free, and the
744     callout function pointed to by pcre_callout, are shared by all threads.
745 nigel 41
746 nigel 77 The compiled form of a regular expression is not altered during match-
747 nigel 73 ing, so the same compiled pattern can safely be used by several threads
748     at once.
749 nigel 41
750    
751 nigel 75 SAVING PRECOMPILED PATTERNS FOR LATER USE
752    
753     The compiled form of a regular expression can be saved and re-used at a
754 nigel 77 later time, possibly by a different program, and even on a host other
755     than the one on which it was compiled. Details are given in the
756 nigel 75 pcreprecompile documentation.
757    
758    
759 nigel 63 CHECKING BUILD-TIME OPTIONS
760 nigel 41
761 nigel 73 int pcre_config(int what, void *where);
762 nigel 63
763 nigel 77 The function pcre_config() makes it possible for a PCRE client to dis-
764 nigel 73 cover which optional features have been compiled into the PCRE library.
765 nigel 77 The pcrebuild documentation has more details about these optional fea-
766 nigel 73 tures.
767 nigel 63
768 nigel 77 The first argument for pcre_config() is an integer, specifying which
769 nigel 73 information is required; the second argument is a pointer to a variable
770 nigel 77 into which the information is placed. The following information is
771 nigel 73 available:
772 nigel 63
773 nigel 73 PCRE_CONFIG_UTF8
774 nigel 63
775 nigel 77 The output is an integer that is set to one if UTF-8 support is avail-
776 nigel 73 able; otherwise it is set to zero.
777 nigel 63
778 nigel 75 PCRE_CONFIG_UNICODE_PROPERTIES
779    
780 nigel 77 The output is an integer that is set to one if support for Unicode
781 nigel 75 character properties is available; otherwise it is set to zero.
782    
783 nigel 73 PCRE_CONFIG_NEWLINE
784 nigel 63
785 nigel 77 The output is an integer that is set to the value of the code that is
786     used for the newline character. It is either linefeed (10) or carriage
787     return (13), and should normally be the standard character for your
788 nigel 73 operating system.
789 nigel 63
790 nigel 73 PCRE_CONFIG_LINK_SIZE
791 nigel 63
792 nigel 77 The output is an integer that contains the number of bytes used for
793 nigel 73 internal linkage in compiled regular expressions. The value is 2, 3, or
794 nigel 77 4. Larger values allow larger regular expressions to be compiled, at
795     the expense of slower matching. The default value of 2 is sufficient
796     for all but the most massive patterns, since it allows the compiled
797 nigel 73 pattern to be up to 64K in size.
798 nigel 63
799 nigel 73 PCRE_CONFIG_POSIX_MALLOC_THRESHOLD
800 nigel 63
801 nigel 77 The output is an integer that contains the threshold above which the
802     POSIX interface uses malloc() for output vectors. Further details are
803 nigel 73 given in the pcreposix documentation.
804 nigel 63
805 nigel 73 PCRE_CONFIG_MATCH_LIMIT
806 nigel 63
807 nigel 73 The output is an integer that gives the default limit for the number of
808 nigel 77 internal matching function calls in a pcre_exec() execution. Further
809 nigel 73 details are given with pcre_exec() below.
810 nigel 63
811 nigel 73 PCRE_CONFIG_STACKRECURSE
812 nigel 63
813 nigel 77 The output is an integer that is set to one if internal recursion when
814     running pcre_exec() is implemented by recursive function calls that use
815     the stack to remember their state. This is the usual way that PCRE is
816     compiled. The output is zero if PCRE was compiled to use blocks of data
817     on the heap instead of recursive function calls. In this case,
818     pcre_stack_malloc and pcre_stack_free are called to manage memory
819     blocks on the heap, thus avoiding the use of the stack.
820 nigel 73
821    
822 nigel 41 COMPILING A PATTERN
823 nigel 63
824 nigel 73 pcre *pcre_compile(const char *pattern, int options,
825     const char **errptr, int *erroffset,
826     const unsigned char *tableptr);
827 nigel 63
828 nigel 77 pcre *pcre_compile2(const char *pattern, int options,
829     int *errorcodeptr,
830     const char **errptr, int *erroffset,
831     const unsigned char *tableptr);
832 nigel 41
833 nigel 77 Either of the functions pcre_compile() or pcre_compile2() can be called
834     to compile a pattern into an internal form. The only difference between
835     the two interfaces is that pcre_compile2() has an additional argument,
836     errorcodeptr, via which a numerical error code can be returned.
837    
838     The pattern is a C string terminated by a binary zero, and is passed in
839     the pattern argument. A pointer to a single block of memory that is
840     obtained via pcre_malloc is returned. This contains the compiled code
841     and related data. The pcre type is defined for the returned block; this
842     is a typedef for a structure whose contents are not externally defined.
843     It is up to the caller to free the memory when it is no longer
844     required.
845    
846     Although the compiled code of a PCRE regex is relocatable, that is, it
847 nigel 73 does not depend on memory location, the complete pcre data block is not
848 nigel 77 fully relocatable, because it may contain a copy of the tableptr argu-
849 nigel 75 ment, which is an address (see below).
850 nigel 41
851 nigel 73 The options argument contains independent bits that affect the compila-
852 nigel 77 tion. It should be zero if no options are required. The available
853     options are described below. Some of them, in particular, those that
854     are compatible with Perl, can also be set and unset from within the
855     pattern (see the detailed description in the pcrepattern documenta-
856     tion). For these options, the contents of the options argument speci-
857     fies their initial settings at the start of compilation and execution.
858     The PCRE_ANCHORED option can be set at the time of matching as well as
859 nigel 75 at compile time.
860 nigel 41
861 nigel 73 If errptr is NULL, pcre_compile() returns NULL immediately. Otherwise,
862 nigel 77 if compilation of a pattern fails, pcre_compile() returns NULL, and
863 nigel 73 sets the variable pointed to by errptr to point to a textual error mes-
864 nigel 77 sage. The offset from the start of the pattern to the character where
865     the error was discovered is placed in the variable pointed to by
866     erroffset, which must not be NULL. If it is, an immediate error is
867 nigel 73 given.
868 nigel 53
869 nigel 77 If pcre_compile2() is used instead of pcre_compile(), and the error-
870     codeptr argument is not NULL, a non-zero error code number is returned
871     via this argument in the event of an error. This is in addition to the
872     textual error message. Error codes and messages are listed below.
873    
874 nigel 73 If the final argument, tableptr, is NULL, PCRE uses a default set of
875 nigel 75 character tables that are built when PCRE is compiled, using the
876     default C locale. Otherwise, tableptr must be an address that is the
877     result of a call to pcre_maketables(). This value is stored with the
878     compiled pattern, and used again by pcre_exec(), unless another table
879     pointer is passed to it. For more discussion, see the section on locale
880     support below.
881 nigel 53
882 nigel 75 This code fragment shows a typical straightforward call to pcre_com-
883 nigel 73 pile():
884 nigel 41
885 nigel 73 pcre *re;
886     const char *error;
887     int erroffset;
888     re = pcre_compile(
889     "^A.*Z", /* the pattern */
890     0, /* default options */
891     &error, /* for error message */
892     &erroffset, /* for error offset */
893     NULL); /* use default character tables */
894 nigel 41
895 nigel 75 The following names for option bits are defined in the pcre.h header
896     file:
897 nigel 41
898 nigel 73 PCRE_ANCHORED
899 nigel 41
900 nigel 73 If this bit is set, the pattern is forced to be "anchored", that is, it
901     is constrained to match only at the first matching point in the string
902 nigel 75 that is being searched (the "subject string"). This effect can also be
903 nigel 73 achieved by appropriate constructs in the pattern itself, which is the
904     only way to do it in Perl.
905 nigel 41
906 nigel 75 PCRE_AUTO_CALLOUT
907    
908     If this bit is set, pcre_compile() automatically inserts callout items,
909     all with number 255, before each pattern item. For discussion of the
910     callout facility, see the pcrecallout documentation.
911    
912 nigel 73 PCRE_CASELESS
913 nigel 41
914 nigel 73 If this bit is set, letters in the pattern match both upper and lower
915     case letters. It is equivalent to Perl's /i option, and it can be
916 nigel 77 changed within a pattern by a (?i) option setting. In UTF-8 mode, PCRE
917     always understands the concept of case for characters whose values are
918     less than 128, so caseless matching is always possible. For characters
919     with higher values, the concept of case is supported if PCRE is com-
920     piled with Unicode property support, but not otherwise. If you want to
921     use caseless matching for characters 128 and above, you must ensure
922     that PCRE is compiled with Unicode property support as well as with
923     UTF-8 support.
924 nigel 41
925 nigel 73 PCRE_DOLLAR_ENDONLY
926 nigel 41
927 nigel 77 If this bit is set, a dollar metacharacter in the pattern matches only
928     at the end of the subject string. Without this option, a dollar also
929     matches immediately before the final character if it is a newline (but
930     not before any other newlines). The PCRE_DOLLAR_ENDONLY option is
931 nigel 73 ignored if PCRE_MULTILINE is set. There is no equivalent to this option
932     in Perl, and no way to set it within a pattern.
933 nigel 41
934 nigel 73 PCRE_DOTALL
935 nigel 41
936 nigel 73 If this bit is set, a dot metacharater in the pattern matches all char-
937 nigel 77 acters, including newlines. Without it, newlines are excluded. This
938     option is equivalent to Perl's /s option, and it can be changed within
939     a pattern by a (?s) option setting. A negative class such as [^a]
940     always matches a newline character, independent of the setting of this
941 nigel 73 option.
942 nigel 63
943 nigel 73 PCRE_EXTENDED
944 nigel 41
945 nigel 77 If this bit is set, whitespace data characters in the pattern are
946     totally ignored except when escaped or inside a character class. White-
947     space does not include the VT character (code 11). In addition, charac-
948     ters between an unescaped # outside a character class and the next new-
949     line character, inclusive, are also ignored. This is equivalent to
950     Perl's /x option, and it can be changed within a pattern by a (?x)
951 nigel 73 option setting.
952 nigel 41
953 nigel 77 This option makes it possible to include comments inside complicated
954     patterns. Note, however, that this applies only to data characters.
955     Whitespace characters may never appear within special character
956     sequences in a pattern, for example within the sequence (?( which
957 nigel 73 introduces a conditional subpattern.
958 nigel 41
959 nigel 73 PCRE_EXTRA
960 nigel 41
961 nigel 77 This option was invented in order to turn on additional functionality
962     of PCRE that is incompatible with Perl, but it is currently of very
963     little use. When set, any backslash in a pattern that is followed by a
964     letter that has no special meaning causes an error, thus reserving
965     these combinations for future expansion. By default, as in Perl, a
966     backslash followed by a letter with no special meaning is treated as a
967     literal. There are at present no other features controlled by this
968 nigel 73 option. It can also be set by a (?X) option setting within a pattern.
969 nigel 41
970 nigel 77 PCRE_FIRSTLINE
971    
972     If this option is set, an unanchored pattern is required to match
973     before or at the first newline character in the subject string, though
974     the matched text may continue over the newline.
975    
976 nigel 73 PCRE_MULTILINE
977 nigel 41
978 nigel 77 By default, PCRE treats the subject string as consisting of a single
979     line of characters (even if it actually contains newlines). The "start
980     of line" metacharacter (^) matches only at the start of the string,
981     while the "end of line" metacharacter ($) matches only at the end of
982 nigel 75 the string, or before a terminating newline (unless PCRE_DOLLAR_ENDONLY
983     is set). This is the same as Perl.
984 nigel 63
985 nigel 77 When PCRE_MULTILINE it is set, the "start of line" and "end of line"
986     constructs match immediately following or immediately before any new-
987     line in the subject string, respectively, as well as at the very start
988     and end. This is equivalent to Perl's /m option, and it can be changed
989 nigel 73 within a pattern by a (?m) option setting. If there are no "\n" charac-
990 nigel 77 ters in a subject string, or no occurrences of ^ or $ in a pattern,
991 nigel 73 setting PCRE_MULTILINE has no effect.
992 nigel 63
993 nigel 73 PCRE_NO_AUTO_CAPTURE
994 nigel 41
995 nigel 73 If this option is set, it disables the use of numbered capturing paren-
996 nigel 77 theses in the pattern. Any opening parenthesis that is not followed by
997     ? behaves as if it were followed by ?: but named parentheses can still
998     be used for capturing (and they acquire numbers in the usual way).
999 nigel 73 There is no equivalent of this option in Perl.
1000 nigel 41
1001 nigel 73 PCRE_UNGREEDY
1002 nigel 41
1003 nigel 77 This option inverts the "greediness" of the quantifiers so that they
1004     are not greedy by default, but become greedy if followed by "?". It is
1005     not compatible with Perl. It can also be set by a (?U) option setting
1006 nigel 73 within the pattern.
1007 nigel 41
1008 nigel 73 PCRE_UTF8
1009 nigel 49
1010 nigel 77 This option causes PCRE to regard both the pattern and the subject as
1011     strings of UTF-8 characters instead of single-byte character strings.
1012     However, it is available only when PCRE is built to include UTF-8 sup-
1013     port. If not, the use of this option provokes an error. Details of how
1014     this option changes the behaviour of PCRE are given in the section on
1015 nigel 75 UTF-8 support in the main pcre page.
1016 nigel 71
1017 nigel 73 PCRE_NO_UTF8_CHECK
1018 nigel 71
1019 nigel 73 When PCRE_UTF8 is set, the validity of the pattern as a UTF-8 string is
1020 nigel 77 automatically checked. If an invalid UTF-8 sequence of bytes is found,
1021     pcre_compile() returns an error. If you already know that your pattern
1022     is valid, and you want to skip this check for performance reasons, you
1023     can set the PCRE_NO_UTF8_CHECK option. When it is set, the effect of
1024 nigel 73 passing an invalid UTF-8 string as a pattern is undefined. It may cause
1025 nigel 77 your program to crash. Note that this option can also be passed to
1026     pcre_exec() and pcre_dfa_exec(), to suppress the UTF-8 validity check-
1027     ing of subject strings.
1028 nigel 71
1029 nigel 73
1030 nigel 77 COMPILATION ERROR CODES
1031    
1032     The following table lists the error codes than may be returned by
1033     pcre_compile2(), along with the error messages that may be returned by
1034     both compiling functions.
1035    
1036     0 no error
1037     1 \ at end of pattern
1038     2 \c at end of pattern
1039     3 unrecognized character follows \
1040     4 numbers out of order in {} quantifier
1041     5 number too big in {} quantifier
1042     6 missing terminating ] for character class
1043     7 invalid escape sequence in character class
1044     8 range out of order in character class
1045     9 nothing to repeat
1046     10 operand of unlimited repeat could match the empty string
1047     11 internal error: unexpected repeat
1048     12 unrecognized character after (?
1049     13 POSIX named classes are supported only within a class
1050     14 missing )
1051     15 reference to non-existent subpattern
1052     16 erroffset passed as NULL
1053     17 unknown option bit(s) set
1054     18 missing ) after comment
1055     19 parentheses nested too deeply
1056     20 regular expression too large
1057     21 failed to get memory
1058     22 unmatched parentheses
1059     23 internal error: code overflow
1060     24 unrecognized character after (?<
1061     25 lookbehind assertion is not fixed length
1062     26 malformed number after (?(
1063     27 conditional group contains more than two branches
1064     28 assertion expected after (?(
1065     29 (?R or (?digits must be followed by )
1066     30 unknown POSIX class name
1067     31 POSIX collating elements are not supported
1068     32 this version of PCRE is not compiled with PCRE_UTF8 support
1069     33 spare error
1070     34 character value in \x{...} sequence is too large
1071     35 invalid condition (?(0)
1072     36 \C not allowed in lookbehind assertion
1073     37 PCRE does not support \L, \l, \N, \U, or \u
1074     38 number after (?C is > 255
1075     39 closing ) for (?C expected
1076     40 recursive call could loop indefinitely
1077     41 unrecognized character after (?P
1078     42 syntax error after (?P
1079     43 two named groups have the same name
1080     44 invalid UTF-8 string
1081     45 support for \P, \p, and \X has not been compiled
1082     46 malformed \P or \p sequence
1083     47 unknown property name after \P or \p
1084    
1085    
1086 nigel 63 STUDYING A PATTERN
1087 nigel 49
1088 nigel 77 pcre_extra *pcre_study(const pcre *code, int options
1089 nigel 73 const char **errptr);
1090 nigel 63
1091 nigel 77 If a compiled pattern is going to be used several times, it is worth
1092 nigel 75 spending more time analyzing it in order to speed up the time taken for
1093 nigel 77 matching. The function pcre_study() takes a pointer to a compiled pat-
1094 nigel 75 tern as its first argument. If studying the pattern produces additional
1095 nigel 77 information that will help speed up matching, pcre_study() returns a
1096     pointer to a pcre_extra block, in which the study_data field points to
1097 nigel 75 the results of the study.
1098 nigel 41
1099 nigel 75 The returned value from pcre_study() can be passed directly to
1100 nigel 77 pcre_exec(). However, a pcre_extra block also contains other fields
1101     that can be set by the caller before the block is passed; these are
1102 nigel 75 described below in the section on matching a pattern.
1103 nigel 63
1104 nigel 77 If studying the pattern does not produce any additional information
1105 nigel 75 pcre_study() returns NULL. In that circumstance, if the calling program
1106 nigel 77 wants to pass any of the other fields to pcre_exec(), it must set up
1107 nigel 75 its own pcre_extra block.
1108 nigel 41
1109 nigel 77 The second argument of pcre_study() contains option bits. At present,
1110 nigel 75 no options are defined, and this argument should always be zero.
1111    
1112 nigel 77 The third argument for pcre_study() is a pointer for an error message.
1113     If studying succeeds (even if no data is returned), the variable it
1114     points to is set to NULL. Otherwise it points to a textual error mes-
1115     sage. You should therefore test the error pointer for NULL after call-
1116 nigel 73 ing pcre_study(), to be sure that it has run successfully.
1117 nigel 41
1118 nigel 73 This is a typical call to pcre_study():
1119 nigel 53
1120 nigel 73 pcre_extra *pe;
1121     pe = pcre_study(
1122     re, /* result of pcre_compile() */
1123     0, /* no options exist */
1124     &error); /* set to NULL or points to a message */
1125 nigel 53
1126 nigel 73 At present, studying a pattern is useful only for non-anchored patterns
1127 nigel 77 that do not have a single fixed starting character. A bitmap of possi-
1128 nigel 75 ble starting bytes is created.
1129 nigel 41
1130    
1131 nigel 63 LOCALE SUPPORT
1132 nigel 41
1133 nigel 77 PCRE handles caseless matching, and determines whether characters are
1134     letters digits, or whatever, by reference to a set of tables, indexed
1135     by character value. When running in UTF-8 mode, this applies only to
1136     characters with codes less than 128. Higher-valued codes never match
1137     escapes such as \w or \d, but can be tested with \p if PCRE is built
1138     with Unicode character property support.
1139 nigel 41
1140 nigel 77 An internal set of tables is created in the default C locale when PCRE
1141     is built. This is used when the final argument of pcre_compile() is
1142     NULL, and is sufficient for many applications. An alternative set of
1143     tables can, however, be supplied. These may be created in a different
1144     locale from the default. As more and more applications change to using
1145 nigel 75 Unicode, the need for this locale support is expected to die away.
1146 nigel 41
1147 nigel 77 External tables are built by calling the pcre_maketables() function,
1148     which has no arguments, in the relevant locale. The result can then be
1149     passed to pcre_compile() or pcre_exec() as often as necessary. For
1150     example, to build and use tables that are appropriate for the French
1151     locale (where accented characters with values greater than 128 are
1152 nigel 75 treated as letters), the following code could be used:
1153    
1154     setlocale(LC_CTYPE, "fr_FR");
1155 nigel 73 tables = pcre_maketables();
1156     re = pcre_compile(..., tables);
1157 nigel 41
1158 nigel 77 When pcre_maketables() runs, the tables are built in memory that is
1159     obtained via pcre_malloc. It is the caller's responsibility to ensure
1160     that the memory containing the tables remains available for as long as
1161 nigel 75 it is needed.
1162 nigel 41
1163 nigel 75 The pointer that is passed to pcre_compile() is saved with the compiled
1164 nigel 77 pattern, and the same tables are used via this pointer by pcre_study()
1165 nigel 75 and normally also by pcre_exec(). Thus, by default, for any single pat-
1166     tern, compilation, studying and matching all happen in the same locale,
1167     but different patterns can be compiled in different locales.
1168 nigel 41
1169 nigel 77 It is possible to pass a table pointer or NULL (indicating the use of
1170     the internal tables) to pcre_exec(). Although not intended for this
1171     purpose, this facility could be used to match a pattern in a different
1172 nigel 75 locale from the one in which it was compiled. Passing table pointers at
1173     run time is discussed below in the section on matching a pattern.
1174    
1175    
1176 nigel 63 INFORMATION ABOUT A PATTERN
1177 nigel 41
1178 nigel 73 int pcre_fullinfo(const pcre *code, const pcre_extra *extra,
1179     int what, void *where);
1180 nigel 63
1181 nigel 77 The pcre_fullinfo() function returns information about a compiled pat-
1182 nigel 73 tern. It replaces the obsolete pcre_info() function, which is neverthe-
1183     less retained for backwards compability (and is documented below).
1184 nigel 43
1185 nigel 77 The first argument for pcre_fullinfo() is a pointer to the compiled
1186     pattern. The second argument is the result of pcre_study(), or NULL if
1187     the pattern was not studied. The third argument specifies which piece
1188     of information is required, and the fourth argument is a pointer to a
1189     variable to receive the data. The yield of the function is zero for
1190 nigel 73 success, or one of the following negative numbers:
1191 nigel 41
1192 nigel 73 PCRE_ERROR_NULL the argument code was NULL
1193     the argument where was NULL
1194     PCRE_ERROR_BADMAGIC the "magic number" was not found
1195     PCRE_ERROR_BADOPTION the value of what was invalid
1196 nigel 53
1197 nigel 77 The "magic number" is placed at the start of each compiled pattern as
1198     an simple check against passing an arbitrary memory pointer. Here is a
1199     typical call of pcre_fullinfo(), to obtain the length of the compiled
1200 nigel 75 pattern:
1201 nigel 53
1202 nigel 73 int rc;
1203     unsigned long int length;
1204     rc = pcre_fullinfo(
1205     re, /* result of pcre_compile() */
1206     pe, /* result of pcre_study(), or NULL */
1207     PCRE_INFO_SIZE, /* what is required */
1208     &length); /* where to put the data */
1209 nigel 43
1210 nigel 77 The possible values for the third argument are defined in pcre.h, and
1211 nigel 73 are as follows:
1212 nigel 43
1213 nigel 73 PCRE_INFO_BACKREFMAX
1214 nigel 41
1215 nigel 77 Return the number of the highest back reference in the pattern. The
1216     fourth argument should point to an int variable. Zero is returned if
1217 nigel 73 there are no back references.
1218 nigel 43
1219 nigel 73 PCRE_INFO_CAPTURECOUNT
1220 nigel 43
1221 nigel 77 Return the number of capturing subpatterns in the pattern. The fourth
1222 nigel 73 argument should point to an int variable.
1223 nigel 43
1224 nigel 77 PCRE_INFO_DEFAULT_TABLES
1225 nigel 75
1226 nigel 77 Return a pointer to the internal default character tables within PCRE.
1227     The fourth argument should point to an unsigned char * variable. This
1228 nigel 75 information call is provided for internal use by the pcre_study() func-
1229 nigel 77 tion. External callers can cause PCRE to use its internal tables by
1230 nigel 75 passing a NULL table pointer.
1231    
1232 nigel 73 PCRE_INFO_FIRSTBYTE
1233 nigel 43
1234 nigel 77 Return information about the first byte of any matched string, for a
1235     non-anchored pattern. (This option used to be called
1236     PCRE_INFO_FIRSTCHAR; the old name is still recognized for backwards
1237 nigel 73 compatibility.)
1238 nigel 41
1239 nigel 77 If there is a fixed first byte, for example, from a pattern such as
1240     (cat|cow|coyote), it is returned in the integer pointed to by where.
1241 nigel 73 Otherwise, if either
1242 nigel 41
1243 nigel 77 (a) the pattern was compiled with the PCRE_MULTILINE option, and every
1244 nigel 73 branch starts with "^", or
1245 nigel 43
1246 nigel 73 (b) every branch of the pattern starts with ".*" and PCRE_DOTALL is not
1247     set (if it were set, the pattern would be anchored),
1248 nigel 41
1249 nigel 77 -1 is returned, indicating that the pattern matches only at the start
1250     of a subject string or after any newline within the string. Otherwise
1251 nigel 73 -2 is returned. For anchored patterns, -2 is returned.
1252 nigel 41
1253 nigel 73 PCRE_INFO_FIRSTTABLE
1254 nigel 41
1255 nigel 77 If the pattern was studied, and this resulted in the construction of a
1256 nigel 73 256-bit table indicating a fixed set of bytes for the first byte in any
1257 nigel 77 matching string, a pointer to the table is returned. Otherwise NULL is
1258     returned. The fourth argument should point to an unsigned char * vari-
1259 nigel 73 able.
1260 nigel 43
1261 nigel 73 PCRE_INFO_LASTLITERAL
1262 nigel 43
1263 nigel 77 Return the value of the rightmost literal byte that must exist in any
1264     matched string, other than at its start, if such a byte has been
1265 nigel 73 recorded. The fourth argument should point to an int variable. If there
1266 nigel 77 is no such byte, -1 is returned. For anchored patterns, a last literal
1267     byte is recorded only if it follows something of variable length. For
1268 nigel 73 example, for the pattern /^a\d+z\d+/ the returned value is "z", but for
1269     /^a\dz\d/ the returned value is -1.
1270 nigel 63
1271 nigel 73 PCRE_INFO_NAMECOUNT
1272     PCRE_INFO_NAMEENTRYSIZE
1273     PCRE_INFO_NAMETABLE
1274 nigel 63
1275 nigel 77 PCRE supports the use of named as well as numbered capturing parenthe-
1276     ses. The names are just an additional way of identifying the parenthe-
1277 nigel 75 ses, which still acquire numbers. A convenience function called
1278 nigel 77 pcre_get_named_substring() is provided for extracting an individual
1279     captured substring by name. It is also possible to extract the data
1280     directly, by first converting the name to a number in order to access
1281     the correct pointers in the output vector (described with pcre_exec()
1282     below). To do the conversion, you need to use the name-to-number map,
1283 nigel 75 which is described by these three values.
1284 nigel 63
1285 nigel 73 The map consists of a number of fixed-size entries. PCRE_INFO_NAMECOUNT
1286     gives the number of entries, and PCRE_INFO_NAMEENTRYSIZE gives the size
1287 nigel 77 of each entry; both of these return an int value. The entry size
1288     depends on the length of the longest name. PCRE_INFO_NAMETABLE returns
1289     a pointer to the first entry of the table (a pointer to char). The
1290 nigel 73 first two bytes of each entry are the number of the capturing parenthe-
1291 nigel 77 sis, most significant byte first. The rest of the entry is the corre-
1292     sponding name, zero terminated. The names are in alphabetical order.
1293     For example, consider the following pattern (assume PCRE_EXTENDED is
1294 nigel 73 set, so white space - including newlines - is ignored):
1295 nigel 63
1296 nigel 73 (?P<date> (?P<year>(\d\d)?\d\d) -
1297     (?P<month>\d\d) - (?P<day>\d\d) )
1298 nigel 63
1299 nigel 77 There are four named subpatterns, so the table has four entries, and
1300     each entry in the table is eight bytes long. The table is as follows,
1301 nigel 75 with non-printing bytes shows in hexadecimal, and undefined bytes shown
1302     as ??:
1303 nigel 63
1304 nigel 73 00 01 d a t e 00 ??
1305     00 05 d a y 00 ?? ??
1306     00 04 m o n t h 00
1307     00 02 y e a r 00 ??
1308 nigel 63
1309 nigel 77 When writing code to extract data from named subpatterns using the
1310 nigel 75 name-to-number map, remember that the length of each entry is likely to
1311     be different for each compiled pattern.
1312 nigel 63
1313 nigel 73 PCRE_INFO_OPTIONS
1314 nigel 63
1315 nigel 77 Return a copy of the options with which the pattern was compiled. The
1316     fourth argument should point to an unsigned long int variable. These
1317 nigel 73 option bits are those specified in the call to pcre_compile(), modified
1318     by any top-level option settings within the pattern itself.
1319 nigel 63
1320 nigel 77 A pattern is automatically anchored by PCRE if all of its top-level
1321 nigel 73 alternatives begin with one of the following:
1322 nigel 63
1323 nigel 73 ^ unless PCRE_MULTILINE is set
1324     \A always
1325     \G always
1326     .* if PCRE_DOTALL is set and there are no back
1327     references to the subpattern in which .* appears
1328 nigel 63
1329 nigel 73 For such patterns, the PCRE_ANCHORED bit is set in the options returned
1330     by pcre_fullinfo().
1331 nigel 63
1332 nigel 73 PCRE_INFO_SIZE
1333 nigel 63
1334 nigel 77 Return the size of the compiled pattern, that is, the value that was
1335 nigel 73 passed as the argument to pcre_malloc() when PCRE was getting memory in
1336     which to place the compiled data. The fourth argument should point to a
1337     size_t variable.
1338 nigel 63
1339 nigel 73 PCRE_INFO_STUDYSIZE
1340 nigel 63
1341 nigel 75 Return the size of the data block pointed to by the study_data field in
1342 nigel 77 a pcre_extra block. That is, it is the value that was passed to
1343 nigel 73 pcre_malloc() when PCRE was getting memory into which to place the data
1344 nigel 77 created by pcre_study(). The fourth argument should point to a size_t
1345 nigel 73 variable.
1346 nigel 63
1347 nigel 73
1348 nigel 63 OBSOLETE INFO FUNCTION
1349    
1350 nigel 73 int pcre_info(const pcre *code, int *optptr, int *firstcharptr);
1351 nigel 63
1352 nigel 77 The pcre_info() function is now obsolete because its interface is too
1353     restrictive to return all the available data about a compiled pattern.
1354     New programs should use pcre_fullinfo() instead. The yield of
1355     pcre_info() is the number of capturing subpatterns, or one of the fol-
1356 nigel 73 lowing negative numbers:
1357 nigel 43
1358 nigel 73 PCRE_ERROR_NULL the argument code was NULL
1359     PCRE_ERROR_BADMAGIC the "magic number" was not found
1360 nigel 43
1361 nigel 77 If the optptr argument is not NULL, a copy of the options with which
1362     the pattern was compiled is placed in the integer it points to (see
1363 nigel 73 PCRE_INFO_OPTIONS above).
1364 nigel 43
1365 nigel 77 If the pattern is not anchored and the firstcharptr argument is not
1366     NULL, it is used to pass back information about the first character of
1367 nigel 73 any matched string (see PCRE_INFO_FIRSTBYTE above).
1368 nigel 43
1369    
1370 nigel 77 REFERENCE COUNTS
1371 nigel 53
1372 nigel 77 int pcre_refcount(pcre *code, int adjust);
1373    
1374     The pcre_refcount() function is used to maintain a reference count in
1375     the data block that contains a compiled pattern. It is provided for the
1376     benefit of applications that operate in an object-oriented manner,
1377     where different parts of the application may be using the same compiled
1378     pattern, but you want to free the block when they are all done.
1379    
1380     When a pattern is compiled, the reference count field is initialized to
1381     zero. It is changed only by calling this function, whose action is to
1382     add the adjust value (which may be positive or negative) to it. The
1383     yield of the function is the new value. However, the value of the count
1384     is constrained to lie between 0 and 65535, inclusive. If the new value
1385     is outside these limits, it is forced to the appropriate limit value.
1386    
1387     Except when it is zero, the reference count is not correctly preserved
1388     if a pattern is compiled on one host and then transferred to a host
1389     whose byte-order is different. (This seems a highly unlikely scenario.)
1390    
1391    
1392     MATCHING A PATTERN: THE TRADITIONAL FUNCTION
1393    
1394 nigel 73 int pcre_exec(const pcre *code, const pcre_extra *extra,
1395     const char *subject, int length, int startoffset,
1396     int options, int *ovector, int ovecsize);
1397 nigel 53
1398 nigel 77 The function pcre_exec() is called to match a subject string against a
1399     compiled pattern, which is passed in the code argument. If the pattern
1400 nigel 75 has been studied, the result of the study should be passed in the extra
1401 nigel 77 argument. This function is the main matching facility of the library,
1402     and it operates in a Perl-like manner. For specialist use there is also
1403     an alternative matching function, which is described below in the sec-
1404     tion about the pcre_dfa_exec() function.
1405 nigel 41
1406 nigel 75 In most applications, the pattern will have been compiled (and option-
1407     ally studied) in the same process that calls pcre_exec(). However, it
1408     is possible to save compiled patterns and study data, and then use them
1409     later in different processes, possibly even on different hosts. For a
1410     discussion about this, see the pcreprecompile documentation.
1411    
1412 nigel 73 Here is an example of a simple call to pcre_exec():
1413 nigel 53
1414 nigel 73 int rc;
1415     int ovector[30];
1416     rc = pcre_exec(
1417     re, /* result of pcre_compile() */
1418     NULL, /* we didn't study the pattern */
1419     "some string", /* the subject string */
1420     11, /* the length of the subject string */
1421     0, /* start at offset 0 in the subject */
1422     0, /* default options */
1423 nigel 75 ovector, /* vector of integers for substring information */
1424 nigel 77 30); /* number of elements (NOT size in bytes) */
1425 nigel 53
1426 nigel 75 Extra data for pcre_exec()
1427 nigel 63
1428 nigel 77 If the extra argument is not NULL, it must point to a pcre_extra data
1429     block. The pcre_study() function returns such a block (when it doesn't
1430     return NULL), but you can also create one for yourself, and pass addi-
1431     tional information in it. The fields in a pcre_extra block are as fol-
1432 nigel 75 lows:
1433    
1434 nigel 73 unsigned long int flags;
1435     void *study_data;
1436     unsigned long int match_limit;
1437     void *callout_data;
1438 nigel 75 const unsigned char *tables;
1439 nigel 63
1440 nigel 77 The flags field is a bitmap that specifies which of the other fields
1441 nigel 73 are set. The flag bits are:
1442 nigel 63
1443 nigel 73 PCRE_EXTRA_STUDY_DATA
1444     PCRE_EXTRA_MATCH_LIMIT
1445     PCRE_EXTRA_CALLOUT_DATA
1446 nigel 75 PCRE_EXTRA_TABLES
1447 nigel 63
1448 nigel 77 Other flag bits should be set to zero. The study_data field is set in
1449     the pcre_extra block that is returned by pcre_study(), together with
1450 nigel 75 the appropriate flag bit. You should not set this yourself, but you may
1451 nigel 77 add to the block by setting the other fields and their corresponding
1452 nigel 75 flag bits.
1453 nigel 63
1454 nigel 73 The match_limit field provides a means of preventing PCRE from using up
1455 nigel 77 a vast amount of resources when running patterns that are not going to
1456     match, but which have a very large number of possibilities in their
1457     search trees. The classic example is the use of nested unlimited
1458 nigel 75 repeats.
1459 nigel 63
1460 nigel 77 Internally, PCRE uses a function called match() which it calls repeat-
1461     edly (sometimes recursively). The limit is imposed on the number of
1462     times this function is called during a match, which has the effect of
1463     limiting the amount of recursion and backtracking that can take place.
1464 nigel 75 For patterns that are not anchored, the count starts from zero for each
1465     position in the subject string.
1466    
1467 nigel 77 The default limit for the library can be set when PCRE is built; the
1468     default default is 10 million, which handles all but the most extreme
1469     cases. You can reduce the default by suppling pcre_exec() with a
1470     pcre_extra block in which match_limit is set to a smaller value, and
1471     PCRE_EXTRA_MATCH_LIMIT is set in the flags field. If the limit is
1472 nigel 73 exceeded, pcre_exec() returns PCRE_ERROR_MATCHLIMIT.
1473 nigel 63
1474 nigel 77 The pcre_callout field is used in conjunction with the "callout" fea-
1475 nigel 73 ture, which is described in the pcrecallout documentation.
1476 nigel 63
1477 nigel 77 The tables field is used to pass a character tables pointer to
1478     pcre_exec(); this overrides the value that is stored with the compiled
1479     pattern. A non-NULL value is stored with the compiled pattern only if
1480     custom tables were supplied to pcre_compile() via its tableptr argu-
1481 nigel 75 ment. If NULL is passed to pcre_exec() using this mechanism, it forces
1482 nigel 77 PCRE's internal tables to be used. This facility is helpful when re-
1483     using patterns that have been saved after compiling with an external
1484     set of tables, because the external tables might be at a different
1485     address when pcre_exec() is called. See the pcreprecompile documenta-
1486 nigel 75 tion for a discussion of saving compiled patterns for later use.
1487 nigel 41
1488 nigel 75 Option bits for pcre_exec()
1489 nigel 71
1490 nigel 77 The unused bits of the options argument for pcre_exec() must be zero.
1491     The only bits that may be set are PCRE_ANCHORED, PCRE_NOTBOL,
1492 nigel 75 PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK and PCRE_PARTIAL.
1493 nigel 41
1494 nigel 75 PCRE_ANCHORED
1495 nigel 41
1496 nigel 77 The PCRE_ANCHORED option limits pcre_exec() to matching at the first
1497     matching position. If a pattern was compiled with PCRE_ANCHORED, or
1498     turned out to be anchored by virtue of its contents, it cannot be made
1499 nigel 75 unachored at matching time.
1500    
1501 nigel 73 PCRE_NOTBOL
1502 nigel 41
1503 nigel 75 This option specifies that first character of the subject string is not
1504 nigel 77 the beginning of a line, so the circumflex metacharacter should not
1505     match before it. Setting this without PCRE_MULTILINE (at compile time)
1506     causes circumflex never to match. This option affects only the behav-
1507     iour of the circumflex metacharacter. It does not affect \A.
1508 nigel 41
1509 nigel 73 PCRE_NOTEOL
1510 nigel 41
1511 nigel 75 This option specifies that the end of the subject string is not the end
1512 nigel 77 of a line, so the dollar metacharacter should not match it nor (except
1513     in multiline mode) a newline immediately before it. Setting this with-
1514 nigel 75 out PCRE_MULTILINE (at compile time) causes dollar never to match. This
1515 nigel 77 option affects only the behaviour of the dollar metacharacter. It does
1516 nigel 75 not affect \Z or \z.
1517 nigel 41
1518 nigel 73 PCRE_NOTEMPTY
1519 nigel 41
1520 nigel 73 An empty string is not considered to be a valid match if this option is
1521 nigel 77 set. If there are alternatives in the pattern, they are tried. If all
1522     the alternatives match the empty string, the entire match fails. For
1523 nigel 73 example, if the pattern
1524 nigel 41
1525 nigel 73 a?b?
1526 nigel 41
1527 nigel 77 is applied to a string not beginning with "a" or "b", it matches the
1528     empty string at the start of the subject. With PCRE_NOTEMPTY set, this
1529 nigel 73 match is not valid, so PCRE searches further into the string for occur-
1530     rences of "a" or "b".
1531 nigel 41
1532 nigel 73 Perl has no direct equivalent of PCRE_NOTEMPTY, but it does make a spe-
1533 nigel 77 cial case of a pattern match of the empty string within its split()
1534     function, and when using the /g modifier. It is possible to emulate
1535 nigel 73 Perl's behaviour after matching a null string by first trying the match
1536 nigel 75 again at the same offset with PCRE_NOTEMPTY and PCRE_ANCHORED, and then
1537 nigel 77 if that fails by advancing the starting offset (see below) and trying
1538 nigel 75 an ordinary match again. There is some code that demonstrates how to do
1539     this in the pcredemo.c sample program.
1540 nigel 41
1541 nigel 75 PCRE_NO_UTF8_CHECK
1542    
1543     When PCRE_UTF8 is set at compile time, the validity of the subject as a
1544 nigel 77 UTF-8 string is automatically checked when pcre_exec() is subsequently
1545     called. The value of startoffset is also checked to ensure that it
1546     points to the start of a UTF-8 character. If an invalid UTF-8 sequence
1547 nigel 75 of bytes is found, pcre_exec() returns the error PCRE_ERROR_BADUTF8. If
1548 nigel 77 startoffset contains an invalid value, PCRE_ERROR_BADUTF8_OFFSET is
1549 nigel 75 returned.
1550    
1551 nigel 77 If you already know that your subject is valid, and you want to skip
1552     these checks for performance reasons, you can set the
1553     PCRE_NO_UTF8_CHECK option when calling pcre_exec(). You might want to
1554     do this for the second and subsequent calls to pcre_exec() if you are
1555     making repeated calls to find all the matches in a single subject
1556     string. However, you should be sure that the value of startoffset
1557     points to the start of a UTF-8 character. When PCRE_NO_UTF8_CHECK is
1558     set, the effect of passing an invalid UTF-8 string as a subject, or a
1559     value of startoffset that does not point to the start of a UTF-8 char-
1560 nigel 75 acter, is undefined. Your program may crash.
1561    
1562     PCRE_PARTIAL
1563    
1564 nigel 77 This option turns on the partial matching feature. If the subject
1565     string fails to match the pattern, but at some point during the match-
1566     ing process the end of the subject was reached (that is, the subject
1567     partially matches the pattern and the failure to match occurred only
1568     because there were not enough subject characters), pcre_exec() returns
1569     PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH. When PCRE_PARTIAL is
1570     used, there are restrictions on what may appear in the pattern. These
1571 nigel 75 are discussed in the pcrepartial documentation.
1572    
1573     The string to be matched by pcre_exec()
1574    
1575 nigel 77 The subject string is passed to pcre_exec() as a pointer in subject, a
1576     length in length, and a starting byte offset in startoffset. In UTF-8
1577     mode, the byte offset must point to the start of a UTF-8 character.
1578     Unlike the pattern string, the subject may contain binary zero bytes.
1579     When the starting offset is zero, the search for a match starts at the
1580 nigel 75 beginning of the subject, and this is by far the most common case.
1581 nigel 63
1582 nigel 77 A non-zero starting offset is useful when searching for another match
1583     in the same subject by calling pcre_exec() again after a previous suc-
1584     cess. Setting startoffset differs from just passing over a shortened
1585     string and setting PCRE_NOTBOL in the case of a pattern that begins
1586 nigel 73 with any kind of lookbehind. For example, consider the pattern
1587 nigel 41
1588 nigel 73 \Biss\B
1589 nigel 41
1590 nigel 77 which finds occurrences of "iss" in the middle of words. (\B matches
1591     only if the current position in the subject is not a word boundary.)
1592     When applied to the string "Mississipi" the first call to pcre_exec()
1593     finds the first occurrence. If pcre_exec() is called again with just
1594     the remainder of the subject, namely "issipi", it does not match,
1595 nigel 73 because \B is always false at the start of the subject, which is deemed
1596 nigel 77 to be a word boundary. However, if pcre_exec() is passed the entire
1597 nigel 75 string again, but with startoffset set to 4, it finds the second occur-
1598 nigel 77 rence of "iss" because it is able to look behind the starting point to
1599 nigel 75 discover that it is preceded by a letter.
1600 nigel 41
1601 nigel 77 If a non-zero starting offset is passed when the pattern is anchored,
1602 nigel 75 one attempt to match at the given offset is made. This can only succeed
1603 nigel 77 if the pattern does not require the match to be at the start of the
1604 nigel 75 subject.
1605 nigel 41
1606 nigel 75 How pcre_exec() returns captured substrings
1607    
1608 nigel 77 In general, a pattern matches a certain portion of the subject, and in
1609     addition, further substrings from the subject may be picked out by
1610     parts of the pattern. Following the usage in Jeffrey Friedl's book,
1611     this is called "capturing" in what follows, and the phrase "capturing
1612     subpattern" is used for a fragment of a pattern that picks out a sub-
1613     string. PCRE supports several other kinds of parenthesized subpattern
1614 nigel 73 that do not cause substrings to be captured.
1615 nigel 65
1616 nigel 77 Captured substrings are returned to the caller via a vector of integer
1617     offsets whose address is passed in ovector. The number of elements in
1618     the vector is passed in ovecsize, which must be a non-negative number.
1619 nigel 75 Note: this argument is NOT the size of ovector in bytes.
1620 nigel 41
1621 nigel 77 The first two-thirds of the vector is used to pass back captured sub-
1622     strings, each substring using a pair of integers. The remaining third
1623     of the vector is used as workspace by pcre_exec() while matching cap-
1624     turing subpatterns, and is not available for passing back information.
1625     The length passed in ovecsize should always be a multiple of three. If
1626 nigel 75 it is not, it is rounded down.
1627    
1628 nigel 77 When a match is successful, information about captured substrings is
1629     returned in pairs of integers, starting at the beginning of ovector,
1630     and continuing up to two-thirds of its length at the most. The first
1631 nigel 73 element of a pair is set to the offset of the first character in a sub-
1632 nigel 77 string, and the second is set to the offset of the first character
1633     after the end of a substring. The first pair, ovector[0] and ovec-
1634     tor[1], identify the portion of the subject string matched by the
1635     entire pattern. The next pair is used for the first capturing subpat-
1636     tern, and so on. The value returned by pcre_exec() is the number of
1637     pairs that have been set. If there are no capturing subpatterns, the
1638     return value from a successful match is 1, indicating that just the
1639 nigel 73 first pair of offsets has been set.
1640 nigel 41
1641 nigel 77 Some convenience functions are provided for extracting the captured
1642     substrings as separate strings. These are described in the following
1643 nigel 73 section.
1644 nigel 41
1645 nigel 77 It is possible for an capturing subpattern number n+1 to match some
1646     part of the subject when subpattern n has not been used at all. For
1647 nigel 73 example, if the string "abc" is matched against the pattern (a|(z))(bc)
1648 nigel 77 subpatterns 1 and 3 are matched, but 2 is not. When this happens, both
1649 nigel 73 offset values corresponding to the unused subpattern are set to -1.
1650 nigel 41
1651 nigel 73 If a capturing subpattern is matched repeatedly, it is the last portion
1652 nigel 75 of the string that it matched that is returned.
1653 nigel 41
1654 nigel 77 If the vector is too small to hold all the captured substring offsets,
1655 nigel 75 it is used as far as possible (up to two-thirds of its length), and the
1656 nigel 77 function returns a value of zero. In particular, if the substring off-
1657 nigel 75 sets are not of interest, pcre_exec() may be called with ovector passed
1658 nigel 77 as NULL and ovecsize as zero. However, if the pattern contains back
1659     references and the ovector is not big enough to remember the related
1660     substrings, PCRE has to get additional memory for use during matching.
1661 nigel 73 Thus it is usually advisable to supply an ovector.
1662 nigel 41
1663 nigel 77 Note that pcre_info() can be used to find out how many capturing sub-
1664 nigel 73 patterns there are in a compiled pattern. The smallest size for ovector
1665 nigel 77 that will allow for n captured substrings, in addition to the offsets
1666 nigel 73 of the substring matched by the whole pattern, is (n+1)*3.
1667 nigel 41
1668 nigel 75 Return values from pcre_exec()
1669    
1670 nigel 77 If pcre_exec() fails, it returns a negative number. The following are
1671 nigel 73 defined in the header file:
1672 nigel 41
1673 nigel 73 PCRE_ERROR_NOMATCH (-1)
1674 nigel 41
1675 nigel 73 The subject string did not match the pattern.
1676 nigel 41
1677 nigel 73 PCRE_ERROR_NULL (-2)
1678 nigel 41
1679 nigel 77 Either code or subject was passed as NULL, or ovector was NULL and
1680 nigel 73 ovecsize was not zero.
1681 nigel 41
1682 nigel 73 PCRE_ERROR_BADOPTION (-3)
1683 nigel 41
1684 nigel 73 An unrecognized bit was set in the options argument.
1685 nigel 41
1686 nigel 73 PCRE_ERROR_BADMAGIC (-4)
1687 nigel 41
1688 nigel 77 PCRE stores a 4-byte "magic number" at the start of the compiled code,
1689 nigel 75 to catch the case when it is passed a junk pointer and to detect when a
1690     pattern that was compiled in an environment of one endianness is run in
1691 nigel 77 an environment with the other endianness. This is the error that PCRE
1692 nigel 75 gives when the magic number is not present.
1693 nigel 41
1694 nigel 73 PCRE_ERROR_UNKNOWN_NODE (-5)
1695 nigel 41
1696 nigel 73 While running the pattern match, an unknown item was encountered in the
1697 nigel 77 compiled pattern. This error could be caused by a bug in PCRE or by
1698 nigel 73 overwriting of the compiled pattern.
1699 nigel 41
1700 nigel 73 PCRE_ERROR_NOMEMORY (-6)
1701 nigel 41
1702 nigel 77 If a pattern contains back references, but the ovector that is passed
1703 nigel 73 to pcre_exec() is not big enough to remember the referenced substrings,
1704 nigel 77 PCRE gets a block of memory at the start of matching to use for this
1705     purpose. If the call via pcre_malloc() fails, this error is given. The
1706 nigel 75 memory is automatically freed at the end of matching.
1707 nigel 41
1708 nigel 73 PCRE_ERROR_NOSUBSTRING (-7)
1709 nigel 53
1710 nigel 77 This error is used by the pcre_copy_substring(), pcre_get_substring(),
1711 nigel 73 and pcre_get_substring_list() functions (see below). It is never
1712     returned by pcre_exec().
1713 nigel 63
1714 nigel 73 PCRE_ERROR_MATCHLIMIT (-8)
1715 nigel 63
1716 nigel 77 The recursion and backtracking limit, as specified by the match_limit
1717     field in a pcre_extra structure (or defaulted) was reached. See the
1718 nigel 73 description above.
1719 nigel 63
1720 nigel 73 PCRE_ERROR_CALLOUT (-9)
1721 nigel 63
1722 nigel 73 This error is never generated by pcre_exec() itself. It is provided for
1723 nigel 77 use by callout functions that want to yield a distinctive error code.
1724 nigel 73 See the pcrecallout documentation for details.
1725 nigel 71
1726 nigel 73 PCRE_ERROR_BADUTF8 (-10)
1727 nigel 71
1728 nigel 77 A string that contains an invalid UTF-8 byte sequence was passed as a
1729 nigel 73 subject.
1730    
1731     PCRE_ERROR_BADUTF8_OFFSET (-11)
1732    
1733     The UTF-8 byte sequence that was passed as a subject was valid, but the
1734 nigel 77 value of startoffset did not point to the beginning of a UTF-8 charac-
1735 nigel 73 ter.
1736    
1737 nigel 77 PCRE_ERROR_PARTIAL (-12)
1738 nigel 73
1739 nigel 77 The subject string did not match, but it did match partially. See the
1740 nigel 75 pcrepartial documentation for details of partial matching.
1741    
1742 nigel 77 PCRE_ERROR_BADPARTIAL (-13)
1743 nigel 75
1744 nigel 77 The PCRE_PARTIAL option was used with a compiled pattern containing
1745     items that are not supported for partial matching. See the pcrepartial
1746 nigel 75 documentation for details of partial matching.
1747    
1748 nigel 77 PCRE_ERROR_INTERNAL (-14)
1749 nigel 75
1750 nigel 77 An unexpected internal error has occurred. This error could be caused
1751 nigel 75 by a bug in PCRE or by overwriting of the compiled pattern.
1752    
1753 nigel 77 PCRE_ERROR_BADCOUNT (-15)
1754 nigel 75
1755 nigel 77 This error is given if the value of the ovecsize argument is negative.
1756 nigel 75
1757    
1758 nigel 63 EXTRACTING CAPTURED SUBSTRINGS BY NUMBER
1759    
1760 nigel 73 int pcre_copy_substring(const char *subject, int *ovector,
1761     int stringcount, int stringnumber, char *buffer,
1762     int buffersize);
1763 nigel 63
1764 nigel 73 int pcre_get_substring(const char *subject, int *ovector,
1765     int stringcount, int stringnumber,
1766     const char **stringptr);
1767 nigel 63
1768 nigel 73 int pcre_get_substring_list(const char *subject,
1769     int *ovector, int stringcount, const char ***listptr);
1770 nigel 63
1771 nigel 77 Captured substrings can be accessed directly by using the offsets
1772     returned by pcre_exec() in ovector. For convenience, the functions
1773 nigel 73 pcre_copy_substring(), pcre_get_substring(), and pcre_get_sub-
1774 nigel 77 string_list() are provided for extracting captured substrings as new,
1775     separate, zero-terminated strings. These functions identify substrings
1776     by number. The next section describes functions for extracting named
1777     substrings. A substring that contains a binary zero is correctly
1778     extracted and has a further zero added on the end, but the result is
1779 nigel 73 not, of course, a C string.
1780 nigel 41
1781 nigel 77 The first three arguments are the same for all three of these func-
1782     tions: subject is the subject string that has just been successfully
1783 nigel 73 matched, ovector is a pointer to the vector of integer offsets that was
1784     passed to pcre_exec(), and stringcount is the number of substrings that
1785 nigel 77 were captured by the match, including the substring that matched the
1786 nigel 75 entire regular expression. This is the value returned by pcre_exec() if
1787 nigel 77 it is greater than zero. If pcre_exec() returned zero, indicating that
1788     it ran out of space in ovector, the value passed as stringcount should
1789 nigel 75 be the number of elements in the vector divided by three.
1790 nigel 41
1791 nigel 77 The functions pcre_copy_substring() and pcre_get_substring() extract a
1792     single substring, whose number is given as stringnumber. A value of
1793     zero extracts the substring that matched the entire pattern, whereas
1794     higher values extract the captured substrings. For pcre_copy_sub-
1795     string(), the string is placed in buffer, whose length is given by
1796     buffersize, while for pcre_get_substring() a new block of memory is
1797     obtained via pcre_malloc, and its address is returned via stringptr.
1798     The yield of the function is the length of the string, not including
1799 nigel 73 the terminating zero, or one of
1800 nigel 41
1801 nigel 73 PCRE_ERROR_NOMEMORY (-6)
1802 nigel 41
1803 nigel 77 The buffer was too small for pcre_copy_substring(), or the attempt to
1804 nigel 73 get memory failed for pcre_get_substring().
1805 nigel 41
1806 nigel 73 PCRE_ERROR_NOSUBSTRING (-7)
1807 nigel 41
1808 nigel 73 There is no substring whose number is stringnumber.
1809 nigel 41
1810 nigel 77 The pcre_get_substring_list() function extracts all available sub-
1811     strings and builds a list of pointers to them. All this is done in a
1812 nigel 75 single block of memory that is obtained via pcre_malloc. The address of
1813 nigel 77 the memory block is returned via listptr, which is also the start of
1814     the list of string pointers. The end of the list is marked by a NULL
1815 nigel 73 pointer. The yield of the function is zero if all went well, or
1816 nigel 41
1817 nigel 73 PCRE_ERROR_NOMEMORY (-6)
1818 nigel 41
1819 nigel 73 if the attempt to get the memory block failed.
1820 nigel 41
1821 nigel 77 When any of these functions encounter a substring that is unset, which
1822     can happen when capturing subpattern number n+1 matches some part of
1823     the subject, but subpattern n has not been used at all, they return an
1824 nigel 73 empty string. This can be distinguished from a genuine zero-length sub-
1825 nigel 77 string by inspecting the appropriate offset in ovector, which is nega-
1826 nigel 73 tive for unset substrings.
1827 nigel 41
1828 nigel 77 The two convenience functions pcre_free_substring() and pcre_free_sub-
1829     string_list() can be used to free the memory returned by a previous
1830 nigel 75 call of pcre_get_substring() or pcre_get_substring_list(), respec-
1831 nigel 77 tively. They do nothing more than call the function pointed to by
1832     pcre_free, which of course could be called directly from a C program.
1833     However, PCRE is used in some situations where it is linked via a spe-
1834 nigel 73 cial interface to another programming language which cannot use
1835 nigel 77 pcre_free directly; it is for these cases that the functions are pro-
1836     vided.
1837 nigel 41
1838 nigel 73
1839 nigel 63 EXTRACTING CAPTURED SUBSTRINGS BY NAME
1840 nigel 41
1841 nigel 75 int pcre_get_stringnumber(const pcre *code,
1842     const char *name);
1843    
1844 nigel 73 int pcre_copy_named_substring(const pcre *code,
1845     const char *subject, int *ovector,
1846     int stringcount, const char *stringname,
1847     char *buffer, int buffersize);
1848 nigel 41
1849 nigel 73 int pcre_get_named_substring(const pcre *code,
1850     const char *subject, int *ovector,
1851     int stringcount, const char *stringname,
1852     const char **stringptr);
1853 nigel 41
1854 nigel 77 To extract a substring by name, you first have to find associated num-
1855 nigel 75 ber. For example, for this pattern
1856 nigel 41
1857 nigel 79 (a+)b(?P<xxx>\d+)...
1858 nigel 63
1859 nigel 75 the number of the subpattern called "xxx" is 2. You can find the number
1860     from the name by calling pcre_get_stringnumber(). The first argument is
1861 nigel 77 the compiled pattern, and the second is the name. The yield of the
1862     function is the subpattern number, or PCRE_ERROR_NOSUBSTRING (-7) if
1863 nigel 75 there is no subpattern of that name.
1864 nigel 63
1865 nigel 75 Given the number, you can extract the substring directly, or use one of
1866     the functions described in the previous section. For convenience, there
1867     are also two functions that do the whole job.
1868    
1869 nigel 77 Most of the arguments of pcre_copy_named_substring() and
1870     pcre_get_named_substring() are the same as those for the similarly
1871     named functions that extract by number. As these are described in the
1872     previous section, they are not re-described here. There are just two
1873 nigel 75 differences:
1874 nigel 63
1875 nigel 77 First, instead of a substring number, a substring name is given. Sec-
1876 nigel 73 ond, there is an extra argument, given at the start, which is a pointer
1877 nigel 77 to the compiled pattern. This is needed in order to gain access to the
1878 nigel 73 name-to-number translation table.
1879 nigel 63
1880 nigel 77 These functions call pcre_get_stringnumber(), and if it succeeds, they
1881     then call pcre_copy_substring() or pcre_get_substring(), as appropri-
1882 nigel 73 ate.
1883 nigel 63
1884 nigel 77
1885     FINDING ALL POSSIBLE MATCHES
1886    
1887     The traditional matching function uses a similar algorithm to Perl,
1888     which stops when it finds the first match, starting at a given point in
1889     the subject. If you want to find all possible matches, or the longest
1890     possible match, consider using the alternative matching function (see
1891     below) instead. If you cannot use the alternative function, but still
1892     need to find all possible matches, you can kludge it up by making use
1893     of the callout facility, which is described in the pcrecallout documen-
1894     tation.
1895    
1896     What you have to do is to insert a callout right at the end of the pat-
1897     tern. When your callout function is called, extract and save the cur-
1898     rent matched substring. Then return 1, which forces pcre_exec() to
1899     backtrack and try other alternatives. Ultimately, when it runs out of
1900     matches, pcre_exec() will yield PCRE_ERROR_NOMATCH.
1901    
1902    
1903     MATCHING A PATTERN: THE ALTERNATIVE FUNCTION
1904    
1905     int pcre_dfa_exec(const pcre *code, const pcre_extra *extra,
1906     const char *subject, int length, int startoffset,
1907     int options, int *ovector, int ovecsize,
1908     int *workspace, int wscount);
1909    
1910     The function pcre_dfa_exec() is called to match a subject string
1911     against a compiled pattern, using a "DFA" matching algorithm. This has
1912     different characteristics to the normal algorithm, and is not compati-
1913     ble with Perl. Some of the features of PCRE patterns are not supported.
1914     Nevertheless, there are times when this kind of matching can be useful.
1915     For a discussion of the two matching algorithms, see the pcrematching
1916     documentation.
1917    
1918     The arguments for the pcre_dfa_exec() function are the same as for
1919     pcre_exec(), plus two extras. The ovector argument is used in a differ-
1920     ent way, and this is described below. The other common arguments are
1921     used in the same way as for pcre_exec(), so their description is not
1922     repeated here.
1923    
1924     The two additional arguments provide workspace for the function. The
1925     workspace vector should contain at least 20 elements. It is used for
1926     keeping track of multiple paths through the pattern tree. More
1927     workspace will be needed for patterns and subjects where there are a
1928     lot of possible matches.
1929    
1930     Here is an example of a simple call to pcre_exec():
1931    
1932     int rc;
1933     int ovector[10];
1934     int wspace[20];
1935     rc = pcre_exec(
1936     re, /* result of pcre_compile() */
1937     NULL, /* we didn't study the pattern */
1938     "some string", /* the subject string */
1939     11, /* the length of the subject string */
1940     0, /* start at offset 0 in the subject */
1941     0, /* default options */
1942     ovector, /* vector of integers for substring information */
1943     10, /* number of elements (NOT size in bytes) */
1944     wspace, /* working space vector */
1945     20); /* number of elements (NOT size in bytes) */
1946    
1947     Option bits for pcre_dfa_exec()
1948    
1949     The unused bits of the options argument for pcre_dfa_exec() must be
1950     zero. The only bits that may be set are PCRE_ANCHORED, PCRE_NOTBOL,
1951     PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK, PCRE_PARTIAL,
1952     PCRE_DFA_SHORTEST, and PCRE_DFA_RESTART. All but the last three of
1953     these are the same as for pcre_exec(), so their description is not
1954     repeated here.
1955    
1956     PCRE_PARTIAL
1957    
1958     This has the same general effect as it does for pcre_exec(), but the
1959     details are slightly different. When PCRE_PARTIAL is set for
1960     pcre_dfa_exec(), the return code PCRE_ERROR_NOMATCH is converted into
1961     PCRE_ERROR_PARTIAL if the end of the subject is reached, there have
1962     been no complete matches, but there is still at least one matching pos-
1963     sibility. The portion of the string that provided the partial match is
1964     set as the first matching string.
1965    
1966     PCRE_DFA_SHORTEST
1967    
1968     Setting the PCRE_DFA_SHORTEST option causes the matching algorithm to
1969     stop as soon as it has found one match. Because of the way the DFA
1970     algorithm works, this is necessarily the shortest possible match at the
1971     first possible matching point in the subject string.
1972    
1973     PCRE_DFA_RESTART
1974    
1975     When pcre_dfa_exec() is called with the PCRE_PARTIAL option, and
1976     returns a partial match, it is possible to call it again, with addi-
1977     tional subject characters, and have it continue with the same match.
1978     The PCRE_DFA_RESTART option requests this action; when it is set, the
1979     workspace and wscount options must reference the same vector as before
1980     because data about the match so far is left in them after a partial
1981     match. There is more discussion of this facility in the pcrepartial
1982     documentation.
1983    
1984     Successful returns from pcre_dfa_exec()
1985    
1986     When pcre_dfa_exec() succeeds, it may have matched more than one sub-
1987     string in the subject. Note, however, that all the matches from one run
1988     of the function start at the same point in the subject. The shorter
1989     matches are all initial substrings of the longer matches. For example,
1990     if the pattern
1991    
1992     <.*>
1993    
1994     is matched against the string
1995    
1996     This is <something> <something else> <something further> no more
1997    
1998     the three matched strings are
1999    
2000     <something>
2001     <something> <something else>
2002     <something> <something else> <something further>
2003    
2004     On success, the yield of the function is a number greater than zero,
2005     which is the number of matched substrings. The substrings themselves
2006     are returned in ovector. Each string uses two elements; the first is
2007     the offset to the start, and the second is the offset to the end. All
2008     the strings have the same start offset. (Space could have been saved by
2009     giving this only once, but it was decided to retain some compatibility
2010     with the way pcre_exec() returns data, even though the meaning of the
2011     strings is different.)
2012    
2013     The strings are returned in reverse order of length; that is, the long-
2014     est matching string is given first. If there were too many matches to
2015     fit into ovector, the yield of the function is zero, and the vector is
2016     filled with the longest matches.
2017    
2018     Error returns from pcre_dfa_exec()
2019    
2020     The pcre_dfa_exec() function returns a negative number when it fails.
2021     Many of the errors are the same as for pcre_exec(), and these are
2022     described above. There are in addition the following errors that are
2023     specific to pcre_dfa_exec():
2024    
2025     PCRE_ERROR_DFA_UITEM (-16)
2026    
2027     This return is given if pcre_dfa_exec() encounters an item in the pat-
2028     tern that it does not support, for instance, the use of \C or a back
2029     reference.
2030    
2031     PCRE_ERROR_DFA_UCOND (-17)
2032    
2033     This return is given if pcre_dfa_exec() encounters a condition item in
2034     a pattern that uses a back reference for the condition. This is not
2035     supported.
2036    
2037     PCRE_ERROR_DFA_UMLIMIT (-18)
2038    
2039     This return is given if pcre_dfa_exec() is called with an extra block
2040     that contains a setting of the match_limit field. This is not supported
2041     (it is meaningless).
2042    
2043     PCRE_ERROR_DFA_WSSIZE (-19)
2044    
2045     This return is given if pcre_dfa_exec() runs out of space in the
2046     workspace vector.
2047    
2048     PCRE_ERROR_DFA_RECURSE (-20)
2049    
2050     When a recursive subpattern is processed, the matching function calls
2051     itself recursively, using private vectors for ovector and workspace.
2052     This error is given if the output vector is not large enough. This
2053     should be extremely rare, as a vector of size 1000 is used.
2054    
2055     Last updated: 16 May 2005
2056     Copyright (c) 1997-2005 University of Cambridge.
2057 nigel 79 ------------------------------------------------------------------------------
2058 nigel 63
2059    
2060 nigel 79 PCRECALLOUT(3) PCRECALLOUT(3)
2061 nigel 63
2062 nigel 79
2063 nigel 73 NAME
2064     PCRE - Perl-compatible regular expressions
2065    
2066 nigel 77
2067 nigel 63 PCRE CALLOUTS
2068    
2069 nigel 73 int (*pcre_callout)(pcre_callout_block *);
2070 nigel 63
2071 nigel 73 PCRE provides a feature called "callout", which is a means of temporar-
2072     ily passing control to the caller of PCRE in the middle of pattern
2073     matching. The caller of PCRE provides an external function by putting
2074     its entry point in the global variable pcre_callout. By default, this
2075     variable contains NULL, which disables all calling out.
2076 nigel 63
2077 nigel 73 Within a regular expression, (?C) indicates the points at which the
2078     external function is to be called. Different callout points can be
2079     identified by putting a number less than 256 after the letter C. The
2080     default value is zero. For example, this pattern has two callout
2081     points:
2082 nigel 63
2083 nigel 75 (?C1)eabc(?C2)def
2084 nigel 63
2085 nigel 75 If the PCRE_AUTO_CALLOUT option bit is set when pcre_compile() is
2086     called, PCRE automatically inserts callouts, all with number 255,
2087     before each item in the pattern. For example, if PCRE_AUTO_CALLOUT is
2088     used with the pattern
2089 nigel 63
2090 nigel 75 A(\d{2}|--)
2091    
2092     it is processed as if it were
2093    
2094     (?C255)A(?C255)((?C255)\d{2}(?C255)|(?C255)-(?C255)-(?C255))(?C255)
2095    
2096     Notice that there is a callout before and after each parenthesis and
2097     alternation bar. Automatic callouts can be used for tracking the
2098     progress of pattern matching. The pcretest command has an option that
2099     sets automatic callouts; when it is used, the output indicates how the
2100     pattern is matched. This is useful information when you are trying to
2101     optimize the performance of a particular pattern.
2102    
2103    
2104     MISSING CALLOUTS
2105    
2106     You should be aware that, because of optimizations in the way PCRE
2107     matches patterns, callouts sometimes do not happen. For example, if the
2108     pattern is
2109    
2110     ab(?C4)cd
2111    
2112     PCRE knows that any matching string must contain the letter "d". If the
2113     subject string is "abyz", the lack of "d" means that matching doesn't
2114     ever start, and the callout is never reached. However, with "abyd",
2115     though the result is still no match, the callout is obeyed.
2116    
2117    
2118     THE CALLOUT INTERFACE
2119    
2120     During matching, when PCRE reaches a callout point, the external func-
2121 nigel 77 tion defined by pcre_callout is called (if it is set). This applies to
2122     both the pcre_exec() and the pcre_dfa_exec() matching functions. The
2123     only argument to the callout function is a pointer to a pcre_callout
2124     block. This structure contains the following fields:
2125 nigel 75
2126 nigel 73 int version;
2127     int callout_number;
2128     int *offset_vector;
2129     const char *subject;
2130     int subject_length;
2131     int start_match;
2132     int current_position;
2133     int capture_top;
2134     int capture_last;
2135     void *callout_data;
2136 nigel 75 int pattern_position;
2137     int next_item_length;
2138 nigel 63
2139 nigel 77 The version field is an integer containing the version number of the
2140     block format. The initial version was 0; the current version is 1. The
2141     version number will change again in future if additional fields are
2142 nigel 75 added, but the intention is never to remove any of the existing fields.
2143 nigel 63
2144 nigel 77 The callout_number field contains the number of the callout, as com-
2145     piled into the pattern (that is, the number after ?C for manual call-
2146 nigel 75 outs, and 255 for automatically generated callouts).
2147 nigel 63
2148 nigel 77 The offset_vector field is a pointer to the vector of offsets that was
2149     passed by the caller to pcre_exec() or pcre_dfa_exec(). When
2150     pcre_exec() is used, the contents can be inspected in order to extract
2151     substrings that have been matched so far, in the same way as for
2152     extracting substrings after a match has completed. For pcre_dfa_exec()
2153     this field is not useful.
2154 nigel 63
2155 nigel 75 The subject and subject_length fields contain copies of the values that
2156 nigel 73 were passed to pcre_exec().
2157 nigel 63
2158 nigel 77 The start_match field contains the offset within the subject at which
2159     the current match attempt started. If the pattern is not anchored, the
2160 nigel 75 callout function may be called several times from the same point in the
2161     pattern for different starting points in the subject.
2162 nigel 63
2163 nigel 77 The current_position field contains the offset within the subject of
2164 nigel 73 the current match pointer.
2165 nigel 63
2166 nigel 77 When the pcre_exec() function is used, the capture_top field contains
2167     one more than the number of the highest numbered captured substring so
2168     far. If no substrings have been captured, the value of capture_top is
2169     one. This is always the case when pcre_dfa_exec() is used, because it
2170     does not support captured substrings.
2171 nigel 63
2172 nigel 77 The capture_last field contains the number of the most recently cap-
2173     tured substring. If no substrings have been captured, its value is -1.
2174     This is always the case when pcre_dfa_exec() is used.
2175 nigel 63
2176 nigel 77 The callout_data field contains a value that is passed to pcre_exec()
2177     or pcre_dfa_exec() specifically so that it can be passed back in call-
2178     outs. It is passed in the pcre_callout field of the pcre_extra data
2179     structure. If no such data was passed, the value of callout_data in a
2180     pcre_callout block is NULL. There is a description of the pcre_extra
2181 nigel 73 structure in the pcreapi documentation.
2182 nigel 63
2183 nigel 77 The pattern_position field is present from version 1 of the pcre_call-
2184 nigel 75 out structure. It contains the offset to the next item to be matched in
2185     the pattern string.
2186 nigel 63
2187 nigel 77 The next_item_length field is present from version 1 of the pcre_call-
2188 nigel 75 out structure. It contains the length of the next item to be matched in
2189 nigel 77 the pattern string. When the callout immediately precedes an alterna-
2190     tion bar, a closing parenthesis, or the end of the pattern, the length
2191     is zero. When the callout precedes an opening parenthesis, the length
2192 nigel 75 is that of the entire subpattern.
2193 nigel 73
2194 nigel 77 The pattern_position and next_item_length fields are intended to help
2195     in distinguishing between different automatic callouts, which all have
2196 nigel 75 the same callout number. However, they are set for all callouts.
2197    
2198    
2199 nigel 63 RETURN VALUES
2200    
2201 nigel 77 The external callout function returns an integer to PCRE. If the value
2202     is zero, matching proceeds as normal. If the value is greater than
2203     zero, matching fails at the current point, but the testing of other
2204     matching possibilities goes ahead, just as if a lookahead assertion had
2205     failed. If the value is less than zero, the match is abandoned, and
2206     pcre_exec() (or pcre_dfa_exec()) returns the negative value.
2207 nigel 63
2208 nigel 77 Negative values should normally be chosen from the set of
2209 nigel 73 PCRE_ERROR_xxx values. In particular, PCRE_ERROR_NOMATCH forces a stan-
2210 nigel 77 dard "no match" failure. The error number PCRE_ERROR_CALLOUT is
2211     reserved for use by callout functions; it will never be used by PCRE
2212 nigel 73 itself.
2213 nigel 63
2214 nigel 77 Last updated: 28 February 2005
2215     Copyright (c) 1997-2005 University of Cambridge.
2216 nigel 79 ------------------------------------------------------------------------------
2217 nigel 63
2218    
2219 nigel 79 PCRECOMPAT(3) PCRECOMPAT(3)
2220 nigel 63
2221 nigel 79
2222 nigel 73 NAME
2223     PCRE - Perl-compatible regular expressions
2224    
2225 nigel 77
2226 nigel 75 DIFFERENCES BETWEEN PCRE AND PERL
2227 nigel 41
2228 nigel 73 This document describes the differences in the ways that PCRE and Perl
2229     handle regular expressions. The differences described here are with
2230     respect to Perl 5.8.
2231 nigel 41
2232 nigel 73 1. PCRE does not have full UTF-8 support. Details of what it does have
2233     are given in the section on UTF-8 support in the main pcre page.
2234 nigel 41
2235 nigel 73 2. PCRE does not allow repeat quantifiers on lookahead assertions. Perl
2236     permits them, but they do not mean what you might think. For example,
2237     (?!a){3} does not assert that the next three characters are not "a". It
2238     just asserts that the next character is not "a" three times.
2239 nigel 41
2240 nigel 73 3. Capturing subpatterns that occur inside negative lookahead asser-
2241     tions are counted, but their entries in the offsets vector are never
2242     set. Perl sets its numerical variables from any such patterns that are
2243     matched before the assertion fails to match something (thereby succeed-
2244     ing), but only if the negative lookahead assertion contains just one
2245     branch.
2246 nigel 41
2247 nigel 73 4. Though binary zero characters are supported in the subject string,
2248     they are not allowed in a pattern string because it is passed as a nor-
2249 nigel 75 mal C string, terminated by zero. The escape sequence \0 can be used in
2250     the pattern to represent a binary zero.
2251 nigel 41
2252 nigel 73 5. The following Perl escape sequences are not supported: \l, \u, \L,
2253 nigel 75 \U, and \N. In fact these are implemented by Perl's general string-han-
2254     dling and are not part of its pattern matching engine. If any of these
2255     are encountered by PCRE, an error is generated.
2256 nigel 41
2257 nigel 75 6. The Perl escape sequences \p, \P, and \X are supported only if PCRE
2258     is built with Unicode character property support. The properties that
2259     can be tested with \p and \P are limited to the general category prop-
2260     erties such as Lu and Nd.
2261    
2262     7. PCRE does support the \Q...\E escape for quoting substrings. Charac-
2263     ters in between are treated as literals. This is slightly different
2264     from Perl in that $ and @ are also handled as literals inside the
2265     quotes. In Perl, they cause variable interpolation (but of course PCRE
2266 nigel 73 does not have variables). Note the following examples:
2267 nigel 49
2268 nigel 73 Pattern PCRE matches Perl matches
2269 nigel 41
2270 nigel 73 \Qabc$xyz\E abc$xyz abc followed by the
2271     contents of $xyz
2272     \Qabc\$xyz\E abc\$xyz abc\$xyz
2273     \Qabc\E\$\Qxyz\E abc$xyz abc$xyz
2274 nigel 41
2275 nigel 75 The \Q...\E sequence is recognized both inside and outside character
2276 nigel 73 classes.
2277 nigel 41
2278 nigel 75 8. Fairly obviously, PCRE does not support the (?{code}) and (?p{code})
2279     constructions. However, there is support for recursive patterns using
2280     the non-Perl items (?R), (?number), and (?P>name). Also, the PCRE
2281     "callout" feature allows an external function to be called during pat-
2282     tern matching. See the pcrecallout documentation for details.
2283 nigel 63
2284 nigel 75 9. There are some differences that are concerned with the settings of
2285     captured strings when part of a pattern is repeated. For example,
2286     matching "aba" against the pattern /^(a(b)?)+$/ in Perl leaves $2
2287 nigel 73 unset, but in PCRE it is set to "b".
2288 nigel 41
2289 nigel 75 10. PCRE provides some extensions to the Perl regular expression facil-
2290     ities:
2291 nigel 41
2292 nigel 75 (a) Although lookbehind assertions must match fixed length strings,
2293 nigel 73 each alternative branch of a lookbehind assertion can match a different
2294     length of string. Perl requires them all to have the same length.
2295 nigel 41
2296 nigel 75 (b) If PCRE_DOLLAR_ENDONLY is set and PCRE_MULTILINE is not set, the $
2297 nigel 73 meta-character matches only at the very end of the string.
2298 nigel 41
2299 nigel 73 (c) If PCRE_EXTRA is set, a backslash followed by a letter with no spe-
2300     cial meaning is faulted.
2301 nigel 41
2302 nigel 75 (d) If PCRE_UNGREEDY is set, the greediness of the repetition quanti-
2303 nigel 73 fiers is inverted, that is, by default they are not greedy, but if fol-
2304     lowed by a question mark they are.
2305 nigel 41
2306 nigel 75 (e) PCRE_ANCHORED can be used at matching time to force a pattern to be
2307     tried only at the first matching position in the subject string.
2308 nigel 41
2309 nigel 75 (f) The PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and PCRE_NO_AUTO_CAP-
2310 nigel 73 TURE options for pcre_exec() have no Perl equivalents.
2311 nigel 41
2312 nigel 75 (g) The (?R), (?number), and (?P>name) constructs allows for recursive
2313     pattern matching (Perl can do this using the (?p{code}) construct,
2314 nigel 73 which PCRE cannot support.)
2315 nigel 41
2316 nigel 75 (h) PCRE supports named capturing substrings, using the Python syntax.
2317 nigel 43
2318 nigel 75 (i) PCRE supports the possessive quantifier "++" syntax, taken from
2319 nigel 73 Sun's Java package.
2320 nigel 63
2321 nigel 73 (j) The (R) condition, for testing recursion, is a PCRE extension.
2322 nigel 63
2323 nigel 73 (k) The callout facility is PCRE-specific.
2324    
2325 nigel 75 (l) The partial matching facility is PCRE-specific.
2326    
2327     (m) Patterns compiled by PCRE can be saved and re-used at a later time,
2328     even on different hosts that have the other endianness.
2329    
2330 nigel 77 (n) The alternative matching function (pcre_dfa_exec()) matches in a
2331     different way and is not Perl-compatible.
2332    
2333     Last updated: 28 February 2005
2334     Copyright (c) 1997-2005 University of Cambridge.
2335 nigel 79 ------------------------------------------------------------------------------
2336 nigel 63
2337    
2338 nigel 79 PCREPATTERN(3) PCREPATTERN(3)
2339 nigel 63
2340 nigel 79
2341 nigel 73 NAME
2342     PCRE - Perl-compatible regular expressions
2343    
2344 nigel 77
2345 nigel 63 PCRE REGULAR EXPRESSION DETAILS
2346    
2347 nigel 73 The syntax and semantics of the regular expressions supported by PCRE
2348     are described below. Regular expressions are also described in the Perl
2349 nigel 75 documentation and in a number of books, some of which have copious
2350     examples. Jeffrey Friedl's "Mastering Regular Expressions", published
2351     by O'Reilly, covers regular expressions in great detail. This descrip-
2352     tion of PCRE's regular expressions is intended as reference material.
2353 nigel 49
2354 nigel 75 The original operation of PCRE was on strings of one-byte characters.
2355     However, there is now also support for UTF-8 character strings. To use
2356     this, you must build PCRE to include UTF-8 support, and then call
2357     pcre_compile() with the PCRE_UTF8 option. How this affects pattern
2358     matching is mentioned in several places below. There is also a summary
2359     of UTF-8 features in the section on UTF-8 support in the main pcre
2360     page.
2361 nigel 41
2362 nigel 77 The remainder of this document discusses the patterns that are sup-
2363     ported by PCRE when its main matching function, pcre_exec(), is used.
2364     From release 6.0, PCRE offers a second matching function,
2365     pcre_dfa_exec(), which matches using a different algorithm that is not
2366     Perl-compatible. The advantages and disadvantages of the alternative
2367     function, and how it differs from the normal function, are discussed in
2368     the pcrematching page.
2369    
2370 nigel 75 A regular expression is a pattern that is matched against a subject
2371     string from left to right. Most characters stand for themselves in a
2372     pattern, and match the corresponding characters in the subject. As a
2373 nigel 73 trivial example, the pattern
2374 nigel 41
2375 nigel 73 The quick brown fox
2376 nigel 41
2377 nigel 77 matches a portion of a subject string that is identical to itself. When
2378     caseless matching is specified (the PCRE_CASELESS option), letters are
2379     matched independently of case. In UTF-8 mode, PCRE always understands
2380     the concept of case for characters whose values are less than 128, so
2381     caseless matching is always possible. For characters with higher val-
2382     ues, the concept of case is supported if PCRE is compiled with Unicode
2383     property support, but not otherwise. If you want to use caseless
2384     matching for characters 128 and above, you must ensure that PCRE is
2385     compiled with Unicode property support as well as with UTF-8 support.
2386 nigel 41
2387 nigel 77 The power of regular expressions comes from the ability to include
2388     alternatives and repetitions in the pattern. These are encoded in the
2389     pattern by the use of metacharacters, which do not stand for themselves
2390     but instead are interpreted in some special way.
2391    
2392     There are two different sets of metacharacters: those that are recog-
2393     nized anywhere in the pattern except within square brackets, and those
2394     that are recognized in square brackets. Outside square brackets, the
2395 nigel 75 metacharacters are as follows:
2396 nigel 41
2397 nigel 73 \ general escape character with several uses
2398     ^ assert start of string (or line, in multiline mode)
2399     $ assert end of string (or line, in multiline mode)
2400     . match any character except newline (by default)
2401     [ start character class definition
2402     | start of alternative branch
2403     ( start subpattern
2404     ) end subpattern
2405     ? extends the meaning of (
2406     also 0 or 1 quantifier
2407     also quantifier minimizer
2408     * 0 or more quantifier
2409     + 1 or more quantifier
2410     also "possessive quantifier"
2411     { start min/max quantifier
2412 nigel 41
2413 nigel 77 Part of a pattern that is in square brackets is called a "character
2414 nigel 75 class". In a character class the only metacharacters are:
2415 nigel 41
2416 nigel 73 \ general escape character
2417     ^ negate the class, but only if the first character
2418     - indicates character range
2419     [ POSIX character class (only if followed by POSIX
2420     syntax)
2421     ] terminates the character class
2422 nigel 41
2423 nigel 77 The following sections describe the use of each of the metacharacters.
2424 nigel 41
2425    
2426 nigel 63 BACKSLASH
2427 nigel 41
2428 nigel 73 The backslash character has several uses. Firstly, if it is followed by
2429 nigel 77 a non-alphanumeric character, it takes away any special meaning that
2430     character may have. This use of backslash as an escape character
2431 nigel 73 applies both inside and outside character classes.
2432 nigel 41
2433 nigel 77 For example, if you want to match a * character, you write \* in the
2434     pattern. This escaping action applies whether or not the following
2435     character would otherwise be interpreted as a metacharacter, so it is
2436     always safe to precede a non-alphanumeric with backslash to specify
2437     that it stands for itself. In particular, if you want to match a back-
2438 nigel 75 slash, you write \\.
2439 nigel 41
2440 nigel 77 If a pattern is compiled with the PCRE_EXTENDED option, whitespace in
2441     the pattern (other than in a character class) and characters between a
2442 nigel 73 # outside a character class and the next newline character are ignored.
2443 nigel 77 An escaping backslash can be used to include a whitespace or # charac-
2444 nigel 73 ter as part of the pattern.
2445 nigel 41
2446 nigel 77 If you want to remove the special meaning from a sequence of charac-
2447     ters, you can do so by putting them between \Q and \E. This is differ-
2448     ent from Perl in that $ and @ are handled as literals in \Q...\E
2449     sequences in PCRE, whereas in Perl, $ and @ cause variable interpola-
2450 nigel 73 tion. Note the following examples:
2451 nigel 63
2452 nigel 73 Pattern PCRE matches Perl matches
2453 nigel 63
2454 nigel 73 \Qabc$xyz\E abc$xyz abc followed by the
2455     contents of $xyz
2456     \Qabc\$xyz\E abc\$xyz abc\$xyz
2457     \Qabc\E\$\Qxyz\E abc$xyz abc$xyz
2458 nigel 63
2459 nigel 77 The \Q...\E sequence is recognized both inside and outside character
2460 nigel 73 classes.
2461 nigel 63
2462 nigel 75 Non-printing characters
2463    
2464 nigel 73 A second use of backslash provides a way of encoding non-printing char-
2465 nigel 77 acters in patterns in a visible manner. There is no restriction on the
2466     appearance of non-printing characters, apart from the binary zero that
2467     terminates a pattern, but when a pattern is being prepared by text
2468     editing, it is usually easier to use one of the following escape
2469 nigel 73 sequences than the binary character it represents:
2470 nigel 63
2471 nigel 73 \a alarm, that is, the BEL character (hex 07)
2472     \cx "control-x", where x is any character
2473     \e escape (hex 1B)
2474     \f formfeed (hex 0C)
2475     \n newline (hex 0A)
2476     \r carriage return (hex 0D)
2477     \t tab (hex 09)
2478     \ddd character with octal code ddd, or backreference
2479     \xhh character with hex code hh
2480     \x{hhh..} character with hex code hhh... (UTF-8 mode only)
2481 nigel 41
2482 nigel 77 The precise effect of \cx is as follows: if x is a lower case letter,
2483     it is converted to upper case. Then bit 6 of the character (hex 40) is
2484     inverted. Thus \cz becomes hex 1A, but \c{ becomes hex 3B, while \c;
2485 nigel 73 becomes hex 7B.
2486 nigel 41
2487 nigel 77 After \x, from zero to two hexadecimal digits are read (letters can be
2488     in upper or lower case). In UTF-8 mode, any number of hexadecimal dig-
2489     its may appear between \x{ and }, but the value of the character code
2490     must be less than 2**31 (that is, the maximum hexadecimal value is
2491     7FFFFFFF). If characters other than hexadecimal digits appear between
2492     \x{ and }, or if there is no terminating }, this form of escape is not
2493     recognized. Instead, the initial \x will be interpreted as a basic
2494     hexadecimal escape, with no following digits, giving a character whose
2495 nigel 75 value is zero.
2496 nigel 41
2497 nigel 73 Characters whose value is less than 256 can be defined by either of the
2498 nigel 77 two syntaxes for \x when PCRE is in UTF-8 mode. There is no difference
2499     in the way they are handled. For example, \xdc is exactly the same as
2500 nigel 73 \x{dc}.
2501 nigel 41
2502 nigel 77 After \0 up to two further octal digits are read. In both cases, if
2503     there are fewer than two digits, just those that are present are used.
2504     Thus the sequence \0\x\07 specifies two binary zeros followed by a BEL
2505     character (code value 7). Make sure you supply two digits after the
2506     initial zero if the pattern character that follows is itself an octal
2507 nigel 75 digit.
2508 nigel 63
2509 nigel 73 The handling of a backslash followed by a digit other than 0 is compli-
2510     cated. Outside a character class, PCRE reads it and any following dig-
2511 nigel 77 its as a decimal number. If the number is less than 10, or if there
2512 nigel 73 have been at least that many previous capturing left parentheses in the
2513 nigel 77 expression, the entire sequence is taken as a back reference. A
2514     description of how this works is given later, following the discussion
2515 nigel 73 of parenthesized subpatterns.
2516 nigel 41
2517 nigel 77 Inside a character class, or if the decimal number is greater than 9
2518     and there have not been that many capturing subpatterns, PCRE re-reads
2519     up to three octal digits following the backslash, and generates a sin-
2520 nigel 73 gle byte from the least significant 8 bits of the value. Any subsequent
2521     digits stand for themselves. For example:
2522 nigel 41
2523 nigel 73 \040 is another way of writing a space
2524     \40 is the same, provided there are fewer than 40
2525     previous capturing subpatterns
2526     \7 is always a back reference
2527     \11 might be a back reference, or another way of
2528     writing a tab
2529     \011 is always a tab
2530     \0113 is a tab followed by the character "3"
2531     \113 might be a back reference, otherwise the
2532     character with octal code 113
2533     \377 might be a back reference, otherwise
2534     the byte consisting entirely of 1 bits
2535     \81 is either a back reference, or a binary zero
2536     followed by the two characters "8" and "1"
2537 nigel 41
2538 nigel 77 Note that octal values of 100 or greater must not be introduced by a
2539 nigel 73 leading zero, because no more than three octal digits are ever read.
2540 nigel 41
2541 nigel 77 All the sequences that define a single byte value or a single UTF-8
2542 nigel 73 character (in UTF-8 mode) can be used both inside and outside character
2543 nigel 77 classes. In addition, inside a character class, the sequence \b is
2544 nigel 75 interpreted as the backspace character (hex 08), and the sequence \X is
2545 nigel 77 interpreted as the character "X". Outside a character class, these
2546 nigel 75 sequences have different meanings (see below).
2547 nigel 43
2548 nigel 75 Generic character types
2549 nigel 41
2550 nigel 77 The third use of backslash is for specifying generic character types.
2551 nigel 75 The following are always recognized:
2552    
2553 nigel 73 \d any decimal digit
2554     \D any character that is not a decimal digit
2555     \s any whitespace character
2556     \S any character that is not a whitespace character
2557     \w any "word" character
2558     \W any "non-word" character
2559 nigel 41
2560 nigel 73 Each pair of escape sequences partitions the complete set of characters
2561 nigel 77 into two disjoint sets. Any given character matches one, and only one,
2562 nigel 73 of each pair.
2563 nigel 41
2564 nigel 75 These character type sequences can appear both inside and outside char-
2565 nigel 77 acter classes. They each match one character of the appropriate type.
2566     If the current matching point is at the end of the subject string, all
2567 nigel 75 of them fail, since there is no character to match.
2568 nigel 41
2569 nigel 77 For compatibility with Perl, \s does not match the VT character (code
2570     11). This makes it different from the the POSIX "space" class. The \s
2571 nigel 73 characters are HT (9), LF (10), FF (12), CR (13), and space (32).
2572 nigel 63
2573 nigel 75 A "word" character is an underscore or any character less than 256 that
2574 nigel 77 is a letter or digit. The definition of letters and digits is con-
2575     trolled by PCRE's low-valued character tables, and may vary if locale-
2576     specific matching is taking place (see "Locale support" in the pcreapi
2577     page). For example, in the "fr_FR" (French) locale, some character
2578     codes greater than 128 are used for accented letters, and these are
2579 nigel 75 matched by \w.
2580 nigel 63
2581 nigel 77 In UTF-8 mode, characters with values greater than 128 never match \d,
2582 nigel 75 \s, or \w, and always match \D, \S, and \W. This is true even when Uni-
2583     code character property support is available.
2584 nigel 41
2585 nigel 75 Unicode character properties
2586    
2587     When PCRE is built with Unicode character property support, three addi-
2588 nigel 77 tional escape sequences to match generic character types are available
2589 nigel 75 when UTF-8 mode is selected. They are:
2590    
2591     \p{xx} a character with the xx property
2592     \P{xx} a character without the xx property
2593     \X an extended Unicode sequence
2594    
2595 nigel 77 The property names represented by xx above are limited to the Unicode
2596     general category properties. Each character has exactly one such prop-
2597     erty, specified by a two-letter abbreviation. For compatibility with
2598     Perl, negation can be specified by including a circumflex between the
2599     opening brace and the property name. For example, \p{^Lu} is the same
2600 nigel 75 as \P{Lu}.
2601    
2602 nigel 77 If only one letter is specified with \p or \P, it includes all the
2603 nigel 75 properties that start with that letter. In this case, in the absence of
2604     negation, the curly brackets in the escape sequence are optional; these
2605     two examples have the same effect:
2606    
2607     \p{L}
2608     \pL
2609    
2610     The following property codes are supported:
2611    
2612     C Other
2613     Cc Control
2614     Cf Format
2615     Cn Unassigned
2616     Co Private use
2617     Cs Surrogate
2618    
2619     L Letter
2620     Ll Lower case letter
2621     Lm Modifier letter
2622     Lo Other letter
2623     Lt Title case letter
2624     Lu Upper case letter
2625    
2626     M Mark
2627     Mc Spacing mark
2628     Me Enclosing mark
2629     Mn Non-spacing mark
2630    
2631     N Number
2632     Nd Decimal number
2633     Nl Letter number
2634     No Other number
2635    
2636     P Punctuation
2637     Pc Connector punctuation
2638     Pd Dash punctuation
2639     Pe Close punctuation
2640     Pf Final punctuation
2641     Pi Initial punctuation
2642     Po Other punctuation
2643     Ps Open punctuation
2644    
2645     S Symbol
2646     Sc Currency symbol
2647     Sk Modifier symbol
2648     Sm Mathematical symbol
2649     So Other symbol
2650    
2651     Z Separator
2652     Zl Line separator
2653     Zp Paragraph separator
2654     Zs Space separator
2655    
2656 nigel 77 Extended properties such as "Greek" or "InMusicalSymbols" are not sup-
2657 nigel 75 ported by PCRE.
2658    
2659 nigel 77 Specifying caseless matching does not affect these escape sequences.
2660 nigel 75 For example, \p{Lu} always matches only upper case letters.
2661    
2662 nigel 77 The \X escape matches any number of Unicode characters that form an
2663 nigel 75 extended Unicode sequence. \X is equivalent to
2664    
2665     (?>\PM\pM*)
2666    
2667 nigel 77 That is, it matches a character without the "mark" property, followed
2668     by zero or more characters with the "mark" property, and treats the
2669     sequence as an atomic group (see below). Characters with the "mark"
2670 nigel 75 property are typically accents that affect the preceding character.
2671    
2672 nigel 77 Matching characters by Unicode property is not fast, because PCRE has
2673     to search a structure that contains data for over fifteen thousand
2674 nigel 75 characters. That is why the traditional escape sequences such as \d and
2675     \w do not use Unicode properties in PCRE.
2676    
2677     Simple assertions
2678    
2679 nigel 73 The fourth use of backslash is for certain simple assertions. An asser-
2680 nigel 77 tion specifies a condition that has to be met at a particular point in
2681     a match, without consuming any characters from the subject string. The
2682     use of subpatterns for more complicated assertions is described below.
2683 nigel 75 The backslashed assertions are:
2684 nigel 41
2685 nigel 73 \b matches at a word boundary
2686     \B matches when not at a word boundary
2687     \A matches at start of subject
2688     \Z matches at end of subject or before newline at end
2689     \z matches at end of subject
2690     \G matches at first matching position in subject
2691 nigel 41
2692 nigel 77 These assertions may not appear in character classes (but note that \b
2693 nigel 73 has a different meaning, namely the backspace character, inside a char-
2694     acter class).
2695 nigel 41
2696 nigel 77 A word boundary is a position in the subject string where the current
2697     character and the previous character do not both match \w or \W (i.e.
2698     one matches \w and the other matches \W), or the start or end of the
2699 nigel 73 string if the first or last character matches \w, respectively.
2700 nigel 43
2701 nigel 77 The \A, \Z, and \z assertions differ from the traditional circumflex
2702 nigel 75 and dollar (described in the next section) in that they only ever match
2703 nigel 77 at the very start and end of the subject string, whatever options are
2704     set. Thus, they are independent of multiline mode. These three asser-
2705 nigel 75 tions are not affected by the PCRE_NOTBOL or PCRE_NOTEOL options, which
2706 nigel 77 affect only the behaviour of the circumflex and dollar metacharacters.
2707     However, if the startoffset argument of pcre_exec() is non-zero, indi-
2708 nigel 75 cating that matching is to start at a point other than the beginning of
2709 nigel 77 the subject, \A can never match. The difference between \Z and \z is
2710     that \Z matches before a newline that is the last character of the
2711     string as well as at the end of the string, whereas \z matches only at
2712 nigel 75 the end.
2713 nigel 63
2714 nigel 77 The \G assertion is true only when the current matching position is at
2715     the start point of the match, as specified by the startoffset argument
2716     of pcre_exec(). It differs from \A when the value of startoffset is
2717     non-zero. By calling pcre_exec() multiple times with appropriate argu-
2718 nigel 73 ments, you can mimic Perl's /g option, and it is in this kind of imple-
2719     mentation where \G can be useful.
2720 nigel 41
2721 nigel 77 Note, however, that PCRE's interpretation of \G, as the start of the
2722 nigel 73 current match, is subtly different from Perl's, which defines it as the
2723 nigel 77 end of the previous match. In Perl, these can be different when the
2724     previously matched string was empty. Because PCRE does just one match
2725 nigel 73 at a time, it cannot reproduce this behaviour.
2726 nigel 41
2727 nigel 77 If all the alternatives of a pattern begin with \G, the expression is
2728 nigel 73 anchored to the starting match position, and the "anchored" flag is set
2729     in the compiled regular expression.
2730 nigel 63
2731    
2732 nigel 41 CIRCUMFLEX AND DOLLAR
2733 nigel 63
2734 nigel 73 Outside a character class, in the default matching mode, the circumflex
2735 nigel 77 character is an assertion that is true only if the current matching
2736     point is at the start of the subject string. If the startoffset argu-
2737     ment of pcre_exec() is non-zero, circumflex can never match if the
2738     PCRE_MULTILINE option is unset. Inside a character class, circumflex
2739 nigel 73 has an entirely different meaning (see below).
2740 nigel 41
2741 nigel 77 Circumflex need not be the first character of the pattern if a number
2742     of alternatives are involved, but it should be the first thing in each
2743     alternative in which it appears if the pattern is ever to match that
2744     branch. If all possible alternatives start with a circumflex, that is,
2745     if the pattern is constrained to match only at the start of the sub-
2746     ject, it is said to be an "anchored" pattern. (There are also other
2747 nigel 73 constructs that can cause a pattern to be anchored.)
2748 nigel 41
2749 nigel 77 A dollar character is an assertion that is true only if the current
2750     matching point is at the end of the subject string, or immediately
2751 nigel 73 before a newline character that is the last character in the string (by
2752 nigel 77 default). Dollar need not be the last character of the pattern if a
2753     number of alternatives are involved, but it should be the last item in
2754     any branch in which it appears. Dollar has no special meaning in a
2755 nigel 73 character class.
2756 nigel 41
2757 nigel 77 The meaning of dollar can be changed so that it matches only at the
2758     very end of the string, by setting the PCRE_DOLLAR_ENDONLY option at
2759 nigel 73 compile time. This does not affect the \Z assertion.
2760 nigel 41
2761 nigel 73 The meanings of the circumflex and dollar characters are changed if the
2762     PCRE_MULTILINE option is set. When this is the case, they match immedi-
2763 nigel 77 ately after and immediately before an internal newline character,
2764     respectively, in addition to matching at the start and end of the sub-
2765     ject string. For example, the pattern /^abc$/ matches the subject
2766     string "def\nabc" (where \n represents a newline character) in multi-
2767 nigel 75 line mode, but not otherwise. Consequently, patterns that are anchored
2768 nigel 77 in single line mode because all branches start with ^ are not anchored
2769     in multiline mode, and a match for circumflex is possible when the
2770     startoffset argument of pcre_exec() is non-zero. The PCRE_DOL-
2771 nigel 75 LAR_ENDONLY option is ignored if PCRE_MULTILINE is set.
2772 nigel 41
2773 nigel 77 Note that the sequences \A, \Z, and \z can be used to match the start
2774     and end of the subject in both modes, and if all branches of a pattern
2775     start with \A it is always anchored, whether PCRE_MULTILINE is set or
2776 nigel 73 not.
2777 nigel 41
2778    
2779 nigel 63 FULL STOP (PERIOD, DOT)
2780 nigel 41
2781 nigel 73 Outside a character class, a dot in the pattern matches any one charac-
2782 nigel 77 ter in the subject, including a non-printing character, but not (by
2783     default) newline. In UTF-8 mode, a dot matches any UTF-8 character,
2784 nigel 75 which might be more than one byte long, except (by default) newline. If
2785 nigel 77 the PCRE_DOTALL option is set, dots match newlines as well. The han-
2786     dling of dot is entirely independent of the handling of circumflex and
2787     dollar, the only relationship being that they both involve newline
2788 nigel 75 characters. Dot has no special meaning in a character class.
2789 nigel 41
2790    
2791 nigel 63 MATCHING A SINGLE BYTE
2792    
2793 nigel 73 Outside a character class, the escape sequence \C matches any one byte,
2794 nigel 77 both in and out of UTF-8 mode. Unlike a dot, it can match a newline.
2795     The feature is provided in Perl in order to match individual bytes in
2796     UTF-8 mode. Because it breaks up UTF-8 characters into individual
2797     bytes, what remains in the string may be a malformed UTF-8 string. For
2798 nigel 75 this reason, the \C escape sequence is best avoided.
2799 nigel 63
2800 nigel 77 PCRE does not allow \C to appear in lookbehind assertions (described
2801     below), because in UTF-8 mode this would make it impossible to calcu-
2802 nigel 75 late the length of the lookbehind.
2803 nigel 63
2804    
2805 nigel 75 SQUARE BRACKETS AND CHARACTER CLASSES
2806 nigel 63
2807 nigel 73 An opening square bracket introduces a character class, terminated by a
2808     closing square bracket. A closing square bracket on its own is not spe-
2809     cial. If a closing square bracket is required as a member of the class,
2810 nigel 77 it should be the first data character in the class (after an initial
2811 nigel 73 circumflex, if present) or escaped with a backslash.
2812 nigel 41
2813 nigel 77 A character class matches a single character in the subject. In UTF-8
2814     mode, the character may occupy more than one byte. A matched character
2815 nigel 73 must be in the set of characters defined by the class, unless the first
2816 nigel 77 character in the class definition is a circumflex, in which case the
2817     subject character must not be in the set defined by the class. If a
2818     circumflex is actually required as a member of the class, ensure it is
2819 nigel 73 not the first character, or escape it with a backslash.
2820 nigel 41
2821 nigel 77 For example, the character class [aeiou] matches any lower case vowel,
2822     while [^aeiou] matches any character that is not a lower case vowel.
2823 nigel 73 Note that a circumflex is just a convenient notation for specifying the
2824 nigel 77 characters that are in the class by enumerating those that are not. A
2825     class that starts with a circumflex is not an assertion: it still con-
2826     sumes a character from the subject string, and therefore it fails if
2827 nigel 75 the current pointer is at the end of the string.
2828 nigel 41
2829 nigel 77 In UTF-8 mode, characters with values greater than 255 can be included
2830     in a class as a literal string of bytes, or by using the \x{ escaping
2831 nigel 73 mechanism.
2832 nigel 63
2833 nigel 77 When caseless matching is set, any letters in a class represent both
2834     their upper case and lower case versions, so for example, a caseless
2835     [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not
2836     match "A", whereas a caseful version would. In UTF-8 mode, PCRE always
2837     understands the concept of case for characters whose values are less
2838     than 128, so caseless matching is always possible. For characters with
2839     higher values, the concept of case is supported if PCRE is compiled
2840     with Unicode property support, but not otherwise. If you want to use
2841     caseless matching for characters 128 and above, you must ensure that
2842     PCRE is compiled with Unicode property support as well as with UTF-8
2843     support.
2844 nigel 41
2845 nigel 75 The newline character is never treated in any special way in character
2846     classes, whatever the setting of the PCRE_DOTALL or PCRE_MULTILINE
2847 nigel 73 options is. A class such as [^a] will always match a newline.
2848 nigel 41
2849 nigel 75 The minus (hyphen) character can be used to specify a range of charac-
2850     ters in a character class. For example, [d-m] matches any letter
2851     between d and m, inclusive. If a minus character is required in a
2852     class, it must be escaped with a backslash or appear in a position
2853     where it cannot be interpreted as indicating a range, typically as the
2854 nigel 73 first or last character in the class.
2855 nigel 41
2856 nigel 73 It is not possible to have the literal character "]" as the end charac-
2857 nigel 75 ter of a range. A pattern such as [W-]46] is interpreted as a class of
2858     two characters ("W" and "-") followed by a literal string "46]", so it
2859     would match "W46]" or "-46]". However, if the "]" is escaped with a
2860     backslash it is interpreted as the end of range, so [W-\]46] is inter-
2861     preted as a class containing a range followed by two other characters.
2862     The octal or hexadecimal representation of "]" can also be used to end
2863     a range.
2864 nigel 41
2865 nigel 75 Ranges operate in the collating sequence of character values. They can
2866     also be used for characters specified numerically, for example
2867     [\000-\037]. In UTF-8 mode, ranges can include characters whose values
2868 nigel 73 are greater than 255, for example [\x{100}-\x{2ff}].
2869 nigel 63
2870 nigel 73 If a range that includes letters is used when caseless matching is set,
2871     it matches the letters in either case. For example, [W-c] is equivalent
2872 nigel 75 to [][\\^_`wxyzabc], matched caselessly, and in non-UTF-8 mode, if
2873     character tables for the "fr_FR" locale are in use, [\xc8-\xcb] matches
2874     accented E characters in both cases. In UTF-8 mode, PCRE supports the
2875     concept of case for characters with values greater than 128 only when
2876     it is compiled with Unicode property support.
2877 nigel 41
2878 nigel 75 The character types \d, \D, \p, \P, \s, \S, \w, and \W may also appear
2879     in a character class, and add the characters that they match to the
2880     class. For example, [\dABCDEF] matches any hexadecimal digit. A circum-
2881     flex can conveniently be used with the upper case character types to
2882     specify a more restricted set of characters than the matching lower
2883     case type. For example, the class [^\W_] matches any letter or digit,
2884     but not underscore.
2885 nigel 41
2886 nigel 75 The only metacharacters that are recognized in character classes are
2887     backslash, hyphen (only where it can be interpreted as specifying a
2888     range), circumflex (only at the start), opening square bracket (only
2889     when it can be interpreted as introducing a POSIX class name - see the
2890     next section), and the terminating closing square bracket. However,
2891     escaping other non-alphanumeric characters does no harm.
2892 nigel 41
2893 nigel 73
2894 nigel 43 POSIX CHARACTER CLASSES
2895    
2896 nigel 75 Perl supports the POSIX notation for character classes. This uses names
2897     enclosed by [: and :] within the enclosing square brackets. PCRE also
2898     supports this notation. For example,
2899 nigel 63
2900 nigel 73 [01[:alpha:]%]
2901 nigel 43
2902 nigel 73 matches "0", "1", any alphabetic character, or "%". The supported class
2903     names are
2904 nigel 43
2905 nigel 73 alnum letters and digits
2906     alpha letters
2907     ascii character codes 0 - 127
2908     blank space or tab only
2909     cntrl control characters
2910     digit decimal digits (same as \d)
2911     graph printing characters, excluding space
2912     lower lower case letters
2913     print printing characters, including space
2914     punct printing characters, excluding letters and digits
2915     space white space (not quite the same as \s)
2916     upper upper case letters
2917     word "word" characters (same as \w)
2918     xdigit hexadecimal digits
2919 nigel 43
2920 nigel 73 The "space" characters are HT (9), LF (10), VT (11), FF (12), CR (13),
2921     and space (32). Notice that this list includes the VT character (code
2922     11). This makes "space" different to \s, which does not include VT (for
2923     Perl compatibility).
2924 nigel 43
2925 nigel 73 The name "word" is a Perl extension, and "blank" is a GNU extension
2926     from Perl 5.8. Another Perl extension is negation, which is indicated
2927     by a ^ character after the colon. For example,
2928 nigel 63
2929 nigel 73 [12[:^digit:]]
2930 nigel 43
2931 nigel 73 matches "1", "2", or any non-digit. PCRE (and Perl) also recognize the
2932     POSIX syntax [.ch.] and [=ch=] where "ch" is a "collating element", but
2933     these are not supported, and an error is given if they are encountered.
2934 nigel 43
2935 nigel 75 In UTF-8 mode, characters with values greater than 128 do not match any
2936 nigel 73 of the POSIX character classes.
2937 nigel 43
2938    
2939 nigel 41 VERTICAL BAR
2940 nigel 63
2941 nigel 73 Vertical bar characters are used to separate alternative patterns. For
2942     example, the pattern
2943 nigel 41
2944 nigel 73 gilbert|sullivan
2945 nigel 41
2946 nigel 73 matches either "gilbert" or "sullivan". Any number of alternatives may
2947     appear, and an empty alternative is permitted (matching the empty
2948     string). The matching process tries each alternative in turn, from
2949     left to right, and the first one that succeeds is used. If the alterna-
2950     tives are within a subpattern (defined below), "succeeds" means match-
2951     ing the rest of the main pattern as well as the alternative in the sub-
2952     pattern.
2953 nigel 41
2954    
2955     INTERNAL OPTION SETTING
2956    
2957 nigel 73 The settings of the PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL, and
2958     PCRE_EXTENDED options can be changed from within the pattern by a
2959     sequence of Perl option letters enclosed between "(?" and ")". The
2960     option letters are
2961 nigel 63
2962 nigel 73 i for PCRE_CASELESS
2963     m for PCRE_MULTILINE
2964     s for PCRE_DOTALL
2965     x for PCRE_EXTENDED
2966 nigel 41
2967 nigel 73 For example, (?im) sets caseless, multiline matching. It is also possi-
2968     ble to unset these options by preceding the letter with a hyphen, and a
2969     combined setting and unsetting such as (?im-sx), which sets PCRE_CASE-
2970     LESS and PCRE_MULTILINE while unsetting PCRE_DOTALL and PCRE_EXTENDED,
2971     is also permitted. If a letter appears both before and after the
2972     hyphen, the option is unset.
2973 nigel 41
2974 nigel 73 When an option change occurs at top level (that is, not inside subpat-
2975     tern parentheses), the change applies to the remainder of the pattern
2976     that follows. If the change is placed right at the start of a pattern,
2977     PCRE extracts it into the global options (and it will therefore show up
2978     in data extracted by the pcre_fullinfo() function).
2979 nigel 41
2980 nigel 73 An option change within a subpattern affects only that part of the cur-
2981     rent pattern that follows it, so
2982 nigel 41
2983 nigel 73 (a(?i)b)c
2984 nigel 41
2985 nigel 73 matches abc and aBc and no other strings (assuming PCRE_CASELESS is not
2986     used). By this means, options can be made to have different settings
2987     in different parts of the pattern. Any changes made in one alternative
2988     do carry on into subsequent branches within the same subpattern. For
2989     example,
2990 nigel 41
2991 nigel 73 (a(?i)b|c)
2992 nigel 41
2993 nigel 73 matches "ab", "aB", "c", and "C", even though when matching "C" the
2994     first branch is abandoned before the option setting. This is because
2995     the effects of option settings happen at compile time. There would be
2996     some very weird behaviour otherwise.
2997 nigel 41
2998 nigel 73 The PCRE-specific options PCRE_UNGREEDY and PCRE_EXTRA can be changed
2999     in the same way as the Perl-compatible options by using the characters
3000     U and X respectively. The (?X) flag setting is special in that it must
3001     always occur earlier in the pattern than any of the additional features
3002 nigel 75 it turns on, even when it is at top level. It is best to put it at the
3003     start.
3004 nigel 41
3005    
3006 nigel 63 SUBPATTERNS
3007 nigel 41
3008 nigel 73 Subpatterns are delimited by parentheses (round brackets), which can be
3009 nigel 75 nested. Turning part of a pattern into a subpattern does two things:
3010 nigel 41
3011 nigel 73 1. It localizes a set of alternatives. For example, the pattern
3012 nigel 41
3013 nigel 73 cat(aract|erpillar|)
3014 nigel 41
3015 nigel 73 matches one of the words "cat", "cataract", or "caterpillar". Without
3016     the parentheses, it would match "cataract", "erpillar" or the empty
3017     string.
3018 nigel 41
3019 nigel 75 2. It sets up the subpattern as a capturing subpattern. This means
3020     that, when the whole pattern matches, that portion of the subject
3021 nigel 73 string that matched the subpattern is passed back to the caller via the
3022     ovector argument of pcre_exec(). Opening parentheses are counted from
3023 nigel 75 left to right (starting from 1) to obtain numbers for the capturing
3024 nigel 73 subpatterns.
3025 nigel 41
3026 nigel 73 For example, if the string "the red king" is matched against the pat-
3027     tern
3028 nigel 41
3029 nigel 73 the ((red|white) (king|queen))
3030 nigel 41
3031 nigel 73 the captured substrings are "red king", "red", and "king", and are num-
3032     bered 1, 2, and 3, respectively.
3033 nigel 41
3034 nigel 73 The fact that plain parentheses fulfil two functions is not always
3035     helpful. There are often times when a grouping subpattern is required
3036     without a capturing requirement. If an opening parenthesis is followed
3037     by a question mark and a colon, the subpattern does not do any captur-
3038     ing, and is not counted when computing the number of any subsequent
3039     capturing subpatterns. For example, if the string "the white queen" is
3040     matched against the pattern
3041 nigel 41
3042 nigel 73 the ((?:red|white) (king|queen))
3043 nigel 41
3044 nigel 73 the captured substrings are "white queen" and "queen", and are numbered
3045     1 and 2. The maximum number of capturing subpatterns is 65535, and the
3046     maximum depth of nesting of all subpatterns, both capturing and non-
3047     capturing, is 200.
3048 nigel 41
3049 nigel 73 As a convenient shorthand, if any option settings are required at the
3050     start of a non-capturing subpattern, the option letters may appear
3051     between the "?" and the ":". Thus the two patterns
3052 nigel 41
3053 nigel 73 (?i:saturday|sunday)
3054     (?:(?i)saturday|sunday)
3055 nigel 41
3056 nigel 73 match exactly the same set of strings. Because alternative branches are
3057     tried from left to right, and options are not reset until the end of
3058     the subpattern is reached, an option setting in one branch does affect
3059     subsequent branches, so the above patterns match "SUNDAY" as well as
3060     "Saturday".
3061 nigel 41
3062    
3063 nigel 63 NAMED SUBPATTERNS
3064 nigel 41
3065 nigel 73 Identifying capturing parentheses by number is simple, but it can be
3066     very hard to keep track of the numbers in complicated regular expres-
3067     sions. Furthermore, if an expression is modified, the numbers may
3068 nigel 75 change. To help with this difficulty, PCRE supports the naming of sub-
3069 nigel 73 patterns, something that Perl does not provide. The Python syntax
3070     (?P<name>...) is used. Names consist of alphanumeric characters and
3071     underscores, and must be unique within a pattern.
3072 nigel 63
3073 nigel 73 Named capturing parentheses are still allocated numbers as well as
3074     names. The PCRE API provides function calls for extracting the name-to-
3075 nigel 75 number translation table from a compiled pattern. There is also a con-
3076     venience function for extracting a captured substring by name. For fur-
3077     ther details see the pcreapi documentation.
3078 nigel 63
3079    
3080 nigel 41 REPETITION
3081 nigel 63
3082 nigel 75 Repetition is specified by quantifiers, which can follow any of the
3083 nigel 73 following items:
3084 nigel 41
3085 nigel 73 a literal data character
3086     the . metacharacter
3087     the \C escape sequence
3088 nigel 75 the \X escape sequence (in UTF-8 mode with Unicode properties)
3089     an escape such as \d that matches a single character
3090 nigel 73 a character class
3091     a back reference (see next section)
3092     a parenthesized subpattern (unless it is an assertion)
3093 nigel 41
3094 nigel 75 The general repetition quantifier specifies a minimum and maximum num-
3095     ber of permitted matches, by giving the two numbers in curly brackets
3096     (braces), separated by a comma. The numbers must be less than 65536,
3097 nigel 73 and the first must be less than or equal to the second. For example:
3098 nigel 41
3099 nigel 73 z{2,4}
3100 nigel 41
3101 nigel 75 matches "zz", "zzz", or "zzzz". A closing brace on its own is not a
3102     special character. If the second number is omitted, but the comma is
3103     present, there is no upper limit; if the second number and the comma
3104     are both omitted, the quantifier specifies an exact number of required
3105 nigel 73 matches. Thus
3106 nigel 41
3107 nigel 73 [aeiou]{3,}
3108 nigel 41
3109 nigel 73 matches at least 3 successive vowels, but may match many more, while
3110 nigel 41
3111 nigel 73 \d{8}
3112 nigel 41
3113 nigel 75 matches exactly 8 digits. An opening curly bracket that appears in a
3114     position where a quantifier is not allowed, or one that does not match
3115     the syntax of a quantifier, is taken as a literal character. For exam-
3116 nigel 73 ple, {,6} is not a quantifier, but a literal string of four characters.
3117 nigel 63
3118 nigel 75 In UTF-8 mode, quantifiers apply to UTF-8 characters rather than to
3119 nigel 73 individual bytes. Thus, for example, \x{100}{2} matches two UTF-8 char-
3120 nigel 75 acters, each of which is represented by a two-byte sequence. Similarly,
3121     when Unicode property support is available, \X{3} matches three Unicode
3122     extended sequences, each of which may be several bytes long (and they
3123     may be of different lengths).
3124 nigel 63
3125 nigel 73 The quantifier {0} is permitted, causing the expression to behave as if
3126     the previous item and the quantifier were not present.
3127 nigel 41
3128 nigel 73 For convenience (and historical compatibility) the three most common
3129     quantifiers have single-character abbreviations:
3130 nigel 41
3131 nigel 73 * is equivalent to {0,}
3132     + is equivalent to {1,}
3133     ? is equivalent to {0,1}
3134 nigel 41
3135 nigel 73 It is possible to construct infinite loops by following a subpattern
3136     that can match no characters with a quantifier that has no upper limit,
3137     for example:
3138 nigel 41
3139 nigel 73 (a?)*
3140 nigel 41
3141 nigel 73 Earlier versions of Perl and PCRE used to give an error at compile time
3142     for such patterns. However, because there are cases where this can be
3143     useful, such patterns are now accepted, but if any repetition of the
3144     subpattern does in fact match no characters, the loop is forcibly bro-
3145     ken.
3146 nigel 41
3147 nigel 73 By default, the quantifiers are "greedy", that is, they match as much
3148     as possible (up to the maximum number of permitted times), without
3149     causing the rest of the pattern to fail. The classic example of where
3150     this gives problems is in trying to match comments in C programs. These
3151 nigel 75 appear between /* and */ and within the comment, individual * and /
3152     characters may appear. An attempt to match C comments by applying the
3153     pattern
3154 nigel 41
3155 nigel 73 /\*.*\*/
3156 nigel 41
3157 nigel 73 to the string
3158 nigel 41
3159 nigel 75 /* first comment */ not comment /* second comment */
3160 nigel 41
3161 nigel 73 fails, because it matches the entire string owing to the greediness of
3162     the .* item.
3163 nigel 41
3164 nigel 73 However, if a quantifier is followed by a question mark, it ceases to
3165     be greedy, and instead matches the minimum number of times possible, so
3166     the pattern
3167 nigel 41
3168 nigel 73 /\*.*?\*/
3169 nigel 41
3170 nigel 73 does the right thing with the C comments. The meaning of the various
3171     quantifiers is not otherwise changed, just the preferred number of
3172     matches. Do not confuse this use of question mark with its use as a
3173     quantifier in its own right. Because it has two uses, it can sometimes
3174     appear doubled, as in
3175 nigel 41
3176 nigel 73 \d??\d
3177 nigel 41
3178 nigel 73 which matches one digit by preference, but can match two if that is the
3179     only way the rest of the pattern matches.
3180 nigel 41
3181 nigel 73 If the PCRE_UNGREEDY option is set (an option which is not available in
3182     Perl), the quantifiers are not greedy by default, but individual ones
3183     can be made greedy by following them with a question mark. In other
3184     words, it inverts the default behaviour.
3185 nigel 41
3186 nigel 73 When a parenthesized subpattern is quantified with a minimum repeat
3187 nigel 75 count that is greater than 1 or with a limited maximum, more memory is
3188 nigel 73 required for the compiled pattern, in proportion to the size of the
3189     minimum or maximum.
3190 nigel 41
3191 nigel 73 If a pattern starts with .* or .{0,} and the PCRE_DOTALL option (equiv-
3192     alent to Perl's /s) is set, thus allowing the . to match newlines, the
3193     pattern is implicitly anchored, because whatever follows will be tried
3194     against every character position in the subject string, so there is no
3195     point in retrying the overall match at any position after the first.
3196     PCRE normally treats such a pattern as though it were preceded by \A.
3197 nigel 63
3198 nigel 73 In cases where it is known that the subject string contains no new-
3199     lines, it is worth setting PCRE_DOTALL in order to obtain this opti-
3200     mization, or alternatively using ^ to indicate anchoring explicitly.
3201 nigel 63
3202 nigel 73 However, there is one situation where the optimization cannot be used.
3203     When .* is inside capturing parentheses that are the subject of a
3204     backreference elsewhere in the pattern, a match at the start may fail,
3205     and a later one succeed. Consider, for example:
3206 nigel 63
3207 nigel 73 (.*)abc\1
3208 nigel 63
3209 nigel 73 If the subject is "xyz123abc123" the match point is the fourth charac-
3210     ter. For this reason, such a pattern is not implicitly anchored.
3211 nigel 41
3212 nigel 73 When a capturing subpattern is repeated, the value captured is the sub-
3213     string that matched the final iteration. For example, after
3214 nigel 41
3215 nigel 73 (tweedle[dume]{3}\s*)+
3216 nigel 41
3217 nigel 73 has matched "tweedledum tweedledee" the value of the captured substring
3218     is "tweedledee". However, if there are nested capturing subpatterns,
3219     the corresponding captured values may have been set in previous itera-
3220     tions. For example, after
3221 nigel 41
3222 nigel 73 /(a|(b))+/
3223 nigel 41
3224 nigel 73 matches "aba" the value of the second captured substring is "b".
3225 nigel 41
3226 nigel 73
3227 nigel 63 ATOMIC GROUPING AND POSSESSIVE QUANTIFIERS
3228 nigel 41
3229 nigel 73 With both maximizing and minimizing repetition, failure of what follows
3230     normally causes the repeated item to be re-evaluated to see if a dif-
3231     ferent number of repeats allows the rest of the pattern to match. Some-
3232     times it is useful to prevent this, either to change the nature of the
3233     match, or to cause it fail earlier than it otherwise might, when the
3234     author of the pattern knows there is no point in carrying on.
3235 nigel 53
3236 nigel 73 Consider, for example, the pattern \d+foo when applied to the subject
3237     line
3238 nigel 53
3239 nigel 73 123456bar
3240 nigel 53
3241 nigel 73 After matching all 6 digits and then failing to match "foo", the normal
3242     action of the matcher is to try again with only 5 digits matching the
3243     \d+ item, and then with 4, and so on, before ultimately failing.
3244     "Atomic grouping" (a term taken from Jeffrey Friedl's book) provides
3245     the means for specifying that once a subpattern has matched, it is not
3246     to be re-evaluated in this way.
3247 nigel 53
3248 nigel 73 If we use atomic grouping for the previous example, the matcher would
3249     give up immediately on failing to match "foo" the first time. The nota-
3250     tion is a kind of special parenthesis, starting with (?> as in this
3251     example:
3252 nigel 53
3253 nigel 73 (?>\d+)foo
3254 nigel 53
3255 nigel 73 This kind of parenthesis "locks up" the part of the pattern it con-
3256     tains once it has matched, and a failure further into the pattern is
3257     prevented from backtracking into it. Backtracking past it to previous
3258     items, however, works as normal.
3259 nigel 53
3260 nigel 73 An alternative description is that a subpattern of this type matches
3261     the string of characters that an identical standalone pattern would
3262     match, if anchored at the current point in the subject string.
3263 nigel 63
3264 nigel 73 Atomic grouping subpatterns are not capturing subpatterns. Simple cases
3265     such as the above example can be thought of as a maximizing repeat that
3266     must swallow everything it can. So, while both \d+ and \d+? are pre-
3267     pared to adjust the number of digits they match in order to make the
3268     rest of the pattern match, (?>\d+) can only match an entire sequence of
3269     digits.
3270 nigel 63
3271 nigel 73 Atomic groups in general can of course contain arbitrarily complicated
3272     subpatterns, and can be nested. However, when the subpattern for an
3273     atomic group is just a single repeated item, as in the example above, a
3274     simpler notation, called a "possessive quantifier" can be used. This
3275     consists of an additional + character following a quantifier. Using
3276     this notation, the previous example can be rewritten as
3277 nigel 63
3278 nigel 75 \d++foo
3279 nigel 63
3280 nigel 73 Possessive quantifiers are always greedy; the setting of the
3281     PCRE_UNGREEDY option is ignored. They are a convenient notation for the
3282     simpler forms of atomic group. However, there is no difference in the
3283     meaning or processing of a possessive quantifier and the equivalent
3284     atomic group.
3285 nigel 63
3286 nigel 73 The possessive quantifier syntax is an extension to the Perl syntax. It
3287     originates in Sun's Java package.
3288 nigel 63
3289 nigel 73 When a pattern contains an unlimited repeat inside a subpattern that
3290     can itself be repeated an unlimited number of times, the use of an
3291     atomic group is the only way to avoid some failing matches taking a
3292     very long time indeed. The pattern
3293 nigel 63
3294 nigel 73 (\D+|<\d+>)*[!?]
3295 nigel 63
3296 nigel 73 matches an unlimited number of substrings that either consist of non-
3297     digits, or digits enclosed in <>, followed by either ! or ?. When it
3298     matches, it runs quickly. However, if it is applied to
3299 nigel 63
3300 nigel 73 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
3301 nigel 63
3302 nigel 73 it takes a long time before reporting failure. This is because the
3303 nigel 75 string can be divided between the internal \D+ repeat and the external
3304     * repeat in a large number of ways, and all have to be tried. (The
3305     example uses [!?] rather than a single character at the end, because
3306     both PCRE and Perl have an optimization that allows for fast failure
3307     when a single character is used. They remember the last single charac-
3308     ter that is required for a match, and fail early if it is not present
3309     in the string.) If the pattern is changed so that it uses an atomic
3310     group, like this:
3311 nigel 63
3312 nigel 73 ((?>\D+)|<\d+>)*[!?]
3313 nigel 63
3314 nigel 75 sequences of non-digits cannot be broken, and failure happens quickly.
3315 nigel 63
3316    
3317     BACK REFERENCES
3318    
3319 nigel 73 Outside a character class, a backslash followed by a digit greater than
3320     0 (and possibly further digits) is a back reference to a capturing sub-
3321 nigel 75 pattern earlier (that is, to its left) in the pattern, provided there
3322 nigel 73 have been that many previous capturing left parentheses.
3323 nigel 41
3324 nigel 73 However, if the decimal number following the backslash is less than 10,
3325 nigel 75 it is always taken as a back reference, and causes an error only if
3326     there are not that many capturing left parentheses in the entire pat-
3327     tern. In other words, the parentheses that are referenced need not be
3328     to the left of the reference for numbers less than 10. See the subsec-
3329     tion entitled "Non-printing characters" above for further details of
3330     the handling of digits following a backslash.
3331 nigel 41
3332 nigel 75 A back reference matches whatever actually matched the capturing sub-
3333     pattern in the current subject string, rather than anything matching
3334 nigel 73 the subpattern itself (see "Subpatterns as subroutines" below for a way
3335     of doing that). So the pattern
3336 nigel 41
3337 nigel 73 (sens|respons)e and \1ibility
3338 nigel 41
3339 nigel 75 matches "sense and sensibility" and "response and responsibility", but
3340     not "sense and responsibility". If caseful matching is in force at the
3341     time of the back reference, the case of letters is relevant. For exam-
3342 nigel 73 ple,
3343 nigel 41
3344 nigel 73 ((?i)rah)\s+\1
3345 nigel 41
3346 nigel 75 matches "rah rah" and "RAH RAH", but not "RAH rah", even though the
3347 nigel 73 original capturing subpattern is matched caselessly.
3348 nigel 41
3349 nigel 75 Back references to named subpatterns use the Python syntax (?P=name).
3350 nigel 73 We could rewrite the above example as follows:
3351 nigel 63
3352 nigel 73 (?<p1>(?i)rah)\s+(?P=p1)
3353 nigel 63
3354 nigel 75 There may be more than one back reference to the same subpattern. If a
3355     subpattern has not actually been used in a particular match, any back
3356 nigel 73 references to it always fail. For example, the pattern
3357 nigel 41
3358 nigel 73 (a|(bc))\2
3359 nigel 41
3360 nigel 75 always fails if it starts to match "a" rather than "bc". Because there
3361     may be many capturing parentheses in a pattern, all digits following
3362     the backslash are taken as part of a potential back reference number.
3363 nigel 73 If the pattern continues with a digit character, some delimiter must be
3364 nigel 75 used to terminate the back reference. If the PCRE_EXTENDED option is
3365     set, this can be whitespace. Otherwise an empty comment (see "Com-
3366     ments" below) can be used.
3367 nigel 41
3368 nigel 73 A back reference that occurs inside the parentheses to which it refers
3369     fails when the subpattern is first used, so, for example, (a\1) never
3370     matches. However, such references can be useful inside repeated sub-
3371     patterns. For example, the pattern
3372 nigel 41
3373 nigel 73 (a|b\1)+
3374 nigel 41
3375 nigel 73 matches any number of "a"s and also "aba", "ababbaa" etc. At each iter-
3376     ation of the subpattern, the back reference matches the character
3377     string corresponding to the previous iteration. In order for this to
3378     work, the pattern must be such that the first iteration does not need
3379     to match the back reference. This can be done using alternation, as in
3380     the example above, or by a quantifier with a minimum of zero.
3381 nigel 41
3382    
3383 nigel 63 ASSERTIONS
3384 nigel 41
3385 nigel 73 An assertion is a test on the characters following or preceding the
3386     current matching point that does not actually consume any characters.
3387     The simple assertions coded as \b, \B, \A, \G, \Z, \z, ^ and $ are
3388 nigel 75 described above.
3389 nigel 43
3390 nigel 75 More complicated assertions are coded as subpatterns. There are two
3391     kinds: those that look ahead of the current position in the subject
3392     string, and those that look behind it. An assertion subpattern is
3393     matched in the normal way, except that it does not cause the current
3394     matching position to be changed.
3395 nigel 41
3396 nigel 75 Assertion subpatterns are not capturing subpatterns, and may not be
3397     repeated, because it makes no sense to assert the same thing several
3398     times. If any kind of assertion contains capturing subpatterns within
3399     it, these are counted for the purposes of numbering the capturing sub-
3400     patterns in the whole pattern. However, substring capturing is carried
3401     out only for positive assertions, because it does not make sense for
3402     negative assertions.
3403    
3404     Lookahead assertions
3405    
3406     Lookahead assertions start with (?= for positive assertions and (?! for
3407     negative assertions. For example,
3408    
3409 nigel 73 \w+(?=;)
3410 nigel 41
3411 nigel 73 matches a word followed by a semicolon, but does not include the semi-
3412     colon in the match, and
3413 nigel 41
3414 nigel 73 foo(?!bar)
3415 nigel 41
3416 nigel 73 matches any occurrence of "foo" that is not followed by "bar". Note
3417     that the apparently similar pattern
3418 nigel 41
3419 nigel 73 (?!foo)bar
3420 nigel 41
3421 nigel 73 does not find an occurrence of "bar" that is preceded by something
3422     other than "foo"; it finds any occurrence of "bar" whatsoever, because
3423     the assertion (?!foo) is always true when the next three characters are
3424 nigel 75 "bar". A lookbehind assertion is needed to achieve the other effect.
3425 nigel 41
3426 nigel 73 If you want to force a matching failure at some point in a pattern, the
3427     most convenient way to do it is with (?!) because an empty string
3428     always matches, so an assertion that requires there not to be an empty
3429     string must always fail.
3430 nigel 63
3431 nigel 75 Lookbehind assertions
3432    
3433 nigel 73 Lookbehind assertions start with (?<= for positive assertions and (?<!
3434     for negative assertions. For example,
3435 nigel 41
3436 nigel 73 (?<!foo)bar
3437 nigel 41
3438 nigel 73 does find an occurrence of "bar" that is not preceded by "foo". The
3439     contents of a lookbehind assertion are restricted such that all the
3440     strings it matches must have a fixed length. However, if there are sev-
3441     eral alternatives, they do not all have to have the same fixed length.
3442     Thus
3443 nigel 41
3444 nigel 73 (?<=bullock|donkey)
3445 nigel 41
3446 nigel 73 is permitted, but
3447 nigel 41
3448 nigel 73 (?<!dogs?|cats?)
3449 nigel 41
3450 nigel 73 causes an error at compile time. Branches that match different length
3451     strings are permitted only at the top level of a lookbehind assertion.
3452     This is an extension compared with Perl (at least for 5.8), which
3453     requires all branches to match the same length of string. An assertion
3454     such as
3455 nigel 41
3456 nigel 73 (?<=ab(c|de))
3457 nigel 41
3458 nigel 73 is not permitted, because its single top-level branch can match two
3459     different lengths, but it is acceptable if rewritten to use two top-
3460     level branches:
3461 nigel 41
3462 nigel 73 (?<=abc|abde)
3463 nigel 41
3464 nigel 73 The implementation of lookbehind assertions is, for each alternative,
3465     to temporarily move the current position back by the fixed width and
3466     then try to match. If there are insufficient characters before the cur-
3467     rent position, the match is deemed to fail.
3468 nigel 41
3469 nigel 73 PCRE does not allow the \C escape (which matches a single byte in UTF-8
3470     mode) to appear in lookbehind assertions, because it makes it impossi-
3471 nigel 75 ble to calculate the length of the lookbehind. The \X escape, which can
3472     match different numbers of bytes, is also not permitted.
3473 nigel 63
3474 nigel 75 Atomic groups can be used in conjunction with lookbehind assertions to
3475 nigel 73 specify efficient matching at the end of the subject string. Consider a
3476     simple pattern such as
3477 nigel 63
3478 nigel 73 abcd$
3479 nigel 63
3480 nigel 75 when applied to a long string that does not match. Because matching
3481 nigel 73 proceeds from left to right, PCRE will look for each "a" in the subject
3482 nigel 75 and then see if what follows matches the rest of the pattern. If the