/[pcre]/code/trunk/doc/pcre.txt
ViewVC logotype

Contents of /code/trunk/doc/pcre.txt

Parent Directory Parent Directory | Revision Log Revision Log


Revision 247 - (hide annotations) (download)
Mon Sep 17 09:38:32 2007 UTC (6 years, 7 months ago) by ph10
File MIME type: text/plain
File size: 301740 byte(s)
(1) Update tests 2,5,7,8 to run when --enable-bsr-anycrlf is used.
(2) Updates files changed by building test release.

1 nigel 75 -----------------------------------------------------------------------------
2 nigel 63 This file contains a concatenation of the PCRE man pages, converted to plain
3     text format for ease of searching with a text editor, or for use on systems
4     that do not have a man page processor. The small individual files that give
5     synopses of each function in the library have not been included. There are
6     separate text files for the pcregrep and pcretest commands.
7     -----------------------------------------------------------------------------
8    
9 nigel 41
10 nigel 79 PCRE(3) PCRE(3)
11 nigel 41
12 nigel 79
13 nigel 73 NAME
14     PCRE - Perl-compatible regular expressions
15    
16 nigel 77
17 nigel 75 INTRODUCTION
18 nigel 41
19 nigel 73 The PCRE library is a set of functions that implement regular expres-
20     sion pattern matching using the same syntax and semantics as Perl, with
21 nigel 93 just a few differences. (Certain features that appeared in Python and
22     PCRE before they appeared in Perl are also available using the Python
23     syntax.)
24 nigel 63
25 nigel 93 The current implementation of PCRE (release 7.x) corresponds approxi-
26     mately with Perl 5.10, including support for UTF-8 encoded strings and
27     Unicode general category properties. However, UTF-8 and Unicode support
28     has to be explicitly enabled; it is not the default. The Unicode tables
29     correspond to Unicode release 5.0.0.
30 nigel 77
31 nigel 93 In addition to the Perl-compatible matching function, PCRE contains an
32     alternative matching function that matches the same compiled patterns
33     in a different way. In certain circumstances, the alternative function
34     has some advantages. For a discussion of the two matching algorithms,
35     see the pcrematching page.
36    
37     PCRE is written in C and released as a C library. A number of people
38     have written wrappers and interfaces of various kinds. In particular,
39     Google Inc. have provided a comprehensive C++ wrapper. This is now
40 nigel 77 included as part of the PCRE distribution. The pcrecpp page has details
41 nigel 93 of this interface. Other people's contributions can be found in the
42 nigel 77 Contrib directory at the primary FTP site, which is:
43 nigel 63
44 nigel 73 ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre
45 nigel 63
46 nigel 93 Details of exactly which Perl regular expression features are and are
47 nigel 73 not supported by PCRE are given in separate documents. See the pcrepat-
48 ph10 208 tern and pcrecompat pages. There is a syntax summary in the pcresyntax
49     page.
50 nigel 63
51 ph10 208 Some features of PCRE can be included, excluded, or changed when the
52     library is built. The pcre_config() function makes it possible for a
53     client to discover which features are available. The features them-
54     selves are described in the pcrebuild page. Documentation about build-
55     ing PCRE for various operating systems can be found in the README file
56 nigel 75 in the source distribution.
57 nigel 63
58 ph10 208 The library contains a number of undocumented internal functions and
59     data tables that are used by more than one of the exported external
60     functions, but which are not intended for use by external callers.
61     Their names all begin with "_pcre_", which hopefully will not provoke
62 nigel 83 any name clashes. In some environments, it is possible to control which
63 ph10 208 external symbols are exported when a shared library is built, and in
64 nigel 83 these cases the undocumented symbols are not exported.
65 nigel 63
66 nigel 77
67 nigel 63 USER DOCUMENTATION
68    
69 ph10 208 The user documentation for PCRE comprises a number of different sec-
70     tions. In the "man" format, each of these is a separate "man page". In
71     the HTML format, each is a separate page, linked from the index page.
72     In the plain text format, all the sections are concatenated, for ease
73 nigel 75 of searching. The sections are as follows:
74 nigel 63
75 nigel 73 pcre this document
76 ph10 153 pcre-config show PCRE installation configuration information
77 nigel 77 pcreapi details of PCRE's native C API
78 nigel 73 pcrebuild options for building PCRE
79     pcrecallout details of the callout feature
80     pcrecompat discussion of Perl compatibility
81 nigel 77 pcrecpp details of the C++ wrapper
82 nigel 73 pcregrep description of the pcregrep command
83 nigel 77 pcrematching discussion of the two matching algorithms
84 nigel 75 pcrepartial details of the partial matching facility
85 nigel 73 pcrepattern syntax and semantics of supported
86     regular expressions
87 ph10 208 pcresyntax quick syntax reference
88 nigel 73 pcreperform discussion of performance issues
89 nigel 77 pcreposix the POSIX-compatible C API
90 nigel 75 pcreprecompile details of saving and re-using precompiled patterns
91 nigel 73 pcresample discussion of the sample program
92 nigel 91 pcrestack discussion of stack usage
93 nigel 75 pcretest description of the pcretest testing command
94 nigel 63
95 ph10 208 In addition, in the "man" and HTML formats, there is a short page for
96 nigel 77 each C library function, listing its arguments and results.
97 nigel 63
98    
99     LIMITATIONS
100    
101 ph10 208 There are some size limitations in PCRE but it is hoped that they will
102 nigel 73 never in practice be relevant.
103 nigel 63
104 ph10 208 The maximum length of a compiled pattern is 65539 (sic) bytes if PCRE
105 nigel 73 is compiled with the default internal linkage size of 2. If you want to
106 ph10 208 process regular expressions that are truly enormous, you can compile
107     PCRE with an internal linkage size of 3 or 4 (see the README file in
108     the source distribution and the pcrebuild documentation for details).
109     In these cases the limit is substantially larger. However, the speed
110 nigel 93 of execution is slower.
111 nigel 63
112 ph10 208 All values in repeating quantifiers must be less than 65536.
113 nigel 63
114 nigel 93 There is no limit to the number of parenthesized subpatterns, but there
115     can be no more than 65535 capturing subpatterns.
116 nigel 63
117 nigel 93 The maximum length of name for a named subpattern is 32 characters, and
118     the maximum number of named subpatterns is 10000.
119 nigel 91
120 ph10 197 The maximum length of a subject string is the largest positive number
121     that an integer variable can hold. However, when using the traditional
122 nigel 77 matching function, PCRE uses recursion to handle subpatterns and indef-
123 ph10 197 inite repetition. This means that the available stack space may limit
124 nigel 77 the size of a subject string that can be processed by certain patterns.
125 nigel 91 For a discussion of stack issues, see the pcrestack documentation.
126 nigel 63
127    
128 nigel 75 UTF-8 AND UNICODE PROPERTY SUPPORT
129 nigel 63
130 ph10 197 From release 3.3, PCRE has had some support for character strings
131     encoded in the UTF-8 format. For release 4.0 this was greatly extended
132     to cover most common requirements, and in release 5.0 additional sup-
133 nigel 75 port for Unicode general category properties was added.
134 nigel 63
135 ph10 197 In order process UTF-8 strings, you must build PCRE to include UTF-8
136     support in the code, and, in addition, you must call pcre_compile()
137     with the PCRE_UTF8 option flag. When you do this, both the pattern and
138     any subject strings that are matched against it are treated as UTF-8
139 nigel 73 strings instead of just strings of bytes.
140 nigel 63
141 ph10 197 If you compile PCRE with UTF-8 support, but do not use it at run time,
142     the library will be a bit bigger, but the additional run time overhead
143 nigel 93 is limited to testing the PCRE_UTF8 flag occasionally, so should not be
144     very big.
145 nigel 63
146 nigel 75 If PCRE is built with Unicode character property support (which implies
147 ph10 197 UTF-8 support), the escape sequences \p{..}, \P{..}, and \X are sup-
148 nigel 75 ported. The available properties that can be tested are limited to the
149 ph10 197 general category properties such as Lu for an upper case letter or Nd
150     for a decimal number, the Unicode script names such as Arabic or Han,
151     and the derived properties Any and L&. A full list is given in the
152 nigel 87 pcrepattern documentation. Only the short names for properties are sup-
153 ph10 197 ported. For example, \p{L} matches a letter. Its Perl synonym, \p{Let-
154     ter}, is not supported. Furthermore, in Perl, many properties may
155     optionally be prefixed by "Is", for compatibility with Perl 5.6. PCRE
156 nigel 87 does not support this.
157 nigel 75
158 ph10 211 Validity of UTF-8 strings
159 nigel 63
160 ph10 211 When you set the PCRE_UTF8 flag, the strings passed as patterns and
161     subjects are (by default) checked for validity on entry to the relevant
162     functions. From release 7.3 of PCRE, the check is according the rules
163     of RFC 3629, which are themselves derived from the Unicode specifica-
164     tion. Earlier releases of PCRE followed the rules of RFC 2279, which
165     allows the full range of 31-bit values (0 to 0x7FFFFFFF). The current
166     check allows only values in the range U+0 to U+10FFFF, excluding U+D800
167     to U+DFFF.
168 nigel 63
169 ph10 211 The excluded code points are the "Low Surrogate Area" of Unicode, of
170     which the Unicode Standard says this: "The Low Surrogate Area does not
171     contain any character assignments, consequently no character code
172     charts or namelists are provided for this area. Surrogates are reserved
173     for use with UTF-16 and then must be used in pairs." The code points
174     that are encoded by UTF-16 pairs are available as independent code
175     points in the UTF-8 encoding. (In other words, the whole surrogate
176     thing is a fudge for UTF-16 which unfortunately messes up UTF-8.)
177    
178     If an invalid UTF-8 string is passed to PCRE, an error return
179     (PCRE_ERROR_BADUTF8) is given. In some situations, you may already know
180     that your strings are valid, and therefore want to skip these checks in
181     order to improve performance. If you set the PCRE_NO_UTF8_CHECK flag at
182     compile time or at run time, PCRE assumes that the pattern or subject
183     it is given (respectively) contains only valid UTF-8 codes. In this
184     case, it does not diagnose an invalid UTF-8 string.
185    
186     If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set,
187     what happens depends on why the string is invalid. If the string con-
188     forms to the "old" definition of UTF-8 (RFC 2279), it is processed as a
189     string of characters in the range 0 to 0x7FFFFFFF. In other words,
190     apart from the initial validity test, PCRE (when in UTF-8 mode) handles
191     strings according to the more liberal rules of RFC 2279. However, if
192     the string does not even conform to RFC 2279, the result is undefined.
193     Your program may crash.
194    
195     If you want to process strings of values in the full range 0 to
196     0x7FFFFFFF, encoded in a UTF-8-like manner as per the old RFC, you can
197     set PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in
198     this situation, you will have to apply your own validity check.
199    
200     General comments about UTF-8 mode
201    
202     1. An unbraced hexadecimal escape sequence (such as \xb3) matches a
203 nigel 87 two-byte UTF-8 character if the value is greater than 127.
204 nigel 63
205 ph10 211 2. Octal numbers up to \777 are recognized, and match two-byte UTF-8
206 nigel 91 characters for values greater than \177.
207    
208 ph10 211 3. Repeat quantifiers apply to complete UTF-8 characters, not to indi-
209 nigel 73 vidual bytes, for example: \x{100}{3}.
210 nigel 63
211 ph10 211 4. The dot metacharacter matches one UTF-8 character instead of a sin-
212 nigel 75 gle byte.
213 nigel 63
214 ph10 211 5. The escape sequence \C can be used to match a single byte in UTF-8
215 ph10 197 mode, but its use can lead to some strange effects. This facility is
216 nigel 77 not available in the alternative matching function, pcre_dfa_exec().
217 nigel 63
218 ph10 211 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly
219 ph10 197 test characters of any code value, but the characters that PCRE recog-
220     nizes as digits, spaces, or word characters remain the same set as
221 nigel 75 before, all with values less than 256. This remains true even when PCRE
222 ph10 197 includes Unicode property support, because to do otherwise would slow
223     down PCRE in many common cases. If you really want to test for a wider
224     sense of, say, "digit", you must use Unicode property tests such as
225 nigel 75 \p{Nd}.
226 nigel 63
227 ph10 211 7. Similarly, characters that match the POSIX named character classes
228 nigel 75 are all low-valued characters.
229 nigel 63
230 ph10 211 8. However, the Perl 5.10 horizontal and vertical whitespace matching
231 ph10 182 escapes (\h, \H, \v, and \V) do match all the appropriate Unicode char-
232     acters.
233    
234 ph10 211 9. Case-insensitive matching applies only to characters whose values
235 ph10 197 are less than 128, unless PCRE is built with Unicode property support.
236     Even when Unicode property support is available, PCRE still uses its
237     own character tables when checking the case of low-valued characters,
238     so as not to degrade performance. The Unicode property information is
239 nigel 87 used only for characters with higher values. Even when Unicode property
240     support is available, PCRE supports case-insensitive matching only when
241 ph10 197 there is a one-to-one mapping between a letter's cases. There are a
242     small number of many-to-one mappings in Unicode; these are not sup-
243 nigel 87 ported by PCRE.
244 nigel 63
245    
246     AUTHOR
247    
248 nigel 77 Philip Hazel
249 ph10 99 University Computing Service
250 nigel 93 Cambridge CB2 3QH, England.
251 nigel 63
252 ph10 197 Putting an actual email address here seems to have been a spam magnet,
253     so I've taken it away. If you want to email me, use my two initials,
254 ph10 153 followed by the two digits 10, at the domain cam.ac.uk.
255 nigel 77
256 nigel 63
257 ph10 99 REVISION
258 nigel 63
259 ph10 211 Last updated: 09 August 2007
260 ph10 99 Copyright (c) 1997-2007 University of Cambridge.
261     ------------------------------------------------------------------------------
262 ph10 111
263    
264 nigel 79 PCREBUILD(3) PCREBUILD(3)
265 nigel 63
266 nigel 79
267 nigel 73 NAME
268     PCRE - Perl-compatible regular expressions
269    
270 nigel 77
271 nigel 63 PCRE BUILD-TIME OPTIONS
272    
273 nigel 73 This document describes the optional features of PCRE that can be
274     selected when the library is compiled. They are all selected, or dese-
275 nigel 75 lected, by providing options to the configure script that is run before
276     the make command. The complete list of options for configure (which
277     includes the standard ones such as the selection of the installation
278     directory) can be obtained by running
279 nigel 63
280 nigel 73 ./configure --help
281 nigel 63
282 ph10 128 The following sections include descriptions of options whose names
283     begin with --enable or --disable. These settings specify changes to the
284     defaults for the configure command. Because of the way that configure
285     works, --enable and --disable always come in pairs, so the complemen-
286     tary option always exists as well, but as it specifies the default, it
287     is not described.
288 nigel 63
289    
290 nigel 83 C++ SUPPORT
291    
292     By default, the configure script will search for a C++ compiler and C++
293     header files. If it finds them, it automatically builds the C++ wrapper
294     library for PCRE. You can disable this by adding
295    
296     --disable-cpp
297    
298     to the configure command.
299    
300    
301 nigel 63 UTF-8 SUPPORT
302    
303 nigel 73 To build PCRE with support for UTF-8 character strings, add
304 nigel 63
305 nigel 73 --enable-utf8
306 nigel 63
307 nigel 73 to the configure command. Of itself, this does not make PCRE treat
308     strings as UTF-8. As well as compiling PCRE with this option, you also
309     have have to set the PCRE_UTF8 option when you call the pcre_compile()
310     function.
311 nigel 63
312    
313 nigel 75 UNICODE CHARACTER PROPERTY SUPPORT
314    
315     UTF-8 support allows PCRE to process character values greater than 255
316     in the strings that it handles. On its own, however, it does not pro-
317     vide any facilities for accessing the properties of such characters. If
318     you want to be able to use the pattern escapes \P, \p, and \X, which
319     refer to Unicode character properties, you must add
320    
321     --enable-unicode-properties
322    
323     to the configure command. This implies UTF-8 support, even if you have
324     not explicitly requested it.
325    
326 ph10 128 Including Unicode property support adds around 30K of tables to the
327     PCRE library. Only the general category properties such as Lu and Nd
328     are supported. Details are given in the pcrepattern documentation.
329 nigel 75
330    
331 nigel 63 CODE VALUE OF NEWLINE
332    
333 ph10 128 By default, PCRE interprets character 10 (linefeed, LF) as indicating
334     the end of a line. This is the normal newline character on Unix-like
335 nigel 91 systems. You can compile PCRE to use character 13 (carriage return, CR)
336     instead, by adding
337 nigel 63
338 nigel 73 --enable-newline-is-cr
339 nigel 63
340 ph10 128 to the configure command. There is also a --enable-newline-is-lf
341 nigel 91 option, which explicitly specifies linefeed as the newline character.
342 nigel 63
343 nigel 91 Alternatively, you can specify that line endings are to be indicated by
344     the two character sequence CRLF. If you want this, add
345 nigel 63
346 nigel 91 --enable-newline-is-crlf
347    
348 nigel 93 to the configure command. There is a fourth option, specified by
349 nigel 91
350 ph10 150 --enable-newline-is-anycrlf
351    
352     which causes PCRE to recognize any of the three sequences CR, LF, or
353     CRLF as indicating a line ending. Finally, a fifth option, specified by
354    
355 nigel 93 --enable-newline-is-any
356 nigel 91
357 ph10 150 causes PCRE to recognize any Unicode newline sequence.
358 nigel 93
359 ph10 128 Whatever line ending convention is selected when PCRE is built can be
360     overridden when the library functions are called. At build time it is
361 nigel 93 conventional to use the standard for your operating system.
362    
363    
364 ph10 231 WHAT \R MATCHES
365    
366     By default, the sequence \R in a pattern matches any Unicode newline
367     sequence, whatever has been selected as the line ending sequence. If
368     you specify
369    
370     --enable-bsr-anycrlf
371    
372     the default is changed so that \R matches only CR, LF, or CRLF. What-
373     ever is selected when PCRE is built can be overridden when the library
374     functions are called.
375    
376    
377 nigel 63 BUILDING SHARED AND STATIC LIBRARIES
378    
379 ph10 128 The PCRE building process uses libtool to build both shared and static
380     Unix libraries by default. You can suppress one of these by adding one
381 nigel 73 of
382 nigel 63
383 nigel 73 --disable-shared
384     --disable-static
385 nigel 63
386 nigel 73 to the configure command, as required.
387 nigel 63
388    
389     POSIX MALLOC USAGE
390    
391 nigel 75 When PCRE is called through the POSIX interface (see the pcreposix doc-
392 ph10 128 umentation), additional working storage is required for holding the
393     pointers to capturing substrings, because PCRE requires three integers
394     per substring, whereas the POSIX interface provides only two. If the
395 nigel 73 number of expected substrings is small, the wrapper function uses space
396     on the stack, because this is faster than using malloc() for each call.
397     The default threshold above which the stack is no longer used is 10; it
398     can be changed by adding a setting such as
399 nigel 63
400 nigel 73 --with-posix-malloc-threshold=20
401 nigel 63
402 nigel 73 to the configure command.
403 nigel 63
404    
405     HANDLING VERY LARGE PATTERNS
406    
407 ph10 128 Within a compiled pattern, offset values are used to point from one
408     part to another (for example, from an opening parenthesis to an alter-
409     nation metacharacter). By default, two-byte values are used for these
410     offsets, leading to a maximum size for a compiled pattern of around
411     64K. This is sufficient to handle all but the most gigantic patterns.
412     Nevertheless, some people do want to process enormous patterns, so it
413     is possible to compile PCRE to use three-byte or four-byte offsets by
414 nigel 73 adding a setting such as
415 nigel 63
416 nigel 73 --with-link-size=3
417 nigel 63
418 ph10 128 to the configure command. The value given must be 2, 3, or 4. Using
419     longer offsets slows down the operation of PCRE because it has to load
420 nigel 73 additional bytes when handling them.
421 nigel 63
422    
423 nigel 73 AVOIDING EXCESSIVE STACK USAGE
424    
425 nigel 77 When matching with the pcre_exec() function, PCRE implements backtrack-
426 nigel 93 ing by making recursive calls to an internal function called match().
427     In environments where the size of the stack is limited, this can se-
428     verely limit PCRE's operation. (The Unix environment does not usually
429 nigel 91 suffer from this problem, but it may sometimes be necessary to increase
430 nigel 93 the maximum stack size. There is a discussion in the pcrestack docu-
431     mentation.) An alternative approach to recursion that uses memory from
432     the heap to remember data, instead of using recursive function calls,
433     has been implemented to work round the problem of limited stack size.
434 nigel 91 If you want to build a version of PCRE that works this way, add
435 nigel 73
436     --disable-stack-for-recursion
437    
438 nigel 93 to the configure command. With this configuration, PCRE will use the
439     pcre_stack_malloc and pcre_stack_free variables to call memory manage-
440 ph10 182 ment functions. By default these point to malloc() and free(), but you
441     can replace the pointers so that your own functions are used.
442 nigel 73
443 ph10 182 Separate functions are provided rather than using pcre_malloc and
444     pcre_free because the usage is very predictable: the block sizes
445     requested are always the same, and the blocks are always freed in
446     reverse order. A calling program might be able to implement optimized
447     functions that perform better than malloc() and free(). PCRE runs
448     noticeably more slowly when built in this way. This option affects only
449     the pcre_exec() function; it is not relevant for the the
450     pcre_dfa_exec() function.
451 nigel 73
452 ph10 182
453 nigel 91 LIMITING PCRE RESOURCE USAGE
454    
455 nigel 93 Internally, PCRE has a function called match(), which it calls repeat-
456     edly (sometimes recursively) when matching a pattern with the
457     pcre_exec() function. By controlling the maximum number of times this
458     function may be called during a single matching operation, a limit can
459     be placed on the resources used by a single call to pcre_exec(). The
460     limit can be changed at run time, as described in the pcreapi documen-
461     tation. The default is 10 million, but this can be changed by adding a
462 nigel 91 setting such as
463    
464     --with-match-limit=500000
465    
466 nigel 93 to the configure command. This setting has no effect on the
467 nigel 91 pcre_dfa_exec() matching function.
468    
469 nigel 93 In some environments it is desirable to limit the depth of recursive
470 nigel 91 calls of match() more strictly than the total number of calls, in order
471 nigel 93 to restrict the maximum amount of stack (or heap, if --disable-stack-
472 nigel 91 for-recursion is specified) that is used. A second limit controls this;
473 nigel 93 it defaults to the value that is set for --with-match-limit, which
474     imposes no additional constraints. However, you can set a lower limit
475 nigel 91 by adding, for example,
476    
477     --with-match-limit-recursion=10000
478    
479 nigel 93 to the configure command. This value can also be overridden at run
480 nigel 91 time.
481    
482    
483 ph10 128 CREATING CHARACTER TABLES AT BUILD TIME
484    
485     PCRE uses fixed tables for processing characters whose code values are
486     less than 256. By default, PCRE is built with a set of tables that are
487     distributed in the file pcre_chartables.c.dist. These tables are for
488     ASCII codes only. If you add
489    
490     --enable-rebuild-chartables
491    
492     to the configure command, the distributed tables are no longer used.
493     Instead, a program called dftables is compiled and run. This outputs
494     the source for new set of tables, created in the default locale of your
495     C runtime system. (This method of replacing the tables does not work if
496     you are cross compiling, because dftables is run on the local host. If
497     you need to create alternative tables when cross compiling, you will
498     have to do so "by hand".)
499    
500    
501 nigel 73 USING EBCDIC CODE
502    
503 ph10 128 PCRE assumes by default that it will run in an environment where the
504     character code is ASCII (or Unicode, which is a superset of ASCII).
505 ph10 197 This is the case for most computer operating systems. PCRE can, how-
506     ever, be compiled to run in an EBCDIC environment by adding
507 nigel 73
508     --enable-ebcdic
509    
510 ph10 128 to the configure command. This setting implies --enable-rebuild-charta-
511 ph10 197 bles. You should only use it if you know that you are in an EBCDIC
512     environment (for example, an IBM mainframe operating system).
513 nigel 73
514 nigel 93
515     SEE ALSO
516    
517     pcreapi(3), pcre_config(3).
518    
519 nigel 63
520 ph10 99 AUTHOR
521 nigel 63
522 ph10 99 Philip Hazel
523     University Computing Service
524     Cambridge CB2 3QH, England.
525    
526    
527     REVISION
528    
529 ph10 231 Last updated: 11 September 2007
530 ph10 99 Copyright (c) 1997-2007 University of Cambridge.
531     ------------------------------------------------------------------------------
532 ph10 111
533    
534 nigel 79 PCREMATCHING(3) PCREMATCHING(3)
535 nigel 63
536 nigel 79
537 nigel 77 NAME
538     PCRE - Perl-compatible regular expressions
539 nigel 73
540 nigel 77
541     PCRE MATCHING ALGORITHMS
542    
543     This document describes the two different algorithms that are available
544     in PCRE for matching a compiled regular expression against a given sub-
545     ject string. The "standard" algorithm is the one provided by the
546     pcre_exec() function. This works in the same was as Perl's matching
547     function, and provides a Perl-compatible matching operation.
548    
549     An alternative algorithm is provided by the pcre_dfa_exec() function;
550     this operates in a different way, and is not Perl-compatible. It has
551     advantages and disadvantages compared with the standard algorithm, and
552     these are described below.
553    
554     When there is only one possible way in which a given subject string can
555     match a pattern, the two algorithms give the same answer. A difference
556     arises, however, when there are multiple possibilities. For example, if
557     the pattern
558    
559     ^<.*>
560    
561     is matched against the string
562    
563     <something> <something else> <something further>
564    
565     there are three possible answers. The standard algorithm finds only one
566 nigel 93 of them, whereas the alternative algorithm finds all three.
567 nigel 77
568    
569     REGULAR EXPRESSIONS AS TREES
570    
571     The set of strings that are matched by a regular expression can be rep-
572     resented as a tree structure. An unlimited repetition in the pattern
573     makes the tree of infinite size, but it is still a tree. Matching the
574     pattern to a given subject string (from a given starting point) can be
575 nigel 91 thought of as a search of the tree. There are two ways to search a
576     tree: depth-first and breadth-first, and these correspond to the two
577     matching algorithms provided by PCRE.
578 nigel 77
579    
580     THE STANDARD MATCHING ALGORITHM
581    
582 ph10 148 In the terminology of Jeffrey Friedl's book "Mastering Regular Expres-
583     sions", the standard algorithm is an "NFA algorithm". It conducts a
584 nigel 77 depth-first search of the pattern tree. That is, it proceeds along a
585     single path through the tree, checking that the subject matches what is
586     required. When there is a mismatch, the algorithm tries any alterna-
587     tives at the current point, and if they all fail, it backs up to the
588     previous branch point in the tree, and tries the next alternative
589     branch at that level. This often involves backing up (moving to the
590     left) in the subject string as well. The order in which repetition
591     branches are tried is controlled by the greedy or ungreedy nature of
592     the quantifier.
593    
594     If a leaf node is reached, a matching string has been found, and at
595     that point the algorithm stops. Thus, if there is more than one possi-
596     ble match, this algorithm returns the first one that it finds. Whether
597     this is the shortest, the longest, or some intermediate length depends
598     on the way the greedy and ungreedy repetition quantifiers are specified
599     in the pattern.
600    
601     Because it ends up with a single path through the tree, it is rela-
602     tively straightforward for this algorithm to keep track of the sub-
603     strings that are matched by portions of the pattern in parentheses.
604     This provides support for capturing parentheses and back references.
605    
606    
607 nigel 93 THE ALTERNATIVE MATCHING ALGORITHM
608 nigel 77
609 nigel 93 This algorithm conducts a breadth-first search of the tree. Starting
610     from the first matching point in the subject, it scans the subject
611     string from left to right, once, character by character, and as it does
612     this, it remembers all the paths through the tree that represent valid
613     matches. In Friedl's terminology, this is a kind of "DFA algorithm",
614     though it is not implemented as a traditional finite state machine (it
615     keeps multiple states active simultaneously).
616 nigel 77
617 nigel 93 The scan continues until either the end of the subject is reached, or
618     there are no more unterminated paths. At this point, terminated paths
619     represent the different matching possibilities (if there are none, the
620     match has failed). Thus, if there is more than one possible match,
621 nigel 77 this algorithm finds all of them, and in particular, it finds the long-
622 nigel 93 est. In PCRE, there is an option to stop the algorithm after the first
623 nigel 77 match (which is necessarily the shortest) has been found.
624    
625     Note that all the matches that are found start at the same point in the
626     subject. If the pattern
627    
628     cat(er(pillar)?)
629    
630 nigel 93 is matched against the string "the caterpillar catchment", the result
631     will be the three strings "cat", "cater", and "caterpillar" that start
632 nigel 77 at the fourth character of the subject. The algorithm does not automat-
633     ically move on to find matches that start at later positions.
634    
635     There are a number of features of PCRE regular expressions that are not
636 nigel 93 supported by the alternative matching algorithm. They are as follows:
637 nigel 77
638 nigel 93 1. Because the algorithm finds all possible matches, the greedy or
639     ungreedy nature of repetition quantifiers is not relevant. Greedy and
640     ungreedy quantifiers are treated in exactly the same way. However, pos-
641     sessive quantifiers can make a difference when what follows could also
642     match what is quantified, for example in a pattern like this:
643 nigel 77
644 nigel 93 ^a++\w!
645    
646     This pattern matches "aaab!" but not "aaa!", which would be matched by
647     a non-possessive quantifier. Similarly, if an atomic group is present,
648     it is matched as if it were a standalone pattern at the current point,
649     and the longest match is then "locked in" for the rest of the overall
650     pattern.
651    
652 nigel 77 2. When dealing with multiple paths through the tree simultaneously, it
653 nigel 93 is not straightforward to keep track of captured substrings for the
654     different matching possibilities, and PCRE's implementation of this
655 nigel 77 algorithm does not attempt to do this. This means that no captured sub-
656     strings are available.
657    
658 nigel 93 3. Because no substrings are captured, back references within the pat-
659 nigel 77 tern are not supported, and cause errors if encountered.
660    
661 nigel 93 4. For the same reason, conditional expressions that use a backrefer-
662     ence as the condition or test for a specific group recursion are not
663     supported.
664 nigel 77
665 ph10 172 5. Because many paths through the tree may be active, the \K escape
666     sequence, which resets the start of the match when encountered (but may
667     be on some paths and not on others), is not supported. It causes an
668     error if encountered.
669    
670     6. Callouts are supported, but the value of the capture_top field is
671 nigel 77 always 1, and the value of the capture_last field is always -1.
672    
673 ph10 211 7. The \C escape sequence, which (in the standard algorithm) matches a
674 ph10 172 single byte, even in UTF-8 mode, is not supported because the alterna-
675     tive algorithm moves through the subject string one character at a
676 nigel 93 time, for all active paths through the tree.
677 nigel 77
678 ph10 211 8. None of the backtracking control verbs such as (*PRUNE) are sup-
679     ported.
680 nigel 77
681 ph10 211
682 nigel 93 ADVANTAGES OF THE ALTERNATIVE ALGORITHM
683 nigel 77
684 ph10 211 Using the alternative matching algorithm provides the following advan-
685 nigel 93 tages:
686 nigel 77
687     1. All possible matches (at a single point in the subject) are automat-
688 ph10 211 ically found, and in particular, the longest match is found. To find
689 nigel 77 more than one match using the standard algorithm, you have to do kludgy
690     things with callouts.
691    
692 ph10 211 2. There is much better support for partial matching. The restrictions
693     on the content of the pattern that apply when using the standard algo-
694     rithm for partial matching do not apply to the alternative algorithm.
695     For non-anchored patterns, the starting position of a partial match is
696 nigel 93 available.
697 nigel 77
698 ph10 211 3. Because the alternative algorithm scans the subject string just
699     once, and never needs to backtrack, it is possible to pass very long
700     subject strings to the matching function in several pieces, checking
701 nigel 93 for partial matching each time.
702 nigel 77
703    
704 nigel 93 DISADVANTAGES OF THE ALTERNATIVE ALGORITHM
705 nigel 77
706 nigel 93 The alternative algorithm suffers from a number of disadvantages:
707 nigel 77
708 ph10 211 1. It is substantially slower than the standard algorithm. This is
709     partly because it has to search for all possible matches, but is also
710 nigel 77 because it is less susceptible to optimization.
711    
712     2. Capturing parentheses and back references are not supported.
713    
714 nigel 93 3. Although atomic groups are supported, their use does not provide the
715     performance advantage that it does for the standard algorithm.
716 nigel 77
717    
718 ph10 99 AUTHOR
719 nigel 77
720 ph10 99 Philip Hazel
721     University Computing Service
722     Cambridge CB2 3QH, England.
723    
724    
725     REVISION
726    
727 ph10 211 Last updated: 08 August 2007
728 ph10 99 Copyright (c) 1997-2007 University of Cambridge.
729     ------------------------------------------------------------------------------
730 ph10 111
731    
732 nigel 79 PCREAPI(3) PCREAPI(3)
733 nigel 77
734 nigel 79
735 nigel 73 NAME
736     PCRE - Perl-compatible regular expressions
737    
738 nigel 77
739 nigel 75 PCRE NATIVE API
740 nigel 63
741 nigel 73 #include <pcre.h>
742 nigel 41
743 nigel 73 pcre *pcre_compile(const char *pattern, int options,
744     const char **errptr, int *erroffset,
745     const unsigned char *tableptr);
746 nigel 41
747 nigel 77 pcre *pcre_compile2(const char *pattern, int options,
748     int *errorcodeptr,
749     const char **errptr, int *erroffset,
750     const unsigned char *tableptr);
751    
752 nigel 73 pcre_extra *pcre_study(const pcre *code, int options,
753     const char **errptr);
754 nigel 41
755 nigel 73 int pcre_exec(const pcre *code, const pcre_extra *extra,
756     const char *subject, int length, int startoffset,
757     int options, int *ovector, int ovecsize);
758 nigel 41
759 nigel 77 int pcre_dfa_exec(const pcre *code, const pcre_extra *extra,
760     const char *subject, int length, int startoffset,
761     int options, int *ovector, int ovecsize,
762     int *workspace, int wscount);
763    
764 nigel 73 int pcre_copy_named_substring(const pcre *code,
765     const char *subject, int *ovector,
766     int stringcount, const char *stringname,
767     char *buffer, int buffersize);
768 nigel 63
769 nigel 73 int pcre_copy_substring(const char *subject, int *ovector,
770     int stringcount, int stringnumber, char *buffer,
771     int buffersize);
772 nigel 41
773 nigel 73 int pcre_get_named_substring(const pcre *code,
774     const char *subject, int *ovector,
775     int stringcount, const char *stringname,
776     const char **stringptr);
777 nigel 63
778 nigel 73 int pcre_get_stringnumber(const pcre *code,
779     const char *name);
780 nigel 63
781 nigel 91 int pcre_get_stringtable_entries(const pcre *code,
782     const char *name, char **first, char **last);
783    
784 nigel 73 int pcre_get_substring(const char *subject, int *ovector,
785     int stringcount, int stringnumber,
786     const char **stringptr);
787 nigel 41
788 nigel 73 int pcre_get_substring_list(const char *subject,
789     int *ovector, int stringcount, const char ***listptr);
790 nigel 41
791 nigel 73 void pcre_free_substring(const char *stringptr);
792 nigel 49
793 nigel 73 void pcre_free_substring_list(const char **stringptr);
794 nigel 49
795 nigel 73 const unsigned char *pcre_maketables(void);
796 nigel 41
797 nigel 73 int pcre_fullinfo(const pcre *code, const pcre_extra *extra,
798     int what, void *where);
799 nigel 43
800 nigel 73 int pcre_info(const pcre *code, int *optptr, int *firstcharptr);
801 nigel 63
802 nigel 77 int pcre_refcount(pcre *code, int adjust);
803    
804 nigel 73 int pcre_config(int what, void *where);
805 nigel 41
806 nigel 73 char *pcre_version(void);
807 nigel 63
808 nigel 73 void *(*pcre_malloc)(size_t);
809 nigel 41
810 nigel 73 void (*pcre_free)(void *);
811 nigel 41
812 nigel 73 void *(*pcre_stack_malloc)(size_t);
813 nigel 41
814 nigel 73 void (*pcre_stack_free)(void *);
815 nigel 41
816 nigel 73 int (*pcre_callout)(pcre_callout_block *);
817 nigel 41
818 nigel 73
819 nigel 75 PCRE API OVERVIEW
820 nigel 41
821 nigel 73 PCRE has its own native API, which is described in this document. There
822 nigel 93 are also some wrapper functions that correspond to the POSIX regular
823 nigel 77 expression API. These are described in the pcreposix documentation.
824     Both of these APIs define a set of C function calls. A C++ wrapper is
825     distributed with PCRE. It is documented in the pcrecpp page.
826 nigel 43
827 nigel 77 The native API C function prototypes are defined in the header file
828     pcre.h, and on Unix systems the library itself is called libpcre. It
829 nigel 75 can normally be accessed by adding -lpcre to the command for linking an
830     application that uses PCRE. The header file defines the macros
831     PCRE_MAJOR and PCRE_MINOR to contain the major and minor release num-
832     bers for the library. Applications can use these to include support
833     for different releases of PCRE.
834 nigel 41
835 nigel 77 The functions pcre_compile(), pcre_compile2(), pcre_study(), and
836     pcre_exec() are used for compiling and matching regular expressions in
837     a Perl-compatible manner. A sample program that demonstrates the sim-
838     plest way of using them is provided in the file called pcredemo.c in
839     the source distribution. The pcresample documentation describes how to
840     run it.
841 nigel 49
842 nigel 77 A second matching function, pcre_dfa_exec(), which is not Perl-compati-
843     ble, is also provided. This uses a different algorithm for the match-
844 nigel 91 ing. The alternative algorithm finds all possible matches (at a given
845 nigel 93 point in the subject), and scans the subject just once. However, this
846     algorithm does not return captured substrings. A description of the two
847     matching algorithms and their advantages and disadvantages is given in
848     the pcrematching documentation.
849 nigel 63
850 nigel 77 In addition to the main compiling and matching functions, there are
851     convenience functions for extracting captured substrings from a subject
852     string that is matched by pcre_exec(). They are:
853    
854 nigel 73 pcre_copy_substring()
855     pcre_copy_named_substring()
856     pcre_get_substring()
857     pcre_get_named_substring()
858     pcre_get_substring_list()
859 nigel 75 pcre_get_stringnumber()
860 nigel 91 pcre_get_stringtable_entries()
861 nigel 63
862 nigel 73 pcre_free_substring() and pcre_free_substring_list() are also provided,
863     to free the memory used for extracted strings.
864 nigel 41
865 nigel 77 The function pcre_maketables() is used to build a set of character
866     tables in the current locale for passing to pcre_compile(),
867     pcre_exec(), or pcre_dfa_exec(). This is an optional facility that is
868     provided for specialist use. Most commonly, no special tables are
869     passed, in which case internal tables that are generated when PCRE is
870     built are used.
871 nigel 49
872 nigel 75 The function pcre_fullinfo() is used to find out information about a
873     compiled pattern; pcre_info() is an obsolete version that returns only
874     some of the available information, but is retained for backwards com-
875     patibility. The function pcre_version() returns a pointer to a string
876 nigel 73 containing the version of PCRE and its date of release.
877 nigel 41
878 nigel 77 The function pcre_refcount() maintains a reference count in a data
879     block containing a compiled pattern. This is provided for the benefit
880     of object-oriented applications.
881    
882 nigel 75 The global variables pcre_malloc and pcre_free initially contain the
883     entry points of the standard malloc() and free() functions, respec-
884 nigel 73 tively. PCRE calls the memory management functions via these variables,
885 nigel 75 so a calling program can replace them if it wishes to intercept the
886 nigel 73 calls. This should be done before calling any PCRE functions.
887 nigel 41
888 nigel 75 The global variables pcre_stack_malloc and pcre_stack_free are also
889     indirections to memory management functions. These special functions
890     are used only when PCRE is compiled to use the heap for remembering
891 nigel 77 data, instead of recursive function calls, when running the pcre_exec()
892 nigel 91 function. See the pcrebuild documentation for details of how to do
893     this. It is a non-standard way of building PCRE, for use in environ-
894     ments that have limited stacks. Because of the greater use of memory
895     management, it runs more slowly. Separate functions are provided so
896     that special-purpose external code can be used for this case. When
897     used, these functions are always called in a stack-like manner (last
898     obtained, first freed), and always for memory blocks of the same size.
899     There is a discussion about PCRE's stack usage in the pcrestack docu-
900     mentation.
901 nigel 41
902 nigel 73 The global variable pcre_callout initially contains NULL. It can be set
903 nigel 77 by the caller to a "callout" function, which PCRE will then call at
904     specified points during a matching operation. Details are given in the
905 nigel 73 pcrecallout documentation.
906 nigel 41
907 nigel 73
908 nigel 91 NEWLINES
909    
910 ph10 150 PCRE supports five different conventions for indicating line breaks in
911 nigel 93 strings: a single CR (carriage return) character, a single LF (line-
912 ph10 150 feed) character, the two-character sequence CRLF, any of the three pre-
913     ceding, or any Unicode newline sequence. The Unicode newline sequences
914     are the three just mentioned, plus the single characters VT (vertical
915     tab, U+000B), FF (formfeed, U+000C), NEL (next line, U+0085), LS (line
916     separator, U+2028), and PS (paragraph separator, U+2029).
917 nigel 93
918     Each of the first three conventions is used by at least one operating
919     system as its standard newline sequence. When PCRE is built, a default
920     can be specified. The default default is LF, which is the Unix stan-
921     dard. When PCRE is run, the default can be overridden, either when a
922     pattern is compiled, or when it is matched.
923    
924 ph10 227 At compile time, the newline convention can be specified by the options
925     argument of pcre_compile(), or it can be specified by special text at
926     the start of the pattern itself; this overrides any other settings. See
927     the pcrepattern page for details of the special character sequences.
928    
929 nigel 91 In the PCRE documentation the word "newline" is used to mean "the char-
930 ph10 227 acter or pair of characters that indicate a line break". The choice of
931     newline convention affects the handling of the dot, circumflex, and
932 nigel 93 dollar metacharacters, the handling of #-comments in /x mode, and, when
933 ph10 227 CRLF is a recognized line ending sequence, the match position advance-
934     ment for a non-anchored pattern. There is more detail about this in the
935 ph10 231 section on pcre_exec() options below.
936 nigel 91
937 ph10 231 The choice of newline convention does not affect the interpretation of
938     the \n or \r escape sequences, nor does it affect what \R matches,
939     which is controlled in a similar way, but by separate options.
940 nigel 91
941 ph10 231
942 nigel 63 MULTITHREADING
943    
944 ph10 231 The PCRE functions can be used in multi-threading applications, with
945 nigel 73 the proviso that the memory management functions pointed to by
946     pcre_malloc, pcre_free, pcre_stack_malloc, and pcre_stack_free, and the
947     callout function pointed to by pcre_callout, are shared by all threads.
948 nigel 41
949 ph10 231 The compiled form of a regular expression is not altered during match-
950 nigel 73 ing, so the same compiled pattern can safely be used by several threads
951     at once.
952 nigel 41
953    
954 nigel 75 SAVING PRECOMPILED PATTERNS FOR LATER USE
955    
956     The compiled form of a regular expression can be saved and re-used at a
957 ph10 231 later time, possibly by a different program, and even on a host other
958     than the one on which it was compiled. Details are given in the
959     pcreprecompile documentation. However, compiling a regular expression
960     with one version of PCRE for use with a different version is not guar-
961 ph10 155 anteed to work and may cause crashes.
962 nigel 75
963    
964 nigel 63 CHECKING BUILD-TIME OPTIONS
965 nigel 41
966 nigel 73 int pcre_config(int what, void *where);
967 nigel 63
968 ph10 231 The function pcre_config() makes it possible for a PCRE client to dis-
969 nigel 73 cover which optional features have been compiled into the PCRE library.
970 ph10 231 The pcrebuild documentation has more details about these optional fea-
971 nigel 73 tures.
972 nigel 63
973 ph10 231 The first argument for pcre_config() is an integer, specifying which
974 nigel 73 information is required; the second argument is a pointer to a variable
975 ph10 231 into which the information is placed. The following information is
976 nigel 73 available:
977 nigel 63
978 nigel 73 PCRE_CONFIG_UTF8
979 nigel 63
980 ph10 231 The output is an integer that is set to one if UTF-8 support is avail-
981 nigel 73 able; otherwise it is set to zero.
982 nigel 63
983 nigel 75 PCRE_CONFIG_UNICODE_PROPERTIES
984    
985 ph10 231 The output is an integer that is set to one if support for Unicode
986 nigel 75 character properties is available; otherwise it is set to zero.
987    
988 nigel 73 PCRE_CONFIG_NEWLINE
989 nigel 63
990 ph10 231 The output is an integer whose value specifies the default character
991     sequence that is recognized as meaning "newline". The four values that
992 ph10 150 are supported are: 10 for LF, 13 for CR, 3338 for CRLF, -2 for ANYCRLF,
993 ph10 231 and -1 for ANY. The default should normally be the standard sequence
994 ph10 150 for your operating system.
995 nigel 63
996 ph10 231 PCRE_CONFIG_BSR
997    
998     The output is an integer whose value indicates what character sequences
999     the \R escape sequence matches by default. A value of 0 means that \R
1000     matches any Unicode line ending sequence; a value of 1 means that \R
1001     matches only CR, LF, or CRLF. The default can be overridden when a pat-
1002     tern is compiled or matched.
1003    
1004 nigel 73 PCRE_CONFIG_LINK_SIZE
1005 nigel 63
1006 ph10 231 The output is an integer that contains the number of bytes used for
1007 nigel 73 internal linkage in compiled regular expressions. The value is 2, 3, or
1008 ph10 231 4. Larger values allow larger regular expressions to be compiled, at
1009     the expense of slower matching. The default value of 2 is sufficient
1010     for all but the most massive patterns, since it allows the compiled
1011 nigel 73 pattern to be up to 64K in size.
1012 nigel 63
1013 nigel 73 PCRE_CONFIG_POSIX_MALLOC_THRESHOLD
1014 nigel 63
1015 ph10 231 The output is an integer that contains the threshold above which the
1016     POSIX interface uses malloc() for output vectors. Further details are
1017 nigel 73 given in the pcreposix documentation.
1018 nigel 63
1019 nigel 73 PCRE_CONFIG_MATCH_LIMIT
1020 nigel 63
1021 nigel 73 The output is an integer that gives the default limit for the number of
1022 ph10 231 internal matching function calls in a pcre_exec() execution. Further
1023 nigel 73 details are given with pcre_exec() below.
1024 nigel 63
1025 nigel 87 PCRE_CONFIG_MATCH_LIMIT_RECURSION
1026    
1027 ph10 231 The output is an integer that gives the default limit for the depth of
1028     recursion when calling the internal matching function in a pcre_exec()
1029 nigel 87 execution. Further details are given with pcre_exec() below.
1030    
1031 nigel 73 PCRE_CONFIG_STACKRECURSE
1032 nigel 63
1033 ph10 231 The output is an integer that is set to one if internal recursion when
1034 nigel 77 running pcre_exec() is implemented by recursive function calls that use
1035 ph10 231 the stack to remember their state. This is the usual way that PCRE is
1036 nigel 77 compiled. The output is zero if PCRE was compiled to use blocks of data
1037 ph10 231 on the heap instead of recursive function calls. In this case,
1038     pcre_stack_malloc and pcre_stack_free are called to manage memory
1039 nigel 77 blocks on the heap, thus avoiding the use of the stack.
1040 nigel 73
1041    
1042 nigel 41 COMPILING A PATTERN
1043 nigel 63
1044 nigel 73 pcre *pcre_compile(const char *pattern, int options,
1045     const char **errptr, int *erroffset,
1046     const unsigned char *tableptr);
1047 nigel 63
1048 nigel 77 pcre *pcre_compile2(const char *pattern, int options,
1049     int *errorcodeptr,
1050     const char **errptr, int *erroffset,
1051     const unsigned char *tableptr);
1052 nigel 41
1053 nigel 77 Either of the functions pcre_compile() or pcre_compile2() can be called
1054     to compile a pattern into an internal form. The only difference between
1055 ph10 231 the two interfaces is that pcre_compile2() has an additional argument,
1056 nigel 77 errorcodeptr, via which a numerical error code can be returned.
1057    
1058     The pattern is a C string terminated by a binary zero, and is passed in
1059 ph10 231 the pattern argument. A pointer to a single block of memory that is
1060     obtained via pcre_malloc is returned. This contains the compiled code
1061 nigel 77 and related data. The pcre type is defined for the returned block; this
1062     is a typedef for a structure whose contents are not externally defined.
1063 nigel 91 It is up to the caller to free the memory (via pcre_free) when it is no
1064     longer required.
1065 nigel 77
1066 ph10 231 Although the compiled code of a PCRE regex is relocatable, that is, it
1067 nigel 73 does not depend on memory location, the complete pcre data block is not
1068 ph10 231 fully relocatable, because it may contain a copy of the tableptr argu-
1069 nigel 75 ment, which is an address (see below).
1070 nigel 41
1071 nigel 93 The options argument contains various bit settings that affect the com-
1072 ph10 231 pilation. It should be zero if no options are required. The available
1073     options are described below. Some of them, in particular, those that
1074     are compatible with Perl, can also be set and unset from within the
1075     pattern (see the detailed description in the pcrepattern documenta-
1076     tion). For these options, the contents of the options argument speci-
1077     fies their initial settings at the start of compilation and execution.
1078     The PCRE_ANCHORED and PCRE_NEWLINE_xxx options can be set at the time
1079 nigel 91 of matching as well as at compile time.
1080 nigel 41
1081 nigel 73 If errptr is NULL, pcre_compile() returns NULL immediately. Otherwise,
1082 ph10 231 if compilation of a pattern fails, pcre_compile() returns NULL, and
1083 nigel 73 sets the variable pointed to by errptr to point to a textual error mes-
1084 nigel 87 sage. This is a static string that is part of the library. You must not
1085     try to free it. The offset from the start of the pattern to the charac-
1086     ter where the error was discovered is placed in the variable pointed to
1087 ph10 231 by erroffset, which must not be NULL. If it is, an immediate error is
1088 nigel 73 given.
1089 nigel 53
1090 ph10 231 If pcre_compile2() is used instead of pcre_compile(), and the error-
1091     codeptr argument is not NULL, a non-zero error code number is returned
1092     via this argument in the event of an error. This is in addition to the
1093 nigel 77 textual error message. Error codes and messages are listed below.
1094    
1095 ph10 231 If the final argument, tableptr, is NULL, PCRE uses a default set of
1096     character tables that are built when PCRE is compiled, using the
1097     default C locale. Otherwise, tableptr must be an address that is the
1098     result of a call to pcre_maketables(). This value is stored with the
1099     compiled pattern, and used again by pcre_exec(), unless another table
1100 nigel 75 pointer is passed to it. For more discussion, see the section on locale
1101     support below.
1102 nigel 53
1103 ph10 231 This code fragment shows a typical straightforward call to pcre_com-
1104 nigel 73 pile():
1105 nigel 41
1106 nigel 73 pcre *re;
1107     const char *error;
1108     int erroffset;
1109     re = pcre_compile(
1110     "^A.*Z", /* the pattern */
1111     0, /* default options */
1112     &error, /* for error message */
1113     &erroffset, /* for error offset */
1114     NULL); /* use default character tables */
1115 nigel 41
1116 ph10 231 The following names for option bits are defined in the pcre.h header
1117 nigel 75 file:
1118 nigel 41
1119 nigel 73 PCRE_ANCHORED
1120 nigel 41
1121 nigel 73 If this bit is set, the pattern is forced to be "anchored", that is, it
1122 ph10 231 is constrained to match only at the first matching point in the string
1123     that is being searched (the "subject string"). This effect can also be
1124     achieved by appropriate constructs in the pattern itself, which is the
1125 nigel 73 only way to do it in Perl.
1126 nigel 41
1127 nigel 75 PCRE_AUTO_CALLOUT
1128    
1129     If this bit is set, pcre_compile() automatically inserts callout items,
1130 ph10 231 all with number 255, before each pattern item. For discussion of the
1131 nigel 75 callout facility, see the pcrecallout documentation.
1132    
1133 ph10 231 PCRE_BSR_ANYCRLF
1134     PCRE_BSR_UNICODE
1135    
1136     These options (which are mutually exclusive) control what the \R escape
1137     sequence matches. The choice is either to match only CR, LF, or CRLF,
1138     or to match any Unicode newline sequence. The default is specified when
1139     PCRE is built. It can be overridden from within the pattern, or by set-
1140     ting an option when a compiled pattern is matched.
1141    
1142 nigel 73 PCRE_CASELESS
1143 nigel 41
1144 ph10 231 If this bit is set, letters in the pattern match both upper and lower
1145     case letters. It is equivalent to Perl's /i option, and it can be
1146     changed within a pattern by a (?i) option setting. In UTF-8 mode, PCRE
1147     always understands the concept of case for characters whose values are
1148     less than 128, so caseless matching is always possible. For characters
1149     with higher values, the concept of case is supported if PCRE is com-
1150     piled with Unicode property support, but not otherwise. If you want to
1151     use caseless matching for characters 128 and above, you must ensure
1152     that PCRE is compiled with Unicode property support as well as with
1153 nigel 77 UTF-8 support.
1154 nigel 41
1155 nigel 73 PCRE_DOLLAR_ENDONLY
1156 nigel 41
1157 ph10 231 If this bit is set, a dollar metacharacter in the pattern matches only
1158     at the end of the subject string. Without this option, a dollar also
1159     matches immediately before a newline at the end of the string (but not
1160     before any other newlines). The PCRE_DOLLAR_ENDONLY option is ignored
1161     if PCRE_MULTILINE is set. There is no equivalent to this option in
1162 nigel 91 Perl, and no way to set it within a pattern.
1163 nigel 41
1164 nigel 73 PCRE_DOTALL
1165 nigel 41
1166 nigel 73 If this bit is set, a dot metacharater in the pattern matches all char-
1167 ph10 231 acters, including those that indicate newline. Without it, a dot does
1168     not match when the current position is at a newline. This option is
1169     equivalent to Perl's /s option, and it can be changed within a pattern
1170     by a (?s) option setting. A negative class such as [^a] always matches
1171 nigel 93 newline characters, independent of the setting of this option.
1172 nigel 63
1173 nigel 91 PCRE_DUPNAMES
1174    
1175 ph10 231 If this bit is set, names used to identify capturing subpatterns need
1176 nigel 91 not be unique. This can be helpful for certain types of pattern when it
1177 ph10 231 is known that only one instance of the named subpattern can ever be
1178     matched. There are more details of named subpatterns below; see also
1179 nigel 91 the pcrepattern documentation.
1180    
1181 nigel 73 PCRE_EXTENDED
1182 nigel 41
1183 ph10 231 If this bit is set, whitespace data characters in the pattern are
1184 nigel 77 totally ignored except when escaped or inside a character class. White-
1185     space does not include the VT character (code 11). In addition, charac-
1186     ters between an unescaped # outside a character class and the next new-
1187 ph10 231 line, inclusive, are also ignored. This is equivalent to Perl's /x
1188     option, and it can be changed within a pattern by a (?x) option set-
1189 nigel 91 ting.
1190 nigel 41
1191 ph10 231 This option makes it possible to include comments inside complicated
1192     patterns. Note, however, that this applies only to data characters.
1193     Whitespace characters may never appear within special character
1194     sequences in a pattern, for example within the sequence (?( which
1195 nigel 73 introduces a conditional subpattern.
1196 nigel 41
1197 nigel 73 PCRE_EXTRA
1198 nigel 41
1199 ph10 231 This option was invented in order to turn on additional functionality
1200     of PCRE that is incompatible with Perl, but it is currently of very
1201     little use. When set, any backslash in a pattern that is followed by a
1202     letter that has no special meaning causes an error, thus reserving
1203     these combinations for future expansion. By default, as in Perl, a
1204     backslash followed by a letter with no special meaning is treated as a
1205     literal. (Perl can, however, be persuaded to give a warning for this.)
1206     There are at present no other features controlled by this option. It
1207 nigel 91 can also be set by a (?X) option setting within a pattern.
1208 nigel 41
1209 nigel 77 PCRE_FIRSTLINE
1210    
1211 ph10 231 If this option is set, an unanchored pattern is required to match
1212     before or at the first newline in the subject string, though the
1213 nigel 91 matched text may continue over the newline.
1214 nigel 77
1215 nigel 73 PCRE_MULTILINE
1216 nigel 41
1217 ph10 231 By default, PCRE treats the subject string as consisting of a single
1218     line of characters (even if it actually contains newlines). The "start
1219     of line" metacharacter (^) matches only at the start of the string,
1220     while the "end of line" metacharacter ($) matches only at the end of
1221 nigel 75 the string, or before a terminating newline (unless PCRE_DOLLAR_ENDONLY
1222     is set). This is the same as Perl.
1223 nigel 63
1224 ph10 231 When PCRE_MULTILINE it is set, the "start of line" and "end of line"
1225     constructs match immediately following or immediately before internal
1226     newlines in the subject string, respectively, as well as at the very
1227     start and end. This is equivalent to Perl's /m option, and it can be
1228 nigel 91 changed within a pattern by a (?m) option setting. If there are no new-
1229 ph10 231 lines in a subject string, or no occurrences of ^ or $ in a pattern,
1230 nigel 73 setting PCRE_MULTILINE has no effect.
1231 nigel 63
1232 nigel 91 PCRE_NEWLINE_CR
1233     PCRE_NEWLINE_LF
1234     PCRE_NEWLINE_CRLF
1235 ph10 150 PCRE_NEWLINE_ANYCRLF
1236 nigel 93 PCRE_NEWLINE_ANY
1237 nigel 91
1238 ph10 231 These options override the default newline definition that was chosen
1239     when PCRE was built. Setting the first or the second specifies that a
1240     newline is indicated by a single character (CR or LF, respectively).
1241     Setting PCRE_NEWLINE_CRLF specifies that a newline is indicated by the
1242     two-character CRLF sequence. Setting PCRE_NEWLINE_ANYCRLF specifies
1243 ph10 150 that any of the three preceding sequences should be recognized. Setting
1244 ph10 231 PCRE_NEWLINE_ANY specifies that any Unicode newline sequence should be
1245 ph10 150 recognized. The Unicode newline sequences are the three just mentioned,
1246 ph10 231 plus the single characters VT (vertical tab, U+000B), FF (formfeed,
1247     U+000C), NEL (next line, U+0085), LS (line separator, U+2028), and PS
1248     (paragraph separator, U+2029). The last two are recognized only in
1249 ph10 150 UTF-8 mode.
1250 nigel 91
1251 ph10 231 The newline setting in the options word uses three bits that are
1252 ph10 150 treated as a number, giving eight possibilities. Currently only six are
1253 ph10 231 used (default plus the five values above). This means that if you set
1254     more than one newline option, the combination may or may not be sensi-
1255 ph10 150 ble. For example, PCRE_NEWLINE_CR with PCRE_NEWLINE_LF is equivalent to
1256 ph10 231 PCRE_NEWLINE_CRLF, but other combinations may yield unused numbers and
1257 ph10 150 cause an error.
1258 nigel 91
1259 ph10 231 The only time that a line break is specially recognized when compiling
1260     a pattern is if PCRE_EXTENDED is set, and an unescaped # outside a
1261     character class is encountered. This indicates a comment that lasts
1262     until after the next line break sequence. In other circumstances, line
1263     break sequences are treated as literal data, except that in
1264 nigel 93 PCRE_EXTENDED mode, both CR and LF are treated as whitespace characters
1265     and are therefore ignored.
1266    
1267     The newline option that is set at compile time becomes the default that
1268 ph10 231 is used for pcre_exec() and pcre_dfa_exec(), but it can be overridden.
1269 nigel 93
1270 nigel 73 PCRE_NO_AUTO_CAPTURE
1271 nigel 41
1272 nigel 73 If this option is set, it disables the use of numbered capturing paren-
1273 ph10 231 theses in the pattern. Any opening parenthesis that is not followed by
1274     ? behaves as if it were followed by ?: but named parentheses can still
1275     be used for capturing (and they acquire numbers in the usual way).
1276 nigel 73 There is no equivalent of this option in Perl.
1277 nigel 41
1278 nigel 73 PCRE_UNGREEDY
1279 nigel 41
1280 ph10 231 This option inverts the "greediness" of the quantifiers so that they
1281     are not greedy by default, but become greedy if followed by "?". It is
1282     not compatible with Perl. It can also be set by a (?U) option setting
1283 nigel 73 within the pattern.
1284 nigel 41
1285 nigel 73 PCRE_UTF8
1286 nigel 49
1287 ph10 231 This option causes PCRE to regard both the pattern and the subject as
1288     strings of UTF-8 characters instead of single-byte character strings.
1289     However, it is available only when PCRE is built to include UTF-8 sup-
1290     port. If not, the use of this option provokes an error. Details of how
1291     this option changes the behaviour of PCRE are given in the section on
1292 nigel 75 UTF-8 support in the main pcre page.
1293 nigel 71
1294 nigel 73 PCRE_NO_UTF8_CHECK
1295 nigel 71
1296 nigel 73 When PCRE_UTF8 is set, the validity of the pattern as a UTF-8 string is
1297 ph10 231 automatically checked. There is a discussion about the validity of
1298     UTF-8 strings in the main pcre page. If an invalid UTF-8 sequence of
1299     bytes is found, pcre_compile() returns an error. If you already know
1300 ph10 211 that your pattern is valid, and you want to skip this check for perfor-
1301 ph10 231 mance reasons, you can set the PCRE_NO_UTF8_CHECK option. When it is
1302     set, the effect of passing an invalid UTF-8 string as a pattern is
1303     undefined. It may cause your program to crash. Note that this option
1304     can also be passed to pcre_exec() and pcre_dfa_exec(), to suppress the
1305 ph10 211 UTF-8 validity checking of subject strings.
1306 nigel 71
1307 nigel 73
1308 nigel 77 COMPILATION ERROR CODES
1309    
1310 ph10 231 The following table lists the error codes than may be returned by
1311     pcre_compile2(), along with the error messages that may be returned by
1312     both compiling functions. As PCRE has developed, some error codes have
1313 nigel 93 fallen out of use. To avoid confusion, they have not been re-used.
1314 nigel 77
1315     0 no error
1316     1 \ at end of pattern
1317     2 \c at end of pattern
1318     3 unrecognized character follows \
1319     4 numbers out of order in {} quantifier
1320     5 number too big in {} quantifier
1321     6 missing terminating ] for character class
1322     7 invalid escape sequence in character class
1323     8 range out of order in character class
1324     9 nothing to repeat
1325 nigel 93 10 [this code is not in use]
1326 nigel 77 11 internal error: unexpected repeat
1327     12 unrecognized character after (?
1328     13 POSIX named classes are supported only within a class
1329     14 missing )
1330     15 reference to non-existent subpattern
1331     16 erroffset passed as NULL
1332     17 unknown option bit(s) set
1333     18 missing ) after comment
1334 nigel 93 19 [this code is not in use]
1335 nigel 77 20 regular expression too large
1336     21 failed to get memory
1337     22 unmatched parentheses
1338     23 internal error: code overflow
1339     24 unrecognized character after (?<
1340     25 lookbehind assertion is not fixed length
1341 nigel 91 26 malformed number or name after (?(
1342 nigel 77 27 conditional group contains more than two branches
1343     28 assertion expected after (?(
1344 ph10 182 29 (?R or (?[+-]digits must be followed by )
1345 nigel 77 30 unknown POSIX class name
1346     31 POSIX collating elements are not supported
1347     32 this version of PCRE is not compiled with PCRE_UTF8 support
1348 nigel 93 33 [this code is not in use]
1349 nigel 77 34 character value in \x{...} sequence is too large
1350     35 invalid condition (?(0)
1351     36 \C not allowed in lookbehind assertion
1352     37 PCRE does not support \L, \l, \N, \U, or \u
1353     38 number after (?C is > 255
1354     39 closing ) for (?C expected
1355     40 recursive call could loop indefinitely
1356     41 unrecognized character after (?P
1357 nigel 93 42 syntax error in subpattern name (missing terminator)
1358 nigel 91 43 two named subpatterns have the same name
1359 nigel 77 44 invalid UTF-8 string
1360     45 support for \P, \p, and \X has not been compiled
1361     46 malformed \P or \p sequence
1362     47 unknown property name after \P or \p
1363 nigel 91 48 subpattern name is too long (maximum 32 characters)
1364     49 too many named subpatterns (maximum 10,000)
1365 ph10 202 50 [this code is not in use]
1366 nigel 91 51 octal value is greater than \377 (not in UTF-8 mode)
1367 nigel 93 52 internal error: overran compiling workspace
1368 ph10 231 53 internal error: previously-checked referenced subpattern not
1369 nigel 93 found
1370     54 DEFINE group contains more than one branch
1371     55 repeating a DEFINE group is not allowed
1372 ph10 231 56 inconsistent NEWLINE options
1373 ph10 182 57 \g is not followed by a braced name or an optionally braced
1374     non-zero number
1375     58 (?+ or (?- or (?(+ or (?(- must be followed by a non-zero number
1376 nigel 77
1377    
1378 nigel 63 STUDYING A PATTERN
1379 nigel 49
1380 nigel 77 pcre_extra *pcre_study(const pcre *code, int options
1381 nigel 73 const char **errptr);
1382 nigel 63
1383 ph10 231 If a compiled pattern is going to be used several times, it is worth
1384 nigel 75 spending more time analyzing it in order to speed up the time taken for
1385 ph10 231 matching. The function pcre_study() takes a pointer to a compiled pat-
1386 nigel 75 tern as its first argument. If studying the pattern produces additional
1387 ph10 231 information that will help speed up matching, pcre_study() returns a
1388     pointer to a pcre_extra block, in which the study_data field points to
1389 nigel 75 the results of the study.
1390 nigel 41
1391 nigel 75 The returned value from pcre_study() can be passed directly to
1392 ph10 231 pcre_exec(). However, a pcre_extra block also contains other fields
1393     that can be set by the caller before the block is passed; these are
1394 nigel 75 described below in the section on matching a pattern.
1395 nigel 63
1396 ph10 231 If studying the pattern does not produce any additional information
1397 nigel 75 pcre_study() returns NULL. In that circumstance, if the calling program
1398 ph10 231 wants to pass any of the other fields to pcre_exec(), it must set up
1399 nigel 75 its own pcre_extra block.
1400 nigel 41
1401 ph10 231 The second argument of pcre_study() contains option bits. At present,
1402 nigel 75 no options are defined, and this argument should always be zero.
1403    
1404 ph10 231 The third argument for pcre_study() is a pointer for an error message.
1405     If studying succeeds (even if no data is returned), the variable it
1406     points to is set to NULL. Otherwise it is set to point to a textual
1407 nigel 87 error message. This is a static string that is part of the library. You
1408 ph10 231 must not try to free it. You should test the error pointer for NULL
1409 nigel 87 after calling pcre_study(), to be sure that it has run successfully.
1410 nigel 41
1411 nigel 73 This is a typical call to pcre_study():
1412 nigel 53
1413 nigel 73 pcre_extra *pe;
1414     pe = pcre_study(
1415     re, /* result of pcre_compile() */
1416     0, /* no options exist */
1417     &error); /* set to NULL or points to a message */
1418 nigel 53
1419 nigel 73 At present, studying a pattern is useful only for non-anchored patterns
1420 ph10 231 that do not have a single fixed starting character. A bitmap of possi-
1421 nigel 75 ble starting bytes is created.
1422 nigel 41
1423    
1424 nigel 63 LOCALE SUPPORT
1425 nigel 41
1426 ph10 231 PCRE handles caseless matching, and determines whether characters are
1427     letters, digits, or whatever, by reference to a set of tables, indexed
1428     by character value. When running in UTF-8 mode, this applies only to
1429     characters with codes less than 128. Higher-valued codes never match
1430     escapes such as \w or \d, but can be tested with \p if PCRE is built
1431     with Unicode character property support. The use of locales with Uni-
1432     code is discouraged. If you are handling characters with codes greater
1433     than 128, you should either use UTF-8 and Unicode, or use locales, but
1434 ph10 142 not try to mix the two.
1435 nigel 41
1436 ph10 231 PCRE contains an internal set of tables that are used when the final
1437     argument of pcre_compile() is NULL. These are sufficient for many
1438 ph10 142 applications. Normally, the internal tables recognize only ASCII char-
1439     acters. However, when PCRE is built, it is possible to cause the inter-
1440     nal tables to be rebuilt in the default "C" locale of the local system,
1441     which may cause them to be different.
1442 nigel 41
1443 ph10 231 The internal tables can always be overridden by tables supplied by the
1444 ph10 142 application that calls PCRE. These may be created in a different locale
1445 ph10 231 from the default. As more and more applications change to using Uni-
1446 ph10 142 code, the need for this locale support is expected to die away.
1447    
1448 ph10 231 External tables are built by calling the pcre_maketables() function,
1449     which has no arguments, in the relevant locale. The result can then be
1450     passed to pcre_compile() or pcre_exec() as often as necessary. For
1451     example, to build and use tables that are appropriate for the French
1452     locale (where accented characters with values greater than 128 are
1453 nigel 75 treated as letters), the following code could be used:
1454    
1455     setlocale(LC_CTYPE, "fr_FR");
1456 nigel 73 tables = pcre_maketables();
1457     re = pcre_compile(..., tables);
1458 nigel 41
1459 ph10 231 The locale name "fr_FR" is used on Linux and other Unix-like systems;
1460 ph10 142 if you are using Windows, the name for the French locale is "french".
1461    
1462 ph10 231 When pcre_maketables() runs, the tables are built in memory that is
1463     obtained via pcre_malloc. It is the caller's responsibility to ensure
1464     that the memory containing the tables remains available for as long as
1465 nigel 75 it is needed.
1466 nigel 41
1467 nigel 75 The pointer that is passed to pcre_compile() is saved with the compiled
1468 ph10 231 pattern, and the same tables are used via this pointer by pcre_study()
1469 nigel 75 and normally also by pcre_exec(). Thus, by default, for any single pat-
1470     tern, compilation, studying and matching all happen in the same locale,
1471     but different patterns can be compiled in different locales.
1472 nigel 41
1473 ph10 231 It is possible to pass a table pointer or NULL (indicating the use of
1474     the internal tables) to pcre_exec(). Although not intended for this
1475     purpose, this facility could be used to match a pattern in a different
1476 nigel 75 locale from the one in which it was compiled. Passing table pointers at
1477     run time is discussed below in the section on matching a pattern.
1478    
1479    
1480 nigel 63 INFORMATION ABOUT A PATTERN
1481 nigel 41
1482 nigel 73 int pcre_fullinfo(const pcre *code, const pcre_extra *extra,
1483     int what, void *where);
1484 nigel 63
1485 ph10 231 The pcre_fullinfo() function returns information about a compiled pat-
1486 nigel 73 tern. It replaces the obsolete pcre_info() function, which is neverthe-
1487     less retained for backwards compability (and is documented below).
1488 nigel 43
1489 ph10 231 The first argument for pcre_fullinfo() is a pointer to the compiled
1490     pattern. The second argument is the result of pcre_study(), or NULL if
1491     the pattern was not studied. The third argument specifies which piece
1492     of information is required, and the fourth argument is a pointer to a
1493     variable to receive the data. The yield of the function is zero for
1494 nigel 73 success, or one of the following negative numbers:
1495 nigel 41
1496 nigel 73 PCRE_ERROR_NULL the argument code was NULL
1497     the argument where was NULL
1498     PCRE_ERROR_BADMAGIC the "magic number" was not found
1499     PCRE_ERROR_BADOPTION the value of what was invalid
1500 nigel 53
1501 ph10 231 The "magic number" is placed at the start of each compiled pattern as
1502     an simple check against passing an arbitrary memory pointer. Here is a
1503     typical call of pcre_fullinfo(), to obtain the length of the compiled
1504 nigel 75 pattern:
1505 nigel 53
1506 nigel 73 int rc;
1507 nigel 91 size_t length;
1508 nigel 73 rc = pcre_fullinfo(
1509     re, /* result of pcre_compile() */
1510     pe, /* result of pcre_study(), or NULL */
1511     PCRE_INFO_SIZE, /* what is required */
1512     &length); /* where to put the data */
1513 nigel 43
1514 ph10 231 The possible values for the third argument are defined in pcre.h, and
1515 nigel 73 are as follows:
1516 nigel 43
1517 nigel 73 PCRE_INFO_BACKREFMAX
1518 nigel 41
1519 ph10 231 Return the number of the highest back reference in the pattern. The
1520     fourth argument should point to an int variable. Zero is returned if
1521 nigel 73 there are no back references.
1522 nigel 43
1523 nigel 73 PCRE_INFO_CAPTURECOUNT
1524 nigel 43
1525 ph10 231 Return the number of capturing subpatterns in the pattern. The fourth
1526 nigel 73 argument should point to an int variable.
1527 nigel 43
1528 nigel 77 PCRE_INFO_DEFAULT_TABLES
1529 nigel 75
1530 ph10 231 Return a pointer to the internal default character tables within PCRE.
1531     The fourth argument should point to an unsigned char * variable. This
1532 nigel 75 information call is provided for internal use by the pcre_study() func-
1533 ph10 231 tion. External callers can cause PCRE to use its internal tables by
1534 nigel 75 passing a NULL table pointer.
1535    
1536 nigel 73 PCRE_INFO_FIRSTBYTE
1537 nigel 43
1538 ph10 231 Return information about the first byte of any matched string, for a
1539     non-anchored pattern. The fourth argument should point to an int vari-
1540     able. (This option used to be called PCRE_INFO_FIRSTCHAR; the old name
1541 nigel 91 is still recognized for backwards compatibility.)
1542 nigel 41
1543 ph10 231 If there is a fixed first byte, for example, from a pattern such as
1544 nigel 93 (cat|cow|coyote), its value is returned. Otherwise, if either
1545 nigel 41
1546 ph10 231 (a) the pattern was compiled with the PCRE_MULTILINE option, and every
1547 nigel 73 branch starts with "^", or
1548 nigel 43
1549 nigel 73 (b) every branch of the pattern starts with ".*" and PCRE_DOTALL is not
1550     set (if it were set, the pattern would be anchored),
1551 nigel 41
1552 ph10 231 -1 is returned, indicating that the pattern matches only at the start
1553     of a subject string or after any newline within the string. Otherwise
1554 nigel 73 -2 is returned. For anchored patterns, -2 is returned.
1555 nigel 41
1556 nigel 73 PCRE_INFO_FIRSTTABLE
1557 nigel 41
1558 ph10 231 If the pattern was studied, and this resulted in the construction of a
1559 nigel 73 256-bit table indicating a fixed set of bytes for the first byte in any
1560 ph10 231 matching string, a pointer to the table is returned. Otherwise NULL is
1561     returned. The fourth argument should point to an unsigned char * vari-
1562 nigel 73 able.
1563 nigel 43
1564 ph10 227 PCRE_INFO_HASCRORLF
1565    
1566 ph10 231 Return 1 if the pattern contains any explicit matches for CR or LF
1567     characters, otherwise 0. The fourth argument should point to an int
1568 ph10 243 variable. An explicit match is either a literal CR or LF character, or
1569     \r or \n.
1570 ph10 227
1571 ph10 172 PCRE_INFO_JCHANGED
1572    
1573 ph10 243 Return 1 if the (?J) option setting is used in the pattern, otherwise
1574 ph10 172 0. The fourth argument should point to an int variable. The (?J) inter-
1575 ph10 182 nal option setting changes the local PCRE_DUPNAMES option.
1576 ph10 172
1577 nigel 73 PCRE_INFO_LASTLITERAL
1578 nigel 43
1579 ph10 243 Return the value of the rightmost literal byte that must exist in any
1580     matched string, other than at its start, if such a byte has been
1581 nigel 73 recorded. The fourth argument should point to an int variable. If there
1582 ph10 243 is no such byte, -1 is returned. For anchored patterns, a last literal
1583     byte is recorded only if it follows something of variable length. For
1584 nigel 73 example, for the pattern /^a\d+z\d+/ the returned value is "z", but for
1585     /^a\dz\d/ the returned value is -1.
1586 nigel 63
1587 nigel 73 PCRE_INFO_NAMECOUNT
1588     PCRE_INFO_NAMEENTRYSIZE
1589     PCRE_INFO_NAMETABLE
1590 nigel 63
1591 ph10 243 PCRE supports the use of named as well as numbered capturing parenthe-
1592     ses. The names are just an additional way of identifying the parenthe-
1593 nigel 91 ses, which still acquire numbers. Several convenience functions such as
1594 ph10 243 pcre_get_named_substring() are provided for extracting captured sub-
1595     strings by name. It is also possible to extract the data directly, by
1596     first converting the name to a number in order to access the correct
1597 nigel 91 pointers in the output vector (described with pcre_exec() below). To do
1598 ph10 243 the conversion, you need to use the name-to-number map, which is
1599 nigel 91 described by these three values.
1600 nigel 63
1601 nigel 73 The map consists of a number of fixed-size entries. PCRE_INFO_NAMECOUNT
1602     gives the number of entries, and PCRE_INFO_NAMEENTRYSIZE gives the size
1603 ph10 243 of each entry; both of these return an int value. The entry size
1604     depends on the length of the longest name. PCRE_INFO_NAMETABLE returns
1605     a pointer to the first entry of the table (a pointer to char). The
1606 nigel 73 first two bytes of each entry are the number of the capturing parenthe-
1607 ph10 243 sis, most significant byte first. The rest of the entry is the corre-
1608     sponding name, zero terminated. The names are in alphabetical order.
1609 nigel 91 When PCRE_DUPNAMES is set, duplicate names are in order of their paren-
1610 ph10 243 theses numbers. For example, consider the following pattern (assume
1611     PCRE_EXTENDED is set, so white space - including newlines - is
1612 nigel 91 ignored):
1613 nigel 63
1614 nigel 93 (?<date> (?<year>(\d\d)?\d\d) -
1615     (?<month>\d\d) - (?<day>\d\d) )
1616 nigel 63
1617 ph10 243 There are four named subpatterns, so the table has four entries, and
1618     each entry in the table is eight bytes long. The table is as follows,
1619 nigel 75 with non-printing bytes shows in hexadecimal, and undefined bytes shown
1620     as ??:
1621 nigel 63
1622 nigel 73 00 01 d a t e 00 ??
1623     00 05 d a y 00 ?? ??
1624     00 04 m o n t h 00
1625     00 02 y e a r 00 ??
1626 nigel 63
1627 ph10 243 When writing code to extract data from named subpatterns using the
1628     name-to-number map, remember that the length of the entries is likely
1629 nigel 91 to be different for each compiled pattern.
1630 nigel 63
1631 ph10 172 PCRE_INFO_OKPARTIAL
1632    
1633 ph10 243 Return 1 if the pattern can be used for partial matching, otherwise 0.
1634     The fourth argument should point to an int variable. The pcrepartial
1635     documentation lists the restrictions that apply to patterns when par-
1636 ph10 172 tial matching is used.
1637    
1638 nigel 73 PCRE_INFO_OPTIONS
1639 nigel 63
1640 ph10 243 Return a copy of the options with which the pattern was compiled. The
1641     fourth argument should point to an unsigned long int variable. These
1642 nigel 73 option bits are those specified in the call to pcre_compile(), modified
1643 ph10 197 by any top-level option settings at the start of the pattern itself. In
1644 ph10 243 other words, they are the options that will be in force when matching
1645     starts. For example, if the pattern /(?im)abc(?-i)d/ is compiled with
1646     the PCRE_EXTENDED option, the result is PCRE_CASELESS, PCRE_MULTILINE,
1647 ph10 197 and PCRE_EXTENDED.
1648 nigel 63
1649 ph10 243 A pattern is automatically anchored by PCRE if all of its top-level
1650 nigel 73 alternatives begin with one of the following:
1651 nigel 63
1652 nigel 73 ^ unless PCRE_MULTILINE is set
1653     \A always
1654     \G always
1655     .* if PCRE_DOTALL is set and there are no back
1656     references to the subpattern in which .* appears
1657 nigel 63
1658 nigel 73 For such patterns, the PCRE_ANCHORED bit is set in the options returned
1659     by pcre_fullinfo().
1660 nigel 63
1661 nigel 73 PCRE_INFO_SIZE
1662 nigel 63
1663 ph10 243 Return the size of the compiled pattern, that is, the value that was
1664 nigel 73 passed as the argument to pcre_malloc() when PCRE was getting memory in
1665     which to place the compiled data. The fourth argument should point to a
1666     size_t variable.
1667 nigel 63
1668 nigel 73 PCRE_INFO_STUDYSIZE
1669 nigel 63
1670 nigel 75 Return the size of the data block pointed to by the study_data field in
1671 ph10 243 a pcre_extra block. That is, it is the value that was passed to
1672 nigel 73 pcre_malloc() when PCRE was getting memory into which to place the data
1673 ph10 243 created by pcre_study(). The fourth argument should point to a size_t
1674 nigel 73 variable.
1675 nigel 63
1676 nigel 73
1677 nigel 63 OBSOLETE INFO FUNCTION
1678    
1679 nigel 73 int pcre_info(const pcre *code, int *optptr, int *firstcharptr);
1680 nigel 63
1681 ph10 243 The pcre_info() function is now obsolete because its interface is too
1682     restrictive to return all the available data about a compiled pattern.
1683     New programs should use pcre_fullinfo() instead. The yield of
1684     pcre_info() is the number of capturing subpatterns, or one of the fol-
1685 nigel 73 lowing negative numbers:
1686 nigel 43
1687 nigel 73 PCRE_ERROR_NULL the argument code was NULL
1688     PCRE_ERROR_BADMAGIC the "magic number" was not found
1689 nigel 43
1690 ph10 243 If the optptr argument is not NULL, a copy of the options with which
1691     the pattern was compiled is placed in the integer it points to (see
1692 nigel 73 PCRE_INFO_OPTIONS above).
1693 nigel 43
1694 ph10 243 If the pattern is not anchored and the firstcharptr argument is not
1695     NULL, it is used to pass back information about the first character of
1696 nigel 73 any matched string (see PCRE_INFO_FIRSTBYTE above).
1697 nigel 43
1698    
1699 nigel 77 REFERENCE COUNTS
1700 nigel 53
1701 nigel 77 int pcre_refcount(pcre *code, int adjust);
1702    
1703 ph10 243 The pcre_refcount() function is used to maintain a reference count in
1704 nigel 77 the data block that contains a compiled pattern. It is provided for the
1705 ph10 243 benefit of applications that operate in an object-oriented manner,
1706 nigel 77 where different parts of the application may be using the same compiled
1707     pattern, but you want to free the block when they are all done.
1708    
1709     When a pattern is compiled, the reference count field is initialized to
1710 ph10 243 zero. It is changed only by calling this function, whose action is to
1711     add the adjust value (which may be positive or negative) to it. The
1712 nigel 77 yield of the function is the new value. However, the value of the count
1713 ph10 243 is constrained to lie between 0 and 65535, inclusive. If the new value
1714 nigel 77 is outside these limits, it is forced to the appropriate limit value.
1715    
1716 ph10 243 Except when it is zero, the reference count is not correctly preserved
1717     if a pattern is compiled on one host and then transferred to a host
1718 nigel 77 whose byte-order is different. (This seems a highly unlikely scenario.)
1719    
1720    
1721     MATCHING A PATTERN: THE TRADITIONAL FUNCTION
1722    
1723 nigel 73 int pcre_exec(const pcre *code, const pcre_extra *extra,
1724     const char *subject, int length, int startoffset,
1725     int options, int *ovector, int ovecsize);
1726 nigel 53
1727 ph10 243 The function pcre_exec() is called to match a subject string against a
1728     compiled pattern, which is passed in the code argument. If the pattern
1729 nigel 75 has been studied, the result of the study should be passed in the extra
1730 ph10 243 argument. This function is the main matching facility of the library,
1731 nigel 77 and it operates in a Perl-like manner. For specialist use there is also
1732 ph10 243 an alternative matching function, which is described below in the sec-
1733 nigel 77 tion about the pcre_dfa_exec() function.
1734 nigel 41
1735 ph10 243 In most applications, the pattern will have been compiled (and option-
1736     ally studied) in the same process that calls pcre_exec(). However, it
1737 nigel 75 is possible to save compiled patterns and study data, and then use them
1738 ph10 243 later in different processes, possibly even on different hosts. For a
1739 nigel 75 discussion about this, see the pcreprecompile documentation.
1740    
1741 nigel 73 Here is an example of a simple call to pcre_exec():
1742 nigel 53
1743 nigel 73 int rc;
1744     int ovector[30];
1745     rc = pcre_exec(
1746     re, /* result of pcre_compile() */
1747     NULL, /* we didn't study the pattern */
1748     "some string", /* the subject string */
1749     11, /* the length of the subject string */
1750     0, /* start at offset 0 in the subject */
1751     0, /* default options */
1752 nigel 75 ovector, /* vector of integers for substring information */
1753 nigel 77 30); /* number of elements (NOT size in bytes) */
1754 nigel 53
1755 nigel 75 Extra data for pcre_exec()
1756 nigel 63
1757 ph10 243 If the extra argument is not NULL, it must point to a pcre_extra data
1758     block. The pcre_study() function returns such a block (when it doesn't
1759     return NULL), but you can also create one for yourself, and pass addi-
1760     tional information in it. The pcre_extra block contains the following
1761 nigel 87 fields (not necessarily in this order):
1762 nigel 75
1763 nigel 73 unsigned long int flags;
1764     void *study_data;
1765     unsigned long int match_limit;
1766 nigel 87 unsigned long int match_limit_recursion;
1767 nigel 73 void *callout_data;
1768 nigel 75 const unsigned char *tables;
1769 nigel 63
1770 ph10 243 The flags field is a bitmap that specifies which of the other fields
1771 nigel 73 are set. The flag bits are:
1772 nigel 63
1773 nigel 73 PCRE_EXTRA_STUDY_DATA
1774     PCRE_EXTRA_MATCH_LIMIT
1775 nigel 87 PCRE_EXTRA_MATCH_LIMIT_RECURSION
1776 nigel 73 PCRE_EXTRA_CALLOUT_DATA
1777 nigel 75 PCRE_EXTRA_TABLES
1778 nigel 63
1779 ph10 243 Other flag bits should be set to zero. The study_data field is set in
1780     the pcre_extra block that is returned by pcre_study(), together with
1781 nigel 75 the appropriate flag bit. You should not set this yourself, but you may
1782 ph10 243 add to the block by setting the other fields and their corresponding
1783 nigel 75 flag bits.
1784 nigel 63
1785 nigel 73 The match_limit field provides a means of preventing PCRE from using up
1786 ph10 243 a vast amount of resources when running patterns that are not going to
1787     match, but which have a very large number of possibilities in their
1788     search trees. The classic example is the use of nested unlimited
1789 nigel 75 repeats.
1790 nigel 63
1791 ph10 243 Internally, PCRE uses a function called match() which it calls repeat-
1792     edly (sometimes recursively). The limit set by match_limit is imposed
1793     on the number of times this function is called during a match, which
1794     has the effect of limiting the amount of backtracking that can take
1795 nigel 87 place. For patterns that are not anchored, the count restarts from zero
1796     for each position in the subject string.
1797 nigel 75
1798 ph10 243 The default value for the limit can be set when PCRE is built; the
1799     default default is 10 million, which handles all but the most extreme
1800     cases. You can override the default by suppling pcre_exec() with a
1801     pcre_extra block in which match_limit is set, and
1802     PCRE_EXTRA_MATCH_LIMIT is set in the flags field. If the limit is
1803 nigel 73 exceeded, pcre_exec() returns PCRE_ERROR_MATCHLIMIT.
1804 nigel 63
1805 ph10 243 The match_limit_recursion field is similar to match_limit, but instead
1806 nigel 87 of limiting the total number of times that match() is called, it limits
1807 ph10 243 the depth of recursion. The recursion depth is a smaller number than
1808     the total number of calls, because not all calls to match() are recur-
1809 nigel 87 sive. This limit is of use only if it is set smaller than match_limit.
1810    
1811 ph10 243 Limiting the recursion depth limits the amount of stack that can be
1812 nigel 87 used, or, when PCRE has been compiled to use memory on the heap instead
1813     of the stack, the amount of heap memory that can be used.
1814    
1815 ph10 243 The default value for match_limit_recursion can be set when PCRE is
1816     built; the default default is the same value as the default for
1817     match_limit. You can override the default by suppling pcre_exec() with
1818     a pcre_extra block in which match_limit_recursion is set, and
1819     PCRE_EXTRA_MATCH_LIMIT_RECURSION is set in the flags field. If the
1820 nigel 87 limit is exceeded, pcre_exec() returns PCRE_ERROR_RECURSIONLIMIT.
1821    
1822 ph10 243 The pcre_callout field is used in conjunction with the "callout" fea-
1823 nigel 73 ture, which is described in the pcrecallout documentation.
1824 nigel 63
1825 ph10 243 The tables field is used to pass a character tables pointer to
1826     pcre_exec(); this overrides the value that is stored with the compiled
1827     pattern. A non-NULL value is stored with the compiled pattern only if
1828     custom tables were supplied to pcre_compile() via its tableptr argu-
1829 nigel 75 ment. If NULL is passed to pcre_exec() using this mechanism, it forces
1830 ph10 243 PCRE's internal tables to be used. This facility is helpful when re-
1831     using patterns that have been saved after compiling with an external
1832     set of tables, because the external tables might be at a different
1833     address when pcre_exec() is called. See the pcreprecompile documenta-
1834 nigel 75 tion for a discussion of saving compiled patterns for later use.
1835 nigel 41
1836 nigel 75 Option bits for pcre_exec()
1837 nigel 71
1838 ph10 243 The unused bits of the options argument for pcre_exec() must be zero.
1839     The only bits that may be set are PCRE_ANCHORED, PCRE_NEWLINE_xxx,
1840 nigel 91 PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK and
1841     PCRE_PARTIAL.
1842 nigel 41
1843 nigel 75 PCRE_ANCHORED
1844 nigel 41
1845 ph10 243 The PCRE_ANCHORED option limits pcre_exec() to matching at the first
1846     matching position. If a pattern was compiled with PCRE_ANCHORED, or
1847     turned out to be anchored by virtue of its contents, it cannot be made
1848 nigel 75 unachored at matching time.
1849    
1850 ph10 231 PCRE_BSR_ANYCRLF
1851     PCRE_BSR_UNICODE
1852    
1853     These options (which are mutually exclusive) control what the \R escape
1854 ph10 243 sequence matches. The choice is either to match only CR, LF, or CRLF,
1855     or to match any Unicode newline sequence. These options override the
1856 ph10 231 choice that was made or defaulted when the pattern was compiled.
1857    
1858 nigel 91 PCRE_NEWLINE_CR
1859     PCRE_NEWLINE_LF
1860     PCRE_NEWLINE_CRLF
1861 ph10 150 PCRE_NEWLINE_ANYCRLF
1862 nigel 93 PCRE_NEWLINE_ANY
1863 nigel 91
1864 ph10 243 These options override the newline definition that was chosen or
1865     defaulted when the pattern was compiled. For details, see the descrip-
1866     tion of pcre_compile() above. During matching, the newline choice
1867     affects the behaviour of the dot, circumflex, and dollar metacharac-
1868     ters. It may also alter the way the match position is advanced after a
1869 ph10 227 match failure for an unanchored pattern.
1870 nigel 91
1871 ph10 243 When PCRE_NEWLINE_CRLF, PCRE_NEWLINE_ANYCRLF, or PCRE_NEWLINE_ANY is
1872     set, and a match attempt for an unanchored pattern fails when the cur-
1873     rent position is at a CRLF sequence, and the pattern contains no
1874     explicit matches for CR or LF characters, the match position is
1875 ph10 227 advanced by two characters instead of one, in other words, to after the
1876     CRLF.
1877    
1878     The above rule is a compromise that makes the most common cases work as
1879 ph10 243 expected. For example, if the pattern is .+A (and the PCRE_DOTALL
1880 ph10 227 option is not set), it does not match the string "\r\nA" because, after
1881 ph10 243 failing at the start, it skips both the CR and the LF before retrying.
1882     However, the pattern [\r\n]A does match that string, because it con-
1883 ph10 227 tains an explicit CR or LF reference, and so advances only by one char-
1884 ph10 231 acter after the first failure.
1885 ph10 227
1886 ph10 231 An explicit match for CR of LF is either a literal appearance of one of
1887 ph10 243 those characters, or one of the \r or \n escape sequences. Implicit
1888     matches such as [^X] do not count, nor does \s (which includes CR and
1889 ph10 231 LF in the characters that it matches).
1890    
1891 ph10 243 Notwithstanding the above, anomalous effects may still occur when CRLF
1892 ph10 227 is a valid newline sequence and explicit \r or \n escapes appear in the
1893     pattern.
1894    
1895 nigel 73 PCRE_NOTBOL
1896 nigel 41
1897 nigel 75 This option specifies that first character of the subject string is not
1898 ph10 243 the beginning of a line, so the circumflex metacharacter should not
1899     match before it. Setting this without PCRE_MULTILINE (at compile time)
1900     causes circumflex never to match. This option affects only the behav-
1901 nigel 77 iour of the circumflex metacharacter. It does not affect \A.
1902 nigel 41
1903 nigel 73 PCRE_NOTEOL
1904 nigel 41
1905 nigel 75 This option specifies that the end of the subject string is not the end
1906 ph10 243 of a line, so the dollar metacharacter should not match it nor (except
1907     in multiline mode) a newline immediately before it. Setting this with-
1908 nigel 75 out PCRE_MULTILINE (at compile time) causes dollar never to match. This
1909 ph10 243 option affects only the behaviour of the dollar metacharacter. It does
1910 nigel 75 not affect \Z or \z.
1911 nigel 41
1912 nigel 73 PCRE_NOTEMPTY
1913 nigel 41
1914 nigel 73 An empty string is not considered to be a valid match if this option is
1915 ph10 243 set. If there are alternatives in the pattern, they are tried. If all
1916     the alternatives match the empty string, the entire match fails. For
1917 nigel 73 example, if the pattern
1918 nigel 41
1919 nigel 73 a?b?
1920 nigel 41
1921 ph10 243 is applied to a string not beginning with "a" or "b", it matches the
1922     empty string at the start of the subject. With PCRE_NOTEMPTY set, this
1923 nigel 73 match is not valid, so PCRE searches further into the string for occur-
1924     rences of "a" or "b".
1925 nigel 41
1926 nigel 73 Perl has no direct equivalent of PCRE_NOTEMPTY, but it does make a spe-
1927 ph10 243 cial case of a pattern match of the empty string within its split()
1928     function, and when using the /g modifier. It is possible to emulate
1929 nigel 73 Perl's behaviour after matching a null string by first trying the match
1930 nigel 75 again at the same offset with PCRE_NOTEMPTY and PCRE_ANCHORED, and then
1931 ph10 243 if that fails by advancing the starting offset (see below) and trying
1932 nigel 75 an ordinary match again. There is some code that demonstrates how to do
1933     this in the pcredemo.c sample program.
1934 nigel 41
1935 nigel 75 PCRE_NO_UTF8_CHECK
1936    
1937     When PCRE_UTF8 is set at compile time, the validity of the subject as a
1938 ph10 243 UTF-8 string is automatically checked when pcre_exec() is subsequently
1939     called. The value of startoffset is also checked to ensure that it
1940     points to the start of a UTF-8 character. There is a discussion about
1941     the validity of UTF-8 strings in the section on UTF-8 support in the
1942     main pcre page. If an invalid UTF-8 sequence of bytes is found,
1943     pcre_exec() returns the error PCRE_ERROR_BADUTF8. If startoffset con-
1944 ph10 211 tains an invalid value, PCRE_ERROR_BADUTF8_OFFSET is returned.
1945 nigel 75
1946 ph10 243 If you already know that your subject is valid, and you want to skip
1947     these checks for performance reasons, you can set the
1948     PCRE_NO_UTF8_CHECK option when calling pcre_exec(). You might want to
1949     do this for the second and subsequent calls to pcre_exec() if you are
1950     making repeated calls to find all the matches in a single subject
1951     string. However, you should be sure that the value of startoffset
1952     points to the start of a UTF-8 character. When PCRE_NO_UTF8_CHECK is
1953     set, the effect of passing an invalid UTF-8 string as a subject, or a
1954     value of startoffset that does not point to the start of a UTF-8 char-
1955 nigel 75 acter, is undefined. Your program may crash.
1956    
1957     PCRE_PARTIAL
1958    
1959 ph10 243 This option turns on the partial matching feature. If the subject
1960     string fails to match the pattern, but at some point during the match-
1961     ing process the end of the subject was reached (that is, the subject
1962     partially matches the pattern and the failure to match occurred only
1963     because there were not enough subject characters), pcre_exec() returns
1964     PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH. When PCRE_PARTIAL is
1965     used, there are restrictions on what may appear in the pattern. These
1966 nigel 75 are discussed in the pcrepartial documentation.
1967    
1968     The string to be matched by pcre_exec()
1969    
1970 ph10 243 The subject string is passed to pcre_exec() as a pointer in subject, a
1971     length in length, and a starting byte offset in startoffset. In UTF-8
1972     mode, the byte offset must point to the start of a UTF-8 character.
1973     Unlike the pattern string, the subject may contain binary zero bytes.
1974     When the starting offset is zero, the search for a match starts at the
1975 nigel 75 beginning of the subject, and this is by far the most common case.
1976 nigel 63
1977 ph10 243 A non-zero starting offset is useful when searching for another match
1978     in the same subject by calling pcre_exec() again after a previous suc-
1979     cess. Setting startoffset differs from just passing over a shortened
1980     string and setting PCRE_NOTBOL in the case of a pattern that begins
1981 nigel 73 with any kind of lookbehind. For example, consider the pattern
1982 nigel 41
1983 nigel 73 \Biss\B
1984 nigel 41
1985 ph10 243 which finds occurrences of "iss" in the middle of words. (\B matches
1986     only if the current position in the subject is not a word boundary.)
1987     When applied to the string "Mississipi" the first call to pcre_exec()
1988     finds the first occurrence. If pcre_exec() is called again with just
1989     the remainder of the subject, namely "issipi", it does not match,
1990 nigel 73 because \B is always false at the start of the subject, which is deemed
1991 ph10 243 to be a word boundary. However, if pcre_exec() is passed the entire
1992 nigel 75 string again, but with startoffset set to 4, it finds the second occur-
1993 ph10 243 rence of "iss" because it is able to look behind the starting point to
1994 nigel 75 discover that it is preceded by a letter.
1995 nigel 41
1996 ph10 243 If a non-zero starting offset is passed when the pattern is anchored,
1997 nigel 75 one attempt to match at the given offset is made. This can only succeed
1998 ph10 243 if the pattern does not require the match to be at the start of the
1999 nigel 75 subject.
2000 nigel 41
2001 nigel 75 How pcre_exec() returns captured substrings
2002    
2003 ph10 243 In general, a pattern matches a certain portion of the subject, and in
2004     addition, further substrings from the subject may be picked out by
2005     parts of the pattern. Following the usage in Jeffrey Friedl's book,
2006     this is called "capturing" in what follows, and the phrase "capturing
2007     subpattern" is used for a fragment of a pattern that picks out a sub-
2008     string. PCRE supports several other kinds of parenthesized subpattern
2009 nigel 73 that do not cause substrings to be captured.
2010 nigel 65
2011 ph10 243 Captured substrings are returned to the caller via a vector of integer
2012     offsets whose address is passed in ovector. The number of elements in
2013     the vector is passed in ovecsize, which must be a non-negative number.
2014 nigel 75 Note: this argument is NOT the size of ovector in bytes.
2015 nigel 41
2016 ph10 243 The first two-thirds of the vector is used to pass back captured sub-
2017     strings, each substring using a pair of integers. The remaining third
2018     of the vector is used as workspace by pcre_exec() while matching cap-
2019     turing subpatterns, and is not available for passing back information.
2020     The length passed in ovecsize should always be a multiple of three. If
2021 nigel 75 it is not, it is rounded down.
2022    
2023 ph10 243 When a match is successful, information about captured substrings is
2024     returned in pairs of integers, starting at the beginning of ovector,
2025     and continuing up to two-thirds of its length at the most. The first
2026 nigel 73 element of a pair is set to the offset of the first character in a sub-
2027 ph10 243 string, and the second is set to the offset of the first character
2028     after the end of a substring. The first pair, ovector[0] and ovec-
2029     tor[1], identify the portion of the subject string matched by the
2030     entire pattern. The next pair is used for the first capturing subpat-
2031 nigel 91 tern, and so on. The value returned by pcre_exec() is one more than the
2032     highest numbered pair that has been set. For example, if two substrings
2033 ph10 243 have been captured, the returned value is 3. If there are no capturing
2034     subpatterns, the return value from a successful match is 1, indicating
2035 nigel 91 that just the first pair of offsets has been set.
2036 nigel 41
2037 nigel 73 If a capturing subpattern is matched repeatedly, it is the last portion
2038 nigel 75 of the string that it matched that is returned.
2039 nigel 41
2040 ph10 243 If the vector is too small to hold all the captured substring offsets,
2041 nigel 75 it is used as far as possible (up to two-thirds of its length), and the
2042 ph10 243 function returns a value of zero. In particular, if the substring off-
2043 nigel 75 sets are not of interest, pcre_exec() may be called with ovector passed
2044 ph10 243 as NULL and ovecsize as zero. However, if the pattern contains back
2045     references and the ovector is not big enough to remember the related
2046     substrings, PCRE has to get additional memory for use during matching.
2047 nigel 73 Thus it is usually advisable to supply an ovector.
2048 nigel 41
2049 ph10 243 The pcre_info() function can be used to find out how many capturing
2050     subpatterns there are in a compiled pattern. The smallest size for
2051     ovector that will allow for n captured substrings, in addition to the
2052 nigel 91 offsets of the substring matched by the whole pattern, is (n+1)*3.
2053 nigel 41
2054 ph10 243 It is possible for capturing subpattern number n+1 to match some part
2055 nigel 91 of the subject when subpattern n has not been used at all. For example,
2056 ph10 243 if the string "abc" is matched against the pattern (a|(z))(bc) the
2057 nigel 91 return from the function is 4, and subpatterns 1 and 3 are matched, but
2058 ph10 243 2 is not. When this happens, both values in the offset pairs corre-
2059 nigel 91 sponding to unused subpatterns are set to -1.
2060 nigel 75
2061 ph10 243 Offset values that correspond to unused subpatterns at the end of the
2062     expression are also set to -1. For example, if the string "abc" is
2063     matched against the pattern (abc)(x(yz)?)? subpatterns 2 and 3 are not
2064     matched. The return from the function is 2, because the highest used
2065 nigel 91 capturing subpattern number is 1. However, you can refer to the offsets
2066 ph10 243 for the second and third capturing subpatterns if you wish (assuming
2067 nigel 91 the vector is large enough, of course).
2068    
2069 ph10 243 Some convenience functions are provided for extracting the captured
2070 nigel 91 substrings as separate strings. These are described below.
2071    
2072     Error return values from pcre_exec()
2073    
2074 ph10 243 If pcre_exec() fails, it returns a negative number. The following are
2075 nigel 73 defined in the header file:
2076 nigel 41
2077 nigel 73 PCRE_ERROR_NOMATCH (-1)
2078 nigel 41
2079 nigel 73 The subject string did not match the pattern.
2080 nigel 41
2081 nigel 73 PCRE_ERROR_NULL (-2)
2082 nigel 41
2083 ph10 243 Either code or subject was passed as NULL, or ovector was NULL and
2084 nigel 73 ovecsize was not zero.
2085 nigel 41
2086 nigel 73 PCRE_ERROR_BADOPTION (-3)
2087 nigel 41
2088 nigel 73 An unrecognized bit was set in the options argument.
2089 nigel 41
2090 nigel 73 PCRE_ERROR_BADMAGIC (-4)
2091 nigel 41
2092 ph10 243 PCRE stores a 4-byte "magic number" at the start of the compiled code,
2093 nigel 75 to catch the case when it is passed a junk pointer and to detect when a
2094     pattern that was compiled in an environment of one endianness is run in
2095 ph10 243 an environment with the other endianness. This is the error that PCRE
2096 nigel 75 gives when the magic number is not present.
2097 nigel 41
2098 nigel 93 PCRE_ERROR_UNKNOWN_OPCODE (-5)
2099 nigel 41
2100 nigel 73 While running the pattern match, an unknown item was encountered in the
2101 ph10 243 compiled pattern. This error could be caused by a bug in PCRE or by
2102 nigel 73 overwriting of the compiled pattern.
2103 nigel 41
2104 nigel 73 PCRE_ERROR_NOMEMORY (-6)
2105 nigel 41
2106 ph10 243 If a pattern contains back references, but the ovector that is passed
2107 nigel 73 to pcre_exec() is not big enough to remember the referenced substrings,
2108 ph10 243 PCRE gets a block of memory at the start of matching to use for this
2109     purpose. If the call via pcre_malloc() fails, this error is given. The
2110 nigel 75 memory is automatically freed at the end of matching.
2111 nigel 41
2112 nigel 73 PCRE_ERROR_NOSUBSTRING (-7)
2113 nigel 53
2114 ph10 243 This error is used by the pcre_copy_substring(), pcre_get_substring(),
2115 nigel 73 and pcre_get_substring_list() functions (see below). It is never
2116     returned by pcre_exec().
2117 nigel 63
2118 nigel 73 PCRE_ERROR_MATCHLIMIT (-8)
2119 nigel 63
2120 ph10 243 The backtracking limit, as specified by the match_limit field in a
2121     pcre_extra structure (or defaulted) was reached. See the description
2122 nigel 87 above.
2123    
2124 nigel 73 PCRE_ERROR_CALLOUT (-9)
2125 nigel 63
2126 nigel 73 This error is never generated by pcre_exec() itself. It is provided for
2127 ph10 243 use by callout functions that want to yield a distinctive error code.
2128 nigel 73 See the pcrecallout documentation for details.
2129 nigel 71
2130 nigel 73 PCRE_ERROR_BADUTF8 (-10)
2131 nigel 71
2132 ph10 243 A string that contains an invalid UTF-8 byte sequence was passed as a
2133 nigel 73 subject.
2134    
2135     PCRE_ERROR_BADUTF8_OFFSET (-11)
2136    
2137     The UTF-8 byte sequence that was passed as a subject was valid, but the
2138 ph10 243 value of startoffset did not point to the beginning of a UTF-8 charac-
2139 nigel 73 ter.
2140    
2141 nigel 77 PCRE_ERROR_PARTIAL (-12)
2142 nigel 73
2143 ph10 243 The subject string did not match, but it did match partially. See the
2144 nigel 75 pcrepartial documentation for details of partial matching.
2145    
2146 nigel 77 PCRE_ERROR_BADPARTIAL (-13)
2147 nigel 75
2148 ph10 243 The PCRE_PARTIAL option was used with a compiled pattern containing
2149     items that are not supported for partial matching. See the pcrepartial
2150 nigel 75 documentation for details of partial matching.
2151    
2152 nigel 77 PCRE_ERROR_INTERNAL (-14)
2153 nigel 75
2154 ph10 243 An unexpected internal error has occurred. This error could be caused
2155 nigel 75 by a bug in PCRE or by overwriting of the compiled pattern.
2156    
2157 nigel 77 PCRE_ERROR_BADCOUNT (-15)
2158 nigel 75
2159 ph10 243 This error is given if the value of the ovecsize argument is negative.
2160 nigel 75
2161 nigel 93 PCRE_ERROR_RECURSIONLIMIT (-21)
2162 nigel 75
2163 nigel 93 The internal recursion limit, as specified by the match_limit_recursion
2164 ph10 243 field in a pcre_extra structure (or defaulted) was reached. See the
2165 nigel 93 description above.
2166    
2167     PCRE_ERROR_BADNEWLINE (-23)
2168    
2169     An invalid combination of PCRE_NEWLINE_xxx options was given.
2170    
2171 ph10 197 Error numbers -16 to -20 and -22 are not used by pcre_exec().
2172 nigel 93
2173    
2174 nigel 63 EXTRACTING CAPTURED SUBSTRINGS BY NUMBER
2175    
2176 nigel 73 int pcre_copy_substring(const char *subject, int *ovector,
2177     int stringcount, int stringnumber, char *buffer,
2178     int buffersize);
2179 nigel 63
2180 nigel 73 int pcre_get_substring(const char *subject, int *ovector,
2181     int stringcount, int stringnumber,
2182     const char **stringptr);
2183 nigel 63
2184 nigel 73 int pcre_get_substring_list(const char *subject,
2185     int *ovector, int stringcount, const char ***listptr);
2186 nigel 63
2187 ph10 243 Captured substrings can be accessed directly by using the offsets
2188     returned by pcre_exec() in ovector. For convenience, the functions
2189 nigel 73 pcre_copy_substring(), pcre_get_substring(), and pcre_get_sub-
2190 ph10 243 string_list() are provided for extracting captured substrings as new,
2191     separate, zero-terminated strings. These functions identify substrings
2192     by number. The next section describes functions for extracting named
2193 nigel 91 substrings.
2194 nigel 41
2195 ph10 243 A substring that contains a binary zero is correctly extracted and has
2196     a further zero added on the end, but the result is not, of course, a C
2197     string. However, you can process such a string by referring to the
2198     length that is returned by pcre_copy_substring() and pcre_get_sub-
2199 nigel 91 string(). Unfortunately, the interface to pcre_get_substring_list() is
2200 ph10 243 not adequate for handling strings containing binary zeros, because the
2201 nigel 91 end of the final string is not independently indicated.
2202    
2203 ph10 243 The first three arguments are the same for all three of these func-
2204     tions: subject is the subject string that has just been successfully
2205 nigel 73 matched, ovector is a pointer to the vector of integer offsets that was
2206     passed to pcre_exec(), and stringcount is the number of substrings that
2207 ph10 243 were captured by the match, including the substring that matched the
2208 nigel 75 entire regular expression. This is the value returned by pcre_exec() if
2209 ph10 243 it is greater than zero. If pcre_exec() returned zero, indicating that
2210     it ran out of space in ovector, the value passed as stringcount should
2211 nigel 75 be the number of elements in the vector divided by three.
2212 nigel 41
2213 ph10 243 The functions pcre_copy_substring() and pcre_get_substring() extract a
2214     single substring, whose number is given as stringnumber. A value of
2215     zero extracts the substring that matched the entire pattern, whereas
2216     higher values extract the captured substrings. For pcre_copy_sub-
2217     string(), the string is placed in buffer, whose length is given by
2218     buffersize, while for pcre_get_substring() a new block of memory is
2219     obtained via pcre_malloc, and its address is returned via stringptr.
2220     The yield of the function is the length of the string, not including
2221 nigel 93 the terminating zero, or one of these error codes:
2222 nigel 41
2223 nigel 73 PCRE_ERROR_NOMEMORY (-6)
2224 nigel 41
2225 ph10 243 The buffer was too small for pcre_copy_substring(), or the attempt to
2226 nigel 73 get memory failed for pcre_get_substring().
2227 nigel 41
2228 nigel 73 PCRE_ERROR_NOSUBSTRING (-7)
2229 nigel 41
2230 nigel 73 There is no substring whose number is stringnumber.
2231 nigel 41
2232 ph10 243 The pcre_get_substring_list() function extracts all available sub-
2233     strings and builds a list of pointers to them. All this is done in a
2234 nigel 75 single block of memory that is obtained via pcre_malloc. The address of
2235 ph10 243 the memory block is returned via listptr, which is also the start of
2236     the list of string pointers. The end of the list is marked by a NULL
2237     pointer. The yield of the function is zero if all went well, or the
2238 nigel 93 error code
2239 nigel 41
2240 nigel 73 PCRE_ERROR_NOMEMORY (-6)
2241 nigel 41
2242 nigel 73 if the attempt to get the memory block failed.
2243 nigel 41
2244 ph10 243 When any of these functions encounter a substring that is unset, which
2245     can happen when capturing subpattern number n+1 matches some part of
2246     the subject, but subpattern n has not been used at all, they return an
2247 nigel 73 empty string. This can be distinguished from a genuine zero-length sub-
2248 ph10 243 string by inspecting the appropriate offset in ovector, which is nega-
2249 nigel 73 tive for unset substrings.
2250 nigel 41
2251 ph10 243 The two convenience functions pcre_free_substring() and pcre_free_sub-
2252     string_list() can be used to free the memory returned by a previous
2253 nigel 75 call of pcre_get_substring() or pcre_get_substring_list(), respec-
2254 ph10 243 tively. They do nothing more than call the function pointed to by
2255     pcre_free, which of course could be called directly from a C program.
2256     However, PCRE is used in some situations where it is linked via a spe-
2257     cial interface to another programming language that cannot use
2258     pcre_free directly; it is for these cases that the functions are pro-
2259 nigel 77 vided.
2260 nigel 41
2261 nigel 73
2262 nigel 63 EXTRACTING CAPTURED SUBSTRINGS BY NAME
2263 nigel 41
2264 nigel 75 int pcre_get_stringnumber(const pcre *code,
2265     const char *name);
2266    
2267 nigel 73 int pcre_copy_named_substring(const pcre *code,
2268     const char *subject, int *ovector,
2269     int stringcount, const char *stringname,
2270     char *buffer, int buffersize);
2271 nigel 41
2272 nigel 73 int pcre_get_named_substring(const pcre *code,
2273     const char *subject, int *ovector,
2274     int stringcount, const char *stringname,
2275     const char **stringptr);
2276 nigel 41
2277 ph10 243 To extract a substring by name, you first have to find associated num-
2278 nigel 75 ber. For example, for this pattern
2279 nigel 41
2280 nigel 93 (a+)b(?<xxx>\d+)...
2281 nigel 63
2282 nigel 91 the number of the subpattern called "xxx" is 2. If the name is known to
2283     be unique (PCRE_DUPNAMES was not set), you can find the number from the
2284     name by calling pcre_get_stringnumber(). The first argument is the com-
2285     piled pattern, and the second is the name. The yield of the function is
2286 ph10 243 the subpattern number, or PCRE_ERROR_NOSUBSTRING (-7) if there is no
2287 nigel 91 subpattern of that name.
2288 nigel 63
2289 nigel 75 Given the number, you can extract the substring directly, or use one of
2290     the functions described in the previous section. For convenience, there
2291     are also two functions that do the whole job.
2292    
2293 ph10 243 Most of the arguments of pcre_copy_named_substring() and
2294     pcre_get_named_substring() are the same as those for the similarly
2295     named functions that extract by number. As these are described in the
2296     previous section, they are not re-described here. There are just two
2297 nigel 75 differences:
2298 nigel 63
2299 ph10 243 First, instead of a substring number, a substring name is given. Sec-
2300 nigel 73 ond, there is an extra argument, given at the start, which is a pointer
2301 ph10 243 to the compiled pattern. This is needed in order to gain access to the
2302 nigel 73 name-to-number translation table.
2303 nigel 63
2304 ph10 243 These functions call pcre_get_stringnumber(), and if it succeeds, they
2305     then call pcre_copy_substring() or pcre_get_substring(), as appropri-
2306     ate. NOTE: If PCRE_DUPNAMES is set and there are duplicate names, the
2307 ph10 128 behaviour may not be what you want (see the next section).
2308 nigel 63
2309 nigel 77
2310 nigel 91 DUPLICATE SUBPATTERN NAMES
2311    
2312     int pcre_get_stringtable_entries(const pcre *code,
2313     const char *name, char **first, char **last);
2314    
2315 ph10 243 When a pattern is compiled with the PCRE_DUPNAMES option, names for
2316     subpatterns are not required to be unique. Normally, patterns with
2317     duplicate names are such that in any one match, only one of the named
2318     subpatterns participates. An example is shown in the pcrepattern docu-
2319 ph10 208 mentation.
2320    
2321 ph10 243 When duplicates are present, pcre_copy_named_substring() and
2322     pcre_get_named_substring() return the first substring corresponding to
2323     the given name that is set. If none are set, PCRE_ERROR_NOSUBSTRING
2324     (-7) is returned; no data is returned. The pcre_get_stringnumber()
2325     function returns one of the numbers that are associated with the name,
2326 ph10 208 but it is not defined which it is.
2327 nigel 91
2328 ph10 243 If you want to get full details of all captured substrings for a given
2329     name, you must use the pcre_get_stringtable_entries() function. The
2330 nigel 91 first argument is the compiled pattern, and the second is the name. The
2331 ph10 243 third and fourth are pointers to variables which are updated by the
2332 nigel 91 function. After it has run, they point to the first and last entries in
2333 ph10 243 the name-to-number table for the given name. The function itself
2334     returns the length of each entry, or PCRE_ERROR_NOSUBSTRING (-7) if
2335     there are none. The format of the table is described above in the sec-
2336     tion entitled Information about a pattern. Given all the relevant
2337     entries for the name, you can extract each of their numbers, and hence
2338 nigel 93 the captured data, if any.
2339 nigel 91
2340    
2341 nigel 77 FINDING ALL POSSIBLE MATCHES
2342    
2343 ph10 243 The traditional matching function uses a similar algorithm to Perl,
2344 nigel 77 which stops when it finds the first match, starting at a given point in
2345 ph10 243 the subject. If you want to find all possible matches, or the longest
2346     possible match, consider using the alternative matching function (see
2347     below) instead. If you cannot use the alternative function, but still
2348     need to find all possible matches, you can kludge it up by making use
2349 nigel 77 of the callout facility, which is described in the pcrecallout documen-
2350     tation.
2351    
2352     What you have to do is to insert a callout right at the end of the pat-
2353 ph10 243 tern. When your callout function is called, extract and save the cur-
2354     rent matched substring. Then return 1, which forces pcre_exec() to
2355     backtrack and try other alternatives. Ultimately, when it runs out of
2356 nigel 77 matches, pcre_exec() will yield PCRE_ERROR_NOMATCH.
2357    
2358    
2359     MATCHING A PATTERN: THE ALTERNATIVE FUNCTION
2360    
2361     int pcre_dfa_exec(const pcre *code, const pcre_extra *extra,
2362     const char *subject, int length, int startoffset,
2363     int options, int *ovector, int ovecsize,
2364     int *workspace, int wscount);
2365    
2366 ph10 243 The function pcre_dfa_exec() is called to match a subject string
2367     against a compiled pattern, using a matching algorithm that scans the
2368     subject string just once, and does not backtrack. This has different
2369     characteristics to the normal algorithm, and is not compatible with
2370     Perl. Some of the features of PCRE patterns are not supported. Never-
2371     theless, there are times when this kind of matching can be useful. For
2372 nigel 93 a discussion of the two matching algorithms, see the pcrematching docu-
2373     mentation.
2374 nigel 77
2375 ph10 243 The arguments for the pcre_dfa_exec() function are the same as for
2376 nigel 77 pcre_exec(), plus two extras. The ovector argument is used in a differ-
2377 ph10 243 ent way, and this is described below. The other common arguments are
2378     used in the same way as for pcre_exec(), so their description is not
2379 nigel 77 repeated here.
2380    
2381 ph10 243 The two additional arguments provide workspace for the function. The
2382     workspace vector should contain at least 20 elements. It is used for
2383 nigel 77 keeping track of multiple paths through the pattern tree. More
2384 ph10 243 workspace will be needed for patterns and subjects where there are a
2385 nigel 91 lot of potential matches.
2386 nigel 77
2387 nigel 87 Here is an example of a simple call to pcre_dfa_exec():
2388 nigel 77
2389     int rc;
2390     int ovector[10];
2391     int wspace[20];
2392 nigel 87 rc = pcre_dfa_exec(
2393 nigel 77 re, /* result of pcre_compile() */
2394     NULL, /* we didn't study the pattern */
2395     "some string", /* the subject string */
2396     11, /* the length of the subject string */
2397     0, /* start at offset 0 in the subject */
2398     0, /* default options */
2399     ovector, /* vector of integers for substring information */
2400     10, /* number of elements (NOT size in bytes) */
2401     wspace, /* working space vector */
2402     20); /* number of elements (NOT size in bytes) */
2403    
2404     Option bits for pcre_dfa_exec()
2405    
2406 ph10 243 The unused bits of the options argument for pcre_dfa_exec() must be
2407     zero. The only bits that may be set are PCRE_ANCHORED, PCRE_NEW-
2408     LINE_xxx, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK,
2409 nigel 91 PCRE_PARTIAL, PCRE_DFA_SHORTEST, and PCRE_DFA_RESTART. All but the last
2410     three of these are the same as for pcre_exec(), so their description is
2411     not repeated here.
2412 nigel 77
2413     PCRE_PARTIAL
2414    
2415 ph10 243 This has the same general effect as it does for pcre_exec(), but the
2416     details are slightly different. When PCRE_PARTIAL is set for
2417     pcre_dfa_exec(), the return code PCRE_ERROR_NOMATCH is converted into
2418     PCRE_ERROR_PARTIAL if the end of the subject is reached, there have
2419 nigel 77 been no complete matches, but there is still at least one matching pos-
2420 ph10 243 sibility. The portion of the string that provided the partial match is
2421 nigel 77 set as the first matching string.
2422    
2423     PCRE_DFA_SHORTEST
2424    
2425 ph10 243 Setting the PCRE_DFA_SHORTEST option causes the matching algorithm to
2426 nigel 93 stop as soon as it has found one match. Because of the way the alterna-
2427 ph10 243 tive algorithm works, this is necessarily the shortest possible match
2428 nigel 93 at the first possible matching point in the subject string.
2429 nigel 77
2430     PCRE_DFA_RESTART
2431    
2432 ph10 243 When pcre_dfa_exec() is called with the PCRE_PARTIAL option, and
2433     returns a partial match, it is possible to call it again, with addi-
2434     tional subject characters, and have it continue with the same match.
2435     The PCRE_DFA_RESTART option requests this action; when it is set, the
2436     workspace and wscount options must reference the same vector as before
2437     because data about the match so far is left in them after a partial
2438     match. There is more discussion of this facility in the pcrepartial
2439 nigel 77 documentation.
2440    
2441     Successful returns from pcre_dfa_exec()
2442    
2443 ph10 243 When pcre_dfa_exec() succeeds, it may have matched more than one sub-
2444 nigel 77 string in the subject. Note, however, that all the matches from one run
2445 ph10 243 of the function start at the same point in the subject. The shorter
2446     matches are all initial substrings of the longer matches. For example,
2447 nigel 77 if the pattern
2448    
2449     <.*>
2450    
2451     is matched against the string
2452    
2453     This is <something> <something else> <something further> no more
2454    
2455     the three matched strings are
2456    
2457     <something>
2458     <something> <something else>
2459     <something> <something else> <something further>
2460    
2461 ph10 243 On success, the yield of the function is a number greater than zero,
2462     which is the number of matched substrings. The substrings themselves
2463     are returned in ovector. Each string uses two elements; the first is
2464     the offset to the start, and the second is the offset to the end. In
2465     fact, all the strings have the same start offset. (Space could have
2466     been saved by giving this only once, but it was decided to retain some
2467     compatibility with the way pcre_exec() returns data, even though the
2468 nigel 93 meaning of the strings is different.)
2469 nigel 77
2470     The strings are returned in reverse order of length; that is, the long-
2471 ph10 243 est matching string is given first. If there were too many matches to
2472     fit into ovector, the yield of the function is zero, and the vector is
2473 nigel 77 filled with the longest matches.
2474    
2475     Error returns from pcre_dfa_exec()
2476    
2477 ph10 243 The pcre_dfa_exec() function returns a negative number when it fails.
2478     Many of the errors are the same as for pcre_exec(), and these are
2479     described above. There are in addition the following errors that are
2480 nigel 77 specific to pcre_dfa_exec():
2481    
2482     PCRE_ERROR_DFA_UITEM (-16)
2483    
2484 ph10 243 This return is given if pcre_dfa_exec() encounters an item in the pat-
2485     tern that it does not support, for instance, the use of \C or a back
2486 nigel 77 reference.
2487    
2488     PCRE_ERROR_DFA_UCOND (-17)
2489    
2490 ph10 243 This return is given if pcre_dfa_exec() encounters a condition item
2491     that uses a back reference for the condition, or a test for recursion
2492 nigel 93 in a specific group. These are not supported.
2493 nigel 77
2494     PCRE_ERROR_DFA_UMLIMIT (-18)
2495    
2496 ph10 243 This return is given if pcre_dfa_exec() is called with an extra block
2497 nigel 77 that contains a setting of the match_limit field. This is not supported
2498     (it is meaningless).
2499    
2500     PCRE_ERROR_DFA_WSSIZE (-19)
2501    
2502 ph10 243 This return is given if pcre_dfa_exec() runs out of space in the
2503 nigel 77 workspace vector.
2504    
2505     PCRE_ERROR_DFA_RECURSE (-20)
2506    
2507 ph10 243 When a recursive subpattern is processed, the matching function calls
2508     itself recursively, using private vectors for ovector and workspace.
2509     This error is given if the output vector is not large enough. This
2510 nigel 77 should be extremely rare, as a vector of size 1000 is used.
2511    
2512 nigel 93
2513     SEE ALSO
2514    
2515 ph10 243 pcrebuild(3), pcrecallout(3), pcrecpp(3)(3), pcrematching(3), pcrepar-
2516     tial(3), pcreposix(3), pcreprecompile(3), pcresample(3), pcrestack(3).
2517 nigel 93
2518 nigel 63
2519 ph10 99 AUTHOR
2520 nigel 63
2521 ph10 99 Philip Hazel
2522     University Computing Service
2523     Cambridge CB2 3QH, England.
2524    
2525    
2526     REVISION
2527    
2528 ph10 231 Last updated: 11 September 2007
2529 ph10 99 Copyright (c) 1997-2007 University of Cambridge.
2530     ------------------------------------------------------------------------------
2531 ph10 111
2532    
2533 nigel 79 PCRECALLOUT(3) PCRECALLOUT(3)
2534 nigel 63
2535 nigel 79
2536 nigel 73 NAME
2537     PCRE - Perl-compatible regular expressions
2538    
2539 nigel 77
2540 nigel 63 PCRE CALLOUTS
2541    
2542 nigel 73 int (*pcre_callout)(pcre_callout_block *);
2543 nigel 63
2544 nigel 73 PCRE provides a feature called "callout", which is a means of temporar-
2545     ily passing control to the caller of PCRE in the middle of pattern
2546     matching. The caller of PCRE provides an external function by putting
2547     its entry point in the global variable pcre_callout. By default, this
2548     variable contains NULL, which disables all calling out.
2549 nigel 63
2550 nigel 73 Within a regular expression, (?C) indicates the points at which the
2551     external function is to be called. Different callout points can be
2552     identified by putting a number less than 256 after the letter C. The
2553     default value is zero. For example, this pattern has two callout
2554     points:
2555 nigel 63
2556 ph10 155 (?C1)abc(?C2)def
2557 nigel 63
2558 nigel 75 If the PCRE_AUTO_CALLOUT option bit is set when pcre_compile() is
2559     called, PCRE automatically inserts callouts, all with number 255,
2560     before each item in the pattern. For example, if PCRE_AUTO_CALLOUT is
2561     used with the pattern
2562 nigel 63
2563 nigel 75 A(\d{2}|--)
2564    
2565     it is processed as if it were
2566    
2567     (?C255)A(?C255)((?C255)\d{2}(?C255)|(?C255)-(?C255)-(?C255))(?C255)
2568    
2569     Notice that there is a callout before and after each parenthesis and
2570     alternation bar. Automatic callouts can be used for tracking the
2571     progress of pattern matching. The pcretest command has an option that
2572     sets automatic callouts; when it is used, the output indicates how the
2573     pattern is matched. This is useful information when you are trying to
2574     optimize the performance of a particular pattern.
2575    
2576    
2577     MISSING CALLOUTS
2578    
2579     You should be aware that, because of optimizations in the way PCRE
2580     matches patterns, callouts sometimes do not happen. For example, if the
2581     pattern is
2582    
2583     ab(?C4)cd
2584    
2585     PCRE knows that any matching string must contain the letter "d". If the
2586     subject string is "abyz", the lack of "d" means that matching doesn't
2587     ever start, and the callout is never reached. However, with "abyd",
2588     though the result is still no match, the callout is obeyed.
2589    
2590    
2591     THE CALLOUT INTERFACE
2592    
2593     During matching, when PCRE reaches a callout point, the external func-
2594 nigel 77 tion defined by pcre_callout is called (if it is set). This applies to
2595     both the pcre_exec() and the pcre_dfa_exec() matching functions. The
2596     only argument to the callout function is a pointer to a pcre_callout
2597     block. This structure contains the following fields:
2598 nigel 75
2599 nigel 73 int version;
2600     int callout_number;
2601     int *offset_vector;
2602     const char *subject;
2603     int subject_length;
2604     int start_match;
2605     int current_position;
2606     int capture_top;
2607     int capture_last;
2608     void *callout_data;
2609 nigel 75 int pattern_position;
2610     int next_item_length;
2611 nigel 63
2612 nigel 77 The version field is an integer containing the version number of the
2613     block format. The initial version was 0; the current version is 1. The
2614     version number will change again in future if additional fields are
2615 nigel 75 added, but the intention is never to remove any of the existing fields.
2616 nigel 63
2617 nigel 77 The callout_number field contains the number of the callout, as com-
2618     piled into the pattern (that is, the number after ?C for manual call-
2619 nigel 75 outs, and 255 for automatically generated callouts).
2620 nigel 63
2621 nigel 77 The offset_vector field is a pointer to the vector of offsets that was
2622     passed by the caller to pcre_exec() or pcre_dfa_exec(). When
2623     pcre_exec() is used, the contents can be inspected in order to extract
2624     substrings that have been matched so far, in the same way as for
2625     extracting substrings after a match has completed. For pcre_dfa_exec()
2626     this field is not useful.
2627 nigel 63
2628 nigel 75 The subject and subject_length fields contain copies of the values that
2629 nigel 73 were passed to pcre_exec().
2630 nigel 63
2631 ph10 172 The start_match field normally contains the offset within the subject
2632     at which the current match attempt started. However, if the escape
2633     sequence \K has been encountered, this value is changed to reflect the
2634     modified starting point. If the pattern is not anchored, the callout
2635     function may be called several times from the same point in the pattern
2636     for different starting points in the subject.
2637 nigel 63
2638 nigel 77 The current_position field contains the offset within the subject of
2639 nigel 73 the current match pointer.
2640 nigel 63
2641 nigel 77 When the pcre_exec() function is used, the capture_top field contains
2642     one more than the number of the highest numbered captured substring so
2643     far. If no substrings have been captured, the value of capture_top is
2644     one. This is always the case when pcre_dfa_exec() is used, because it
2645     does not support captured substrings.
2646 nigel 63
2647 nigel 77 The capture_last field contains the number of the most recently cap-
2648     tured substring. If no substrings have been captured, its value is -1.
2649     This is always the case when pcre_dfa_exec() is used.
2650 nigel 63
2651 nigel 77 The callout_data field contains a value that is passed to pcre_exec()
2652     or pcre_dfa_exec() specifically so that it can be passed back in call-
2653     outs. It is passed in the pcre_callout field of the pcre_extra data
2654     structure. If no such data was passed, the value of callout_data in a
2655     pcre_callout block is NULL. There is a description of the pcre_extra
2656 nigel 73 structure in the pcreapi documentation.
2657 nigel 63
2658 nigel 77 The pattern_position field is present from version 1 of the pcre_call-
2659 nigel 75 out structure. It contains the offset to the next item to be matched in
2660     the pattern string.
2661 nigel 63
2662 nigel 77 The next_item_length field is present from version 1 of the pcre_call-
2663 nigel 75 out structure. It contains the length of the next item to be matched in
2664 nigel 77 the pattern string. When the callout immediately precedes an alterna-
2665     tion bar, a closing parenthesis, or the end of the pattern, the length
2666     is zero. When the callout precedes an opening parenthesis, the length
2667 nigel 75 is that of the entire subpattern.
2668 nigel 73
2669 nigel 77 The pattern_position and next_item_length fields are intended to help
2670     in distinguishing between different automatic callouts, which all have
2671 nigel 75 the same callout number. However, they are set for all callouts.
2672    
2673    
2674 nigel 63 RETURN VALUES
2675    
2676 nigel 77 The external callout function returns an integer to PCRE. If the value
2677     is zero, matching proceeds as normal. If the value is greater than
2678     zero, matching fails at the current point, but the testing of other
2679     matching possibilities goes ahead, just as if a lookahead assertion had
2680     failed. If the value is less than zero, the match is abandoned, and
2681     pcre_exec() (or pcre_dfa_exec()) returns the negative value.
2682 nigel 63
2683 nigel 77 Negative values should normally be chosen from the set of
2684 nigel 73 PCRE_ERROR_xxx values. In particular, PCRE_ERROR_NOMATCH forces a stan-
2685 nigel 77 dard "no match" failure. The error number PCRE_ERROR_CALLOUT is
2686     reserved for use by callout functions; it will never be used by PCRE
2687 nigel 73 itself.
2688 nigel 63
2689    
2690 ph10 99 AUTHOR
2691 nigel 63
2692 ph10 99 Philip Hazel
2693     University Computing Service
2694     Cambridge CB2 3QH, England.
2695    
2696    
2697     REVISION
2698    
2699 ph10 172 Last updated: 29 May 2007
2700 ph10 99 Copyright (c) 1997-2007 University of Cambridge.
2701     ------------------------------------------------------------------------------
2702 ph10 111
2703    
2704 nigel 79 PCRECOMPAT(3) PCRECOMPAT(3)
2705 nigel 63
2706 nigel 79
2707 nigel 73 NAME
2708     PCRE - Perl-compatible regular expressions
2709    
2710 nigel 77
2711 nigel 75 DIFFERENCES BETWEEN PCRE AND PERL
2712 nigel 41
2713 nigel 73 This document describes the differences in the ways that PCRE and Perl
2714 nigel 93 handle regular expressions. The differences described here are mainly
2715 ph10 182 with respect to Perl 5.8, though PCRE versions 7.0 and later contain
2716     some features that are expected to be in the forthcoming Perl 5.10.
2717 nigel 41
2718 nigel 93 1. PCRE has only a subset of Perl's UTF-8 and Unicode support. Details
2719     of what it does have are given in the section on UTF-8 support in the
2720 nigel 87 main pcre page.
2721 nigel 41
2722 nigel 73 2. PCRE does not allow repeat quantifiers on lookahead assertions. Perl
2723 nigel 93 permits them, but they do not mean what you might think. For example,
2724 nigel 73 (?!a){3} does not assert that the next three characters are not "a". It
2725     just asserts that the next character is not "a" three times.
2726 nigel 41
2727 nigel 93 3. Capturing subpatterns that occur inside negative lookahead asser-
2728     tions are counted, but their entries in the offsets vector are never
2729     set. Perl sets its numerical variables from any such patterns that are
2730 nigel 73 matched before the assertion fails to match something (thereby succeed-
2731 nigel 93 ing), but only if the negative lookahead assertion contains just one
2732 nigel 73 branch.
2733 nigel 41
2734 nigel 93 4. Though binary zero characters are supported in the subject string,
2735 nigel 73 they are not allowed in a pattern string because it is passed as a nor-
2736 nigel 75 mal C string, terminated by zero. The escape sequence \0 can be used in
2737     the pattern to represent a binary zero.
2738 nigel 41
2739 nigel 93 5. The following Perl escape sequences are not supported: \l, \u, \L,
2740 nigel 75 \U, and \N. In fact these are implemented by Perl's general string-han-
2741 nigel 93 dling and are not part of its pattern matching engine. If any of these
2742 nigel 75 are encountered by PCRE, an error is generated.
2743 nigel 41
2744 nigel 93 6. The Perl escape sequences \p, \P, and \X are supported only if PCRE
2745     is built with Unicode character property support. The properties that
2746     can be tested with \p and \P are limited to the general category prop-
2747     erties such as Lu and Nd, script names such as Greek or Han, and the
2748 nigel 87 derived properties Any and L&.
2749 nigel 75
2750     7. PCRE does support the \Q...\E escape for quoting substrings. Charac-
2751 nigel 93 ters in between are treated as literals. This is slightly different
2752     from Perl in that $ and @ are also handled as literals inside the
2753     quotes. In Perl, they cause variable interpolation (but of course PCRE
2754 nigel 73 does not have variables). Note the following examples:
2755 nigel 49
2756 nigel 73 Pattern PCRE matches Perl matches
2757 nigel 41
2758 nigel 73 \Qabc$xyz\E abc$xyz abc followed by the
2759     contents of $xyz
2760     \Qabc\$xyz\E abc\$xyz abc\$xyz
2761     \Qabc\E\$\Qxyz\E abc$xyz abc$xyz
2762 nigel 41
2763 nigel 93 The \Q...\E sequence is recognized both inside and outside character
2764 nigel 73 classes.
2765 nigel 41
2766 nigel 93 8. Fairly obviously, PCRE does not support the (?{code}) and (??{code})
2767     constructions. However, there is support for recursive patterns. This
2768     is not available in Perl 5.8, but will be in Perl 5.10. Also, the PCRE
2769     "callout" feature allows an external function to be called during pat-
2770 nigel 75 tern matching. See the pcrecallout documentation for details.
2771 nigel 63
2772 nigel 93 9. Subpatterns that are called recursively or as "subroutines" are
2773     always treated as atomic groups in PCRE. This is like Python, but
2774     unlike Perl.
2775    
2776     10. There are some differences that are concerned with the settings of
2777     captured strings when part of a pattern is repeated. For example,
2778     matching "aba" against the pattern /^(a(b)?)+$/ in Perl leaves $2
2779 nigel 73 unset, but in PCRE it is set to "b".
2780 nigel 41
2781 ph10 211 11. PCRE does support Perl 5.10's backtracking verbs (*ACCEPT),
2782     (*FAIL), (*F), (*COMMIT), (*PRUNE), (*SKIP), and (*THEN), but only in
2783     the forms without an argument. PCRE does not support (*MARK). If
2784     (*ACCEPT) is within capturing parentheses, PCRE does not set that cap-
2785     ture group; this is different to Perl.
2786    
2787     12. PCRE provides some extensions to the Perl regular expression facil-
2788 nigel 93 ities. Perl 5.10 will include new features that are not in earlier
2789     versions, some of which (such as named parentheses) have been in PCRE
2790     for some time. This list is with respect to Perl 5.10:
2791 nigel 41
2792 nigel 93 (a) Although lookbehind assertions must match fixed length strings,
2793 nigel 73 each alternative branch of a lookbehind assertion can match a different
2794     length of string. Perl requires them all to have the same length.
2795 nigel 41
2796 nigel 93 (b) If PCRE_DOLLAR_ENDONLY is set and PCRE_MULTILINE is not set, the $
2797 nigel 73 meta-character matches only at the very end of the string.
2798 nigel 41
2799 nigel 73 (c) If PCRE_EXTRA is set, a backslash followed by a letter with no spe-
2800 ph10 182 cial meaning is faulted. Otherwise, like Perl, the backslash is quietly
2801     ignored. (Perl can be made to issue a warning.)
2802 nigel 41
2803 nigel 93 (d) If PCRE_UNGREEDY is set, the greediness of the repetition quanti-
2804 nigel 73 fiers is inverted, that is, by default they are not greedy, but if fol-
2805     lowed by a question mark they are.
2806 nigel 41
2807 nigel 75 (e) PCRE_ANCHORED can be used at matching time to force a pattern to be
2808     tried only at the first matching position in the subject string.
2809 nigel 41
2810 nigel 93 (f) The PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and PCRE_NO_AUTO_CAP-
2811 nigel 73 TURE options for pcre_exec() have no Perl equivalents.
2812 nigel 41
2813 ph10 231 (g) The \R escape sequence can be restricted to match only CR, LF, or
2814     CRLF by the PCRE_BSR_ANYCRLF option.
2815 nigel 41
2816 ph10 231 (h) The callout facility is PCRE-specific.
2817 nigel 43
2818 ph10 231 (i) The partial matching facility is PCRE-specific.
2819    
2820     (j) Patterns compiled by PCRE can be saved and re-used at a later time,
2821 nigel 75 even on different hosts that have the other endianness.
2822    
2823 ph10 231 (k) The alternative matching function (pcre_dfa_exec()) matches in a
2824 nigel 77 different way and is not Perl-compatible.
2825    
2826 ph10 231 (l) PCRE recognizes some special sequences such as (*CR) at the start
2827     of a pattern that set overall options that cannot be changed within the
2828     pattern.
2829 nigel 63
2830 ph10 231
2831 ph10 99 AUTHOR
2832 nigel 63
2833 ph10 99 Philip Hazel
2834     University Computing Service
2835     Cambridge CB2 3QH, England.
2836    
2837    
2838     REVISION
2839    
2840 ph10 231 Last updated: 11 September 2007
2841 ph10 99 Copyright (c) 1997-2007 University of Cambridge.
2842     ------------------------------------------------------------------------------
2843 ph10 111
2844    
2845 nigel 79 PCREPATTERN(3) PCREPATTERN(3)
2846 nigel 63
2847 nigel 79
2848 nigel 73 NAME
2849     PCRE - Perl-compatible regular expressions
2850    
2851 nigel 77
2852 nigel 63 PCRE REGULAR EXPRESSION DETAILS
2853    
2854 ph10 208 The syntax and semantics of the regular expressions that are supported
2855     by PCRE are described in detail below. There is a quick-reference syn-
2856     tax summary in the pcresyntax page. Perl's regular expressions are
2857     described in its own documentation, and regular expressions in general
2858     are covered in a number of books, some of which have copious examples.
2859     Jeffrey Friedl's "Mastering Regular Expressions", published by
2860     O'Reilly, covers regular expressions in great detail. This description
2861     of PCRE's regular expressions is intended as reference material.
2862 nigel 49
2863 nigel 75 The original operation of PCRE was on strings of one-byte characters.
2864     However, there is now also support for UTF-8 character strings. To use
2865     this, you must build PCRE to include UTF-8 support, and then call
2866     pcre_compile() with the PCRE_UTF8 option. How this affects pattern
2867     matching is mentioned in several places below. There is also a summary
2868     of UTF-8 features in the section on UTF-8 support in the main pcre
2869     page.
2870 nigel 41
2871 nigel 77 The remainder of this document discusses the patterns that are sup-
2872     ported by PCRE when its main matching function, pcre_exec(), is used.
2873     From release 6.0, PCRE offers a second matching function,
2874     pcre_dfa_exec(), which matches using a different algorithm that is not
2875 ph10 172 Perl-compatible. Some of the features discussed below are not available
2876     when pcre_dfa_exec() is used. The advantages and disadvantages of the
2877     alternative function, and how it differs from the normal function, are
2878     discussed in the pcrematching page.
2879 nigel 77
2880 nigel 93
2881 ph10 227 NEWLINE CONVENTIONS
2882    
2883     PCRE supports five different conventions for indicating line breaks in
2884     strings: a single CR (carriage return) character, a single LF (line-
2885     feed) character, the two-character sequence CRLF, any of the three pre-
2886     ceding, or any Unicode newline sequence. The pcreapi page has further
2887     discussion about newlines, and shows how to set the newline convention
2888     in the options arguments for the compiling and matching functions.
2889    
2890     It is also possible to specify a newline convention by starting a pat-
2891     tern string with one of the following five sequences:
2892    
2893     (*CR) carriage return
2894     (*LF) linefeed
2895     (*CRLF) carriage return, followed by linefeed
2896     (*ANYCRLF) any of the three above
2897     (*ANY) all Unicode newline sequences
2898    
2899     These override the default and the options given to pcre_compile(). For
2900     example, on a Unix system where LF is the default newline sequence, the
2901     pattern
2902    
2903     (*CR)a.b
2904    
2905     changes the convention to CR. That pattern matches "a\nb" because LF is
2906     no longer a newline. Note that these special settings, which are not
2907     Perl-compatible, are recognized only at the very start of a pattern,
2908 ph10 231 and that they must be in upper case. If more than one of them is
2909     present, the last one is used.
2910 ph10 227
2911 ph10 231 The newline convention does not affect what the \R escape sequence
2912     matches. By default, this is any Unicode newline sequence, for Perl
2913     compatibility. However, this can be changed; see the description of \R
2914 ph10 247 in the section entitled "Newline sequences" below. A change of \R set-
2915     ting can be combined with a change of newline convention.
2916 ph10 227
2917 ph10 231
2918 nigel 93 CHARACTERS AND METACHARACTERS
2919    
2920 ph10 247 A regular expression is a pattern that is matched against a subject
2921     string from left to right. Most characters stand for themselves in a
2922     pattern, and match the corresponding characters in the subject. As a
2923 nigel 73 trivial example, the pattern
2924 nigel 41
2925 nigel 73 The quick brown fox
2926 nigel 41
2927 nigel 77 matches a portion of a subject string that is identical to itself. When
2928 ph10 247 caseless matching is specified (the PCRE_CASELESS option), letters are
2929     matched independently of case. In UTF-8 mode, PCRE always understands
2930     the concept of case for characters whose values are less than 128, so
2931     caseless matching is always possible. For characters with higher val-
2932     ues, the concept of case is supported if PCRE is compiled with Unicode
2933     property support, but not otherwise. If you want to use caseless
2934     matching for characters 128 and above, you must ensure that PCRE is
2935 nigel 77 compiled with Unicode property support as well as with UTF-8 support.
2936 nigel 41
2937 ph10 247 The power of regular expressions comes from the ability to include
2938     alternatives and repetitions in the pattern. These are encoded in the
2939 nigel 77 pattern by the use of metacharacters, which do not stand for themselves
2940     but instead are interpreted in some special way.
2941    
2942 ph10 247 There are two different sets of metacharacters: those that are recog-
2943     nized anywhere in the pattern except within square brackets, and those
2944     that are recognized within square brackets. Outside square brackets,
2945 nigel 93 the metacharacters are as follows:
2946 nigel 41
2947 nigel 73 \ general escape character with several uses
2948     ^ assert start of string (or line, in multiline mode)
2949     $ assert end of string (or line, in multiline mode)
2950     . match any character except newline (by default)
2951     [ start character class definition
2952     | start of alternative branch
2953     ( start subpattern
2954     ) end subpattern
2955     ? extends the meaning of (
2956     also 0 or 1 quantifier
2957     also quantifier minimizer
2958     * 0 or more quantifier
2959     + 1 or more quantifier
2960     also "possessive quantifier"
2961     { start min/max quantifier
2962 nigel 41
2963 ph10 247 Part of a pattern that is in square brackets is called a "character
2964 nigel 75 class". In a character class the only metacharacters are:
2965 nigel 41
2966 nigel 73 \ general escape character
2967     ^ negate the class, but only if the first character
2968     - indicates character range
2969     [ POSIX character class (only if followed by POSIX
2970     syntax)
2971     ] terminates the character class
2972 nigel 41
2973 ph10 247 The following sections describe the use of each of the metacharacters.
2974 nigel 41
2975    
2976 nigel 63 BACKSLASH
2977 nigel 41
2978 nigel 73 The backslash character has several uses. Firstly, if it is followed by
2979 ph10 247 a non-alphanumeric character, it takes away any special meaning that
2980     character may have. This use of backslash as an escape character
2981 nigel 73 applies both inside and outside character classes.
2982 nigel 41
2983 ph10 247 For example, if you want to match a * character, you write \* in the
2984     pattern. This escaping action applies whether or not the following
2985     character would otherwise be interpreted as a metacharacter, so it is
2986     always safe to precede a non-alphanumeric with backslash to specify
2987     that it stands for itself. In particular, if you want to match a back-
2988 nigel 75 slash, you write \\.
2989 nigel 41
2990 ph10 247 If a pattern is compiled with the PCRE_EXTENDED option, whitespace in
2991     the pattern (other than in a character class) and characters between a
2992 nigel 91 # outside a character class and the next newline are ignored. An escap-
2993 ph10 247 ing backslash can be used to include a whitespace or # character as
2994 nigel 91 part of the pattern.
2995 nigel 41
2996 ph10 247 If you want to remove the special meaning from a sequence of charac-
2997     ters, you can do so by putting them between \Q and \E. This is differ-
2998     ent from Perl in that $ and @ are handled as literals in \Q...\E
2999     sequences in PCRE, whereas in Perl, $ and @ cause variable interpola-
3000 nigel 73 tion. Note the following examples:
3001 nigel 63
3002 nigel 73 Pattern PCRE matches Perl matches
3003 nigel 63
3004 nigel 73 \Qabc$xyz\E abc$xyz abc followed by the
3005     contents of $xyz
3006     \Qabc\$xyz\E abc\$xyz abc\$xyz
3007     \Qabc\E\$\Qxyz\E abc$xyz abc$xyz
3008 nigel 63
3009 ph10 247 The \Q...\E sequence is recognized both inside and outside character
3010 nigel 73 classes.
3011 nigel 63
3012 nigel 75 Non-printing characters
3013    
3014 nigel 73 A second use of backslash provides a way of encoding non-printing char-
3015 ph10 247 acters in patterns in a visible manner. There is no restriction on the
3016     appearance of non-printing characters, apart from the binary zero that
3017     terminates a pattern, but when a pattern is being prepared by text
3018     editing, it is usually easier to use one of the following escape
3019 nigel 73 sequences than the binary character it represents:
3020 nigel 63
3021 nigel 73 \a alarm, that is, the BEL character (hex 07)
3022     \cx "control-x", where x is any character
3023     \e escape (hex 1B)
3024     \f formfeed (hex 0C)
3025 ph10 227 \n linefeed (hex 0A)
3026 nigel 73 \r carriage return (hex 0D)
3027     \t tab (hex 09)
3028     \ddd character with octal code ddd, or backreference
3029     \xhh character with hex code hh
3030 nigel 87 \x{hhh..} character with hex code hhh..
3031 nigel 41
3032 ph10 247 The precise effect of \cx is as follows: if x is a lower case letter,
3033     it is converted to upper case. Then bit 6 of the character (hex 40) is
3034     inverted. Thus \cz becomes hex 1A, but \c{ becomes hex 3B, while \c;
3035 nigel 73 becomes hex 7B.
3036 nigel 41
3037 ph10 247 After \x, from zero to two hexadecimal digits are read (letters can be
3038     in upper or lower case). Any number of hexadecimal digits may appear
3039     between \x{ and }, but the value of the character code must be less
3040 ph10 211 than 256 in non-UTF-8 mode, and less than 2**31 in UTF-8 mode. That is,
3041 ph10 247 the maximum value in hexadecimal is 7FFFFFFF. Note that this is bigger
3042 ph10 211 than the largest Unicode code point, which is 10FFFF.
3043 nigel 41
3044 ph10 247 If characters other than hexadecimal digits appear between \x{ and },
3045 ph10 211 or if there is no terminating }, this form of escape is not recognized.
3046 ph10 247 Instead, the initial \x will be interpreted as a basic hexadecimal
3047     escape, with no following digits, giving a character whose value is
3048 ph10 211 zero.
3049    
3050 nigel 73 Characters whose value is less than 256 can be defined by either of the
3051 ph10 247 two syntaxes for \x. There is no difference in the way they are han-
3052 nigel 87 dled. For example, \xdc is exactly the same as \x{dc}.
3053 nigel 41
3054 ph10 247 After \0 up to two further octal digits are read. If there are fewer
3055     than two digits, just those that are present are used. Thus the
3056 nigel 91 sequence \0\x\07 specifies two binary zeros followed by a BEL character
3057 ph10 247 (code value 7). Make sure you supply two digits after the initial zero
3058 nigel 91 if the pattern character that follows is itself an octal digit.
3059 nigel 63
3060 nigel 73 The handling of a backslash followed by a digit other than 0 is compli-
3061     cated. Outside a character class, PCRE reads it and any following dig-
3062 ph10 247 its as a decimal number. If the number is less than 10, or if there
3063 nigel 73 have been at least that many previous capturing left parentheses in the
3064 ph10 247 expression, the entire sequence is taken as a back reference. A
3065     description of how this works is given later, following the discussion
3066 nigel 73 of parenthesized subpatterns.
3067 nigel 41
3068 ph10 247 Inside a character class, or if the decimal number is greater than 9
3069     and there have not been that many capturing subpatterns, PCRE re-reads
3070 nigel 93 up to three octal digits following the backslash, and uses them to gen-
3071 ph10 247 erate a data character. Any subsequent digits stand for themselves. In
3072     non-UTF-8 mode, the value of a character specified in octal must be
3073     less than \400. In UTF-8 mode, values up to \777 are permitted. For
3074 nigel 91 example:
3075 nigel 41
3076 nigel 73 \040 is another way of writing a space
3077     \40 is the same, provided there are fewer than 40
3078     previous capturing subpatterns
3079     \7 is always a back reference
3080     \11 might be a back reference, or another way of
3081     writing a tab
3082     \011 is always a tab
3083     \0113 is a tab followed by the character "3"
3084     \113 might be a back reference, otherwise the
3085     character with octal code 113
3086     \377 might be a back reference, otherwise
3087     the byte consisting entirely of 1 bits
3088     \81 is either a back reference, or a binary zero
3089     followed by the two characters "8" and "1"
3090 nigel 41
3091 ph10 247 Note that octal values of 100 or greater must not be introduced by a
3092 nigel 73 leading zero, because no more than three octal digits are ever read.
3093 nigel 41
3094 nigel 91 All the sequences that define a single character value can be used both
3095 ph10 247 inside and outside character classes. In addition, inside a character
3096     class, the sequence \b is interpreted as the backspace character (hex
3097     08), and the sequences \R and \X are interpreted as the characters "R"
3098     and "X", respectively. Outside a character class, these sequences have
3099 nigel 93 different meanings (see below).
3100 nigel 43
3101 nigel 93 Absolute and relative back references
3102    
3103 ph10 247 The sequence \g followed by an unsigned or a negative number, option-
3104     ally enclosed in braces, is an absolute or relative back reference. A
3105 ph10 208 named back reference can be coded as \g{name}. Back references are dis-
3106     cussed later, following the discussion of parenthesized subpatterns.
3107 nigel 93
3108 nigel 75 Generic character types
3109 nigel 41
3110 nigel 93 Another use of backslash is for specifying generic character types. The
3111     following are always recognized:
3112 nigel 75
3113 nigel 73 \d any decimal digit
3114     \D any character that is not a decimal digit
3115 ph10 182 \h any horizontal whitespace character
3116     \H any character that is not a horizontal whitespace character
3117 nigel 73 \s any whitespace character
3118     \S any character that is not a whitespace character
3119 ph10 182 \v any vertical whitespace character
3120     \V any character that is not a vertical whitespace character
3121 nigel 73 \w any "word" character
3122     \W any "non-word" character
3123 nigel 41
3124 nigel 73 Each pair of escape sequences partitions the complete set of characters
3125 ph10 247 into two disjoint sets. Any given character matches one, and only one,
3126 nigel 73 of each pair.
3127 nigel 41
3128 nigel 75 These character type sequences can appear both inside and outside char-
3129 ph10 247 acter classes. They each match one character of the appropriate type.
3130     If the current matching point is at the end of the subject string, all
3131 nigel 75 of them fail, since there is no character to match.
3132 nigel 41
3133 ph10 247 For compatibility with Perl, \s does not match the VT character (code
3134     11). This makes it different from the the POSIX "space" class. The \s
3135     characters are HT (9), LF (10), FF (12), CR (13), and space (32). If
3136 nigel 91 "use locale;" is included in a Perl script, \s may match the VT charac-
3137 ph10 182 ter. In PCRE, it never does.
3138 nigel 63
3139 ph10 247 In UTF-8 mode, characters with values greater than 128 never match \d,
3140 ph10 182 \s, or \w, and always match \D, \S, and \W. This is true even when Uni-
3141 ph10 247 code character property support is available. These sequences retain
3142 ph10 182 their original meanings from before UTF-8 support was available, mainly
3143     for efficiency reasons.
3144    
3145     The sequences \h, \H, \v, and \V are Perl 5.10 features. In contrast to
3146 ph10 247 the other sequences, these do match certain high-valued codepoints in
3147 ph10 182 UTF-8 mode. The horizontal space characters are:
3148    
3149     U+0009 Horizontal tab
3150     U+0020 Space
3151     U+00A0 Non-break space
3152     U+1680 Ogham space mark
3153     U+180E Mongolian vowel separator
3154     U+2000 En quad
3155     U+2001 Em quad
3156     U+2002 En space
3157     U+2003 Em space
3158     U+2004 Three-per-em space
3159     U+2005 Four-per-em space
3160     U+2006 Six-per-em space
3161     U+2007 Figure space
3162     U+2008 Punctuation space
3163     U+2009 Thin space
3164     U+200A Hair space
3165     U+202F Narrow no-break space
3166     U+205F Medium mathematical space
3167     U+3000 Ideographic space
3168    
3169     The vertical space characters are:
3170    
3171     U+000A Linefeed
3172     U+000B Vertical tab
3173     U+000C Formfeed
3174     U+000D Carriage return
3175     U+0085 Next line
3176     U+2028 Line separator
3177     U+2029 Paragraph separator
3178    
3179 nigel 75 A "word" character is an underscore or any character less than 256 that
3180 ph10 247 is a letter or digit. The definition of letters and digits is con-
3181     trolled by PCRE's low-valued character tables, and may vary if locale-
3182     specific matching is taking place (see "Locale support" in the pcreapi
3183     page). For example, in a French locale such as "fr_FR" in Unix-like
3184     systems, or "french" in Windows, some character codes greater than 128
3185     are used for accented letters, and these are matched by \w. The use of
3186 ph10 182 locales with Unicode is discouraged.
3187 nigel 63
3188 nigel 93 Newline sequences
3189    
3190 ph10 247 Outside a character class, by default, the escape sequence \R matches
3191 ph10 231 any Unicode newline sequence. This is a Perl 5.10 feature. In non-UTF-8
3192     mode \R is equivalent to the following:
3193 nigel 93
3194     (?>\r\n|\n|\x0b|\f|\r|\x85)
3195    
3196 ph10 247 This is an example of an "atomic group", details of which are given
3197 nigel 93 below. This particular group matches either the two-character sequence
3198 ph10 247 CR followed by LF, or one of the single characters LF (linefeed,
3199 nigel 93 U+000A), VT (vertical tab, U+000B), FF (formfeed, U+000C), CR (carriage
3200     return, U+000D), or NEL (next line, U+0085). The two-character sequence
3201     is treated as a single unit that cannot be split.
3202    
3203 ph10 247 In UTF-8 mode, two additional characters whose codepoints are greater
3204 nigel 93 than 255 are added: LS (line separator, U+2028) and PS (paragraph sepa-
3205 ph10 247 rator, U+2029). Unicode character property support is not needed for
3206 nigel 93 these characters to be recognized.
3207    
3208 ph10 231 It is possible to restrict \R to match only CR, LF, or CRLF (instead of
3209 ph10 247 the complete set of Unicode line endings) by setting the option
3210 ph10 231 PCRE_BSR_ANYCRLF either at compile time or when the pattern is matched.
3211 ph10 247 (BSR is an abbrevation for "backslash R".) This can be made the default
3212     when PCRE is built; if this is the case, the other behaviour can be
3213     requested via the PCRE_BSR_UNICODE option. It is also possible to
3214     specify these settings by starting a pattern string with one of the
3215     following sequences:
3216 ph10 231
3217     (*BSR_ANYCRLF) CR, LF, or CRLF only
3218     (*BSR_UNICODE) any Unicode newline sequence
3219    
3220     These override the default and the options given to pcre_compile(), but
3221     they can be overridden by options given to pcre_exec(). Note that these
3222     special settings, which are not Perl-compatible, are recognized only at
3223     the very start of a pattern, and that they must be in upper case. If
3224 ph10 247 more than one of them is present, the last one is used. They can be
3225     combined with a change of newline convention, for example, a pattern
3226     can start with:
3227 ph10 231
3228 ph10 247 (*ANY)(*BSR_ANYCRLF)
3229    
3230 nigel 93 Inside a character class, \R matches the letter "R".
3231    
3232 nigel 75 Unicode character properties
3233    
3234     When PCRE is built with Unicode character property support, three addi-
3235 ph10 227 tional escape sequences that match characters with specific properties
3236     are available. When not in UTF-8 mode, these sequences are of course
3237     limited to testing characters whose codepoints are less than 256, but
3238 ph10 185 they do work in this mode. The extra escape sequences are:
3239 nigel 75
3240 nigel 87 \p{xx} a character with the xx property
3241     \P{xx} a character without the xx property
3242     \X an extended Unicode sequence
3243 nigel 75
3244 ph10 227 The property names represented by xx above are limited to the Unicode
3245 nigel 87 script names, the general category properties, and "Any", which matches
3246     any character (including newline). Other properties such as "InMusical-
3247 ph10 227 Symbols" are not currently supported by PCRE. Note that \P{Any} does
3248 nigel 87 not match any characters, so always causes a match failure.
3249 nigel 75
3250 nigel 87 Sets of Unicode characters are defined as belonging to certain scripts.
3251 ph10 227 A character from one of these sets can be matched using a script name.
3252 nigel 87 For example:
3253 nigel 75
3254 nigel 87 \p{Greek}
3255     \P{Han}
3256    
3257 ph10 227 Those that are not part of an identified script are lumped together as
3258 nigel 87 "Common". The current list of scripts is:
3259    
3260 nigel 93 Arabic, Armenian, Balinese, Bengali, Bopomofo, Braille, Buginese,
3261 ph10 227 Buhid, Canadian_Aboriginal, Cherokee, Common, Coptic, Cuneiform,
3262 nigel 93 Cypriot, Cyrillic, Deseret, Devanagari, Ethiopic, Georgian, Glagolitic,
3263 ph10 227 Gothic, Greek, Gujarati, Gurmukhi, Han, Hangul, Hanunoo, Hebrew, Hira-
3264     gana, Inherited, Kannada, Katakana, Kharoshthi, Khmer, Lao, Latin,
3265 nigel 93 Limbu, Linear_B, Malayalam, Mongolian, Myanmar, New_Tai_Lue, Nko,
3266 ph10 227 Ogham, Old_Italic, Old_Persian, Oriya, Osmanya, Phags_Pa, Phoenician,
3267 nigel 93 Runic, Shavian, Sinhala, Syloti_Nagri, Syriac, Tagalog, Tagbanwa,
3268     Tai_Le, Tamil, Telugu, Thaana, Thai, Tibetan, Tifinagh, Ugaritic, Yi.
3269 nigel 87
3270 ph10 227 Each character has exactly one general category property, specified by
3271 nigel 87 a two-letter abbreviation. For compatibility with Perl, negation can be
3272 ph10 227 specified by including a circumflex between the opening brace and the
3273 nigel 87 property name. For example, \p{^Lu} is the same as \P{Lu}.
3274    
3275     If only one letter is specified with \p or \P, it includes all the gen-
3276 ph10 227 eral category properties that start with that letter. In this case, in
3277     the absence of negation, the curly brackets in the escape sequence are
3278 nigel 87 optional; these two examples have the same effect:
3279    
3280 nigel 75 \p{L}
3281     \pL
3282    
3283 nigel 87 The following general category property codes are supported:
3284 nigel 75
3285     C Other
3286     Cc Control
3287     Cf Format
3288     Cn Unassigned
3289     Co Private use
3290     Cs Surrogate
3291    
3292     L Letter
3293     Ll Lower case letter
3294     Lm Modifier letter
3295     Lo Other letter
3296     Lt Title case letter
3297     Lu Upper case letter
3298    
3299     M Mark
3300     Mc Spacing mark
3301     Me Enclosing mark
3302     Mn Non-spacing mark
3303    
3304     N Number
3305     Nd Decimal number
3306     Nl Letter number
3307     No Other number
3308    
3309     P Punctuation
3310     Pc Connector punctuation
3311     Pd Dash punctuation
3312     Pe Close punctuation
3313     Pf Final punctuation
3314     Pi Initial punctuation
3315     Po Other punctuation
3316     Ps Open punctuation
3317    
3318     S Symbol
3319     Sc Currency symbol
3320     Sk Modifier symbol
3321     Sm Mathematical symbol
3322     So Other symbol
3323    
3324     Z Separator
3325     Zl Line separator
3326     Zp Paragraph separator
3327     Zs Space separator
3328    
3329 ph10 227 The special property L& is also supported: it matches a character that
3330     has the Lu, Ll, or Lt property, in other words, a letter that is not
3331 nigel 87 classified as a modifier or "other".
3332 nigel 75
3333 ph10 227 The Cs (Surrogate) property applies only to characters in the range
3334     U+D800 to U+DFFF. Such characters are not valid in UTF-8 strings (see
3335 ph10 211 RFC 3629) and so cannot be tested by PCRE, unless UTF-8 validity check-
3336 ph10 227 ing has been turned off (see the discussion of PCRE_NO_UTF8_CHECK in
3337 ph10 211 the pcreapi page).
3338    
3339 ph10 227 The long synonyms for these properties that Perl supports (such as
3340     \p{Letter}) are not supported by PCRE, nor is it permitted to prefix
3341 nigel 87 any of these properties with "Is".
3342    
3343     No character that is in the Unicode table has the Cn (unassigned) prop-
3344     erty. Instead, this property is assumed for any code point that is not
3345     in the Unicode table.
3346    
3347 ph10 227 Specifying caseless matching does not affect these escape sequences.
3348 nigel 75 For example, \p{Lu} always matches only upper case letters.
3349    
3350 ph10 227 The \X escape matches any number of Unicode characters that form an
3351 nigel 75 extended Unicode sequence. \X is equivalent to
3352    
3353     (?>\PM\pM*)
3354    
3355 ph10 227 That is, it matches a character without the "mark" property, followed
3356     by zero or more characters with the "mark" property, and treats the
3357     sequence as an atomic group (see below). Characters with the "mark"
3358     property are typically accents that affect the preceding character.
3359     None of them have codepoints less than 256, so in non-UTF-8 mode \X
3360 ph10 185 matches any one character.
3361 nigel 75
3362 ph10 227 Matching characters by Unicode property is not fast, because PCRE has
3363     to search a structure that contains data for over fifteen thousand
3364 nigel 75 characters. That is why the traditional escape sequences such as \d and
3365     \w do not use Unicode properties in PCRE.
3366    
3367 ph10 172 Resetting the match start
3368    
3369     The escape sequence \K, which is a Perl 5.10 feature, causes any previ-
3370 ph10 227 ously matched characters not to be included in the final matched
3371 ph10 172 sequence. For example, the pattern:
3372    
3373     foo\Kbar
3374    
3375 ph10 227 matches "foobar", but reports that it has matched "bar". This feature
3376     is similar to a lookbehind assertion (described below). However, in
3377     this case, the part of the subject before the real match does not have
3378     to be of fixed length, as lookbehind assertions do. The use of \K does
3379     not interfere with the setting of captured substrings. For example,
3380 ph10 172 when the pattern
3381    
3382     (foo)\Kbar
3383    
3384     matches "foobar", the first substring is still set to "foo".
3385    
3386 nigel 75 Simple assertions
3387    
3388 ph10 227 The final use of backslash is for certain simple assertions. An asser-
3389     tion specifies a condition that has to be met at a particular point in
3390     a match, without consuming any characters from the subject string. The
3391     use of subpatterns for more complicated assertions is described below.
3392 nigel 75 The backslashed assertions are:
3393 nigel 41
3394 nigel 73 \b matches at a word boundary
3395     \B matches when not at a word boundary
3396 nigel 93 \A matches at the start of the subject
3397     \Z matches at the end of the subject
3398     also matches before a newline at the end of the subject
3399     \z matches only at the end of the subject
3400     \G matches at the first matching position in the subject
3401 nigel 41
3402 ph10 227 These assertions may not appear in character classes (but note that \b
3403 nigel 73 has a different meaning, namely the backspace character, inside a char-
3404     acter class).
3405 nigel 41
3406 ph10 227 A word boundary is a position in the subject string where the current
3407     character and the previous character do not both match \w or \W (i.e.
3408     one matches \w and the other matches \W), or the start or end of the
3409