/[pcre]/code/trunk/doc/pcre.txt
ViewVC logotype

Contents of /code/trunk/doc/pcre.txt

Parent Directory Parent Directory | Revision Log Revision Log


Revision 545 - (hide annotations) (download)
Wed Jun 16 10:51:15 2010 UTC (4 years, 4 months ago) by ph10
File MIME type: text/plain
File size: 354147 byte(s)
Tidyup for 8.10-RC2 test release.

1 nigel 75 -----------------------------------------------------------------------------
2 nigel 63 This file contains a concatenation of the PCRE man pages, converted to plain
3     text format for ease of searching with a text editor, or for use on systems
4     that do not have a man page processor. The small individual files that give
5 ph10 535 synopses of each function in the library have not been included. Neither has
6 ph10 429 the pcredemo program. There are separate text files for the pcregrep and
7     pcretest commands.
8 nigel 63 -----------------------------------------------------------------------------
9    
10 nigel 41
11 nigel 79 PCRE(3) PCRE(3)
12 nigel 41
13 nigel 79
14 nigel 73 NAME
15     PCRE - Perl-compatible regular expressions
16    
17 nigel 77
18 nigel 75 INTRODUCTION
19 nigel 41
20 nigel 73 The PCRE library is a set of functions that implement regular expres-
21     sion pattern matching using the same syntax and semantics as Perl, with
22 ph10 461 just a few differences. Some features that appeared in Python and PCRE
23     before they appeared in Perl are also available using the Python syn-
24     tax, there is some support for one or two .NET and Oniguruma syntax
25     items, and there is an option for requesting some minor changes that
26     give better JavaScript compatibility.
27 nigel 63
28 ph10 461 The current implementation of PCRE corresponds approximately with Perl
29 ph10 518 5.10/5.11, including support for UTF-8 encoded strings and Unicode gen-
30     eral category properties. However, UTF-8 and Unicode support has to be
31 ph10 461 explicitly enabled; it is not the default. The Unicode tables corre-
32 ph10 507 spond to Unicode release 5.2.0.
33 nigel 77
34 nigel 93 In addition to the Perl-compatible matching function, PCRE contains an
35 ph10 461 alternative function that matches the same compiled patterns in a dif-
36     ferent way. In certain circumstances, the alternative function has some
37     advantages. For a discussion of the two matching algorithms, see the
38     pcrematching page.
39 nigel 93
40     PCRE is written in C and released as a C library. A number of people
41     have written wrappers and interfaces of various kinds. In particular,
42     Google Inc. have provided a comprehensive C++ wrapper. This is now
43 nigel 77 included as part of the PCRE distribution. The pcrecpp page has details
44 nigel 93 of this interface. Other people's contributions can be found in the
45 nigel 77 Contrib directory at the primary FTP site, which is:
46 nigel 63
47 nigel 73 ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre
48 nigel 63
49 nigel 93 Details of exactly which Perl regular expression features are and are
50 nigel 73 not supported by PCRE are given in separate documents. See the pcrepat-
51 ph10 208 tern and pcrecompat pages. There is a syntax summary in the pcresyntax
52     page.
53 nigel 63
54 ph10 208 Some features of PCRE can be included, excluded, or changed when the
55     library is built. The pcre_config() function makes it possible for a
56     client to discover which features are available. The features them-
57     selves are described in the pcrebuild page. Documentation about build-
58 ph10 461 ing PCRE for various operating systems can be found in the README and
59     NON-UNIX-USE files in the source distribution.
60 nigel 63
61 ph10 208 The library contains a number of undocumented internal functions and
62     data tables that are used by more than one of the exported external
63     functions, but which are not intended for use by external callers.
64     Their names all begin with "_pcre_", which hopefully will not provoke
65 nigel 83 any name clashes. In some environments, it is possible to control which
66 ph10 208 external symbols are exported when a shared library is built, and in
67 nigel 83 these cases the undocumented symbols are not exported.
68 nigel 63
69 nigel 77
70 nigel 63 USER DOCUMENTATION
71    
72 ph10 208 The user documentation for PCRE comprises a number of different sec-
73     tions. In the "man" format, each of these is a separate "man page". In
74     the HTML format, each is a separate page, linked from the index page.
75 ph10 429 In the plain text format, all the sections, except the pcredemo sec-
76     tion, are concatenated, for ease of searching. The sections are as fol-
77     lows:
78 nigel 63
79 nigel 73 pcre this document
80 ph10 153 pcre-config show PCRE installation configuration information
81 nigel 77 pcreapi details of PCRE's native C API
82 nigel 73 pcrebuild options for building PCRE
83     pcrecallout details of the callout feature
84     pcrecompat discussion of Perl compatibility
85 nigel 77 pcrecpp details of the C++ wrapper
86 ph10 429 pcredemo a demonstration C program that uses PCRE
87 nigel 73 pcregrep description of the pcregrep command
88 nigel 77 pcrematching discussion of the two matching algorithms
89 nigel 75 pcrepartial details of the partial matching facility
90 nigel 73 pcrepattern syntax and semantics of supported
91     regular expressions
92     pcreperform discussion of performance issues
93 nigel 77 pcreposix the POSIX-compatible C API
94 nigel 75 pcreprecompile details of saving and re-using precompiled patterns
95 ph10 429 pcresample discussion of the pcredemo program
96 nigel 91 pcrestack discussion of stack usage
97 ph10 461 pcresyntax quick syntax reference
98 nigel 75 pcretest description of the pcretest testing command
99 nigel 63
100 ph10 429 In addition, in the "man" and HTML formats, there is a short page for
101 nigel 77 each C library function, listing its arguments and results.
102 nigel 63
103    
104     LIMITATIONS
105    
106 ph10 429 There are some size limitations in PCRE but it is hoped that they will
107 nigel 73 never in practice be relevant.
108 nigel 63
109 ph10 429 The maximum length of a compiled pattern is 65539 (sic) bytes if PCRE
110 nigel 73 is compiled with the default internal linkage size of 2. If you want to
111 ph10 429 process regular expressions that are truly enormous, you can compile
112     PCRE with an internal linkage size of 3 or 4 (see the README file in
113     the source distribution and the pcrebuild documentation for details).
114     In these cases the limit is substantially larger. However, the speed
115 nigel 93 of execution is slower.
116 nigel 63
117 ph10 208 All values in repeating quantifiers must be less than 65536.
118 nigel 63
119 nigel 93 There is no limit to the number of parenthesized subpatterns, but there
120     can be no more than 65535 capturing subpatterns.
121 nigel 63
122 nigel 93 The maximum length of name for a named subpattern is 32 characters, and
123     the maximum number of named subpatterns is 10000.
124 nigel 91
125 ph10 429 The maximum length of a subject string is the largest positive number
126     that an integer variable can hold. However, when using the traditional
127 nigel 77 matching function, PCRE uses recursion to handle subpatterns and indef-
128 ph10 429 inite repetition. This means that the available stack space may limit
129 nigel 77 the size of a subject string that can be processed by certain patterns.
130 nigel 91 For a discussion of stack issues, see the pcrestack documentation.
131 nigel 63
132    
133 nigel 75 UTF-8 AND UNICODE PROPERTY SUPPORT
134 nigel 63
135 ph10 429 From release 3.3, PCRE has had some support for character strings
136     encoded in the UTF-8 format. For release 4.0 this was greatly extended
137     to cover most common requirements, and in release 5.0 additional sup-
138 nigel 75 port for Unicode general category properties was added.
139 nigel 63
140 ph10 429 In order process UTF-8 strings, you must build PCRE to include UTF-8
141     support in the code, and, in addition, you must call pcre_compile()
142     with the PCRE_UTF8 option flag, or the pattern must start with the
143     sequence (*UTF8). When either of these is the case, both the pattern
144     and any subject strings that are matched against it are treated as
145 ph10 461 UTF-8 strings instead of strings of 1-byte characters.
146 nigel 63
147 ph10 429 If you compile PCRE with UTF-8 support, but do not use it at run time,
148     the library will be a bit bigger, but the additional run time overhead
149 nigel 93 is limited to testing the PCRE_UTF8 flag occasionally, so should not be
150     very big.
151 nigel 63
152 nigel 75 If PCRE is built with Unicode character property support (which implies
153 ph10 429 UTF-8 support), the escape sequences \p{..}, \P{..}, and \X are sup-
154 nigel 75 ported. The available properties that can be tested are limited to the
155 ph10 429 general category properties such as Lu for an upper case letter or Nd
156     for a decimal number, the Unicode script names such as Arabic or Han,
157     and the derived properties Any and L&. A full list is given in the
158 nigel 87 pcrepattern documentation. Only the short names for properties are sup-
159 ph10 429 ported. For example, \p{L} matches a letter. Its Perl synonym, \p{Let-
160     ter}, is not supported. Furthermore, in Perl, many properties may
161     optionally be prefixed by "Is", for compatibility with Perl 5.6. PCRE
162 nigel 87 does not support this.
163 nigel 75
164 ph10 211 Validity of UTF-8 strings
165 nigel 63
166 ph10 429 When you set the PCRE_UTF8 flag, the strings passed as patterns and
167 ph10 211 subjects are (by default) checked for validity on entry to the relevant
168 ph10 429 functions. From release 7.3 of PCRE, the check is according the rules
169     of RFC 3629, which are themselves derived from the Unicode specifica-
170     tion. Earlier releases of PCRE followed the rules of RFC 2279, which
171     allows the full range of 31-bit values (0 to 0x7FFFFFFF). The current
172 ph10 211 check allows only values in the range U+0 to U+10FFFF, excluding U+D800
173     to U+DFFF.
174 nigel 63
175 ph10 429 The excluded code points are the "Low Surrogate Area" of Unicode, of
176     which the Unicode Standard says this: "The Low Surrogate Area does not
177     contain any character assignments, consequently no character code
178 ph10 211 charts or namelists are provided for this area. Surrogates are reserved
179 ph10 429 for use with UTF-16 and then must be used in pairs." The code points
180     that are encoded by UTF-16 pairs are available as independent code
181     points in the UTF-8 encoding. (In other words, the whole surrogate
182 ph10 211 thing is a fudge for UTF-16 which unfortunately messes up UTF-8.)
183    
184 ph10 429 If an invalid UTF-8 string is passed to PCRE, an error return
185 ph10 211 (PCRE_ERROR_BADUTF8) is given. In some situations, you may already know
186     that your strings are valid, and therefore want to skip these checks in
187     order to improve performance. If you set the PCRE_NO_UTF8_CHECK flag at
188 ph10 429 compile time or at run time, PCRE assumes that the pattern or subject
189     it is given (respectively) contains only valid UTF-8 codes. In this
190 ph10 211 case, it does not diagnose an invalid UTF-8 string.
191    
192 ph10 429 If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set,
193     what happens depends on why the string is invalid. If the string con-
194 ph10 211 forms to the "old" definition of UTF-8 (RFC 2279), it is processed as a
195 ph10 429 string of characters in the range 0 to 0x7FFFFFFF. In other words,
196 ph10 211 apart from the initial validity test, PCRE (when in UTF-8 mode) handles
197 ph10 429 strings according to the more liberal rules of RFC 2279. However, if
198     the string does not even conform to RFC 2279, the result is undefined.
199 ph10 211 Your program may crash.
200    
201 ph10 429 If you want to process strings of values in the full range 0 to
202     0x7FFFFFFF, encoded in a UTF-8-like manner as per the old RFC, you can
203 ph10 211 set PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in
204     this situation, you will have to apply your own validity check.
205    
206     General comments about UTF-8 mode
207    
208 ph10 429 1. An unbraced hexadecimal escape sequence (such as \xb3) matches a
209 nigel 87 two-byte UTF-8 character if the value is greater than 127.
210 nigel 63
211 ph10 429 2. Octal numbers up to \777 are recognized, and match two-byte UTF-8
212 nigel 91 characters for values greater than \177.
213    
214 ph10 429 3. Repeat quantifiers apply to complete UTF-8 characters, not to indi-
215 nigel 73 vidual bytes, for example: \x{100}{3}.
216 nigel 63
217 ph10 429 4. The dot metacharacter matches one UTF-8 character instead of a sin-
218 nigel 75 gle byte.
219 nigel 63
220 ph10 429 5. The escape sequence \C can be used to match a single byte in UTF-8
221     mode, but its use can lead to some strange effects. This facility is
222 nigel 77 not available in the alternative matching function, pcre_dfa_exec().
223 nigel 63
224 ph10 429 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly
225 ph10 518 test characters of any code value, but, by default, the characters that
226     PCRE recognizes as digits, spaces, or word characters remain the same
227     set as before, all with values less than 256. This remains true even
228     when PCRE is built to include Unicode property support, because to do
229     otherwise would slow down PCRE in many common cases. Note that this
230     also applies to \b, because it is defined in terms of \w and \W. If you
231     really want to test for a wider sense of, say, "digit", you can use
232     explicit Unicode property tests such as \p{Nd}. Alternatively, if you
233     set the PCRE_UCP option, the way that the character escapes work is
234     changed so that Unicode properties are used to determine which charac-
235     ters match. There are more details in the section on generic character
236     types in the pcrepattern documentation.
237 nigel 63
238 ph10 429 7. Similarly, characters that match the POSIX named character classes
239 ph10 518 are all low-valued characters, unless the PCRE_UCP option is set.
240 nigel 63
241 ph10 429 8. However, the Perl 5.10 horizontal and vertical whitespace matching
242 ph10 182 escapes (\h, \H, \v, and \V) do match all the appropriate Unicode char-
243 ph10 518 acters, whether or not PCRE_UCP is set.
244 ph10 182
245 ph10 429 9. Case-insensitive matching applies only to characters whose values
246     are less than 128, unless PCRE is built with Unicode property support.
247     Even when Unicode property support is available, PCRE still uses its
248     own character tables when checking the case of low-valued characters,
249     so as not to degrade performance. The Unicode property information is
250 nigel 87 used only for characters with higher values. Even when Unicode property
251     support is available, PCRE supports case-insensitive matching only when
252 ph10 429 there is a one-to-one mapping between a letter's cases. There are a
253     small number of many-to-one mappings in Unicode; these are not sup-
254 nigel 87 ported by PCRE.
255 nigel 63
256    
257     AUTHOR
258    
259 nigel 77 Philip Hazel
260 ph10 99 University Computing Service
261 nigel 93 Cambridge CB2 3QH, England.
262 nigel 63
263 ph10 429 Putting an actual email address here seems to have been a spam magnet,
264     so I've taken it away. If you want to email me, use my two initials,
265 ph10 153 followed by the two digits 10, at the domain cam.ac.uk.
266 nigel 77
267 nigel 63
268 ph10 99 REVISION
269 nigel 63
270 ph10 518 Last updated: 12 May 2010
271 ph10 507 Copyright (c) 1997-2010 University of Cambridge.
272 ph10 99 ------------------------------------------------------------------------------
273 ph10 535
274    
275 nigel 79 PCREBUILD(3) PCREBUILD(3)
276 nigel 63
277 nigel 79
278 nigel 73 NAME
279     PCRE - Perl-compatible regular expressions
280    
281 nigel 77
282 nigel 63 PCRE BUILD-TIME OPTIONS
283    
284 nigel 73 This document describes the optional features of PCRE that can be
285 ph10 261 selected when the library is compiled. It assumes use of the configure
286     script, where the optional features are selected or deselected by pro-
287     viding options to configure before running the make command. However,
288     the same options can be selected in both Unix-like and non-Unix-like
289 ph10 453 environments using the GUI facility of cmake-gui if you are using CMake
290     instead of configure to build PCRE.
291 nigel 63
292 ph10 453 There is a lot more information about building PCRE in non-Unix-like
293     environments in the file called NON_UNIX_USE, which is part of the PCRE
294     distribution. You should consult this file as well as the README file
295     if you are building in a non-Unix-like environment.
296    
297 ph10 261 The complete list of options for configure (which includes the standard
298 ph10 453 ones such as the selection of the installation directory) can be
299 ph10 261 obtained by running
300    
301 nigel 73 ./configure --help
302 nigel 63
303 ph10 453 The following sections include descriptions of options whose names
304 ph10 128 begin with --enable or --disable. These settings specify changes to the
305 ph10 453 defaults for the configure command. Because of the way that configure
306     works, --enable and --disable always come in pairs, so the complemen-
307     tary option always exists as well, but as it specifies the default, it
308 ph10 128 is not described.
309 nigel 63
310    
311 nigel 83 C++ SUPPORT
312    
313     By default, the configure script will search for a C++ compiler and C++
314     header files. If it finds them, it automatically builds the C++ wrapper
315     library for PCRE. You can disable this by adding
316    
317     --disable-cpp
318    
319     to the configure command.
320    
321    
322 nigel 63 UTF-8 SUPPORT
323    
324 ph10 392 To build PCRE with support for UTF-8 Unicode character strings, add
325 nigel 63
326 nigel 73 --enable-utf8
327 nigel 63
328 ph10 453 to the configure command. Of itself, this does not make PCRE treat
329     strings as UTF-8. As well as compiling PCRE with this option, you also
330     have have to set the PCRE_UTF8 option when you call the pcre_compile()
331 ph10 461 or pcre_compile2() functions.
332 nigel 63
333 ph10 453 If you set --enable-utf8 when compiling in an EBCDIC environment, PCRE
334 ph10 392 expects its input to be either ASCII or UTF-8 (depending on the runtime
335 ph10 453 option). It is not possible to support both EBCDIC and UTF-8 codes in
336     the same version of the library. Consequently, --enable-utf8 and
337 ph10 392 --enable-ebcdic are mutually exclusive.
338 nigel 63
339 ph10 392
340 nigel 75 UNICODE CHARACTER PROPERTY SUPPORT
341    
342 ph10 453 UTF-8 support allows PCRE to process character values greater than 255
343     in the strings that it handles. On its own, however, it does not pro-
344 nigel 75 vide any facilities for accessing the properties of such characters. If
345 ph10 453 you want to be able to use the pattern escapes \P, \p, and \X, which
346 nigel 75 refer to Unicode character properties, you must add
347    
348     --enable-unicode-properties
349    
350 ph10 453 to the configure command. This implies UTF-8 support, even if you have
351 nigel 75 not explicitly requested it.
352    
353 ph10 453 Including Unicode property support adds around 30K of tables to the
354     PCRE library. Only the general category properties such as Lu and Nd
355 ph10 128 are supported. Details are given in the pcrepattern documentation.
356 nigel 75
357    
358 nigel 63 CODE VALUE OF NEWLINE
359    
360 ph10 453 By default, PCRE interprets the linefeed (LF) character as indicating
361     the end of a line. This is the normal newline character on Unix-like
362     systems. You can compile PCRE to use carriage return (CR) instead, by
363 ph10 392 adding
364 nigel 63
365 nigel 73 --enable-newline-is-cr
366 nigel 63
367 ph10 453 to the configure command. There is also a --enable-newline-is-lf
368 nigel 91 option, which explicitly specifies linefeed as the newline character.
369 nigel 63
370 nigel 91 Alternatively, you can specify that line endings are to be indicated by
371     the two character sequence CRLF. If you want this, add
372 nigel 63
373 nigel 91 --enable-newline-is-crlf
374    
375 nigel 93 to the configure command. There is a fourth option, specified by
376 nigel 91
377 ph10 150 --enable-newline-is-anycrlf
378    
379 ph10 453 which causes PCRE to recognize any of the three sequences CR, LF, or
380 ph10 150 CRLF as indicating a line ending. Finally, a fifth option, specified by
381    
382 nigel 93 --enable-newline-is-any
383 nigel 91
384 ph10 150 causes PCRE to recognize any Unicode newline sequence.
385 nigel 93
386 ph10 453 Whatever line ending convention is selected when PCRE is built can be
387     overridden when the library functions are called. At build time it is
388 nigel 93 conventional to use the standard for your operating system.
389    
390    
391 ph10 231 WHAT \R MATCHES
392    
393 ph10 453 By default, the sequence \R in a pattern matches any Unicode newline
394     sequence, whatever has been selected as the line ending sequence. If
395 ph10 231 you specify
396    
397     --enable-bsr-anycrlf
398    
399 ph10 453 the default is changed so that \R matches only CR, LF, or CRLF. What-
400     ever is selected when PCRE is built can be overridden when the library
401 ph10 231 functions are called.
402    
403    
404 nigel 63 BUILDING SHARED AND STATIC LIBRARIES
405    
406 ph10 453 The PCRE building process uses libtool to build both shared and static
407     Unix libraries by default. You can suppress one of these by adding one
408 nigel 73 of
409 nigel 63
410 nigel 73 --disable-shared
411     --disable-static
412 nigel 63
413 nigel 73 to the configure command, as required.
414 nigel 63
415    
416     POSIX MALLOC USAGE
417    
418 nigel 75 When PCRE is called through the POSIX interface (see the pcreposix doc-
419 ph10 453 umentation), additional working storage is required for holding the
420     pointers to capturing substrings, because PCRE requires three integers
421     per substring, whereas the POSIX interface provides only two. If the
422 nigel 73 number of expected substrings is small, the wrapper function uses space
423     on the stack, because this is faster than using malloc() for each call.
424     The default threshold above which the stack is no longer used is 10; it
425     can be changed by adding a setting such as
426 nigel 63
427 nigel 73 --with-posix-malloc-threshold=20
428 nigel 63
429 nigel 73 to the configure command.
430 nigel 63
431    
432     HANDLING VERY LARGE PATTERNS
433    
434 ph10 453 Within a compiled pattern, offset values are used to point from one
435     part to another (for example, from an opening parenthesis to an alter-
436     nation metacharacter). By default, two-byte values are used for these
437     offsets, leading to a maximum size for a compiled pattern of around
438     64K. This is sufficient to handle all but the most gigantic patterns.
439 ph10 461 Nevertheless, some people do want to process truyl enormous patterns,
440     so it is possible to compile PCRE to use three-byte or four-byte off-
441     sets by adding a setting such as
442 nigel 63
443 nigel 73 --with-link-size=3
444 nigel 63
445 ph10 453 to the configure command. The value given must be 2, 3, or 4. Using
446     longer offsets slows down the operation of PCRE because it has to load
447 nigel 73 additional bytes when handling them.
448 nigel 63
449    
450 nigel 73 AVOIDING EXCESSIVE STACK USAGE
451    
452 nigel 77 When matching with the pcre_exec() function, PCRE implements backtrack-
453 ph10 453 ing by making recursive calls to an internal function called match().
454     In environments where the size of the stack is limited, this can se-
455     verely limit PCRE's operation. (The Unix environment does not usually
456 nigel 91 suffer from this problem, but it may sometimes be necessary to increase
457 ph10 453 the maximum stack size. There is a discussion in the pcrestack docu-
458     mentation.) An alternative approach to recursion that uses memory from
459     the heap to remember data, instead of using recursive function calls,
460     has been implemented to work round the problem of limited stack size.
461 nigel 91 If you want to build a version of PCRE that works this way, add
462 nigel 73
463     --disable-stack-for-recursion
464    
465 ph10 453 to the configure command. With this configuration, PCRE will use the
466     pcre_stack_malloc and pcre_stack_free variables to call memory manage-
467     ment functions. By default these point to malloc() and free(), but you
468 ph10 461 can replace the pointers so that your own functions are used instead.
469 nigel 73
470 ph10 453 Separate functions are provided rather than using pcre_malloc and
471     pcre_free because the usage is very predictable: the block sizes
472     requested are always the same, and the blocks are always freed in
473     reverse order. A calling program might be able to implement optimized
474     functions that perform better than malloc() and free(). PCRE runs
475 ph10 182 noticeably more slowly when built in this way. This option affects only
476 ph10 461 the pcre_exec() function; it is not relevant for pcre_dfa_exec().
477 nigel 73
478 ph10 182
479 nigel 91 LIMITING PCRE RESOURCE USAGE
480    
481 ph10 461 Internally, PCRE has a function called match(), which it calls repeat-
482     edly (sometimes recursively) when matching a pattern with the
483     pcre_exec() function. By controlling the maximum number of times this
484     function may be called during a single matching operation, a limit can
485     be placed on the resources used by a single call to pcre_exec(). The
486     limit can be changed at run time, as described in the pcreapi documen-
487     tation. The default is 10 million, but this can be changed by adding a
488 nigel 91 setting such as
489    
490     --with-match-limit=500000
491    
492 ph10 461 to the configure command. This setting has no effect on the
493 nigel 91 pcre_dfa_exec() matching function.
494    
495 ph10 461 In some environments it is desirable to limit the depth of recursive
496 nigel 91 calls of match() more strictly than the total number of calls, in order
497 ph10 461 to restrict the maximum amount of stack (or heap, if --disable-stack-
498 nigel 91 for-recursion is specified) that is used. A second limit controls this;
499 ph10 461 it defaults to the value that is set for --with-match-limit, which
500     imposes no additional constraints. However, you can set a lower limit
501 nigel 91 by adding, for example,
502    
503     --with-match-limit-recursion=10000
504    
505 ph10 461 to the configure command. This value can also be overridden at run
506 nigel 91 time.
507    
508    
509 ph10 128 CREATING CHARACTER TABLES AT BUILD TIME
510    
511 ph10 461 PCRE uses fixed tables for processing characters whose code values are
512     less than 256. By default, PCRE is built with a set of tables that are
513     distributed in the file pcre_chartables.c.dist. These tables are for
514 ph10 128 ASCII codes only. If you add
515    
516     --enable-rebuild-chartables
517    
518 ph10 461 to the configure command, the distributed tables are no longer used.
519     Instead, a program called dftables is compiled and run. This outputs
520 ph10 128 the source for new set of tables, created in the default locale of your
521     C runtime system. (This method of replacing the tables does not work if
522 ph10 461 you are cross compiling, because dftables is run on the local host. If
523     you need to create alternative tables when cross compiling, you will
524 ph10 128 have to do so "by hand".)
525    
526    
527 nigel 73 USING EBCDIC CODE
528    
529 ph10 461 PCRE assumes by default that it will run in an environment where the
530     character code is ASCII (or Unicode, which is a superset of ASCII).
531     This is the case for most computer operating systems. PCRE can, how-
532 ph10 197 ever, be compiled to run in an EBCDIC environment by adding
533 nigel 73
534     --enable-ebcdic
535    
536 ph10 128 to the configure command. This setting implies --enable-rebuild-charta-
537 ph10 461 bles. You should only use it if you know that you are in an EBCDIC
538     environment (for example, an IBM mainframe operating system). The
539 ph10 392 --enable-ebcdic option is incompatible with --enable-utf8.
540 nigel 73
541 nigel 93
542 ph10 286 PCREGREP OPTIONS FOR COMPRESSED FILE SUPPORT
543    
544     By default, pcregrep reads all files as plain text. You can build it so
545     that it recognizes files whose names end in .gz or .bz2, and reads them
546     with libz or libbz2, respectively, by adding one or both of
547    
548     --enable-pcregrep-libz
549     --enable-pcregrep-libbz2
550    
551     to the configure command. These options naturally require that the rel-
552 ph10 461 evant libraries are installed on your system. Configuration will fail
553 ph10 286 if they are not.
554    
555    
556 ph10 289 PCRETEST OPTION FOR LIBREADLINE SUPPORT
557    
558     If you add
559    
560     --enable-pcretest-libreadline
561    
562 ph10 461 to the configure command, pcretest is linked with the libreadline
563     library, and when its input is from a terminal, it reads it using the
564 ph10 289 readline() function. This provides line-editing and history facilities.
565 ph10 461 Note that libreadline is GPL-licensed, so if you distribute a binary of
566 ph10 289 pcretest linked in this way, there may be licensing issues.
567    
568 ph10 461 Setting this option causes the -lreadline option to be added to the
569     pcretest build. In many operating environments with a sytem-installed
570 ph10 345 libreadline this is sufficient. However, in some environments (e.g. if
571 ph10 461 an unmodified distribution version of readline is in use), some extra
572     configuration may be necessary. The INSTALL file for libreadline says
573 ph10 345 this:
574 ph10 289
575 ph10 345 "Readline uses the termcap functions, but does not link with the
576     termcap or curses library itself, allowing applications which link
577     with readline the to choose an appropriate library."
578    
579 ph10 461 If your environment has not been set up so that an appropriate library
580 ph10 345 is automatically included, you may need to add something like
581    
582     LIBS="-ncurses"
583    
584     immediately before the configure command.
585    
586    
587 nigel 93 SEE ALSO
588    
589     pcreapi(3), pcre_config(3).
590    
591 nigel 63
592 ph10 99 AUTHOR
593 nigel 63
594 ph10 99 Philip Hazel
595     University Computing Service
596     Cambridge CB2 3QH, England.
597    
598    
599     REVISION
600    
601 ph10 461 Last updated: 29 September 2009
602 ph10 392 Copyright (c) 1997-2009 University of Cambridge.
603 ph10 99 ------------------------------------------------------------------------------
604 ph10 535
605    
606 nigel 79 PCREMATCHING(3) PCREMATCHING(3)
607 nigel 63
608 nigel 79
609 nigel 77 NAME
610     PCRE - Perl-compatible regular expressions
611 nigel 73
612 nigel 77
613     PCRE MATCHING ALGORITHMS
614    
615     This document describes the two different algorithms that are available
616     in PCRE for matching a compiled regular expression against a given sub-
617     ject string. The "standard" algorithm is the one provided by the
618     pcre_exec() function. This works in the same was as Perl's matching
619     function, and provides a Perl-compatible matching operation.
620    
621     An alternative algorithm is provided by the pcre_dfa_exec() function;
622     this operates in a different way, and is not Perl-compatible. It has
623     advantages and disadvantages compared with the standard algorithm, and
624     these are described below.
625    
626     When there is only one possible way in which a given subject string can
627     match a pattern, the two algorithms give the same answer. A difference
628     arises, however, when there are multiple possibilities. For example, if
629     the pattern
630    
631     ^<.*>
632    
633     is matched against the string
634    
635     <something> <something else> <something further>
636    
637     there are three possible answers. The standard algorithm finds only one
638 nigel 93 of them, whereas the alternative algorithm finds all three.
639 nigel 77
640    
641     REGULAR EXPRESSIONS AS TREES
642    
643     The set of strings that are matched by a regular expression can be rep-
644     resented as a tree structure. An unlimited repetition in the pattern
645     makes the tree of infinite size, but it is still a tree. Matching the
646     pattern to a given subject string (from a given starting point) can be
647 nigel 91 thought of as a search of the tree. There are two ways to search a
648     tree: depth-first and breadth-first, and these correspond to the two
649     matching algorithms provided by PCRE.
650 nigel 77
651    
652     THE STANDARD MATCHING ALGORITHM
653    
654 ph10 148 In the terminology of Jeffrey Friedl's book "Mastering Regular Expres-
655     sions", the standard algorithm is an "NFA algorithm". It conducts a
656 nigel 77 depth-first search of the pattern tree. That is, it proceeds along a
657     single path through the tree, checking that the subject matches what is
658     required. When there is a mismatch, the algorithm tries any alterna-
659     tives at the current point, and if they all fail, it backs up to the
660     previous branch point in the tree, and tries the next alternative
661     branch at that level. This often involves backing up (moving to the
662     left) in the subject string as well. The order in which repetition
663     branches are tried is controlled by the greedy or ungreedy nature of
664     the quantifier.
665    
666     If a leaf node is reached, a matching string has been found, and at
667     that point the algorithm stops. Thus, if there is more than one possi-
668     ble match, this algorithm returns the first one that it finds. Whether
669     this is the shortest, the longest, or some intermediate length depends
670     on the way the greedy and ungreedy repetition quantifiers are specified
671     in the pattern.
672    
673     Because it ends up with a single path through the tree, it is rela-
674     tively straightforward for this algorithm to keep track of the sub-
675     strings that are matched by portions of the pattern in parentheses.
676     This provides support for capturing parentheses and back references.
677    
678    
679 nigel 93 THE ALTERNATIVE MATCHING ALGORITHM
680 nigel 77
681 nigel 93 This algorithm conducts a breadth-first search of the tree. Starting
682     from the first matching point in the subject, it scans the subject
683     string from left to right, once, character by character, and as it does
684     this, it remembers all the paths through the tree that represent valid
685     matches. In Friedl's terminology, this is a kind of "DFA algorithm",
686     though it is not implemented as a traditional finite state machine (it
687     keeps multiple states active simultaneously).
688 nigel 77
689 ph10 461 Although the general principle of this matching algorithm is that it
690     scans the subject string only once, without backtracking, there is one
691     exception: when a lookaround assertion is encountered, the characters
692     following or preceding the current point have to be independently
693     inspected.
694    
695 nigel 93 The scan continues until either the end of the subject is reached, or
696     there are no more unterminated paths. At this point, terminated paths
697     represent the different matching possibilities (if there are none, the
698     match has failed). Thus, if there is more than one possible match,
699 nigel 77 this algorithm finds all of them, and in particular, it finds the long-
700 ph10 461 est. There is an option to stop the algorithm after the first match
701     (which is necessarily the shortest) is found.
702 nigel 77
703     Note that all the matches that are found start at the same point in the
704     subject. If the pattern
705    
706     cat(er(pillar)?)
707    
708 nigel 93 is matched against the string "the caterpillar catchment", the result
709     will be the three strings "cat", "cater", and "caterpillar" that start
710 nigel 77 at the fourth character of the subject. The algorithm does not automat-
711     ically move on to find matches that start at later positions.
712    
713     There are a number of features of PCRE regular expressions that are not
714 nigel 93 supported by the alternative matching algorithm. They are as follows:
715 nigel 77
716 ph10 461 1. Because the algorithm finds all possible matches, the greedy or
717     ungreedy nature of repetition quantifiers is not relevant. Greedy and
718 nigel 93 ungreedy quantifiers are treated in exactly the same way. However, pos-
719 ph10 461 sessive quantifiers can make a difference when what follows could also
720 nigel 93 match what is quantified, for example in a pattern like this:
721 nigel 77
722 nigel 93 ^a++\w!
723    
724 ph10 461 This pattern matches "aaab!" but not "aaa!", which would be matched by
725     a non-possessive quantifier. Similarly, if an atomic group is present,
726     it is matched as if it were a standalone pattern at the current point,
727     and the longest match is then "locked in" for the rest of the overall
728 nigel 93 pattern.
729    
730 nigel 77 2. When dealing with multiple paths through the tree simultaneously, it
731 ph10 461 is not straightforward to keep track of captured substrings for the
732     different matching possibilities, and PCRE's implementation of this
733 nigel 77 algorithm does not attempt to do this. This means that no captured sub-
734     strings are available.
735    
736 ph10 461 3. Because no substrings are captured, back references within the pat-
737 nigel 77 tern are not supported, and cause errors if encountered.
738    
739 ph10 461 4. For the same reason, conditional expressions that use a backrefer-
740     ence as the condition or test for a specific group recursion are not
741 nigel 93 supported.
742 nigel 77
743 ph10 461 5. Because many paths through the tree may be active, the \K escape
744 ph10 172 sequence, which resets the start of the match when encountered (but may
745 ph10 461 be on some paths and not on others), is not supported. It causes an
746 ph10 172 error if encountered.
747    
748 ph10 461 6. Callouts are supported, but the value of the capture_top field is
749 nigel 77 always 1, and the value of the capture_last field is always -1.
750    
751 ph10 461 7. The \C escape sequence, which (in the standard algorithm) matches a
752     single byte, even in UTF-8 mode, is not supported because the alterna-
753     tive algorithm moves through the subject string one character at a
754 nigel 93 time, for all active paths through the tree.
755 nigel 77
756 ph10 461 8. Except for (*FAIL), the backtracking control verbs such as (*PRUNE)
757     are not supported. (*FAIL) is supported, and behaves like a failing
758 ph10 345 negative assertion.
759 nigel 77
760 ph10 211
761 nigel 93 ADVANTAGES OF THE ALTERNATIVE ALGORITHM
762 nigel 77
763 ph10 461 Using the alternative matching algorithm provides the following advan-
764 nigel 93 tages:
765 nigel 77
766     1. All possible matches (at a single point in the subject) are automat-
767 ph10 461 ically found, and in particular, the longest match is found. To find
768 nigel 77 more than one match using the standard algorithm, you have to do kludgy
769     things with callouts.
770    
771 ph10 461 2. Because the alternative algorithm scans the subject string just
772     once, and never needs to backtrack, it is possible to pass very long
773     subject strings to the matching function in several pieces, checking
774     for partial matching each time. The pcrepartial documentation gives
775     details of partial matching.
776 nigel 77
777    
778 nigel 93 DISADVANTAGES OF THE ALTERNATIVE ALGORITHM
779 nigel 77
780 nigel 93 The alternative algorithm suffers from a number of disadvantages:
781 nigel 77
782 ph10 453 1. It is substantially slower than the standard algorithm. This is
783     partly because it has to search for all possible matches, but is also
784 nigel 77 because it is less susceptible to optimization.
785    
786     2. Capturing parentheses and back references are not supported.
787    
788 nigel 93 3. Although atomic groups are supported, their use does not provide the
789     performance advantage that it does for the standard algorithm.
790 nigel 77
791    
792 ph10 99 AUTHOR
793 nigel 77
794 ph10 99 Philip Hazel
795     University Computing Service
796     Cambridge CB2 3QH, England.
797    
798    
799     REVISION
800    
801 ph10 461 Last updated: 29 September 2009
802 ph10 429 Copyright (c) 1997-2009 University of Cambridge.
803 ph10 99 ------------------------------------------------------------------------------
804 ph10 535
805    
806 nigel 79 PCREAPI(3) PCREAPI(3)
807 nigel 77
808 nigel 79
809 nigel 73 NAME
810     PCRE - Perl-compatible regular expressions
811    
812 nigel 77
813 nigel 75 PCRE NATIVE API
814 nigel 63
815 nigel 73 #include <pcre.h>
816 nigel 41
817 nigel 73 pcre *pcre_compile(const char *pattern, int options,
818     const char **errptr, int *erroffset,
819     const unsigned char *tableptr);
820 nigel 41
821 nigel 77 pcre *pcre_compile2(const char *pattern, int options,
822     int *errorcodeptr,
823     const char **errptr, int *erroffset,
824     const unsigned char *tableptr);
825    
826 nigel 73 pcre_extra *pcre_study(const pcre *code, int options,
827     const char **errptr);
828 nigel 41
829 nigel 73 int pcre_exec(const pcre *code, const pcre_extra *extra,
830     const char *subject, int length, int startoffset,
831     int options, int *ovector, int ovecsize);
832 nigel 41
833 nigel 77 int pcre_dfa_exec(const pcre *code, const pcre_extra *extra,
834     const char *subject, int length, int startoffset,
835     int options, int *ovector, int ovecsize,
836     int *workspace, int wscount);
837    
838 nigel 73 int pcre_copy_named_substring(const pcre *code,
839     const char *subject, int *ovector,
840     int stringcount, const char *stringname,
841     char *buffer, int buffersize);
842 nigel 63
843 nigel 73 int pcre_copy_substring(const char *subject, int *ovector,
844     int stringcount, int stringnumber, char *buffer,
845     int buffersize);
846 nigel 41
847 nigel 73 int pcre_get_named_substring(const pcre *code,
848     const char *subject, int *ovector,
849     int stringcount, const char *stringname,
850     const char **stringptr);
851 nigel 63
852 nigel 73 int pcre_get_stringnumber(const pcre *code,
853     const char *name);
854 nigel 63
855 nigel 91 int pcre_get_stringtable_entries(const pcre *code,
856     const char *name, char **first, char **last);
857    
858 nigel 73 int pcre_get_substring(const char *subject, int *ovector,
859     int stringcount, int stringnumber,
860     const char **stringptr);
861 nigel 41
862 nigel 73 int pcre_get_substring_list(const char *subject,
863     int *ovector, int stringcount, const char ***listptr);
864 nigel 41
865 nigel 73 void pcre_free_substring(const char *stringptr);
866 nigel 49
867 nigel 73 void pcre_free_substring_list(const char **stringptr);
868 nigel 49
869 nigel 73 const unsigned char *pcre_maketables(void);
870 nigel 41
871 nigel 73 int pcre_fullinfo(const pcre *code, const pcre_extra *extra,
872     int what, void *where);
873 nigel 43
874 nigel 73 int pcre_info(const pcre *code, int *optptr, int *firstcharptr);
875 nigel 63
876 nigel 77 int pcre_refcount(pcre *code, int adjust);
877    
878 nigel 73 int pcre_config(int what, void *where);
879 nigel 41
880 nigel 73 char *pcre_version(void);
881 nigel 63
882 nigel 73 void *(*pcre_malloc)(size_t);
883 nigel 41
884 nigel 73 void (*pcre_free)(void *);
885 nigel 41
886 nigel 73 void *(*pcre_stack_malloc)(size_t);
887 nigel 41
888 nigel 73 void (*pcre_stack_free)(void *);
889 nigel 41
890 nigel 73 int (*pcre_callout)(pcre_callout_block *);
891 nigel 41
892 nigel 73
893 nigel 75 PCRE API OVERVIEW
894 nigel 41
895 nigel 73 PCRE has its own native API, which is described in this document. There
896 nigel 93 are also some wrapper functions that correspond to the POSIX regular
897 nigel 77 expression API. These are described in the pcreposix documentation.
898     Both of these APIs define a set of C function calls. A C++ wrapper is
899     distributed with PCRE. It is documented in the pcrecpp page.
900 nigel 43
901 nigel 77 The native API C function prototypes are defined in the header file
902     pcre.h, and on Unix systems the library itself is called libpcre. It
903 nigel 75 can normally be accessed by adding -lpcre to the command for linking an
904     application that uses PCRE. The header file defines the macros
905     PCRE_MAJOR and PCRE_MINOR to contain the major and minor release num-
906     bers for the library. Applications can use these to include support
907     for different releases of PCRE.
908 nigel 41
909 ph10 535 In a Windows environment, if you want to statically link an application
910     program against a non-dll pcre.a file, you must define PCRE_STATIC
911     before including pcre.h or pcrecpp.h, because otherwise the pcre_mal-
912     loc() and pcre_free() exported functions will be declared
913     __declspec(dllimport), with unwanted results.
914    
915 nigel 77 The functions pcre_compile(), pcre_compile2(), pcre_study(), and
916     pcre_exec() are used for compiling and matching regular expressions in
917     a Perl-compatible manner. A sample program that demonstrates the sim-
918     plest way of using them is provided in the file called pcredemo.c in
919 ph10 429 the PCRE source distribution. A listing of this program is given in the
920     pcredemo documentation, and the pcresample documentation describes how
921     to compile and run it.
922 nigel 49
923 nigel 77 A second matching function, pcre_dfa_exec(), which is not Perl-compati-
924 ph10 429 ble, is also provided. This uses a different algorithm for the match-
925     ing. The alternative algorithm finds all possible matches (at a given
926 ph10 453 point in the subject), and scans the subject just once (unless there
927     are lookbehind assertions). However, this algorithm does not return
928     captured substrings. A description of the two matching algorithms and
929     their advantages and disadvantages is given in the pcrematching docu-
930     mentation.
931 nigel 63
932 ph10 453 In addition to the main compiling and matching functions, there are
933 nigel 77 convenience functions for extracting captured substrings from a subject
934     string that is matched by pcre_exec(). They are:
935    
936 nigel 73 pcre_copy_substring()
937     pcre_copy_named_substring()
938     pcre_get_substring()
939     pcre_get_named_substring()
940     pcre_get_substring_list()
941 nigel 75 pcre_get_stringnumber()
942 nigel 91 pcre_get_stringtable_entries()
943 nigel 63
944 nigel 73 pcre_free_substring() and pcre_free_substring_list() are also provided,
945     to free the memory used for extracted strings.
946 nigel 41
947 ph10 453 The function pcre_maketables() is used to build a set of character
948     tables in the current locale for passing to pcre_compile(),
949     pcre_exec(), or pcre_dfa_exec(). This is an optional facility that is
950     provided for specialist use. Most commonly, no special tables are
951     passed, in which case internal tables that are generated when PCRE is
952 nigel 77 built are used.
953 nigel 49
954 ph10 453 The function pcre_fullinfo() is used to find out information about a
955     compiled pattern; pcre_info() is an obsolete version that returns only
956     some of the available information, but is retained for backwards com-
957     patibility. The function pcre_version() returns a pointer to a string
958 nigel 73 containing the version of PCRE and its date of release.
959 nigel 41
960 ph10 453 The function pcre_refcount() maintains a reference count in a data
961     block containing a compiled pattern. This is provided for the benefit
962 nigel 77 of object-oriented applications.
963    
964 ph10 453 The global variables pcre_malloc and pcre_free initially contain the
965     entry points of the standard malloc() and free() functions, respec-
966 nigel 73 tively. PCRE calls the memory management functions via these variables,
967 ph10 453 so a calling program can replace them if it wishes to intercept the
968 nigel 73 calls. This should be done before calling any PCRE functions.
969 nigel 41
970 ph10 453 The global variables pcre_stack_malloc and pcre_stack_free are also
971     indirections to memory management functions. These special functions
972     are used only when PCRE is compiled to use the heap for remembering
973 nigel 77 data, instead of recursive function calls, when running the pcre_exec()
974 ph10 453 function. See the pcrebuild documentation for details of how to do
975     this. It is a non-standard way of building PCRE, for use in environ-
976     ments that have limited stacks. Because of the greater use of memory
977     management, it runs more slowly. Separate functions are provided so
978     that special-purpose external code can be used for this case. When
979     used, these functions are always called in a stack-like manner (last
980     obtained, first freed), and always for memory blocks of the same size.
981     There is a discussion about PCRE's stack usage in the pcrestack docu-
982 nigel 91 mentation.
983 nigel 41
984 nigel 73 The global variable pcre_callout initially contains NULL. It can be set
985 ph10 453 by the caller to a "callout" function, which PCRE will then call at
986     specified points during a matching operation. Details are given in the
987 nigel 73 pcrecallout documentation.
988 nigel 41
989 nigel 73
990 nigel 91 NEWLINES
991    
992 ph10 453 PCRE supports five different conventions for indicating line breaks in
993     strings: a single CR (carriage return) character, a single LF (line-
994 ph10 150 feed) character, the two-character sequence CRLF, any of the three pre-
995 ph10 453 ceding, or any Unicode newline sequence. The Unicode newline sequences
996     are the three just mentioned, plus the single characters VT (vertical
997     tab, U+000B), FF (formfeed, U+000C), NEL (next line, U+0085), LS (line
998 ph10 150 separator, U+2028), and PS (paragraph separator, U+2029).
999 nigel 93
1000 ph10 453 Each of the first three conventions is used by at least one operating
1001     system as its standard newline sequence. When PCRE is built, a default
1002     can be specified. The default default is LF, which is the Unix stan-
1003     dard. When PCRE is run, the default can be overridden, either when a
1004 nigel 93 pattern is compiled, or when it is matched.
1005    
1006 ph10 227 At compile time, the newline convention can be specified by the options
1007 ph10 453 argument of pcre_compile(), or it can be specified by special text at
1008 ph10 227 the start of the pattern itself; this overrides any other settings. See
1009     the pcrepattern page for details of the special character sequences.
1010    
1011 nigel 91 In the PCRE documentation the word "newline" is used to mean "the char-
1012 ph10 453 acter or pair of characters that indicate a line break". The choice of
1013     newline convention affects the handling of the dot, circumflex, and
1014 nigel 93 dollar metacharacters, the handling of #-comments in /x mode, and, when
1015 ph10 453 CRLF is a recognized line ending sequence, the match position advance-
1016 ph10 227 ment for a non-anchored pattern. There is more detail about this in the
1017 ph10 231 section on pcre_exec() options below.
1018 nigel 91
1019 ph10 453 The choice of newline convention does not affect the interpretation of
1020     the \n or \r escape sequences, nor does it affect what \R matches,
1021 ph10 231 which is controlled in a similar way, but by separate options.
1022 nigel 91
1023 ph10 231
1024 nigel 63 MULTITHREADING
1025    
1026 ph10 453 The PCRE functions can be used in multi-threading applications, with
1027 nigel 73 the proviso that the memory management functions pointed to by
1028     pcre_malloc, pcre_free, pcre_stack_malloc, and pcre_stack_free, and the
1029     callout function pointed to by pcre_callout, are shared by all threads.
1030 nigel 41
1031 ph10 453 The compiled form of a regular expression is not altered during match-
1032 nigel 73 ing, so the same compiled pattern can safely be used by several threads
1033     at once.
1034 nigel 41
1035    
1036 nigel 75 SAVING PRECOMPILED PATTERNS FOR LATER USE
1037    
1038     The compiled form of a regular expression can be saved and re-used at a
1039 ph10 453 later time, possibly by a different program, and even on a host other
1040     than the one on which it was compiled. Details are given in the
1041     pcreprecompile documentation. However, compiling a regular expression
1042     with one version of PCRE for use with a different version is not guar-
1043 ph10 155 anteed to work and may cause crashes.
1044 nigel 75
1045    
1046 nigel 63 CHECKING BUILD-TIME OPTIONS
1047 nigel 41
1048 nigel 73 int pcre_config(int what, void *where);
1049 nigel 63
1050 ph10 453 The function pcre_config() makes it possible for a PCRE client to dis-
1051 nigel 73 cover which optional features have been compiled into the PCRE library.
1052 ph10 453 The pcrebuild documentation has more details about these optional fea-
1053 nigel 73 tures.
1054 nigel 63
1055 ph10 453 The first argument for pcre_config() is an integer, specifying which
1056 nigel 73 information is required; the second argument is a pointer to a variable
1057 ph10 453 into which the information is placed. The following information is
1058 nigel 73 available:
1059 nigel 63
1060 nigel 73 PCRE_CONFIG_UTF8
1061 nigel 63
1062 ph10 453 The output is an integer that is set to one if UTF-8 support is avail-
1063 nigel 73 able; otherwise it is set to zero.
1064 nigel 63
1065 nigel 75 PCRE_CONFIG_UNICODE_PROPERTIES
1066    
1067 ph10 453 The output is an integer that is set to one if support for Unicode
1068 nigel 75 character properties is available; otherwise it is set to zero.
1069    
1070 nigel 73 PCRE_CONFIG_NEWLINE
1071 nigel 63
1072 ph10 453 The output is an integer whose value specifies the default character
1073     sequence that is recognized as meaning "newline". The four values that
1074 ph10 150 are supported are: 10 for LF, 13 for CR, 3338 for CRLF, -2 for ANYCRLF,
1075 ph10 453 and -1 for ANY. Though they are derived from ASCII, the same values
1076 ph10 392 are returned in EBCDIC environments. The default should normally corre-
1077     spond to the standard sequence for your operating system.
1078 nigel 63
1079 ph10 231 PCRE_CONFIG_BSR
1080    
1081     The output is an integer whose value indicates what character sequences
1082 ph10 453 the \R escape sequence matches by default. A value of 0 means that \R
1083     matches any Unicode line ending sequence; a value of 1 means that \R
1084 ph10 231 matches only CR, LF, or CRLF. The default can be overridden when a pat-
1085     tern is compiled or matched.
1086    
1087 nigel 73 PCRE_CONFIG_LINK_SIZE
1088 nigel 63
1089 ph10 453 The output is an integer that contains the number of bytes used for
1090 nigel 73 internal linkage in compiled regular expressions. The value is 2, 3, or
1091 ph10 453 4. Larger values allow larger regular expressions to be compiled, at
1092     the expense of slower matching. The default value of 2 is sufficient
1093     for all but the most massive patterns, since it allows the compiled
1094 nigel 73 pattern to be up to 64K in size.
1095 nigel 63
1096 nigel 73 PCRE_CONFIG_POSIX_MALLOC_THRESHOLD
1097 nigel 63
1098 ph10 453 The output is an integer that contains the threshold above which the
1099     POSIX interface uses malloc() for output vectors. Further details are
1100 nigel 73 given in the pcreposix documentation.
1101 nigel 63
1102 nigel 73 PCRE_CONFIG_MATCH_LIMIT
1103 nigel 63
1104 ph10 453 The output is a long integer that gives the default limit for the num-
1105     ber of internal matching function calls in a pcre_exec() execution.
1106 ph10 392 Further details are given with pcre_exec() below.
1107 nigel 63
1108 nigel 87 PCRE_CONFIG_MATCH_LIMIT_RECURSION
1109    
1110 ph10 392 The output is a long integer that gives the default limit for the depth
1111 ph10 453 of recursion when calling the internal matching function in a
1112     pcre_exec() execution. Further details are given with pcre_exec()
1113 ph10 392 below.
1114 nigel 87
1115 nigel 73 PCRE_CONFIG_STACKRECURSE
1116 nigel 63
1117 ph10 453 The output is an integer that is set to one if internal recursion when
1118 nigel 77 running pcre_exec() is implemented by recursive function calls that use
1119 ph10 453 the stack to remember their state. This is the usual way that PCRE is
1120 nigel 77 compiled. The output is zero if PCRE was compiled to use blocks of data
1121 ph10 453 on the heap instead of recursive function calls. In this case,
1122     pcre_stack_malloc and pcre_stack_free are called to manage memory
1123 nigel 77 blocks on the heap, thus avoiding the use of the stack.
1124 nigel 73
1125    
1126 nigel 41 COMPILING A PATTERN
1127 nigel 63
1128 nigel 73 pcre *pcre_compile(const char *pattern, int options,
1129     const char **errptr, int *erroffset,
1130     const unsigned char *tableptr);
1131 nigel 63
1132 nigel 77 pcre *pcre_compile2(const char *pattern, int options,
1133     int *errorcodeptr,
1134     const char **errptr, int *erroffset,
1135     const unsigned char *tableptr);
1136 nigel 41
1137 nigel 77 Either of the functions pcre_compile() or pcre_compile2() can be called
1138     to compile a pattern into an internal form. The only difference between
1139 ph10 453 the two interfaces is that pcre_compile2() has an additional argument,
1140 ph10 461 errorcodeptr, via which a numerical error code can be returned. To
1141     avoid too much repetition, we refer just to pcre_compile() below, but
1142     the information applies equally to pcre_compile2().
1143 nigel 77
1144     The pattern is a C string terminated by a binary zero, and is passed in
1145 ph10 453 the pattern argument. A pointer to a single block of memory that is
1146     obtained via pcre_malloc is returned. This contains the compiled code
1147 nigel 77 and related data. The pcre type is defined for the returned block; this
1148     is a typedef for a structure whose contents are not externally defined.
1149 nigel 91 It is up to the caller to free the memory (via pcre_free) when it is no
1150     longer required.
1151 nigel 77
1152 ph10 453 Although the compiled code of a PCRE regex is relocatable, that is, it
1153 nigel 73 does not depend on memory location, the complete pcre data block is not
1154 ph10 453 fully relocatable, because it may contain a copy of the tableptr argu-
1155 nigel 75 ment, which is an address (see below).
1156 nigel 41
1157 nigel 93 The options argument contains various bit settings that affect the com-
1158 ph10 453 pilation. It should be zero if no options are required. The available
1159     options are described below. Some of them (in particular, those that
1160 ph10 461 are compatible with Perl, but some others as well) can also be set and
1161 ph10 453 unset from within the pattern (see the detailed description in the
1162     pcrepattern documentation). For those options that can be different in
1163     different parts of the pattern, the contents of the options argument
1164 ph10 461 specifies their settings at the start of compilation and execution. The
1165     PCRE_ANCHORED, PCRE_BSR_xxx, and PCRE_NEWLINE_xxx options can be set at
1166     the time of matching as well as at compile time.
1167 nigel 41
1168 nigel 73 If errptr is NULL, pcre_compile() returns NULL immediately. Otherwise,
1169 ph10 453 if compilation of a pattern fails, pcre_compile() returns NULL, and
1170 nigel 73 sets the variable pointed to by errptr to point to a textual error mes-
1171 nigel 87 sage. This is a static string that is part of the library. You must not
1172 ph10 454 try to free it. The byte offset from the start of the pattern to the
1173 ph10 461 character that was being processed when the error was discovered is
1174 ph10 454 placed in the variable pointed to by erroffset, which must not be NULL.
1175     If it is, an immediate error is given. Some errors are not detected
1176     until checks are carried out when the whole pattern has been scanned;
1177     in this case the offset is set to the end of the pattern.
1178 nigel 53
1179 ph10 453 If pcre_compile2() is used instead of pcre_compile(), and the error-
1180     codeptr argument is not NULL, a non-zero error code number is returned
1181     via this argument in the event of an error. This is in addition to the
1182 nigel 77 textual error message. Error codes and messages are listed below.
1183    
1184 ph10 453 If the final argument, tableptr, is NULL, PCRE uses a default set of
1185     character tables that are built when PCRE is compiled, using the
1186     default C locale. Otherwise, tableptr must be an address that is the
1187     result of a call to pcre_maketables(). This value is stored with the
1188     compiled pattern, and used again by pcre_exec(), unless another table
1189 nigel 75 pointer is passed to it. For more discussion, see the section on locale
1190     support below.
1191 nigel 53
1192 ph10 453 This code fragment shows a typical straightforward call to pcre_com-
1193 nigel 73 pile():
1194 nigel 41
1195 nigel 73 pcre *re;
1196     const char *error;
1197     int erroffset;
1198     re = pcre_compile(
1199     "^A.*Z", /* the pattern */
1200     0, /* default options */
1201     &error, /* for error message */
1202     &erroffset, /* for error offset */
1203     NULL); /* use default character tables */
1204 nigel 41
1205 ph10 453 The following names for option bits are defined in the pcre.h header
1206 nigel 75 file:
1207 nigel 41
1208 nigel 73 PCRE_ANCHORED
1209 nigel 41
1210 nigel 73 If this bit is set, the pattern is forced to be "anchored", that is, it
1211 ph10 453 is constrained to match only at the first matching point in the string
1212     that is being searched (the "subject string"). This effect can also be
1213     achieved by appropriate constructs in the pattern itself, which is the
1214 nigel 73 only way to do it in Perl.
1215 nigel 41
1216 nigel 75 PCRE_AUTO_CALLOUT
1217    
1218     If this bit is set, pcre_compile() automatically inserts callout items,
1219 ph10 453 all with number 255, before each pattern item. For discussion of the
1220 nigel 75 callout facility, see the pcrecallout documentation.
1221    
1222 ph10 231 PCRE_BSR_ANYCRLF
1223     PCRE_BSR_UNICODE
1224    
1225     These options (which are mutually exclusive) control what the \R escape
1226 ph10 453 sequence matches. The choice is either to match only CR, LF, or CRLF,
1227 ph10 231 or to match any Unicode newline sequence. The default is specified when
1228     PCRE is built. It can be overridden from within the pattern, or by set-
1229     ting an option when a compiled pattern is matched.
1230    
1231 nigel 73 PCRE_CASELESS
1232 nigel 41
1233 ph10 453 If this bit is set, letters in the pattern match both upper and lower
1234     case letters. It is equivalent to Perl's /i option, and it can be
1235     changed within a pattern by a (?i) option setting. In UTF-8 mode, PCRE
1236     always understands the concept of case for characters whose values are
1237     less than 128, so caseless matching is always possible. For characters
1238     with higher values, the concept of case is supported if PCRE is com-
1239     piled with Unicode property support, but not otherwise. If you want to
1240     use caseless matching for characters 128 and above, you must ensure
1241     that PCRE is compiled with Unicode property support as well as with
1242 nigel 77 UTF-8 support.
1243 nigel 41
1244 nigel 73 PCRE_DOLLAR_ENDONLY
1245 nigel 41
1246 ph10 453 If this bit is set, a dollar metacharacter in the pattern matches only
1247     at the end of the subject string. Without this option, a dollar also
1248     matches immediately before a newline at the end of the string (but not
1249     before any other newlines). The PCRE_DOLLAR_ENDONLY option is ignored
1250     if PCRE_MULTILINE is set. There is no equivalent to this option in
1251 nigel 91 Perl, and no way to set it within a pattern.
1252 nigel 41
1253 nigel 73 PCRE_DOTALL
1254 nigel 41
1255 nigel 73 If this bit is set, a dot metacharater in the pattern matches all char-
1256 ph10 453 acters, including those that indicate newline. Without it, a dot does
1257     not match when the current position is at a newline. This option is
1258     equivalent to Perl's /s option, and it can be changed within a pattern
1259     by a (?s) option setting. A negative class such as [^a] always matches
1260 nigel 93 newline characters, independent of the setting of this option.
1261 nigel 63
1262 nigel 91 PCRE_DUPNAMES
1263    
1264 ph10 453 If this bit is set, names used to identify capturing subpatterns need
1265 nigel 91 not be unique. This can be helpful for certain types of pattern when it
1266 ph10 453 is known that only one instance of the named subpattern can ever be
1267     matched. There are more details of named subpatterns below; see also
1268 nigel 91 the pcrepattern documentation.
1269    
1270 nigel 73 PCRE_EXTENDED
1271 nigel 41
1272 ph10 453 If this bit is set, whitespace data characters in the pattern are
1273 nigel 77 totally ignored except when escaped or inside a character class. White-
1274     space does not include the VT character (code 11). In addition, charac-
1275     ters between an unescaped # outside a character class and the next new-
1276 ph10 453 line, inclusive, are also ignored. This is equivalent to Perl's /x
1277     option, and it can be changed within a pattern by a (?x) option set-
1278 nigel 91 ting.
1279 nigel 41
1280 ph10 453 This option makes it possible to include comments inside complicated
1281     patterns. Note, however, that this applies only to data characters.
1282     Whitespace characters may never appear within special character
1283     sequences in a pattern, for example within the sequence (?( which
1284 nigel 73 introduces a conditional subpattern.
1285 nigel 41
1286 nigel 73 PCRE_EXTRA
1287 nigel 41
1288 ph10 453 This option was invented in order to turn on additional functionality
1289     of PCRE that is incompatible with Perl, but it is currently of very
1290     little use. When set, any backslash in a pattern that is followed by a
1291     letter that has no special meaning causes an error, thus reserving
1292     these combinations for future expansion. By default, as in Perl, a
1293     backslash followed by a letter with no special meaning is treated as a
1294 ph10 518 literal. (Perl can, however, be persuaded to give an error for this, by
1295     running it with the -w option.) There are at present no other features
1296     controlled by this option. It can also be set by a (?X) option setting
1297     within a pattern.
1298 nigel 41
1299 nigel 77 PCRE_FIRSTLINE
1300    
1301 ph10 518 If this option is set, an unanchored pattern is required to match
1302     before or at the first newline in the subject string, though the
1303 nigel 91 matched text may continue over the newline.
1304 nigel 77
1305 ph10 345 PCRE_JAVASCRIPT_COMPAT
1306    
1307     If this option is set, PCRE's behaviour is changed in some ways so that
1308 ph10 518 it is compatible with JavaScript rather than Perl. The changes are as
1309 ph10 345 follows:
1310    
1311 ph10 518 (1) A lone closing square bracket in a pattern causes a compile-time
1312     error, because this is illegal in JavaScript (by default it is treated
1313 ph10 345 as a data character). Thus, the pattern AB]CD becomes illegal when this
1314     option is set.
1315    
1316 ph10 518 (2) At run time, a back reference to an unset subpattern group matches
1317     an empty string (by default this causes the current matching alterna-
1318     tive to fail). A pattern such as (\1)(a) succeeds when this option is
1319     set (assuming it can find an "a" in the subject), whereas it fails by
1320 ph10 345 default, for Perl compatibility.
1321    
1322 nigel 73 PCRE_MULTILINE
1323 nigel 41
1324 ph10 518 By default, PCRE treats the subject string as consisting of a single
1325     line of characters (even if it actually contains newlines). The "start
1326     of line" metacharacter (^) matches only at the start of the string,
1327     while the "end of line" metacharacter ($) matches only at the end of
1328 nigel 75 the string, or before a terminating newline (unless PCRE_DOLLAR_ENDONLY
1329     is set). This is the same as Perl.
1330 nigel 63
1331 ph10 518 When PCRE_MULTILINE it is set, the "start of line" and "end of line"
1332     constructs match immediately following or immediately before internal
1333     newlines in the subject string, respectively, as well as at the very
1334     start and end. This is equivalent to Perl's /m option, and it can be
1335 nigel 91 changed within a pattern by a (?m) option setting. If there are no new-
1336 ph10 518 lines in a subject string, or no occurrences of ^ or $ in a pattern,
1337 nigel 73 setting PCRE_MULTILINE has no effect.
1338 nigel 63
1339 nigel 91 PCRE_NEWLINE_CR
1340     PCRE_NEWLINE_LF
1341     PCRE_NEWLINE_CRLF
1342 ph10 150 PCRE_NEWLINE_ANYCRLF
1343 nigel 93 PCRE_NEWLINE_ANY
1344 nigel 91
1345 ph10 518 These options override the default newline definition that was chosen
1346     when PCRE was built. Setting the first or the second specifies that a
1347     newline is indicated by a single character (CR or LF, respectively).
1348     Setting PCRE_NEWLINE_CRLF specifies that a newline is indicated by the
1349     two-character CRLF sequence. Setting PCRE_NEWLINE_ANYCRLF specifies
1350 ph10 150 that any of the three preceding sequences should be recognized. Setting
1351 ph10 518 PCRE_NEWLINE_ANY specifies that any Unicode newline sequence should be
1352 ph10 150 recognized. The Unicode newline sequences are the three just mentioned,
1353 ph10 518 plus the single characters VT (vertical tab, U+000B), FF (formfeed,
1354     U+000C), NEL (next line, U+0085), LS (line separator, U+2028), and PS
1355     (paragraph separator, U+2029). The last two are recognized only in
1356 ph10 150 UTF-8 mode.
1357 nigel 91
1358 ph10 518 The newline setting in the options word uses three bits that are
1359 ph10 150 treated as a number, giving eight possibilities. Currently only six are
1360 ph10 518 used (default plus the five values above). This means that if you set
1361     more than one newline option, the combination may or may not be sensi-
1362 ph10 150 ble. For example, PCRE_NEWLINE_CR with PCRE_NEWLINE_LF is equivalent to
1363 ph10 518 PCRE_NEWLINE_CRLF, but other combinations may yield unused numbers and
1364 ph10 150 cause an error.
1365 nigel 91
1366 ph10 518 The only time that a line break is specially recognized when compiling
1367     a pattern is if PCRE_EXTENDED is set, and an unescaped # outside a
1368     character class is encountered. This indicates a comment that lasts
1369     until after the next line break sequence. In other circumstances, line
1370     break sequences are treated as literal data, except that in
1371 nigel 93 PCRE_EXTENDED mode, both CR and LF are treated as whitespace characters
1372     and are therefore ignored.
1373    
1374     The newline option that is set at compile time becomes the default that
1375 ph10 392 is used for pcre_exec() and pcre_dfa_exec(), but it can be overridden.
1376 nigel 93
1377 nigel 73 PCRE_NO_AUTO_CAPTURE
1378 nigel 41
1379 nigel 73 If this option is set, it disables the use of numbered capturing paren-
1380 ph10 518 theses in the pattern. Any opening parenthesis that is not followed by
1381     ? behaves as if it were followed by ?: but named parentheses can still
1382     be used for capturing (and they acquire numbers in the usual way).
1383 nigel 73 There is no equivalent of this option in Perl.
1384 nigel 41
1385 ph10 535 PCRE_UCP
1386    
1387     This option changes the way PCRE processes \b, \d, \s, \w, and some of
1388     the POSIX character classes. By default, only ASCII characters are rec-
1389     ognized, but if PCRE_UCP is set, Unicode properties are used instead to
1390     classify characters. More details are given in the section on generic
1391     character types in the pcrepattern page. If you set PCRE_UCP, matching
1392     one of the items it affects takes much longer. The option is available
1393     only if PCRE has been compiled with Unicode property support.
1394    
1395 nigel 73 PCRE_UNGREEDY
1396 nigel 41
1397 ph10 518 This option inverts the "greediness" of the quantifiers so that they
1398     are not greedy by default, but become greedy if followed by "?". It is
1399     not compatible with Perl. It can also be set by a (?U) option setting
1400 nigel 73 within the pattern.
1401 nigel 41
1402 nigel 73 PCRE_UTF8
1403 nigel 49
1404 ph10 518 This option causes PCRE to regard both the pattern and the subject as
1405     strings of UTF-8 characters instead of single-byte character strings.
1406     However, it is available only when PCRE is built to include UTF-8 sup-
1407     port. If not, the use of this option provokes an error. Details of how
1408     this option changes the behaviour of PCRE are given in the section on
1409 nigel 75 UTF-8 support in the main pcre page.
1410 nigel 71
1411 nigel 73 PCRE_NO_UTF8_CHECK
1412 nigel 71
1413 nigel 73 When PCRE_UTF8 is set, the validity of the pattern as a UTF-8 string is
1414 ph10 518 automatically checked. There is a discussion about the validity of
1415     UTF-8 strings in the main pcre page. If an invalid UTF-8 sequence of
1416     bytes is found, pcre_compile() returns an error. If you already know
1417 ph10 211 that your pattern is valid, and you want to skip this check for perfor-
1418 ph10 518 mance reasons, you can set the PCRE_NO_UTF8_CHECK option. When it is
1419     set, the effect of passing an invalid UTF-8 string as a pattern is
1420     undefined. It may cause your program to crash. Note that this option
1421     can also be passed to pcre_exec() and pcre_dfa_exec(), to suppress the
1422 ph10 211 UTF-8 validity checking of subject strings.
1423 nigel 71
1424 nigel 73
1425 nigel 77 COMPILATION ERROR CODES
1426    
1427 ph10 518 The following table lists the error codes than may be returned by
1428     pcre_compile2(), along with the error messages that may be returned by
1429     both compiling functions. As PCRE has developed, some error codes have
1430 nigel 93 fallen out of use. To avoid confusion, they have not been re-used.
1431 nigel 77
1432     0 no error
1433     1 \ at end of pattern
1434     2 \c at end of pattern
1435     3 unrecognized character follows \
1436     4 numbers out of order in {} quantifier
1437     5 number too big in {} quantifier
1438     6 missing terminating ] for character class
1439     7 invalid escape sequence in character class
1440     8 range out of order in character class
1441     9 nothing to repeat
1442 nigel 93 10 [this code is not in use]
1443 nigel 77 11 internal error: unexpected repeat
1444 ph10 292 12 unrecognized character after (? or (?-
1445 nigel 77 13 POSIX named classes are supported only within a class
1446     14 missing )
1447     15 reference to non-existent subpattern
1448     16 erroffset passed as NULL
1449     17 unknown option bit(s) set
1450     18 missing ) after comment
1451 nigel 93 19 [this code is not in use]
1452 ph10 292 20 regular expression is too large
1453 nigel 77 21 failed to get memory
1454     22 unmatched parentheses
1455     23 internal error: code overflow
1456     24 unrecognized character after (?<
1457     25 lookbehind assertion is not fixed length
1458 nigel 91 26 malformed number or name after (?(
1459 nigel 77 27 conditional group contains more than two branches
1460     28 assertion expected after (?(
1461 ph10 182 29 (?R or (?[+-]digits must be followed by )
1462 nigel 77 30 unknown POSIX class name
1463     31 POSIX collating elements are not supported
1464     32 this version of PCRE is not compiled with PCRE_UTF8 support
1465 nigel 93 33 [this code is not in use]
1466 nigel 77 34 character value in \x{...} sequence is too large
1467     35 invalid condition (?(0)
1468     36 \C not allowed in lookbehind assertion
1469     37 PCRE does not support \L, \l, \N, \U, or \u
1470     38 number after (?C is > 255
1471     39 closing ) for (?C expected
1472     40 recursive call could loop indefinitely
1473     41 unrecognized character after (?P
1474 nigel 93 42 syntax error in subpattern name (missing terminator)
1475 nigel 91 43 two named subpatterns have the same name
1476 nigel 77 44 invalid UTF-8 string
1477     45 support for \P, \p, and \X has not been compiled
1478     46 malformed \P or \p sequence
1479     47 unknown property name after \P or \p
1480 nigel 91 48 subpattern name is too long (maximum 32 characters)
1481 ph10 292 49 too many named subpatterns (maximum 10000)
1482 ph10 202 50 [this code is not in use]
1483 nigel 91 51 octal value is greater than \377 (not in UTF-8 mode)
1484 nigel 93 52 internal error: overran compiling workspace
1485 ph10 518 53 internal error: previously-checked referenced subpattern not
1486 nigel 93 found
1487     54 DEFINE group contains more than one branch
1488     55 repeating a DEFINE group is not allowed
1489 ph10 231 56 inconsistent NEWLINE options
1490 ph10 345 57 \g is not followed by a braced, angle-bracketed, or quoted
1491     name/number or by a plain number
1492     58 a numbered reference must not be zero
1493 ph10 512 59 an argument is not allowed for (*ACCEPT), (*FAIL), or (*COMMIT)
1494 ph10 292 60 (*VERB) not recognized
1495     61 number is too big
1496     62 subpattern name expected
1497     63 digit expected after (?+
1498 ph10 345 64 ] is an invalid data character in JavaScript compatibility mode
1499 ph10 518 65 different names for subpatterns of the same number are not
1500 ph10 512 allowed
1501     66 (*MARK) must have an argument
1502 ph10 535 67 this version of PCRE is not compiled with PCRE_UCP support
1503 nigel 77
1504 ph10 518 The numbers 32 and 10000 in errors 48 and 49 are defaults; different
1505 ph10 292 values may be used if the limits were changed when PCRE was built.
1506 nigel 77
1507 ph10 292
1508 nigel 63 STUDYING A PATTERN
1509 nigel 49
1510 nigel 77 pcre_extra *pcre_study(const pcre *code, int options
1511 nigel 73 const char **errptr);
1512 nigel 63
1513 ph10 518 If a compiled pattern is going to be used several times, it is worth
1514 nigel 75 spending more time analyzing it in order to speed up the time taken for
1515 ph10 518 matching. The function pcre_study() takes a pointer to a compiled pat-
1516 nigel 75 tern as its first argument. If studying the pattern produces additional
1517 ph10 518 information that will help speed up matching, pcre_study() returns a
1518     pointer to a pcre_extra block, in which the study_data field points to
1519 nigel 75 the results of the study.
1520 nigel 41
1521 nigel 75 The returned value from pcre_study() can be passed directly to
1522 ph10 518 pcre_exec() or pcre_dfa_exec(). However, a pcre_extra block also con-
1523     tains other fields that can be set by the caller before the block is
1524 ph10 461 passed; these are described below in the section on matching a pattern.
1525 nigel 63
1526 ph10 518 If studying the pattern does not produce any useful information,
1527 nigel 75 pcre_study() returns NULL. In that circumstance, if the calling program
1528 ph10 518 wants to pass any of the other fields to pcre_exec() or
1529 ph10 461 pcre_dfa_exec(), it must set up its own pcre_extra block.
1530 nigel 41
1531 ph10 518 The second argument of pcre_study() contains option bits. At present,
1532 nigel 75 no options are defined, and this argument should always be zero.
1533    
1534 ph10 518 The third argument for pcre_study() is a pointer for an error message.
1535     If studying succeeds (even if no data is returned), the variable it
1536     points to is set to NULL. Otherwise it is set to point to a textual
1537 nigel 87 error message. This is a static string that is part of the library. You
1538 ph10 518 must not try to free it. You should test the error pointer for NULL
1539 nigel 87 after calling pcre_study(), to be sure that it has run successfully.
1540 nigel 41
1541 nigel 73 This is a typical call to pcre_study():
1542 nigel 53
1543 nigel 73 pcre_extra *pe;
1544     pe = pcre_study(
1545     re, /* result of pcre_compile() */
1546     0, /* no options exist */
1547     &error); /* set to NULL or points to a message */
1548 nigel 53
1549 ph10 461 Studying a pattern does two things: first, a lower bound for the length
1550     of subject string that is needed to match the pattern is computed. This
1551     does not mean that there are any strings of that length that match, but
1552 ph10 518 it does guarantee that no shorter strings match. The value is used by
1553     pcre_exec() and pcre_dfa_exec() to avoid wasting time by trying to
1554     match strings that are shorter than the lower bound. You can find out
1555 ph10 461 the value in a calling program via the pcre_fullinfo() function.
1556 nigel 41
1557 ph10 461 Studying a pattern is also useful for non-anchored patterns that do not
1558 ph10 518 have a single fixed starting character. A bitmap of possible starting
1559     bytes is created. This speeds up finding a position in the subject at
1560 ph10 461 which to start matching.
1561 nigel 41
1562 ph10 461
1563 nigel 63 LOCALE SUPPORT
1564 nigel 41
1565 ph10 518 PCRE handles caseless matching, and determines whether characters are
1566     letters, digits, or whatever, by reference to a set of tables, indexed
1567     by character value. When running in UTF-8 mode, this applies only to
1568 ph10 535 characters with codes less than 128. By default, higher-valued codes
1569     never match escapes such as \w or \d, but they can be tested with \p if
1570     PCRE is built with Unicode character property support. Alternatively,
1571     the PCRE_UCP option can be set at compile time; this causes \w and
1572     friends to use Unicode property support instead of built-in tables. The
1573     use of locales with Unicode is discouraged. If you are handling charac-
1574     ters with codes greater than 128, you should either use UTF-8 and Uni-
1575     code, or use locales, but not try to mix the two.
1576 nigel 41
1577 ph10 518 PCRE contains an internal set of tables that are used when the final
1578     argument of pcre_compile() is NULL. These are sufficient for many
1579 ph10 142 applications. Normally, the internal tables recognize only ASCII char-
1580     acters. However, when PCRE is built, it is possible to cause the inter-
1581     nal tables to be rebuilt in the default "C" locale of the local system,
1582     which may cause them to be different.
1583 nigel 41
1584 ph10 518 The internal tables can always be overridden by tables supplied by the
1585 ph10 142 application that calls PCRE. These may be created in a different locale
1586 ph10 518 from the default. As more and more applications change to using Uni-
1587 ph10 142 code, the need for this locale support is expected to die away.
1588    
1589 ph10 518 External tables are built by calling the pcre_maketables() function,
1590     which has no arguments, in the relevant locale. The result can then be
1591     passed to pcre_compile() or pcre_exec() as often as necessary. For
1592     example, to build and use tables that are appropriate for the French
1593     locale (where accented characters with values greater than 128 are
1594 nigel 75 treated as letters), the following code could be used:
1595    
1596     setlocale(LC_CTYPE, "fr_FR");
1597 nigel 73 tables = pcre_maketables();
1598     re = pcre_compile(..., tables);
1599 nigel 41
1600 ph10 518 The locale name "fr_FR" is used on Linux and other Unix-like systems;
1601 ph10 142 if you are using Windows, the name for the French locale is "french".
1602    
1603 ph10 518 When pcre_maketables() runs, the tables are built in memory that is
1604     obtained via pcre_malloc. It is the caller's responsibility to ensure
1605     that the memory containing the tables remains available for as long as
1606 nigel 75 it is needed.
1607 nigel 41
1608 nigel 75 The pointer that is passed to pcre_compile() is saved with the compiled
1609 ph10 518 pattern, and the same tables are used via this pointer by pcre_study()
1610 nigel 75 and normally also by pcre_exec(). Thus, by default, for any single pat-
1611     tern, compilation, studying and matching all happen in the same locale,
1612     but different patterns can be compiled in different locales.
1613 nigel 41
1614 ph10 518 It is possible to pass a table pointer or NULL (indicating the use of
1615     the internal tables) to pcre_exec(). Although not intended for this
1616     purpose, this facility could be used to match a pattern in a different
1617 nigel 75 locale from the one in which it was compiled. Passing table pointers at
1618     run time is discussed below in the section on matching a pattern.
1619    
1620    
1621 nigel 63 INFORMATION ABOUT A PATTERN
1622 nigel 41
1623 nigel 73 int pcre_fullinfo(const pcre *code, const pcre_extra *extra,
1624     int what, void *where);
1625 nigel 63
1626 ph10 518 The pcre_fullinfo() function returns information about a compiled pat-
1627 nigel 73 tern. It replaces the obsolete pcre_info() function, which is neverthe-
1628     less retained for backwards compability (and is documented below).
1629 nigel 43
1630 ph10 518 The first argument for pcre_fullinfo() is a pointer to the compiled
1631     pattern. The second argument is the result of pcre_study(), or NULL if
1632     the pattern was not studied. The third argument specifies which piece
1633     of information is required, and the fourth argument is a pointer to a
1634     variable to receive the data. The yield of the function is zero for
1635 nigel 73 success, or one of the following negative numbers:
1636 nigel 41
1637 nigel 73 PCRE_ERROR_NULL the argument code was NULL
1638     the argument where was NULL
1639     PCRE_ERROR_BADMAGIC the "magic number" was not found
1640     PCRE_ERROR_BADOPTION the value of what was invalid
1641 nigel 53
1642 ph10 518 The "magic number" is placed at the start of each compiled pattern as
1643     an simple check against passing an arbitrary memory pointer. Here is a
1644     typical call of pcre_fullinfo(), to obtain the length of the compiled
1645 nigel 75 pattern:
1646 nigel 53
1647 nigel 73 int rc;
1648 nigel 91 size_t length;
1649 nigel 73 rc = pcre_fullinfo(
1650     re, /* result of pcre_compile() */
1651     pe, /* result of pcre_study(), or NULL */
1652     PCRE_INFO_SIZE, /* what is required */
1653     &length); /* where to put the data */
1654 nigel 43
1655 ph10 518 The possible values for the third argument are defined in pcre.h, and
1656 nigel 73 are as follows:
1657 nigel 43
1658 nigel 73 PCRE_INFO_BACKREFMAX
1659 nigel 41
1660 ph10 518 Return the number of the highest back reference in the pattern. The
1661     fourth argument should point to an int variable. Zero is returned if
1662 nigel 73 there are no back references.
1663 nigel 43
1664 nigel 73 PCRE_INFO_CAPTURECOUNT
1665 nigel 43
1666 ph10 518 Return the number of capturing subpatterns in the pattern. The fourth
1667 nigel 73 argument should point to an int variable.
1668 nigel 43
1669 nigel 77 PCRE_INFO_DEFAULT_TABLES
1670 nigel 75
1671 ph10 518 Return a pointer to the internal default character tables within PCRE.
1672     The fourth argument should point to an unsigned char * variable. This
1673 nigel 75 information call is provided for internal use by the pcre_study() func-
1674 ph10 518 tion. External callers can cause PCRE to use its internal tables by
1675 nigel 75 passing a NULL table pointer.
1676    
1677 nigel 73 PCRE_INFO_FIRSTBYTE
1678 nigel 43
1679 ph10 518 Return information about the first byte of any matched string, for a
1680     non-anchored pattern. The fourth argument should point to an int vari-
1681     able. (This option used to be called PCRE_INFO_FIRSTCHAR; the old name
1682 nigel 91 is still recognized for backwards compatibility.)
1683 nigel 41
1684 ph10 518 If there is a fixed first byte, for example, from a pattern such as
1685 nigel 93 (cat|cow|coyote), its value is returned. Otherwise, if either
1686 nigel 41
1687 ph10 518 (a) the pattern was compiled with the PCRE_MULTILINE option, and every
1688 nigel 73 branch starts with "^", or
1689 nigel 43
1690 nigel 73 (b) every branch of the pattern starts with ".*" and PCRE_DOTALL is not
1691     set (if it were set, the pattern would be anchored),
1692 nigel 41
1693 ph10 518 -1 is returned, indicating that the pattern matches only at the start
1694     of a subject string or after any newline within the string. Otherwise
1695 nigel 73 -2 is returned. For anchored patterns, -2 is returned.
1696 nigel 41
1697 nigel 73 PCRE_INFO_FIRSTTABLE
1698 nigel 41
1699 ph10 518 If the pattern was studied, and this resulted in the construction of a
1700 nigel 73 256-bit table indicating a fixed set of bytes for the first byte in any
1701 ph10 518 matching string, a pointer to the table is returned. Otherwise NULL is
1702     returned. The fourth argument should point to an unsigned char * vari-
1703 nigel 73 able.
1704 nigel 43
1705 ph10 227 PCRE_INFO_HASCRORLF
1706    
1707 ph10 518 Return 1 if the pattern contains any explicit matches for CR or LF
1708     characters, otherwise 0. The fourth argument should point to an int
1709     variable. An explicit match is either a literal CR or LF character, or
1710 ph10 243 \r or \n.
1711 ph10 227
1712 ph10 172 PCRE_INFO_JCHANGED
1713    
1714 ph10 518 Return 1 if the (?J) or (?-J) option setting is used in the pattern,
1715     otherwise 0. The fourth argument should point to an int variable. (?J)
1716 ph10 286 and (?-J) set and unset the local PCRE_DUPNAMES option, respectively.
1717 ph10 172
1718 nigel 73 PCRE_INFO_LASTLITERAL
1719 nigel 43
1720 ph10 518 Return the value of the rightmost literal byte that must exist in any
1721     matched string, other than at its start, if such a byte has been
1722 nigel 73 recorded. The fourth argument should point to an int variable. If there
1723 ph10 518 is no such byte, -1 is returned. For anchored patterns, a last literal
1724     byte is recorded only if it follows something of variable length. For
1725 nigel 73 example, for the pattern /^a\d+z\d+/ the returned value is "z", but for
1726     /^a\dz\d/ the returned value is -1.
1727 nigel 63
1728 ph10 461 PCRE_INFO_MINLENGTH
1729    
1730 ph10 518 If the pattern was studied and a minimum length for matching subject
1731     strings was computed, its value is returned. Otherwise the returned
1732     value is -1. The value is a number of characters, not bytes (this may
1733     be relevant in UTF-8 mode). The fourth argument should point to an int
1734     variable. A non-negative value is a lower bound to the length of any
1735     matching string. There may not be any strings of that length that do
1736 ph10 461 actually match, but every string that does match is at least that long.
1737    
1738 nigel 73 PCRE_INFO_NAMECOUNT
1739     PCRE_INFO_NAMEENTRYSIZE
1740     PCRE_INFO_NAMETABLE
1741 nigel 63
1742 ph10 518 PCRE supports the use of named as well as numbered capturing parenthe-
1743     ses. The names are just an additional way of identifying the parenthe-
1744 nigel 91 ses, which still acquire numbers. Several convenience functions such as
1745 ph10 518 pcre_get_named_substring() are provided for extracting captured sub-
1746     strings by name. It is also possible to extract the data directly, by
1747     first converting the name to a number in order to access the correct
1748 nigel 91 pointers in the output vector (described with pcre_exec() below). To do
1749 ph10 518 the conversion, you need to use the name-to-number map, which is
1750 nigel 91 described by these three values.
1751 nigel 63
1752 nigel 73 The map consists of a number of fixed-size entries. PCRE_INFO_NAMECOUNT
1753     gives the number of entries, and PCRE_INFO_NAMEENTRYSIZE gives the size
1754 ph10 518 of each entry; both of these return an int value. The entry size
1755     depends on the length of the longest name. PCRE_INFO_NAMETABLE returns
1756     a pointer to the first entry of the table (a pointer to char). The
1757 nigel 73 first two bytes of each entry are the number of the capturing parenthe-
1758 ph10 518 sis, most significant byte first. The rest of the entry is the corre-
1759 ph10 461 sponding name, zero terminated.
1760 nigel 63
1761 ph10 518 The names are in alphabetical order. Duplicate names may appear if (?|
1762 ph10 461 is used to create multiple groups with the same number, as described in
1763 ph10 518 the section on duplicate subpattern numbers in the pcrepattern page.
1764     Duplicate names for subpatterns with different numbers are permitted
1765     only if PCRE_DUPNAMES is set. In all cases of duplicate names, they
1766     appear in the table in the order in which they were found in the pat-
1767     tern. In the absence of (?| this is the order of increasing number;
1768 ph10 461 when (?| is used this is not necessarily the case because later subpat-
1769     terns may have lower numbers.
1770    
1771 ph10 518 As a simple example of the name/number table, consider the following
1772     pattern (assume PCRE_EXTENDED is set, so white space - including new-
1773 ph10 461 lines - is ignored):
1774    
1775 nigel 93 (?<date> (?<year>(\d\d)?\d\d) -
1776     (?<month>\d\d) - (?<day>\d\d) )
1777 nigel 63
1778 ph10 518 There are four named subpatterns, so the table has four entries, and
1779     each entry in the table is eight bytes long. The table is as follows,
1780 nigel 75 with non-printing bytes shows in hexadecimal, and undefined bytes shown
1781     as ??:
1782 nigel 63
1783 nigel 73 00 01 d a t e 00 ??
1784     00 05 d a y 00 ?? ??
1785     00 04 m o n t h 00
1786     00 02 y e a r 00 ??
1787 nigel 63
1788 ph10 518 When writing code to extract data from named subpatterns using the
1789     name-to-number map, remember that the length of the entries is likely
1790 nigel 91 to be different for each compiled pattern.
1791 nigel 63
1792 ph10 172 PCRE_INFO_OKPARTIAL
1793    
1794 ph10 518 Return 1 if the pattern can be used for partial matching with
1795     pcre_exec(), otherwise 0. The fourth argument should point to an int
1796     variable. From release 8.00, this always returns 1, because the
1797     restrictions that previously applied to partial matching have been
1798     lifted. The pcrepartial documentation gives details of partial match-
1799 ph10 453 ing.
1800 ph10 172
1801 nigel 73 PCRE_INFO_OPTIONS
1802 nigel 63
1803 ph10 518 Return a copy of the options with which the pattern was compiled. The
1804     fourth argument should point to an unsigned long int variable. These
1805 nigel 73 option bits are those specified in the call to pcre_compile(), modified
1806 ph10 197 by any top-level option settings at the start of the pattern itself. In
1807 ph10 518 other words, they are the options that will be in force when matching
1808     starts. For example, if the pattern /(?im)abc(?-i)d/ is compiled with
1809     the PCRE_EXTENDED option, the result is PCRE_CASELESS, PCRE_MULTILINE,
1810 ph10 197 and PCRE_EXTENDED.
1811 nigel 63
1812 ph10 518 A pattern is automatically anchored by PCRE if all of its top-level
1813 nigel 73 alternatives begin with one of the following:
1814 nigel 63
1815 nigel 73 ^ unless PCRE_MULTILINE is set
1816     \A always
1817     \G always
1818     .* if PCRE_DOTALL is set and there are no back
1819     references to the subpattern in which .* appears
1820 nigel 63
1821 nigel 73 For such patterns, the PCRE_ANCHORED bit is set in the options returned
1822     by pcre_fullinfo().
1823 nigel 63
1824 nigel 73 PCRE_INFO_SIZE
1825 nigel 63
1826 ph10 518 Return the size of the compiled pattern, that is, the value that was
1827 nigel 73 passed as the argument to pcre_malloc() when PCRE was getting memory in
1828     which to place the compiled data. The fourth argument should point to a
1829     size_t variable.
1830 nigel 63
1831 nigel 73 PCRE_INFO_STUDYSIZE
1832 nigel 63
1833 nigel 75 Return the size of the data block pointed to by the study_data field in
1834 ph10 518 a pcre_extra block. That is, it is the value that was passed to
1835 nigel 73 pcre_malloc() when PCRE was getting memory into which to place the data
1836 ph10 518 created by pcre_study(). If pcre_extra is NULL, or there is no study
1837     data, zero is returned. The fourth argument should point to a size_t
1838 nigel 73 variable.
1839 nigel 63
1840 nigel 73
1841 nigel 63 OBSOLETE INFO FUNCTION
1842    
1843 nigel 73 int pcre_info(const pcre *code, int *optptr, int *firstcharptr);
1844 nigel 63
1845 ph10 518 The pcre_info() function is now obsolete because its interface is too
1846     restrictive to return all the available data about a compiled pattern.
1847     New programs should use pcre_fullinfo() instead. The yield of
1848     pcre_info() is the number of capturing subpatterns, or one of the fol-
1849 nigel 73 lowing negative numbers:
1850 nigel 43
1851 nigel 73 PCRE_ERROR_NULL the argument code was NULL
1852     PCRE_ERROR_BADMAGIC the "magic number" was not found
1853 nigel 43
1854 ph10 518 If the optptr argument is not NULL, a copy of the options with which
1855     the pattern was compiled is placed in the integer it points to (see
1856 nigel 73 PCRE_INFO_OPTIONS above).
1857 nigel 43
1858 ph10 518 If the pattern is not anchored and the firstcharptr argument is not
1859     NULL, it is used to pass back information about the first character of
1860 nigel 73 any matched string (see PCRE_INFO_FIRSTBYTE above).
1861 nigel 43
1862    
1863 nigel 77 REFERENCE COUNTS
1864 nigel 53
1865 nigel 77 int pcre_refcount(pcre *code, int adjust);
1866    
1867 ph10 518 The pcre_refcount() function is used to maintain a reference count in
1868 nigel 77 the data block that contains a compiled pattern. It is provided for the
1869 ph10 518 benefit of applications that operate in an object-oriented manner,
1870 nigel 77 where different parts of the application may be using the same compiled
1871     pattern, but you want to free the block when they are all done.
1872    
1873     When a pattern is compiled, the reference count field is initialized to
1874 ph10 518 zero. It is changed only by calling this function, whose action is to
1875     add the adjust value (which may be positive or negative) to it. The
1876 nigel 77 yield of the function is the new value. However, the value of the count
1877 ph10 518 is constrained to lie between 0 and 65535, inclusive. If the new value
1878 nigel 77 is outside these limits, it is forced to the appropriate limit value.
1879    
1880 ph10 518 Except when it is zero, the reference count is not correctly preserved
1881     if a pattern is compiled on one host and then transferred to a host
1882 nigel 77 whose byte-order is different. (This seems a highly unlikely scenario.)
1883    
1884    
1885     MATCHING A PATTERN: THE TRADITIONAL FUNCTION
1886    
1887 nigel 73 int pcre_exec(const pcre *code, const pcre_extra *extra,
1888     const char *subject, int length, int startoffset,
1889     int options, int *ovector, int ovecsize);
1890 nigel 53
1891 ph10 518 The function pcre_exec() is called to match a subject string against a
1892     compiled pattern, which is passed in the code argument. If the pattern
1893     was studied, the result of the study should be passed in the extra
1894     argument. This function is the main matching facility of the library,
1895 nigel 77 and it operates in a Perl-like manner. For specialist use there is also
1896 ph10 518 an alternative matching function, which is described below in the sec-
1897 nigel 77 tion about the pcre_dfa_exec() function.
1898 nigel 41
1899 ph10 518 In most applications, the pattern will have been compiled (and option-
1900     ally studied) in the same process that calls pcre_exec(). However, it
1901 nigel 75 is possible to save compiled patterns and study data, and then use them
1902 ph10 518 later in different processes, possibly even on different hosts. For a
1903 nigel 75 discussion about this, see the pcreprecompile documentation.
1904    
1905 nigel 73 Here is an example of a simple call to pcre_exec():
1906 nigel 53
1907 nigel 73 int rc;
1908     int ovector[30];
1909     rc = pcre_exec(
1910     re, /* result of pcre_compile() */
1911     NULL, /* we didn't study the pattern */
1912     "some string", /* the subject string */
1913     11, /* the length of the subject string */
1914     0, /* start at offset 0 in the subject */
1915     0, /* default options */
1916 nigel 75 ovector, /* vector of integers for substring information */
1917 nigel 77 30); /* number of elements (NOT size in bytes) */
1918 nigel 53
1919 nigel 75 Extra data for pcre_exec()
1920 nigel 63
1921 ph10 518 If the extra argument is not NULL, it must point to a pcre_extra data
1922     block. The pcre_study() function returns such a block (when it doesn't
1923     return NULL), but you can also create one for yourself, and pass addi-
1924     tional information in it. The pcre_extra block contains the following
1925 nigel 87 fields (not necessarily in this order):
1926 nigel 75
1927 nigel 73 unsigned long int flags;
1928     void *study_data;
1929     unsigned long int match_limit;
1930 nigel 87 unsigned long int match_limit_recursion;
1931 nigel 73 void *callout_data;
1932 nigel 75 const unsigned char *tables;
1933 ph10 512 unsigned char **mark;
1934 nigel 63
1935 ph10 518 The flags field is a bitmap that specifies which of the other fields
1936 nigel 73 are set. The flag bits are:
1937 nigel 63
1938 nigel 73 PCRE_EXTRA_STUDY_DATA
1939     PCRE_EXTRA_MATCH_LIMIT
1940 nigel 87 PCRE_EXTRA_MATCH_LIMIT_RECURSION
1941 nigel 73 PCRE_EXTRA_CALLOUT_DATA
1942 nigel 75 PCRE_EXTRA_TABLES
1943 ph10 512 PCRE_EXTRA_MARK
1944 nigel 63
1945 ph10 518 Other flag bits should be set to zero. The study_data field is set in
1946     the pcre_extra block that is returned by pcre_study(), together with
1947 nigel 75 the appropriate flag bit. You should not set this yourself, but you may
1948 ph10 518 add to the block by setting the other fields and their corresponding
1949 nigel 75 flag bits.
1950 nigel 63
1951 nigel 73 The match_limit field provides a means of preventing PCRE from using up
1952 ph10 518 a vast amount of resources when running patterns that are not going to
1953     match, but which have a very large number of possibilities in their
1954     search trees. The classic example is a pattern that uses nested unlim-
1955 ph10 461 ited repeats.
1956 nigel 63
1957 ph10 518 Internally, PCRE uses a function called match() which it calls repeat-
1958     edly (sometimes recursively). The limit set by match_limit is imposed
1959     on the number of times this function is called during a match, which
1960     has the effect of limiting the amount of backtracking that can take
1961 nigel 87 place. For patterns that are not anchored, the count restarts from zero
1962     for each position in the subject string.
1963 nigel 75
1964 ph10 518 The default value for the limit can be set when PCRE is built; the
1965     default default is 10 million, which handles all but the most extreme
1966     cases. You can override the default by suppling pcre_exec() with a
1967     pcre_extra block in which match_limit is set, and
1968     PCRE_EXTRA_MATCH_LIMIT is set in the flags field. If the limit is
1969 nigel 73 exceeded, pcre_exec() returns PCRE_ERROR_MATCHLIMIT.
1970 nigel 63
1971 ph10 518 The match_limit_recursion field is similar to match_limit, but instead
1972 nigel 87 of limiting the total number of times that match() is called, it limits
1973 ph10 518 the depth of recursion. The recursion depth is a smaller number than
1974     the total number of calls, because not all calls to match() are recur-
1975 nigel 87 sive. This limit is of use only if it is set smaller than match_limit.
1976    
1977 ph10 518 Limiting the recursion depth limits the amount of stack that can be
1978 nigel 87 used, or, when PCRE has been compiled to use memory on the heap instead
1979     of the stack, the amount of heap memory that can be used.
1980    
1981 ph10 518 The default value for match_limit_recursion can be set when PCRE is
1982     built; the default default is the same value as the default for
1983     match_limit. You can override the default by suppling pcre_exec() with
1984     a pcre_extra block in which match_limit_recursion is set, and
1985     PCRE_EXTRA_MATCH_LIMIT_RECURSION is set in the flags field. If the
1986 nigel 87 limit is exceeded, pcre_exec() returns PCRE_ERROR_RECURSIONLIMIT.
1987    
1988 ph10 518 The callout_data field is used in conjunction with the "callout" fea-
1989 ph10 453 ture, and is described in the pcrecallout documentation.
1990 nigel 63
1991 ph10 518 The tables field is used to pass a character tables pointer to
1992     pcre_exec(); this overrides the value that is stored with the compiled
1993     pattern. A non-NULL value is stored with the compiled pattern only if
1994     custom tables were supplied to pcre_compile() via its tableptr argu-
1995 nigel 75 ment. If NULL is passed to pcre_exec() using this mechanism, it forces
1996 ph10 518 PCRE's internal tables to be used. This facility is helpful when re-
1997     using patterns that have been saved after compiling with an external
1998     set of tables, because the external tables might be at a different
1999     address when pcre_exec() is called. See the pcreprecompile documenta-
2000 nigel 75 tion for a discussion of saving compiled patterns for later use.
2001 nigel 41
2002 ph10 518 If PCRE_EXTRA_MARK is set in the flags field, the mark field must be
2003     set to point to a char * variable. If the pattern contains any back-
2004     tracking control verbs such as (*MARK:NAME), and the execution ends up
2005     with a name to pass back, a pointer to the name string (zero termi-
2006     nated) is placed in the variable pointed to by the mark field. The
2007     names are within the compiled pattern; if you wish to retain such a
2008     name you must copy it before freeing the memory of a compiled pattern.
2009     If there is no name to pass back, the variable pointed to by the mark
2010     field set to NULL. For details of the backtracking control verbs, see
2011 ph10 512 the section entitled "Backtracking control" in the pcrepattern documen-
2012     tation.
2013    
2014 nigel 75 Option bits for pcre_exec()
2015 nigel 71
2016 ph10 518 The unused bits of the options argument for pcre_exec() must be zero.
2017     The only bits that may be set are PCRE_ANCHORED, PCRE_NEWLINE_xxx,
2018     PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NOTEMPTY_ATSTART,
2019     PCRE_NO_START_OPTIMIZE, PCRE_NO_UTF8_CHECK, PCRE_PARTIAL_SOFT, and
2020 ph10 453 PCRE_PARTIAL_HARD.
2021 nigel 41
2022 nigel 75 PCRE_ANCHORED
2023 nigel 41
2024 ph10 518 The PCRE_ANCHORED option limits pcre_exec() to matching at the first
2025     matching position. If a pattern was compiled with PCRE_ANCHORED, or
2026     turned out to be anchored by virtue of its contents, it cannot be made
2027 nigel 75 unachored at matching time.
2028    
2029 ph10 231 PCRE_BSR_ANYCRLF
2030     PCRE_BSR_UNICODE
2031    
2032     These options (which are mutually exclusive) control what the \R escape
2033 ph10 518 sequence matches. The choice is either to match only CR, LF, or CRLF,
2034     or to match any Unicode newline sequence. These options override the
2035 ph10 231 choice that was made or defaulted when the pattern was compiled.
2036    
2037 nigel 91 PCRE_NEWLINE_CR
2038     PCRE_NEWLINE_LF
2039     PCRE_NEWLINE_CRLF
2040 ph10 150 PCRE_NEWLINE_ANYCRLF
2041 nigel 93 PCRE_NEWLINE_ANY
2042 nigel 91
2043 ph10 518 These options override the newline definition that was chosen or
2044     defaulted when the pattern was compiled. For details, see the descrip-
2045     tion of pcre_compile() above. During matching, the newline choice
2046     affects the behaviour of the dot, circumflex, and dollar metacharac-
2047     ters. It may also alter the way the match position is advanced after a
2048 ph10 227 match failure for an unanchored pattern.
2049 nigel 91
2050 ph10 518 When PCRE_NEWLINE_CRLF, PCRE_NEWLINE_ANYCRLF, or PCRE_NEWLINE_ANY is
2051     set, and a match attempt for an unanchored pattern fails when the cur-
2052     rent position is at a CRLF sequence, and the pattern contains no
2053     explicit matches for CR or LF characters, the match position is
2054 ph10 227 advanced by two characters instead of one, in other words, to after the
2055     CRLF.
2056    
2057     The above rule is a compromise that makes the most common cases work as
2058 ph10 518 expected. For example, if the pattern is .+A (and the PCRE_DOTALL
2059 ph10 227 option is not set), it does not match the string "\r\nA" because, after
2060 ph10 518 failing at the start, it skips both the CR and the LF before retrying.
2061     However, the pattern [\r\n]A does match that string, because it con-
2062 ph10 227 tains an explicit CR or LF reference, and so advances only by one char-
2063 ph10 231 acter after the first failure.
2064 ph10 227
2065 ph10 231 An explicit match for CR of LF is either a literal appearance of one of
2066 ph10 518 those characters, or one of the \r or \n escape sequences. Implicit
2067     matches such as [^X] do not count, nor does \s (which includes CR and
2068 ph10 231 LF in the characters that it matches).
2069    
2070 ph10 518 Notwithstanding the above, anomalous effects may still occur when CRLF
2071 ph10 227 is a valid newline sequence and explicit \r or \n escapes appear in the
2072     pattern.
2073    
2074 nigel 73 PCRE_NOTBOL
2075 nigel 41
2076 nigel 75 This option specifies that first character of the subject string is not
2077 ph10 518 the beginning of a line, so the circumflex metacharacter should not
2078     match before it. Setting this without PCRE_MULTILINE (at compile time)
2079     causes circumflex never to match. This option affects only the behav-
2080 nigel 77 iour of the circumflex metacharacter. It does not affect \A.
2081 nigel 41
2082 nigel 73 PCRE_NOTEOL
2083 nigel 41
2084 nigel 75 This option specifies that the end of the subject string is not the end
2085 ph10 518 of a line, so the dollar metacharacter should not match it nor (except
2086     in multiline mode) a newline immediately before it. Setting this with-
2087 nigel 75 out PCRE_MULTILINE (at compile time) causes dollar never to match. This
2088 ph10 518 option affects only the behaviour of the dollar metacharacter. It does
2089 nigel 75 not affect \Z or \z.
2090 nigel 41
2091 nigel 73 PCRE_NOTEMPTY
2092 nigel 41
2093 nigel 73 An empty string is not considered to be a valid match if this option is
2094 ph10 518 set. If there are alternatives in the pattern, they are tried. If all
2095     the alternatives match the empty string, the entire match fails. For
2096 nigel 73 example, if the pattern
2097 nigel 41
2098 nigel 73 a?b?
2099 nigel 41
2100 ph10 518 is applied to a string not beginning with "a" or "b", it matches an
2101     empty string at the start of the subject. With PCRE_NOTEMPTY set, this
2102 nigel 73 match is not valid, so PCRE searches further into the string for occur-
2103     rences of "a" or "b".
2104 nigel 41
2105 ph10 453 PCRE_NOTEMPTY_ATSTART
2106 nigel 41
2107 ph10 518 This is like PCRE_NOTEMPTY, except that an empty string match that is
2108     not at the start of the subject is permitted. If the pattern is
2109 ph10 453 anchored, such a match can occur only if the pattern contains \K.
2110    
2111 ph10 518 Perl has no direct equivalent of PCRE_NOTEMPTY or
2112     PCRE_NOTEMPTY_ATSTART, but it does make a special case of a pattern
2113     match of the empty string within its split() function, and when using
2114     the /g modifier. It is possible to emulate Perl's behaviour after
2115 ph10 453 matching a null string by first trying the match again at the same off-
2116 ph10 518 set with PCRE_NOTEMPTY_ATSTART and PCRE_ANCHORED, and then if that
2117 ph10 453 fails, by advancing the starting offset (see below) and trying an ordi-
2118 ph10 518 nary match again. There is some code that demonstrates how to do this
2119 ph10 453 in the pcredemo sample program.
2120    
2121 ph10 392 PCRE_NO_START_OPTIMIZE
2122    
2123 ph10 518 There are a number of optimizations that pcre_exec() uses at the start
2124     of a match, in order to speed up the process. For example, if it is
2125 ph10 545 known that an unanchored match must start with a specific character, it
2126     searches the subject for that character, and fails immediately if it
2127     cannot find it, without actually running the main matching function.
2128     This means that a special item such as (*COMMIT) at the start of a pat-
2129     tern is not considered until after a suitable starting point for the
2130     match has been found. When callouts are in use, these "start-up" opti-
2131     mizations can cause them to be skipped if the pattern is never actually
2132     used. The PCRE_NO_START_OPTIMIZE option disables the start-up optimiza-
2133     tions, causing performance to suffer, but ensuring that the callouts do
2134     occur, and that items such as (*COMMIT) are considered at every possi-
2135     ble starting position in the subject string.
2136 ph10 392
2137 nigel 75 PCRE_NO_UTF8_CHECK
2138    
2139     When PCRE_UTF8 is set at compile time, the validity of the subject as a
2140 ph10 545 UTF-8 string is automatically checked when pcre_exec() is subsequently
2141     called. The value of startoffset is also checked to ensure that it
2142     points to the start of a UTF-8 character. There is a discussion about
2143     the validity of UTF-8 strings in the section on UTF-8 support in the
2144     main pcre page. If an invalid UTF-8 sequence of bytes is found,
2145     pcre_exec() returns the error PCRE_ERROR_BADUTF8. If startoffset con-
2146 ph10 211 tains an invalid value, PCRE_ERROR_BADUTF8_OFFSET is returned.
2147 nigel 75
2148 ph10 545 If you already know that your subject is valid, and you want to skip
2149     these checks for performance reasons, you can set the
2150     PCRE_NO_UTF8_CHECK option when calling pcre_exec(). You might want to
2151     do this for the second and subsequent calls to pcre_exec() if you are
2152     making repeated calls to find all the matches in a single subject
2153     string. However, you should be sure that the value of startoffset
2154     points to the start of a UTF-8 character. When PCRE_NO_UTF8_CHECK is
2155     set, the effect of passing an invalid UTF-8 string as a subject, or a
2156     value of startoffset that does not point to the start of a UTF-8 char-
2157 nigel 75 acter, is undefined. Your program may crash.
2158    
2159 ph10 429 PCRE_PARTIAL_HARD
2160     PCRE_PARTIAL_SOFT
2161 nigel 75
2162 ph10 545 These options turn on the partial matching feature. For backwards com-
2163     patibility, PCRE_PARTIAL is a synonym for PCRE_PARTIAL_SOFT. A partial
2164     match occurs if the end of the subject string is reached successfully,
2165     but there are not enough subject characters to complete the match. If
2166     this happens when PCRE_PARTIAL_HARD is set, pcre_exec() immediately
2167     returns PCRE_ERROR_PARTIAL. Otherwise, if PCRE_PARTIAL_SOFT is set,
2168     matching continues by testing any other alternatives. Only if they all
2169     fail is PCRE_ERROR_PARTIAL returned (instead of PCRE_ERROR_NOMATCH).
2170 ph10 453 The portion of the string that was inspected when the partial match was
2171 ph10 545 found is set as the first matching string. There is a more detailed
2172 ph10 453 discussion in the pcrepartial documentation.
2173 nigel 75
2174     The string to be matched by pcre_exec()
2175    
2176 ph10 545 The subject string is passed to pcre_exec() as a pointer in subject, a
2177 ph10 371 length (in bytes) in length, and a starting byte offset in startoffset.
2178     In UTF-8 mode, the byte offset must point to the start of a UTF-8 char-
2179 ph10 545 acter. Unlike the pattern string, the subject may contain binary zero
2180     bytes. When the starting offset is zero, the search for a match starts
2181     at the beginning of the subject, and this is by far the most common
2182 ph10 371 case.
2183 nigel 63
2184 ph10 545 A non-zero starting offset is useful when searching for another match
2185     in the same subject by calling pcre_exec() again after a previous suc-
2186     cess. Setting startoffset differs from just passing over a shortened
2187     string and setting PCRE_NOTBOL in the case of a pattern that begins
2188 nigel 73 with any kind of lookbehind. For example, consider the pattern
2189 nigel 41
2190 nigel 73 \Biss\B
2191 nigel 41
2192 ph10 545 which finds occurrences of "iss" in the middle of words. (\B matches
2193     only if the current position in the subject is not a word boundary.)
2194     When applied to the string "Mississipi" the first call to pcre_exec()
2195     finds the first occurrence. If pcre_exec() is called again with just
2196     the remainder of the subject, namely "issipi", it does not match,
2197 nigel 73 because \B is always false at the start of the subject, which is deemed
2198 ph10 545 to be a word boundary. However, if pcre_exec() is passed the entire
2199 nigel 75 string again, but with startoffset set to 4, it finds the second occur-
2200 ph10 545 rence of "iss" because it is able to look behind the starting point to
2201 nigel 75 discover that it is preceded by a letter.
2202 nigel 41
2203 ph10 545 If a non-zero starting offset is passed when the pattern is anchored,
2204 nigel 75 one attempt to match at the given offset is made. This can only succeed
2205 ph10 545 if the pattern does not require the match to be at the start of the
2206 nigel 75 subject.
2207 nigel 41
2208 nigel 75 How pcre_exec() returns captured substrings
2209    
2210 ph10 545 In general, a pattern matches a certain portion of the subject, and in
2211     addition, further substrings from the subject may be picked out by
2212     parts of the pattern. Following the usage in Jeffrey Friedl's book,
2213     this is called "capturing" in what follows, and the phrase "capturing
2214     subpattern" is used for a fragment of a pattern that picks out a sub-
2215     string. PCRE supports several other kinds of parenthesized subpattern
2216 nigel 73 that do not cause substrings to be captured.
2217 nigel 65
2218 ph10 371 Captured substrings are returned to the caller via a vector of integers
2219 ph10 545 whose address is passed in ovector. The number of elements in the vec-
2220     tor is passed in ovecsize, which must be a non-negative number. Note:
2221 ph10 371 this argument is NOT the size of ovector in bytes.
2222 nigel 41
2223 ph10 545 The first two-thirds of the vector is used to pass back captured sub-
2224     strings, each substring using a pair of integers. The remaining third
2225     of the vector is used as workspace by pcre_exec() while matching cap-
2226     turing subpatterns, and is not available for passing back information.
2227     The number passed in ovecsize should always be a multiple of three. If
2228 nigel 75 it is not, it is rounded down.
2229    
2230 ph10 545 When a match is successful, information about captured substrings is
2231     returned in pairs of integers, starting at the beginning of ovector,
2232     and continuing up to two-thirds of its length at the most. The first
2233     element of each pair is set to the byte offset of the first character
2234     in a substring, and the second is set to the byte offset of the first
2235     character after the end of a substring. Note: these values are always
2236 ph10 371 byte offsets, even in UTF-8 mode. They are not character counts.
2237 nigel 41
2238 ph10 545 The first pair of integers, ovector[0] and ovector[1], identify the
2239     portion of the subject string matched by the entire pattern. The next
2240     pair is used for the first capturing subpattern, and so on. The value
2241 ph10 371 returned by pcre_exec() is one more than the highest numbered pair that
2242 ph10 545 has been set. For example, if two substrings have been captured, the
2243     returned value is 3. If there are no capturing subpatterns, the return
2244 ph10 371 value from a successful match is 1, indicating that just the first pair
2245     of offsets has been set.
2246    
2247 nigel 73 If a capturing subpattern is matched repeatedly, it is the last portion
2248 nigel 75 of the string that it matched that is returned.
2249 nigel 41
2250 ph10 545 If the vector is too small to hold all the captured substring offsets,
2251 nigel 75 it is used as far as possible (up to two-thirds of its length), and the
2252 ph10 545 function returns a value of zero. If the substring offsets are not of
2253     interest, pcre_exec() may be called with ovector passed as NULL and
2254     ovecsize as zero. However, if the pattern contains back references and
2255     the ovector is not big enough to remember the related substrings, PCRE
2256     has to get additional memory for use during matching. Thus it is usu-
2257 ph10 371 ally advisable to supply an ovector.
2258 nigel 41
2259 ph10 461 The pcre_fullinfo() function can be used to find out how many capturing
2260 ph10 545 subpatterns there are in a compiled pattern. The smallest size for
2261     ovector that will allow for n captured substrings, in addition to the
2262 nigel 91 offsets of the substring matched by the whole pattern, is (n+1)*3.
2263 nigel 41
2264 ph10 545 It is possible for capturing subpattern number n+1 to match some part
2265 nigel 91 of the subject when subpattern n has not been used at all. For example,
2266 ph10 545 if the string "abc" is matched against the pattern (a|(z))(bc) the
2267 nigel 91 return from the function is 4, and subpatterns 1 and 3 are matched, but
2268 ph10 545 2 is not. When this happens, both values in the offset pairs corre-
2269 nigel 91 sponding to unused subpatterns are set to -1.
2270 nigel 75
2271 ph10 545 Offset values that correspond to unused subpatterns at the end of the
2272     expression are also set to -1. For example, if the string "abc" is
2273     matched against the pattern (abc)(x(yz)?)? subpatterns 2 and 3 are not
2274     matched. The return from the function is 2, because the highest used
2275 nigel 91 capturing subpattern number is 1. However, you can refer to the offsets
2276 ph10 545 for the second and third capturing subpatterns if you wish (assuming
2277 nigel 91 the vector is large enough, of course).
2278    
2279 ph10 545 Some convenience functions are provided for extracting the captured
2280 nigel 91 substrings as separate strings. These are described below.
2281    
2282     Error return values from pcre_exec()
2283    
2284 ph10 545 If pcre_exec() fails, it returns a negative number. The following are
2285 nigel 73 defined in the header file:
2286 nigel 41
2287 nigel 73 PCRE_ERROR_NOMATCH (-1)
2288 nigel 41
2289 nigel 73 The subject string did not match the pattern.
2290 nigel 41
2291 nigel 73 PCRE_ERROR_NULL (-2)
2292 nigel 41
2293 ph10 545 Either code or subject was passed as NULL, or ovector was NULL and
2294 nigel 73 ovecsize was not zero.
2295 nigel 41
2296 nigel 73 PCRE_ERROR_BADOPTION (-3)
2297 nigel 41
2298 nigel 73 An unrecognized bit was set in the options argument.
2299 nigel 41
2300 nigel 73 PCRE_ERROR_BADMAGIC (-4)
2301 nigel 41
2302 ph10 545 PCRE stores a 4-byte "magic number" at the start of the compiled code,
2303 nigel 75 to catch the case when it is passed a junk pointer and to detect when a
2304     pattern that was compiled in an environment of one endianness is run in
2305 ph10 545 an environment with the other endianness. This is the error that PCRE
2306 nigel 75 gives when the magic number is not present.
2307 nigel 41
2308 nigel 93 PCRE_ERROR_UNKNOWN_OPCODE (-5)
2309 nigel 41
2310 nigel 73 While running the pattern match, an unknown item was encountered in the
2311 ph10 545 compiled pattern. This error could be caused by a bug in PCRE or by
2312 nigel 73 overwriting of the compiled pattern.
2313 nigel 41
2314 nigel 73 PCRE_ERROR_NOMEMORY (-6)
2315 nigel 41
2316 ph10 545 If a pattern contains back references, but the ovector that is passed
2317 nigel 73 to pcre_exec() is not big enough to remember the referenced substrings,
2318 ph10 545 PCRE gets a block of memory at the start of matching to use for this
2319     purpose. If the call via pcre_malloc() fails, this error is given. The
2320 nigel 75 memory is automatically freed at the end of matching.
2321 nigel 41
2322 ph10 545 This error is also given if pcre_stack_malloc() fails in pcre_exec().
2323     This can happen only when PCRE has been compiled with --disable-stack-
2324 ph10 535 for-recursion.
2325    
2326 nigel 73 PCRE_ERROR_NOSUBSTRING (-7)
2327 nigel 53
2328 ph10 545 This error is used by the pcre_copy_substring(), pcre_get_substring(),
2329 nigel 73 and pcre_get_substring_list() functions (see below). It is never
2330     returned by pcre_exec().
2331 nigel 63
2332 nigel 73 PCRE_ERROR_MATCHLIMIT (-8)
2333 nigel 63
2334 ph10 545 The backtracking limit, as specified by the match_limit field in a
2335     pcre_extra structure (or defaulted) was reached. See the description
2336 nigel 87 above.
2337    
2338 nigel 73 PCRE_ERROR_CALLOUT (-9)
2339 nigel 63
2340 nigel 73 This error is never generated by pcre_exec() itself. It is provided for
2341 ph10 545 use by callout functions that want to yield a distinctive error code.
2342 nigel 73 See the pcrecallout documentation for details.
2343 nigel 71
2344 nigel 73 PCRE_ERROR_BADUTF8 (-10)
2345 nigel 71
2346 ph10 545 A string that contains an invalid UTF-8 byte sequence was passed as a
2347 nigel 73 subject.
2348    
2349     PCRE_ERROR_BADUTF8_OFFSET (-11)
2350    
2351     The UTF-8 byte sequence that was passed as a subject was valid, but the
2352 ph10 545 value of startoffset did not point to the beginning of a UTF-8 charac-
2353 nigel 73 ter.
2354    
2355 nigel 77 PCRE_ERROR_PARTIAL (-12)
2356 nigel 73
2357 ph10 545 The subject string did not match, but it did match partially. See the
2358 nigel 75 pcrepartial documentation for details of partial matching.
2359    
2360 nigel 77 PCRE_ERROR_BADPARTIAL (-13)
2361 nigel 75
2362 ph10 545 This code is no longer in use. It was formerly returned when the
2363     PCRE_PARTIAL option was used with a compiled pattern containing items
2364     that were not supported for partial matching. From release 8.00
2365 ph10 429 onwards, there are no restrictions on partial matching.
2366 nigel 75
2367 nigel 77 PCRE_ERROR_INTERNAL (-14)
2368 nigel 75
2369 ph10 545 An unexpected internal error has occurred. This error could be caused
2370 nigel 75 by a bug in PCRE or by overwriting of the compiled pattern.
2371    
2372 nigel 77 PCRE_ERROR_BADCOUNT (-15)
2373 nigel 75
2374 ph10 392 This error is given if the value of the ovecsize argument is negative.
2375 nigel 75
2376 nigel 93 PCRE_ERROR_RECURSIONLIMIT (-21)
2377 nigel 75
2378 nigel 93 The internal recursion limit, as specified by the match_limit_recursion
2379 ph10 545 field in a pcre_extra structure (or defaulted) was reached. See the
2380 nigel 93 description above.
2381    
2382     PCRE_ERROR_BADNEWLINE (-23)
2383    
2384     An invalid combination of PCRE_NEWLINE_xxx options was given.
2385    
2386 ph10 197 Error numbers -16 to -20 and -22 are not used by pcre_exec().
2387 nigel 93
2388    
2389 nigel 63 EXTRACTING CAPTURED SUBSTRINGS BY NUMBER
2390    
2391 nigel 73 int pcre_copy_substring(const char *subject, int *ovector,
2392     int stringcount, int stringnumber, char *buffer,
2393     int buffersize);
2394 nigel 63
2395 nigel 73 int pcre_get_substring(const char *subject, int *ovector,
2396     int stringcount, int stringnumber,
2397     const char **stringptr);
2398 nigel 63
2399 nigel 73 int pcre_get_substring_list(const char *subject,
2400     int *ovector, int stringcount, const char ***listptr);
2401 nigel 63
2402 ph10 545 Captured substrings can be accessed directly by using the offsets
2403     returned by pcre_exec() in ovector. For convenience, the functions
2404 nigel 73 pcre_copy_substring(), pcre_get_substring(), and pcre_get_sub-
2405 ph10 545 string_list() are provided for extracting captured substrings as new,
2406     separate, zero-terminated strings. These functions identify substrings
2407     by number. The next section describes functions for extracting named
2408 nigel 91 substrings.
2409 nigel 41
2410 ph10 545 A substring that contains a binary zero is correctly extracted and has
2411     a further zero added on the end, but the result is not, of course, a C
2412     string. However, you can process such a string by referring to the
2413     length that is returned by pcre_copy_substring() and pcre_get_sub-
2414 nigel 91 string(). Unfortunately, the interface to pcre_get_substring_list() is
2415 ph10 545 not adequate for handling strings containing binary zeros, because the
2416 nigel 91 end of the final string is not independently indicated.
2417    
2418 ph10 545 The first three arguments are the same for all three of these func-
2419     tions: subject is the subject string that has just been successfully
2420 nigel 73 matched, ovector is a pointer to the vector of integer offsets that was
2421     passed to pcre_exec(), and stringcount is the number of substrings that
2422 ph10 545 were captured by the match, including the substring that matched the
2423 nigel 75 entire regular expression. This is the value returned by pcre_exec() if
2424 ph10 545 it is greater than zero. If pcre_exec() returned zero, indicating that
2425     it ran out of space in ovector, the value passed as stringcount should
2426 nigel 75 be the number of elements in the vector divided by three.
2427 nigel 41
2428 ph10 545 The functions pcre_copy_substring() and pcre_get_substring() extract a
2429     single substring, whose number is given as stringnumber. A value of
2430     zero extracts the substring that matched the entire pattern, whereas
2431     higher values extract the captured substrings. For pcre_copy_sub-
2432     string(), the string is placed in buffer, whose length is given by
2433     buffersize, while for pcre_get_substring() a new block of memory is
2434     obtained via pcre_malloc, and its address is returned via stringptr.
2435     The yield of the function is the length of the string, not including
2436 nigel 93 the terminating zero, or one of these error codes:
2437 nigel 41
2438 nigel 73 PCRE_ERROR_NOMEMORY (-6)
2439 nigel 41
2440 ph10 545 The buffer was too small for pcre_copy_substring(), or the attempt to
2441 nigel 73 get memory failed for pcre_get_substring().
2442 nigel 41
2443 nigel 73 PCRE_ERROR_NOSUBSTRING (-7)
2444 nigel 41
2445 nigel 73 There is no substring whose number is stringnumber.
2446 nigel 41
2447 ph10 545 The pcre_get_substring_list() function extracts all available sub-
2448     strings and builds a list of pointers to them. All this is done in a
2449 nigel 75 single block of memory that is obtained via pcre_malloc. The address of
2450 ph10 545 the memory block is returned via listptr, which is also the start of
2451     the list of string pointers. The end of the list is marked by a NULL
2452     pointer. The yield of the function is zero if all went well, or the
2453 nigel 93 error code
2454 nigel 41
2455 nigel 73 PCRE_ERROR_NOMEMORY (-6)
2456 nigel 41
2457 nigel 73 if the attempt to get the memory block failed.
2458 nigel 41
2459 ph10 545 When any of these functions encounter a substring that is unset, which
2460     can happen when capturing subpattern number n+1 matches some part of
2461     the subject, but subpattern n has not been used at all, they return an
2462 nigel 73 empty string. This can be distinguished from a genuine zero-length sub-
2463 ph10 545 string by inspecting the appropriate offset in ovector, which is nega-
2464 nigel 73 tive for unset substrings.
2465 nigel 41
2466 ph10 545 The two convenience functions pcre_free_substring() and pcre_free_sub-
2467     string_list() can be used to free the memory returned by a previous
2468 nigel 75 call of pcre_get_substring() or pcre_get_substring_list(), respec-
2469 ph10 545 tively. They do nothing more than call the function pointed to by
2470     pcre_free, which of course could be called directly from a C program.
2471     However, PCRE is used in some situations where it is linked via a spe-
2472     cial interface to another programming language that cannot use
2473     pcre_free directly; it is for these cases that the functions are pro-
2474 nigel 77 vided.
2475 nigel 41
2476 nigel 73
2477 nigel 63 EXTRACTING CAPTURED SUBSTRINGS BY NAME
2478 nigel 41
2479 nigel 75 int pcre_get_stringnumber(const pcre *code,
2480     const char *name);
2481    
2482 nigel 73 int pcre_copy_named_substring(const pcre *code,
2483     const char *subject, int *ovector,
2484     int stringcount, const char *stringname,
2485     char *buffer, int buffersize);
2486 nigel 41
2487 nigel 73 int pcre_get_named_substring(const pcre *code,
2488     const char *subject, int *ovector,
2489     int stringcount, const char *stringname,
2490     const char **stringptr);
2491 nigel 41
2492 ph10 545 To extract a substring by name, you first have to find associated num-
2493 nigel 75 ber. For example, for this pattern
2494 nigel 41
2495 nigel 93 (a+)b(?<xxx>\d+)...
2496 nigel 63
2497 nigel 91 the number of the subpattern called "xxx" is 2. If the name is known to
2498     be unique (PCRE_DUPNAMES was not set), you can find the number from the
2499     name by calling pcre_get_stringnumber(). The first argument is the com-
2500     piled pattern, and the second is the name. The yield of the function is
2501 ph10 545 the subpattern number, or PCRE_ERROR_NOSUBSTRING (-7) if there is no
2502 nigel 91 subpattern of that name.
2503 nigel 63
2504 nigel 75 Given the number, you can extract the substring directly, or use one of
2505     the functions described in the previous section. For convenience, there
2506     are also two functions that do the whole job.
2507    
2508 ph10 545 Most of the arguments of pcre_copy_named_substring() and
2509     pcre_get_named_substring() are the same as those for the similarly
2510     named functions that extract by number. As these are described in the
2511     previous section, they are not re-described here. There are just two
2512 nigel 75 differences:
2513 nigel 63
2514 ph10 545 First, instead of a substring number, a substring name is given. Sec-
2515 nigel 73 ond, there is an extra argument, given at the start, which is a pointer
2516 ph10 545 to the compiled pattern. This is needed in order to gain access to the
2517 nigel 73 name-to-number translation table.
2518 nigel 63
2519 ph10 545 These functions call pcre_get_stringnumber(), and if it succeeds, they
2520     then call pcre_copy_substring() or pcre_get_substring(), as appropri-
2521     ate. NOTE: If PCRE_DUPNAMES is set and there are duplicate names, the
2522 ph10 128 behaviour may not be what you want (see the next section).
2523 nigel 63
2524 ph10 461 Warning: If the pattern uses the (?| feature to set up multiple subpat-
2525 ph10 545 terns with the same number, as described in the section on duplicate
2526     subpattern numbers in the pcrepattern page, you cannot use names to
2527     distinguish the different subpatterns, because names are not included
2528     in the compiled code. The matching process uses only numbers. For this
2529     reason, the use of different names for subpatterns of the same number
2530 ph10 461 causes an error at compile time.
2531 nigel 77
2532 ph10 392
2533 nigel 91 DUPLICATE SUBPATTERN NAMES
2534    
2535     int pcre_get_stringtable_entries(const pcre *code,
2536     const char *name, char **first, char **last);
2537    
2538 ph10 545 When a pattern is compiled with the PCRE_DUPNAMES option, names for
2539     subpatterns are not required to be unique. (Duplicate names are always
2540     allowed for subpatterns with the same number, created by using the (?|
2541     feature. Indeed, if such subpatterns are named, they are required to
2542 ph10 461 use the same names.)
2543 ph10 208
2544 ph10 461 Normally, patterns with duplicate names are such that in any one match,
2545 ph10 545 only one of the named subpatterns participates. An example is shown in
2546 ph10 461 the pcrepattern documentation.
2547    
2548 ph10 545 When duplicates are present, pcre_copy_named_substring() and
2549     pcre_get_named_substring() return the first substring corresponding to
2550     the given name that is set. If none are set, PCRE_ERROR_NOSUBSTRING
2551     (-7) is returned; no data is returned. The pcre_get_stringnumber()
2552     function returns one of the numbers that are associated with the name,
2553 ph10 208 but it is not defined which it is.
2554 nigel 91
2555 ph10 545 If you want to get full details of all captured substrings for a given
2556     name, you must use the pcre_get_stringtable_entries() function. The
2557 nigel 91 first argument is the compiled pattern, and the second is the name. The
2558 ph10 545 third and fourth are pointers to variables which are updated by the
2559 nigel 91 function. After it has run, they point to the first and last entries in
2560 ph10 545 the name-to-number table for the given name. The function itself
2561     returns the length of each entry, or PCRE_ERROR_NOSUBSTRING (-7) if
2562     there are none. The format of the table is described above in the sec-
2563     tion entitled Information about a pattern. Given all the relevant
2564     entries for the name, you can extract each of their numbers, and hence
2565 nigel 93 the captured data, if any.
2566 nigel 91
2567    
2568 nigel 77 FINDING ALL POSSIBLE MATCHES
2569    
2570 ph10 545 The traditional matching function uses a similar algorithm to Perl,
2571 nigel 77 which stops when it finds the first match, starting at a given point in
2572 ph10 545 the subject. If you want to find all possible matches, or the longest
2573     possible match, consider using the alternative matching function (see
2574     below) instead. If you cannot use the alternative function, but still
2575     need to find all possible matches, you can kludge it up by making use
2576 nigel 77 of the callout facility, which is described in the pcrecallout documen-
2577     tation.
2578    
2579     What you have to do is to insert a callout right at the end of the pat-
2580 ph10 545 tern. When your callout function is called, extract and save the cur-
2581     rent matched substring. Then return 1, which forces pcre_exec() to
2582     backtrack and try other alternatives. Ultimately, when it runs out of
2583 nigel 77 matches, pcre_exec() will yield PCRE_ERROR_NOMATCH.
2584    
2585    
2586     MATCHING A PATTERN: THE ALTERNATIVE FUNCTION
2587    
2588     int pcre_dfa_exec(const pcre *code, const pcre_extra *extra,
2589     const char *subject, int length, int startoffset,
2590     int options, int *ovector, int ovecsize,
2591     int *workspace, int wscount);
2592    
2593 ph10 545 The function pcre_dfa_exec() is called to match a subject string
2594     against a compiled pattern, using a matching algorithm that scans the
2595     subject string just once, and does not backtrack. This has different
2596     characteristics to the normal algorithm, and is not compatible with
2597     Perl. Some of the features of PCRE patterns are not supported. Never-
2598     theless, there are times when this kind of matching can be useful. For
2599     a discussion of the two matching algorithms, and a list of features
2600     that pcre_dfa_exec() does not support, see the pcrematching documenta-
2601 ph10 453 tion.
2602 nigel 77
2603 ph10 545 The arguments for the pcre_dfa_exec() function are the same as for
2604 nigel 77 pcre_exec(), plus two extras. The ovector argument is used in a differ-
2605 ph10 545 ent way, and this is described below. The other common arguments are
2606     used in the same way as for pcre_exec(), so their description is not
2607 nigel 77 repeated here.
2608    
2609 ph10 545 The two additional arguments provide workspace for the function. The
2610     workspace vector should contain at least 20 elements. It is used for
2611 nigel 77 keeping track of multiple paths through the pattern tree. More
2612 ph10 545 workspace will be needed for patterns and subjects where there are a
2613 nigel 91 lot of potential matches.
2614 nigel 77
2615 nigel 87 Here is an example of a simple call to pcre_dfa_exec():
2616 nigel 77
2617     int rc;
2618     int ovector[10];
2619     int wspace[20];
2620 nigel 87 rc = pcre_dfa_exec(
2621 nigel 77 re, /* result of pcre_compile() */
2622     NULL, /* we didn't study the pattern */
2623     "some string", /* the subject string */
2624     11, /* the length of the subject string */
2625     0, /* start at offset 0 in the subject */
2626     0, /* default options */
2627     ovector, /* vector of integers for substring information */
2628     10, /* number of elements (NOT size in bytes) */
2629     wspace, /* working space vector */
2630     20); /* number of elements (NOT size in bytes) */
2631    
2632     Option bits for pcre_dfa_exec()
2633    
2634 ph10 545 The unused bits of the options argument for pcre_dfa_exec() must be
2635     zero. The only bits that may be set are PCRE_ANCHORED, PCRE_NEW-
2636 ph10 453 LINE_xxx, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY,
2637 ph10 545 PCRE_NOTEMPTY_ATSTART, PCRE_NO_UTF8_CHECK, PCRE_BSR_ANYCRLF,
2638     PCRE_BSR_UNICODE, PCRE_NO_START_OPTIMIZE, PCRE_PARTIAL_HARD, PCRE_PAR-
2639     TIAL_SOFT, PCRE_DFA_SHORTEST, and PCRE_DFA_RESTART. All but the last
2640 ph10 518 four of these are exactly the same as for pcre_exec(), so their
2641 ph10 453 description is not repeated here.
2642 nigel 77
2643 ph10 429 PCRE_PARTIAL_HARD
2644     PCRE_PARTIAL_SOFT
2645 nigel 77
2646 ph10 518 These have the same general effect as they do for pcre_exec(), but the
2647     details are slightly different. When PCRE_PARTIAL_HARD is set for
2648     pcre_dfa_exec(), it returns PCRE_ERROR_PARTIAL if the end of the sub-
2649     ject is reached and there is still at least one matching possibility
2650 ph10 429 that requires additional characters. This happens even if some complete
2651     matches have also been found. When PCRE_PARTIAL_SOFT is set, the return
2652     code PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if the end
2653 ph10 518 of the subject is reached, there have been no complete matches, but
2654     there is still at least one matching possibility. The portion of the
2655     string that was inspected when the longest partial match was found is
2656 ph10 453 set as the first matching string in both cases.
2657 nigel 77
2658     PCRE_DFA_SHORTEST
2659    
2660 ph10 518 Setting the PCRE_DFA_SHORTEST option causes the matching algorithm to
2661 nigel 93 stop as soon as it has found one match. Because of the way the alterna-
2662 ph10 518 tive algorithm works, this is necessarily the shortest possible match
2663 nigel 93 at the first possible matching point in the subject string.
2664 nigel 77
2665     PCRE_DFA_RESTART
2666    
2667 ph10 429 When pcre_dfa_exec() returns a partial match, it is possible to call it
2668 ph10 518 again, with additional subject characters, and have it continue with
2669     the same match. The PCRE_DFA_RESTART option requests this action; when
2670     it is set, the workspace and wscount options must reference the same
2671     vector as before because data about the match so far is left in them
2672 ph10 429 after a partial match. There is more discussion of this facility in the
2673     pcrepartial documentation.
2674 nigel 77
2675     Successful returns from pcre_dfa_exec()
2676    
2677 ph10 518 When pcre_dfa_exec() succeeds, it may have matched more than one sub-
2678 nigel 77 string in the subject. Note, however, that all the matches from one run
2679 ph10 518 of the function start at the same point in the subject. The shorter
2680     matches are all initial substrings of the longer matches. For example,
2681 nigel 77 if the pattern
2682    
2683     <.*>
2684    
2685     is matched against the string
2686    
2687     This is <something> <something else> <something further> no more
2688    
2689     the three matched strings are
2690    
2691     <something>
2692     <something> <something else>
2693     <something> <something else> <something further>
2694    
2695 ph10 518 On success, the yield of the function is a number greater than zero,
2696     which is the number of matched substrings. The substrings themselves
2697     are returned in ovector. Each string uses two elements; the first is
2698     the offset to the start, and the second is the offset to the end. In
2699     fact, all the strings have the same start offset. (Space could have
2700     been saved by giving this only once, but it was decided to retain some
2701     compatibility with the way pcre_exec() returns data, even though the
2702 nigel 93 meaning of the strings is different.)
2703 nigel 77
2704     The strings are returned in reverse order of length; that is, the long-
2705 ph10 518 est matching string is given first. If there were too many matches to
2706     fit into ovector, the yield of the function is zero, and the vector is
2707 nigel 77 filled with the longest matches.
2708    
2709     Error returns from pcre_dfa_exec()
2710    
2711 ph10 518 The pcre_dfa_exec() function returns a negative number when it fails.
2712     Many of the errors are the same as for pcre_exec(), and these are
2713     described above. There are in addition the following errors that are
2714 nigel 77 specific to pcre_dfa_exec():
2715    
2716     PCRE_ERROR_DFA_UITEM (-16)
2717    
2718 ph10 518 This return is given if pcre_dfa_exec() encounters an item in the pat-
2719     tern that it does not support, for instance, the use of \C or a back
2720 nigel 77 reference.
2721    
2722     PCRE_ERROR_DFA_UCOND (-17)
2723    
2724 ph10 518 This return is given if pcre_dfa_exec() encounters a condition item
2725     that uses a back reference for the condition, or a test for recursion
2726 nigel 93 in a specific group. These are not supported.
2727 nigel 77
2728     PCRE_ERROR_DFA_UMLIMIT (-18)
2729    
2730 ph10 518 This return is given if pcre_dfa_exec() is called with an extra block
2731 nigel 77 that contains a setting of the match_limit field. This is not supported
2732     (it is meaningless).
2733    
2734     PCRE_ERROR_DFA_WSSIZE (-19)
2735    
2736 ph10 518 This return is given if pcre_dfa_exec() runs out of space in the
2737 nigel 77 workspace vector.
2738    
2739     PCRE_ERROR_DFA_RECURSE (-20)
2740    
2741 ph10 518 When a recursive subpattern is processed, the matching function calls
2742     itself recursively, using private vectors for ovector and workspace.
2743     This error is given if the output vector is not large enough. This
2744 nigel 77 should be extremely rare, as a vector of size 1000 is used.
2745    
2746 nigel 93
2747     SEE ALSO
2748    
2749 ph10 518 pcrebuild(3), pcrecallout(3), pcrecpp(3)(3), pcrematching(3), pcrepar-
2750 ph10 392 tial(3), pcreposix(3), pcreprecompile(3), pcresample(3), pcrestack(3).
2751 nigel 93
2752 nigel 63
2753 ph10 99 AUTHOR
2754 nigel 63
2755 ph10 99 Philip Hazel
2756     University Computing Service
2757     Cambridge CB2 3QH, England.
2758    
2759    
2760     REVISION
2761    
2762 ph10 545 Last updated: 15 June 2010
2763 ph10 512 Copyright (c) 1997-2010 University of Cambridge.
2764 ph10 99 ------------------------------------------------------------------------------
2765 ph10 535
2766    
2767 nigel 79 PCRECALLOUT(3) PCRECALLOUT(3)
2768 nigel 63
2769 nigel 79
2770 nigel 73 NAME
2771     PCRE - Perl-compatible regular expressions
2772    
2773 nigel 77
2774 nigel 63 PCRE CALLOUTS
2775    
2776 nigel 73 int (*pcre_callout)(pcre_callout_block *);
2777 nigel 63
2778 nigel 73 PCRE provides a feature called "callout", which is a means of temporar-
2779     ily passing control to the caller of PCRE in the middle of pattern
2780     matching. The caller of PCRE provides an external function by putting
2781     its entry point in the global variable pcre_callout. By default, this
2782     variable contains NULL, which disables all calling out.
2783 nigel 63
2784 nigel 73 Within a regular expression, (?C) indicates the points at which the
2785     external function is to be called. Different callout points can be
2786     identified by putting a number less than 256 after the letter C. The
2787     default value is zero. For example, this pattern has two callout
2788     points:
2789 nigel 63
2790 ph10 155 (?C1)abc(?C2)def
2791 nigel 63
2792 ph10 461 If the PCRE_AUTO_CALLOUT option bit is set when pcre_compile() or
2793     pcre_compile2() is called, PCRE automatically inserts callouts, all
2794     with number 255, before each item in the pattern. For example, if
2795     PCRE_AUTO_CALLOUT is used with the pattern
2796 nigel 63
2797 nigel 75 A(\d{2}|--)
2798    
2799     it is processed as if it were
2800    
2801     (?C255)A(?C255)((?C255)\d{2}(?C255)|(?C255)-(?C255)-(?C255))(?C255)
2802    
2803     Notice that there is a callout before and after each parenthesis and
2804     alternation bar. Automatic callouts can be used for tracking the
2805     progress of pattern matching. The pcretest command has an option that
2806     sets automatic callouts; when it is used, the output indicates how the
2807     pattern is matched. This is useful information when you are trying to
2808     optimize the performance of a particular pattern.
2809    
2810    
2811     MISSING CALLOUTS
2812    
2813     You should be aware that, because of optimizations in the way PCRE
2814 ph10 392 matches patterns by default, callouts sometimes do not happen. For
2815     example, if the pattern is
2816 nigel 75
2817     ab(?C4)cd
2818    
2819     PCRE knows that any matching string must contain the letter "d". If the
2820     subject string is "abyz", the lack of "d" means that matching doesn't
2821     ever start, and the callout is never reached. However, with "abyd",
2822     though the result is still no match, the callout is obeyed.
2823    
2824 ph10 461 If the pattern is studied, PCRE knows the minimum length of a matching
2825     string, and will immediately give a "no match" return without actually
2826     running a match if the subject is not long enough, or, for unanchored
2827     patterns, if it has been scanned far enough.
2828    
2829     You can disable these optimizations by passing the PCRE_NO_START_OPTI-
2830     MIZE option to pcre_exec() or pcre_dfa_exec(). This slows down the
2831     matching process, but does ensure that callouts such as the example
2832 ph10 392 above are obeyed.
2833 nigel 75
2834 ph10 392
2835 nigel 75 THE CALLOUT INTERFACE
2836    
2837 ph10 461 During matching, when PCRE reaches a callout point, the external func-
2838     tion defined by pcre_callout is called (if it is set). This applies to
2839     both the pcre_exec() and the pcre_dfa_exec() matching functions. The
2840     only argument to the callout function is a pointer to a pcre_callout
2841 nigel 77 block. This structure contains the following fields:
2842 nigel 75
2843 nigel 73 int version;
2844     int callout_number;
2845     int *offset_vector;
2846     const char *subject;
2847     int subject_length;
2848     int start_match;
2849     int current_position;
2850     int capture_top;
2851     int capture_last;
2852     void *callout_data;
2853 nigel 75 int pattern_position;
2854     int next_item_length;
2855 nigel 63
2856 ph10 461 The version field is an integer containing the version number of the
2857     block format. The initial version was 0; the current version is 1. The
2858     version number will change again in future if additional fields are
2859 nigel 75 added, but the intention is never to remove any of the existing fields.
2860 nigel 63
2861 ph10 461 The callout_number field contains the number of the callout, as com-
2862     piled into the pattern (that is, the number after ?C for manual call-
2863 nigel 75 outs, and 255 for automatically generated callouts).
2864 nigel 63
2865 ph10 461 The offset_vector field is a pointer to the vector of offsets that was
2866     passed by the caller to pcre_exec() or pcre_dfa_exec(). When
2867     pcre_exec() is used, the contents can be inspected in order to extract
2868     substrings that have been matched so far, in the same way as for
2869     extracting substrings after a match has completed. For pcre_dfa_exec()
2870 nigel 77 this field is not useful.
2871 nigel 63
2872 nigel 75 The subject and subject_length fields contain copies of the values that
2873 nigel 73 were passed to pcre_exec().
2874 nigel 63
2875 ph10 461 The start_match field normally contains the offset within the subject
2876     at which the current match attempt started. However, if the escape
2877     sequence \K has been encountered, this value is changed to reflect the
2878     modified starting point. If the pattern is not anchored, the callout
2879 ph10 172 function may be called several times from the same point in the pattern
2880     for different starting points in the subject.
2881 nigel 63
2882 ph10 461 The current_position field contains the offset within the subject of
2883 nigel 73 the current match pointer.
2884 nigel 63
2885 ph10 461 When the pcre_exec() function is used, the capture_top field contains
2886     one more than the number of the highest numbered captured substring so
2887     far. If no substrings have been captured, the value of capture_top is
2888     one. This is always the case when pcre_dfa_exec() is used, because it
2889 nigel 77 does not support captured substrings.
2890 nigel 63
2891 ph10 461 The capture_last field contains the number of the most recently cap-
2892     tured substring. If no substrings have been captured, its value is -1.
2893 nigel 77 This is always the case when pcre_dfa_exec() is used.
2894 nigel 63
2895 ph10 461 The callout_data field contains a value that is passed to pcre_exec()
2896     or pcre_dfa_exec() specifically so that it can be passed back in call-
2897     outs. It is passed in the pcre_callout field of the pcre_extra data
2898     structure. If no such data was passed, the value of callout_data in a
2899     pcre_callout block is NULL. There is a description of the pcre_extra
2900 nigel 73 structure in the pcreapi documentation.
2901 nigel 63
2902 ph10 461 The pattern_position field is present from version 1 of the pcre_call-
2903 nigel 75 out structure. It contains the offset to the next item to be matched in
2904     the pattern string.
2905 nigel 63
2906 ph10 461 The next_item_length field is present from version 1 of the pcre_call-
2907 nigel 75 out structure. It contains the length of the next item to be matched in
2908 ph10 461 the pattern string. When the callout immediately precedes an alterna-
2909     tion bar, a closing parenthesis, or the end of the pattern, the length
2910     is zero. When the callout precedes an opening parenthesis, the length
2911 nigel 75 is that of the entire subpattern.
2912 nigel 73
2913 ph10 461 The pattern_position and next_item_length fields are intended to help
2914     in distinguishing between different automatic callouts, which all have
2915 nigel 75 the same callout number. However, they are set for all callouts.
2916    
2917    
2918 nigel 63 RETURN VALUES
2919    
2920 ph10 461 The external callout function returns an integer to PCRE. If the value
2921     is zero, matching proceeds as normal. If the value is greater than
2922     zero, matching fails at the current point, but the testing of other
2923 nigel 77 matching possibilities goes ahead, just as if a lookahead assertion had
2924 ph10 461 failed. If the value is less than zero, the match is abandoned, and
2925     pcre_exec() or pcre_dfa_exec() returns the negative value.
2926 nigel 63
2927 ph10 461 Negative values should normally be chosen from the set of
2928 nigel 73 PCRE_ERROR_xxx values. In particular, PCRE_ERROR_NOMATCH forces a stan-
2929 ph10 461 dard "no match" failure. The error number PCRE_ERROR_CALLOUT is
2930     reserved for use by callout functions; it will never be used by PCRE
2931 nigel 73 itself.
2932 nigel 63
2933    
2934 ph10 99 AUTHOR
2935 nigel 63
2936 ph10 99 Philip Hazel
2937     University Computing Service
2938     Cambridge CB2 3QH, England.
2939    
2940    
2941     REVISION
2942    
2943 ph10 461 Last updated: 29 September 2009
2944 ph10 392 Copyright (c) 1997-2009 University of Cambridge.
2945 ph10 99 ------------------------------------------------------------------------------
2946 ph10 535
2947    
2948 nigel 79 PCRECOMPAT(3) PCRECOMPAT(3)
2949 nigel 63
2950 nigel 79
2951 nigel 73 NAME
2952     PCRE - Perl-compatible regular expressions
2953    
2954 nigel 77
2955 nigel 75 DIFFERENCES BETWEEN PCRE AND PERL
2956 nigel 41
2957 nigel 73 This document describes the differences in the ways that PCRE and Perl
2958 ph10 461 handle regular expressions. The differences described here are with
2959 ph10 535 respect to Perl 5.10/5.11.
2960 nigel 41
2961 ph10 461 1. PCRE has only a subset of Perl's UTF-8 and Unicode support. Details
2962     of what it does have are given in the section on UTF-8 support in the
2963 nigel 87 main pcre page.
2964 nigel 41
2965 nigel 73 2. PCRE does not allow repeat quantifiers on lookahead assertions. Perl
2966 ph10 461 permits them, but they do not mean what you might think. For example,
2967 nigel 73 (?!a){3} does not assert that the next three characters are not "a". It
2968     just asserts that the next character is not "a" three times.
2969 nigel 41
2970 ph10 461 3. Capturing subpatterns that occur inside negative lookahead asser-
2971     tions are counted, but their entries in the offsets vector are never
2972     set. Perl sets its numerical variables from any such patterns that are
2973 nigel 73 matched before the assertion fails to match something (thereby succeed-
2974 ph10 461 ing), but only if the negative lookahead assertion contains just one
2975 nigel 73 branch.
2976 nigel 41
2977 ph10 461 4. Though binary zero characters are supported in the subject string,
2978 nigel 73 they are not allowed in a pattern string because it is passed as a nor-
2979 nigel 75 mal C string, terminated by zero. The escape sequence \0 can be used in
2980     the pattern to represent a binary zero.
2981 nigel 41
2982 ph10 461 5. The following Perl escape sequences are not supported: \l, \u, \L,
2983 nigel 75 \U, and \N. In fact these are implemented by Perl's general string-han-
2984 ph10 461 dling and are not part of its pattern matching engine. If any of these
2985 nigel 75 are encountered by PCRE, an error is generated.
2986 nigel 41
2987 ph10 461 6. The Perl escape sequences \p, \P, and \X are supported only if PCRE
2988     is built with Unicode character property support. The properties that
2989     can be tested with \p and \P are limited to the general category prop-
2990     erties such as Lu and Nd, script names such as Greek or Han, and the
2991     derived properties Any and L&. PCRE does support the Cs (surrogate)
2992     property, which Perl does not; the Perl documentation says "Because
2993 ph10 453 Perl hides the need for the user to understand the internal representa-
2994 ph10 461 tion of Unicode characters, there is no need to implement the somewhat
2995 ph10 453 messy concept of surrogates."
2996 nigel 75
2997     7. PCRE does support the \Q...\E escape for quoting substrings. Charac-
2998 ph10 461 ters in between are treated as literals. This is slightly different
2999     from Perl in that $ and @ are also handled as literals inside the
3000     quotes. In Perl, they cause variable interpolation (but of course PCRE
3001 nigel 73 does not have variables). Note the following examples:
3002 nigel 49
3003 nigel 73 Pattern PCRE matches Perl matches
3004 nigel 41
3005 nigel 73 \Qabc$xyz\E abc$xyz abc followed by the
3006     contents of $xyz
3007     \Qabc\$xyz\E abc\$xyz abc\$xyz
3008     \Qabc\E\$\Qxyz\E abc$xyz abc$xyz
3009 nigel 41
3010 ph10 461 The \Q...\E sequence is recognized both inside and outside character
3011 nigel 73 classes.
3012 nigel 41
3013 nigel 93 8. Fairly obviously, PCRE does not support the (?{code}) and (??{code})
3014 ph10 461 constructions. However, there is support for recursive patterns. This
3015     is not available in Perl 5.8, but it is in Perl 5.10. Also, the PCRE
3016     "callout" feature allows an external function to be called during pat-
3017 nigel 75 tern matching. See the pcrecallout documentation for details.
3018 nigel 63
3019 ph10 461 9. Subpatterns that are called recursively or as "subroutines" are
3020     always treated as atomic groups in PCRE. This is like Python, but
3021     unlike Perl. There is a discussion of an example that explains this in
3022     more detail in the section on recursion differences from Perl in the
3023     pcrepattern page.
3024 nigel 93
3025 ph10 461 10. There are some differences that are concerned with the settings of
3026     captured strings when part of a pattern is repeated. For example,
3027     matching "aba" against the pattern /^(a(b)?)+$/ in Perl leaves $2
3028 nigel 73 unset, but in PCRE it is set to "b".
3029 nigel 41
3030 ph10 535 11. PCRE's handling of duplicate subpattern numbers and duplicate sub-
3031 ph10 461 pattern names is not as general as Perl's. This is a consequence of the
3032     fact the PCRE works internally just with numbers, using an external ta-
3033     ble to translate between numbers and names. In particular, a pattern
3034     such as (?|(?<a>A)|(?<b)B), where the two capturing parentheses have
3035     the same number but different names, is not supported, and causes an
3036     error at compile time. If it were allowed, it would not be possible to
3037     distinguish which parentheses matched, because both names map to cap-
3038     turing subpattern number 1. To avoid this confusing situation, an error
3039     is given at compile time.
3040 nigel 41
3041 ph10 535 12. PCRE provides some extensions to the Perl regular expression facil-
3042 ph10 461 ities. Perl 5.10 includes new features that are not in earlier ver-
3043     sions of Perl, some of which (such as named parentheses) have been in
3044     PCRE for some time. This list is with respect to Perl 5.10:
3045 nigel 41
3046 ph10 461 (a) Although lookbehind assertions in PCRE must match fixed length
3047     strings, each alternative branch of a lookbehind assertion can match a
3048     different length of string. Perl requires them all to have the same
3049     length.
3050    
3051     (b) If PCRE_DOLLAR_ENDONLY is set and PCRE_MULTILINE is not set, the $
3052 nigel 73 meta-character matches only at the very end of the string.
3053 nigel 41
3054 nigel 73 (c) If PCRE_EXTRA is set, a backslash followed by a letter with no spe-
3055 ph10 182 cial meaning is faulted. Otherwise, like Perl, the backslash is quietly
3056     ignored. (Perl can be made to issue a warning.)
3057 nigel 41
3058 ph10 461 (d) If PCRE_UNGREEDY is set, the greediness of the repetition quanti-
3059 nigel 73 fiers is inverted, that is, by default they are not greedy, but if fol-
3060     lowed by a question mark they are.
3061 nigel 41
3062 nigel 75 (e) PCRE_ANCHORED can be used at matching time to force a pattern to be
3063     tried only at the first matching position in the subject string.
3064 nigel 41
3065 ph10 453 (f) The PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NOTEMPTY_ATSTART,
3066 ph10 461 and PCRE_NO_AUTO_CAPTURE options for pcre_exec() have no Perl equiva-
3067 ph10 453 lents.
3068 nigel 41
3069 ph10 461 (g) The \R escape sequence can be restricted to match only CR, LF, or
3070 ph10 231 CRLF by the PCRE_BSR_ANYCRLF option.
3071 nigel 41
3072 ph10 231 (h) The callout facility is PCRE-specific.
3073 nigel 43
3074 ph10 231 (i) The partial matching facility is PCRE-specific.
3075    
3076     (j) Patterns compiled by PCRE can be saved and re-used at a later time,
3077 nigel 75 even on different hosts that have the other endianness.
3078    
3079 ph10 461 (k) The alternative matching function (pcre_dfa_exec()) matches in a
3080 nigel 77 different way and is not Perl-compatible.
3081    
3082 ph10 461 (l) PCRE recognizes some special sequences such as (*CR) at the start
3083 ph10 231 of a pattern that set overall options that cannot be changed within the
3084     pattern.
3085 nigel 63
3086 ph10 231
3087 ph10 99 AUTHOR
3088 nigel 63
3089 ph10 99 Philip Hazel
3090     University Computing Service
3091     Cambridge CB2 3QH, England.
3092    
3093    
3094     REVISION
3095    
3096 ph10 535 Last updated: 12 May 2010
3097     Copyright (c) 1997-2010 University of Cambridge.
3098 ph10 99 ------------------------------------------------------------------------------
3099 ph10 535
3100    
3101 nigel 79 PCREPATTERN(3) PCREPATTERN(3)
3102 nigel 63
3103 nigel 79
3104 nigel 73 NAME
3105     PCRE - Perl-compatible regular expressions
3106    
3107 nigel 77
3108 nigel 63 PCRE REGULAR EXPRESSION DETAILS
3109    
3110 ph10 208 The syntax and semantics of the regular expressions that are supported
3111     by PCRE are described in detail below. There is a quick-reference syn-
3112 ph10 345 tax summary in the pcresyntax page. PCRE tries to match Perl syntax and
3113     semantics as closely as it can. PCRE also supports some alternative
3114     regular expression syntax (which does not conflict with the Perl syn-
3115     tax) in order to provide some compatibility with regular expressions in
3116     Python, .NET, and Oniguruma.
3117 nigel 49
3118 ph10 345 Perl's regular expressions are described in its own documentation, and
3119     regular expressions in general are covered in a number of books, some
3120     of which have copious examples. Jeffrey Friedl's "Mastering Regular
3121     Expressions", published by O'Reilly, covers regular expressions in
3122     great detail. This description of PCRE's regular expressions is
3123     intended as reference material.
3124    
3125 nigel 75 The original operation of PCRE was on strings of one-byte characters.
3126     However, there is now also support for UTF-8 character strings. To use
3127 ph10 461 this, PCRE must be built to include UTF-8 support, and you must call
3128     pcre_compile() or pcre_compile2() with the PCRE_UTF8 option. There is
3129     also a special sequence that can be given at the start of a pattern:
3130 nigel 41
3131 ph10 416 (*UTF8)
3132    
3133     Starting a pattern with this sequence is equivalent to setting the
3134     PCRE_UTF8 option. This feature is not Perl-compatible. How setting
3135     UTF-8 mode affects pattern matching is mentioned in several places
3136     below. There is also a summary of UTF-8 features in the section on
3137     UTF-8 support in the main pcre page.
3138    
3139 ph10 535 Another special sequence that may appear at the start of a pattern or
3140     in combination with (*UTF8) is:
3141    
3142     (*UCP)
3143    
3144     This has the same effect as setting the PCRE_UCP option: it causes
3145     sequences such as \d and \w to use Unicode properties to determine
3146     character types, instead of recognizing only characters with codes less
3147     than 128 via a lookup table.
3148    
3149 nigel 77 The remainder of this document discusses the patterns that are sup-
3150     ported by PCRE when its main matching function, pcre_exec(), is used.
3151     From release 6.0, PCRE offers a second matching function,
3152     pcre_dfa_exec(), which matches using a different algorithm that is not
3153 ph10 172 Perl-compatible. Some of the features discussed below are not available
3154     when pcre_dfa_exec() is used. The advantages and disadvantages of the
3155     alternative function, and how it differs from the normal function, are
3156     discussed in the pcrematching page.
3157 nigel 77
3158 nigel 93
3159 ph10 227 NEWLINE CONVENTIONS
3160    
3161     PCRE supports five different conventions for indicating line breaks in
3162     strings: a single CR (carriage return) character, a single LF (line-
3163     feed) character, the two-character sequence CRLF, any of the three pre-
3164     ceding, or any Unicode newline sequence. The pcreapi page has further
3165     discussion about newlines, and shows how to set the newline convention
3166     in the options arguments for the compiling and matching functions.
3167    
3168     It is also possible to specify a newline convention by starting a pat-
3169     tern string with one of the following five sequences:
3170    
3171     (*CR) carriage return
3172     (*LF) linefeed
3173     (*CRLF) carriage return, followed by linefeed
3174     (*ANYCRLF) any of the three above
3175     (*ANY) all Unicode newline sequences
3176    
3177 ph10 461 These override the default and the options given to pcre_compile() or
3178     pcre_compile2(). For example, on a Unix system where LF is the default
3179     newline sequence, the pattern
3180 ph10 227
3181     (*CR)a.b
3182    
3183     changes the convention to CR. That pattern matches "a\nb" because LF is
3184     no longer a newline. Note that these special settings, which are not
3185     Perl-compatible, are recognized only at the very start of a pattern,
3186 ph10 231 and that they must be in upper case. If more than one of them is
3187     present, the last one is used.
3188 ph10 227
3189 ph10 518 The newline convention affects the interpretation of the dot metachar-
3190     acter when PCRE_DOTALL is not set, and also the behaviour of \N. How-
3191     ever, it does not affect what the \R escape sequence matches. By
3192     default, this is any Unicode newline sequence, for Perl compatibility.
3193     However, this can be changed; see the description of \R in the section
3194     entitled "Newline sequences" below. A change of \R setting can be com-
3195     bined with a change of newline convention.
3196 ph10 227
3197 ph10 231
3198 nigel 93 CHARACTERS AND METACHARACTERS
3199    
3200 ph10 247 A regular expression is a pattern that is matched against a subject
3201     string from left to right. Most characters stand for themselves in a
3202     pattern, and match the corresponding characters in the subject. As a
3203 nigel 73 trivial example, the pattern
3204 nigel 41
3205 nigel 73 The quick brown fox
3206 nigel 41
3207 nigel 77 matches a portion of a subject string that is identical to itself. When
3208 ph10 247 caseless matching is specified (the PCRE_CASELESS option), letters are
3209     matched independently of case. In UTF-8 mode, PCRE always understands
3210     the concept of case for characters whose values are less than 128, so
3211     caseless matching is always possible. For characters with higher val-
3212     ues, the concept of case is supported if PCRE is compiled with Unicode
3213     property support, but not otherwise. If you want to use caseless
3214     matching for characters 128 and above, you must ensure that PCRE is
3215 nigel 77 compiled with Unicode property support as well as with UTF-8 support.
3216 nigel 41
3217 ph10 247 The power of regular expressions comes from the ability to include
3218     alternatives and repetitions in the pattern. These are encoded in the
3219 nigel 77 pattern by the use of metacharacters, which do not stand for themselves
3220     but instead are interpreted in some special way.
3221    
3222 ph10 247 There are two different sets of metacharacters: those that are recog-
3223     nized anywhere in the pattern except within square brackets, and those
3224     that are recognized within square brackets. Outside square brackets,
3225 nigel 93 the metacharacters are as follows:
3226 nigel 41
3227 nigel 73 \ general escape character with several uses
3228     ^ assert start of string (or line, in multiline mode)
3229     $ assert end of string (or line, in multiline mode)
3230     . match any character except newline (by default)
3231     [ start character class definition
3232     | start of alternative branch
3233     ( start subpattern
3234     ) end subpattern
3235     ? extends the meaning of (
3236     also 0 or 1 quantifier
3237     also quantifier minimizer
3238     * 0 or more quantifier
3239     + 1 or more quantifier
3240     also "possessive quantifier"
3241     { start min/max quantifier
3242 nigel 41
3243 ph10 247 Part of a pattern that is in square brackets is called a "character
3244 nigel 75 class". In a character class the only metacharacters are:
3245 nigel 41
3246 nigel 73 \ general escape character
3247     ^ negate the class, but only if the first character
3248     - indicates character range
3249     [ POSIX character class (only if followed by POSIX
3250     syntax)
3251     ] terminates the character class
3252 nigel 41
3253 ph10 392 The following sections describe the use of each of the metacharacters.
3254 nigel 41
3255    
3256 nigel 63 BACKSLASH
3257 nigel 41
3258 nigel 73 The backslash character has several uses. Firstly, if it is followed by
3259 ph10 392 a non-alphanumeric character, it takes away any special meaning that
3260     character may have. This use of backslash as an escape character
3261 nigel 73 applies both inside and outside character classes.
3262 nigel 41
3263 ph10 392 For example, if you want to match a * character, you write \* in the
3264     pattern. This escaping action applies whether or not the following
3265     character would otherwise be interpreted as a metacharacter, so it is
3266     always safe to precede a non-alphanumeric with backslash to specify
3267     that it stands for itself. In particular, if you want to match a back-
3268 nigel 75 slash, you write \\.
3269 nigel 41
3270 ph10 392 If a pattern is compiled with the PCRE_EXTENDED option, whitespace in
3271     the pattern (other than in a character class) and characters between a
3272 nigel 91 # outside a character class and the next newline are ignored. An escap-
3273 ph10 392 ing backslash can be used to include a whitespace or # character as
3274 nigel 91 part of the pattern.
3275 nigel 41
3276 ph10 392 If you want to remove the special meaning from a sequence of charac-
3277     ters, you can do so by putting them between \Q and \E. This is differ-
3278     ent from Perl in that $ and @ are handled as literals in \Q...\E
3279     sequences in PCRE, whereas in Perl, $ and @ cause variable interpola-
3280 nigel 73 tion. Note the following examples:
3281 nigel 63
3282 nigel 73 Pattern PCRE matches Perl matches
3283 nigel 63
3284 nigel 73 \Qabc$xyz\E abc$xyz abc followed by the
3285     contents of $xyz
3286     \Qabc\$xyz\E abc\$xyz abc\$xyz
3287     \Qabc\E\$\Qxyz\E abc$xyz abc$xyz
3288 nigel 63
3289 ph10 392 The \Q...\E sequence is recognized both inside and outside character
3290 nigel 73 classes.
3291 nigel 63
3292 nigel 75 Non-printing characters
3293    
3294 nigel 73 A second use of backslash provides a way of encoding non-printing char-
3295 ph10 392 acters in patterns in a visible manner. There is no restriction on the
3296     appearance of non-printing characters, apart from the binary zero that
3297     terminates a pattern, but when a pattern is being prepared by text
3298 ph10 461 editing, it is often easier to use one of the following escape
3299 nigel 73 sequences than the binary character it represents:
3300 nigel 63
3301 nigel 73 \a alarm, that is, the BEL character (hex 07)
3302     \cx "control-x", where x is any character
3303     \e escape (hex 1B)
3304     \f formfeed (hex 0C)
3305 ph10 227 \n linefeed (hex 0A)
3306 nigel 73 \r carriage return (hex 0D)
3307     \t tab (hex 09)
3308 ph10 488 \ddd character with octal code ddd, or back reference
3309 nigel 73 \xhh character with hex code hh
3310 nigel 87 \x{hhh..} character with hex code hhh..
3311 nigel 41
3312 ph10 392 The precise effect of \cx is as follows: if x is a lower case letter,
3313     it is converted to upper case. Then bit 6 of the character (hex 40) is
3314     inverted. Thus \cz becomes hex 1A, but \c{ becomes hex 3B, while \c;
3315 nigel 73 becomes hex 7B.
3316 nigel 41
3317 ph10 392 After \x, from zero to two hexadecimal digits are read (letters can be
3318     in upper or lower case). Any number of hexadecimal digits may appear
3319     between \x{ and }, but the value of the character code must be less
3320 ph10 211 than 256 in non-UTF-8 mode, and less than 2**31 in UTF-8 mode. That is,
3321 ph10 392 the maximum value in hexadecimal is 7FFFFFFF. Note that this is bigger
3322 ph10 211 than the largest Unicode code point, which is 10FFFF.
3323 nigel 41
3324 ph10 392 If characters other than hexadecimal digits appear between \x{ and },
3325 ph10 211 or if there is no terminating }, this form of escape is not recognized.
3326 ph10 392 Instead, the initial \x will be interpreted as a basic hexadecimal
3327     escape, with no following digits, giving a character whose value is
3328 ph10 211 zero.
3329    
3330 nigel 73 Characters whose value is less than 256 can be defined by either of the
3331 ph10 392 two syntaxes for \x. There is no difference in the way they are han-
3332 nigel 87 dled. For example, \xdc is exactly the same as \x{dc}.
3333 nigel 41
3334 ph10 392 After \0 up to two further octal digits are read. If there are fewer
3335     than two digits, just those that are present are used. Thus the
3336 nigel 91 sequence \0\x\07 specifies two binary zeros followed by a BEL character
3337 ph10 392 (code value 7). Make sure you supply two digits after the initial zero
3338 nigel 91 if the pattern character that follows is itself an octal digit.
3339 nigel 63
3340 nigel 73 The handlin