/[pcre]/code/trunk/doc/pcre.txt
ViewVC logotype

Contents of /code/trunk/doc/pcre.txt

Parent Directory Parent Directory | Revision Log Revision Log


Revision 313 - (hide annotations) (download)
Wed Jan 23 18:05:06 2008 UTC (6 years, 10 months ago) by ph10
File MIME type: text/plain
File size: 303842 byte(s)
Tidies for 7.6-RC2.

1 nigel 75 -----------------------------------------------------------------------------
2 nigel 63 This file contains a concatenation of the PCRE man pages, converted to plain
3     text format for ease of searching with a text editor, or for use on systems
4     that do not have a man page processor. The small individual files that give
5     synopses of each function in the library have not been included. There are
6     separate text files for the pcregrep and pcretest commands.
7     -----------------------------------------------------------------------------
8    
9 nigel 41
10 nigel 79 PCRE(3) PCRE(3)
11 nigel 41
12 nigel 79
13 nigel 73 NAME
14     PCRE - Perl-compatible regular expressions
15    
16 nigel 77
17 nigel 75 INTRODUCTION
18 nigel 41
19 nigel 73 The PCRE library is a set of functions that implement regular expres-
20     sion pattern matching using the same syntax and semantics as Perl, with
21 nigel 93 just a few differences. (Certain features that appeared in Python and
22     PCRE before they appeared in Perl are also available using the Python
23     syntax.)
24 nigel 63
25 nigel 93 The current implementation of PCRE (release 7.x) corresponds approxi-
26     mately with Perl 5.10, including support for UTF-8 encoded strings and
27     Unicode general category properties. However, UTF-8 and Unicode support
28     has to be explicitly enabled; it is not the default. The Unicode tables
29     correspond to Unicode release 5.0.0.
30 nigel 77
31 nigel 93 In addition to the Perl-compatible matching function, PCRE contains an
32     alternative matching function that matches the same compiled patterns
33     in a different way. In certain circumstances, the alternative function
34     has some advantages. For a discussion of the two matching algorithms,
35     see the pcrematching page.
36    
37     PCRE is written in C and released as a C library. A number of people
38     have written wrappers and interfaces of various kinds. In particular,
39     Google Inc. have provided a comprehensive C++ wrapper. This is now
40 nigel 77 included as part of the PCRE distribution. The pcrecpp page has details
41 nigel 93 of this interface. Other people's contributions can be found in the
42 nigel 77 Contrib directory at the primary FTP site, which is:
43 nigel 63
44 nigel 73 ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre
45 nigel 63
46 nigel 93 Details of exactly which Perl regular expression features are and are
47 nigel 73 not supported by PCRE are given in separate documents. See the pcrepat-
48 ph10 208 tern and pcrecompat pages. There is a syntax summary in the pcresyntax
49     page.
50 nigel 63
51 ph10 208 Some features of PCRE can be included, excluded, or changed when the
52     library is built. The pcre_config() function makes it possible for a
53     client to discover which features are available. The features them-
54     selves are described in the pcrebuild page. Documentation about build-
55     ing PCRE for various operating systems can be found in the README file
56 nigel 75 in the source distribution.
57 nigel 63
58 ph10 208 The library contains a number of undocumented internal functions and
59     data tables that are used by more than one of the exported external
60     functions, but which are not intended for use by external callers.
61     Their names all begin with "_pcre_", which hopefully will not provoke
62 nigel 83 any name clashes. In some environments, it is possible to control which
63 ph10 208 external symbols are exported when a shared library is built, and in
64 nigel 83 these cases the undocumented symbols are not exported.
65 nigel 63
66 nigel 77
67 nigel 63 USER DOCUMENTATION
68    
69 ph10 208 The user documentation for PCRE comprises a number of different sec-
70     tions. In the "man" format, each of these is a separate "man page". In
71     the HTML format, each is a separate page, linked from the index page.
72     In the plain text format, all the sections are concatenated, for ease
73 nigel 75 of searching. The sections are as follows:
74 nigel 63
75 nigel 73 pcre this document
76 ph10 153 pcre-config show PCRE installation configuration information
77 nigel 77 pcreapi details of PCRE's native C API
78 nigel 73 pcrebuild options for building PCRE
79     pcrecallout details of the callout feature
80     pcrecompat discussion of Perl compatibility
81 nigel 77 pcrecpp details of the C++ wrapper
82 nigel 73 pcregrep description of the pcregrep command
83 nigel 77 pcrematching discussion of the two matching algorithms
84 nigel 75 pcrepartial details of the partial matching facility
85 nigel 73 pcrepattern syntax and semantics of supported
86     regular expressions
87 ph10 208 pcresyntax quick syntax reference
88 nigel 73 pcreperform discussion of performance issues
89 nigel 77 pcreposix the POSIX-compatible C API
90 nigel 75 pcreprecompile details of saving and re-using precompiled patterns
91 nigel 73 pcresample discussion of the sample program
92 nigel 91 pcrestack discussion of stack usage
93 nigel 75 pcretest description of the pcretest testing command
94 nigel 63
95 ph10 208 In addition, in the "man" and HTML formats, there is a short page for
96 nigel 77 each C library function, listing its arguments and results.
97 nigel 63
98    
99     LIMITATIONS
100    
101 ph10 208 There are some size limitations in PCRE but it is hoped that they will
102 nigel 73 never in practice be relevant.
103 nigel 63
104 ph10 208 The maximum length of a compiled pattern is 65539 (sic) bytes if PCRE
105 nigel 73 is compiled with the default internal linkage size of 2. If you want to
106 ph10 208 process regular expressions that are truly enormous, you can compile
107     PCRE with an internal linkage size of 3 or 4 (see the README file in
108     the source distribution and the pcrebuild documentation for details).
109     In these cases the limit is substantially larger. However, the speed
110 nigel 93 of execution is slower.
111 nigel 63
112 ph10 208 All values in repeating quantifiers must be less than 65536.
113 nigel 63
114 nigel 93 There is no limit to the number of parenthesized subpatterns, but there
115     can be no more than 65535 capturing subpatterns.
116 nigel 63
117 nigel 93 The maximum length of name for a named subpattern is 32 characters, and
118     the maximum number of named subpatterns is 10000.
119 nigel 91
120 ph10 197 The maximum length of a subject string is the largest positive number
121     that an integer variable can hold. However, when using the traditional
122 nigel 77 matching function, PCRE uses recursion to handle subpatterns and indef-
123 ph10 197 inite repetition. This means that the available stack space may limit
124 nigel 77 the size of a subject string that can be processed by certain patterns.
125 nigel 91 For a discussion of stack issues, see the pcrestack documentation.
126 nigel 63
127    
128 nigel 75 UTF-8 AND UNICODE PROPERTY SUPPORT
129 nigel 63
130 ph10 197 From release 3.3, PCRE has had some support for character strings
131     encoded in the UTF-8 format. For release 4.0 this was greatly extended
132     to cover most common requirements, and in release 5.0 additional sup-
133 nigel 75 port for Unicode general category properties was added.
134 nigel 63
135 ph10 197 In order process UTF-8 strings, you must build PCRE to include UTF-8
136     support in the code, and, in addition, you must call pcre_compile()
137     with the PCRE_UTF8 option flag. When you do this, both the pattern and
138     any subject strings that are matched against it are treated as UTF-8
139 nigel 73 strings instead of just strings of bytes.
140 nigel 63
141 ph10 197 If you compile PCRE with UTF-8 support, but do not use it at run time,
142     the library will be a bit bigger, but the additional run time overhead
143 nigel 93 is limited to testing the PCRE_UTF8 flag occasionally, so should not be
144     very big.
145 nigel 63
146 nigel 75 If PCRE is built with Unicode character property support (which implies
147 ph10 197 UTF-8 support), the escape sequences \p{..}, \P{..}, and \X are sup-
148 nigel 75 ported. The available properties that can be tested are limited to the
149 ph10 197 general category properties such as Lu for an upper case letter or Nd
150     for a decimal number, the Unicode script names such as Arabic or Han,
151     and the derived properties Any and L&. A full list is given in the
152 nigel 87 pcrepattern documentation. Only the short names for properties are sup-
153 ph10 197 ported. For example, \p{L} matches a letter. Its Perl synonym, \p{Let-
154     ter}, is not supported. Furthermore, in Perl, many properties may
155     optionally be prefixed by "Is", for compatibility with Perl 5.6. PCRE
156 nigel 87 does not support this.
157 nigel 75
158 ph10 211 Validity of UTF-8 strings
159 nigel 63
160 ph10 211 When you set the PCRE_UTF8 flag, the strings passed as patterns and
161     subjects are (by default) checked for validity on entry to the relevant
162     functions. From release 7.3 of PCRE, the check is according the rules
163     of RFC 3629, which are themselves derived from the Unicode specifica-
164     tion. Earlier releases of PCRE followed the rules of RFC 2279, which
165     allows the full range of 31-bit values (0 to 0x7FFFFFFF). The current
166     check allows only values in the range U+0 to U+10FFFF, excluding U+D800
167     to U+DFFF.
168 nigel 63
169 ph10 211 The excluded code points are the "Low Surrogate Area" of Unicode, of
170     which the Unicode Standard says this: "The Low Surrogate Area does not
171     contain any character assignments, consequently no character code
172     charts or namelists are provided for this area. Surrogates are reserved
173     for use with UTF-16 and then must be used in pairs." The code points
174     that are encoded by UTF-16 pairs are available as independent code
175     points in the UTF-8 encoding. (In other words, the whole surrogate
176     thing is a fudge for UTF-16 which unfortunately messes up UTF-8.)
177    
178     If an invalid UTF-8 string is passed to PCRE, an error return
179     (PCRE_ERROR_BADUTF8) is given. In some situations, you may already know
180     that your strings are valid, and therefore want to skip these checks in
181     order to improve performance. If you set the PCRE_NO_UTF8_CHECK flag at
182     compile time or at run time, PCRE assumes that the pattern or subject
183     it is given (respectively) contains only valid UTF-8 codes. In this
184     case, it does not diagnose an invalid UTF-8 string.
185    
186     If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set,
187     what happens depends on why the string is invalid. If the string con-
188     forms to the "old" definition of UTF-8 (RFC 2279), it is processed as a
189     string of characters in the range 0 to 0x7FFFFFFF. In other words,
190     apart from the initial validity test, PCRE (when in UTF-8 mode) handles
191     strings according to the more liberal rules of RFC 2279. However, if
192     the string does not even conform to RFC 2279, the result is undefined.
193     Your program may crash.
194    
195     If you want to process strings of values in the full range 0 to
196     0x7FFFFFFF, encoded in a UTF-8-like manner as per the old RFC, you can
197     set PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in
198     this situation, you will have to apply your own validity check.
199    
200     General comments about UTF-8 mode
201    
202     1. An unbraced hexadecimal escape sequence (such as \xb3) matches a
203 nigel 87 two-byte UTF-8 character if the value is greater than 127.
204 nigel 63
205 ph10 211 2. Octal numbers up to \777 are recognized, and match two-byte UTF-8
206 nigel 91 characters for values greater than \177.
207    
208 ph10 211 3. Repeat quantifiers apply to complete UTF-8 characters, not to indi-
209 nigel 73 vidual bytes, for example: \x{100}{3}.
210 nigel 63
211 ph10 211 4. The dot metacharacter matches one UTF-8 character instead of a sin-
212 nigel 75 gle byte.
213 nigel 63
214 ph10 211 5. The escape sequence \C can be used to match a single byte in UTF-8
215 ph10 197 mode, but its use can lead to some strange effects. This facility is
216 nigel 77 not available in the alternative matching function, pcre_dfa_exec().
217 nigel 63
218 ph10 211 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly
219 ph10 197 test characters of any code value, but the characters that PCRE recog-
220     nizes as digits, spaces, or word characters remain the same set as
221 nigel 75 before, all with values less than 256. This remains true even when PCRE
222 ph10 197 includes Unicode property support, because to do otherwise would slow
223     down PCRE in many common cases. If you really want to test for a wider
224     sense of, say, "digit", you must use Unicode property tests such as
225 nigel 75 \p{Nd}.
226 nigel 63
227 ph10 211 7. Similarly, characters that match the POSIX named character classes
228 nigel 75 are all low-valued characters.
229 nigel 63
230 ph10 211 8. However, the Perl 5.10 horizontal and vertical whitespace matching
231 ph10 182 escapes (\h, \H, \v, and \V) do match all the appropriate Unicode char-
232     acters.
233    
234 ph10 211 9. Case-insensitive matching applies only to characters whose values
235 ph10 197 are less than 128, unless PCRE is built with Unicode property support.
236     Even when Unicode property support is available, PCRE still uses its
237     own character tables when checking the case of low-valued characters,
238     so as not to degrade performance. The Unicode property information is
239 nigel 87 used only for characters with higher values. Even when Unicode property
240     support is available, PCRE supports case-insensitive matching only when
241 ph10 197 there is a one-to-one mapping between a letter's cases. There are a
242     small number of many-to-one mappings in Unicode; these are not sup-
243 nigel 87 ported by PCRE.
244 nigel 63
245    
246     AUTHOR
247    
248 nigel 77 Philip Hazel
249 ph10 99 University Computing Service
250 nigel 93 Cambridge CB2 3QH, England.
251 nigel 63
252 ph10 197 Putting an actual email address here seems to have been a spam magnet,
253     so I've taken it away. If you want to email me, use my two initials,
254 ph10 153 followed by the two digits 10, at the domain cam.ac.uk.
255 nigel 77
256 nigel 63
257 ph10 99 REVISION
258 nigel 63
259 ph10 211 Last updated: 09 August 2007
260 ph10 99 Copyright (c) 1997-2007 University of Cambridge.
261     ------------------------------------------------------------------------------
262 ph10 259
263    
264 nigel 79 PCREBUILD(3) PCREBUILD(3)
265 nigel 63
266 nigel 79
267 nigel 73 NAME
268     PCRE - Perl-compatible regular expressions
269    
270 nigel 77
271 nigel 63 PCRE BUILD-TIME OPTIONS
272    
273 nigel 73 This document describes the optional features of PCRE that can be
274 ph10 261 selected when the library is compiled. It assumes use of the configure
275     script, where the optional features are selected or deselected by pro-
276     viding options to configure before running the make command. However,
277     the same options can be selected in both Unix-like and non-Unix-like
278     environments using the GUI facility of CMakeSetup if you are using
279     CMake instead of configure to build PCRE.
280 nigel 63
281 ph10 261 The complete list of options for configure (which includes the standard
282     ones such as the selection of the installation directory) can be
283     obtained by running
284    
285 nigel 73 ./configure --help
286 nigel 63
287 ph10 261 The following sections include descriptions of options whose names
288 ph10 128 begin with --enable or --disable. These settings specify changes to the
289 ph10 261 defaults for the configure command. Because of the way that configure
290     works, --enable and --disable always come in pairs, so the complemen-
291     tary option always exists as well, but as it specifies the default, it
292 ph10 128 is not described.
293 nigel 63
294    
295 nigel 83 C++ SUPPORT
296    
297     By default, the configure script will search for a C++ compiler and C++
298     header files. If it finds them, it automatically builds the C++ wrapper
299     library for PCRE. You can disable this by adding
300    
301     --disable-cpp
302    
303     to the configure command.
304    
305    
306 nigel 63 UTF-8 SUPPORT
307    
308 nigel 73 To build PCRE with support for UTF-8 character strings, add
309 nigel 63
310 nigel 73 --enable-utf8
311 nigel 63
312 ph10 261 to the configure command. Of itself, this does not make PCRE treat
313     strings as UTF-8. As well as compiling PCRE with this option, you also
314     have have to set the PCRE_UTF8 option when you call the pcre_compile()
315 nigel 73 function.
316 nigel 63
317    
318 nigel 75 UNICODE CHARACTER PROPERTY SUPPORT
319    
320 ph10 261 UTF-8 support allows PCRE to process character values greater than 255
321     in the strings that it handles. On its own, however, it does not pro-
322 nigel 75 vide any facilities for accessing the properties of such characters. If
323 ph10 261 you want to be able to use the pattern escapes \P, \p, and \X, which
324 nigel 75 refer to Unicode character properties, you must add
325    
326     --enable-unicode-properties
327    
328 ph10 261 to the configure command. This implies UTF-8 support, even if you have
329 nigel 75 not explicitly requested it.
330    
331 ph10 261 Including Unicode property support adds around 30K of tables to the
332     PCRE library. Only the general category properties such as Lu and Nd
333 ph10 128 are supported. Details are given in the pcrepattern documentation.
334 nigel 75
335    
336 nigel 63 CODE VALUE OF NEWLINE
337    
338 ph10 261 By default, PCRE interprets character 10 (linefeed, LF) as indicating
339     the end of a line. This is the normal newline character on Unix-like
340 nigel 91 systems. You can compile PCRE to use character 13 (carriage return, CR)
341     instead, by adding
342 nigel 63
343 nigel 73 --enable-newline-is-cr
344 nigel 63
345 ph10 261 to the configure command. There is also a --enable-newline-is-lf
346 nigel 91 option, which explicitly specifies linefeed as the newline character.
347 nigel 63
348 nigel 91 Alternatively, you can specify that line endings are to be indicated by
349     the two character sequence CRLF. If you want this, add
350 nigel 63
351 nigel 91 --enable-newline-is-crlf
352    
353 nigel 93 to the configure command. There is a fourth option, specified by
354 nigel 91
355 ph10 150 --enable-newline-is-anycrlf
356    
357 ph10 261 which causes PCRE to recognize any of the three sequences CR, LF, or
358 ph10 150 CRLF as indicating a line ending. Finally, a fifth option, specified by
359    
360 nigel 93 --enable-newline-is-any
361 nigel 91
362 ph10 150 causes PCRE to recognize any Unicode newline sequence.
363 nigel 93
364 ph10 261 Whatever line ending convention is selected when PCRE is built can be
365     overridden when the library functions are called. At build time it is
366 nigel 93 conventional to use the standard for your operating system.
367    
368    
369 ph10 231 WHAT \R MATCHES
370    
371 ph10 261 By default, the sequence \R in a pattern matches any Unicode newline
372     sequence, whatever has been selected as the line ending sequence. If
373 ph10 231 you specify
374    
375     --enable-bsr-anycrlf
376    
377 ph10 261 the default is changed so that \R matches only CR, LF, or CRLF. What-
378     ever is selected when PCRE is built can be overridden when the library
379 ph10 231 functions are called.
380    
381    
382 nigel 63 BUILDING SHARED AND STATIC LIBRARIES
383    
384 ph10 261 The PCRE building process uses libtool to build both shared and static
385     Unix libraries by default. You can suppress one of these by adding one
386 nigel 73 of
387 nigel 63
388 nigel 73 --disable-shared
389     --disable-static
390 nigel 63
391 nigel 73 to the configure command, as required.
392 nigel 63
393    
394     POSIX MALLOC USAGE
395    
396 nigel 75 When PCRE is called through the POSIX interface (see the pcreposix doc-
397 ph10 261 umentation), additional working storage is required for holding the
398     pointers to capturing substrings, because PCRE requires three integers
399     per substring, whereas the POSIX interface provides only two. If the
400 nigel 73 number of expected substrings is small, the wrapper function uses space
401     on the stack, because this is faster than using malloc() for each call.
402     The default threshold above which the stack is no longer used is 10; it
403     can be changed by adding a setting such as
404 nigel 63
405 nigel 73 --with-posix-malloc-threshold=20
406 nigel 63
407 nigel 73 to the configure command.
408 nigel 63
409    
410     HANDLING VERY LARGE PATTERNS
411    
412 ph10 261 Within a compiled pattern, offset values are used to point from one
413     part to another (for example, from an opening parenthesis to an alter-
414     nation metacharacter). By default, two-byte values are used for these
415     offsets, leading to a maximum size for a compiled pattern of around
416     64K. This is sufficient to handle all but the most gigantic patterns.
417     Nevertheless, some people do want to process enormous patterns, so it
418     is possible to compile PCRE to use three-byte or four-byte offsets by
419 nigel 73 adding a setting such as
420 nigel 63
421 nigel 73 --with-link-size=3
422 nigel 63
423 ph10 261 to the configure command. The value given must be 2, 3, or 4. Using
424     longer offsets slows down the operation of PCRE because it has to load
425 nigel 73 additional bytes when handling them.
426 nigel 63
427    
428 nigel 73 AVOIDING EXCESSIVE STACK USAGE
429    
430 nigel 77 When matching with the pcre_exec() function, PCRE implements backtrack-
431 ph10 261 ing by making recursive calls to an internal function called match().
432     In environments where the size of the stack is limited, this can se-
433     verely limit PCRE's operation. (The Unix environment does not usually
434 nigel 91 suffer from this problem, but it may sometimes be necessary to increase
435 ph10 261 the maximum stack size. There is a discussion in the pcrestack docu-
436     mentation.) An alternative approach to recursion that uses memory from
437     the heap to remember data, instead of using recursive function calls,
438     has been implemented to work round the problem of limited stack size.
439 nigel 91 If you want to build a version of PCRE that works this way, add
440 nigel 73
441     --disable-stack-for-recursion
442    
443 ph10 261 to the configure command. With this configuration, PCRE will use the
444     pcre_stack_malloc and pcre_stack_free variables to call memory manage-
445     ment functions. By default these point to malloc() and free(), but you
446 ph10 182 can replace the pointers so that your own functions are used.
447 nigel 73
448 ph10 261 Separate functions are provided rather than using pcre_malloc and
449     pcre_free because the usage is very predictable: the block sizes
450     requested are always the same, and the blocks are always freed in
451     reverse order. A calling program might be able to implement optimized
452     functions that perform better than malloc() and free(). PCRE runs
453 ph10 182 noticeably more slowly when built in this way. This option affects only
454 ph10 261 the pcre_exec() function; it is not relevant for the the
455 ph10 182 pcre_dfa_exec() function.
456 nigel 73
457 ph10 182
458 nigel 91 LIMITING PCRE RESOURCE USAGE
459    
460 ph10 261 Internally, PCRE has a function called match(), which it calls repeat-
461     edly (sometimes recursively) when matching a pattern with the
462     pcre_exec() function. By controlling the maximum number of times this
463     function may be called during a single matching operation, a limit can
464     be placed on the resources used by a single call to pcre_exec(). The
465     limit can be changed at run time, as described in the pcreapi documen-
466     tation. The default is 10 million, but this can be changed by adding a
467 nigel 91 setting such as
468    
469     --with-match-limit=500000
470    
471 ph10 261 to the configure command. This setting has no effect on the
472 nigel 91 pcre_dfa_exec() matching function.
473    
474 ph10 261 In some environments it is desirable to limit the depth of recursive
475 nigel 91 calls of match() more strictly than the total number of calls, in order
476 ph10 261 to restrict the maximum amount of stack (or heap, if --disable-stack-
477 nigel 91 for-recursion is specified) that is used. A second limit controls this;
478 ph10 261 it defaults to the value that is set for --with-match-limit, which
479     imposes no additional constraints. However, you can set a lower limit
480 nigel 91 by adding, for example,
481    
482     --with-match-limit-recursion=10000
483    
484 ph10 261 to the configure command. This value can also be overridden at run
485 nigel 91 time.
486    
487    
488 ph10 128 CREATING CHARACTER TABLES AT BUILD TIME
489    
490 ph10 261 PCRE uses fixed tables for processing characters whose code values are
491     less than 256. By default, PCRE is built with a set of tables that are
492     distributed in the file pcre_chartables.c.dist. These tables are for
493 ph10 128 ASCII codes only. If you add
494    
495     --enable-rebuild-chartables
496    
497 ph10 261 to the configure command, the distributed tables are no longer used.
498     Instead, a program called dftables is compiled and run. This outputs
499 ph10 128 the source for new set of tables, created in the default locale of your
500     C runtime system. (This method of replacing the tables does not work if
501 ph10 261 you are cross compiling, because dftables is run on the local host. If
502     you need to create alternative tables when cross compiling, you will
503 ph10 128 have to do so "by hand".)
504    
505    
506 nigel 73 USING EBCDIC CODE
507    
508 ph10 261 PCRE assumes by default that it will run in an environment where the
509     character code is ASCII (or Unicode, which is a superset of ASCII).
510     This is the case for most computer operating systems. PCRE can, how-
511 ph10 197 ever, be compiled to run in an EBCDIC environment by adding
512 nigel 73
513     --enable-ebcdic
514    
515 ph10 128 to the configure command. This setting implies --enable-rebuild-charta-
516 ph10 261 bles. You should only use it if you know that you are in an EBCDIC
517 ph10 197 environment (for example, an IBM mainframe operating system).
518 nigel 73
519 nigel 93
520 ph10 286 PCREGREP OPTIONS FOR COMPRESSED FILE SUPPORT
521    
522     By default, pcregrep reads all files as plain text. You can build it so
523     that it recognizes files whose names end in .gz or .bz2, and reads them
524     with libz or libbz2, respectively, by adding one or both of
525    
526     --enable-pcregrep-libz
527     --enable-pcregrep-libbz2
528    
529     to the configure command. These options naturally require that the rel-
530     evant libraries are installed on your system. Configuration will fail
531     if they are not.
532    
533    
534 ph10 289 PCRETEST OPTION FOR LIBREADLINE SUPPORT
535    
536     If you add
537    
538     --enable-pcretest-libreadline
539    
540     to the configure command, pcretest is linked with the libreadline
541     library, and when its input is from a terminal, it reads it using the
542     readline() function. This provides line-editing and history facilities.
543     Note that libreadline is GPL-licenced, so if you distribute a binary of
544     pcretest linked in this way, there may be licensing issues.
545    
546    
547 nigel 93 SEE ALSO
548    
549     pcreapi(3), pcre_config(3).
550    
551 nigel 63
552 ph10 99 AUTHOR
553 nigel 63
554 ph10 99 Philip Hazel
555     University Computing Service
556     Cambridge CB2 3QH, England.
557    
558    
559     REVISION
560    
561 ph10 289 Last updated: 18 December 2007
562 ph10 99 Copyright (c) 1997-2007 University of Cambridge.
563     ------------------------------------------------------------------------------
564 ph10 259
565    
566 nigel 79 PCREMATCHING(3) PCREMATCHING(3)
567 nigel 63
568 nigel 79
569 nigel 77 NAME
570     PCRE - Perl-compatible regular expressions
571 nigel 73
572 nigel 77
573     PCRE MATCHING ALGORITHMS
574    
575     This document describes the two different algorithms that are available
576     in PCRE for matching a compiled regular expression against a given sub-
577     ject string. The "standard" algorithm is the one provided by the
578     pcre_exec() function. This works in the same was as Perl's matching
579     function, and provides a Perl-compatible matching operation.
580    
581     An alternative algorithm is provided by the pcre_dfa_exec() function;
582     this operates in a different way, and is not Perl-compatible. It has
583     advantages and disadvantages compared with the standard algorithm, and
584     these are described below.
585    
586     When there is only one possible way in which a given subject string can
587     match a pattern, the two algorithms give the same answer. A difference
588     arises, however, when there are multiple possibilities. For example, if
589     the pattern
590    
591     ^<.*>
592    
593     is matched against the string
594    
595     <something> <something else> <something further>
596    
597     there are three possible answers. The standard algorithm finds only one
598 nigel 93 of them, whereas the alternative algorithm finds all three.
599 nigel 77
600    
601     REGULAR EXPRESSIONS AS TREES
602    
603     The set of strings that are matched by a regular expression can be rep-
604     resented as a tree structure. An unlimited repetition in the pattern
605     makes the tree of infinite size, but it is still a tree. Matching the
606     pattern to a given subject string (from a given starting point) can be
607 nigel 91 thought of as a search of the tree. There are two ways to search a
608     tree: depth-first and breadth-first, and these correspond to the two
609     matching algorithms provided by PCRE.
610 nigel 77
611    
612     THE STANDARD MATCHING ALGORITHM
613    
614 ph10 148 In the terminology of Jeffrey Friedl's book "Mastering Regular Expres-
615     sions", the standard algorithm is an "NFA algorithm". It conducts a
616 nigel 77 depth-first search of the pattern tree. That is, it proceeds along a
617     single path through the tree, checking that the subject matches what is
618     required. When there is a mismatch, the algorithm tries any alterna-
619     tives at the current point, and if they all fail, it backs up to the
620     previous branch point in the tree, and tries the next alternative
621     branch at that level. This often involves backing up (moving to the
622     left) in the subject string as well. The order in which repetition
623     branches are tried is controlled by the greedy or ungreedy nature of
624     the quantifier.
625    
626     If a leaf node is reached, a matching string has been found, and at
627     that point the algorithm stops. Thus, if there is more than one possi-
628     ble match, this algorithm returns the first one that it finds. Whether
629     this is the shortest, the longest, or some intermediate length depends
630     on the way the greedy and ungreedy repetition quantifiers are specified
631     in the pattern.
632    
633     Because it ends up with a single path through the tree, it is rela-
634     tively straightforward for this algorithm to keep track of the sub-
635     strings that are matched by portions of the pattern in parentheses.
636     This provides support for capturing parentheses and back references.
637    
638    
639 nigel 93 THE ALTERNATIVE MATCHING ALGORITHM
640 nigel 77
641 nigel 93 This algorithm conducts a breadth-first search of the tree. Starting
642     from the first matching point in the subject, it scans the subject
643     string from left to right, once, character by character, and as it does
644     this, it remembers all the paths through the tree that represent valid
645     matches. In Friedl's terminology, this is a kind of "DFA algorithm",
646     though it is not implemented as a traditional finite state machine (it
647     keeps multiple states active simultaneously).
648 nigel 77
649 nigel 93 The scan continues until either the end of the subject is reached, or
650     there are no more unterminated paths. At this point, terminated paths
651     represent the different matching possibilities (if there are none, the
652     match has failed). Thus, if there is more than one possible match,
653 nigel 77 this algorithm finds all of them, and in particular, it finds the long-
654 nigel 93 est. In PCRE, there is an option to stop the algorithm after the first
655 nigel 77 match (which is necessarily the shortest) has been found.
656    
657     Note that all the matches that are found start at the same point in the
658     subject. If the pattern
659    
660     cat(er(pillar)?)
661    
662 nigel 93 is matched against the string "the caterpillar catchment", the result
663     will be the three strings "cat", "cater", and "caterpillar" that start
664 nigel 77 at the fourth character of the subject. The algorithm does not automat-
665     ically move on to find matches that start at later positions.
666    
667     There are a number of features of PCRE regular expressions that are not
668 nigel 93 supported by the alternative matching algorithm. They are as follows:
669 nigel 77
670 nigel 93 1. Because the algorithm finds all possible matches, the greedy or
671     ungreedy nature of repetition quantifiers is not relevant. Greedy and
672     ungreedy quantifiers are treated in exactly the same way. However, pos-
673     sessive quantifiers can make a difference when what follows could also
674     match what is quantified, for example in a pattern like this:
675 nigel 77
676 nigel 93 ^a++\w!
677    
678     This pattern matches "aaab!" but not "aaa!", which would be matched by
679     a non-possessive quantifier. Similarly, if an atomic group is present,
680     it is matched as if it were a standalone pattern at the current point,
681     and the longest match is then "locked in" for the rest of the overall
682     pattern.
683    
684 nigel 77 2. When dealing with multiple paths through the tree simultaneously, it
685 nigel 93 is not straightforward to keep track of captured substrings for the
686     different matching possibilities, and PCRE's implementation of this
687 nigel 77 algorithm does not attempt to do this. This means that no captured sub-
688     strings are available.
689    
690 nigel 93 3. Because no substrings are captured, back references within the pat-
691 nigel 77 tern are not supported, and cause errors if encountered.
692    
693 nigel 93 4. For the same reason, conditional expressions that use a backrefer-
694     ence as the condition or test for a specific group recursion are not
695     supported.
696 nigel 77
697 ph10 172 5. Because many paths through the tree may be active, the \K escape
698     sequence, which resets the start of the match when encountered (but may
699     be on some paths and not on others), is not supported. It causes an
700     error if encountered.
701    
702     6. Callouts are supported, but the value of the capture_top field is
703 nigel 77 always 1, and the value of the capture_last field is always -1.
704    
705 ph10 211 7. The \C escape sequence, which (in the standard algorithm) matches a
706 ph10 172 single byte, even in UTF-8 mode, is not supported because the alterna-
707     tive algorithm moves through the subject string one character at a
708 nigel 93 time, for all active paths through the tree.
709 nigel 77
710 ph10 211 8. None of the backtracking control verbs such as (*PRUNE) are sup-
711     ported.
712 nigel 77
713 ph10 211
714 nigel 93 ADVANTAGES OF THE ALTERNATIVE ALGORITHM
715 nigel 77
716 ph10 211 Using the alternative matching algorithm provides the following advan-
717 nigel 93 tages:
718 nigel 77
719     1. All possible matches (at a single point in the subject) are automat-
720 ph10 211 ically found, and in particular, the longest match is found. To find
721 nigel 77 more than one match using the standard algorithm, you have to do kludgy
722     things with callouts.
723    
724 ph10 211 2. There is much better support for partial matching. The restrictions
725     on the content of the pattern that apply when using the standard algo-
726     rithm for partial matching do not apply to the alternative algorithm.
727     For non-anchored patterns, the starting position of a partial match is
728 nigel 93 available.
729 nigel 77
730 ph10 211 3. Because the alternative algorithm scans the subject string just
731     once, and never needs to backtrack, it is possible to pass very long
732     subject strings to the matching function in several pieces, checking
733 nigel 93 for partial matching each time.
734 nigel 77
735    
736 nigel 93 DISADVANTAGES OF THE ALTERNATIVE ALGORITHM
737 nigel 77
738 nigel 93 The alternative algorithm suffers from a number of disadvantages:
739 nigel 77
740 ph10 211 1. It is substantially slower than the standard algorithm. This is
741     partly because it has to search for all possible matches, but is also
742 nigel 77 because it is less susceptible to optimization.
743    
744     2. Capturing parentheses and back references are not supported.
745    
746 nigel 93 3. Although atomic groups are supported, their use does not provide the
747     performance advantage that it does for the standard algorithm.
748 nigel 77
749    
750 ph10 99 AUTHOR
751 nigel 77
752 ph10 99 Philip Hazel
753     University Computing Service
754     Cambridge CB2 3QH, England.
755    
756    
757     REVISION
758    
759 ph10 211 Last updated: 08 August 2007
760 ph10 99 Copyright (c) 1997-2007 University of Cambridge.
761     ------------------------------------------------------------------------------
762 ph10 259
763    
764 nigel 79 PCREAPI(3) PCREAPI(3)
765 nigel 77
766 nigel 79
767 nigel 73 NAME
768     PCRE - Perl-compatible regular expressions
769    
770 nigel 77
771 nigel 75 PCRE NATIVE API
772 nigel 63
773 nigel 73 #include <pcre.h>
774 nigel 41
775 nigel 73 pcre *pcre_compile(const char *pattern, int options,
776     const char **errptr, int *erroffset,
777     const unsigned char *tableptr);
778 nigel 41
779 nigel 77 pcre *pcre_compile2(const char *pattern, int options,
780     int *errorcodeptr,
781     const char **errptr, int *erroffset,
782     const unsigned char *tableptr);
783    
784 nigel 73 pcre_extra *pcre_study(const pcre *code, int options,
785     const char **errptr);
786 nigel 41
787 nigel 73 int pcre_exec(const pcre *code, const pcre_extra *extra,
788     const char *subject, int length, int startoffset,
789     int options, int *ovector, int ovecsize);
790 nigel 41
791 nigel 77 int pcre_dfa_exec(const pcre *code, const pcre_extra *extra,
792     const char *subject, int length, int startoffset,
793     int options, int *ovector, int ovecsize,
794     int *workspace, int wscount);
795    
796 nigel 73 int pcre_copy_named_substring(const pcre *code,
797     const char *subject, int *ovector,
798     int stringcount, const char *stringname,
799     char *buffer, int buffersize);
800 nigel 63
801 nigel 73 int pcre_copy_substring(const char *subject, int *ovector,
802     int stringcount, int stringnumber, char *buffer,
803     int buffersize);
804 nigel 41
805 nigel 73 int pcre_get_named_substring(const pcre *code,
806     const char *subject, int *ovector,
807     int stringcount, const char *stringname,
808     const char **stringptr);
809 nigel 63
810 nigel 73 int pcre_get_stringnumber(const pcre *code,
811     const char *name);
812 nigel 63
813 nigel 91 int pcre_get_stringtable_entries(const pcre *code,
814     const char *name, char **first, char **last);
815    
816 nigel 73 int pcre_get_substring(const char *subject, int *ovector,
817     int stringcount, int stringnumber,
818     const char **stringptr);
819 nigel 41
820 nigel 73 int pcre_get_substring_list(const char *subject,
821     int *ovector, int stringcount, const char ***listptr);
822 nigel 41
823 nigel 73 void pcre_free_substring(const char *stringptr);
824 nigel 49
825 nigel 73 void pcre_free_substring_list(const char **stringptr);
826 nigel 49
827 nigel 73 const unsigned char *pcre_maketables(void);
828 nigel 41
829 nigel 73 int pcre_fullinfo(const pcre *code, const pcre_extra *extra,
830     int what, void *where);
831 nigel 43
832 nigel 73 int pcre_info(const pcre *code, int *optptr, int *firstcharptr);
833 nigel 63
834 nigel 77 int pcre_refcount(pcre *code, int adjust);
835    
836 nigel 73 int pcre_config(int what, void *where);
837 nigel 41
838 nigel 73 char *pcre_version(void);
839 nigel 63
840 nigel 73 void *(*pcre_malloc)(size_t);
841 nigel 41
842 nigel 73 void (*pcre_free)(void *);
843 nigel 41
844 nigel 73 void *(*pcre_stack_malloc)(size_t);
845 nigel 41
846 nigel 73 void (*pcre_stack_free)(void *);
847 nigel 41
848 nigel 73 int (*pcre_callout)(pcre_callout_block *);
849 nigel 41
850 nigel 73
851 nigel 75 PCRE API OVERVIEW
852 nigel 41
853 nigel 73 PCRE has its own native API, which is described in this document. There
854 nigel 93 are also some wrapper functions that correspond to the POSIX regular
855 nigel 77 expression API. These are described in the pcreposix documentation.
856     Both of these APIs define a set of C function calls. A C++ wrapper is
857     distributed with PCRE. It is documented in the pcrecpp page.
858 nigel 43
859 nigel 77 The native API C function prototypes are defined in the header file
860     pcre.h, and on Unix systems the library itself is called libpcre. It
861 nigel 75 can normally be accessed by adding -lpcre to the command for linking an
862     application that uses PCRE. The header file defines the macros
863     PCRE_MAJOR and PCRE_MINOR to contain the major and minor release num-
864     bers for the library. Applications can use these to include support
865     for different releases of PCRE.
866 nigel 41
867 nigel 77 The functions pcre_compile(), pcre_compile2(), pcre_study(), and
868     pcre_exec() are used for compiling and matching regular expressions in
869     a Perl-compatible manner. A sample program that demonstrates the sim-
870     plest way of using them is provided in the file called pcredemo.c in
871     the source distribution. The pcresample documentation describes how to
872 ph10 313 compile and run it.
873 nigel 49
874 nigel 77 A second matching function, pcre_dfa_exec(), which is not Perl-compati-
875     ble, is also provided. This uses a different algorithm for the match-
876 nigel 91 ing. The alternative algorithm finds all possible matches (at a given
877 nigel 93 point in the subject), and scans the subject just once. However, this
878     algorithm does not return captured substrings. A description of the two
879     matching algorithms and their advantages and disadvantages is given in
880     the pcrematching documentation.
881 nigel 63
882 nigel 77 In addition to the main compiling and matching functions, there are
883     convenience functions for extracting captured substrings from a subject
884     string that is matched by pcre_exec(). They are:
885    
886 nigel 73 pcre_copy_substring()
887     pcre_copy_named_substring()
888     pcre_get_substring()
889     pcre_get_named_substring()
890     pcre_get_substring_list()
891 nigel 75 pcre_get_stringnumber()
892 nigel 91 pcre_get_stringtable_entries()
893 nigel 63
894 nigel 73 pcre_free_substring() and pcre_free_substring_list() are also provided,
895     to free the memory used for extracted strings.
896 nigel 41
897 nigel 77 The function pcre_maketables() is used to build a set of character
898     tables in the current locale for passing to pcre_compile(),
899     pcre_exec(), or pcre_dfa_exec(). This is an optional facility that is
900     provided for specialist use. Most commonly, no special tables are
901     passed, in which case internal tables that are generated when PCRE is
902     built are used.
903 nigel 49
904 nigel 75 The function pcre_fullinfo() is used to find out information about a
905     compiled pattern; pcre_info() is an obsolete version that returns only
906     some of the available information, but is retained for backwards com-
907     patibility. The function pcre_version() returns a pointer to a string
908 nigel 73 containing the version of PCRE and its date of release.
909 nigel 41
910 nigel 77 The function pcre_refcount() maintains a reference count in a data
911     block containing a compiled pattern. This is provided for the benefit
912     of object-oriented applications.
913    
914 nigel 75 The global variables pcre_malloc and pcre_free initially contain the
915     entry points of the standard malloc() and free() functions, respec-
916 nigel 73 tively. PCRE calls the memory management functions via these variables,
917 nigel 75 so a calling program can replace them if it wishes to intercept the
918 nigel 73 calls. This should be done before calling any PCRE functions.
919 nigel 41
920 nigel 75 The global variables pcre_stack_malloc and pcre_stack_free are also
921     indirections to memory management functions. These special functions
922     are used only when PCRE is compiled to use the heap for remembering
923 nigel 77 data, instead of recursive function calls, when running the pcre_exec()
924 nigel 91 function. See the pcrebuild documentation for details of how to do
925     this. It is a non-standard way of building PCRE, for use in environ-
926     ments that have limited stacks. Because of the greater use of memory
927     management, it runs more slowly. Separate functions are provided so
928     that special-purpose external code can be used for this case. When
929     used, these functions are always called in a stack-like manner (last
930     obtained, first freed), and always for memory blocks of the same size.
931     There is a discussion about PCRE's stack usage in the pcrestack docu-
932     mentation.
933 nigel 41
934 nigel 73 The global variable pcre_callout initially contains NULL. It can be set
935 nigel 77 by the caller to a "callout" function, which PCRE will then call at
936     specified points during a matching operation. Details are given in the
937 nigel 73 pcrecallout documentation.
938 nigel 41
939 nigel 73
940 nigel 91 NEWLINES
941    
942 ph10 150 PCRE supports five different conventions for indicating line breaks in
943 nigel 93 strings: a single CR (carriage return) character, a single LF (line-
944 ph10 150 feed) character, the two-character sequence CRLF, any of the three pre-
945     ceding, or any Unicode newline sequence. The Unicode newline sequences
946     are the three just mentioned, plus the single characters VT (vertical
947     tab, U+000B), FF (formfeed, U+000C), NEL (next line, U+0085), LS (line
948     separator, U+2028), and PS (paragraph separator, U+2029).
949 nigel 93
950     Each of the first three conventions is used by at least one operating
951     system as its standard newline sequence. When PCRE is built, a default
952     can be specified. The default default is LF, which is the Unix stan-
953     dard. When PCRE is run, the default can be overridden, either when a
954     pattern is compiled, or when it is matched.
955    
956 ph10 227 At compile time, the newline convention can be specified by the options
957     argument of pcre_compile(), or it can be specified by special text at
958     the start of the pattern itself; this overrides any other settings. See
959     the pcrepattern page for details of the special character sequences.
960    
961 nigel 91 In the PCRE documentation the word "newline" is used to mean "the char-
962 ph10 227 acter or pair of characters that indicate a line break". The choice of
963     newline convention affects the handling of the dot, circumflex, and
964 nigel 93 dollar metacharacters, the handling of #-comments in /x mode, and, when
965 ph10 227 CRLF is a recognized line ending sequence, the match position advance-
966     ment for a non-anchored pattern. There is more detail about this in the
967 ph10 231 section on pcre_exec() options below.
968 nigel 91
969 ph10 231 The choice of newline convention does not affect the interpretation of
970     the \n or \r escape sequences, nor does it affect what \R matches,
971     which is controlled in a similar way, but by separate options.
972 nigel 91
973 ph10 231
974 nigel 63 MULTITHREADING
975    
976 ph10 231 The PCRE functions can be used in multi-threading applications, with
977 nigel 73 the proviso that the memory management functions pointed to by
978     pcre_malloc, pcre_free, pcre_stack_malloc, and pcre_stack_free, and the
979     callout function pointed to by pcre_callout, are shared by all threads.
980 nigel 41
981 ph10 231 The compiled form of a regular expression is not altered during match-
982 nigel 73 ing, so the same compiled pattern can safely be used by several threads
983     at once.
984 nigel 41
985    
986 nigel 75 SAVING PRECOMPILED PATTERNS FOR LATER USE
987    
988     The compiled form of a regular expression can be saved and re-used at a
989 ph10 231 later time, possibly by a different program, and even on a host other
990     than the one on which it was compiled. Details are given in the
991     pcreprecompile documentation. However, compiling a regular expression
992     with one version of PCRE for use with a different version is not guar-
993 ph10 155 anteed to work and may cause crashes.
994 nigel 75
995    
996 nigel 63 CHECKING BUILD-TIME OPTIONS
997 nigel 41
998 nigel 73 int pcre_config(int what, void *where);
999 nigel 63
1000 ph10 231 The function pcre_config() makes it possible for a PCRE client to dis-
1001 nigel 73 cover which optional features have been compiled into the PCRE library.
1002 ph10 231 The pcrebuild documentation has more details about these optional fea-
1003 nigel 73 tures.
1004 nigel 63
1005 ph10 231 The first argument for pcre_config() is an integer, specifying which
1006 nigel 73 information is required; the second argument is a pointer to a variable
1007 ph10 231 into which the information is placed. The following information is
1008 nigel 73 available:
1009 nigel 63
1010 nigel 73 PCRE_CONFIG_UTF8
1011 nigel 63
1012 ph10 231 The output is an integer that is set to one if UTF-8 support is avail-
1013 nigel 73 able; otherwise it is set to zero.
1014 nigel 63
1015 nigel 75 PCRE_CONFIG_UNICODE_PROPERTIES
1016    
1017 ph10 231 The output is an integer that is set to one if support for Unicode
1018 nigel 75 character properties is available; otherwise it is set to zero.
1019    
1020 nigel 73 PCRE_CONFIG_NEWLINE
1021 nigel 63
1022 ph10 231 The output is an integer whose value specifies the default character
1023     sequence that is recognized as meaning "newline". The four values that
1024 ph10 150 are supported are: 10 for LF, 13 for CR, 3338 for CRLF, -2 for ANYCRLF,
1025 ph10 231 and -1 for ANY. The default should normally be the standard sequence
1026 ph10 150 for your operating system.
1027 nigel 63
1028 ph10 231 PCRE_CONFIG_BSR
1029    
1030     The output is an integer whose value indicates what character sequences
1031     the \R escape sequence matches by default. A value of 0 means that \R
1032     matches any Unicode line ending sequence; a value of 1 means that \R
1033     matches only CR, LF, or CRLF. The default can be overridden when a pat-
1034     tern is compiled or matched.
1035    
1036 nigel 73 PCRE_CONFIG_LINK_SIZE
1037 nigel 63
1038 ph10 231 The output is an integer that contains the number of bytes used for
1039 nigel 73 internal linkage in compiled regular expressions. The value is 2, 3, or
1040 ph10 231 4. Larger values allow larger regular expressions to be compiled, at
1041     the expense of slower matching. The default value of 2 is sufficient
1042     for all but the most massive patterns, since it allows the compiled
1043 nigel 73 pattern to be up to 64K in size.
1044 nigel 63
1045 nigel 73 PCRE_CONFIG_POSIX_MALLOC_THRESHOLD
1046 nigel 63
1047 ph10 231 The output is an integer that contains the threshold above which the
1048     POSIX interface uses malloc() for output vectors. Further details are
1049 nigel 73 given in the pcreposix documentation.
1050 nigel 63
1051 nigel 73 PCRE_CONFIG_MATCH_LIMIT
1052 nigel 63
1053 nigel 73 The output is an integer that gives the default limit for the number of
1054 ph10 231 internal matching function calls in a pcre_exec() execution. Further
1055 nigel 73 details are given with pcre_exec() below.
1056 nigel 63
1057 nigel 87 PCRE_CONFIG_MATCH_LIMIT_RECURSION
1058    
1059 ph10 231 The output is an integer that gives the default limit for the depth of
1060     recursion when calling the internal matching function in a pcre_exec()
1061 nigel 87 execution. Further details are given with pcre_exec() below.
1062    
1063 nigel 73 PCRE_CONFIG_STACKRECURSE
1064 nigel 63
1065 ph10 231 The output is an integer that is set to one if internal recursion when
1066 nigel 77 running pcre_exec() is implemented by recursive function calls that use
1067 ph10 231 the stack to remember their state. This is the usual way that PCRE is
1068 nigel 77 compiled. The output is zero if PCRE was compiled to use blocks of data
1069 ph10 231 on the heap instead of recursive function calls. In this case,
1070     pcre_stack_malloc and pcre_stack_free are called to manage memory
1071 nigel 77 blocks on the heap, thus avoiding the use of the stack.
1072 nigel 73
1073    
1074 nigel 41 COMPILING A PATTERN
1075 nigel 63
1076 nigel 73 pcre *pcre_compile(const char *pattern, int options,
1077     const char **errptr, int *erroffset,
1078     const unsigned char *tableptr);
1079 nigel 63
1080 nigel 77 pcre *pcre_compile2(const char *pattern, int options,
1081     int *errorcodeptr,
1082     const char **errptr, int *erroffset,
1083     const unsigned char *tableptr);
1084 nigel 41
1085 nigel 77 Either of the functions pcre_compile() or pcre_compile2() can be called
1086     to compile a pattern into an internal form. The only difference between
1087 ph10 231 the two interfaces is that pcre_compile2() has an additional argument,
1088 nigel 77 errorcodeptr, via which a numerical error code can be returned.
1089    
1090     The pattern is a C string terminated by a binary zero, and is passed in
1091 ph10 231 the pattern argument. A pointer to a single block of memory that is
1092     obtained via pcre_malloc is returned. This contains the compiled code
1093 nigel 77 and related data. The pcre type is defined for the returned block; this
1094     is a typedef for a structure whose contents are not externally defined.
1095 nigel 91 It is up to the caller to free the memory (via pcre_free) when it is no
1096     longer required.
1097 nigel 77
1098 ph10 231 Although the compiled code of a PCRE regex is relocatable, that is, it
1099 nigel 73 does not depend on memory location, the complete pcre data block is not
1100 ph10 231 fully relocatable, because it may contain a copy of the tableptr argu-
1101 nigel 75 ment, which is an address (see below).
1102 nigel 41
1103 nigel 93 The options argument contains various bit settings that affect the com-
1104 ph10 231 pilation. It should be zero if no options are required. The available
1105     options are described below. Some of them, in particular, those that
1106     are compatible with Perl, can also be set and unset from within the
1107     pattern (see the detailed description in the pcrepattern documenta-
1108     tion). For these options, the contents of the options argument speci-
1109     fies their initial settings at the start of compilation and execution.
1110     The PCRE_ANCHORED and PCRE_NEWLINE_xxx options can be set at the time
1111 nigel 91 of matching as well as at compile time.
1112 nigel 41
1113 nigel 73 If errptr is NULL, pcre_compile() returns NULL immediately. Otherwise,
1114 ph10 231 if compilation of a pattern fails, pcre_compile() returns NULL, and
1115 nigel 73 sets the variable pointed to by errptr to point to a textual error mes-
1116 nigel 87 sage. This is a static string that is part of the library. You must not
1117     try to free it. The offset from the start of the pattern to the charac-
1118     ter where the error was discovered is placed in the variable pointed to
1119 ph10 231 by erroffset, which must not be NULL. If it is, an immediate error is
1120 nigel 73 given.
1121 nigel 53
1122 ph10 231 If pcre_compile2() is used instead of pcre_compile(), and the error-
1123     codeptr argument is not NULL, a non-zero error code number is returned
1124     via this argument in the event of an error. This is in addition to the
1125 nigel 77 textual error message. Error codes and messages are listed below.
1126    
1127 ph10 231 If the final argument, tableptr, is NULL, PCRE uses a default set of
1128     character tables that are built when PCRE is compiled, using the
1129     default C locale. Otherwise, tableptr must be an address that is the
1130     result of a call to pcre_maketables(). This value is stored with the
1131     compiled pattern, and used again by pcre_exec(), unless another table
1132 nigel 75 pointer is passed to it. For more discussion, see the section on locale
1133     support below.
1134 nigel 53
1135 ph10 231 This code fragment shows a typical straightforward call to pcre_com-
1136 nigel 73 pile():
1137 nigel 41
1138 nigel 73 pcre *re;
1139     const char *error;
1140     int erroffset;
1141     re = pcre_compile(
1142     "^A.*Z", /* the pattern */
1143     0, /* default options */
1144     &error, /* for error message */
1145     &erroffset, /* for error offset */
1146     NULL); /* use default character tables */
1147 nigel 41
1148 ph10 231 The following names for option bits are defined in the pcre.h header
1149 nigel 75 file:
1150 nigel 41
1151 nigel 73 PCRE_ANCHORED
1152 nigel 41
1153 nigel 73 If this bit is set, the pattern is forced to be "anchored", that is, it
1154 ph10 231 is constrained to match only at the first matching point in the string
1155     that is being searched (the "subject string"). This effect can also be
1156     achieved by appropriate constructs in the pattern itself, which is the
1157 nigel 73 only way to do it in Perl.
1158 nigel 41
1159 nigel 75 PCRE_AUTO_CALLOUT
1160    
1161     If this bit is set, pcre_compile() automatically inserts callout items,
1162 ph10 231 all with number 255, before each pattern item. For discussion of the
1163 nigel 75 callout facility, see the pcrecallout documentation.
1164    
1165 ph10 231 PCRE_BSR_ANYCRLF
1166     PCRE_BSR_UNICODE
1167    
1168     These options (which are mutually exclusive) control what the \R escape
1169     sequence matches. The choice is either to match only CR, LF, or CRLF,
1170     or to match any Unicode newline sequence. The default is specified when
1171     PCRE is built. It can be overridden from within the pattern, or by set-
1172     ting an option when a compiled pattern is matched.
1173    
1174 nigel 73 PCRE_CASELESS
1175 nigel 41
1176 ph10 231 If this bit is set, letters in the pattern match both upper and lower
1177     case letters. It is equivalent to Perl's /i option, and it can be
1178     changed within a pattern by a (?i) option setting. In UTF-8 mode, PCRE
1179     always understands the concept of case for characters whose values are
1180     less than 128, so caseless matching is always possible. For characters
1181     with higher values, the concept of case is supported if PCRE is com-
1182     piled with Unicode property support, but not otherwise. If you want to
1183     use caseless matching for characters 128 and above, you must ensure
1184     that PCRE is compiled with Unicode property support as well as with
1185 nigel 77 UTF-8 support.
1186 nigel 41
1187 nigel 73 PCRE_DOLLAR_ENDONLY
1188 nigel 41
1189 ph10 231 If this bit is set, a dollar metacharacter in the pattern matches only
1190     at the end of the subject string. Without this option, a dollar also
1191     matches immediately before a newline at the end of the string (but not
1192     before any other newlines). The PCRE_DOLLAR_ENDONLY option is ignored
1193     if PCRE_MULTILINE is set. There is no equivalent to this option in
1194 nigel 91 Perl, and no way to set it within a pattern.
1195 nigel 41
1196 nigel 73 PCRE_DOTALL
1197 nigel 41
1198 nigel 73 If this bit is set, a dot metacharater in the pattern matches all char-
1199 ph10 231 acters, including those that indicate newline. Without it, a dot does
1200     not match when the current position is at a newline. This option is
1201     equivalent to Perl's /s option, and it can be changed within a pattern
1202     by a (?s) option setting. A negative class such as [^a] always matches
1203 nigel 93 newline characters, independent of the setting of this option.
1204 nigel 63
1205 nigel 91 PCRE_DUPNAMES
1206    
1207 ph10 231 If this bit is set, names used to identify capturing subpatterns need
1208 nigel 91 not be unique. This can be helpful for certain types of pattern when it
1209 ph10 231 is known that only one instance of the named subpattern can ever be
1210     matched. There are more details of named subpatterns below; see also
1211 nigel 91 the pcrepattern documentation.
1212    
1213 nigel 73 PCRE_EXTENDED
1214 nigel 41
1215 ph10 231 If this bit is set, whitespace data characters in the pattern are
1216 nigel 77 totally ignored except when escaped or inside a character class. White-
1217     space does not include the VT character (code 11). In addition, charac-
1218     ters between an unescaped # outside a character class and the next new-
1219 ph10 231 line, inclusive, are also ignored. This is equivalent to Perl's /x
1220     option, and it can be changed within a pattern by a (?x) option set-
1221 nigel 91 ting.
1222 nigel 41
1223 ph10 231 This option makes it possible to include comments inside complicated
1224     patterns. Note, however, that this applies only to data characters.
1225     Whitespace characters may never appear within special character
1226     sequences in a pattern, for example within the sequence (?( which
1227 nigel 73 introduces a conditional subpattern.
1228 nigel 41
1229 nigel 73 PCRE_EXTRA
1230 nigel 41
1231 ph10 231 This option was invented in order to turn on additional functionality
1232     of PCRE that is incompatible with Perl, but it is currently of very
1233     little use. When set, any backslash in a pattern that is followed by a
1234     letter that has no special meaning causes an error, thus reserving
1235     these combinations for future expansion. By default, as in Perl, a
1236     backslash followed by a letter with no special meaning is treated as a
1237     literal. (Perl can, however, be persuaded to give a warning for this.)
1238     There are at present no other features controlled by this option. It
1239 nigel 91 can also be set by a (?X) option setting within a pattern.
1240 nigel 41
1241 nigel 77 PCRE_FIRSTLINE
1242    
1243 ph10 231 If this option is set, an unanchored pattern is required to match
1244     before or at the first newline in the subject string, though the
1245 nigel 91 matched text may continue over the newline.
1246 nigel 77
1247 nigel 73 PCRE_MULTILINE
1248 nigel 41
1249 ph10 231 By default, PCRE treats the subject string as consisting of a single
1250     line of characters (even if it actually contains newlines). The "start
1251     of line" metacharacter (^) matches only at the start of the string,
1252     while the "end of line" metacharacter ($) matches only at the end of
1253 nigel 75 the string, or before a terminating newline (unless PCRE_DOLLAR_ENDONLY
1254     is set). This is the same as Perl.
1255 nigel 63
1256 ph10 231 When PCRE_MULTILINE it is set, the "start of line" and "end of line"
1257     constructs match immediately following or immediately before internal
1258     newlines in the subject string, respectively, as well as at the very
1259     start and end. This is equivalent to Perl's /m option, and it can be
1260 nigel 91 changed within a pattern by a (?m) option setting. If there are no new-
1261 ph10 231 lines in a subject string, or no occurrences of ^ or $ in a pattern,
1262 nigel 73 setting PCRE_MULTILINE has no effect.
1263 nigel 63
1264 nigel 91 PCRE_NEWLINE_CR
1265     PCRE_NEWLINE_LF
1266     PCRE_NEWLINE_CRLF
1267 ph10 150 PCRE_NEWLINE_ANYCRLF
1268 nigel 93 PCRE_NEWLINE_ANY
1269 nigel 91
1270 ph10 231 These options override the default newline definition that was chosen
1271     when PCRE was built. Setting the first or the second specifies that a
1272     newline is indicated by a single character (CR or LF, respectively).
1273     Setting PCRE_NEWLINE_CRLF specifies that a newline is indicated by the
1274     two-character CRLF sequence. Setting PCRE_NEWLINE_ANYCRLF specifies
1275 ph10 150 that any of the three preceding sequences should be recognized. Setting
1276 ph10 231 PCRE_NEWLINE_ANY specifies that any Unicode newline sequence should be
1277 ph10 150 recognized. The Unicode newline sequences are the three just mentioned,
1278 ph10 231 plus the single characters VT (vertical tab, U+000B), FF (formfeed,
1279     U+000C), NEL (next line, U+0085), LS (line separator, U+2028), and PS
1280     (paragraph separator, U+2029). The last two are recognized only in
1281 ph10 150 UTF-8 mode.
1282 nigel 91
1283 ph10 231 The newline setting in the options word uses three bits that are
1284 ph10 150 treated as a number, giving eight possibilities. Currently only six are
1285 ph10 231 used (default plus the five values above). This means that if you set
1286     more than one newline option, the combination may or may not be sensi-
1287 ph10 150 ble. For example, PCRE_NEWLINE_CR with PCRE_NEWLINE_LF is equivalent to
1288 ph10 231 PCRE_NEWLINE_CRLF, but other combinations may yield unused numbers and
1289 ph10 150 cause an error.
1290 nigel 91
1291 ph10 231 The only time that a line break is specially recognized when compiling
1292     a pattern is if PCRE_EXTENDED is set, and an unescaped # outside a
1293     character class is encountered. This indicates a comment that lasts
1294     until after the next line break sequence. In other circumstances, line
1295     break sequences are treated as literal data, except that in
1296 nigel 93 PCRE_EXTENDED mode, both CR and LF are treated as whitespace characters
1297     and are therefore ignored.
1298    
1299     The newline option that is set at compile time becomes the default that
1300 ph10 231 is used for pcre_exec() and pcre_dfa_exec(), but it can be overridden.
1301 nigel 93
1302 nigel 73 PCRE_NO_AUTO_CAPTURE
1303 nigel 41
1304 nigel 73 If this option is set, it disables the use of numbered capturing paren-
1305 ph10 231 theses in the pattern. Any opening parenthesis that is not followed by
1306     ? behaves as if it were followed by ?: but named parentheses can still
1307     be used for capturing (and they acquire numbers in the usual way).
1308 nigel 73 There is no equivalent of this option in Perl.
1309 nigel 41
1310 nigel 73 PCRE_UNGREEDY
1311 nigel 41
1312 ph10 231 This option inverts the "greediness" of the quantifiers so that they
1313     are not greedy by default, but become greedy if followed by "?". It is
1314     not compatible with Perl. It can also be set by a (?U) option setting
1315 nigel 73 within the pattern.
1316 nigel 41
1317 nigel 73 PCRE_UTF8
1318 nigel 49
1319 ph10 231 This option causes PCRE to regard both the pattern and the subject as
1320     strings of UTF-8 characters instead of single-byte character strings.
1321     However, it is available only when PCRE is built to include UTF-8 sup-
1322     port. If not, the use of this option provokes an error. Details of how
1323     this option changes the behaviour of PCRE are given in the section on
1324 nigel 75 UTF-8 support in the main pcre page.
1325 nigel 71
1326 nigel 73 PCRE_NO_UTF8_CHECK
1327 nigel 71
1328 nigel 73 When PCRE_UTF8 is set, the validity of the pattern as a UTF-8 string is
1329 ph10 231 automatically checked. There is a discussion about the validity of
1330     UTF-8 strings in the main pcre page. If an invalid UTF-8 sequence of
1331     bytes is found, pcre_compile() returns an error. If you already know
1332 ph10 211 that your pattern is valid, and you want to skip this check for perfor-
1333 ph10 231 mance reasons, you can set the PCRE_NO_UTF8_CHECK option. When it is
1334     set, the effect of passing an invalid UTF-8 string as a pattern is
1335     undefined. It may cause your program to crash. Note that this option
1336     can also be passed to pcre_exec() and pcre_dfa_exec(), to suppress the
1337 ph10 211 UTF-8 validity checking of subject strings.
1338 nigel 71
1339 nigel 73
1340 nigel 77 COMPILATION ERROR CODES
1341    
1342 ph10 231 The following table lists the error codes than may be returned by
1343     pcre_compile2(), along with the error messages that may be returned by
1344     both compiling functions. As PCRE has developed, some error codes have
1345 nigel 93 fallen out of use. To avoid confusion, they have not been re-used.
1346 nigel 77
1347     0 no error
1348     1 \ at end of pattern
1349     2 \c at end of pattern
1350     3 unrecognized character follows \
1351     4 numbers out of order in {} quantifier
1352     5 number too big in {} quantifier
1353     6 missing terminating ] for character class
1354     7 invalid escape sequence in character class
1355     8 range out of order in character class
1356     9 nothing to repeat
1357 nigel 93 10 [this code is not in use]
1358 nigel 77 11 internal error: unexpected repeat
1359 ph10 292 12 unrecognized character after (? or (?-
1360 nigel 77 13 POSIX named classes are supported only within a class
1361     14 missing )
1362     15 reference to non-existent subpattern
1363     16 erroffset passed as NULL
1364     17 unknown option bit(s) set
1365     18 missing ) after comment
1366 nigel 93 19 [this code is not in use]
1367 ph10 292 20 regular expression is too large
1368 nigel 77 21 failed to get memory
1369     22 unmatched parentheses
1370     23 internal error: code overflow
1371     24 unrecognized character after (?<
1372     25 lookbehind assertion is not fixed length
1373 nigel 91 26 malformed number or name after (?(
1374 nigel 77 27 conditional group contains more than two branches
1375     28 assertion expected after (?(
1376 ph10 182 29 (?R or (?[+-]digits must be followed by )
1377 nigel 77 30 unknown POSIX class name
1378     31 POSIX collating elements are not supported
1379     32 this version of PCRE is not compiled with PCRE_UTF8 support
1380 nigel 93 33 [this code is not in use]
1381 nigel 77 34 character value in \x{...} sequence is too large
1382     35 invalid condition (?(0)
1383     36 \C not allowed in lookbehind assertion
1384     37 PCRE does not support \L, \l, \N, \U, or \u
1385     38 number after (?C is > 255
1386     39 closing ) for (?C expected
1387     40 recursive call could loop indefinitely
1388     41 unrecognized character after (?P
1389 nigel 93 42 syntax error in subpattern name (missing terminator)
1390 nigel 91 43 two named subpatterns have the same name
1391 nigel 77 44 invalid UTF-8 string
1392     45 support for \P, \p, and \X has not been compiled
1393     46 malformed \P or \p sequence
1394     47 unknown property name after \P or \p
1395 nigel 91 48 subpattern name is too long (maximum 32 characters)
1396 ph10 292 49 too many named subpatterns (maximum 10000)
1397 ph10 202 50 [this code is not in use]
1398 nigel 91 51 octal value is greater than \377 (not in UTF-8 mode)
1399 nigel 93 52 internal error: overran compiling workspace
1400 ph10 231 53 internal error: previously-checked referenced subpattern not
1401 nigel 93 found
1402     54 DEFINE group contains more than one branch
1403     55 repeating a DEFINE group is not allowed
1404 ph10 231 56 inconsistent NEWLINE options
1405 ph10 182 57 \g is not followed by a braced name or an optionally braced
1406     non-zero number
1407     58 (?+ or (?- or (?(+ or (?(- must be followed by a non-zero number
1408 ph10 292 59 (*VERB) with an argument is not supported
1409     60 (*VERB) not recognized
1410     61 number is too big
1411     62 subpattern name expected
1412     63 digit expected after (?+
1413 nigel 77
1414 ph10 292 The numbers 32 and 10000 in errors 48 and 49 are defaults; different
1415     values may be used if the limits were changed when PCRE was built.
1416 nigel 77
1417 ph10 292
1418 nigel 63 STUDYING A PATTERN
1419 nigel 49
1420 nigel 77 pcre_extra *pcre_study(const pcre *code, int options
1421 nigel 73 const char **errptr);
1422 nigel 63
1423 ph10 292 If a compiled pattern is going to be used several times, it is worth
1424 nigel 75 spending more time analyzing it in order to speed up the time taken for
1425 ph10 292 matching. The function pcre_study() takes a pointer to a compiled pat-
1426 nigel 75 tern as its first argument. If studying the pattern produces additional
1427 ph10 292 information that will help speed up matching, pcre_study() returns a
1428     pointer to a pcre_extra block, in which the study_data field points to
1429 nigel 75 the results of the study.
1430 nigel 41
1431 nigel 75 The returned value from pcre_study() can be passed directly to
1432 ph10 292 pcre_exec(). However, a pcre_extra block also contains other fields
1433     that can be set by the caller before the block is passed; these are
1434 nigel 75 described below in the section on matching a pattern.
1435 nigel 63
1436 ph10 292 If studying the pattern does not produce any additional information
1437 nigel 75 pcre_study() returns NULL. In that circumstance, if the calling program
1438 ph10 292 wants to pass any of the other fields to pcre_exec(), it must set up
1439 nigel 75 its own pcre_extra block.
1440 nigel 41
1441 ph10 292 The second argument of pcre_study() contains option bits. At present,
1442 nigel 75 no options are defined, and this argument should always be zero.
1443    
1444 ph10 292 The third argument for pcre_study() is a pointer for an error message.
1445     If studying succeeds (even if no data is returned), the variable it
1446     points to is set to NULL. Otherwise it is set to point to a textual
1447 nigel 87 error message. This is a static string that is part of the library. You
1448 ph10 292 must not try to free it. You should test the error pointer for NULL
1449 nigel 87 after calling pcre_study(), to be sure that it has run successfully.
1450 nigel 41
1451 nigel 73 This is a typical call to pcre_study():
1452 nigel 53
1453 nigel 73 pcre_extra *pe;
1454     pe = pcre_study(
1455     re, /* result of pcre_compile() */
1456     0, /* no options exist */
1457     &error); /* set to NULL or points to a message */
1458 nigel 53
1459 nigel 73 At present, studying a pattern is useful only for non-anchored patterns
1460 ph10 292 that do not have a single fixed starting character. A bitmap of possi-
1461 nigel 75 ble starting bytes is created.
1462 nigel 41
1463    
1464 nigel 63 LOCALE SUPPORT
1465 nigel 41
1466 ph10 292 PCRE handles caseless matching, and determines whether characters are
1467     letters, digits, or whatever, by reference to a set of tables, indexed
1468     by character value. When running in UTF-8 mode, this applies only to
1469     characters with codes less than 128. Higher-valued codes never match
1470     escapes such as \w or \d, but can be tested with \p if PCRE is built
1471     with Unicode character property support. The use of locales with Uni-
1472     code is discouraged. If you are handling characters with codes greater
1473     than 128, you should either use UTF-8 and Unicode, or use locales, but
1474 ph10 142 not try to mix the two.
1475 nigel 41
1476 ph10 292 PCRE contains an internal set of tables that are used when the final
1477     argument of pcre_compile() is NULL. These are sufficient for many
1478 ph10 142 applications. Normally, the internal tables recognize only ASCII char-
1479     acters. However, when PCRE is built, it is possible to cause the inter-
1480     nal tables to be rebuilt in the default "C" locale of the local system,
1481     which may cause them to be different.
1482 nigel 41
1483 ph10 292 The internal tables can always be overridden by tables supplied by the
1484 ph10 142 application that calls PCRE. These may be created in a different locale
1485 ph10 292 from the default. As more and more applications change to using Uni-
1486 ph10 142 code, the need for this locale support is expected to die away.
1487    
1488 ph10 292 External tables are built by calling the pcre_maketables() function,
1489     which has no arguments, in the relevant locale. The result can then be
1490     passed to pcre_compile() or pcre_exec() as often as necessary. For
1491     example, to build and use tables that are appropriate for the French
1492     locale (where accented characters with values greater than 128 are
1493 nigel 75 treated as letters), the following code could be used:
1494    
1495     setlocale(LC_CTYPE, "fr_FR");
1496 nigel 73 tables = pcre_maketables();
1497     re = pcre_compile(..., tables);
1498 nigel 41
1499 ph10 292 The locale name "fr_FR" is used on Linux and other Unix-like systems;
1500 ph10 142 if you are using Windows, the name for the French locale is "french".
1501    
1502 ph10 292 When pcre_maketables() runs, the tables are built in memory that is
1503     obtained via pcre_malloc. It is the caller's responsibility to ensure
1504     that the memory containing the tables remains available for as long as
1505 nigel 75 it is needed.
1506 nigel 41
1507 nigel 75 The pointer that is passed to pcre_compile() is saved with the compiled
1508 ph10 292 pattern, and the same tables are used via this pointer by pcre_study()
1509 nigel 75 and normally also by pcre_exec(). Thus, by default, for any single pat-
1510     tern, compilation, studying and matching all happen in the same locale,
1511     but different patterns can be compiled in different locales.
1512 nigel 41
1513 ph10 292 It is possible to pass a table pointer or NULL (indicating the use of
1514     the internal tables) to pcre_exec(). Although not intended for this
1515     purpose, this facility could be used to match a pattern in a different
1516 nigel 75 locale from the one in which it was compiled. Passing table pointers at
1517     run time is discussed below in the section on matching a pattern.
1518    
1519    
1520 nigel 63 INFORMATION ABOUT A PATTERN
1521 nigel 41
1522 nigel 73 int pcre_fullinfo(const pcre *code, const pcre_extra *extra,
1523     int what, void *where);
1524 nigel 63
1525 ph10 292 The pcre_fullinfo() function returns information about a compiled pat-
1526 nigel 73 tern. It replaces the obsolete pcre_info() function, which is neverthe-
1527     less retained for backwards compability (and is documented below).
1528 nigel 43
1529 ph10 292 The first argument for pcre_fullinfo() is a pointer to the compiled
1530     pattern. The second argument is the result of pcre_study(), or NULL if
1531     the pattern was not studied. The third argument specifies which piece
1532     of information is required, and the fourth argument is a pointer to a
1533     variable to receive the data. The yield of the function is zero for
1534 nigel 73 success, or one of the following negative numbers:
1535 nigel 41
1536 nigel 73 PCRE_ERROR_NULL the argument code was NULL
1537     the argument where was NULL
1538     PCRE_ERROR_BADMAGIC the "magic number" was not found
1539     PCRE_ERROR_BADOPTION the value of what was invalid
1540 nigel 53
1541 ph10 292 The "magic number" is placed at the start of each compiled pattern as
1542     an simple check against passing an arbitrary memory pointer. Here is a
1543     typical call of pcre_fullinfo(), to obtain the length of the compiled
1544 nigel 75 pattern:
1545 nigel 53
1546 nigel 73 int rc;
1547 nigel 91 size_t length;
1548 nigel 73 rc = pcre_fullinfo(
1549     re, /* result of pcre_compile() */
1550     pe, /* result of pcre_study(), or NULL */
1551     PCRE_INFO_SIZE, /* what is required */
1552     &length); /* where to put the data */
1553 nigel 43
1554 ph10 292 The possible values for the third argument are defined in pcre.h, and
1555 nigel 73 are as follows:
1556 nigel 43
1557 nigel 73 PCRE_INFO_BACKREFMAX
1558 nigel 41
1559 ph10 292 Return the number of the highest back reference in the pattern. The
1560     fourth argument should point to an int variable. Zero is returned if
1561 nigel 73 there are no back references.
1562 nigel 43
1563 nigel 73 PCRE_INFO_CAPTURECOUNT
1564 nigel 43
1565 ph10 292 Return the number of capturing subpatterns in the pattern. The fourth
1566 nigel 73 argument should point to an int variable.
1567 nigel 43
1568 nigel 77 PCRE_INFO_DEFAULT_TABLES
1569 nigel 75
1570 ph10 292 Return a pointer to the internal default character tables within PCRE.
1571     The fourth argument should point to an unsigned char * variable. This
1572 nigel 75 information call is provided for internal use by the pcre_study() func-
1573 ph10 292 tion. External callers can cause PCRE to use its internal tables by
1574 nigel 75 passing a NULL table pointer.
1575    
1576 nigel 73 PCRE_INFO_FIRSTBYTE
1577 nigel 43
1578 ph10 292 Return information about the first byte of any matched string, for a
1579     non-anchored pattern. The fourth argument should point to an int vari-
1580     able. (This option used to be called PCRE_INFO_FIRSTCHAR; the old name
1581 nigel 91 is still recognized for backwards compatibility.)
1582 nigel 41
1583 ph10 292 If there is a fixed first byte, for example, from a pattern such as
1584 nigel 93 (cat|cow|coyote), its value is returned. Otherwise, if either
1585 nigel 41
1586 ph10 292 (a) the pattern was compiled with the PCRE_MULTILINE option, and every
1587 nigel 73 branch starts with "^", or
1588 nigel 43
1589 nigel 73 (b) every branch of the pattern starts with ".*" and PCRE_DOTALL is not
1590     set (if it were set, the pattern would be anchored),
1591 nigel 41
1592 ph10 292 -1 is returned, indicating that the pattern matches only at the start
1593     of a subject string or after any newline within the string. Otherwise
1594 nigel 73 -2 is returned. For anchored patterns, -2 is returned.
1595 nigel 41
1596 nigel 73 PCRE_INFO_FIRSTTABLE
1597 nigel 41
1598 ph10 292 If the pattern was studied, and this resulted in the construction of a
1599 nigel 73 256-bit table indicating a fixed set of bytes for the first byte in any
1600 ph10 292 matching string, a pointer to the table is returned. Otherwise NULL is
1601     returned. The fourth argument should point to an unsigned char * vari-
1602 nigel 73 able.
1603 nigel 43
1604 ph10 227 PCRE_INFO_HASCRORLF
1605    
1606 ph10 292 Return 1 if the pattern contains any explicit matches for CR or LF
1607     characters, otherwise 0. The fourth argument should point to an int
1608     variable. An explicit match is either a literal CR or LF character, or
1609 ph10 243 \r or \n.
1610 ph10 227
1611 ph10 172 PCRE_INFO_JCHANGED
1612    
1613 ph10 292 Return 1 if the (?J) or (?-J) option setting is used in the pattern,
1614     otherwise 0. The fourth argument should point to an int variable. (?J)
1615 ph10 286 and (?-J) set and unset the local PCRE_DUPNAMES option, respectively.
1616 ph10 172
1617 nigel 73 PCRE_INFO_LASTLITERAL
1618 nigel 43
1619 ph10 292 Return the value of the rightmost literal byte that must exist in any
1620     matched string, other than at its start, if such a byte has been
1621 nigel 73 recorded. The fourth argument should point to an int variable. If there
1622 ph10 292 is no such byte, -1 is returned. For anchored patterns, a last literal
1623     byte is recorded only if it follows something of variable length. For
1624 nigel 73 example, for the pattern /^a\d+z\d+/ the returned value is "z", but for
1625     /^a\dz\d/ the returned value is -1.
1626 nigel 63
1627 nigel 73 PCRE_INFO_NAMECOUNT
1628     PCRE_INFO_NAMEENTRYSIZE
1629     PCRE_INFO_NAMETABLE
1630 nigel 63
1631 ph10 292 PCRE supports the use of named as well as numbered capturing parenthe-
1632     ses. The names are just an additional way of identifying the parenthe-
1633 nigel 91 ses, which still acquire numbers. Several convenience functions such as
1634 ph10 292 pcre_get_named_substring() are provided for extracting captured sub-
1635     strings by name. It is also possible to extract the data directly, by
1636     first converting the name to a number in order to access the correct
1637 nigel 91 pointers in the output vector (described with pcre_exec() below). To do
1638 ph10 292 the conversion, you need to use the name-to-number map, which is
1639 nigel 91 described by these three values.
1640 nigel 63
1641 nigel 73 The map consists of a number of fixed-size entries. PCRE_INFO_NAMECOUNT
1642     gives the number of entries, and PCRE_INFO_NAMEENTRYSIZE gives the size
1643 ph10 292 of each entry; both of these return an int value. The entry size
1644     depends on the length of the longest name. PCRE_INFO_NAMETABLE returns
1645     a pointer to the first entry of the table (a pointer to char). The
1646 nigel 73 first two bytes of each entry are the number of the capturing parenthe-
1647 ph10 292 sis, most significant byte first. The rest of the entry is the corre-
1648     sponding name, zero terminated. The names are in alphabetical order.
1649 nigel 91 When PCRE_DUPNAMES is set, duplicate names are in order of their paren-
1650 ph10 292 theses numbers. For example, consider the following pattern (assume
1651     PCRE_EXTENDED is set, so white space - including newlines - is
1652 nigel 91 ignored):
1653 nigel 63
1654 nigel 93 (?<date> (?<year>(\d\d)?\d\d) -
1655     (?<month>\d\d) - (?<day>\d\d) )
1656 nigel 63
1657 ph10 292 There are four named subpatterns, so the table has four entries, and
1658     each entry in the table is eight bytes long. The table is as follows,
1659 nigel 75 with non-printing bytes shows in hexadecimal, and undefined bytes shown
1660     as ??:
1661 nigel 63
1662 nigel 73 00 01 d a t e 00 ??
1663     00 05 d a y 00 ?? ??
1664     00 04 m o n t h 00
1665     00 02 y e a r 00 ??
1666 nigel 63
1667 ph10 292 When writing code to extract data from named subpatterns using the
1668     name-to-number map, remember that the length of the entries is likely
1669 nigel 91 to be different for each compiled pattern.
1670 nigel 63
1671 ph10 172 PCRE_INFO_OKPARTIAL
1672    
1673 ph10 292 Return 1 if the pattern can be used for partial matching, otherwise 0.
1674     The fourth argument should point to an int variable. The pcrepartial
1675     documentation lists the restrictions that apply to patterns when par-
1676 ph10 172 tial matching is used.
1677    
1678 nigel 73 PCRE_INFO_OPTIONS
1679 nigel 63
1680 ph10 292 Return a copy of the options with which the pattern was compiled. The
1681     fourth argument should point to an unsigned long int variable. These
1682 nigel 73 option bits are those specified in the call to pcre_compile(), modified
1683 ph10 197 by any top-level option settings at the start of the pattern itself. In
1684 ph10 292 other words, they are the options that will be in force when matching
1685     starts. For example, if the pattern /(?im)abc(?-i)d/ is compiled with
1686     the PCRE_EXTENDED option, the result is PCRE_CASELESS, PCRE_MULTILINE,
1687 ph10 197 and PCRE_EXTENDED.
1688 nigel 63
1689 ph10 292 A pattern is automatically anchored by PCRE if all of its top-level
1690 nigel 73 alternatives begin with one of the following:
1691 nigel 63
1692 nigel 73 ^ unless PCRE_MULTILINE is set
1693     \A always
1694     \G always
1695     .* if PCRE_DOTALL is set and there are no back
1696     references to the subpattern in which .* appears
1697 nigel 63
1698 nigel 73 For such patterns, the PCRE_ANCHORED bit is set in the options returned
1699     by pcre_fullinfo().
1700 nigel 63
1701 nigel 73 PCRE_INFO_SIZE
1702 nigel 63
1703 ph10 292 Return the size of the compiled pattern, that is, the value that was
1704 nigel 73 passed as the argument to pcre_malloc() when PCRE was getting memory in
1705     which to place the compiled data. The fourth argument should point to a
1706     size_t variable.
1707 nigel 63
1708 nigel 73 PCRE_INFO_STUDYSIZE
1709 nigel 63
1710 nigel 75 Return the size of the data block pointed to by the study_data field in
1711 ph10 292 a pcre_extra block. That is, it is the value that was passed to
1712 nigel 73 pcre_malloc() when PCRE was getting memory into which to place the data
1713 ph10 292 created by pcre_study(). The fourth argument should point to a size_t
1714 nigel 73 variable.
1715 nigel 63
1716 nigel 73
1717 nigel 63 OBSOLETE INFO FUNCTION
1718    
1719 nigel 73 int pcre_info(const pcre *code, int *optptr, int *firstcharptr);
1720 nigel 63
1721 ph10 292 The pcre_info() function is now obsolete because its interface is too
1722     restrictive to return all the available data about a compiled pattern.
1723     New programs should use pcre_fullinfo() instead. The yield of
1724     pcre_info() is the number of capturing subpatterns, or one of the fol-
1725 nigel 73 lowing negative numbers:
1726 nigel 43
1727 nigel 73 PCRE_ERROR_NULL the argument code was NULL
1728     PCRE_ERROR_BADMAGIC the "magic number" was not found
1729 nigel 43
1730 ph10 292 If the optptr argument is not NULL, a copy of the options with which
1731     the pattern was compiled is placed in the integer it points to (see
1732 nigel 73 PCRE_INFO_OPTIONS above).
1733 nigel 43
1734 ph10 292 If the pattern is not anchored and the firstcharptr argument is not
1735     NULL, it is used to pass back information about the first character of
1736 nigel 73 any matched string (see PCRE_INFO_FIRSTBYTE above).
1737 nigel 43
1738    
1739 nigel 77 REFERENCE COUNTS
1740 nigel 53
1741 nigel 77 int pcre_refcount(pcre *code, int adjust);
1742    
1743 ph10 292 The pcre_refcount() function is used to maintain a reference count in
1744 nigel 77 the data block that contains a compiled pattern. It is provided for the
1745 ph10 292 benefit of applications that operate in an object-oriented manner,
1746 nigel 77 where different parts of the application may be using the same compiled
1747     pattern, but you want to free the block when they are all done.
1748    
1749     When a pattern is compiled, the reference count field is initialized to
1750 ph10 292 zero. It is changed only by calling this function, whose action is to
1751     add the adjust value (which may be positive or negative) to it. The
1752 nigel 77 yield of the function is the new value. However, the value of the count
1753 ph10 292 is constrained to lie between 0 and 65535, inclusive. If the new value
1754 nigel 77 is outside these limits, it is forced to the appropriate limit value.
1755    
1756 ph10 292 Except when it is zero, the reference count is not correctly preserved
1757     if a pattern is compiled on one host and then transferred to a host
1758 nigel 77 whose byte-order is different. (This seems a highly unlikely scenario.)
1759    
1760    
1761     MATCHING A PATTERN: THE TRADITIONAL FUNCTION
1762    
1763 nigel 73 int pcre_exec(const pcre *code, const pcre_extra *extra,
1764     const char *subject, int length, int startoffset,
1765     int options, int *ovector, int ovecsize);
1766 nigel 53
1767 ph10 292 The function pcre_exec() is called to match a subject string against a
1768     compiled pattern, which is passed in the code argument. If the pattern
1769 nigel 75 has been studied, the result of the study should be passed in the extra
1770 ph10 292 argument. This function is the main matching facility of the library,
1771 nigel 77 and it operates in a Perl-like manner. For specialist use there is also
1772 ph10 292 an alternative matching function, which is described below in the sec-
1773 nigel 77 tion about the pcre_dfa_exec() function.
1774 nigel 41
1775 ph10 292 In most applications, the pattern will have been compiled (and option-
1776     ally studied) in the same process that calls pcre_exec(). However, it
1777 nigel 75 is possible to save compiled patterns and study data, and then use them
1778 ph10 292 later in different processes, possibly even on different hosts. For a
1779 nigel 75 discussion about this, see the pcreprecompile documentation.
1780    
1781 nigel 73 Here is an example of a simple call to pcre_exec():
1782 nigel 53
1783 nigel 73 int rc;
1784     int ovector[30];
1785     rc = pcre_exec(
1786     re, /* result of pcre_compile() */
1787     NULL, /* we didn't study the pattern */
1788     "some string", /* the subject string */
1789     11, /* the length of the subject string */
1790     0, /* start at offset 0 in the subject */
1791     0, /* default options */
1792 nigel 75 ovector, /* vector of integers for substring information */
1793 nigel 77 30); /* number of elements (NOT size in bytes) */
1794 nigel 53
1795 nigel 75 Extra data for pcre_exec()
1796 nigel 63
1797 ph10 292 If the extra argument is not NULL, it must point to a pcre_extra data
1798     block. The pcre_study() function returns such a block (when it doesn't
1799     return NULL), but you can also create one for yourself, and pass addi-
1800     tional information in it. The pcre_extra block contains the following
1801 nigel 87 fields (not necessarily in this order):
1802 nigel 75
1803 nigel 73 unsigned long int flags;
1804     void *study_data;
1805     unsigned long int match_limit;
1806 nigel 87 unsigned long int match_limit_recursion;
1807 nigel 73 void *callout_data;
1808 nigel 75 const unsigned char *tables;
1809 nigel 63
1810 ph10 292 The flags field is a bitmap that specifies which of the other fields
1811 nigel 73 are set. The flag bits are:
1812 nigel 63
1813 nigel 73 PCRE_EXTRA_STUDY_DATA
1814     PCRE_EXTRA_MATCH_LIMIT
1815 nigel 87 PCRE_EXTRA_MATCH_LIMIT_RECURSION
1816 nigel 73 PCRE_EXTRA_CALLOUT_DATA
1817 nigel 75 PCRE_EXTRA_TABLES
1818 nigel 63
1819 ph10 292 Other flag bits should be set to zero. The study_data field is set in
1820     the pcre_extra block that is returned by pcre_study(), together with
1821 nigel 75 the appropriate flag bit. You should not set this yourself, but you may
1822 ph10 292 add to the block by setting the other fields and their corresponding
1823 nigel 75 flag bits.
1824 nigel 63
1825 nigel 73 The match_limit field provides a means of preventing PCRE from using up
1826 ph10 292 a vast amount of resources when running patterns that are not going to
1827     match, but which have a very large number of possibilities in their
1828     search trees. The classic example is the use of nested unlimited
1829 nigel 75 repeats.
1830 nigel 63
1831 ph10 292 Internally, PCRE uses a function called match() which it calls repeat-
1832     edly (sometimes recursively). The limit set by match_limit is imposed
1833     on the number of times this function is called during a match, which
1834     has the effect of limiting the amount of backtracking that can take
1835 nigel 87 place. For patterns that are not anchored, the count restarts from zero
1836     for each position in the subject string.
1837 nigel 75
1838 ph10 292 The default value for the limit can be set when PCRE is built; the
1839     default default is 10 million, which handles all but the most extreme
1840     cases. You can override the default by suppling pcre_exec() with a
1841     pcre_extra block in which match_limit is set, and
1842     PCRE_EXTRA_MATCH_LIMIT is set in the flags field. If the limit is
1843 nigel 73 exceeded, pcre_exec() returns PCRE_ERROR_MATCHLIMIT.
1844 nigel 63
1845 ph10 292 The match_limit_recursion field is similar to match_limit, but instead
1846 nigel 87 of limiting the total number of times that match() is called, it limits
1847 ph10 292 the depth of recursion. The recursion depth is a smaller number than
1848     the total number of calls, because not all calls to match() are recur-
1849 nigel 87 sive. This limit is of use only if it is set smaller than match_limit.
1850    
1851 ph10 292 Limiting the recursion depth limits the amount of stack that can be
1852 nigel 87 used, or, when PCRE has been compiled to use memory on the heap instead
1853     of the stack, the amount of heap memory that can be used.
1854    
1855 ph10 292 The default value for match_limit_recursion can be set when PCRE is
1856     built; the default default is the same value as the default for
1857     match_limit. You can override the default by suppling pcre_exec() with
1858     a pcre_extra block in which match_limit_recursion is set, and
1859     PCRE_EXTRA_MATCH_LIMIT_RECURSION is set in the flags field. If the
1860 nigel 87 limit is exceeded, pcre_exec() returns PCRE_ERROR_RECURSIONLIMIT.
1861    
1862 ph10 292 The pcre_callout field is used in conjunction with the "callout" fea-
1863 nigel 73 ture, which is described in the pcrecallout documentation.
1864 nigel 63
1865 ph10 292 The tables field is used to pass a character tables pointer to
1866     pcre_exec(); this overrides the value that is stored with the compiled
1867     pattern. A non-NULL value is stored with the compiled pattern only if
1868     custom tables were supplied to pcre_compile() via its tableptr argu-
1869 nigel 75 ment. If NULL is passed to pcre_exec() using this mechanism, it forces
1870 ph10 292 PCRE's internal tables to be used. This facility is helpful when re-
1871     using patterns that have been saved after compiling with an external
1872     set of tables, because the external tables might be at a different
1873     address when pcre_exec() is called. See the pcreprecompile documenta-
1874 nigel 75 tion for a discussion of saving compiled patterns for later use.
1875 nigel 41
1876 nigel 75 Option bits for pcre_exec()
1877 nigel 71
1878 ph10 292 The unused bits of the options argument for pcre_exec() must be zero.
1879     The only bits that may be set are PCRE_ANCHORED, PCRE_NEWLINE_xxx,
1880 nigel 91 PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK and
1881     PCRE_PARTIAL.
1882 nigel 41
1883 nigel 75 PCRE_ANCHORED
1884 nigel 41
1885 ph10 292 The PCRE_ANCHORED option limits pcre_exec() to matching at the first
1886     matching position. If a pattern was compiled with PCRE_ANCHORED, or
1887     turned out to be anchored by virtue of its contents, it cannot be made
1888 nigel 75 unachored at matching time.
1889    
1890 ph10 231 PCRE_BSR_ANYCRLF
1891     PCRE_BSR_UNICODE
1892    
1893     These options (which are mutually exclusive) control what the \R escape
1894 ph10 292 sequence matches. The choice is either to match only CR, LF, or CRLF,
1895     or to match any Unicode newline sequence. These options override the
1896 ph10 231 choice that was made or defaulted when the pattern was compiled.
1897    
1898 nigel 91 PCRE_NEWLINE_CR
1899     PCRE_NEWLINE_LF
1900     PCRE_NEWLINE_CRLF
1901 ph10 150 PCRE_NEWLINE_ANYCRLF
1902 nigel 93 PCRE_NEWLINE_ANY
1903 nigel 91
1904 ph10 292 These options override the newline definition that was chosen or
1905     defaulted when the pattern was compiled. For details, see the descrip-
1906     tion of pcre_compile() above. During matching, the newline choice
1907     affects the behaviour of the dot, circumflex, and dollar metacharac-
1908     ters. It may also alter the way the match position is advanced after a
1909 ph10 227 match failure for an unanchored pattern.
1910 nigel 91
1911 ph10 292 When PCRE_NEWLINE_CRLF, PCRE_NEWLINE_ANYCRLF, or PCRE_NEWLINE_ANY is
1912     set, and a match attempt for an unanchored pattern fails when the cur-
1913     rent position is at a CRLF sequence, and the pattern contains no
1914     explicit matches for CR or LF characters, the match position is
1915 ph10 227 advanced by two characters instead of one, in other words, to after the
1916     CRLF.
1917    
1918     The above rule is a compromise that makes the most common cases work as
1919 ph10 292 expected. For example, if the pattern is .+A (and the PCRE_DOTALL
1920 ph10 227 option is not set), it does not match the string "\r\nA" because, after
1921 ph10 292 failing at the start, it skips both the CR and the LF before retrying.
1922     However, the pattern [\r\n]A does match that string, because it con-
1923 ph10 227 tains an explicit CR or LF reference, and so advances only by one char-
1924 ph10 231 acter after the first failure.
1925 ph10 227
1926 ph10 231 An explicit match for CR of LF is either a literal appearance of one of
1927 ph10 292 those characters, or one of the \r or \n escape sequences. Implicit
1928     matches such as [^X] do not count, nor does \s (which includes CR and
1929 ph10 231 LF in the characters that it matches).
1930    
1931 ph10 292 Notwithstanding the above, anomalous effects may still occur when CRLF
1932 ph10 227 is a valid newline sequence and explicit \r or \n escapes appear in the
1933     pattern.
1934    
1935 nigel 73 PCRE_NOTBOL
1936 nigel 41
1937 nigel 75 This option specifies that first character of the subject string is not
1938 ph10 292 the beginning of a line, so the circumflex metacharacter should not
1939     match before it. Setting this without PCRE_MULTILINE (at compile time)
1940     causes circumflex never to match. This option affects only the behav-
1941 nigel 77 iour of the circumflex metacharacter. It does not affect \A.
1942 nigel 41
1943 nigel 73 PCRE_NOTEOL
1944 nigel 41
1945 nigel 75 This option specifies that the end of the subject string is not the end
1946 ph10 292 of a line, so the dollar metacharacter should not match it nor (except
1947     in multiline mode) a newline immediately before it. Setting this with-
1948 nigel 75 out PCRE_MULTILINE (at compile time) causes dollar never to match. This
1949 ph10 292 option affects only the behaviour of the dollar metacharacter. It does
1950 nigel 75 not affect \Z or \z.
1951 nigel 41
1952 nigel 73 PCRE_NOTEMPTY
1953 nigel 41
1954 nigel 73 An empty string is not considered to be a valid match if this option is
1955 ph10 292 set. If there are alternatives in the pattern, they are tried. If all
1956     the alternatives match the empty string, the entire match fails. For
1957 nigel 73 example, if the pattern
1958 nigel 41
1959 nigel 73 a?b?
1960 nigel 41
1961 ph10 292 is applied to a string not beginning with "a" or "b", it matches the
1962     empty string at the start of the subject. With PCRE_NOTEMPTY set, this
1963 nigel 73 match is not valid, so PCRE searches further into the string for occur-
1964     rences of "a" or "b".
1965 nigel 41
1966 nigel 73 Perl has no direct equivalent of PCRE_NOTEMPTY, but it does make a spe-
1967 ph10 292 cial case of a pattern match of the empty string within its split()
1968     function, and when using the /g modifier. It is possible to emulate
1969 nigel 73 Perl's behaviour after matching a null string by first trying the match
1970 nigel 75 again at the same offset with PCRE_NOTEMPTY and PCRE_ANCHORED, and then
1971 ph10 292 if that fails by advancing the starting offset (see below) and trying
1972 nigel 75 an ordinary match again. There is some code that demonstrates how to do
1973     this in the pcredemo.c sample program.
1974 nigel 41
1975 nigel 75 PCRE_NO_UTF8_CHECK
1976    
1977     When PCRE_UTF8 is set at compile time, the validity of the subject as a
1978 ph10 292 UTF-8 string is automatically checked when pcre_exec() is subsequently
1979     called. The value of startoffset is also checked to ensure that it
1980     points to the start of a UTF-8 character. There is a discussion about
1981     the validity of UTF-8 strings in the section on UTF-8 support in the
1982     main pcre page. If an invalid UTF-8 sequence of bytes is found,
1983     pcre_exec() returns the error PCRE_ERROR_BADUTF8. If startoffset con-
1984 ph10 211 tains an invalid value, PCRE_ERROR_BADUTF8_OFFSET is returned.
1985 nigel 75
1986 ph10 292 If you already know that your subject is valid, and you want to skip
1987     these checks for performance reasons, you can set the
1988     PCRE_NO_UTF8_CHECK option when calling pcre_exec(). You might want to
1989     do this for the second and subsequent calls to pcre_exec() if you are
1990     making repeated calls to find all the matches in a single subject
1991     string. However, you should be sure that the value of startoffset
1992     points to the start of a UTF-8 character. When PCRE_NO_UTF8_CHECK is
1993     set, the effect of passing an invalid UTF-8 string as a subject, or a
1994     value of startoffset that does not point to the start of a UTF-8 char-
1995 nigel 75 acter, is undefined. Your program may crash.
1996    
1997     PCRE_PARTIAL
1998    
1999 ph10 292 This option turns on the partial matching feature. If the subject
2000     string fails to match the pattern, but at some point during the match-
2001     ing process the end of the subject was reached (that is, the subject
2002     partially matches the pattern and the failure to match occurred only
2003     because there were not enough subject characters), pcre_exec() returns
2004     PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH. When PCRE_PARTIAL is
2005     used, there are restrictions on what may appear in the pattern. These
2006 nigel 75 are discussed in the pcrepartial documentation.
2007    
2008     The string to be matched by pcre_exec()
2009    
2010 ph10 292 The subject string is passed to pcre_exec() as a pointer in subject, a
2011     length in length, and a starting byte offset in startoffset. In UTF-8
2012     mode, the byte offset must point to the start of a UTF-8 character.
2013     Unlike the pattern string, the subject may contain binary zero bytes.
2014     When the starting offset is zero, the search for a match starts at the
2015 nigel 75 beginning of the subject, and this is by far the most common case.
2016 nigel 63
2017 ph10 292 A non-zero starting offset is useful when searching for another match
2018     in the same subject by calling pcre_exec() again after a previous suc-
2019     cess. Setting startoffset differs from just passing over a shortened
2020     string and setting PCRE_NOTBOL in the case of a pattern that begins
2021 nigel 73 with any kind of lookbehind. For example, consider the pattern
2022 nigel 41
2023 nigel 73 \Biss\B
2024 nigel 41
2025 ph10 292 which finds occurrences of "iss" in the middle of words. (\B matches
2026     only if the current position in the subject is not a word boundary.)
2027     When applied to the string "Mississipi" the first call to pcre_exec()
2028     finds the first occurrence. If pcre_exec() is called again with just
2029     the remainder of the subject, namely "issipi", it does not match,
2030 nigel 73 because \B is always false at the start of the subject, which is deemed
2031 ph10 292 to be a word boundary. However, if pcre_exec() is passed the entire
2032 nigel 75 string again, but with startoffset set to 4, it finds the second occur-
2033 ph10 292 rence of "iss" because it is able to look behind the starting point to
2034 nigel 75 discover that it is preceded by a letter.
2035 nigel 41
2036 ph10 292 If a non-zero starting offset is passed when the pattern is anchored,
2037 nigel 75 one attempt to match at the given offset is made. This can only succeed
2038 ph10 292 if the pattern does not require the match to be at the start of the
2039 nigel 75 subject.
2040 nigel 41
2041 nigel 75 How pcre_exec() returns captured substrings
2042    
2043 ph10 292 In general, a pattern matches a certain portion of the subject, and in
2044     addition, further substrings from the subject may be picked out by
2045     parts of the pattern. Following the usage in Jeffrey Friedl's book,
2046     this is called "capturing" in what follows, and the phrase "capturing
2047     subpattern" is used for a fragment of a pattern that picks out a sub-
2048     string. PCRE supports several other kinds of parenthesized subpattern
2049 nigel 73 that do not cause substrings to be captured.
2050 nigel 65
2051 ph10 292 Captured substrings are returned to the caller via a vector of integer
2052     offsets whose address is passed in ovector. The number of elements in
2053     the vector is passed in ovecsize, which must be a non-negative number.
2054 nigel 75 Note: this argument is NOT the size of ovector in bytes.
2055 nigel 41
2056 ph10 292 The first two-thirds of the vector is used to pass back captured sub-
2057     strings, each substring using a pair of integers. The remaining third
2058     of the vector is used as workspace by pcre_exec() while matching cap-
2059     turing subpatterns, and is not available for passing back information.
2060     The length passed in ovecsize should always be a multiple of three. If
2061 nigel 75 it is not, it is rounded down.
2062    
2063 ph10 292 When a match is successful, information about captured substrings is
2064     returned in pairs of integers, starting at the beginning of ovector,
2065     and continuing up to two-thirds of its length at the most. The first
2066 nigel 73 element of a pair is set to the offset of the first character in a sub-
2067 ph10 292 string, and the second is set to the offset of the first character
2068     after the end of a substring. The first pair, ovector[0] and ovec-
2069     tor[1], identify the portion of the subject string matched by the
2070     entire pattern. The next pair is used for the first capturing subpat-
2071 nigel 91 tern, and so on. The value returned by pcre_exec() is one more than the
2072     highest numbered pair that has been set. For example, if two substrings
2073 ph10 292 have been captured, the returned value is 3. If there are no capturing
2074     subpatterns, the return value from a successful match is 1, indicating
2075 nigel 91 that just the first pair of offsets has been set.
2076 nigel 41
2077 nigel 73 If a capturing subpattern is matched repeatedly, it is the last portion
2078 nigel 75 of the string that it matched that is returned.
2079 nigel 41
2080 ph10 292 If the vector is too small to hold all the captured substring offsets,
2081 nigel 75 it is used as far as possible (up to two-thirds of its length), and the
2082 ph10 292 function returns a value of zero. In particular, if the substring off-
2083 nigel 75 sets are not of interest, pcre_exec() may be called with ovector passed
2084 ph10 292 as NULL and ovecsize as zero. However, if the pattern contains back
2085     references and the ovector is not big enough to remember the related
2086     substrings, PCRE has to get additional memory for use during matching.
2087 nigel 73 Thus it is usually advisable to supply an ovector.
2088 nigel 41
2089 ph10 292 The pcre_info() function can be used to find out how many capturing
2090     subpatterns there are in a compiled pattern. The smallest size for
2091     ovector that will allow for n captured substrings, in addition to the
2092 nigel 91 offsets of the substring matched by the whole pattern, is (n+1)*3.
2093 nigel 41
2094 ph10 292 It is possible for capturing subpattern number n+1 to match some part
2095 nigel 91 of the subject when subpattern n has not been used at all. For example,
2096 ph10 292 if the string "abc" is matched against the pattern (a|(z))(bc) the
2097 nigel 91 return from the function is 4, and subpatterns 1 and 3 are matched, but
2098 ph10 292 2 is not. When this happens, both values in the offset pairs corre-
2099 nigel 91 sponding to unused subpatterns are set to -1.
2100 nigel 75
2101 ph10 292 Offset values that correspond to unused subpatterns at the end of the
2102     expression are also set to -1. For example, if the string "abc" is
2103     matched against the pattern (abc)(x(yz)?)? subpatterns 2 and 3 are not
2104     matched. The return from the function is 2, because the highest used
2105 nigel 91 capturing subpattern number is 1. However, you can refer to the offsets
2106 ph10 292 for the second and third capturing subpatterns if you wish (assuming
2107 nigel 91 the vector is large enough, of course).
2108    
2109 ph10 292 Some convenience functions are provided for extracting the captured
2110 nigel 91 substrings as separate strings. These are described below.
2111    
2112     Error return values from pcre_exec()
2113    
2114 ph10 292 If pcre_exec() fails, it returns a negative number. The following are
2115 nigel 73 defined in the header file:
2116 nigel 41
2117 nigel 73 PCRE_ERROR_NOMATCH (-1)
2118 nigel 41
2119 nigel 73 The subject string did not match the pattern.
2120 nigel 41
2121 nigel 73 PCRE_ERROR_NULL (-2)
2122 nigel 41
2123 ph10 292 Either code or subject was passed as NULL, or ovector was NULL and
2124 nigel 73 ovecsize was not zero.
2125 nigel 41
2126 nigel 73 PCRE_ERROR_BADOPTION (-3)
2127 nigel 41
2128 nigel 73 An unrecognized bit was set in the options argument.
2129 nigel 41
2130 nigel 73 PCRE_ERROR_BADMAGIC (-4)
2131 nigel 41
2132 ph10 292 PCRE stores a 4-byte "magic number" at the start of the compiled code,
2133 nigel 75 to catch the case when it is passed a junk pointer and to detect when a
2134     pattern that was compiled in an environment of one endianness is run in
2135 ph10 292 an environment with the other endianness. This is the error that PCRE
2136 nigel 75 gives when the magic number is not present.
2137 nigel 41
2138 nigel 93 PCRE_ERROR_UNKNOWN_OPCODE (-5)
2139 nigel 41
2140 nigel 73 While running the pattern match, an unknown item was encountered in the
2141 ph10 292 compiled pattern. This error could be caused by a bug in PCRE or by
2142 nigel 73 overwriting of the compiled pattern.
2143 nigel 41
2144 nigel 73 PCRE_ERROR_NOMEMORY (-6)
2145 nigel 41
2146 ph10 292 If a pattern contains back references, but the ovector that is passed
2147 nigel 73 to pcre_exec() is not big enough to remember the referenced substrings,
2148 ph10 292 PCRE gets a block of memory at the start of matching to use for this
2149     purpose. If the call via pcre_malloc() fails, this error is given. The
2150 nigel 75 memory is automatically freed at the end of matching.
2151 nigel 41
2152 nigel 73 PCRE_ERROR_NOSUBSTRING (-7)
2153 nigel 53
2154 ph10 292 This error is used by the pcre_copy_substring(), pcre_get_substring(),
2155 nigel 73 and pcre_get_substring_list() functions (see below). It is never
2156     returned by pcre_exec().
2157 nigel 63
2158 nigel 73 PCRE_ERROR_MATCHLIMIT (-8)
2159 nigel 63
2160 ph10 292 The backtracking limit, as specified by the match_limit field in a
2161     pcre_extra structure (or defaulted) was reached. See the description
2162 nigel 87 above.
2163    
2164 nigel 73 PCRE_ERROR_CALLOUT (-9)
2165 nigel 63
2166 nigel 73 This error is never generated by pcre_exec() itself. It is provided for
2167 ph10 292 use by callout functions that want to yield a distinctive error code.
2168 nigel 73 See the pcrecallout documentation for details.
2169 nigel 71
2170 nigel 73 PCRE_ERROR_BADUTF8 (-10)
2171 nigel 71
2172 ph10 292 A string that contains an invalid UTF-8 byte sequence was passed as a
2173 nigel 73 subject.
2174    
2175     PCRE_ERROR_BADUTF8_OFFSET (-11)
2176    
2177     The UTF-8 byte sequence that was passed as a subject was valid, but the
2178 ph10 292 value of startoffset did not point to the beginning of a UTF-8 charac-
2179 nigel 73 ter.
2180    
2181 nigel 77 PCRE_ERROR_PARTIAL (-12)
2182 nigel 73
2183 ph10 292 The subject string did not match, but it did match partially. See the
2184 nigel 75 pcrepartial documentation for details of partial matching.
2185    
2186 nigel 77 PCRE_ERROR_BADPARTIAL (-13)
2187 nigel 75
2188 ph10 292 The PCRE_PARTIAL option was used with a compiled pattern containing
2189     items that are not supported for partial matching. See the pcrepartial
2190 nigel 75 documentation for details of partial matching.
2191    
2192 nigel 77 PCRE_ERROR_INTERNAL (-14)
2193 nigel 75
2194 ph10 292 An unexpected internal error has occurred. This error could be caused
2195 nigel 75 by a bug in PCRE or by overwriting of the compiled pattern.
2196    
2197 nigel 77 PCRE_ERROR_BADCOUNT (-15)
2198 nigel 75
2199 ph10 292 This error is given if the value of the ovecsize argument is negative.
2200 nigel 75
2201 nigel 93 PCRE_ERROR_RECURSIONLIMIT (-21)
2202 nigel 75
2203 nigel 93 The internal recursion limit, as specified by the match_limit_recursion
2204 ph10 292 field in a pcre_extra structure (or defaulted) was reached. See the
2205 nigel 93 description above.
2206    
2207     PCRE_ERROR_BADNEWLINE (-23)
2208    
2209     An invalid combination of PCRE_NEWLINE_xxx options was given.
2210    
2211 ph10 197 Error numbers -16 to -20 and -22 are not used by pcre_exec().
2212 nigel 93
2213    
2214 nigel 63 EXTRACTING CAPTURED SUBSTRINGS BY NUMBER
2215    
2216 nigel 73 int pcre_copy_substring(const char *subject, int *ovector,
2217     int stringcount, int stringnumber, char *buffer,
2218     int buffersize);
2219 nigel 63
2220 nigel 73 int pcre_get_substring(const char *subject, int *ovector,
2221     int stringcount, int stringnumber,
2222     const char **stringptr);
2223 nigel 63
2224 nigel 73 int pcre_get_substring_list(const char *subject,
2225     int *ovector, int stringcount, const char ***listptr);
2226 nigel 63
2227 ph10 292 Captured substrings can be accessed directly by using the offsets
2228     returned by pcre_exec() in ovector. For convenience, the functions
2229 nigel 73 pcre_copy_substring(), pcre_get_substring(), and pcre_get_sub-
2230 ph10 292 string_list() are provided for extracting captured substrings as new,
2231     separate, zero-terminated strings. These functions identify substrings
2232     by number. The next section describes functions for extracting named
2233 nigel 91 substrings.
2234 nigel 41
2235 ph10 292 A substring that contains a binary zero is correctly extracted and has
2236     a further zero added on the end, but the result is not, of course, a C
2237     string. However, you can process such a string by referring to the
2238     length that is returned by pcre_copy_substring() and pcre_get_sub-
2239 nigel 91 string(). Unfortunately, the interface to pcre_get_substring_list() is
2240 ph10 292 not adequate for handling strings containing binary zeros, because the
2241 nigel 91 end of the final string is not independently indicated.
2242    
2243 ph10 292 The first three arguments are the same for all three of these func-
2244     tions: subject is the subject string that has just been successfully
2245 nigel 73 matched, ovector is a pointer to the vector of integer offsets that was
2246     passed to pcre_exec(), and stringcount is the number of substrings that
2247 ph10 292 were captured by the match, including the substring that matched the
2248 nigel 75 entire regular expression. This is the value returned by pcre_exec() if
2249 ph10 292 it is greater than zero. If pcre_exec() returned zero, indicating that
2250     it ran out of space in ovector, the value passed as stringcount should
2251 nigel 75 be the number of elements in the vector divided by three.
2252 nigel 41
2253 ph10 292 The functions pcre_copy_substring() and pcre_get_substring() extract a
2254     single substring, whose number is given as stringnumber. A value of
2255     zero extracts the substring that matched the entire pattern, whereas
2256     higher values extract the captured substrings. For pcre_copy_sub-
2257     string(), the string is placed in buffer, whose length is given by
2258     buffersize, while for pcre_get_substring() a new block of memory is
2259     obtained via pcre_malloc, and its address is returned via stringptr.
2260     The yield of the function is the length of the string, not including
2261 nigel 93 the terminating zero, or one of these error codes:
2262 nigel 41
2263 nigel 73 PCRE_ERROR_NOMEMORY (-6)
2264 nigel 41
2265 ph10 292 The buffer was too small for pcre_copy_substring(), or the attempt to
2266 nigel 73 get memory failed for pcre_get_substring().
2267 nigel 41
2268 nigel 73 PCRE_ERROR_NOSUBSTRING (-7)
2269 nigel 41
2270 nigel 73 There is no substring whose number is stringnumber.
2271 nigel 41
2272 ph10 292 The pcre_get_substring_list() function extracts all available sub-
2273     strings and builds a list of pointers to them. All this is done in a
2274 nigel 75 single block of memory that is obtained via pcre_malloc. The address of
2275 ph10 292 the memory block is returned via listptr, which is also the start of
2276     the list of string pointers. The end of the list is marked by a NULL
2277     pointer. The yield of the function is zero if all went well, or the
2278 nigel 93 error code
2279 nigel 41
2280 nigel 73 PCRE_ERROR_NOMEMORY (-6)
2281 nigel 41
2282 nigel 73 if the attempt to get the memory block failed.
2283 nigel 41
2284 ph10 292 When any of these functions encounter a substring that is unset, which
2285     can happen when capturing subpattern number n+1 matches some part of
2286     the subject, but subpattern n has not been used at all, they return an
2287 nigel 73 empty string. This can be distinguished from a genuine zero-length sub-
2288 ph10 292 string by inspecting the appropriate offset in ovector, which is nega-
2289 nigel 73 tive for unset substrings.
2290 nigel 41
2291 ph10 292 The two convenience functions pcre_free_substring() and pcre_free_sub-
2292     string_list() can be used to free the memory returned by a previous
2293 nigel 75 call of pcre_get_substring() or pcre_get_substring_list(), respec-
2294 ph10 292 tively. They do nothing more than call the function pointed to by
2295     pcre_free, which of course could be called directly from a C program.
2296     However, PCRE is used in some situations where it is linked via a spe-
2297     cial interface to another programming language that cannot use
2298     pcre_free directly; it is for these cases that the functions are pro-
2299 nigel 77 vided.
2300 nigel 41
2301 nigel 73
2302 nigel 63 EXTRACTING CAPTURED SUBSTRINGS BY NAME
2303 nigel 41
2304 nigel 75 int pcre_get_stringnumber(const pcre *code,
2305     const char *name);
2306    
2307 nigel 73 int pcre_copy_named_substring(const pcre *code,
2308     const char *subject, int *ovector,
2309     int stringcount, const char *stringname,
2310     char *buffer, int buffersize);
2311 nigel 41
2312 nigel 73 int pcre_get_named_substring(const pcre *code,
2313     const char *subject, int *ovector,
2314     int stringcount, const char *stringname,
2315     const char **stringptr);
2316 nigel 41
2317 ph10 292 To extract a substring by name, you first have to find associated num-
2318 nigel 75 ber. For example, for this pattern
2319 nigel 41
2320 nigel 93 (a+)b(?<xxx>\d+)...
2321 nigel 63
2322 nigel 91 the number of the subpattern called "xxx" is 2. If the name is known to
2323     be unique (PCRE_DUPNAMES was not set), you can find the number from the
2324     name by calling pcre_get_stringnumber(). The first argument is the com-
2325     piled pattern, and the second is the name. The yield of the function is
2326 ph10 292 the subpattern number, or PCRE_ERROR_NOSUBSTRING (-7) if there is no
2327 nigel 91 subpattern of that name.
2328 nigel 63
2329 nigel 75 Given the number, you can extract the substring directly, or use one of
2330     the functions described in the previous section. For convenience, there
2331     are also two functions that do the whole job.
2332    
2333 ph10 292 Most of the arguments of pcre_copy_named_substring() and
2334     pcre_get_named_substring() are the same as those for the similarly
2335     named functions that extract by number. As these are described in the
2336     previous section, they are not re-described here. There are just two
2337 nigel 75 differences:
2338 nigel 63
2339 ph10 292 First, instead of a substring number, a substring name is given. Sec-
2340 nigel 73 ond, there is an extra argument, given at the start, which is a pointer
2341 ph10 292 to the compiled pattern. This is needed in order to gain access to the
2342 nigel 73 name-to-number translation table.
2343 nigel 63
2344 ph10 292 These functions call pcre_get_stringnumber(), and if it succeeds, they
2345     then call pcre_copy_substring() or pcre_get_substring(), as appropri-
2346     ate. NOTE: If PCRE_DUPNAMES is set and there are duplicate names, the
2347 ph10 128 behaviour may not be what you want (see the next section).
2348 nigel 63
2349 nigel 77
2350 nigel 91 DUPLICATE SUBPATTERN NAMES
2351    
2352     int pcre_get_stringtable_entries(const pcre *code,
2353     const char *name, char **first, char **last);
2354    
2355 ph10 292 When a pattern is compiled with the PCRE_DUPNAMES option, names for
2356     subpatterns are not required to be unique. Normally, patterns with
2357     duplicate names are such that in any one match, only one of the named
2358     subpatterns participates. An example is shown in the pcrepattern docu-
2359 ph10 208 mentation.
2360    
2361 ph10 292 When duplicates are present, pcre_copy_named_substring() and
2362     pcre_get_named_substring() return the first substring corresponding to
2363     the given name that is set. If none are set, PCRE_ERROR_NOSUBSTRING
2364     (-7) is returned; no data is returned. The pcre_get_stringnumber()
2365     function returns one of the numbers that are associated with the name,
2366 ph10 208 but it is not defined which it is.
2367 nigel 91
2368 ph10 292 If you want to get full details of all captured substrings for a given
2369     name, you must use the pcre_get_stringtable_entries() function. The
2370 nigel 91 first argument is the compiled pattern, and the second is the name. The
2371 ph10 292 third and fourth are pointers to variables which are updated by the
2372 nigel 91 function. After it has run, they point to the first and last entries in
2373 ph10 292 the name-to-number table for the given name. The function itself
2374     returns the length of each entry, or PCRE_ERROR_NOSUBSTRING (-7) if
2375     there are none. The format of the table is described above in the sec-
2376     tion entitled Information about a pattern. Given all the relevant
2377     entries for the name, you can extract each of their numbers, and hence
2378 nigel 93 the captured data, if any.
2379 nigel 91
2380    
2381 nigel 77 FINDING ALL POSSIBLE MATCHES
2382    
2383 ph10 292 The traditional matching function uses a similar algorithm to Perl,
2384 nigel 77 which stops when it finds the first match, starting at a given point in
2385 ph10 292 the subject. If you want to find all possible matches, or the longest
2386     possible match, consider using the alternative matching function (see
2387     below) instead. If you cannot use the alternative function, but still
2388     need to find all possible matches, you can kludge it up by making use
2389 nigel 77 of the callout facility, which is described in the pcrecallout documen-
2390     tation.
2391    
2392     What you have to do is to insert a callout right at the end of the pat-
2393 ph10 292 tern. When your callout function is called, extract and save the cur-
2394     rent matched substring. Then return 1, which forces pcre_exec() to
2395     backtrack and try other alternatives. Ultimately, when it runs out of
2396 nigel 77 matches, pcre_exec() will yield PCRE_ERROR_NOMATCH.
2397    
2398    
2399     MATCHING A PATTERN: THE ALTERNATIVE FUNCTION
2400    
2401     int pcre_dfa_exec(const pcre *code, const pcre_extra *extra,
2402     const char *subject, int length, int startoffset,
2403     int options, int *ovector, int ovecsize,
2404     int *workspace, int wscount);
2405    
2406 ph10 292 The function pcre_dfa_exec() is called to match a subject string
2407     against a compiled pattern, using a matching algorithm that scans the
2408     subject string just once, and does not backtrack. This has different
2409     characteristics to the normal algorithm, and is not compatible with
2410     Perl. Some of the features of PCRE patterns are not supported. Never-
2411     theless, there are times when this kind of matching can be useful. For
2412 nigel 93 a discussion of the two matching algorithms, see the pcrematching docu-
2413     mentation.
2414 nigel 77
2415 ph10 292 The arguments for the pcre_dfa_exec() function are the same as for
2416 nigel 77 pcre_exec(), plus two extras. The ovector argument is used in a differ-
2417 ph10 292 ent way, and this is described below. The other common arguments are
2418     used in the same way as for pcre_exec(), so their description is not
2419 nigel 77 repeated here.
2420    
2421 ph10 292 The two additional arguments provide workspace for the function. The
2422     workspace vector should contain at least 20 elements. It is used for
2423 nigel 77 keeping track of multiple paths through the pattern tree. More
2424 ph10 292 workspace will be needed for patterns and subjects where there are a
2425 nigel 91 lot of potential matches.
2426 nigel 77
2427 nigel 87 Here is an example of a simple call to pcre_dfa_exec():
2428 nigel 77
2429     int rc;
2430     int ovector[10];
2431     int wspace[20];
2432 nigel 87 rc = pcre_dfa_exec(
2433 nigel 77 re, /* result of pcre_compile() */
2434     NULL, /* we didn't study the pattern */
2435     "some string", /* the subject string */
2436     11, /* the length of the subject string */
2437     0, /* start at offset 0 in the subject */
2438     0, /* default options */
2439     ovector, /* vector of integers for substring information */
2440     10, /* number of elements (NOT size in bytes) */
2441     wspace, /* working space vector */
2442     20); /* number of elements (NOT size in bytes) */
2443    
2444     Option bits for pcre_dfa_exec()
2445    
2446 ph10 292 The unused bits of the options argument for pcre_dfa_exec() must be
2447     zero. The only bits that may be set are PCRE_ANCHORED, PCRE_NEW-
2448     LINE_xxx, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK,
2449 nigel 91 PCRE_PARTIAL, PCRE_DFA_SHORTEST, and PCRE_DFA_RESTART. All but the last
2450     three of these are the same as for pcre_exec(), so their description is
2451     not repeated here.
2452 nigel 77
2453     PCRE_PARTIAL
2454    
2455 ph10 292 This has the same general effect as it does for pcre_exec(), but the
2456     details are slightly different. When PCRE_PARTIAL is set for
2457     pcre_dfa_exec(), the return code PCRE_ERROR_NOMATCH is converted into
2458     PCRE_ERROR_PARTIAL if the end of the subject is reached, there have
2459 nigel 77 been no complete matches, but there is still at least one matching pos-
2460 ph10 292 sibility. The portion of the string that provided the partial match is
2461 nigel 77 set as the first matching string.
2462    
2463     PCRE_DFA_SHORTEST
2464    
2465 ph10 292 Setting the PCRE_DFA_SHORTEST option causes the matching algorithm to
2466 nigel 93 stop as soon as it has found one match. Because of the way the alterna-
2467 ph10 292 tive algorithm works, this is necessarily the shortest possible match
2468 nigel 93 at the first possible matching point in the subject string.
2469 nigel 77
2470     PCRE_DFA_RESTART
2471    
2472 ph10 292 When pcre_dfa_exec() is called with the PCRE_PARTIAL option, and
2473     returns a partial match, it is possible to call it again, with addi-
2474     tional subject characters, and have it continue with the same match.
2475     The PCRE_DFA_RESTART option requests this action; when it is set, the
2476     workspace and wscount options must reference the same vector as before
2477     because data about the match so far is left in them after a partial
2478     match. There is more discussion of this facility in the pcrepartial
2479 nigel 77 documentation.
2480    
2481     Successful returns from pcre_dfa_exec()
2482    
2483 ph10 292 When pcre_dfa_exec() succeeds, it may have matched more than one sub-
2484 nigel 77 string in the subject. Note, however, that all the matches from one run
2485 ph10 292 of the function start at the same point in the subject. The shorter
2486     matches are all initial substrings of the longer matches. For example,
2487 nigel 77 if the pattern
2488    
2489     <.*>
2490    
2491     is matched against the string
2492    
2493     This is <something> <something else> <something further> no more
2494    
2495     the three matched strings are
2496    
2497     <something>
2498     <something> <something else>
2499     <something> <something else> <something further>
2500    
2501 ph10 292 On success, the yield of the function is a number greater than zero,
2502     which is the number of matched substrings. The substrings themselves
2503     are returned in ovector. Each string uses two elements; the first is
2504     the offset to the start, and the second is the offset to the end. In
2505     fact, all the strings have the same start offset. (Space could have
2506     been saved by giving this only once, but it was decided to retain some
2507     compatibility with the way pcre_exec() returns data, even though the
2508 nigel 93 meaning of the strings is different.)
2509 nigel 77
2510     The strings are returned in reverse order of length; that is, the long-
2511 ph10 292 est matching string is given first. If there were too many matches to
2512     fit into ovector, the yield of the function is zero, and the vector is
2513 nigel 77 filled with the longest matches.
2514    
2515     Error returns from pcre_dfa_exec()
2516    
2517 ph10 292 The pcre_dfa_exec() function returns a negative number when it fails.
2518     Many of the errors are the same as for pcre_exec(), and these are
2519     described above. There are in addition the following errors that are
2520 nigel 77 specific to pcre_dfa_exec():
2521    
2522     PCRE_ERROR_DFA_UITEM (-16)
2523    
2524 ph10 292 This return is given if pcre_dfa_exec() encounters an item in the pat-
2525     tern that it does not support, for instance, the use of \C or a back
2526 nigel 77 reference.
2527    
2528     PCRE_ERROR_DFA_UCOND (-17)
2529    
2530 ph10 292 This return is given if pcre_dfa_exec() encounters a condition item
2531     that uses a back reference for the condition, or a test for recursion
2532 nigel 93 in a specific group. These are not supported.
2533 nigel 77
2534     PCRE_ERROR_DFA_UMLIMIT (-18)
2535    
2536 ph10 292 This return is given if pcre_dfa_exec() is called with an extra block
2537 nigel 77 that contains a setting of the match_limit field. This is not supported
2538     (it is meaningless).
2539    
2540     PCRE_ERROR_DFA_WSSIZE (-19)
2541    
2542 ph10 292 This return is given if pcre_dfa_exec() runs out of space in the
2543 nigel 77 workspace vector.
2544    
2545     PCRE_ERROR_DFA_RECURSE (-20)
2546    
2547 ph10 292 When a recursive subpattern is processed, the matching function calls
2548     itself recursively, using private vectors for ovector and workspace.
2549     This error is given if the output vector is not large enough. This
2550 nigel 77 should be extremely rare, as a vector of size 1000 is used.
2551    
2552 nigel 93
2553     SEE ALSO
2554    
2555 ph10 292 pcrebuild(3), pcrecallout(3), pcrecpp(3)(3), pcrematching(3), pcrepar-
2556     tial(3), pcreposix(3), pcreprecompile(3), pcresample(3), pcrestack(3).
2557 nigel 93
2558 nigel 63
2559 ph10 99 AUTHOR
2560 nigel 63
2561 ph10 99 Philip Hazel
2562     University Computing Service
2563     Cambridge CB2 3QH, England.
2564    
2565    
2566     REVISION
2567    
2568 ph10 313 Last updated: 23 January 2008
2569     Copyright (c) 1997-2008 University of Cambridge.
2570 ph10 99 ------------------------------------------------------------------------------
2571 ph10 259
2572    
2573 nigel 79 PCRECALLOUT(3) PCRECALLOUT(3)
2574 nigel 63
2575 nigel 79
2576 nigel 73 NAME
2577     PCRE - Perl-compatible regular expressions
2578    
2579 nigel 77
2580 nigel 63 PCRE CALLOUTS
2581    
2582 nigel 73 int (*pcre_callout)(pcre_callout_block *);
2583 nigel 63
2584 nigel 73 PCRE provides a feature called "callout", which is a means of temporar-
2585     ily passing control to the caller of PCRE in the middle of pattern
2586     matching. The caller of PCRE provides an external function by putting
2587     its entry point in the global variable pcre_callout. By default, this
2588     variable contains NULL, which disables all calling out.
2589 nigel 63
2590 nigel 73 Within a regular expression, (?C) indicates the points at which the
2591     external function is to be called. Different callout points can be
2592     identified by putting a number less than 256 after the letter C. The
2593     default value is zero. For example, this pattern has two callout
2594     points:
2595 nigel 63
2596 ph10 155 (?C1)abc(?C2)def
2597 nigel 63
2598 nigel 75 If the PCRE_AUTO_CALLOUT option bit is set when pcre_compile() is
2599     called, PCRE automatically inserts callouts, all with number 255,
2600     before each item in the pattern. For example, if PCRE_AUTO_CALLOUT is
2601     used with the pattern
2602 nigel 63
2603 nigel 75 A(\d{2}|--)
2604    
2605     it is processed as if it were
2606    
2607     (?C255)A(?C255)((?C255)\d{2}(?C255)|(?C255)-(?C255)-(?C255))(?C255)
2608    
2609     Notice that there is a callout before and after each parenthesis and
2610     alternation bar. Automatic callouts can be used for tracking the
2611     progress of pattern matching. The pcretest command has an option that
2612     sets automatic callouts; when it is used, the output indicates how the
2613     pattern is matched. This is useful information when you are trying to
2614     optimize the performance of a particular pattern.
2615    
2616    
2617     MISSING CALLOUTS
2618    
2619     You should be aware that, because of optimizations in the way PCRE
2620     matches patterns, callouts sometimes do not happen. For example, if the
2621     pattern is
2622    
2623     ab(?C4)cd
2624    
2625     PCRE knows that any matching string must contain the letter "d". If the
2626     subject string is "abyz", the lack of "d" means that matching doesn't
2627     ever start, and the callout is never reached. However, with "abyd",
2628     though the result is still no match, the callout is obeyed.
2629    
2630    
2631     THE CALLOUT INTERFACE
2632    
2633     During matching, when PCRE reaches a callout point, the external func-
2634 nigel 77 tion defined by pcre_callout is called (if it is set). This applies to
2635     both the pcre_exec() and the pcre_dfa_exec() matching functions. The
2636     only argument to the callout function is a pointer to a pcre_callout
2637     block. This structure contains the following fields:
2638 nigel 75
2639 nigel 73 int version;
2640     int callout_number;
2641     int *offset_vector;
2642     const char *subject;
2643     int subject_length;
2644     int start_match;
2645     int current_position;
2646     int capture_top;
2647     int capture_last;
2648     void *callout_data;
2649 nigel 75 int pattern_position;
2650     int next_item_length;
2651 nigel 63
2652 nigel 77 The version field is an integer containing the version number of the
2653     block format. The initial version was 0; the current version is 1. The
2654     version number will change again in future if additional fields are
2655 nigel 75 added, but the intention is never to remove any of the existing fields.
2656 nigel 63
2657 nigel 77 The callout_number field contains the number of the callout, as com-
2658     piled into the pattern (that is, the number after ?C for manual call-
2659 nigel 75 outs, and 255 for automatically generated callouts).
2660 nigel 63
2661 nigel 77 The offset_vector field is a pointer to the vector of offsets that was
2662     passed by the caller to pcre_exec() or pcre_dfa_exec(). When
2663     pcre_exec() is used, the contents can be inspected in order to extract
2664     substrings that have been matched so far, in the same way as for
2665     extracting substrings after a match has completed. For pcre_dfa_exec()
2666     this field is not useful.
2667 nigel 63
2668 nigel 75 The subject and subject_length fields contain copies of the values that
2669 nigel 73 were passed to pcre_exec().
2670 nigel 63
2671 ph10 172 The start_match field normally contains the offset within the subject
2672     at which the current match attempt started. However, if the escape
2673     sequence \K has been encountered, this value is changed to reflect the
2674     modified starting point. If the pattern is not anchored, the callout
2675     function may be called several times from the same point in the pattern
2676     for different starting points in the subject.
2677 nigel 63
2678 nigel 77 The current_position field contains the offset within the subject of
2679 nigel 73 the current match pointer.
2680 nigel 63
2681 nigel 77 When the pcre_exec() function is used, the capture_top field contains
2682     one more than the number of the highest numbered captured substring so
2683     far. If no substrings have been captured, the value of capture_top is
2684     one. This is always the case when pcre_dfa_exec() is used, because it
2685     does not support captured substrings.
2686 nigel 63
2687 nigel 77 The capture_last field contains the number of the most recently cap-
2688     tured substring. If no substrings have been captured, its value is -1.
2689     This is always the case when pcre_dfa_exec() is used.
2690 nigel 63
2691 nigel 77 The callout_data field contains a value that is passed to pcre_exec()
2692     or pcre_dfa_exec() specifically so that it can be passed back in call-
2693     outs. It is passed in the pcre_callout field of the pcre_extra data
2694     structure. If no such data was passed, the value of callout_data in a
2695     pcre_callout block is NULL. There is a description of the pcre_extra
2696 nigel 73 structure in the pcreapi documentation.
2697 nigel 63
2698 nigel 77 The pattern_position field is present from version 1 of the pcre_call-
2699 nigel 75 out structure. It contains the offset to the next item to be matched in
2700     the pattern string.
2701 nigel 63
2702 nigel 77 The next_item_length field is present from version 1 of the pcre_call-
2703 nigel 75 out structure. It contains the length of the next item to be matched in
2704 nigel 77 the pattern string. When the callout immediately precedes an alterna-
2705     tion bar, a closing parenthesis, or the end of the pattern, the length
2706     is zero. When the callout precedes an opening parenthesis, the length
2707 nigel 75 is that of the entire subpattern.
2708 nigel 73
2709 nigel 77 The pattern_position and next_item_length fields are intended to help
2710     in distinguishing between different automatic callouts, which all have
2711 nigel 75 the same callout number. However, they are set for all callouts.
2712    
2713    
2714 nigel 63 RETURN VALUES
2715    
2716 nigel 77 The external callout function returns an integer to PCRE. If the value
2717     is zero, matching proceeds as normal. If the value is greater than
2718     zero, matching fails at the current point, but the testing of other
2719     matching possibilities goes ahead, just as if a lookahead assertion had
2720     failed. If the value is less than zero, the match is abandoned, and
2721     pcre_exec() (or pcre_dfa_exec()) returns the negative value.
2722 nigel 63
2723 nigel 77 Negative values should normally be chosen from the set of
2724 nigel 73 PCRE_ERROR_xxx values. In particular, PCRE_ERROR_NOMATCH forces a stan-
2725 nigel 77 dard "no match" failure. The error number PCRE_ERROR_CALLOUT is
2726     reserved for use by callout functions; it will never be used by PCRE
2727 nigel 73 itself.
2728 nigel 63
2729    
2730 ph10 99 AUTHOR
2731 nigel 63
2732 ph10 99 Philip Hazel
2733     University Computing Service
2734     Cambridge CB2 3QH, England.
2735    
2736    
2737     REVISION
2738    
2739 ph10 172 Last updated: 29 May 2007
2740 ph10 99 Copyright (c) 1997-2007 University of Cambridge.
2741     ------------------------------------------------------------------------------
2742 ph10 259
2743    
2744 nigel 79 PCRECOMPAT(3) PCRECOMPAT(3)
2745 nigel 63
2746 nigel 79
2747 nigel 73 NAME
2748     PCRE - Perl-compatible regular expressions
2749    
2750 nigel 77
2751 nigel 75 DIFFERENCES BETWEEN PCRE AND PERL
2752 nigel 41
2753 nigel 73 This document describes the differences in the ways that PCRE and Perl
2754 nigel 93 handle regular expressions. The differences described here are mainly
2755 ph10 182 with respect to Perl 5.8, though PCRE versions 7.0 and later contain
2756     some features that are expected to be in the forthcoming Perl 5.10.
2757 nigel 41
2758 nigel 93 1. PCRE has only a subset of Perl's UTF-8 and Unicode support. Details
2759     of what it does have are given in the section on UTF-8 support in the
2760 nigel 87 main pcre page.
2761 nigel 41
2762 nigel 73 2. PCRE does not allow repeat quantifiers on lookahead assertions. Perl
2763 nigel 93 permits them, but they do not mean what you might think. For example,
2764 nigel 73 (?!a){3} does not assert that the next three characters are not "a". It
2765     just asserts that the next character is not "a" three times.
2766 nigel 41
2767 nigel 93 3. Capturing subpatterns that occur inside negative lookahead asser-
2768     tions are counted, but their entries in the offsets vector are never
2769     set. Perl sets its numerical variables from any such patterns that are
2770 nigel 73 matched before the assertion fails to match something (thereby succeed-
2771 nigel 93 ing), but only if the negative lookahead assertion contains just one
2772 nigel 73 branch.
2773 nigel 41
2774 nigel 93 4. Though binary zero characters are supported in the subject string,
2775 nigel 73 they are not allowed in a pattern string because it is passed as a nor-
2776 nigel 75 mal C string, terminated by zero. The escape sequence \0 can be used in
2777     the pattern to represent a binary zero.
2778 nigel 41
2779 nigel 93 5. The following Perl escape sequences are not supported: \l, \u, \L,
2780 nigel 75 \U, and \N. In fact these are implemented by Perl's general string-han-
2781 nigel 93 dling and are not part of its pattern matching engine. If any of these
2782 nigel 75 are encountered by PCRE, an error is generated.
2783 nigel 41
2784 nigel 93 6. The Perl escape sequences \p, \P, and \X are supported only if PCRE
2785     is built with Unicode character property support. The properties that
2786     can be tested with \p and \P are limited to the general category prop-
2787     erties such as Lu and Nd, script names such as Greek or Han, and the
2788 nigel 87 derived properties Any and L&.
2789 nigel 75
2790     7. PCRE does support the \Q...\E escape for quoting substrings. Charac-
2791 nigel 93 ters in between are treated as literals. This is slightly different
2792     from Perl in that $ and @ are also handled as literals inside the
2793     quotes. In Perl, they cause variable interpolation (but of course PCRE
2794 nigel 73 does not have variables). Note the following examples:
2795 nigel 49
2796 nigel 73 Pattern PCRE matches Perl matches
2797 nigel 41
2798 nigel 73 \Qabc$xyz\E abc$xyz abc followed by the
2799     contents of $xyz
2800     \Qabc\$xyz\E abc\$xyz abc\$xyz
2801     \Qabc\E\$\Qxyz\E abc$xyz abc$xyz
2802 nigel 41
2803 nigel 93 The \Q...\E sequence is recognized both inside and outside character
2804 nigel 73 classes.
2805 nigel 41
2806 nigel 93 8. Fairly obviously, PCRE does not support the (?{code}) and (??{code})
2807     constructions. However, there is support for recursive patterns. This
2808     is not available in Perl 5.8, but will be in Perl 5.10. Also, the PCRE
2809     "callout" feature allows an external function to be called during pat-
2810 nigel 75 tern matching. See the pcrecallout documentation for details.
2811 nigel 63
2812 nigel 93 9. Subpatterns that are called recursively or as "subroutines" are
2813     always treated as atomic groups in PCRE. This is like Python, but
2814     unlike Perl.
2815    
2816     10. There are some differences that are concerned with the settings of
2817     captured strings when part of a pattern is repeated. For example,
2818     matching "aba" against the pattern /^(a(b)?)+$/ in Perl leaves $2
2819 nigel 73 unset, but in PCRE it is set to "b".
2820 nigel 41
2821 ph10 211 11. PCRE does support Perl 5.10's backtracking verbs (*ACCEPT),
2822     (*FAIL), (*F), (*COMMIT), (*PRUNE), (*SKIP), and (*THEN), but only in
2823     the forms without an argument. PCRE does not support (*MARK). If
2824     (*ACCEPT) is within capturing parentheses, PCRE does not set that cap-
2825     ture group; this is different to Perl.
2826    
2827     12. PCRE provides some extensions to the Perl regular expression facil-
2828 nigel 93 ities. Perl 5.10 will include new features that are not in earlier
2829     versions, some of which (such as named parentheses) have been in PCRE
2830     for some time. This list is with respect to Perl 5.10:
2831 nigel 41
2832 nigel 93 (a) Although lookbehind assertions must match fixed length strings,
2833 nigel 73 each alternative branch of a lookbehind assertion can match a different
2834     length of string. Perl requires them all to have the same length.
2835 nigel 41
2836 nigel 93 (b) If PCRE_DOLLAR_ENDONLY is set and PCRE_MULTILINE is not set, the $
2837 nigel 73 meta-character matches only at the very end of the string.
2838 nigel 41
2839 nigel 73 (c) If PCRE_EXTRA is set, a backslash followed by a letter with no spe-
2840 ph10 182 cial meaning is faulted. Otherwise, like Perl, the backslash is quietly
2841     ignored. (Perl can be made to issue a warning.)
2842 nigel 41
2843 nigel 93 (d) If PCRE_UNGREEDY is set, the greediness of the repetition quanti-
2844 nigel 73 fiers is inverted, that is, by default they are not greedy, but if fol-
2845     lowed by a question mark they are.
2846 nigel 41
2847 nigel 75 (e) PCRE_ANCHORED can be used at matching time to force a pattern to be
2848     tried only at the first matching position in the subject string.
2849 nigel 41
2850 nigel 93 (f) The PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and PCRE_NO_AUTO_CAP-
2851 nigel 73 TURE options for pcre_exec() have no Perl equivalents.
2852 nigel 41
2853 ph10 231 (g) The \R escape sequence can be restricted to match only CR, LF, or
2854     CRLF by the PCRE_BSR_ANYCRLF option.
2855 nigel 41
2856 ph10 231 (h) The callout facility is PCRE-specific.
2857 nigel 43
2858 ph10 231 (i) The partial matching facility is PCRE-specific.
2859    
2860     (j) Patterns compiled by PCRE can be saved and re-used at a later time,
2861 nigel 75 even on different hosts that have the other endianness.
2862    
2863 ph10 231 (k) The alternative matching function (pcre_dfa_exec()) matches in a
2864 nigel 77 different way and is not Perl-compatible.
2865    
2866 ph10 231 (l) PCRE recognizes some special sequences such as (*CR) at the start
2867     of a pattern that set overall options that cannot be changed within the
2868     pattern.
2869 nigel 63
2870 ph10 231
2871 ph10 99 AUTHOR
2872 nigel 63
2873 ph10 99 Philip Hazel
2874     University Computing Service
2875     Cambridge CB2 3QH, England.
2876    
2877    
2878     REVISION
2879    
2880 ph10 231 Last updated: 11 September 2007
2881 ph10 99 Copyright (c) 1997-2007 University of Cambridge.
2882     ------------------------------------------------------------------------------
2883 ph10 259
2884    
2885 nigel 79 PCREPATTERN(3) PCREPATTERN(3)
2886 nigel 63
2887 nigel 79
2888 nigel 73 NAME
2889     PCRE - Perl-compatible regular expressions
2890    
2891 nigel 77
2892 nigel 63 PCRE REGULAR EXPRESSION DETAILS
2893    
2894 ph10 208 The syntax and semantics of the regular expressions that are supported
2895     by PCRE are described in detail below. There is a quick-reference syn-
2896     tax summary in the pcresyntax page. Perl's regular expressions are
2897     described in its own documentation, and regular expressions in general
2898     are covered in a number of books, some of which have copious examples.
2899     Jeffrey Friedl's "Mastering Regular Expressions", published by
2900     O'Reilly, covers regular expressions in great detail. This description
2901     of PCRE's regular expressions is intended as reference material.
2902 nigel 49
2903 nigel 75 The original operation of PCRE was on strings of one-byte characters.
2904     However, there is now also support for UTF-8 character strings. To use
2905     this, you must build PCRE to include UTF-8 support, and then call
2906     pcre_compile() with the PCRE_UTF8 option. How this affects pattern
2907     matching is mentioned in several places below. There is also a summary
2908     of UTF-8 features in the section on UTF-8 support in the main pcre
2909     page.
2910 nigel 41
2911 nigel 77 The remainder of this document discusses the patterns that are sup-
2912     ported by PCRE when its main matching function, pcre_exec(), is used.
2913     From release 6.0, PCRE offers a second matching function,
2914     pcre_dfa_exec(), which matches using a different algorithm that is not
2915 ph10 172 Perl-compatible. Some of the features discussed below are not available
2916     when pcre_dfa_exec() is used. The advantages and disadvantages of the
2917     alternative function, and how it differs from the normal function, are
2918     discussed in the pcrematching page.
2919 nigel 77
2920 nigel 93
2921 ph10 227 NEWLINE CONVENTIONS
2922    
2923     PCRE supports five different conventions for indicating line breaks in
2924     strings: a single CR (carriage return) character, a single LF (line-
2925     feed) character, the two-character sequence CRLF, any of the three pre-
2926     ceding, or any Unicode newline sequence. The pcreapi page has further
2927     discussion about newlines, and shows how to set the newline convention
2928     in the options arguments for the compiling and matching functions.
2929    
2930     It is also possible to specify a newline convention by starting a pat-
2931     tern string with one of the following five sequences:
2932    
2933     (*CR) carriage return
2934     (*LF) linefeed
2935     (*CRLF) carriage return, followed by linefeed
2936     (*ANYCRLF) any of the three above
2937     (*ANY) all Unicode newline sequences
2938    
2939     These override the default and the options given to pcre_compile(). For
2940     example, on a Unix system where LF is the default newline sequence, the
2941     pattern
2942    
2943     (*CR)a.b
2944    
2945     changes the convention to CR. That pattern matches "a\nb" because LF is
2946     no longer a newline. Note that these special settings, which are not
2947     Perl-compatible, are recognized only at the very start of a pattern,
2948 ph10 231 and that they must be in upper case. If more than one of them is
2949     present, the last one is used.
2950 ph10 227
2951 ph10 231 The newline convention does not affect what the \R escape sequence
2952     matches. By default, this is any Unicode newline sequence, for Perl
2953     compatibility. However, this can be changed; see the description of \R
2954 ph10 247 in the section entitled "Newline sequences" below. A change of \R set-
2955     ting can be combined with a change of newline convention.
2956 ph10 227
2957 ph10 231
2958 nigel 93 CHARACTERS AND METACHARACTERS
2959    
2960 ph10 247 A regular expression is a pattern that is matched against a subject
2961     string from left to right. Most characters stand for themselves in a
2962     pattern, and match the corresponding characters in the subject. As a
2963 nigel 73 trivial example, the pattern
2964 nigel 41
2965 nigel 73 The quick brown fox
2966 nigel 41
2967 nigel 77 matches a portion of a subject string that is identical to itself. When
2968 ph10 247 caseless matching is specified (the PCRE_CASELESS option), letters are
2969     matched independently of case. In UTF-8 mode, PCRE always understands
2970     the concept of case for characters whose values are less than 128, so
2971     caseless matching is always possible. For characters with higher val-
2972     ues, the concept of case is supported if PCRE is compiled with Unicode
2973     property support, but not otherwise. If you want to use caseless
2974     matching for characters 128 and above, you must ensure that PCRE is
2975 nigel 77 compiled with Unicode property support as well as with UTF-8 support.
2976 nigel 41
2977 ph10 247 The power of regular expressions comes from the ability to include
2978     alternatives and repetitions in the pattern. These are encoded in the
2979 nigel 77 pattern by the use of metacharacters, which do not stand for themselves
2980     but instead are interpreted in some special way.
2981    
2982 ph10 247 There are two different sets of metacharacters: those that are recog-
2983     nized anywhere in the pattern except within square brackets, and those
2984     that are recognized within square brackets. Outside square brackets,
2985 nigel 93 the metacharacters are as follows:
2986 nigel 41
2987 nigel 73 \ general escape character with several uses
2988     ^ assert start of string (or line, in multiline mode)
2989     $ assert end of string (or line, in multiline mode)
2990     . match any character except newline (by default)
2991     [ start character class definition
2992     | start of alternative branch
2993     ( start subpattern
2994     ) end subpattern
2995     ? extends the meaning of (
2996     also 0 or 1 quantifier
2997     also quantifier minimizer
2998     * 0 or more quantifier
2999     + 1 or more quantifier
3000     also "possessive quantifier"
3001     { start min/max quantifier
3002 nigel 41
3003 ph10 247 Part of a pattern that is in square brackets is called a "character
3004 nigel 75 class". In a character class the only metacharacters are:
3005 nigel 41
3006 nigel 73 \ general escape character
3007     ^ negate the class, but only if the first character
3008     - indicates character range
3009     [ POSIX character class (only if followed by POSIX
3010     syntax)
3011     ] terminates the character class
3012 nigel 41
3013 ph10 247 The following sections describe the use of each of the metacharacters.
3014 nigel 41
3015    
3016 nigel 63 BACKSLASH
3017 nigel 41
3018 nigel 73 The backslash character has several uses. Firstly, if it is followed by
3019 ph10 247 a non-alphanumeric character, it takes away any special meaning that
3020     character may have. This use of backslash as an escape character
3021 nigel 73 applies both inside and outside character classes.
3022 nigel 41
3023 ph10 247 For example, if you want to match a * character, you write \* in the
3024     pattern. This escaping action applies whether or not the following
3025     character would otherwise be interpreted as a metacharacter, so it is
3026     always safe to precede a non-alphanumeric with backslash to specify
3027     that it stands for itself. In particular, if you want to match a back-
3028 nigel 75 slash, you write \\.
3029 nigel 41
3030 ph10 247 If a pattern is compiled with the PCRE_EXTENDED option, whitespace in
3031     the pattern (other than in a character class) and characters between a
3032 nigel 91 # outside a character class and the next newline are ignored. An escap-
3033 ph10 247 ing backslash can be used to include a whitespace or # character as
3034 nigel 91 part of the pattern.
3035 nigel 41
3036 ph10 247 If you want to remove the special meaning from a sequence of charac-
3037     ters, you can do so by putting them between \Q and \E. This is differ-
3038     ent from Perl in that $ and @ are handled as literals in \Q...\E
3039     sequences in PCRE, whereas in Perl, $ and @ cause variable interpola-
3040 nigel 73 tion. Note the following examples:
3041 nigel 63
3042 nigel 73 Pattern PCRE matches Perl matches
3043 nigel 63
3044 nigel 73 \Qabc$xyz\E abc$xyz abc followed by the
3045     contents of $xyz
3046     \Qabc\$xyz\E abc\$xyz abc\$xyz
3047     \Qabc\E\$\Qxyz\E abc$xyz abc$xyz
3048 nigel 63
3049 ph10 247 The \Q...\E sequence is recognized both inside and outside character
3050 nigel 73 classes.
3051 nigel 63
3052 nigel 75 Non-printing characters
3053    
3054 nigel 73 A second use of backslash provides a way of encoding non-printing char-
3055 ph10 247 acters in patterns in a visible manner. There is no restriction on the
3056     appearance of non-printing characters, apart from the binary zero that
3057     terminates a pattern, but when a pattern is being prepared by text
3058     editing, it is usually easier to use one of the following escape
3059 nigel 73 sequences than the binary character it represents:
3060 nigel 63
3061 nigel 73 \a alarm, that is, the BEL character (hex 07)
3062     \cx "control-x", where x is any character
3063     \e escape (hex 1B)
3064     \f formfeed (hex 0C)
3065 ph10 227 \n linefeed (hex 0A)
3066 nigel 73 \r carriage return (hex 0D)
3067     \t tab (hex 09)
3068     \ddd character with octal code ddd, or backreference
3069     \xhh character with hex code hh
3070 nigel 87 \x{hhh..} character with hex code hhh..
3071 nigel 41
3072 ph10 247 The precise effect of \cx is as follows: if x is a lower case letter,
3073     it is converted to upper case. Then bit 6 of the character (hex 40) is
3074     inverted. Thus \cz becomes hex 1A, but \c{ becomes hex 3B, while \c;
3075 nigel 73 becomes hex 7B.
3076 nigel 41
3077 ph10 247 After \x, from zero to two hexadecimal digits are read (letters can be
3078     in upper or lower case). Any number of hexadecimal digits may appear
3079     between \x{ and }, but the value of the character code must be less
3080 ph10 211 than 256 in non-UTF-8 mode, and less than 2**31 in UTF-8 mode. That is,
3081 ph10 247 the maximum value in hexadecimal is 7FFFFFFF. Note that this is bigger
3082 ph10 211 than the largest Unicode code point, which is 10FFFF.
3083 nigel 41
3084 ph10 247 If characters other than hexadecimal digits appear between \x{ and },
3085 ph10 211 or if there is no terminating }, this form of escape is not recognized.
3086 ph10 247 Instead, the initial \x will be interpreted as a basic hexadecimal
3087     escape, with no following digits, giving a character whose value is
3088 ph10 211 zero.
3089    
3090 nigel 73 Characters whose value is less than 256 can be defined by either of the
3091 ph10 247 two syntaxes for \x. There is no difference in the way they are han-
3092 nigel 87 dled. For example, \xdc is exactly the same as \x{dc}.
3093 nigel 41
3094 ph10 247 After \0 up to two further octal digits are read. If there are fewer
3095     than two digits, just those that are present are used. Thus the
3096 nigel 91 sequence \0\x\07 specifies two binary zeros followed by a BEL character
3097 ph10 247 (code value 7). Make sure you supply two digits after the initial zero
3098 nigel 91 if the pattern character that follows is itself an octal digit.
3099 nigel 63
3100 nigel 73 The handling of a backslash followed by a digit other than 0 is compli-
3101     cated. Outside a character class, PCRE reads it and any following dig-
3102 ph10 247 its as a decimal number. If the number is less than 10, or if there
3103 nigel 73 have been at least that many previous capturing left parentheses in the
3104 ph10 247 expression, the entire sequence is taken as a back reference. A
3105     description of how this works is given later, following the discussion
3106 nigel 73 of parenthesized subpatterns.
3107 nigel 41
3108 ph10 247 Inside a character class, or if the decimal number is greater than 9
3109     and there have not been that many capturing subpatterns, PCRE re-reads
3110 nigel 93 up to three octal digits following the backslash, and uses them to gen-
3111 ph10 247 erate a data character. Any subsequent digits stand for themselves. In
3112     non-UTF-8 mode, the value of a character specified in octal must be
3113     less than \400. In UTF-8 mode, values up to \777 are permitted. For
3114 nigel 91 example:
3115 nigel 41
3116 nigel 73 \040 is another way of writing a space
3117     \40 is the same, provided there are fewer than 40
3118     previous capturing subpatterns
3119     \7 is always a back reference
3120     \11 might be a back reference, or another way of
3121     writing a tab
3122     \011 is always a tab
3123     \0113 is a tab followed by the character "3"
3124     \113 might be a back reference, otherwise the
3125     character with octal code 113
3126     \377 might be a back reference, otherwise
3127     the byte consisting entirely of 1 bits
3128     \81 is either a back reference, or a binary zero
3129     followed by the two characters "8" and "1"
3130 nigel 41
3131 ph10 247 Note that octal values of 100 or greater must not be introduced by a
3132 nigel 73 leading zero, because no more than three octal digits are ever read.
3133 nigel 41
3134 nigel 91 All the sequences that define a single character value can be used both
3135 ph10 247 inside and outside character classes. In addition, inside a character
3136     class, the sequence \b is interpreted as the backspace character (hex
3137     08), and the sequences \R and \X are interpreted as the characters "R"
3138     and "X", respectively. Outside a character class, these sequences have
3139 nigel 93 different meanings (see below).
3140 nigel 43
3141 nigel 93 Absolute and relative back references
3142    
3143 ph10 247 The sequence \g followed by an unsigned or a negative number, option-
3144     ally enclosed in braces, is an absolute or relative back reference. A
3145 ph10 208 named back reference can be coded as \g{name}. Back references are dis-
3146     cussed later, following the discussion of parenthesized subpatterns.
3147 nigel 93
3148 nigel 75 Generic character types
3149 nigel 41
3150 nigel 93 Another use of backslash is for specifying generic character types. The
3151     following are always recognized:
3152 nigel 75
3153 nigel 73 \d any decimal digit
3154     \D any character that is not a decimal digit
3155 ph10 182 \h any horizontal whitespace character
3156     \H any character that is not a horizontal whitespace character
3157 nigel 73 \s any whitespace character
3158     \S any character that is not a whitespace character
3159 ph10 182 \v any vertical whitespace character
3160     \V any character that is not a vertical whitespace character
3161 nigel 73 \w any "word" character
3162     \W any "non-word" character
3163 nigel 41
3164 nigel 73 Each pair of escape sequences partitions the complete set of characters
3165 ph10 247 into two disjoint sets. Any given character matches one, and only one,
3166 nigel 73 of each pair.
3167 nigel 41
3168 nigel 75 These character type sequences can appear both inside and outside char-
3169 ph10 247 acter classes. They each match one character of the appropriate type.
3170     If the current matching point is at the end of the subject string, all
3171 nigel 75 of them fail, since there is no character to match.
3172 nigel 41
3173 ph10 247 For compatibility with Perl, \s does not match the VT character (code
3174     11). This makes it different from the the POSIX "space" class. The \s
3175     characters are HT (9), LF (10), FF (12), CR (13), and space (32). If
3176 nigel 91 "use locale;" is included in a Perl script, \s may match the VT charac-
3177 ph10 182 ter. In PCRE, it never does.
3178 nigel 63
3179 ph10 247 In UTF-8 mode, characters with values greater than 128 never match \d,
3180 ph10 182 \s, or \w, and always match \D, \S, and \W. This is true even when Uni-
3181 ph10 247 code character property support is available. These sequences retain
3182 ph10 182 their original meanings from before UTF-8 support was available, mainly
3183     for efficiency reasons.
3184    
3185     The sequences \h, \H, \v, and \V are Perl 5.10 features. In contrast to
3186 ph10 247 the other sequences, these do match certain high-valued codepoints in
3187 ph10 182 UTF-8 mode. The horizontal space characters are:
3188    
3189     U+0009 Horizontal tab
3190     U+0020 Space
3191     U+00A0 Non-break space
3192     U+1680 Ogham space mark
3193     U+180E Mongolian vowel separator
3194     U+2000 En quad
3195     U+2001 Em quad
3196     U+2002 En space
3197     U+2003 Em space
3198     U+2004 Three-per-em space
3199     U+2005 Four-per-em space
3200     U+2006 Six-per-em space
3201     U+2007 Figure space
3202     U+2008 Punctuation space
3203     U+2009 Thin space
3204     U+200A Hair space
3205     U+202F Narrow no-break space
3206     U+205F Medium mathematical space
3207     U+3000 Ideographic space
3208    
3209     The vertical space characters are:
3210    
3211     U+000A Linefeed
3212     U+000B Vertical tab
3213     U+000C Formfeed
3214     U+000D Carriage return
3215     U+0085 Next line
3216     U+2028 Line separator
3217     U+2029 Paragraph separator
3218    
3219 nigel 75 A "word" character is an underscore or any character less than 256 that
3220 ph10 247 is a letter or digit. The definition of letters and digits is con-
3221     trolled by PCRE's low-valued character tables, and may vary if locale-
3222     specific matching is taking place (see "Locale support" in the pcreapi
3223     page). For example, in a French locale such as "fr_FR" in Unix-like
3224     systems, or "french" in Windows, some character codes greater than 128
3225     are used for accented letters, and these are matched by \w. The use of
3226 ph10 182 locales with Unicode is discouraged.
3227 nigel 63
3228 nigel 93 Newline sequences
3229    
3230 ph10 247 Outside a character class, by default, the escape sequence \R matches
3231 ph10 231 any Unicode newline sequence. This is a Perl 5.10 feature. In non-UTF-8
3232     mode \R is equivalent to the following:
3233 nigel 93
3234     (?>\r\n|\n|\x0b|\f|\r|\x85)
3235    
3236 ph10 247 This is an example of an "atomic group", details of which are given
3237 nigel 93 below. This particular group matches either the two-character sequence
3238 ph10 247 CR followed by LF, or one of the single characters LF (linefeed,
3239 nigel 93 U+000A), VT (vertical tab, U+000B), FF (formfeed, U+000C), CR (carriage
3240     return, U+000D), or NEL (next line, U+0085). The two-character sequence
3241     is treated as a single unit that cannot be split.
3242    
3243 ph10 247 In UTF-8 mode, two additional characters whose codepoints are greater
3244 nigel 93 than 255 are added: LS (line separator, U+2028) and PS (paragraph sepa-
3245 ph10 247 rator, U+2029). Unicode character property support is not needed for
3246 nigel 93 these characters to be recognized.
3247    
3248 ph10 231 It is possible to restrict \R to match only CR, LF, or CRLF (instead of
3249 ph10 247 the complete set of Unicode line endings) by setting the option
3250 ph10 231 PCRE_BSR_ANYCRLF either at compile time or when the pattern is matched.
3251 ph10 247 (BSR is an abbrevation for "backslash R".) This can be made the default
3252     when PCRE is built; if this is the case, the other behaviour can be
3253     requested via the PCRE_BSR_UNICODE option. It is also possible to
3254     specify these settings by starting a pattern string with one of the
3255     following sequences:
3256 ph10 231
3257     (*BSR_ANYCRLF) CR, LF, or CRLF only
3258     (*BSR_UNICODE) any Unicode newline sequence
3259    
3260     These override the default and the options given to pcre_compile(), but
3261     they can be overridden by options given to pcre_exec(). Note that these
3262     special settings, which are not Perl-compatible, are recognized only at
3263     the very start of a pattern, and that they must be in upper case. If
3264 ph10 247 more than one of them is present, the last one is used. They can be
3265     combined with a change of newline convention, for example, a pattern
3266     can start with:
3267 ph10 231
3268 ph10 247 (*ANY)(*BSR_ANYCRLF)
3269    
3270 nigel 93 Inside a character class, \R matches the letter "R".
3271    
3272 nigel 75 Unicode character properties
3273    
3274     When PCRE is built with Unicode character property support, three addi-
3275 ph10 227 tional escape sequences that match characters with specific properties
3276     are available. When not in UTF-8 mode, these sequences are of course
3277     limited to testing characters whose codepoints are less than 256, but
3278 ph10 185 they do work in this mode. The extra escape sequences are:
3279 nigel 75
3280 nigel 87 \p{xx} a character with the xx property
3281     \P{xx} a character without the xx property
3282     \X an extended Unicode sequence
3283 nigel 75
3284 ph10 227 The property names represented by xx above are limited to the Unicode
3285 nigel 87 script names, the general category properties, and "Any", which matches
3286     any character (including newline). Other properties such as "InMusical-
3287 ph10 227 Symbols" are not currently supported by PCRE. Note that \P{Any} does
3288 nigel 87 not match any characters, so always causes a match failure.
3289 nigel 75
3290 nigel 87 Sets of Unicode characters are defined as belonging to certain scripts.
3291 ph10 227 A character from one of these sets can be matched using a script name.
3292 nigel 87 For example:
3293 nigel 75
3294 nigel 87 \p{Greek}
3295     \P{Han}
3296    
3297 ph10 227 Those that are not part of an identified script are lumped together as
3298 nigel 87 "Common". The current list of scripts is:
3299    
3300 nigel 93 Arabic, Armenian, Balinese, Bengali, Bopomofo, Braille, Buginese,
3301 ph10 227 Buhid, Canadian_Aboriginal, Cherokee, Common, Coptic, Cuneiform,
3302 nigel 93 Cypriot, Cyrillic, Deseret, Devanagari, Ethiopic, Georgian, Glagolitic,
3303 ph10 227 Gothic, Greek, Gujarati, Gurmukhi, Han, Hangul, Hanunoo, Hebrew, Hira-
3304     gana, Inherited, Kannada, Katakana, Kharoshthi, Khmer, Lao, Latin,
3305 nigel 93 Limbu, Linear_B, Malayalam, Mongolian, Myanmar, New_Tai_Lue, Nko,
3306 ph10 227 Ogham, Old_Italic, Old_Persian, Oriya, Osmanya, Phags_Pa, Phoenician,
3307 nigel 93 Runic, Shavian, Sinhala, Syloti_Nagri, Syriac, Tagalog, Tagbanwa,
3308     Tai_Le, Tamil, Telugu, Thaana, Thai, Tibetan, Tifinagh, Ugaritic, Yi.
3309 nigel 87
3310 ph10 227 Each character has exactly one general category property, specified by
3311 nigel 87 a two-letter abbreviation. For compatibility with Perl, negation can be
3312 ph10 227 specified by including a circumflex between the opening brace and the
3313 nigel 87 property name. For example, \p{^Lu} is the same as \P{Lu}.
3314    
3315     If only one letter is specified with \p or \P, it includes all the gen-
3316 ph10 227 eral category properties that start with that letter. In this case, in
3317     the absence of negation, the curly brackets in the escape sequence are
3318 nigel 87 optional; these two examples have the same effect:
3319    
3320 nigel 75 \p{L}
3321     \pL
3322    
3323 nigel 87 The following general category property codes are supported:
3324 nigel 75
3325     C Other
3326     Cc Control
3327     Cf Format
3328     Cn Unassigned
3329     Co Private use
3330     Cs Surrogate
3331    
3332     L Letter
3333     Ll Lower case letter
3334     Lm Modifier letter
3335     Lo Other letter
3336     Lt Title case letter
3337     Lu Upper case letter
3338    
3339     M Mark
3340     Mc Spacing mark
3341     Me Enclosing mark
3342     Mn Non-spacing mark
3343    
3344     N Number
3345     Nd Decimal number
3346     Nl Letter number
3347     No Other number
3348    
3349     P Punctuation
3350     Pc Connector punctuation
3351     Pd Dash punctuation
3352     Pe Close punctuation
3353     Pf Final punctuation
3354     Pi Initial punctuation
3355     Po Other punctuation
3356     Ps Open punctuation
3357    
3358     S Symbol
3359     Sc Currency symbol
3360     Sk Modifier symbol
3361     Sm Mathematical symbol
3362     So Other symbol
3363    
3364     Z Separator
3365     Zl Line separator
3366     Zp Paragraph separator
3367     Zs Space separator
3368    
3369 ph10 227 The special property L& is also supported: it matches a character that
3370     has the Lu, Ll, or Lt property, in other words, a letter that is not
3371 nigel 87 classified as a modifier or "other".
3372 nigel 75
3373 ph10 227 The Cs (Surrogate) property applies only to characters in the range
3374     U+D800 to U+DFFF. Such characters are not valid in UTF-8 strings (see
3375 ph10 211 RFC 3629) and so cannot be tested by PCRE, unless UTF-8 validity check-
3376 ph10 227 ing has been turned off (see the discussion of PCRE_NO_UTF8_CHECK in
3377 ph10 211 the pcreapi page).
3378    
3379 ph10 227 The long synonyms for these properties that Perl supports (such as
3380     \p{Letter}) are not supported by PCRE, nor is it permitted to prefix
3381 nigel 87 any of these properties with "Is".
3382    
3383     No character that is in the Unicode table has the Cn (unassigned) prop-
3384     erty. Instead, this property is assumed for any code point that is not
3385     in the Unicode table.
3386    
3387 ph10 227 Specifying caseless matching does not affect these escape sequences.
3388 nigel 75 For example, \p{Lu} always matches only upper case letters.
3389    
3390 ph10 227 The \X escape matches any number of Unicode characters that form an
3391 nigel 75 extended Unicode sequence. \X is equivalent to